An image depth estimation method, apparatus, electronic device, and storage medium

Abstract

This invention discloses an image depth estimation method, apparatus, electronic device, and storage medium. The method acquires an image of the target device to be estimated during inspection; processes the image to be estimated using a hybrid encoder to obtain hybrid encoded features; the hybrid encoded features are obtained by fusing local convolutional features and global window self-attention features of the image to be estimated; the local convolutional features are obtained by performing depth-separable convolution processing on the image to be estimated; the global window self-attention features are obtained by dynamically dividing the image to be estimated into windows and performing self-attention processing under each window; feature alignment is performed on the hybrid encoded features to obtain a depth feature map of the image to be estimated; the predicted depth value and depth prediction confidence of each pixel are determined based on the depth feature map, thus achieving image depth estimation. This invention improves the real-time performance and accuracy of depth prediction, thereby enhancing inspection safety.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image depth estimation method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the comprehensive advancement of Industry 4.0, digital twins, and intelligent operation and maintenance, critical infrastructure such as oil and gas, chemical industry, power, and rail transportation have placed higher demands on "all-weather, all-element, and all-space" inspections: they must achieve stable 24-hour monitoring in extreme environments such as high temperature, high dust, strong electromagnetic radiation, and confined spaces, while ensuring the real-time, accuracy, and safety redundancy of defect detection.

[0003] Existing inspection terminals are typically deployed on embedded platforms or drones with strictly limited power consumption, resulting in limited processor computing power. Most existing high-precision deep networks rely on deep convolutions or large-scale self-attention operations, making it difficult to achieve millisecond-level response on such devices. Furthermore, existing inspection equipment is susceptible to the influence of complex and extreme environments, and images acquired in the monitoring environment are prone to deformation, leading to inaccurate monitoring results. Existing inspection methods only output predicted depth values ​​and use them directly, resulting in security issues due to the uncertainty of depth prediction accuracy. Summary of the Invention

[0004] This invention provides an image depth estimation method, apparatus, electronic device, and storage medium to improve the real-time performance and accuracy of depth estimation, thereby enhancing inspection safety.

[0005] According to one aspect of the present invention, an image depth estimation method is provided, comprising:

[0006] During the inspection process, acquire the image to be estimated of the target equipment;

[0007] The image to be estimated is processed by a hybrid encoder to obtain hybrid coding features of the image to be estimated; the hybrid coding features are obtained by fusing local convolutional features and global window self-attention features of the image to be estimated; the local convolutional features are obtained by performing depth-separable convolution on the image to be estimated; the global window self-attention features are obtained by dynamically dividing the image to be estimated into windows and performing self-attention processing under each window;

[0008] The hybrid encoded features are aligned to obtain the depth feature map of the image to be estimated;

[0009] Based on the depth feature map, the predicted depth value and depth prediction confidence of each pixel are determined to achieve image depth estimation.

[0010] According to another aspect of the present invention, an image depth estimation apparatus is provided, comprising:

[0011] The image acquisition module is used to acquire the image of the target equipment to be estimated during the inspection process;

[0012] The hybrid coding feature acquisition module is used to process the image to be estimated through a hybrid encoder to obtain the hybrid coding features of the image to be estimated. The hybrid coding features are obtained by fusing the local convolutional features and global window self-attention features of the image to be estimated. The local convolutional features are obtained by performing depth-separable convolution processing on the image to be estimated. The global window self-attention features are obtained by dynamically dividing the image to be estimated into windows and performing self-attention processing under each window.

[0013] The depth feature map acquisition module is used to perform feature alignment on the hybrid coded features to obtain the depth feature map of the image to be estimated;

[0014] The depth prediction module is used to determine the predicted depth value and depth prediction confidence of each pixel based on the depth feature map, thereby realizing image depth estimation.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the image depth estimation method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the image depth estimation method according to any embodiment of the present invention.

[0020] The technical solution of this invention acquires an image of the target device to be estimated during inspection; processes the image to be estimated using a hybrid encoder to obtain hybrid encoded features; the hybrid encoded features are obtained by fusing local convolutional features and global window self-attention features of the image to be estimated; the local convolutional features are obtained by performing depth-separable convolution on the image to be estimated; the global window self-attention features are obtained by dynamically dividing the image to be estimated into windows and performing self-attention processing under each window; the hybrid encoded features are aligned to obtain a depth feature map of the image to be estimated; the predicted depth value and depth prediction confidence of each pixel are determined based on the depth feature map to achieve image depth estimation. This solution employs a hybrid approach of local convolution and dynamic window self-attention, deformable convolution for feature alignment, and depth-confidence dual-branch output. This addresses the problems of poor real-time performance, inaccurate monitoring results, and security issues caused by unclear depth prediction accuracy in existing technologies, improving the real-time performance and accuracy of depth prediction, thereby enhancing inspection safety.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1a This is a flowchart of an image depth estimation method provided in Embodiment 1 of the present invention;

[0024] Figure 1b This is a schematic diagram of the structure of a hybrid encoder module provided in Embodiment 1 of the present invention;

[0025] Figure 1c This is a schematic diagram of the structure of an image monocular depth estimation system provided in this embodiment;

[0026] Figure 2 This is a schematic diagram of the structure of an image depth estimation device provided in Embodiment 2 of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the image depth estimation method of this invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1a This is a flowchart of an image depth estimation method provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation of estimating the depth value of industrial equipment images during inspection. The method can be executed by an image depth estimation device, which can be implemented in hardware and / or software, and can be configured in the processor of an industrial camera. Figure 1a As shown, the method includes:

[0032] S110. During the inspection process, acquire the image to be estimated of the target equipment.

[0033] The target equipment can refer to industrial equipment that requires safety inspection. The image to be estimated can be the original image of the target equipment captured by an industrial camera.

[0034] Optionally, after obtaining the image to be estimated, the process may further include: performing distortion correction and color normalization on the image to be estimated to obtain a preprocessed image to be estimated.

[0035] In this embodiment, the original acquired image (i.e., the image to be estimated) can undergo standardization processing such as distortion correction and color normalization. For example, the image to be estimated is... The normalized image obtained after preprocessing can be ,in and These represent the mean and standard deviation of the image pixels, respectively. This embodiment can also enhance image contrast through adaptive histogram equalization, ensuring feature discernibility under extreme lighting conditions, thereby obtaining the preprocessed image to be estimated.

[0036] S120. The image to be estimated is processed by a hybrid encoder to obtain the hybrid coding features of the image to be estimated. The hybrid coding features are obtained by fusing the local convolutional features and the global window self-attention features of the image to be estimated. The local convolutional features are obtained by performing depth-separable convolution on the image to be estimated. The global window self-attention features are obtained by dynamically dividing the image to be estimated into windows and performing self-attention processing under each window.

[0037] Specifically, local convolutional features can be obtained by processing the preprocessed image to be estimated through depth-separable structures in local convolutional branches. Global window self-attention features can be obtained by processing through window self-attention branches. Hybrid coding features can be obtained by gating and fusing local convolutional features and global window self-attention features.

[0038] In one optional implementation, local convolutional features can be obtained by: extracting features from the preprocessed image to be estimated according to preset convolutional parameters to obtain a first feature extraction result; processing the first feature extraction result sequentially according to a first preset depth convolutional parameter and a first preset pointwise convolutional parameter to obtain a second feature extraction result; and processing the second feature extraction result sequentially according to a second preset depth convolutional parameter and a second preset pointwise convolutional parameter to obtain local convolutional features.

[0039] For example, such as Figure 1b As shown, this embodiment can be performed according to preset convolution parameters (convolution kernel). (C / 4, step size 2) preprocess the image to be estimated (image dimension H) W 3) Perform feature extraction to obtain the first feature extraction result; then, using depthwise separable block 1, perform convolution according to the first preset depth parameters (convolution kernel 3). 3) and the first preset pointwise convolution parameters (convolution kernel 1) The first feature extraction result is processed by (1, C / 2) to obtain the second feature extraction result; the depth-separable block 2 is then processed according to the second preset depth convolution parameters (convolution kernel 3). 3) and the second preset pointwise convolution parameters (convolution kernel 1) 1, C) The second feature extraction result is processed to obtain the local convolutional feature (H / 4) W / 4 C), Figure 1bThe dashed lines represent auxiliary connections, and the solid lines represent the main data flow. This embodiment uses depthwise separable convolution to reduce computational complexity while preserving local feature extraction.

[0040] In one optional implementation, the global window self-attention features can be obtained as follows: The preprocessed image to be estimated is divided into blocks to obtain multiple image blocks, and each image block is linearly embedded to obtain an embedding vector; based on the embedding vector of each image block, a preset horizontal gradient operator, and a preset vertical gradient operator, the edge pixel distribution density of each image block is calculated; based on the edge pixel distribution density of each image block and a preset edge density threshold, the window size of each image block is dynamically adjusted; multi-head self-attention is calculated within each image block after the window size adjustment to obtain the respective attention calculation results; the attention calculation results are concatenated and fused through a linear transformation to obtain the global window self-attention features.

[0041] For example, such as Figure 1b As shown, this embodiment can perform patch embedding (i.e., Patch embedding, with parameters such as 4) on the preprocessed image to be estimated. 4, dim=C), the preprocessed image to be estimated is divided into 4 Image blocks of 4 pixels each, each containing 16 pixels, are transformed using a linear transformation. The original pixel values ​​of the 4 image blocks are mapped to C-dimensional space to obtain the embedding vector.

[0042] Then, adaptive window partitioning is performed. Specifically, edge density (i.e., edge pixel distribution density) is calculated for each image block based on preset horizontal and vertical gradient operators, which can be expressed as follows: ,in and Let represent the gradient operators in the horizontal and vertical directions, respectively, and I represent the preprocessed image to be estimated.

[0043] Furthermore, the edge density of each image patch can be compared with a preset edge density threshold, and the window size of each image patch can be dynamically adjusted based on the comparison result. For example, it can be represented as... ,in, Indicates window size. It can represent the preset upper limit value of edge density. It can represent a preset lower limit value for edge density. This adaptive strategy allows high-resolution features to be maintained in textured areas and computational redundancy to be reduced in smooth areas.

[0044] Furthermore, multi-head attention can be calculated for each image patch after dynamically adjusting the window size, yielding the self-attention calculation result for each window. Let the feature sequence within the specified window be... Where n represents the sequence length and d represents the feature dimension, the self-attention calculation process is as follows: Where Q represents the query matrix, K represents the key matrix, and V represents the value matrix, and , This represents the learnable projection matrix. This setting, by restricting attention computation to a local window, reduces computational complexity from that of global self-attention. Reduce to Furthermore, the results of each attention calculation can be concatenated and linearly transformed to obtain the global window self-attention feature (H / 4). W / 4 C).

[0045] In one optional implementation, the hybrid coding features can be obtained by: weighting and concatenating local convolutional features and global window self-attention features according to preset gating weights, and calculating a gating value according to a preset activation function and the weighted concatenation result; and weighting and fusing local convolutional features and global window self-attention features according to the gating value to obtain hybrid coding features.

[0046] Examples such as Figure 1b The gated fusion module represents local convolutional features as The global window self-attention feature is represented as ;pass Calculate the gate value, where, This represents the activation function. and Indicates the preset gating weight. This indicates that local convolutional features and global self-attention features are weighted and concatenated according to preset gating weights; through Calculate the hybrid coding features, where, This represents element-wise product. This gating mechanism enables the network to adaptively select the feature representation that best suits the current region.

[0047] Optionally, this embodiment can introduce a global modeling module composed of dynamic sparse attention or sparse linear transformation to replace the local self-attention after window partitioning. Dynamic sparse attention predicts saliency masks and performs long-range correlation calculations only at high-response locations, achieving an equivalent reduction in computational complexity and storage overhead compared to the local windowing strategy. It can also maintain a global perspective while meeting the real-time requirements of embedded platforms.

[0048] Optionally, the hybrid coding module can be replaced with a lightweight convolutional architecture that employs grouped convolutions, layer-by-layer variable resolution, and conditional computation, such as implementing computation path pruning in the form of channel rearrangement or dynamic depthwise separable convolutions; and a small number of multi-head self-attention layers can be superimposed at the end of the backbone to capture cross-regional semantic dependencies.

[0049] S130. Align the hybrid coded features to obtain the depth feature map of the image to be estimated.

[0050] In one optional implementation, feature alignment of the hybrid coded features to obtain a depth feature map of the image to be estimated may include: predicting offsets and modulation scalars for the sampling points of the hybrid coded features using deformable convolution; determining the actual sampling position of the hybrid coded features based on the offsets, predefined sampling offsets, and specified output positions; processing the hybrid coded features based on the actual sampling position, modulation scalars, and preset sampling weights to obtain the feature response result of the hybrid coded features at the target position; and obtaining the depth feature map of the image to be estimated based on the feature response result.

[0051] In this embodiment, it can be achieved by... The depth value at position p in the output depth feature map is obtained, thus yielding the depth feature map of the image to be estimated. Here, x represents the hybrid encoded feature. Indicates a predefined sampling offset. This represents the predicted offset of sampling point k. This represents the modulation scalar at sampling point k. Indicates the actual sampling location. This represents the sampling weight. This setting can effectively solve the problem of spatial misalignment of features across scales.

[0052] Optionally, this embodiment can also perform sparse upsampling on the results after deformable convolution alignment to efficiently recover high-resolution features, reduce computational redundancy, and then perform skip connection fusion to balance local details and global semantics, thus solving the problem of target scale variation in the image.

[0053] Alternatively, multi-scale content-aware convolutions with learnable biases or pixel-wise modulation convolutions can be used instead of deformable convolutions. Content-aware convolutions, by introducing modulation coefficients dynamically generated from neighborhood features into the kernel weights, can adaptively adjust sampling weights for cross-scale displacements, thus providing equivalent alignment and distortion suppression capabilities to deformable convolutions.

[0054] S140. Determine the predicted depth value and depth prediction confidence of each pixel based on the depth feature map to achieve image depth estimation.

[0055] In one optional implementation, determining the predicted depth value and depth prediction confidence of each pixel based on the depth feature map may include: mapping the depth feature map to a predefined depth range through a depth prediction branch to obtain the predicted depth value of each pixel; calculating the depth feature prediction variance of the depth feature map through a confidence prediction branch and calculating the error between the predicted depth and the actual depth; and determining the depth prediction confidence of each pixel based on the prediction variance and the error.

[0056] In this embodiment, the depth prediction branch can use scale-invariant log-space regression to map the depth feature map to a predefined depth range, obtaining the depth prediction value for each pixel. For example, it can be achieved through... To achieve, among which, and These represent the predefined lower depth limit and upper depth limit, respectively. This represents the depth features in the depth feature map of the image to be estimated. This represents the activation function.

[0057] The confidence prediction branch can quantify the reliability of deep prediction results by learning variance uncertainty. For example, confidence prediction results can be obtained using negative log-likelihood loss. ,in For pixels The predicted variance at that location Indicates the actual depth.

[0058] Optionally, pixel confidence estimation can also be achieved using methods based on Monte Carlo dropout reasoning, channel-level heterogeneous integration, or evidence theory-based deep regression.

[0059] The technical solution of this embodiment acquires the image to be estimated of the target device during the inspection process; processes the image to be estimated using a hybrid encoder to obtain the hybrid coding features of the image to be estimated; the hybrid coding features are obtained by fusing the local convolutional features and global window self-attention features of the image to be estimated; the local convolutional features are obtained by performing depth-separable convolution processing on the image to be estimated; the global window self-attention features are obtained by dynamically dividing the image to be estimated into windows and performing self-attention processing under each window; the hybrid coding features are aligned to obtain the depth feature map of the image to be estimated; the predicted depth value and depth prediction confidence of each pixel are determined based on the depth feature map to achieve image depth estimation. By employing the technical means of hybrid local convolution and dynamic window self-attention, deformable convolution to achieve feature alignment, and depth-confidence dual-branch output, this solution solves the problems of poor real-time performance, inaccurate monitoring results, and security issues caused by unclear depth prediction accuracy in existing technical solutions, thereby improving the real-time performance and accuracy of depth prediction and thus improving inspection security.

[0060] To enable those skilled in the art to better understand the image depth estimation method of this embodiment, Figure 1c This embodiment provides a schematic diagram of an image monocular depth estimation system. The system comprises an image acquisition module (industrial camera), a preprocessing module, a depth estimation main network (hybrid encoder module, deformable decoder module, and depth prediction branch), a confidence evaluation subnetwork (confidence prediction branch), and a sequence processing module. These modules are cascaded via data streams, allowing for both offline training and online inference. The specific neural network diagram of the hybrid encoder module can be shown below. Figure 1b As shown, it includes local convolutional branches, window self-attention branches, and gating fusion modules.

[0061] It needs to be explained that, Figure 1c The system shown can be optimized end-to-end using a multi-task joint loss function, and the total system loss can be expressed as follows: The depth loss, expressed using the scale-invariant logarithmic error, can be described as follows: The confidence loss can be expressed as Smoothing loss is used to preserve the local continuity of the depth map and can be expressed as: Normal consistency loss ensures the correctness of the geometric structure and can be expressed as: ,in and These are the predicted normal vector and the true normal vector, respectively.

[0062] The above describes the process of depth estimation for a single image. If the input... Figure 1c The system shown is composed of consecutive frame images, and can employ a confidence-weighted TSDF (Truncated Signed Distance Function) fusion strategy. Given new depth observations... and corresponding confidence level voxels The TSDF value is updated as follows: , ,in For cumulative weighting, As the current observation weight, This represents the signed distance function value of the current frame. This fusion strategy can effectively suppress the negative impact of low-confidence regions on the 3D reconstruction of target devices in inspection environments. Optionally, TSDF fusion can be replaced by incremental concave hull updates based on voxel hashing or hierarchical voxel merging strategies based on sparse octrees. Voxel hashing has lower memory consumption in large scenes, and octree fusion can adapt to resolution in the spatial domain. Both can also complete real-time dense point cloud accumulation within the computational and storage constraints of mobile platforms.

[0063] This embodiment achieves a dynamic balance between local texture detail capture and long-range context modeling by embedding ROI (Region of Interest) constraints into a lightweight convolutional backbone of window self-attention. This significantly reduces parameter size and computational redundancy while maintaining the global receptive field. With the help of a unified operator fusion and fixed-point quantization strategy, the system can run smoothly on power-constrained embedded platforms, greatly shortening the overall latency from image acquisition to depth output, and providing a guarantee for real-time defect alarm and closed-loop path planning.

[0064] The depth-confidence dual-branch output provides a quantitative uncertainty assessment for each pixel, which the inspection system can use to dynamically adjust thresholds, perform redundancy verification, or perform multi-sensor supplementary testing, thereby reducing the false alarm rate while ensuring the defect detection recall rate. The confidence heatmap can also serve as a weight in the downstream 3D point cloud stitching and digital twin rendering process, further improving the integrity and consistency of the global reconstruction.

[0065] This embodiment of incremental TSDF fusion pipeline, through confidence weighting and abnormal pixel suppression, realizes online dense point cloud construction in environments such as long-distance pipe corridors, tank areas, and narrow cavities, providing an accurate three-dimensional basis for subsequent crack evolution monitoring, corrosion quantification assessment, and equipment remaining life prediction. The interface with external visual odometry, inertial navigation, or lidar is loosely coupled, enabling the system to be flexibly integrated into existing inspection robots, drones, and fixed monitoring nodes, reducing modification costs and simplifying operation and maintenance processes.

[0066] This embodiment achieves real-time, high-precision, and robust perception of depth information in complex industrial scenarios while ensuring low power consumption and small size. It significantly improves the safety redundancy capability of defect alarms through confidence quantification and multi-domain adaptation, and provides a unified and reliable depth foundation for large-scale 3D digital modeling. It has outstanding engineering application value and broad industrial promotion prospects.

[0067] Example 2

[0068] Figure 2 This is a schematic diagram of an image depth estimation device provided in Embodiment 2 of the present invention. Figure 2 As shown, the device includes: an image acquisition module 210 to be estimated, a hybrid coding feature acquisition module 220, a depth feature map acquisition module 230, and a depth prediction module 240. Wherein:

[0069] The image acquisition module 210 is used to acquire the image of the target equipment to be estimated during the inspection process;

[0070] The hybrid coding feature acquisition module 220 is used to process the image to be estimated through a hybrid encoder to obtain the hybrid coding features of the image to be estimated; the hybrid coding features are obtained by fusing the local convolutional features and global window self-attention features of the image to be estimated; the local convolutional features are obtained by performing depth-separable convolution processing on the image to be estimated; the global window self-attention features are obtained by dynamically dividing the image to be estimated into windows and performing self-attention processing under each window;

[0071] The depth feature map acquisition module 230 is used to perform feature alignment on the hybrid coded features to obtain the depth feature map of the image to be estimated;

[0072] The depth prediction module 240 is used to determine the predicted depth value and depth prediction confidence of each pixel based on the depth feature map, thereby realizing image depth estimation.

[0073] The technical solution of this embodiment acquires the image to be estimated of the target device during the inspection process; processes the image to be estimated using a hybrid encoder to obtain the hybrid coding features of the image to be estimated; the hybrid coding features are obtained by fusing the local convolutional features and global window self-attention features of the image to be estimated; the local convolutional features are obtained by performing depth-separable convolution processing on the image to be estimated; the global window self-attention features are obtained by dynamically dividing the image to be estimated into windows and performing self-attention processing under each window; the hybrid coding features are aligned to obtain the depth feature map of the image to be estimated; the predicted depth value and depth prediction confidence of each pixel are determined based on the depth feature map to achieve image depth estimation. By employing the technical means of hybrid local convolution and dynamic window self-attention, deformable convolution to achieve feature alignment, and depth-confidence dual-branch output, this solution solves the problems of poor real-time performance, inaccurate monitoring results, and security issues caused by unclear depth prediction accuracy in existing technical solutions, thereby improving the real-time performance and accuracy of depth prediction and thus improving inspection security.

[0074] Optionally, the image depth estimation device further includes an image preprocessing module, used after acquiring the image to be estimated:

[0075] The image to be estimated is subjected to distortion correction and color normalization to obtain a preprocessed image to be estimated.

[0076] Optionally, the image depth estimation device further includes a local convolutional feature acquisition module, used for:

[0077] The preprocessed image to be estimated is subjected to feature extraction according to preset convolution parameters to obtain the first feature extraction result;

[0078] The first feature extraction result is processed sequentially according to the first preset depth convolution parameter and the first preset pointwise convolution parameter to obtain the second feature extraction result;

[0079] The second feature extraction result is processed sequentially according to the second preset depth convolution parameter and the second preset pointwise convolution parameter to obtain the local convolution feature.

[0080] Optionally, the image depth estimation device further includes a global window self-attention feature acquisition module, used for:

[0081] The preprocessed image to be estimated is divided into blocks to obtain multiple image blocks, and linear embedding is performed on each image block to obtain the embedding vector of each image block.

[0082] Based on the embedding vector of each image block, the preset horizontal gradient operator, and the preset vertical gradient operator, the edge pixel distribution density is calculated for each image block;

[0083] The window size of each image block is dynamically adjusted based on the edge pixel distribution density and a preset edge density threshold.

[0084] Multi-head self-attention is calculated separately for each image block after the window size is adjusted, and the attention calculation results are obtained for each block.

[0085] The attention calculation results are concatenated and fused through a linear transformation to obtain the global window self-attention feature.

[0086] Optionally, the image depth estimation device further includes a hybrid coded feature acquisition module, used for:

[0087] The local convolutional features and the global window self-attention features are weighted and concatenated according to a preset gating weight, and a gating value is calculated based on a preset activation function and the weighted concatenation result.

[0088] The local convolutional features and the global window self-attention features are weighted and fused according to the gating value to obtain the hybrid coding features.

[0089] Optional, the depth feature map acquisition module 230 can be used for:

[0090] The offset and modulation scalar are predicted for the sampling points of the hybrid coded features using deformable convolution;

[0091] The actual sampling position of the hybrid coded feature is determined based on the offset, the predefined sampling offset, and the specified output position.

[0092] The hybrid coding features are processed according to the actual sampling position, the modulation scalar and the preset sampling weight to obtain the feature response result of the hybrid coding features at the target position;

[0093] The depth feature map of the image to be estimated is obtained based on the feature response results.

[0094] Optional, the depth prediction module 240 can be used for:

[0095] The depth feature map is mapped to a predefined depth range through the depth prediction branch to obtain the depth prediction value of each pixel;

[0096] The depth feature prediction variance of the depth feature map is predicted by the confidence prediction branch, and the error between the predicted depth and the actual depth is calculated. The depth prediction confidence of each pixel is determined based on the prediction variance and the error.

[0097] The image depth estimation device provided in the embodiments of the present invention can execute the image depth estimation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0098] Example 3

[0099] Figure 3 A schematic diagram of an electronic device 300 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers or various forms of mobile devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0100] like Figure 3 As shown, the electronic device 300 includes at least one processor 301 and a memory, such as a read-only memory (ROM) 302 or a random access memory (RAM) 303, communicatively connected to the at least one processor 301. The memory stores computer programs executable by the at least one processor. The processor 301 can perform various appropriate actions and processes based on the computer program stored in the ROM 302 or loaded into the RAM 303 from storage unit 308. The RAM 303 can also store various programs and data required for the operation of the electronic device 300. The processor 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0101] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0102] Processor 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 301 performs the various methods and processes described above, such as image depth estimation methods.

[0103] In some embodiments, the image depth estimation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by processor 301, one or more steps of the image depth estimation method described above may be performed. Alternatively, in other embodiments, processor 301 may be configured to perform the image depth estimation method by any other suitable means (e.g., by means of firmware).

[0104] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0106] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0107] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0108] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0109] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0110] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0111] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An image depth estimation method, characterized in that, include: During the inspection process, acquire the image to be estimated of the target equipment; The image to be estimated is processed by a hybrid encoder to obtain hybrid coding features of the image to be estimated; the hybrid coding features are obtained by fusing local convolutional features and global window self-attention features of the image to be estimated; the local convolutional features are obtained by performing depth-separable convolution on the image to be estimated; the global window self-attention features are obtained by dynamically dividing the image to be estimated into windows and performing self-attention processing under each window; The hybrid encoded features are aligned to obtain the depth feature map of the image to be estimated; Based on the depth feature map, the predicted depth value and depth prediction confidence of each pixel are determined to achieve image depth estimation.

2. The method according to claim 1, characterized in that, After obtaining the image to be estimated, the following steps are also included: The image to be estimated is subjected to distortion correction and color normalization to obtain a preprocessed image to be estimated.

3. The method according to claim 2, characterized in that, The local convolutional features are obtained in the following way: The preprocessed image to be estimated is subjected to feature extraction according to preset convolution parameters to obtain the first feature extraction result; The first feature extraction result is processed sequentially according to the first preset depth convolution parameter and the first preset pointwise convolution parameter to obtain the second feature extraction result; The second feature extraction result is processed sequentially according to the second preset depth convolution parameter and the second preset pointwise convolution parameter to obtain the local convolution feature.

4. The method according to claim 2, characterized in that, The global window self-attention feature is obtained in the following way: The preprocessed image to be estimated is divided into blocks to obtain multiple image blocks, and linear embedding is performed on each image block to obtain the embedding vector of each image block. Based on the embedding vector of each image block, the preset horizontal gradient operator, and the preset vertical gradient operator, the edge pixel distribution density is calculated for each image block; The window size of each image block is dynamically adjusted based on the edge pixel distribution density and a preset edge density threshold. Multi-head self-attention is calculated separately for each image block after the window size is adjusted, and the attention calculation results are obtained for each block. The attention calculation results are concatenated and fused through a linear transformation to obtain the global window self-attention feature.

5. The method according to claim 1, characterized in that, The hybrid coding features are obtained in the following way: The local convolutional features and the global window self-attention features are weighted and concatenated according to a preset gating weight, and a gating value is calculated based on a preset activation function and the weighted concatenation result. The local convolutional features and the global window self-attention features are weighted and fused according to the gating value to obtain the hybrid coding features.

6. The method according to claim 1, characterized in that, The hybrid encoded features are aligned to obtain a depth feature map of the image to be estimated, including: The offset and modulation scalar are predicted for the sampling points of the hybrid coded features using deformable convolution; The actual sampling position of the hybrid coded feature is determined based on the offset, the predefined sampling offset, and the specified output position. The hybrid coding features are processed according to the actual sampling position, the modulation scalar and the preset sampling weight to obtain the feature response result of the hybrid coding features at the target position; The depth feature map of the image to be estimated is obtained based on the feature response results.

7. The method according to claim 6, characterized in that, Determining the predicted depth value and depth prediction confidence for each pixel based on the depth feature map includes: The depth feature map is mapped to a predefined depth range through the depth prediction branch to obtain the depth prediction value of each pixel; The depth feature prediction variance of the depth feature map is predicted by the confidence prediction branch, and the error between the predicted depth and the actual depth is calculated. The depth prediction confidence of each pixel is determined based on the prediction variance and the error.

8. An image depth estimation device, characterized in that, include: The image acquisition module is used to acquire the image of the target equipment to be estimated during the inspection process; The hybrid coding feature acquisition module is used to process the image to be estimated through a hybrid encoder to obtain the hybrid coding features of the image to be estimated. The hybrid coding features are obtained by fusing the local convolutional features and global window self-attention features of the image to be estimated. The local convolutional features are obtained by performing depth-separable convolution processing on the image to be estimated. The global window self-attention features are obtained by dynamically dividing the image to be estimated into windows and performing self-attention processing under each window. The depth feature map acquisition module is used to perform feature alignment on the hybrid coded features to obtain the depth feature map of the image to be estimated; The depth prediction module is used to determine the predicted depth value and depth prediction confidence of each pixel based on the depth feature map, thereby realizing image depth estimation.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform an image depth estimation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute an image depth estimation method according to any one of claims 1-7.

Images