Monocular image semantic completion method based on attention mechanism, medium and equipment

Abstract

The invention discloses a monocular image semantic completion method based on an attention mechanism, a medium and equipment. The method comprises the following steps: receiving a monocular indoor scene image and preprocessing the monocular indoor scene image; extracting an original feature map including a depth feature map and a semantic feature map through a convolutional neural network; performing channel dimension splicing on the depth feature map and the semantic feature map to form a spliced feature map; processing the spliced feature map through a learnable convolutional layer, calculating a normalized attention weight based on a coupling relationship between a depth feature and a semantic feature, and performing weighted optimization on a feature value to obtain an attention feature map; adding and fusing the attention feature map and the original feature map element by element; and finally, recovering the image size through up-sampling, predicting the semantic category of each pixel point, and generating a semantic completion result. According to the method, through dynamic coupling of the depth features and the semantic features and an attention weighting mechanism, the problem of insufficient fusion of the depth information and the semantic information in a traditional method is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a semantic completion method of monocular image based on attention mechanism, medium and equipment. BACKGROUND

[0002] Traditional monocular indoor scene semantic completion methods are generally based on the encoder-decoder structure (such as U-Net) of convolutional neural network (CNN), which transmits features through a skip connection to preserve details, and then realizes semantic completion of missing areas. However, such methods have significant limitations in practical applications, and the core problem lies in the rationality of depth information processing and feature extraction:

[0003] Firstly, it performs poorly in areas with dramatic depth changes (such as staircases, room-to-corridor transition zones). Due to the lack of dynamic perception ability to the importance of different depth areas, it is unable to allocate attention weights according to the complexity of depth changes, resulting in inaccurate depth details in the completion results, large deviation from the actual scene, and serious loss of key area information.

[0004] Secondly, the fusion method of depth and semantic information is too simple. The existing technology mostly adopts a direct combination method, which fails to fully utilize the complementary advantages of the two types of information, limiting the accuracy of the completion results.

[0005] In addition, the feature extraction lacks a dynamic adjustment mechanism. Traditional methods treat all areas on the image equally, without differentiating the extraction focus according to the complexity of depth changes, resulting in loss of useful information in key areas, while non-key areas may retain excessive useless information, further affecting the completion effect. SUMMARY

[0006] In view of the above problems, the present application provides a semantic completion method of monocular image based on attention mechanism, medium and equipment, which solves the problem of key area information loss in the image semantic completion method of the prior art, affecting the accuracy of the completion results.

[0007] To solve the above problems, in a first aspect, the present application provides a semantic completion method of monocular image based on attention mechanism, which comprises the following steps:

[0008] S1: receiving a monocular indoor scene image, pre-processing the monocular indoor scene image to obtain a first scene image;

[0009] S2: extracting an original feature map of the first scene image through a convolutional neural network, the original feature map comprising a depth feature map and a semantic feature map;

[0010] S3: concatenating the depth feature map and the semantic feature map in the channel dimension to form a concatenated feature map;

[0011] S4: processing the spliced feature map through a learnable convolutional layer to obtain feature values corresponding to each spatial coordinate position on the spliced feature map, calculating normalized attention weights based on a coupling relationship between deep features and semantic features, and performing weighted optimization on the feature values corresponding to each spatial coordinate position on the spliced feature map according to the attention weights to obtain an attention feature map;

[0012] S5: performing element-by-element addition fusion on the attention feature map and the original feature map to obtain a fusion feature map;

[0013] S6: restoring the fusion feature map to the same size as the monocular indoor scene image received in step S1 through an upsampling operation to obtain a second scene image, and predicting semantic categories of each pixel point on the second scene image using a convolution operation to generate a semantic completion result.

[0014] Further, the normalized attention weights are calculated based on the feature values, and the calculation formula is as follows:

[0015]

[0016] wherein w i is the attention weight corresponding to position i of the spliced feature map obtained by processing through the learnable convolutional layer in step S4, exp(·) is a natural exponential function, z i represents the maximum feature value of position i belonging to a certain semantic list, z k represents the feature value of position i belonging to the kth semantic category, N represents the total number of semantic categories of position i predicted, and k ranges from 1 to N.

[0017] Further, the method comprises:

[0018] S41: calculating the gradient amplitude of the original depth feature map using a Sobel operator and comparing the gradient amplitude with a preset threshold τ, if then determining it as a sudden change region, otherwise determining it as a flat region;

[0019] S42: configuring a first modulation factor of the attention formula greater than 1 for the sudden change region, adjusting the feature values of the sudden change region participating in the calculation of the attention weights according to the first modulation factor, and configuring a second modulation factor of the attention formula less than 1 for the flat region, adjusting the feature values of the flat region participating in the calculation of the attention weights according to the second modulation factor.

[0020] Further, according to the first modulation factor, the eigenvalue of the abrupt region participating in the calculation of the attention weight is adjusted, and the calculation formula of the attention weight is as follows:

[0021]

[0022] wherein ||·|| represents a norm function, λ represents an enhancement coefficient, and the value range of λ is [0.1, 1], represents the gradient amplitude of position i on the original depth feature map, represents the first modulation factor;

[0023] According to the second modulation factor, the eigenvalue of the flat region participating in the calculation of the attention weight is adjusted, and the calculation formula of the attention weight is as follows:

[0024]

[0025] wherein I(i) represents the flat region indication function value corresponding to position i, I(i) is a binary function, and if the depth variance σ 2 (D i ) of the neighborhood of position i is less than the variance threshold , the value of I(i) is 1, otherwise the value of I(i) is 0, β is a decay function, and the value range of β is [0.1, 0.3], 1-β·I(i) represents the second modulation factor.

[0026] Further, the semantic class of each pixel point on the second scene image is predicted by using a convolution operation, and the semantic completion result is generated, including:

[0027] The semantic class number and convolution value of each pixel point are output by using the convolution operation, the convolution value is converted into a probability distribution by using a softmax activation function, the output dimension of the softmax activation function is consistent with the predicted semantic class number, the probability value corresponding to each pixel point represents the probability that the pixel point belongs to a certain semantic class, and the semantic class with the maximum probability value is taken as the predicted semantic class of the current pixel point.

[0028] All pixel points on the second scene image are traversed, and the above operation is repeated to generate the semantic completion result.

[0029] Further, before the step of taking the semantic class with the maximum probability value as the predicted semantic class of the current pixel point, the step of judging whether the difference between the maximum probability value and other probability values is greater than a preset error range is further included, if yes, the semantic class with the maximum probability value is taken as the predicted semantic class of the current pixel point, and if no, the following steps are performed:

[0030] A preferred probability set G i corresponding to the current pixel point is constructed.i The maximum probability value of the current pixel point and the probability value within the preset error range from the maximum probability value, G i Each probability value in G i corresponds to a semantic category.

[0031] The maximum probability value of the semantic category of the neighborhood pixel point corresponding to the current pixel point is obtained, and the maximum probability value of the neighborhood pixel point with semantic category overlap is screened out.

[0032] Further, the method further comprises:

[0033] After obtaining the semantic completion result, the second scene image is divided into different category regions according to the predicted semantic category corresponding to the pixel point.

[0034] The edges of each category region are smoothed using a Gaussian filter, and the hollow part of each category region is completed using a texture propagation algorithm.

[0035] Completing the hollow part of each category region specifically includes:

[0036] The gradient amplitude of each spatial position of the original depth feature map is calculated using a Sobel operator, and the gradient amplitudes are arranged in ascending order.

[0037] A number of pixel points with the smallest gradient amplitude in the same category region in the current second scene image are obtained according to the number of pixel points contained in the hollow part, and the obtained pixel points are used to complete the hollow part.

[0038] Further, the method further comprises:

[0039] A plurality of indoor scene images with semantic annotations are collected to obtain an indoor scene image dataset.

[0040] The indoor scene image dataset is input into a pre-trained model, and the model is trained through a composite loss function to enable the model to learn semantic features and depth features simultaneously, wherein the composite loss function includes a semantic segmentation loss and a depth estimation loss, and the calculation formula of the composite loss function is as follows:

[0041] L=xL sem +yL depth ;

[0042] wherein L represents the composite loss, x and y represent learnable parameters and are dynamically adjusted through backpropagation, L sem represents the semantic segmentation loss, and L depthdenotes a depth estimation loss.

[0043] In a second aspect, the present application provides a computer-readable storage medium, which stores computer program instructions, the computer program instructions, when executed by a processor, implement the method of the first aspect of the present application.

[0044] In a third aspect, the present application also provides an electronic device, comprising a memory and a processor, the memory being configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of the first aspect.

[0045] Distinguished from the prior art, the above technical solution discloses a semantic completion method of monocular image based on attention mechanism, medium and equipment, the method comprising: receiving a monocular indoor scene image and performing preprocessing; extracting an original feature map containing a depth feature map and a semantic feature map through a convolutional neural network; channel dimensionally splicing the depth feature map and the semantic feature map to form a spliced feature map; processing the spliced feature map through a learnable convolutional layer, calculating normalized attention weights based on the coupling relationship between the depth feature and the semantic feature, and performing weighted optimization on the feature values to obtain an attention feature map; element-wise adding and fusing the attention feature map and the original feature map; finally, restoring the image size through upsampling and predicting the semantic category of each pixel point to generate a semantic completion result. Through the dynamic coupling of the depth feature and the semantic feature and the attention weighting mechanism, the present application effectively solves the problems of insufficient fusion of depth information and semantic information and weak feature response in key areas in the traditional method, and significantly improves the accuracy and detail preservation ability of indoor scene semantic completion.

[0046] The above invention content related records are only summaries of the technical solutions of the present application. In order to enable those skilled in the art to more clearly understand the technical solutions of the present application, and then implement the contents recorded in the specification and drawings, and in order to enable the above purposes and other purposes, features and advantages of the present application to be more easily understood, the following describes the specific embodiments of the present application and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings only serve to illustrate the principles, implementation manners, applications, characteristics and effects of the specific embodiments of the present application and other related contents, and cannot be considered as limitations of the present application.

[0048] In the drawings of the specification:

[0049] Figure 1 a flowchart of the semantic completion method of monocular image based on attention mechanism described in the first exemplary embodiment of the present application;

[0050] Figure 2A flow chart of the method for semantic completion of a monocular image based on an attention mechanism according to the second exemplary embodiment of the present application;

[0051] Figure 3 A flow chart of the method for semantic completion of a monocular image based on an attention mechanism according to the third exemplary embodiment of the present application;

[0052] Figure 4 A flow chart of the method for semantic completion of a monocular image based on an attention mechanism according to the fourth exemplary embodiment of the present application;

[0053] Figure 5 A flow chart of the method for semantic completion of a monocular image based on an attention mechanism according to the fifth exemplary embodiment of the present application;

[0054] Figure 6 A flow chart of the method for semantic completion of a monocular image based on an attention mechanism according to the sixth exemplary embodiment of the present application;

[0055] Figure 7 A flow chart of the method for semantic completion of a monocular image based on an attention mechanism according to the seventh exemplary embodiment of the present application;

[0056] Figure 8 A module schematic diagram of the device for semantic completion of a monocular image based on an attention mechanism according to the seventh exemplary embodiment of the present application;

[0057] Figure 9 A module schematic diagram of the electronic device according to an exemplary embodiment of the present application.

[0058] The reference signs involved in the above-mentioned figures are explained as follows:

[0059] 10, electronic device;

[0060] 101, processor;

[0061] 102, storage medium. DETAILED DESCRIPTION

[0062] To explain possible application scenarios, technical principles, specific schemes that can be implemented, purposes and effects that can be achieved of the present application in detail, the following will be explained in detail in combination with specific embodiments listed and with the aid of the drawings. The embodiments described in this document are only used to more clearly explain the technical schemes of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0063] The term 'embodiment' is mentioned in the present document means that the specific features, structures or properties described in connection with the embodiment can be included in at least one embodiment of the present application. The term 'embodiment' appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit the independence or association between other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, each technical feature mentioned in each embodiment can be combined in any way to form a corresponding implementable technical solution.

[0064] Unless otherwise defined, the meanings of the technical terms used in the present document are the same as those commonly understood by the person skilled in the art to which the present application belongs; the use of related terms in the present document is only for the purpose of describing specific embodiments, and is not intended to limit the present application.

[0065] In the description of the present application, the phrase 'and / or' is a description of the logical relationship between the objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases: A exists, B exists, and A and B exist at the same time. In addition, the character ' / ' in the present document generally represents that the associated objects before and after are a kind of 'or' logical relationship.

[0066] In the present application, the terms such as 'first' and'second' are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual quantity, primary and secondary or order relationship between the entities or operations.

[0067] In the present application, without more limitation, the phrase 'includes', 'contains', 'has' or other similar open expressions used in the sentence are intended to cover non-exclusive inclusion, and these expressions do not exclude the presence of other elements in the process, method or product including the described elements, so that the process, method or product including a series of elements can not only include those limited elements, but also include other elements not explicitly listed, or also include the elements inherent to such process, method or product.

[0068] As the same as the understanding in the 'Guidelines for Examination', in the present application, the expressions such as 'greater than', 'less than', 'exceed' are understood as not including the number; the expressions such as 'above', 'below', 'within' are understood as including the number. In addition, in the description of the embodiments of the present application, the meaning of'multiple' is more than two (including two), and similar expressions related to'multiple' are also understood in this way, for example,'multiple groups','multiple times', etc., unless otherwise explicitly limited.

[0069] As Figure 1 shown, in a first aspect, the present application provides a semantic completion method of monocular image based on attention mechanism, which comprises the following steps:

[0070] S1: receive a monocular indoor scene image, preprocess the monocular indoor scene image to obtain a first scene image.

[0071] In step S1, the monocular indoor scene image refers to a two-dimensional image of an indoor space scene captured by a monocular camera, containing visual information such as objects, structures, textures, etc. in the scene, but lacking direct depth information.

[0072] Preprocessing refers to a series of operations such as normalization and size adjustment on the original monocular indoor scene image, so that the image meets the needs of subsequent convolutional neural network processing. Among them, normalization is to adjust the image pixel value to a certain range (such as [-1, 1]) to speed up model convergence and improve stability; size adjustment is to scale the image to a uniform size (such as 256x256 pixels) required by the model input.

[0073] After receiving the monocular indoor scene image, normalization and size adjustment and other preprocessing operations can eliminate the scale difference of image data, so that the model can converge faster, and at the same time ensure that the input image size meets the requirements of the subsequent convolutional neural network, providing standardized data for feature extraction.

[0074] S2: extract the original feature map of the first scene image through a convolutional neural network, the original feature map including a depth feature map and a semantic feature map.

[0075] In step S2, the convolutional neural network (CNN) is a deep learning model specially designed for processing grid-structured data (such as images), composed of multiple convolutional layers, pooling layers, and fully connected layers, etc. The convolutional layer performs convolution operation by sliding the convolution kernel on the image to extract local features of the image; the pooling layer reduces the dimension of the feature map output by the convolutional layer, reducing the number of parameters while preserving important features; the fully connected layer integrates all features for the final classification or regression task.

[0076] The original feature map refers to the preliminary feature representation obtained after the CNN extracts features from the preprocessed image, where the depth feature map reflects the depth information of different regions in the image, and the semantic feature map contains the semantic category information of objects, structures, etc. in the scene.

[0077] By processing the preprocessed first scene image through the convolutional neural network, the convolution operation of the convolutional layer extracts the depth features and semantic features of the image at different scales, generating the depth feature map and the semantic feature map respectively. Convolution operation performs dot product operation between convolution kernel and image region to capture local patterns in the image, and gradually extracts features from low level (such as edge, texture) to high level (such as object shape, semantic category) as the network level deepens.

[0078] Specifically, the preprocessed image can be subjected to feature extraction based on a convolutional neural network of ResNet-50. Multi-layer convolutional operations gradually extract deep features and semantic features. The input image is subjected to a convolutional operation with a convolution kernel size of 7x7 and a stride of 2 to generate a feature map with a size of 128x128x64. Subsequently, the feature map resolution is reduced through a max-pooling layer. Then, the feature map enters multiple residual blocks, each of which contains three convolutional layers. After processing through multiple residual blocks, deep features and semantic features are gradually extracted. After the third residual block, a feature map with a size of 32x32x1024 is obtained.

[0079] S3: concatenating the deep feature map and the semantic feature map in a channel dimension to form a concatenated feature map.

[0080] In step S3, the concatenated feature map refers to concatenating the deep feature map and the semantic feature map in a channel dimension to form a new feature map that integrates deep and semantic information, thereby providing a basis for the subsequent application of the attention mechanism.

[0081] By concatenating the deep feature map and the semantic feature map in a channel dimension, the two different types of feature information are integrated into one feature map, so that the subsequent operation can consider both deep and semantic information, thereby providing a basis for more accurate analysis of the scene.

[0082] S4: processing the concatenated feature map through a learnable convolutional layer to obtain feature values corresponding to each spatial coordinate position on the concatenated feature map, calculating normalized attention weights based on the coupling relationship between the deep features and the semantic features according to the feature values, and weighting and optimizing the feature values corresponding to each spatial coordinate position on the concatenated feature map according to the attention weights to obtain an attention feature map.

[0083] In step S4, the learnable convolutional layer refers to a convolutional kernel parameter in the convolutional layer that is learned through training data, which can adaptively extract features according to the characteristics of the data. In the present application, the learnable convolutional layer is used to process the concatenated feature map to obtain feature values corresponding to each spatial coordinate position.

[0084] The normalized attention weight refers to a weight calculated according to the feature values of each position on the concatenated feature map, which is used to measure the importance of the features at that position. After normalization, the sum of the attention weights of all positions is 1.

[0085] The attention feature map refers to a feature map obtained by weighting and optimizing the feature values corresponding to each spatial coordinate position on the concatenated feature map using the calculated normalized attention weights, which highlights the features of key regions and suppresses redundant information.

[0086] In step S4, the spliced feature map is first processed by a learnable convolutional layer, the convolutional kernel parameters of which are continuously optimized in the training process to adapt to the feature patterns in the data. After processing, the feature values corresponding to each spatial coordinate position on the spliced feature map are obtained, and based on the coupling relationship between the depth features and the semantic features, the normalized attention weights are calculated according to these feature values, so that the attention weight corresponding to the position with a larger feature value is higher, i.e. more attention is paid. Then, the feature values corresponding to each spatial coordinate position on the spliced feature map are weighted and optimized according to the calculated attention weights, to obtain an attention feature map, highlighting the features of the depth change complex region and the semantic key region.

[0087] S5: Element-wise addition fusion of the attention feature map and the original feature map is performed to obtain a fusion feature map.

[0088] In step S5, the fusion feature map refers to the feature map obtained after element-wise addition fusion of the attention feature map and the original feature map. The fusion feature map not only utilizes the key information highlighted by the attention mechanism, but also retains the basic information in the original feature map, further enhancing the richness and effectiveness of the feature representation.

[0089] S6: The fusion feature map is restored to the same size as the monocular indoor scene image received in step S1 through an upsampling operation to obtain a second scene image, and a convolution operation is used to predict the semantic category of each pixel point on the second scene image to generate a semantic completion result.

[0090] In step S6, the upsampling operation is the opposite of downsampling (such as pooling operation), which restores the low-resolution feature map to high-resolution through interpolation and other methods, so that its size is the same as the original monocular indoor scene image, and a second scene image is obtained.

[0091] The convolution operation refers to applying a convolution kernel to the image or feature map for convolution operation to extract features or make classification prediction. In the present application, it is used to predict the semantic category of each pixel point on the second scene image.

[0092] The semantic category refers to the classification label of different objects, structures, etc. in the indoor scene, such as walls, floors, furniture, and people.

[0093] In the process of image semantic completion, firstly, the fusion feature map is restored to the same size as the monocular indoor scene image received in step S1 through upsampling operation, to obtain a second scene image. The upsampling operation can use methods such as bilinear interpolation and transpose convolution to restore the low-resolution feature map to the original image resolution, providing a feature map of appropriate size for subsequent pixel-level semantic prediction. Then, the semantic category of each pixel point on the second scene image is predicted by using convolution operation to generate the semantic completion result. The convolution operation calculates the convolution value of the features in the neighborhood of each pixel point on the second scene image with the convolution kernel by sliding the convolution kernel on the second scene image, and converts the convolution value into a probability distribution through a softmax activation function. By traversing all the pixel points on the second scene image and repeating the above operation, a complete semantic completion result is finally generated.

[0094] The above scheme has the following beneficial effects:

[0095] (1) Improve the depth detail processing capability: by introducing the attention mechanism, the importance of different depth regions can be dynamically perceived, and more attention is allocated to regions with large depth changes (such as the transition area between stairs, rooms and corridors), effectively solving the problem of insufficient processing of depth change sensitive regions in traditional methods, and improving the depth detail accuracy of the completion result in these regions.

[0096] (2) Optimize feature fusion: after splicing the depth feature map and the semantic feature map, process them through a learnable convolution layer and an attention mechanism, fully explore the coupling relationship between depth and semantic information, and realize effective fusion of the two. Compared with the traditional simple combination method, it can better play the complementary advantages of depth information and semantic information, improve the accuracy and smoothness of the completion result.

[0097] (3) Enhance the pertinence of feature extraction: the attention mechanism enables the model to dynamically adjust the attention degree to different regions during feature extraction according to the complexity of depth change, focusing on key areas and reducing the interference of useless information in non-key areas, improving the pertinence and effectiveness of feature extraction, and thus improving the adaptability to complex indoor scenes.

[0098] In some embodiments, the normalized attention weight is calculated according to the feature value, and the calculation formula is as follows:

[0099]

[0100] where w i is the attention weight corresponding to position i of the spliced feature map obtained by processing through the learnable convolution layer in step S4, exp(·) is the natural exponential function, z i represents the maximum feature value of position i belonging to a certain semantic list, and z kwherein z represents a feature value of a position i belonging to a k-th semantic category, N represents a total number of predicted semantic categories of the position i, and k ranges from 1 to N.

[0101] For example, for a certain spatial position i on the spliced feature map, the maximum feature value z i = 2 after the convolution layer processing, and the sum of feature values of other positions is

[0102] The attention weight of the position is:

[0103]

[0104] The weight value reflects the importance of the position in the fusion process of the depth feature map and the semantic feature map. The greater the weight value, the more the features of the position are retained in the subsequent fusion process.

[0105] The above formula is based on the principle of the softmax function. By performing exponential operation on the feature value z i of each position, the feature value is mapped to a larger value range, highlighting the difference between larger feature values, and then normalizing the exponential feature values of all positions, so that the sum of the attention weights w i of all positions is 1, thereby obtaining the importance weight of each position relative to other positions.

[0106] By this way of calculating and normalizing the attention weight, the importance of each position can be intuitively and effectively determined according to the feature value size of each position on the spliced feature map, providing a reasonable weight distribution basis for subsequent weighted optimization of the feature map, so that the model can automatically focus on the feature significant area, improving the capture ability of key information, and thus improving the effect of semantic completion.

[0107] In some embodiments, the method comprises:

[0108] S41: calculating the gradient amplitude of the original depth feature map using a Sobel operator and comparing the gradient amplitude with a preset threshold τ, if then determining as a mutation region, otherwise determining as a flat region.

[0109] In step S41, the Sobel operator is an operator for edge detection, which detects the edge information in the image by calculating the gradient of the image in the horizontal and vertical directions. In this embodiment, the Sobel operator is used to calculate the gradient amplitude of the original depth feature map to determine the depth change. Specifically, the Sobel operator performs convolution operation with the depth map in the horizontal and vertical directions to obtain the gradient values in the horizontal and vertical directions, and then calculates the gradient amplitude according to the Pythagorean theorem.

[0110] The gradient magnitude represents the size of the gradient at a certain point in the image, reflecting the degree of depth change at the point. The larger the gradient magnitude, the more intense the depth change, which may correspond to a depth mutation region. The smaller the gradient magnitude, the relatively flat the depth change, which may belong to a flat region.

[0111] The preset threshold τ is a gradient magnitude limit value for distinguishing between depth mutation regions and flat regions, which is determined by experiment or experience and used for subsequent determination of the region type of the depth map.

[0112] The mutation region is a region in the depth map with a gradient magnitude greater than or equal to the preset threshold τ, which usually corresponds to a part of the indoor scene with intense depth change, such as the steps of a staircase, the structural change at the transition between a room and a corridor, etc.

[0113] The flat region is a region in the depth map with a gradient magnitude less than the preset threshold τ, i.e., a region with relatively flat depth change, such as a large area of wall, flat ground, etc.

[0114] S42: configuring a first modulation factor of the attention formula greater than 1 for the mutation region, adjusting the feature values of the mutation region involved in the calculation of the attention weight according to the first modulation factor, and configuring a second modulation factor of the attention formula less than 1 for the flat region, adjusting the feature values of the flat region involved in the calculation of the attention weight according to the second modulation factor.

[0115] In step S42, the first modulation factor refers to a value greater than 1 configured for the mutation region, which is used to adjust the feature values of the mutation region involved in the calculation of the attention weight, enhancing the importance of the mutation region in attention calculation.

[0116] The second modulation factor refers to a value less than 1 configured for the flat region, which is used to adjust the feature values of the flat region involved in the calculation of the attention weight, relatively weakening the importance of the flat region in attention calculation.

[0117] By classifying the depth change regions and configuring corresponding modulation factors for different types of regions to adjust the attention weight calculation, the model's adaptive ability to depth change is further enhanced. In the depth mutation region, the attention allocation is increased, which can more accurately capture the complex depth information and semantic information of these regions, reducing the completion bias; in the flat region, the attention is appropriately reduced to avoid wasting too many resources in simple regions, improving the overall processing efficiency and completion effect of the model, and further improving the semantic completion result in terms of depth details and accuracy.

[0118] In some embodiments, after adjusting the feature values of the mutation region involved in the calculation of the attention weight according to the first modulation factor, the calculation formula of the attention weight is as follows:

[0119]

[0120] where ||·|| represents a norm function, λ represents an enhancement coefficient, the value range of λ is [0.1, 1], and λ can determine its optimal value through an experiment or a training process to control the enhancement degree of the eigenvalue of the mutation region; represents the gradient amplitude of position i on the original depth feature map, represents the first modulation factor.

[0121] The above formula multiplies the modulation factor related to the gradient amplitude on the basis of the original eigenvalue z i , so that the position with a more intense depth change (a larger gradient amplitude) has a greater influence on the calculation of the attention weight, thereby further highlighting the importance of the mutation region.

[0122] After adjusting the eigenvalue of the flat region participating in the calculation of the attention weight according to the second modulation factor, the calculation formula of the attention weight is as follows:

[0123]

[0124] where I(i) represents the flat region indicator function value corresponding to position i, I(i) is a binary function, and if the depth variance σ 2 (D i ) of the neighborhood of position i is less than the variance threshold , the value of I(i) is 1, otherwise the value of I(i) is 0, β is a decay function, the value range of β is [0.1, 0.3], and 1-β·I(i) represents the second modulation factor. β can determine its appropriate value through an experiment or training to control the decay degree of the eigenvalue of the flat region. The above formula subtracts the decay term related to the flat region indicator function on the basis of the original eigenvalue z i , so that the position belonging to the flat region (with a smaller depth variance) has a relatively reduced influence on the calculation of the attention weight, thereby realizing the reasonable distribution of attention on the flat region.

[0125] By the above scheme, different regions can be processed differently according to the depth variation. Specifically, in the mutation region, the feature importance of the complex depth variation region is further highlighted through the modulation factor related to the gradient amplitude, so that the model can more accurately capture the key information of these regions and improve the accuracy of completion; in the flat region, the weight of the feature value in the attention calculation of the flat region is reasonably reduced through the attenuation term based on the depth variance judgment, avoiding the model from investing too many resources in simple regions, optimizing the attention allocation strategy, and improving the adaptability and completion effect of the semantic completion method for regions with different depth characteristics.

[0126] In some embodiments, as shown in Figure 2 the semantic completion result is generated by predicting the semantic category of each pixel point on the second scene image using a convolution operation, including:

[0127] Step S201: The semantic category number and convolution value of each pixel point are output by using the convolution operation, and the convolution value is converted into a probability distribution by a softmax activation function. The output dimension of the softmax activation function is consistent with the predicted semantic category number, and the probability value corresponding to each pixel point represents the probability that the pixel point belongs to a certain semantic category. The semantic category with the maximum probability value is taken as the predicted semantic category of the current pixel point.

[0128] Step S202: All pixel points on the second scene image are traversed, and the above operation is repeated to generate the semantic completion result.

[0129] The above scheme uses a convolution operation combined with a softmax activation function to predict the semantic category, fully utilizes the ability of a convolutional neural network to extract local features of an image, and can effectively infer the semantic information of each pixel point from the image features. The convolution value is converted into a probability distribution by a softmax function, so that the prediction result has a probabilistic meaning and can intuitively reflect the possibility of each pixel point belonging to different semantic categories, thereby improving the accuracy and reliability of semantic category prediction.

[0130] In some embodiments, before the step of taking the semantic category with the maximum probability value as the predicted semantic category of the current pixel point, there is a step of judging whether the difference between the maximum probability value and other probability values is greater than a preset error range. If yes, the semantic category with the maximum probability value is taken as the predicted semantic category of the current pixel point. If not, as shown in Figure 3 the following steps are performed:

[0131] Step S301: An optimal probability set G i corresponding to the current pixel point is constructed i , which contains the maximum probability value of the current pixel point and the probability values whose differences from the maximum probability value are within a preset error range.i each probability value in the set G corresponds to a semantic category;

[0132] Step S302: Obtain the maximum probability value of the semantic category of the neighborhood pixel point corresponding to the current pixel point, and filter out the maximum probability value of the neighborhood pixel point with the semantic category overlapping with G i The maximum probability value of the neighborhood pixel point with the semantic category overlapping with G i is re-calculated, and the semantic category corresponding to the probability value with the maximum average value is taken as the predicted semantic category of the current pixel point.

[0133] In step S301, the preset error range is a threshold range for judging the difference degree of the maximum probability value and other probability values, and if the difference between the maximum probability value and at least one other probability value is within the range, it indicates that the semantic category prediction of the pixel point has a certain uncertainty.

[0134] The preferred probability set G i is a set containing the maximum probability value of the current pixel point and the probability value with the difference within the preset error range, each probability value in the set corresponds to a semantic category, and is used for further filtering the possible semantic category when the prediction has uncertainty.

[0135] The neighborhood pixel point refers to the pixel point within a certain range around the current pixel point, and the neighborhood size is usually set according to actual needs, such as the pixel points within a 3x3 or 5x5 pixel region.

[0136] The semantic category overlap refers to the case that the semantic category corresponding to the maximum probability value of the neighborhood pixel point is the same as the semantic category in the preferred probability set G i .

[0137] Through the above scheme, when the semantic category of a single pixel point is difficult to be determined by its own probability value, the semantic information of the neighborhood pixel point is introduced, the continuity and correlation of the semantic category in space are utilized, the possible semantic category is re-evaluated and filtered, the accuracy and reliability of the semantic category prediction are improved, the misjudgment caused by the single pixel point feature blur or noise interference is avoided, and the semantic completion result is more reasonable and coherent.

[0138] As shown in FIG. Figure 4 , in some embodiments, the method further comprises:

[0139] S401: After obtaining the semantic completion result, divide the second scene image into different category regions according to the predicted semantic category of the pixel point;

[0140] S402: Smooth the edge of each category region using a Gaussian filter, and complete the hollow part of each category region using a texture propagation algorithm.

[0141] As shown in Figure 5 completing the hollow part of each category region specifically includes:

[0142] S501: using a Sobel operator to calculate the gradient amplitude of each spatial position of the original depth feature map, and arranging the gradient amplitudes in ascending order;

[0143] S502: obtaining a corresponding number of pixel points with the smallest gradient amplitude in the same category region in the current second scene image according to the number of pixel points contained in the hollow part, and using the obtained pixel points to complete the hollow part.

[0144] In step S401, the category region refers to a continuous region with the same semantic category divided from the second scene image according to the predicted semantic category of the pixel points, such as a wall region, a floor region, a furniture region, etc.

[0145] In step S402, the Gaussian filter is a linear smoothing filter, which suppresses the high-frequency components (such as edge noise) in the image by weighted average processing, thereby achieving smoothing processing of the image edges. The filtering effect is determined by the size of the filter and the standard deviation σ, and in this embodiment, a 5x5 filter can be used, σ = 1.

[0146] The texture propagation algorithm is an algorithm for filling the hollow or missing regions in the image based on the existing texture information in the image, which propagates the information of the surrounding similar texture region to the hollow region, so that the filled region is naturally integrated with the surrounding environment.

[0147] The hollow part refers to the region of pixel points in the semantic completion result that is not correctly predicted or lacks semantic information, which is usually manifested as a region that is not coherent with the surrounding semantic category.

[0148] The above scheme optimizes the post-processing of the semantic completion result, significantly improving the visual quality of the completion result. The Gaussian filter smoothes the edges of the category region, eliminating the jagged and incoherent phenomenon, and making the transition between different category regions more natural. The texture propagation algorithm based on gradient amplitude screening of pixel points completes the hollow part, and the region where the selected pixel points are located has a gentle depth change and relatively consistent texture. Using the obtained pixel points to complete the hollow part makes the texture of the hollow region consistent with that of the surrounding same category region, avoiding the conspicuousness of the hollow region, making the entire semantic completion result more complete, natural and real, and improving the user's visual experience of the completion result.

[0149] In some embodiments, the method further comprises:

[0150] Collect multiple indoor scene images with semantic annotations to obtain an indoor scene image dataset;

[0151] Input the indoor scene image dataset into a pre-trained model, train the model through a composite loss function, and make the model learn semantic features and depth features at the same time, wherein the composite loss function comprises a semantic segmentation loss and a depth estimation loss, and the calculation formula of the composite loss function is as follows:

[0152] L = xL sem +yL depth ;

[0153] wherein L represents the composite loss, x and y represent learnable parameters and are dynamically adjusted through back propagation, L sem represents the semantic segmentation loss, and L depth represents the depth estimation loss.

[0154] In this embodiment, the indoor scene image with semantic annotations refers to an indoor scene image containing explicit semantic class labels, each pixel point corresponds to a known semantic class, and is used for model training and evaluation.

[0155] The indoor scene image dataset is a set composed of multiple indoor scene images with semantic annotations, provides sufficient sample data for model training, covers indoor scenes of different types and different layouts, and improves the generalization ability of the model.

[0156] The pre-trained model refers to a model pre-trained on a large-scale general image dataset (such as ImageNet), which already has certain feature extraction capability, and is used as the initial parameters of the model of the present application, which can accelerate the convergence speed of the model and improve the performance of the model.

[0157] The composite loss function refers to a loss function composed of multiple loss functions, which includes the semantic segmentation loss and the depth estimation loss in the present application, and is used to optimize the performance of the model in both semantic class prediction and depth information estimation tasks.

[0158] The semantic segmentation loss (L sem ) is a loss function for measuring the difference between the predicted semantic class of the model and the real semantic annotation, and common ones include cross-entropy loss function, etc.

[0159] The depth estimation loss (L depth ) is a loss function for measuring the difference between the predicted depth information of the model and the real depth information, such as mean square error loss function, etc.

[0160] The learnable parameters x and y in the composite loss function are parameters for adjusting the weights of the semantic segmentation loss and the depth estimation loss, and their values are dynamically adjusted through back propagation during the model training process to balance the learning importance of the two tasks.

[0161] Backpropagation is an algorithm for training neural networks by calculating the gradient of the loss function with respect to the network parameters, propagating the gradient from the output layer to the input layer, and adjusting the network parameters according to the gradient descent method to minimize the loss function.

[0162] Taking cross-entropy loss as an example, for each pixel point, the cross-entropy of the predicted semantic class and the real semantic class is calculated:

[0163]

[0164] where C is the number of semantic classes, y c is the real semantic class label (one-hot encoding), p c is the predicted probability of belonging to class c. Suppose at a certain pixel point, the real class is "wall", the corresponding y 墙壁 = 1, and the other classes are 0. The predicted probability p 墙壁 = 0.8, then the semantic segmentation loss of this pixel point is:

[0165] L sem = -1 x log 0.8 ≈ 0.223;

[0166] For depth estimation loss, L1 loss is used. Suppose the predicted depth value is d pred , the real depth value is d true , then the L1 loss is:

[0167] L depth = |d pred -d true |.

[0168] If the predicted depth value is 2.5 meters and the real depth value is 2.0 meters, then the L1 loss is:

[0169] L depth = |2.5-2.0| = 0.5;

[0170] Suppose in the total loss function, x = 0.7, y = 0.3, then the total loss is:

[0171] L = 0.7 x 0.223 + 0.3 x 0.5 ≈ 0.156 + 0.15 = 0.306;

[0172] Then the model can be trained using the Adam optimization algorithm, with an initial learning rate of 0.001, x = 0.9, y = 0.999. During training, the network parameters are adjusted according to the value of the loss function. When the value of the loss function no longer decreases significantly, the model is considered to have converged. By continuously adjusting the parameters, the value of the loss function is reduced, and the performance of the model in semantic completion is improved.

[0173] The above scheme is trained by using an indoor scene image dataset with semantic annotation, and a composite loss function is used to optimize the semantic segmentation and depth estimation tasks at the same time, so that the model can learn accurate semantic and depth features at the same time, and improve the joint understanding ability of the model to semantic and depth information. The use of pre-trained model accelerates the convergence process of the model, reduces the training time and data requirements. The dynamic adjustment of learnable parameters enables the model to automatically balance the learning weights of the two tasks according to the actual situation during training, improving the overall performance of the model.

[0174] As shown in Figure 6 and Figure 7 , suppose there is an indoor scene image containing a staircase area. In the feature extraction stage, the depth feature map and the semantic feature map are obtained after the above steps, and their sizes are both 32x32x512. Then the two feature maps are spliced in the channel dimension to obtain a spliced feature map with a size of 32x32x1024, and the spliced feature map is input into the adaptive depth perception attention mechanism module.

[0175] In the adaptive depth perception attention mechanism module, first, a 1x1 convolution layer is used to process the spliced feature map, reducing the channel number to 512. The output formula of the convolution layer is as follows:

[0176] Z = Conv(F 拼接 );

[0177] Where F 拼接 is the spliced feature map, and Z is the output feature map of the convolution layer.

[0178] Suppose at a pixel position i in the staircase area, the feature value z i output by the convolution layer is 1.5, and the feature value sum of other positions is calculated as follows:

[0179]

[0180] The attention weight at this position can be represented as follows:

[0181]

[0182] Then feature optimization and fusion can be performed, which is to fuse the feature map processed by the adaptive depth perception attention mechanism with the original feature map, using the element-by-element addition method to enhance the key information in the feature map and suppress unimportant information.

[0183] Suppose the feature value of the original feature map at a certain position is F 原始 = [1.2, 0.8, 1.5], and the feature map processed by the attention mechanism is F注意力 = [0.5, 0.3, 0.7], the calculation formula of the fused feature value is as follows:

[0184] F 融合 = F 原始 + F 注意力 = [1.2 + 0.5, 0.8 + 0.3, 1.5 + 0.7] = [1.7, 1.1, 2.2].

[0185] Then semantic completion can be performed, specifically based on the optimized feature map, using upsampling and convolution operations for semantic completion. Gradual upsampling restores the feature map resolution to the same size as the input image, and uses convolution operations to predict the semantic category of each pixel point to generate a semantic completion result. A multi-scale upsampling strategy is adopted to ensure the details and accuracy of the completion result.

[0186] Suppose now that a semantic completion needs to be performed on an indoor scene containing stairs. The depth of the stair area changes greatly, and in traditional methods, it is prone to inaccurate completion. When the system of the present application processes this scene, first, the depth features and semantic features are obtained through the feature extraction module. Then, in the adaptive depth perception attention mechanism module, the attention weight of the stair area is calculated to be high because the depth of this area changes greatly and is important for overall scene understanding. After the feature optimization and fusion processing steps, the features of the stair area are enhanced. In the semantic completion generation stage, the optimized feature map is used to complete the following steps:

[0187] Suppose that after feature optimization and fusion, the obtained feature map size is 32x32x512. In the semantic completion generation stage, first, multi-scale feature maps are generated, and features of different levels are extracted, specifically marked as follows:

[0188] F1: resolution 128x128, channel number 256;

[0189] F2: resolution 64x64, channel number 512;

[0190] F3: resolution 32x32, channel number 1024;

[0191] F4: resolution 16x16, channel number 2048;

[0192] Starting from F4, it is upsampled using bilinear interpolation to increase the resolution of F4 to 32x32, obtaining Then add and F3 element by element to obtain the fused feature map F 34 , the calculation formula is as follows:

[0193]

[0194] Then F 34 is up-sampled to 64x64 to obtain F and F is element-wise added to F2 to obtain the fused feature map F 234 , and the calculation formula is as follows:

[0195]

[0196] Then F 234 is up-sampled to 128x128 to obtain F and F is element-wise added to F1 to obtain the fused feature map F 1234 , and the calculation formula is as follows:

[0197]

[0198] Finally, F 1234 is up-sampled to the original image size 256x256, and a convolution operation is performed on the up-sampled image of the same size as the original image to predict the probability of each pixel belonging to a certain semantic category, and the calculation formula is as follows:

[0199] P = Conv(UpSample(F 1234 ));

[0200] where P is the probability of a pixel belonging to a certain semantic category, UpSample(·) represents the up-sampling operation, which is a process of restoring a low-resolution feature map to the same size (256x256) as the original image, and Conv(·) represents the convolution operation, which is a process of sliding calculation on the up-sampled feature map using a convolution kernel to extract local features and predict semantic categories.

[0201] Then, through the softmax activation function, the output value is converted to a probability distribution:

[0202] P softmax = Softmax(P);

[0203] where Softmax(·) represents the softmax activation function, which is a mathematical function that converts the original prediction value to a probability distribution. P softmax represents the probability distribution matrix obtained after the softmax activation function processing. Its dimension is also [256, 256, C], where each element P softmax(i,j,c) represents the probability that the pixel point at coordinate position (i,j) in the original image belongs to the c-th semantic category. Through the probability distribution, the most probable semantic category (i.e., the category with the maximum probability value) of each pixel point can be directly determined, and finally the semantic completion result is generated. It should be noted that the pixel point at coordinate position (i,j) in this application has the same meaning as the position i described in the foregoing, and the foregoing uses the expression of position i for simplification of description.

[0204] Specifically, for each pixel point in the feature map, the score value of each semantic category is input into the softmax activation function. Assuming that in the staircase scene, a pixel point is located at the edge of the staircase, after the convolution operation, the score value of each semantic category of the pixel point is shown in Table 1 below (assuming there are only 5 categories: floor, wall, staircase, door, and window):

[0205] Table 1

[0206] Category Floor Wall Stair Door Window Score value 1.2 0.8 2.5 0.3 0.1

[0207] Then, the softmax activation function is applied to calculate all the score values in Table 1 respectively, and the calculation formula is as follows:

[0208] Floor: e 1.2 ≈3.32;

[0209] Wall: e 0.8 ≈2.23;

[0210] Staircase: e 2.5 ≈12.18;

[0211] Door: e 0.3 ≈1.35;

[0212] Window: e 0.1 ≈1.11.

[0213] Then, the exponential sum is calculated, and the calculation formula is as follows:

[0214]

[0215] Then, the probability value of each predicted category at the current pixel position is calculated, and the calculation formula is as follows:

[0216]

[0217]

[0218] Therefore, we can conclude that the probability distribution of pixels belonging to each semantic category is as follows: floor approximately 0.164, wall approximately 0.110, stairs approximately 0.603, door approximately 0.067, and window approximately 0.056. Clearly, the pixel has the highest probability of belonging to "stairs" (0.603), so the predicted semantic category for this pixel is "stairs". By performing this calculation and classification on each pixel in the image, the semantic completion result of the entire indoor scene is finally generated.

[0219] If the staircase area in the semantic prediction map after semantic completion still has some jagged edges and holes, this application also uses a Gaussian filter to smooth the edges. Specifically, for each pixel in the semantic prediction map, the weighted average of its neighboring pixels is calculated, resulting in smoother edges and reduced jaggedness. For hole filling, a texture synthesis-based method can be used. Texture samples are extracted from the known parts of the staircase area, and textures are synthesized for the hole areas based on these samples. Specifically, a texture propagation algorithm can be used to fill the holes gradually from the edge. Each time a hole is filled, an appropriate texture block is selected based on the direction and color of the surrounding textures to ensure that the filled staircase area blends naturally with its surrounding environment.

[0220] like Figure 8 As shown, in a second aspect, this application also provides a semantic completion device, the device comprising:

[0221] An image input module is used to receive a monocular indoor scene image, preprocess the monocular indoor scene image, and obtain a first scene image;

[0222] The feature extraction module is used to extract the original feature map of the first scene image through a convolutional neural network. The original feature map includes a depth feature map and a semantic feature map. The depth feature map and the semantic feature map are concatenated by channel dimension to form a concatenated feature map.

[0223] An adaptive deep-perception attention mechanism module is used to process the concatenated feature map through a learnable convolutional layer to obtain the feature values ​​corresponding to each spatial coordinate position on the concatenated feature map. Based on the coupling relationship between deep features and semantic features, normalized attention weights are calculated according to the feature values. The feature values ​​corresponding to each spatial coordinate position on the concatenated feature map are weighted and optimized according to the attention weights to obtain an attention feature map. The attention feature map and the original feature map are added and fused element by element to obtain a fused feature map.

[0224] The semantic completion module is configured to restore the fused feature map to the same size of the monocular indoor scene image received in step S1 by an up-sampling operation to obtain a second scene image, and to generate a semantic completion result by predicting a semantic category of each pixel point on the second scene image by using a convolution operation.

[0225] The result optimization and post-processing module is configured to perform edge smoothing and hole filling operations on the semantic completion result.

[0226] In a third aspect, the present application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the semantic completion method of a monocular image based on an attention mechanism according to the first aspect of the present application.

[0227] The computer readable storage medium can be a volatile memory or a non-volatile memory, or both.

[0228] The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD ROM).

[0229] The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synclink dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The computer-readable storage media described in the embodiments of the present invention are intended to include these and any other suitable types of memory.

[0230] like Figure 9 As shown, in a fourth aspect, the present invention provides an electronic device 10, including a processor 101 and a storage medium 102, wherein a computer program is stored on the storage medium, and the computer program, when executed by the processor, implements the semantic completion method for monocular images based on an attention mechanism as described in the first aspect of the present invention.

[0231] In some embodiments, the processor can be implemented by software, hardware, firmware or a combination thereof, and can use at least one of circuit, single or multiple Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, microprocessor, so that the processor can execute part or all of the steps or any combination of steps in the semantic completion method of monocular image based on attention mechanism in various embodiments of the present application.

[0232] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of the present application, the patent protection scope of the present application should not be limited. Any technical solutions obtained by replacing or modifying the equivalent structure or equivalent process based on the essential concept of the present application, using the content described in the specification and drawings of the present application, and directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are all included in the patent protection scope of the present application.

Claims

1. A semantic completion method for monocular images based on an attention mechanism, characterized in that, The method includes the following steps: S1: Receive a monocular indoor scene image, preprocess the monocular indoor scene image to obtain a first scene image; S2: Extract the original feature map of the first scene image through a convolutional neural network, wherein the original feature map includes a depth feature map and a semantic feature map; S3: Concatenate the depth feature map and the semantic feature map along the channel dimension to form a concatenated feature map; S4: The concatenated feature map is processed by a learnable convolutional layer to obtain the feature values ​​corresponding to each spatial coordinate position on the concatenated feature map. Based on the coupling relationship between deep features and semantic features, normalized attention weights are calculated according to the feature values. The feature values ​​corresponding to each spatial coordinate position on the concatenated feature map are weighted and optimized according to the attention weights to obtain the attention feature map. S5: The attention feature map and the original feature map are added and fused element by element to obtain a fused feature map; S6: The fused feature map is restored to the same size as the monocular indoor scene image received in step S1 by upsampling operation to obtain the second scene image, and the semantic category of each pixel in the second scene image is predicted by convolution operation to generate semantic completion result.

2. The semantic completion method for monocular images based on an attention mechanism as described in claim 1, characterized in that, The normalized attention weights are calculated based on the aforementioned feature values, using the following formula: Among them, w i The attention weight at position i is the concatenated feature map obtained through learnable convolutional layers in step S4, where exp(·) is the natural exponential function, and z i z represents the maximum feature value of position i belonging to a certain semantic list. k The feature value representing position i belongs to the k-th semantic category is N, where N represents the total number of semantic categories predicted for position i, and k ranges from [1, N].

3. The semantic completion method for monocular images based on an attention mechanism as described in claim 2, characterized in that, The method includes: S41: Calculate the gradient magnitude of the original depth feature map using the Sobel operator. And compare the gradient magnitude with a preset threshold τ, if If it is a mutation region, it is determined to be a flat region; otherwise, it is determined to be a flat region. S42: Configure a first modulation factor of the attention formula greater than 1 for the mutation region, adjust the feature value of the mutation region participating in the calculation of attention weight according to the first modulation factor, and configure a second modulation factor of the attention formula less than 1 for the flat region, adjust the feature value of the flat region participating in the calculation of attention weight according to the second modulation factor.

4. The semantic completion method for monocular images based on an attention mechanism as described in claim 3, characterized in that, After adjusting the feature values ​​of the abrupt change regions involved in calculating the attention weights based on the first modulation factor, the formula for calculating the attention weights is as follows: Where ||·|| represents the norm function, λ represents the enhancement coefficient, and the value of λ ranges from [0.1, 1]. This represents the gradient magnitude at position i in the original depth feature map. Indicates the first modulation factor; After adjusting the feature values ​​of the flat region involved in calculating the attention weights according to the second modulation factor, the formula for calculating the attention weights is as follows: Where I(i) represents the value of the flat region indicator function corresponding to position i, and I(i) is a binary function. If the depth variance σ of the neighborhood of position i is... 2 (D i Less than the variance threshold If I(i) is 1, then I(i) is 0 otherwise. β is the attenuation function, and the value of β is in the range of [0.1, 0.3]. 1-β·I(i) represents the second modulation factor.

5. The semantic completion method for monocular images based on an attention mechanism as described in claim 1, characterized in that, Using convolution operations to predict the semantic category of each pixel in the second scene image, the semantic completion results are generated as follows: The convolution operation outputs the number of semantic categories and the convolution value for each pixel. The convolution value is then converted into a probability distribution using the softmax activation function. The output dimension of the softmax activation function is the same as the number of predicted semantic categories. The probability value corresponding to each pixel represents the probability that the pixel belongs to a certain semantic category. The semantic category with the highest probability value is taken as the predicted semantic category of the current pixel. Iterate through all pixels in the second scene image and repeat the above operation to generate semantic completion results.

6. The semantic completion method for monocular images based on an attention mechanism as described in claim 1, characterized in that, Before using the semantic category with the highest probability value as the predicted semantic category of the current pixel, the following step is also included: determining whether the difference between the highest probability value and other probability values ​​is greater than a preset error range. If so, the semantic category with the highest probability value is used as the predicted semantic category of the current pixel; otherwise, the following steps are performed: Construct the optimal probability set G corresponding to the current pixel. i G i G contains the maximum probability value of the current pixel and the probability value that the difference between it and the current pixel is within a preset error range. i Each probability value in the equation corresponds to a semantic category; Obtain the maximum probability value of the semantic category of the neighboring pixels corresponding to the current pixel, and filter out those that are similar to G. i The maximum probability value of neighboring pixels with overlapping semantic categories is used to divide the pixels according to semantic categories. The average of the maximum probability values ​​of each semantic category is then recalculated, and the semantic category corresponding to the probability value with the largest average is taken as the predicted semantic category of the current pixel.

7. The semantic completion method for monocular images based on an attention mechanism as described in claim 1, characterized in that, The method further includes: After obtaining the semantic completion result, the second scene image is divided into different category regions according to the predicted semantic category corresponding to the pixel; The edges of each category region are smoothed using a Gaussian filter, and the hole portions of each category region are filled using a texture propagation algorithm; Filling in the gaps in each category area specifically includes: The Sobel operator is used to calculate the gradient magnitude at each spatial location of the original depth feature map, and the gradient magnitudes are arranged in ascending order. Based on the number of pixels contained in the hole, obtain a corresponding number of pixels with the smallest gradient magnitude in the same category region of the current second scene image, and use the obtained pixels to fill in the hole.

8. The semantic completion method for monocular images based on an attention mechanism as described in claim 1, characterized in that, The method further includes: Multiple semantically labeled indoor scene images were collected to obtain an indoor scene image dataset; An indoor scene image dataset is input into a pre-trained model, which is then trained using a composite loss function. This allows the model to learn both semantic and depth features simultaneously. The composite loss function includes semantic segmentation loss and depth estimation loss, and its calculation formula is as follows: L=xL sem +yL depth ; Where L represents the composite loss, and x and y represent learnable parameters that are dynamically adjusted through backpropagation. sem L represents the semantic segmentation loss. depth This indicates the depth estimation loss.

9. A computer-readable storage medium storing computer program instructions thereon, characterized in that, The computer program instructions, when executed by a processor, implement the method as described in any one of claims 1-8.

10. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-8.

Images