Indoor scene spatial information recognition detection framework based on multi-task feature fusion sharing

Abstract

Based on the multi-task feature fusion sharing indoor scene space information recognition detection framework, through the design and embedding of the depth estimation and instance segmentation exclusive expert module in the multi-task learning framework, the adaptive spatial feature fusion module is introduced in the depth estimation model and instance segmentation model, and the cross-task feature fusion perception module is introduced after the two task models, which respectively solves the "seesaw" problem existing in the traditional multi-task learning framework, the feature extraction error problem caused by the direct fusion of different size feature images in the pyramid pooling feature fusion module, and the conflict and noise problem caused by the direct fusion of different types of task model features, and realizes the cross-task collaborative reasoning function of the model, thereby breaking the information "black box" problem existing in the depth learning, and further improving the learning accuracy of the framework in the depth estimation and instance segmentation tasks.

Description

Technical Field

[0001] This invention relates to the fields of instance segmentation technology, depth estimation technology and multi-task joint learning technology, and in particular to an indoor scene spatial information recognition and detection framework based on multi-task feature fusion and sharing. Background Technology

[0002] With the rapid development of artificial intelligence, robotics, and augmented / virtual reality (AR / VR), the intelligent perception and understanding of indoor environments by robots has become crucial. Indoor scene spatial information recognition and detection refers to the ability of computer vision systems to analyze various geometric and semantic information within indoor scenes. Its core tasks typically include: scene segmentation, classifying each pixel in an image and identifying structural elements such as walls, floors, ceilings, doors, and windows; object detection and recognition, locating and identifying objects such as furniture and appliances in the scene; and depth estimation, predicting the distance information from each pixel to the camera from monocular or RGB images.

[0003] Traditional solutions typically treat these tasks as independent "single-task" problems, designing and training dedicated neural network models for each task. However, there are strong intrinsic connections between different visual information in indoor scenes. For example, the position of an object is often closely related to the scene layout (e.g., a chair is on the floor or near a wall) and instance segmentation regions (e.g., a sofa is above the floor). Independent single-task models cannot effectively utilize these intrinsic connections between tasks. For instance, scene layout information can provide strong contextual priors for object detection (objects are usually located inside the room rather than on a wall), and accurate object detection results, in turn, can help optimize the boundaries of the scene layout. For complex indoor scenes, single-task models lack this cross-task collaborative reasoning ability, so they cannot integrate complementary cues provided by other tasks. Therefore, single-task models are prone to low accuracy or even failure when faced with challenging and complex environmental samples. In multi-task learning, since each task has different feature representations and objective functions, and there may be complex connections and interactions between different tasks, achieving collaboration and interaction between tasks to obtain better learning results is a challenging problem. Summary of the Invention

[0004] The purpose of this invention is to provide an indoor scene spatial information recognition and detection framework based on multi-task feature fusion and sharing.

[0005] The technical solution adopted by this invention to solve its technical problem is:

[0006] This invention provides an indoor scene spatial information recognition and detection framework based on multi-task feature fusion and sharing, comprising:

[0007] The dedicated expert module extracts image features simultaneously from the shared expert module and the dedicated expert for each task when the input image enters the model, and fuses the two different image features through a fully connected layer (FC) and the activation function Softmax.

[0008] An adaptive spatial feature fusion module is used to fuse feature images of four different sizes with pixel ratios of 16×16, 32×32, 64×64, and 128×128, output by the pyramid pooling feature fusion module.

[0009] A cross-task feature fusion perception module, composed of a channel attention mechanism and a spatial attention mechanism, is used to adaptively interact features between two different tasks—depth estimation and instance segmentation—through the attention mechanism, thereby fully utilizing the complementary information between the two tasks. The extracted image features are first passed through the channel attention module of the cross-task feature fusion perception module. The channel attention mechanism extracts global information of the image features through global average pooling and max pooling operations. Then, the importance weights of each channel are learned through a fully connected layer and superimposed to obtain the channel feature weights CAM(F) used to emphasize those more meaningful to the current task. D ) and CAM (F S Then, these channel feature weights are combined with the original image features F for another task. D or F S Perform element-wise multiplication to obtain new image features F' D and F' S .

[0010] Furthermore, the formula for the input feature information of the dedicated expert module is expressed as follows:

[0011] ;

[0012] Among them, S D (x) is a selected matrix consisting of all selected vectors, which include the features E extracted by the shared expert module. T (S,a) Features E extracted by the dedicated expert module T (D,a) ;

[0013] This allows each task to obtain a weighted average of multiple expert modules. Finally, the fused feature images extracted by the dedicated expert module and the shared expert module for each task are output to the instance segmentation model and the depth estimation model, respectively. The output expression is as follows:

[0014] ;

[0015] Where, m S and m D represents the number of shared expert modules and the number of dedicated experts for the depth estimation task, respectively, and d represents the dimension of the input image features.

[0016] Furthermore, the four different sizes of feature image fusion expressions of the adaptive spatial feature fusion module are as follows:

[0017] First layer: ;

[0018] Second layer: ;

[0019] Third layer: ;

[0020] Fourth layer: ;

[0021] Among them, Y c X represents the total feature image after fusing the four sub-feature images of layer C; a,b α represents the feature value of the sub-feature image of layer a mapped to the sub-feature image of layer b; α represents the feature image spatial weight when layer 1 is mapped to other layers; β represents the feature image spatial weight when layer 2 is mapped to other layers; γ represents the feature image spatial weight when layer 3 is mapped to other layers; and χ represents the feature image spatial weight when layer 4 is mapped to other layers.

[0022] Furthermore, new image features F' are obtained through the cross-task feature fusion perception module. D and F' S The expressions are as follows:

[0023] ;

[0024] Subsequently, F' D and F' S Entering the spatial attention module of CFFP, the spatial attention mechanism extracts spatial information of image features by concatenating max pooling and global average pooling and then connecting a convolutional layer at the back end. It then generates a spatial attention map to emphasize more important spatial regions in the feature map, thereby obtaining the spatial feature weights SAM (F'). D ) and SAM (F' S Finally, similar to the previous operations, the obtained spatial feature weights SAM (F') are calculated. D ) and SAM (F' S ) respectively with the original image features F of another task D or F S Perform element-wise multiplication to obtain the output image features F”.D and F” S , where F” D and F” S The expressions are as follows:

[0025] ;

[0026] The cross-task feature fusion perception module can interactively share the extracted features through channel attention and spatial attention mechanisms.

[0027] In summary, the beneficial effects of the present invention are as follows:

[0028] This invention presents an indoor scene spatial information recognition and detection framework based on multi-task feature fusion and sharing. Through code innovation—designing and introducing dedicated expert modules for depth estimation and instance segmentation tasks within a multi-task learning framework, introducing an adaptive spatial feature fusion module within the depth estimation and instance segmentation models, and introducing a cross-task feature fusion perception module after the two task models—it solves the common "seesaw" problem in traditional multi-task learning frameworks, the feature extraction error caused by directly fusing feature images of different sizes in the pyramid pooling feature fusion module, and the conflict and noise problems that easily arise when directly fusing feature information extracted from different types of task models. Furthermore, it achieves cross-task collaborative inference functionality, thus breaking the information "black box" problem prevalent in deep learning. Therefore, the instance segmentation and depth estimation multi-task feature fusion interactive learning framework proposed in this invention exhibits superior spatial instance segmentation accuracy and spatial depth information estimation performance compared to other previous methods or frameworks. Attached Figure Description

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0030] Figure 1 This is the network structure of the Cross-Task Feature Fusion Perception Module (CFFP) proposed in this invention;

[0031] Figure 2 This is the multi-task learning network structure of the dedicated expert module introduced in this invention;

[0032] Figure 3 This is the pyramid pooling feature fusion network structure of the adaptive spatial feature fusion module (ASFF) proposed in this invention;

[0033] Figure 4 This is the overall network structure of the indoor scene spatial information recognition and detection framework based on multi-task feature fusion and sharing proposed in this invention. Detailed Implementation

[0034] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0035] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" or "linked" should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral connection; it can refer to a mechanical connection or an electrical connection; it can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0036] Please see Figures 1-4 This invention provides an indoor scene spatial information recognition and detection framework based on multi-task feature fusion and sharing, comprising:

[0037] The dedicated expert module, when the input image enters the model, simultaneously extracts image features from the shared expert module and the dedicated expert for each task. These two different image features are then fused using a fully connected layer (FC) and the softmax activation function. The formula for expressing the input feature information of the depth estimation task model is as follows:

[0038] ;

[0039] Among them, S D (x) is a selected matrix consisting of all selected vectors, which include the features E extracted by the shared expert module. T (S,a) Features E extracted by the dedicated expert module T (D,a) .

[0040] This allows each task to obtain a weighted average of multiple expert modules. Finally, the fused feature images extracted by the dedicated expert module and the shared expert module for each task are output to the instance segmentation model and the depth estimation model, respectively. The output expression is as follows:

[0041] ;

[0042] Where, m S and m D represents the number of shared expert modules and the number of dedicated experts for the depth estimation task, respectively, and d represents the dimension of the input image features;

[0043] Since the feature image processing process extracted by each expert module in the segmentation task is the same as that in the depth estimation task, the final output G is obtained. S (x);

[0044] Since each task's dedicated expert module is only used to extract the features required for that task, its extraction accuracy is not affected by other tasks, thus effectively solving the "seesaw" problem in multi-task learning models.

[0045] In one embodiment, the indoor scene spatial information recognition and detection framework based on multi-task feature fusion and sharing of the present invention further includes an adaptive spatial feature fusion module (ASFF) embedded in the connection channel between the pyramid pooling feature fusion module and the decoder of the depth estimation task model and the instance segmentation task model. The ASFF module can fuse features of four different sizes of feature images with pixel ratios of 16×16, 32×32, 64×64, and 128×128 output by the pyramid pooling feature fusion module. For the depth estimation model, the feature image fusion expressions of its four-layer ASFF module are as follows:

[0046] First layer: ;

[0047] Second layer: ;

[0048] Third layer: ;

[0049] Fourth layer: ;

[0050] Among them, Y c X represents the total feature image after fusing the four sub-feature images of layer C; a,b α represents the feature value of the sub-feature image of layer a mapped to the sub-feature image of layer b; α represents the feature image spatial weight when layer 1 is mapped to other layers; β represents the feature image spatial weight when layer 2 is mapped to other layers; γ represents the feature image spatial weight when layer 3 is mapped to other layers; and χ represents the feature image spatial weight when layer 4 is mapped to other layers.

[0051] The above formulas yield four levels of Y, all with equal dimensions. This aligns the four different-sized feature images extracted by the pyramid pooling feature fusion module. The ASFF module fuses these four sets of feature images of different sizes, enabling information exchange between them and resulting in four sets of feature information with different content but the same size. This is then used for feature fusion. This effectively reduces the error caused by fusing features with large size differences, thus solving the problem of feature extraction errors caused by direct fusion of feature images with different sizes, and improving the detection performance of the depth estimation model. The feature image processing of the instance segmentation model is the same as described above.

[0052] In one embodiment, the indoor scene spatial information recognition and detection framework based on multi-task feature fusion and sharing of the present invention further includes a cross-task feature fusion perception module (CFFP) embedded in the front end of the output heads of the depth estimation model and the instance segmentation model. This module consists of a channel attention mechanism and a spatial attention mechanism. The purpose of the CFFP module is to adaptively interact features between the two different tasks, depth estimation and instance segmentation, through an attention mechanism, thereby fully utilizing the complementary information between the two tasks. The image features extracted by the two task modules are first processed by the channel attention module of the CFFP. The channel attention mechanism extracts global information of the image features through global average pooling and max pooling operations. Then, the importance weights of each channel are learned through a fully connected layer and superimposed to obtain the channel feature weights CAM (F) used to emphasize the channel features more meaningful to the current task. D ) and CAM (F S Then, these channel feature weights are combined with the original image features F for another task. D or F S Perform element-wise multiplication to obtain new image features F' D and F' S , where F' D and F' S The expressions are as follows:

[0053] ;

[0054] Subsequently, F' D and F' S Entering the spatial attention module of CFFP, the spatial attention mechanism extracts spatial information of image features by concatenating max pooling and global average pooling and then connecting a convolutional layer at the back end. It then generates a spatial attention map to emphasize more important spatial regions in the feature map, thereby obtaining the spatial feature weights SAM (F'). D ) and SAM (F' S Finally, similar to the previous operations, the obtained spatial feature weights SAM (F') are calculated. D ) and SAM (F' S ) respectively with the original image features F of another task D or F S Perform element-wise multiplication to obtain the output image features F”. D and F” S , where F” D and F” S The expressions are as follows:

[0055] ;

[0056] In this way, CFFP allows features extracted from two different tasks to interact and share through channel attention and spatial attention mechanisms, enabling cross-task collaborative reasoning—that is, one task provides complementary feature cues to another. This breaks the information "black box" problem commonly found in deep learning. Furthermore, the introduction of attention mechanisms reduces noise and conflicts arising from the fusion of features from different tasks, thus significantly improving the model's detection accuracy in complex environments.

[0057] Understandably, in existing technologies, during the recognition and detection of spatial information in indoor scenes, different task models emphasize different aspects of image feature extraction. For example, depth estimation focuses on spatial distance information, while instance segmentation focuses on object contours and category information. Although their feature content and representation differ, they are interconnected. If the features extracted by the two tasks could be shared between the two task models, cross-task collaborative reasoning could be achieved, thus overcoming the common "black box" problem in deep learning. Furthermore, complementary feature cues provided by one task to another could significantly improve the model's detection accuracy in complex environments. However, because the feature information extracted by different task models differs, direct fusion can easily lead to conflicts and noise, making it difficult for the model to extract effective features for accurate recognition of spatial information in indoor scenes.

[0058] Based on this, this invention embeds a Cross-task Feature Fusion Perception Module (CFFP) composed of channel attention and spatial attention mechanisms at the front end of the output heads of the depth estimation model and the instance segmentation model. The purpose of the CFFP module is to adaptively interact features between the two different tasks of depth estimation and instance segmentation through attention mechanisms, thereby fully utilizing the complementary information between the two tasks. The image features extracted by the two task modules are first processed by the channel attention module of the CFFP. The channel attention mechanism extracts global information of image features through global average pooling and max pooling operations. Then, the importance weights of each channel are learned through fully connected layers and superimposed to obtain the channel feature weights CAM (F) used to emphasize the channel features that are more meaningful to the current task. D ) and CAM (F S Then, these channel feature weights are combined with the original image features F for another task. D or F S Perform element-wise multiplication to obtain new image features F' D and F' S , where F' D and F'S The expressions are as follows:

[0059] ;

[0060] Subsequently, F' D and F' S Entering the spatial attention module of CFFP, the spatial attention mechanism extracts spatial information of image features by concatenating max pooling and global average pooling and then connecting a convolutional layer at the back end. It then generates a spatial attention map to emphasize more important spatial regions in the feature map, thereby obtaining the spatial feature weights SAM (F'). D ) and SAM (F' S Finally, similar to the previous operations, the obtained spatial feature weights SAM (F') are calculated. D ) and SAM (F' S ) respectively with the original image features F of another task D or F S Perform element-wise multiplication to obtain the output image features F”. D and F” S , where F” D and F” S The expressions are as follows:

[0061] ;

[0062] In this way, features extracted from two different tasks can be interactively shared within CFFP through channel attention and spatial attention mechanisms, realizing the model's cross-task collaborative reasoning function, i.e., one task provides complementary feature cues to another. This breaks the information "black box" problem commonly found in deep learning. Furthermore, the introduction of the attention mechanism reduces noise and conflicts arising from the fusion of features from different tasks. Therefore, the introduction of the CFFP module can significantly improve the model's detection accuracy in complex environments. The network structure of the CFFP module is as follows: Figure 1 As shown.

[0063] Furthermore, in existing technologies, in multi-task learning, the underlying Expert module of traditional multi-task learning models is shared. Therefore, when the weight of the spatial instance segmentation task on the shared Expert module increases, its detection accuracy will rise. However, this will inevitably preempt the weight of the spatial depth information estimation task, causing its estimation performance to decline. This is the "seesaw" problem that always exists in traditional multi-task learning, where an improvement in the accuracy of one task is often accompanied by a decrease in the accuracy of another. In indoor scenarios, the robot's spatial information recognition and detection capabilities aim to simultaneously improve the accuracy of both types of tasks, balance their weight ratios, reduce the constraints between the two tasks, and enhance the ability to share and interact with feature information between them.

[0064] Based on this, this invention introduces the concept of a specific expert module in the multi-task learning framework. In addition to the shared expert module among tasks, the multi-task learning model of this invention also introduces a unique specific expert module for each task. Thus, when an input image enters the model, the shared expert module and the specific expert module for each task simultaneously extract image features, and then fuse the two different image features through a fully connected layer (FC) and the softmax activation function. The input feature information can be expressed by the formula:

[0065] ;

[0066] Among them, S D (x) is a selected matrix consisting of all selected vectors, which include the features E extracted by the shared expert module. T (S,a) Features E extracted by the dedicated expert module T (D,a) .

[0067] This allows each task to obtain a weighted average of multiple expert modules. Finally, the fused feature images extracted by the dedicated expert module and the shared expert module for each task are output to the instance segmentation model and the depth estimation model, respectively. The grid structure is as follows: Figure 2 As shown. Taking the depth estimation task as an example, its output expression is:

[0068] ;

[0069] Where, m S and m D represents the number of shared expert modules and the number of dedicated experts for the depth estimation task, respectively, and d represents the dimension of the input image features.

[0070] Since each task's dedicated expert module is only used to extract the features required for that task, its extraction accuracy is not affected by other tasks, thus effectively solving the "seesaw" problem in multi-task learning models.

[0071] On the other hand, in existing technologies, to improve the detection performance and accuracy of single-task modules within multi-task learning frameworks, a pyramid pooling module (PPM), or multi-scale feature fusion module, is often introduced. The PPM module uses four encoders to obtain four different sized feature images from the input image. These feature images contain different feature information: smaller feature images tend to retain detailed information of the input image, while larger feature images retain more contextual information, i.e., global information. By fusing these different sized feature images, the PPM module can ensure that the fused output feature image retains both the detailed and global features of the input image, thus improving the model's performance in estimating the depth of objects in the input image. However, pyramid pooling feature fusion also has many drawbacks. The most significant one is that larger feature images primarily extract global information, while smaller feature images mainly extract small targets or details. This leads to inconsistencies between feature images of different sizes. When the size difference between feature images is large, direct fusion results in significant feature extraction errors. These errors can cause robots to miss spatial details or misjudge depth information in indoor scenes, potentially leading to collisions with other objects in complex indoor environments.

[0072] Based on this, the present invention incorporates an adaptive spatial feature fusion module (ASFF) into the connection channel between the pyramid pooling feature fusion module and the decoder of the depth estimation model and the instance segmentation model. The ASFF module can perform feature fusion on feature images of different sizes, and the feature fusion formula of its fourth-layer ASFF module is expressed as follows:

[0073] ;

[0074] Among them, Y c X represents the total feature image after fusing the four sub-feature images of layer C; a,bα represents the feature value of the sub-feature image of layer a mapped to the sub-feature image of layer b; α represents the feature image spatial weight when layer 1 is mapped to other layers; β represents the feature image spatial weight when layer 2 is mapped to other layers; γ represents the feature image spatial weight when layer 3 is mapped to other layers; and χ represents the feature image spatial weight when layer 4 is mapped to other layers.

[0075] The above formulas yield four levels of Y, all with equal dimensions. This aligns the four different sized feature images extracted by the pyramid pooling feature fusion module. The ASFF module fuses these four sets of feature maps of different sizes, enabling information exchange between feature maps of different scales. This results in four sets of feature information with different extracted content but the same size, which are then further fused. The grid structure is as follows: Figure 3 As shown, this effectively reduces the error generated when fusing features with large size differences, thus solving the problem of feature extraction error caused by direct fusion of feature images with different sizes, and thereby improving the spatial information perception performance of the model.

[0076] Specifically, the indoor scene spatial information recognition and detection framework based on multi-task feature fusion and sharing of this invention is based on a multi-task learning framework with embedded dedicated expert modules. It improves the pyramid pooling feature fusion method in the depth estimation and instance segmentation task models by utilizing an adaptive spatial feature fusion module (ASFF). Simultaneously, it reduces noise and conflicts arising from the fusion of features from different tasks by introducing a cross-task feature fusion perception module (CFFP), thereby truly realizing the model's cross-task collaborative reasoning function. This improves the training accuracy of the model in both depth estimation and instance segmentation tasks, better adapting to the recognition and detection of complex indoor scene spatial information. The specific steps are as follows:

[0077] 1) Under natural lighting conditions, an image dataset of an indoor scene is collected using a depth camera, and the collected image data, camera parameters, and depth parameters are input into the multi-task framework.

[0078] 2) The input images are fed into the dedicated expert modules and shared expert modules for the two tasks respectively, and these expert modules extract the features required for each task.

[0079] 3) The features extracted by the dedicated expert module and the shared expert module are fused through a subsequent fully connected layer (FC) and the softmax activation function. Taking the depth estimation task as an example, its input information can be expressed by the formula:

[0080] ;

[0081] Where S D(x) is a selected matrix consisting of all selected vectors, which include the features E extracted by the shared expert module for the depth estimation task. T (S,a) Features E extracted by the depth estimation expert module T (D,a) .

[0082] The total feature set S for depth estimation D (x) The output is obtained through a subsequent fully connected layer (FC) and the activation function Softmax:

[0083] ;

[0084] Where, m S and m D represents the number of shared expert modules and the number of dedicated experts for the depth estimation task, respectively, and d represents the dimension of the input image features.

[0085] The feature image processing procedure extracted by each expert module in the instance segmentation task is the same as that in the depth estimation task, ultimately yielding the output G. S (x), then G D (x) and G S (x) are input into the subsequent depth estimation model and instance segmentation model, respectively.

[0086] 4) For depth estimation tasks, the output G of the multi-task learning framework will be... D (x) is used as the total input image information. The Pyramid Pooling Module (PPM) in the depth estimation model has four different encoders (Encoder1, Encoder2, Encoder3, Encoder4), which respectively process the input G... D (x) performs feature re-extraction and outputs feature images of four different sizes with pixel ratios of 16×16, 32×32, 64×64, and 128×128. Then, four adaptive spatial feature fusion modules (ASFF) are used to fuse and align the four feature images of different sizes. The feature image fusion expressions for the four ASFF modules are as follows:

[0087] First layer: ;

[0088] Second layer: ;

[0089] Third layer: ;

[0090] Fourth layer: ;

[0091] Among them, Y c X represents the total feature image after fusing the four sub-feature images of layer C; a,b α represents the feature value of the sub-feature image of layer a mapped to the sub-feature image of layer b; α represents the feature image spatial weight when layer 1 is mapped to other layers; β represents the feature image spatial weight when layer 2 is mapped to other layers; γ represents the feature image spatial weight when layer 3 is mapped to other layers; and χ represents the feature image spatial weight when layer 4 is mapped to other layers.

[0092] Four ASFF modules are used to fuse features of different sizes to obtain four sets of fused features of the same size: Y1, Y2, Y3, and Y4. These four sets of features are then fused through a decoder to obtain the final depth image output F. D .

[0093] 5) For instance segmentation tasks, similar to depth estimation tasks, the output G of the multi-task learning framework is first processed. S (x) is used as the total input image information. The Pyramid Pooling Module (PPM) in the instance segmentation model also has four different encoders (Encoder5, Encoder6, Encoder7, Encoder8), which respectively process the input G... S (x) performs feature re-extraction and outputs feature images of four different sizes with pixel ratios of 16×16, 32×32, 64×64, and 128×128. Then, four adaptive spatial feature fusion modules (ASFF) are used to fuse and align the four feature images of different sizes. The feature image fusion expressions for the four ASFF modules are as follows:

[0094] First layer: ;

[0095] Second layer: ;

[0096] Third layer: ;

[0097] Fourth layer: ;

[0098] Among them, y c This represents the total feature image after fusing the four sub-feature images of layer C; x a,b α represents the feature value of the sub-feature image of layer a mapped to the sub-feature image of layer b; α represents the feature image spatial weight when layer 1 is mapped to other layers; β represents the feature image spatial weight when layer 2 is mapped to other layers; γ represents the feature image spatial weight when layer 3 is mapped to other layers; and χ represents the feature image spatial weight when layer 4 is mapped to other layers.

[0099] Four ASFF modules are used to fuse features of different sizes to obtain four sets of fused features of the same size: y1, y2, y3, and y4. These four sets of features are then fused through a decoder to obtain the final segmented image output F. S .

[0100] 6) Image features F extracted by the two task modules D and F S Subsequently, the channel attention module of the Cross-Task Feature Fusion Perception Module (CFFP) is used. The channel attention mechanism extracts global information of image features through global average pooling and max pooling operations. Then, the importance weights of each channel are learned through a fully connected layer and superimposed to obtain the channel feature weights CAM (F) used to emphasize the channel features that are more meaningful to the current task. D ) and CAM (F S Then, these channel feature weights are combined with the original image features F for another task. D or F S Perform element-wise multiplication to obtain new image features F' D and F' S , where F' D and F' S The expressions are as follows:

[0101] ;

[0102] Subsequently, F' D and F' S Entering the spatial attention module of CFFP, the spatial attention mechanism extracts spatial information of image features by concatenating max pooling and global average pooling and then connecting a convolutional layer at the back end. It then generates a spatial attention map to emphasize more important spatial regions in the feature map, thereby obtaining the spatial feature weights SAM (F'). D ) and SAM (F' S Finally, similar to the previous operations, the obtained spatial feature weights SAM (F') are calculated. D ) and SAM (F' S ) respectively with the original image features F of another task D or F S Perform element-wise multiplication to obtain the output image features F”. D and F” S , where F” D and F” S The expressions are as follows:

[0103] ;

[0104] In this way, CFFP allows features extracted from two different tasks to interact and share through channel attention and spatial attention mechanisms, enabling cross-task collaborative reasoning and overcoming the information "black box" problem prevalent in deep learning. Furthermore, the introduction of attention mechanisms reduces noise and conflicts arising from the fusion of features from different tasks, thus significantly improving the model's detection accuracy in complex environments.

[0105] This invention presents an indoor scene spatial information recognition and detection framework based on multi-task feature fusion and sharing. Through code innovation—designing and introducing dedicated expert modules for depth estimation and instance segmentation tasks within a multi-task learning framework, introducing an adaptive spatial feature fusion module (ASFF) within the depth estimation and instance segmentation models, and introducing a cross-task feature fusion perception module (CFFP) after the two task models—it solves the common "seesaw" problem in traditional multi-task learning frameworks, the feature extraction error caused by directly fusing feature images of different sizes in the pyramid pooling feature fusion module, and the conflict and noise problems that easily arise when directly fusing feature information extracted from different types of task models. Furthermore, it achieves cross-task collaborative reasoning functionality, thus breaking the information "black box" problem prevalent in deep learning. Therefore, the instance segmentation and depth estimation multi-task feature fusion interactive learning framework proposed in this invention exhibits superior spatial instance segmentation accuracy and spatial depth information estimation performance compared to previous methods or frameworks.

[0106] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may utilize the disclosed technical content to make changes or equivalent variations to other fields. However, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention, without departing from the scope of the present invention, shall still fall within the protection scope of the present invention. In the description of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. For those skilled in the art, the specific meaning of the above terms in the present invention can be understood through specific circumstances.

Claims

1. A framework for indoor scene spatial information recognition and detection based on multi-task feature fusion and sharing, characterized in that, include: The dedicated expert module extracts image features simultaneously from the shared expert module and the dedicated expert for each task when the input image enters the model, and fuses the two different image features through a fully connected layer (FC) and the activation function Softmax. The fused feature images extracted by the dedicated expert module and the shared expert module for each task are respectively output to the instance segmentation model and the depth estimation model. Both the instance segmentation model and the depth estimation model include a pyramid pooling feature fusion module. The pyramid pooling feature fusion module includes four encoders and outputs feature images of four different sizes with pixel ratios of 16×16, 32×32, 64×64 and 128×128. An adaptive spatial feature fusion module is used to fuse features from four different sizes of output feature images. A cross-task feature fusion perception module, composed of a channel attention mechanism and a spatial attention mechanism, adaptively interacts features between two different tasks—depth estimation and instance segmentation—through the attention mechanism, thereby fully utilizing the complementary information between the two tasks. The depth feature image output by the depth estimation model and the segmentation feature image output by the instance segmentation model are input into the channel attention module of the cross-task feature fusion perception module. The channel attention mechanism extracts global information of image features through global average pooling and max pooling operations, then learns the importance weight of each channel through a fully connected layer, and superimposes them to obtain the channel feature weight CAM (F) used to emphasize the channel features more meaningful to the current task. D ) and CAM (F S Then, these channel feature weights are combined with the original image features F for another task. D or F S Perform element-wise multiplication to obtain new image features F' D and F' S； The new image feature F' is obtained through the cross-task feature fusion perception module. D and F' S The expressions are as follows: ； Subsequently, F' D and F' S Entering the spatial attention module of CFFP, the spatial attention mechanism extracts spatial information of image features by concatenating max pooling and global average pooling and then connecting a convolutional layer at the back end. It then generates a spatial attention map to emphasize more important spatial regions in the feature map, thereby obtaining the spatial feature weights SAM (F'). D ) and SAM(F' S Finally, similar to the previous operations, the obtained spatial feature weights SAM (F') are calculated. D ) and SAM (F' S ) respectively with the original image features F of another task D or F S Perform element-wise multiplication to obtain the output image features F”. D and F” S , where F” D and F” S The expressions are as follows: ； The cross-task feature fusion perception module can interactively share the extracted features through channel attention and spatial attention mechanisms.

2. The indoor scene spatial information recognition and detection framework based on multi-task feature fusion and sharing as described in claim 1, characterized in that: The formula for the input feature information of the dedicated expert module is expressed as follows: ； Among them, S D (x) is a selected matrix consisting of all selected vectors, which include the features E extracted by the shared expert module. T (S,a) Features E extracted by the dedicated expert module T (D,a) ; This allows each task to obtain a weighted average of multiple expert modules. Finally, the fused feature images extracted by the dedicated expert module and the shared expert module for each task are output to the instance segmentation model and the depth estimation model, respectively. The output expression is as follows: ； Where, m S and m D represents the number of shared expert modules and the number of dedicated experts for the depth estimation task, respectively, and d represents the dimension of the input image features.

3. The indoor scene spatial information recognition and detection framework based on multi-task feature fusion and sharing as described in claim 1, characterized in that: The four different sizes of feature image fusion expressions of the adaptive spatial feature fusion module are as follows: First layer: ; Second layer: ; Third layer: ; Fourth layer: ; Among them, Y c X represents the total feature image after fusing the four sub-feature images of layer C; a,b α represents the feature value of the sub-feature image of layer a mapped to the sub-feature image of layer b; α represents the feature image spatial weight when layer 1 is mapped to other layers; β represents the feature image spatial weight when layer 2 is mapped to other layers; γ represents the feature image spatial weight when layer 3 is mapped to other layers; and χ represents the feature image spatial weight when layer 4 is mapped to other layers.

Images