Indoor semantic segmentation method and device using color and depth map multi-modal fusion

Through deep learning methods, multimodal data and features are fused to color and depth images in indoor scenes. The dual-branch encoder and attention mechanism are used to solve the robustness and accuracy of semantic segmentation of indoor scenes in the prior art. , achieving more efficient semantic segmentation effect.

CN118982667APending Publication Date: 2024-11-19WUHAN UNIV

Patent Information

Application Number
CN202411023552.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The existing semantic segmentation method based on deep learning in indoor scenes due to problems such as lighting changes, object overlap and category imbalance, causing color images to be unable to effectively distinguish multiple objects, and the feature expression of depth images is insufficient, resulting in robustness and accuracy. Poor.

Method used

Deep learning method is used to fuse multimodal data and features of color images and deep images. By building a dual-branch encoder-decoder structure and attention mechanism, encoding and decoding layer by layer, combining pyramid supersupervision strategy, semantic segmentation is improved Robustness and efficiency.

Benefits of technology

It effectively improves the accuracy and robustness of indoor image semantic segmentation, reduces misidentification and category confusion, improves the accuracy and practical application of semantic segmentation, and is suitable for smart home and indoor robot recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982667A_ABST
    Figure CN118982667A_ABST
Patent Text Reader

Abstract

The invention provides an indoor semantic segmentation method and equipment using color and depth map multi-modal fusion, and the method comprises the steps: carrying out the image collection of an indoor scene through a depth camera, obtaining corresponding color and depth images, and carrying out the registration and preprocessing; the depth image adopts an HHA image; constructing a pixel-level semantic information label in the image, and establishing and dividing a test data set and a training data set; building an image semantic segmentation deep learning model for fusing multi-modal data, and extracting features from the color image and the HHA image; in a network structure of the image semantic segmentation deep learning model, two modes of a color image and an HHA image are coded layer by layer through a coding module, after multi-layer coding, fused features are combined, then layer-by-layer decoding is carried out through a decoding module, and finally a semantic segmentation result is generated. And training an image semantic segmentation deep learning model used for fusing the multi-modal data, and obtaining an image pixel level semantic automatic segmentation result based on the trained model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent indoor scene recognition, and in particular to a technical solution for indoor image semantic segmentation using multimodal fusion of color and depth images. Background Art

[0002] With the continuous development of urbanization, people's demand for indoor environment services continues to rise. Intelligent recognition of indoor scenes can assist in indoor robot applications, smart homes, indoor navigation and other fields, bringing convenience to citizens' indoor life and service experience.

[0003] Indoor images contain structures such as walls, floors, ceilings, doors, etc., as well as movable objects such as chairs, tables, and electrical appliances, which have relatively complex environmental properties. As an important part of intelligent recognition, semantic segmentation is also one of the key technologies in scene understanding technology. Semantic segmentation aims to classify each pixel in the image and help identify and segment scene elements.

[0004] For the earlier image segmentation methods, some methods achieve this effect by clustering pixels in the image into different categories according to the defined criteria. Some other methods use edge detection to identify edge lines in the image and thus identify different areas in the image. However, these methods cannot take into account the characteristics of the pixels themselves, so as to achieve the mapping of pixels to object categories. In order to further obtain the mapping of pixels to object categories, it can be achieved through hand-crafted features and machine learning algorithms. Some methods first obtain common image features such as SIFT features, and then use machine learning classification methods such as random forests or support vector machines to map pixels to objects. However, as the complexity of the scene increases, the accuracy of these methods is low. With the great success of deep learning in natural language processing and computer vision, it has become the mainstream trend of current research. Some methods use convolutional neural networks (CNNs) with encoder-decoder structures. These networks can learn the complex hierarchical features of images and provide accurate segmentation results.

[0005] However, the existing semantic segmentation methods based on deep learning methods always make it impossible to distinguish many objects using only color images due to lighting changes in indoor scenes, overlaps between objects, and unbalanced representation of object categories. Depth images, as a kind of data that can reflect the three-dimensional information of the scene, can make up for this shortcoming. Therefore, how to cleverly fuse color images and depth images and how to improve the robustness of semantic segmentation remain unsolved problems. In short, due to the large difference between the information expression patterns of color images and depth images, it is easy to lead to inappropriate feature fusion and poor performance.

[0006] For example, CN 201911391140.5 proposed a deep learning semantic segmentation method for the metal facade surface scene of a storage tank. This method failed to combine multimodal data, limiting its potential for application in more robust analysis and recognition. CN201911064946.3 proposed a two-stream semantic segmentation network for RGB images and depth images. However, this network structure has certain limitations when processing single-channel depth images, fails to fully express its features, and performs poorly in high-level feature extraction.

[0007] Therefore, in view of the existing technical difficulties, it is urgent to propose a fusion method for color images and depth images, that is, multimodal data, to improve the efficiency of image semantic segmentation in indoor scenes and have good robustness. Summary of the invention

[0008] Faced with the complex structure and texture conditions in indoor scenes, the existing methods are not robust and universal, and the efficiency of automated segmentation is low. This paper uses deep learning methods to fuse data and features of color images and depth images to improve the efficiency and accuracy of semantic segmentation of indoor images, and proposes a deep learning semantic segmentation technology solution based on the fusion of multimodal data (color images and depth images).

[0009] In order to achieve the above object, the technical solution proposed by the present invention includes an indoor semantic segmentation method using multimodal fusion of color and depth images, performing the following processing: Use a depth camera to capture images of indoor scenes, obtain corresponding color and depth images, and perform registration and preprocessing; the depth image uses an HHA image; Construct pixel-level semantic information annotation in images, and establish and divide test data sets and training data sets; A deep learning model for image semantic segmentation is constructed for fusing multimodal data, and features are extracted from color images and HHA images respectively; in the network structure of the deep learning model for image semantic segmentation, the two modalities of color image and HHA image are encoded layer by layer through an encoding module, and at each layer, the features of the color image and the depth image are fused through a fusion module to enhance feature representation; after multi-layer encoding, the fused features are merged through an addition operation, and then decoded layer by layer through a decoding module to finally generate a semantic segmentation result; Train a deep learning model for image semantic segmentation that integrates multimodal data, and obtain pixel-level semantic automatic segmentation results based on the trained model.

[0010] Moreover, the depth camera adopts an infrared camera, and decodes the single-channel depth image acquired by the infrared camera into an HHA image as a network input of the image semantic segmentation deep learning model.

[0011] Moreover, in the network structure of the image semantic segmentation deep learning model, two encoding structures are set, namely the color image encoding structure and the depth image encoding structure; each encoding structure has several encoding modules, and the intermediate features of the two encoding structures are extracted and optimized through the fusion module of the attention mechanism to obtain the fusion features of multimodal data, and the fused features are passed back to the two encoding structures to assist feature extraction.

[0012] Moreover, each encoding module is composed of a combination of several encoding units.

[0013] Moreover, in the fusion module of the attention mechanism, the following processing is performed to achieve the feature fusion and expression of the color image and the depth image: Perform global maximum pooling on the color image feature tensor at the channel level to obtain the color image channel feature tensor; perform global maximum pooling on the depth image feature tensor at the channel level to obtain the depth image channel feature tensor; In the color processing path and the depth processing path, the color image channel features and the depth image channel features are added according to corresponding preset weights and then input into the multi-layer perceptron module; The weights output by the multilayer perceptron module are multiplied by the original color image features to obtain the color image spatial features, and the weights output by the multilayer perceptron module are multiplied by the original depth image features to obtain the depth image spatial features; Input the spatial features of the color image and the depth image into the corresponding convolution modules respectively to obtain the corresponding convolution features; The convolutional features of the color image and the depth image are fused.

[0014] Moreover, when training the deep learning model for image semantic segmentation that fuses multimodal data, a pyramid hyper-supervision strategy is used to calculate the cross entropy loss function at different scales, and utilize features at different scales to improve the final semantic performance.

[0015] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the indoor semantic segmentation method utilizing multimodal fusion of color and depth maps as described above is implemented.

[0016] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the indoor semantic segmentation method using multimodal fusion of color and depth images as described above.

[0017] On the other hand, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the indoor semantic segmentation method using multimodal fusion of color and depth images as described above.

[0018] The technical solution proposed in the present invention is to solve the many challenges faced by existing semantic segmentation methods based on deep learning when processing indoor scenes. These challenges mainly include the variability of indoor lighting, the overlap between numerous objects indoors, and the unbalanced representation of object categories. In these cases, relying solely on color images often cannot effectively distinguish multiple objects in the scene. This is because color images only provide two-dimensional information of the scene and lack a direct description of depth and spatial relationships. Therefore, semantic segmentation methods based on color images alone will result in poor robustness, misidentification, and category confusion. For example, color images cannot well identify curtains and sofas with similar textures. Because in color images, curtains with similar textures have the same color information attributes as sofas, which will produce misidentification and cross-robustness. The use of three-dimensional information in depth images can better solve this problem, because curtains are usually not similar to sofas in geometric structure. Therefore, in the face of various particularities and complexities in indoor environments, the present invention integrates color image and depth image information to improve the robustness and efficiency of indoor image semantic segmentation. By effectively extracting and fusing color image and depth image features, it reduces problems such as misrecognition, category confusion, and insensitivity to scale changes in indoor image semantic segmentation, thereby improving the accuracy and robustness of semantic segmentation.

[0019] Compared with the deep learning semantic segmentation method suitable for the metal facade surface scene of storage tanks proposed in CN201911391140.5, the present invention combines multimodal data, including color images and depth images, improves the robustness of the algorithm, and redesigns the network structure to enable it to more effectively realize the extraction and fusion of multimodal data.

[0020] Compared with the dual-stream semantic segmentation network for RGB images and depth images proposed in CN201911064946.3, the present invention proposes a method for transcoding single-channel depth images into HHA depth images before inputting the data into the network, which effectively optimizes the feature expression of the depth image.

[0021] Compared with the prior art, the present invention has the following beneficial effects: The present invention effectively utilizes multimodal data and uses different modal image data of color and depth to enhance the robustness of semantic segmentation of indoor scenes. The present invention adopts a dual encoder-decoder framework to extract specific features of color and depth images respectively; at the same time, the decoder generates a multi-scale semantic map, which can improve the final segmentation result. A series of test results show the effectiveness and robustness of the method of the present invention.

[0022] The present invention effectively improves the feature expression problem of depth image data, transcodes single-channel depth images into HHA depth images, and improves the feature extraction, expression and fusion capabilities of the network from a mechanism perspective.

[0023] The present invention contributes to the automatic recognition of indoor scenes by semantically segmenting indoor scene images. The present invention is applied to indoor robots and indoor furniture, and can effectively identify and classify indoor objects, providing a reference and technical support for future smart furniture and indoor navigation.

[0024] The solution of the present invention is simple and convenient to implement, has strong practicality, solves the problems of low practicality and inconvenience in actual application existing in related technologies, can improve user experience, and has important market value. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.

[0026] Figure 2 Schematic diagram of a multimodal image according to an embodiment of the present invention, which includes a color image, a single-channel depth image, and a HHA depth image from left to right.

[0027] Figure 3 This is an example diagram of a data set according to an embodiment of the present invention, which includes a color image, a depth image, and a label image from left to right.

[0028] Figure 4 Schematic diagram of the structure of an indoor image semantic segmentation model based on multimodal data fusion according to an embodiment of the present invention.

[0029] Figure 5 Schematic diagram of the structure of the coding block according to an embodiment of the present invention.

[0030] Figure 6 Schematic diagram of a multimodal fusion structure based on an attention mechanism according to an embodiment of the present invention.

[0031] Figure 7 It is a multimodal processing module of the attention mechanism of an embodiment of the present invention.

[0032] Figure 8It is a schematic diagram of the automatic recognition result of the indoor image semantic segmentation model based on multimodal data fusion according to an embodiment of the present invention.

[0033] Fig. 9 It is a framework structure diagram of the hardware system of an embodiment of the present invention.

[0034] Fig.10 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0036] The present invention first uses depth sensor data to collect raw data, including a series of corresponding color images and depth images. After image calibration and preprocessing, the collected color and depth images are calibrated and preprocessed to ensure data quality as the basis for data set production. The data set is then divided and augmented to improve the generalization ability of the model. At the same time, the model framework is determined and multiple convolutional layers are stacked in the model. In addition, feature extraction and multimodal feature fusion modules are added to the network, and a pyramid supervised loss function is set to optimize model training. After the hyperparameters are set and the model is trained, the final model is obtained, followed by model testing and result visualization. Finally, the trained model and software environment are deployed on the application hardware platform.

[0037] like Figure 1 As shown, an indoor semantic segmentation method using multimodal fusion of color and depth images provided by an embodiment of the present invention includes the following steps: Step 1, data acquisition and processing, including using a depth camera to collect images of indoor scenes, obtain corresponding color and depth images, and perform registration and preprocessing; This step includes the acquisition and preprocessing of data, where the data is color and depth images in indoor environments. The specific contents include: acquisition of color images and depth images, registration between image pairs, and decoding of single-channel depth images.

[0038] Furthermore, the color image and the depth image are collected using corresponding sensors. The color image can be collected by an ordinary optical camera, and the depth camera is collected by an infrared camera. The Kinect camera used in the embodiment of the present invention is equipped with an optical camera and an infrared camera, which can simultaneously capture paired color and grayscale images to construct image pairs.

[0039] Furthermore, through calibration and data registration between multiple sensors, the perspective error caused by the coordinate error of the two sensors can be reduced. The calibration between sensors can be considered as a coordinate system conversion problem, that is, a rigid body rotation problem between sensors. , needs to be converted to optical camera coordinates , the rigid body change can be achieved through the transformation matrix Implementation, the formula can be expressed as: .

[0040] Furthermore, the present invention innovatively proposes to decode the depth image (single-channel grayscale image) obtained by the infrared camera into an HHA image as the network input, replacing the single-channel depth image used in the general method, and enhancing the representation ability of the network model. The three channels of the HHA image represent the horizontal difference, the height from the ground, and the angle between the local surface normal of the pixel and the inferred gravity direction. The schematic diagram of the multimodal image can be seen Figure 2 , from left to right are color image, single-channel depth image and HHA depth image.

[0041] For ease of implementation reference, the generation process of HHA depth image is described as follows: (1) Read depth image: Read the original depth image and convert it into depth values ​​in meters.

[0042] (2) Calculate height information (Height above ground): Calculate the height from the ground to each pixel and use the ground detection algorithm to determine the depth value of the ground.

[0043] First, use a ground detection algorithm (such as RANSAC plane fitting) to detect the ground plane equation ,in are the parameters of the plane equation.

[0044] Secondly, for each pixel Depth value , calculate its height information :

[0045] in, are the parameters of the ground plane, are the pixel coordinates and depth values ​​respectively. is the norm of the plane normal vector, used for normalization.

[0046] (3) Calculate horizontal and vertical gradients: Use edge detection methods to calculate the horizontal and vertical gradients of the depth image. Horizontal gradient and vertical gradient The calculation formula is:

[0047] in,

[0048] (4) Calculate Surface Normals: Calculate the normals based on the horizontal and vertical gradients. Specifically, the normals can be calculated based on the relationship between the gradient direction and the depth value.

[0049] Among them, the normal vector , and The quantities are:

[0050] in, is the component of the normal vector in the X direction, is the component of the normal vector in the Y direction, is the component of the normal vector in the Z direction. and are the gradients in the horizontal and vertical directions respectively.

[0051] Normalized normal vector The calculation method is:

[0052] (5) Combine into HHA image: Combine the height information, horizontal and vertical components of the normal vector into a three-channel RGB image, which is the HHA image. See the schematic diagram of the multimodal image provided in Figure 2, from left to right are color image, single-channel depth image and HHA depth image.

[0053] Step 2, data set preparation, including constructing pixel-level semantic information annotation in the image and establishing a data set, which is divided into a test data set and a training data set; This step requires annotating the preprocessed color image and depth image to facilitate the implementation and verification of the subsequent multimodal data fusion indoor image semantic segmentation method. The specific contents include: semantic annotation of images and division of data sets.

[0054] Furthermore, suitable annotation tools can be used to annotate different objects in the image at the pixel level. During annotation, it is recommended to reasonably select the data volume and data distribution according to the specific implementation situation to ensure the comprehensiveness and representativeness of the annotation.

[0055] Furthermore, based on the labeled indoor color images and depth images, we analyze and balance the data distribution and divide the data set. One part is used as a training set to train the indoor image semantic segmentation model of multimodal data fusion; the other part is used as a test set to detect and verify the automatic recognition effect of the model. Figure 3 , from left to right are color image, depth image and label image.

[0056] Step 3: Establish an image semantic segmentation model that integrates multimodal data, including building a semantic segmentation deep learning model that integrates multimodal data according to the characteristics of the data set, effectively integrating multimodal data, and improving the efficiency of automatic image recognition; This step requires building an image semantic segmentation deep learning model that can be used to fuse multimodal data, and fusing multimodal data based on data characteristics. The specific contents include: building a deep learning model and selecting a loss function.

[0057] Furthermore, the network structure of the image semantic segmentation deep learning model extracts features from the color image and the depth image (HHA) respectively, and encodes the two modalities layer by layer through the encoding module. During specific implementation, the number of layers can be set according to the specific situation, such as 3, 4 or 5 layers. This embodiment preferably uses 4 layers. At each layer, the features of the color image and the depth image are fused through a fusion module to enhance the feature representation. After multiple layers of encoding, the fused features are merged through an addition operation, and then decoded layer by layer by a decoding module to finally generate a semantic segmentation result. In general, the semantic segmentation model consists of an encoder and a decoder part, and the encoder and the decoder are connected in series to form the model. The structure of the indoor image semantic segmentation model based on multimodal data fusion proposed in the embodiment of the present invention can be seen Figure 4 : The features extracted by the color image coding module 1 and the depth image coding module 1 are fused through the fusion module 1 and then passed back to the color image coding module 2 and the depth image coding module 2; the features extracted by the color image coding module 2 and the depth image coding module 2 are fused through the fusion module 2 and then passed back to the color image coding module 3 and the depth image coding module 3…The features extracted by the color image coding module 4 and the depth image coding module 4 are added together and output to the decoding module 1, which are decoded layer by layer through the decoding modules 1, 2, 3, and 4 to finally generate the semantic segmentation result.

[0058] Furthermore, since the input data is a color image and a depth image, in order to improve the feature representation capability, the encoder is a dual-branch encoder, including two independent encoding structures. One encoding structure is used to extract features from the color image, namely the color image encoding structure, and the other encoding structure is used to extract features with the depth image, namely the depth image encoding structure. The color image encoding structure and the depth image encoding structure have the same structural form.

[0059] Furthermore, each coding structure is composed of a number of coding modules (the number of which corresponds to the number of layers), and different coding modules in a coding structure have similar structures but do not share parameters, and are used to characterize features of different scales. Figure 5, each encoding module (color image encoding module or depth image encoding module) is composed of a combination of several (preferably 3 or 5) encoding units, and each encoding unit includes a 1×1 convolution, a 3×3 convolution and a 1×1 convolution in sequence.

[0060] In general, the embodiment has two coding structures, one coding structure has four coding modules, and each coding module is composed of a combination of 3 or 5 (1×1 convolution, 3×3 convolution, 1×1 convolution). The color image coding structure in the embodiment includes color image coding module 1, color image coding module 2, color image coding module 3, and color image coding module 4, which respectively include 3 coding units, 3 coding units, 5 coding units, and 5 coding units; the depth image coding structure in the embodiment includes depth image coding module 1, depth image coding module 2, depth image coding module 3, and depth image coding module 4, which also respectively include 3 coding units, 3 coding units, 5 coding units, and 5 coding units.

[0061] Furthermore, in order to improve the fusion of color image features and depth image features, the intermediate features of the two encoding structures are extracted and optimized through the fusion module of the attention mechanism to obtain the fusion features of multimodal data, where the intermediate features represent the results after each encoding block is processed. Figure 4 Taking the first layer as an example, the original features are processed by the color image encoding module 1 of the color encoder to obtain the color intermediate features. Similarly, the depth intermediate features can be obtained. The color intermediate features and the depth intermediate features are simultaneously input into the fusion module of the attention mechanism for feature representation enhancement and fusion. This process can be regarded as: ,in is the fused feature, It is a color feature. is a deep feature, Indicates the fusion process.

[0062] Furthermore, it is necessary to transfer the fused features back to the two encoding structures to assist in feature extraction. The present invention proposes a bidirectional feature propagation process, which can be expressed as: and ,in represents the fusion feature, and Represent the color transfer feature and depth transfer feature respectively, and Represent color image features and depth image features respectively, Represents fusion features. Figure 4 As shown, taking the first layer as an example, the color intermediate features and the depth intermediate features are simultaneously input into the fusion module 1 of the attention mechanism to obtain the enhanced features after representation enhancement, that is, For the color coding branch, it is necessary to fuse this feature with the original color intermediate feature to improve the coding effect. and The fusion is performed as the input feature of the color image encoding module 2. Similarly, the input feature of the depth image encoding module 2 can be obtained.

[0063] The multimodal fusion structure based on the attention mechanism proposed in the present invention can be seen Figure 6 , the color image features (color intermediate features) and the depth image features (depth intermediate features) are simultaneously input into the fusion module (multimodal processing module) of the attention mechanism for feature representation enhancement and fusion, and the color transfer features and depth transfer features are obtained. The fusion module of the attention mechanism uses a cross-modal attention method to effectively process the feature fusion and expression of the two modalities (color image and depth image). The specific implementation of this process can be seen Figure 7 , mainly includes the following key steps: 1. First, the global pooling method is used to perform global maximum pooling on the channel level of the original color image feature tensor of size H×W×n, and a color image channel feature tensor of size n is obtained, where H represents the height of the image, W represents the width of the image, and n represents the feature dimension.

[0064] Similar to color images, features are extracted from the input depth image. That is, the depth image feature tensor is subjected to global maximum pooling at the channel level to obtain the depth image channel feature tensor.

[0065] These features can represent important structures in the depth information.

[0066] 2. In the color processing path and the depth processing path, the color image channel features and the depth image channel features are added according to the corresponding preset weights and then input into the multi-layer perceptron (MLP) module respectively. The MLP module is used to obtain the weight of the feature importance. In the specific implementation, the existing MLP module in the prior art can be used, which will not be described in detail in the present invention. There are two parallel paths in this step, one for processing the color image channel features and the other for processing the depth image channel features. The method of adding features depends on the different paths. Preferably, in the color processing path, the addition weight of the color image channel features and the depth image channel features is 2:1, because the main purpose of this path is to use the depth features to enhance the color features. A similar weight addition method is used in the depth processing path, that is, the addition weight of the color image channel features and the depth image channel features is 1:2.

[0067] 3. Multiply the weights output by the MLP module with the original features to achieve the effect of feature enhancement, where the size of the output by the MLP module is n×n, and multiply it with the original features of size H×W×n to obtain spatial features of size H×W×n. Multiply the weights output by the MLP module with the original color image features to obtain color image spatial features, and multiply the weights output by the MLP module with the original depth image features to obtain depth image spatial features.

[0068] 4. Input the spatial features of the color image and depth image into the corresponding convolution modules respectively. The convolution module can further extract and enhance the features and output the convolution features of the color image and depth image. The convolution operation at this stage helps to capture the local patterns and textures of the image.

[0069] 5. Finally, the convolutional features of the color image and the depth image are fused. This fusion process is performed in an additive manner. By fusing features of different modalities, a richer and more comprehensive image representation can be obtained.

[0070] Furthermore, the decoder restores the image resolution by bidirectional interpolation upsampling and maps the high-dimensional features into category classification features. The length of the category feature is the number of categories to be classified, where the largest dimension in the category feature is the category to which the pixel belongs.

[0071] Furthermore, it is necessary to select a suitable loss function to constrain and learn the parameters. For semantic segmentation tasks, the cross entropy function is usually used as the loss function. The formula of the cross entropy loss function is: ,in Indicates the total number of samples; Represents the index of the sample and traverses all samples; Indicates The loss of samples, is the total loss; Indicates the number of categories; Indicates the index of the category, ranging from 1 to That is, traverse all categories; Indicates The sample in The true label of the class; The model predicts The samples belong to The probability of the class.

[0072] Furthermore, in order to alleviate the gradient vanishing and enhance the recognition of images with different resolutions, the present invention uses a pyramid super-supervision strategy to calculate the cross entropy loss function at different scales and use the features of different scales to improve the final semantic performance. Through the pyramid super-supervision strategy, supervision in the low spatial resolution output helps higher output by identifying boundaries and large objects; at the same time, supervision in the high spatial resolution output can refine the semantic information. The loss function of the pyramid super-supervision strategy is ,in , in, The index representing the number of layers of the pyramid, Indicates the total number of pyramid layers, corresponding to the number of encoding and decoding layers. =4; N represents the total number of samples; Indicates n The loss function value of the layer pyramid; i Indicates the index of the sample; It is i The predicted score vector of samples; It is i The true label index of each sample; Indicates i The samples belong to The predicted score of the category; Indicates i The samples belong to The predicted score of the category; c is the category index, traversing all categories.

[0073] Step 4: Model training and saving, including using the training data set, setting the corresponding hyperparameters (number of batches, optimizer parameters, number of iterations), and training the parameters of the semantic segmentation deep learning model of multimodal data fusion to obtain the neural network model parameters and obtain the trained model; In this step, after the neural network is built, its parameters need to be trained through the existing training data set to obtain a model that can be automatically recognized. The specific contents include: data augmentation processing, optimizer and hyperparameter setting.

[0074] Furthermore, in order to prevent the model from overfitting during training, data augmentation is required for the training dataset. The data augmentation strategy randomly rotates and scales the color and depth images, and finally crops them to the same size to ensure the heterogeneity of the image size of the input model.

[0075] Furthermore, the data augmentation method used in the present invention includes: using random rotation (rotating 90 degrees or 180 degrees clockwise), random flipping (flipping from left to right or from top to bottom), random scaling (scaling factor is randomly between 1 times and 1.6 times), random color change (brightness, saturation, contrast change) and random cropping (to an image of 480 × 640 pixels).

[0076] Furthermore, in model training, the present invention uses a stochastic gradient descent optimizer. In parameter optimization, the gradient represents the rate of change of the objective function with respect to the parameter, indicating in which direction the function decreases fastest at the current parameter value. The stochastic gradient descent optimizer first calculates the gradient of the model parameters, uses the learning rate multiplied by the negative direction of the gradient as the update direction of the parameters, and updates the model parameters. The process can be expressed as: ,in represents the model parameters, represents the learning rate, represents the gradient, and t represents the current training batch.

[0077] Furthermore, it is necessary to select corresponding hyperparameters to optimize the training process, including learning rate, momentum, batch, number of training times and other hyperparameters.

[0078] Furthermore, the gradients are calculated and the parameters are updated iteratively to minimize the objective function, and finally the parameters in the model are stabilized to obtain a well-trained model.

[0079] Furthermore, the trained model and parameters are saved as a ".pth" file, and the parameters are loaded and recognized when they are needed.

[0080] Step 5, model accuracy assessment and visualization, includes inputting the test dataset into the trained model to obtain pixel-level semantic automatic segmentation results of the image.

[0081] In this step, after obtaining the optimal parameters of the model, the recognition efficiency is tested on the test data set to verify its generalization. The specific contents include: the selection of evaluation indicators and the test process.

[0082] Furthermore, the present invention uses intersection-over-union (IoU) to evaluate the automated recognition results in the test. IoU is used to measure the degree of overlap between the predicted results and the true labels in target detection or image segmentation tasks, and its calculation method is: , where region C is the true label region, and G is the identified result region. Indicates the corresponding area.

[0083] Furthermore, the parameters are first loaded into the model, and then the test data set is input in batches to calculate its overall evaluation index. When the evaluation index result meets a certain range, it is determined that the automatic recognition effect is high and it is an effective model. Figure 8 , from left to right are color image, depth image, label image and recognition image.

[0084] In order to facilitate understanding of the technical effects of the present invention, the following table gives a comparison with some existing semantic segmentation networks as a comparison. Under the training and testing of the same data set, the performance of multiple semantic segmentation methods on three key indicators: global pixel-level accuracy, average category progress, and average intersection-over-union ratio. This method performed well in all three indicators and was significantly better than other comparison methods. This method achieved a global pixel-level accuracy of 74.8, which is higher than the highest 73.2 (RAFNet network) of other methods. This shows that this method has higher accuracy in pixel classification of the entire image and can more accurately identify and classify pixels in the image. This method achieved a high score of 59.1 in average category progress, which is significantly better than other methods. This means that this method is more balanced and stable when dealing with different categories, and the classification accuracy of each category has been effectively improved. In comparison, the best performing RAFNet network among other methods is only 58.2. Finally, in terms of average intersection-over-union ratio, this method reached 48.1, which is also ahead of other methods. This shows that this method works best in terms of the overlap between the segmentation results and the true labels, providing more accurate semantic segmentation results, exceeding the 47.2 of the RAFNet network. In summary, this method performs well in three key indicators: global pixel-level accuracy, average category progress, and average intersection-over-union, showing its significant superiority in semantic segmentation tasks.

[0085]

[0086] During specific implementation, after verifying the model, the corresponding model and environment can be deployed to the corresponding hardware platform.

[0087] The embodiment utilizes color image information representation and depth image information representation to perform end-to-end multimodal feature extraction and fusion based on color images and depth images, adopts a deep learning method, uses a codec as the overall framework, and an attention module to fuse and extract multimodal features, effectively improving the robustness and effectiveness of indoor image semantic segmentation, solving problems such as misidentification, category confusion, and insensitivity to scale changes in indoor image semantic segmentation, improving the accuracy and robustness of semantic segmentation, and can be deployed on a hardware platform.

[0088] The present invention also designs a hardware system based on an indoor robot, and the corresponding embodiment specifically includes an image acquisition module, a processing module, and a robot module. Fig. 9 This is the framework diagram of the hardware system.

[0089] Furthermore, the image acquisition module includes a camera module (including a color camera and a depth camera) and a sensor module (including an inertial measurement unit, etc.), wherein the camera module is used for real-time acquisition of color images and depth images, and the sensor platform is used to monitor the real-time position change of the robot platform. It is worth mentioning that the various sensors (camera, inertial measurement unit, etc.) need to be calibrated in advance.

[0090] Furthermore, the processing module includes a graphics processor, a central processing unit, and an output module. The graphics processor deploys the indoor image semantic segmentation method based on multimodal data fusion in Example 1, which is used to perform real-time recognition and automatic semantic segmentation of images. The central processing unit is used to perform real-time image processing, including image preprocessing (correction, analysis) and data augmentation and reading. The output module is used to output and visualize the recognition results of real-time images.

[0091] Furthermore, the robot module is used to carry modules such as the image acquisition platform, processing platform, power supply, and action module.

[0092] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. The processor may call the logic instructions in the memory to execute the indoor semantic segmentation method using multimodal fusion of color and depth images.

[0093] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0094] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the indoor semantic segmentation method using multimodal fusion of color and depth maps provided by the above methods.

[0095] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the indoor semantic segmentation method using multimodal fusion of color and depth maps provided by the above-mentioned methods.

[0096] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0097] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An indoor semantic segmentation method using multimodal fusion of color and depth images, characterized by: Perform the following processing, Use a depth camera to capture images of indoor scenes, obtain corresponding color and depth images, and perform registration and preprocessing; the depth image uses an HHA image; Construct pixel-level semantic information annotation in images, and establish and divide test data sets and training data sets; A deep learning model for image semantic segmentation is constructed for fusing multimodal data, and features are extracted from color images and HHA images respectively; in the network structure of the deep learning model for image semantic segmentation, the two modalities of color image and HHA image are encoded layer by layer through an encoding module, and at each layer, the features of the color image and the depth image are fused through a fusion module to enhance feature representation; after multi-layer encoding, the fused features are merged through an addition operation, and then decoded layer by layer through a decoding module to finally generate a semantic segmentation result; Train a deep learning model for image semantic segmentation that integrates multimodal data, and obtain pixel-level semantic automatic segmentation results based on the trained model.

2. The indoor semantic segmentation method using multimodal fusion of color and depth images according to claim 1, characterized in that: The depth camera adopts an infrared camera, and decodes the single-channel depth image acquired by the infrared camera into an HHA image as the network input of the image semantic segmentation deep learning model.

3. The indoor semantic segmentation method using multimodal fusion of color and depth images according to claim 1, characterized in that: In the network structure of the image semantic segmentation deep learning model, two encoding structures are set, namely a color image encoding structure and a depth image encoding structure; each encoding structure has several encoding modules, and the intermediate features of the two encoding structures are extracted and optimized through the fusion module of the attention mechanism to obtain the fusion features of multimodal data, and the fused features are passed back to the two encoding structures to assist feature extraction.

4. The indoor semantic segmentation method using multimodal fusion of color and depth images according to claim 3, characterized in that: Each encoding module is composed of a combination of several encoding units.

5. The indoor semantic segmentation method using multimodal fusion of color and depth images according to claim 3, characterized in that: In the fusion module of the attention mechanism, the following processing is performed to achieve the feature fusion and expression of color images and depth images: Perform global maximum pooling on the color image feature tensor at the channel level to obtain the color image channel feature tensor; perform global maximum pooling on the depth image feature tensor at the channel level to obtain the depth image channel feature tensor; In the color processing path and the depth processing path, the color image channel features and the depth image channel features are added according to corresponding preset weights and then input into the multi-layer perceptron module; The weights output by the multilayer perceptron module are multiplied by the original color image features to obtain the color image spatial features, and the weights output by the multilayer perceptron module are multiplied by the original depth image features to obtain the depth image spatial features; Input the spatial features of the color image and the depth image into the corresponding convolution modules respectively to obtain the corresponding convolution features; The convolutional features of the color image and the depth image are fused.

6. The indoor semantic segmentation method using multimodal fusion of color and depth images according to claim 1, characterized in that: When training deep learning models for image semantic segmentation that fuse multimodal data, a pyramid hyper-supervision strategy is used to calculate cross-entropy loss functions at different scales, and to use features at different scales to improve the final semantic performance.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the indoor semantic segmentation method using multimodal fusion of color and depth images is implemented as described in any one of claims 1 to 6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the indoor semantic segmentation method using multimodal fusion of color and depth images is implemented as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the indoor semantic segmentation method using multimodal fusion of color and depth images is implemented as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • RGBD image semantic segmentation method

    CN107403430A

  • Image semantic segmentation method, device and equipment based on deep neural network

    CN117115448A

  • Multi-modal semantic segmentation method and system based on depth camera

    CN117635946A

  • Image retrieval method based on deep learning and semantic segmentation

    WO2019237646A1

Cited By

  • Robot control method and system, electronic equipment and storage medium

    CN119369412A

  • Plant leaf scab segmentation method and system based on RGB-D cross-modal fusion

    CN120374650A

  • A plant leaf spot segmentation method and system based on RGB-D cross-modal fusion

    CN120374650B

  • Robot control method and system, electronic device, and storage medium

    WO2026138201A1