Scene semantic segmentation-based in-station equipment interaction method and system of space station

Abstract

The invention provides a scene semantic segmentation-based in-station equipment interaction method and system of a space station, and relates to the technical field of computer vision. The method comprises the steps of collecting equipment images in a space station, and obtaining training data; constructing a semantic segmentation model of the image to be trained based on the model structure of the DeepLabv3 + network; based on the image segmentation loss function, using the training data to train a to-be-trained image semantic segmentation model to obtain an image semantic segmentation model; acquiring a current scene equipment image; inputting the current scene equipment image into the image semantic segmentation model to obtain an equipment segmentation image; obtaining an interaction demand; based on the space station equipment database, space station equipment information is called from the UI according to the interaction requirement and the equipment segmentation image. The method is an accurate and efficient in-station equipment interaction method based on scene semantic segmentation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for space station in-station device interaction based on scene semantic segmentation. Background Art

[0002] The confined and confined space of the space station necessitates that internal air circulation rely on a ventilation system. Astronauts' breathing, sweat, and moisture generated by equipment operation make the station's equipment susceptible to oxidation and mold, seriously threatening its performance and service life, impacting the station's safe and stable operation, as well as the astronauts' health and work efficiency. To ensure the station's long-term and reliable operation, astronauts must regularly maintain the equipment. New ways for space station equipment to interact are currently a hot topic of research. Early equipment detection methods relied on traditional image processing methods, typically employing computer vision and image processing techniques such as edge detection, color segmentation, and shape matching.

[0003] In recent years, deep learning technology has made significant progress in device recognition. Deep learning models can learn complex feature representations from large amounts of data, improving the accuracy and robustness of device recognition. We use the DeepLabV3+ network to perform semantic segmentation of target devices, employ the Visual Background Extractor (ViBe) combined with grayscale histograms for double Gaussian fitting, and implement object detection using the RetinaNet network.

[0004] As a classic semantic segmentation network, DeepLabV3+ has achieved remarkable results in complex scene segmentation tasks by introducing a dilated spatial pyramid pooling module and an encoder-decoder structure. However, some limitations remain. The original DeepLabV3+ network has limited ability to integrate contextual information from multi-scale features, and is particularly prone to information loss when recovering edge details and segmenting small objects. Backbone networks based on the Xception neural network architecture or the Residual Neural Network (ResNet) architecture have large parameter counts, making them difficult to deploy on mobile devices or resource-constrained devices. The traditional dilated convolution of Atrous Spatial Pyramid Pooling (ASPP) uses discrete interval sampling, resulting in less dense receptive field coverage and difficulty capturing continuous multi-scale context. The importance of feature channels and spatial positions is not dynamically weighted, making it susceptible to background noise in complex scenes.

[0005] In the existing technology, there is a lack of an accurate and efficient method for device interaction within a space station based on scene semantic segmentation. Summary of the Invention

[0006] To address the technical issues in existing technologies such as scene semantic segmentation, which suffer from insufficient coverage of the receptive field, difficulty in capturing continuous multi-scale context, and susceptibility to background noise in complex scenes, the present invention provides a method and system for in-station device interaction in a space station based on scene semantic segmentation. The technical solution is as follows:

[0007] On the one hand, a method for interacting with devices within a space station based on scene semantic segmentation is provided. The method is implemented by a device interaction device within the space station, and the method includes:

[0008] Collect images of equipment inside the space station and obtain training data;

[0009] Build a semantic segmentation model for the image to be trained based on the model structure of the DeepLabv3+ network;

[0010] Based on the image segmentation loss function, the training data is used to train the image semantic segmentation model to obtain the image semantic segmentation model;

[0011] Obtain the current scene device image; input the current scene device image into the image semantic segmentation model to obtain the device segmentation image;

[0012] Obtain interaction requirements; based on the space station equipment database, according to interaction requirements and equipment segmentation images, retrieve space station equipment information in the UI interface.

[0013] On the other hand, a space station in-station device interaction system based on scene semantic segmentation is provided. The system is applied to a space station in-station device interaction method based on scene semantic segmentation. The system includes a camera, an electronic device, and a touch screen, wherein:

[0014] The camera is used to obtain the current scene device image;

[0015] The electronic device is used to collect images of devices in the space station and obtain training data; construct a semantic segmentation model for the image to be trained based on the model structure of the DeepLabv3+ network; train the semantic segmentation model for the image to be trained using the training data based on the image segmentation loss function to obtain an image semantic segmentation model; input the current scene device image into the image semantic segmentation model to obtain a device segmentation image;

[0016] The touch screen is used to obtain interaction requirements; based on the space station equipment database, according to the interaction requirements and the equipment segmentation image, the space station equipment information is retrieved in the UI interface.

[0017] On the other hand, a space station in-station device interaction device is provided, and the space station in-station device interaction device includes: a processor; a memory, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, any one of the above-mentioned space station in-station device interaction methods based on scene semantic segmentation is implemented.

[0018] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned methods for in-space station device interaction based on scene semantic segmentation.

[0019] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0020] This paper proposes a method for in-station device interaction in a space station based on scene semantic segmentation. It introduces a channel-spatial dual attention module into the encoder portion of the DeepLabV3+ network to improve the model's robustness to complex scenes. It also replaces the original backbone network, Xception, with MobileNetV2, leveraging its inverted residual structure and linear bottleneck layer to reduce the number of parameters and computational complexity. The original ASPP module is replaced with DenseASPP, which, by stacking densely connected dilated convolutional layers, constructs a denser and wider receptive field, enhancing the contextual modeling capabilities for small targets and multi-scale objects. This paper provides an accurate and efficient method for in-station device interaction in a space station based on scene semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flow chart of a method for device interaction within a space station based on scene semantic segmentation provided by an embodiment of the present invention;

[0023] Figure 2 This is a block diagram of a space station in-station device interaction system based on scene semantic segmentation provided by an embodiment of the present invention;

[0024] Figure 3 This is a structural diagram of a space station in-station device interaction device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0026] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0027] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0028] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0029] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0030] The embodiment of the present invention provides a method for space station in-station device interaction based on scene semantic segmentation, which can be implemented by a space station in-station device interaction device, which can be a terminal or a server. Figure 1 The flowchart of the method for interacting with devices in a space station based on scene semantic segmentation is shown. The processing flow of the method may include the following steps:

[0031] S1. Collect images of equipment in the space station and obtain training data.

[0032] In one feasible implementation, the present invention collects a large number of images of instruments and components within the space station and annotates each component with a semantic label. Labeling tools such as Labelme are used to generate the annotated data in common formats such as Pascal VOC. Basic image operations are performed using image processing libraries such as the Open Source Computer Vision Library (OpenCV). The collected images are then preprocessed, including cropping, scaling, and denoising, to improve image quality. The training data images have a resolution of 3840×2160.

[0033] S2. Build a semantic segmentation model for the image to be trained based on the model structure of the DeepLabv3+ network.

[0034] In one feasible implementation, a deep learning model for semantic segmentation (DeepLab Version 3Plus, DeeplabV3+) introduces a new encoder-decoder architecture. The encoder consists of a deep convolutional neural network (DCNN) with dilated convolutions, which can be used with a common classification network. At the end of the encoder, DeeplabV3+ introduces an atrous spatial pyramid pooling (ASPP) module with dilated convolutions to achieve multi-scale object segmentation capabilities.

[0035] ASPP performs 1x1 convolutions on the input feature maps, followed by 3x3 convolutions with dilation rates of 6, 12, and 18, and global average pooling. It then fuses the feature maps and performs 1x1 convolutions to reduce the number of channels to 256. ASPP can extract and distinguish feature information from objects of different scales, effectively achieving multi-scale object segmentation. This module aims to capture contextual information at different scales to better understand the semantics of an image.

[0036] Compared with DeepLabv3, DeepLabv3+ introduces a Decoder module, which performs necessary downsampling operations on the input image, further fuses the underlying features with the high-level features, fuses the low-level features in the feature map recovery process, and restores the boundary information of the target part. The feature map recovery uses a linear interpolation method, which ultimately improves the accuracy of network segmentation.

[0037] S3. Based on the image segmentation loss function, the training data is used to train the image semantic segmentation model to obtain the image semantic segmentation model.

[0038] Among them, the image semantic segmentation model includes an encoder part and a decoder part;

[0039] The backbone network of the encoder part is the lightweight convolutional neural network MobileNetV2;

[0040] The multi-scale feature extraction module of the encoder part is the DenseASPP module of dense void spatial pyramid pooling;

[0041] The image semantic segmentation model adopts the CBAM module that combines the channel attention mechanism and the spatial attention mechanism.

[0042] One possible implementation uses the Xception network as the backbone of the classic DeeplabV3+ network. The Xception model has a relatively complex architecture and powerful expressive capabilities, making it suitable for complex tasks and large datasets. However, due to its relatively large model size, it requires more computing resources and storage space, and training time is generally longer, making it less suitable for resource-constrained devices.

[0043] Therefore, the classic DeeplabV3+ network architecture was improved, with the backbone network replaced with the lightweight convolutional neural network MobileNetV2. Compared to the Xception network, MobileNetV2 has a lower model size and computational complexity, making it suitable for mobile devices and embedded systems. Due to its lightweight design, MobileNetV2 also boasts faster inference performance.

[0044] MobileNet primarily replaces ordinary convolution with depthwise convolution and introduces two hyperparameters, the width factor and the resolution factor, to flexibly control the size of the network model. By optimizing the network structure, the network accuracy exceeds that of most neural networks with fewer parameters and computational complexity.

[0045] MobileNetv2 adds two key modules to its architecture: a linear bottleneck relationship between network layers and residual connections between bottleneck blocks. The bottleneck block design effectively encodes feature information at both the input and output ends of the model. The inner layers of the network encapsulate information from lower-level layers (such as object edges and outlines) into higher-level, abstract representations.

[0046] MobileNetv2 adds point convolution before depth-wise separable convolution, which allows for free adjustment of channels. The activation function can effectively increase the nonlinear representation of the network model in high-dimensional space, but will destroy features in low-dimensional feature space. This is the design of the inverted residual module. The input first undergoes 1x1 convolution for channel expansion, then uses 3x3 depth-wise convolution, and finally uses 1x1 point convolution to compress the channel to a low dimension. The whole process is "expansion-convolution-compression".

[0047] The Convolutional Block Attention Module (CBAM) is a widely used attention mechanism in deep learning. Its purpose is to improve model performance by focusing on important parts of an image. CBAM processes both the channel and spatial dimensions. First, the channel attention module focuses on "which channels are important," and then the spatial attention module focuses on "where" the informative parts are. This dual attention mechanism enables CBAM to fully capture key information in features.

[0048] Traditional atrous spatial pyramid pooling (ASPP) uses parallel atrous convolutions and atrous convolutions with different dilation rates for feature extraction. It can capture multi-scale information through convolutions with different receptive fields.

[0049] However, as the dilation rate increases (especially when it exceeds 24), the effective weight of the dilated convolution decreases, and the feature extraction capability also decreases. Therefore, this paper replaces the traditional ASPP module with the Densely Connected Atrous Spatial Pyramid Pooling (DenseASPP) module. DenseASPP can better capture the semantic information in the image.

[0050] Among them, the image segmentation loss function is obtained by weighted calculation of multiple image semantic category loss functions; the category loss function is the cross entropy loss function.

[0051] In one feasible implementation, the present invention divides the training dataset into training, validation, and test sets to ensure model generalization. Model hyperparameters (such as learning rate and batch size) are adjusted, and transfer learning techniques are used to improve training efficiency. Training is performed on an Nvidia Jetson Xavier NX development board, with loss and accuracy monitored during training, and the optimal model saved.

[0052] During training, for each pixel in the image, the cross-entropy loss is calculated between the predicted category and the true category. Cross-entropy loss is commonly used in classification problems and is particularly applicable to per-pixel classification tasks in image semantic segmentation. To address class imbalance, different weights are assigned to different categories. By assigning weights to different categories, the loss contribution of each category is balanced, preventing some categories from having a disproportionate impact on the overall loss.

[0053] S4. Obtain a device image of the current scene; input the device image of the current scene into an image semantic segmentation model to obtain a device segmentation image.

[0054] Optionally, the current scene device image is input into an image semantic segmentation model to obtain a device segmentation image, including:

[0055] Preprocess the current scene device image to obtain a processed device image;

[0056] Based on the processed device image, preliminary feature extraction is performed through the backbone network to obtain the first feature map;

[0057] According to the first feature map, the CBAM module is used to perform feature enhancement to obtain a third feature map;

[0058] The third feature map is input into the DenseASPP module for multi-scale feature extraction to obtain a multi-scale dense feature map;

[0059] Based on the first feature map, resolution recovery is performed according to the multi-scale dense feature map to obtain a device segmentation image.

[0060] In one feasible implementation, after the camera captures the image, it is transmitted to the electronic device for processing in real time. A pre-processing module is used to perform operations such as cropping, scaling, and denoising on the image to improve image quality.

[0061] DenseASPP adopts a densely connected and cascaded dilated convolutional layer design. The output of each dilated convolutional layer is not only passed to the next layer, but also connected to all subsequent unvisited layers, forming a dense feature transfer path. The expansion rate of each layer of DenseASPP increases layer by layer, and the final output is a multi-expansion rate, multi-scale dense feature map as shown in formula (1):

[0062] (1);

[0063] in, represents the expansion rate of the lth layer; () represents dilated convolution; Represents a feature map formed by concatenating the outputs of all previous layers. DenseASPP not only gradually accumulates multi-scale features through dense connections, but also the receptive field is determined by multiple consecutive atrous convolutional layers, avoiding the performance degradation caused by a single high dilation rate.

[0064] In the decoder, high-level features are fused with low-level features. The low-level features, derived from the first feature map, contain more spatial detail, helping to restore spatial details in the image. The feature map is upsampled to the original image resolution using bilinear interpolation or deconvolution to generate the final segmentation result. Upsampling gradually restores the image resolution, resulting in more refined segmentation results.

[0065] Optionally, according to the first feature map, using a CBAM module to perform feature enhancement to obtain a third feature map includes:

[0066] Based on the multi-layer perceptron, channel attention calculation is performed according to the first feature map to obtain a channel attention map;

[0067] Based on the channel attention map, the channel feature of the first feature map is enhanced to obtain the second feature map;

[0068] Based on the 7×7 convolutional layer, spatial attention calculation is performed according to the second feature map to obtain the spatial attention map;

[0069] Based on the channel attention map and the spatial attention map, feature enhancement is performed according to the first feature map to obtain a third feature map.

[0070] In one feasible implementation, channel attention obtains two different feature descriptions (one average and one maximum) for each channel of the input feature map through global average pooling and global maximum pooling. These two descriptions are processed by a multi-layer perceptron with shared weights, and the output feature map is then element-wise summed, and a sigmoid function is applied to generate a channel attention map M. c , the shape is consistent with the number of input channels. The formula is as follows (2):

[0071] (2);

[0072] Among them, MLP is a multi-layer perceptron; AvgPool represents average pooling; MaxPool represents maximum pooling; F is the first feature map; σ represents the Sigmoid activation function.

[0073] The feature map after spatial attention is applied to the channel attention is used to generate two two-dimensional feature maps through average pooling and maximum pooling in the channel direction. These two feature maps are stacked in the channel dimension, passed through a 7×7 convolution layer, and the sigmoid function is used to generate the spatial attention map M. s The formula is as follows (3):

[0074] (3);

[0075] Among them, Conv represents convolution operation; Concat represents concatenation operation.

[0076] The third feature map is obtained by combining the first feature map F with the channel attention map and the spatial attention map by element-wise multiplication. The process is as follows (4):

[0077] (4);

[0078] Through such a structure, CBAM can effectively enhance the network's response to important features in the image, improve the model's performance in various visual tasks, especially in image recognition and segmentation of complex scenes.

[0079] S5. Obtain interaction requirements: Based on the space station equipment database, according to the interaction requirements and the equipment segmentation image, retrieve the space station equipment information in the UI interface.

[0080] Optionally, based on the space station device database, according to interaction requirements and device segmentation images, the space station device information is retrieved in the UI interface, including:

[0081] Extract information based on the device segmentation image to obtain device image information;

[0082] Based on the space station equipment database, data query is performed according to the equipment image information to obtain detailed equipment information;

[0083] Based on interaction needs, space station device information is retrieved through the UI interface according to device image information and device detailed information.

[0084] In a feasible implementation manner, the present invention calls preset device images in a space station device database based on the device segmentation image to generate device detailed information with labeled images.

[0085] The annotated image is displayed on the UI, and you can tap an area of interest to view detailed information. Manually tapping an annotated area on the touchscreen will bring up a 3D model of the component in that area and related information. Learn about the component's name, function, and operation instructions through the 3D model and text description.

[0086] The interactive UI used in this paper is developed based on the Tkinter UI framework and supports interactive functions such as exploded view operation, zooming in, zooming out, and rotating the 3D models of the space station equipment. The paper creates a 3D model for each industrial instrument and its parts, using SketchUp software to design and export the 3D model files.

[0087] This paper proposes a method for in-station device interaction in a space station based on scene semantic segmentation. It introduces a channel-spatial dual attention module into the encoder portion of the DeepLabV3+ network to improve the model's robustness to complex scenes. It also replaces the original backbone network, Xception, with MobileNetV2, leveraging its inverted residual structure and linear bottleneck layer to reduce the number of parameters and computational complexity. The original ASPP module is replaced with DenseASPP, which, by stacking densely connected dilated convolutional layers, constructs a denser and wider receptive field, enhancing the contextual modeling capabilities for small targets and multi-scale objects. This paper provides an accurate and efficient method for in-station device interaction in a space station based on scene semantic segmentation.

[0088] Figure 2 This is a block diagram of a space station device interaction system based on scene semantic segmentation according to an exemplary embodiment. The system is used for a space station device interaction method based on scene semantic segmentation. Figure 2 The system includes a camera 210, an electronic device 220 and a touch screen 230, wherein:

[0089] Camera 210, used to obtain the current scene device image;

[0090] Electronic device 220 is used to collect images of devices in the space station and obtain training data; construct a semantic segmentation model for the image to be trained based on the model structure of the DeepLabv3+ network; train the semantic segmentation model for the image to be trained using the training data based on the image segmentation loss function to obtain an image semantic segmentation model; input the current scene device image into the image semantic segmentation model to obtain a segmented image of the device;

[0091] The touch screen 230 is used to obtain interaction requirements; based on the space station equipment database, according to the interaction requirements and the equipment segmentation image, the space station equipment information is retrieved in the UI interface.

[0092] The image semantic segmentation model includes an encoder part and a decoder part;

[0093] The backbone network of the encoder part is a lightweight convolutional neural network MobileNetV2;

[0094] The multi-scale feature extraction module of the encoder part is a DenseASPP module of dense void spatial pyramid pooling;

[0095] The image semantic segmentation model adopts the CBAM module that combines the channel attention mechanism and the spatial attention mechanism.

[0096] The image segmentation loss function is obtained by weighted calculation of multiple image semantic category loss functions; the category loss function is a cross entropy loss function.

[0097] Optionally, the electronic device 220 is further configured to:

[0098] Preprocess the current scene device image to obtain a processed device image;

[0099] Based on the processed device image, preliminary feature extraction is performed through the backbone network to obtain the first feature map;

[0100] According to the first feature map, the CBAM module is used to perform feature enhancement to obtain a third feature map;

[0101] The third feature map is input into the DenseASPP module for multi-scale feature extraction to obtain a multi-scale dense feature map;

[0102] Based on the first feature map, resolution recovery is performed according to the multi-scale dense feature map to obtain a device segmentation image.

[0103] Optionally, the electronic device 220 is further configured to:

[0104] Based on the multi-layer perceptron, channel attention calculation is performed according to the first feature map to obtain a channel attention map;

[0105] Based on the channel attention map, the channel feature of the first feature map is enhanced to obtain the second feature map;

[0106] Based on the 7×7 convolutional layer, spatial attention calculation is performed according to the second feature map to obtain the spatial attention map;

[0107] Based on the channel attention map and the spatial attention map, feature enhancement is performed according to the first feature map to obtain a third feature map.

[0108] Optionally, the touch screen 230 is further configured to:

[0109] Extract information based on the device segmentation image to obtain device image information;

[0110] Based on the space station equipment database, data query is performed according to the equipment image information to obtain detailed equipment information;

[0111] Based on interaction needs, space station device information is retrieved through the UI interface according to device image information and device detailed information.

[0112] This paper proposes a method for in-station device interaction in a space station based on scene semantic segmentation. It introduces a channel-spatial dual attention module into the encoder portion of the DeepLabV3+ network to improve the model's robustness to complex scenes. It also replaces the original backbone network, Xception, with MobileNetV2, leveraging its inverted residual structure and linear bottleneck layer to reduce the number of parameters and computational complexity. The original ASPP module is replaced with DenseASPP, which, by stacking densely connected dilated convolutional layers, constructs a denser and wider receptive field, enhancing the contextual modeling capabilities for small targets and multi-scale objects. This paper provides an accurate and efficient method for in-station device interaction in a space station based on scene semantic segmentation.

[0113] Figure 3 This is a schematic diagram of the structure of a space station device interaction device provided by an embodiment of the present invention. Figure 3 As shown, the space station equipment interaction equipment may include the above Figure 2 Optionally, the space station in-station device interaction device 310 may include a first processor 2001.

[0114] Optionally, the space station in-station device interaction device 310 may also include a memory 2002 and a transceiver 2003.

[0115] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0116] The following combination Figure 3 The components of the space station device interaction device 310 are described in detail:

[0117] The first processor 2001 is the control center of the space station's in-station device interaction device 310 and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0118] Optionally, the first processor 2001 can perform various functions of the space station in-station device interaction device 310 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0119] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0120] In a specific implementation, as an embodiment, the space station in-station device interaction device 310 may also include multiple processors, such as Figure 3 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0121] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0122] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently and be accessed through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0123] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0124] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0125] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and communicate with the space station through the interface circuit of the device interaction device 310 ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0126] It should be noted that Figure 3 The structure of the space station in-station device interaction device 310 shown in the figure does not constitute a limitation on the router. The actual knowledge structure identification device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0127] In addition, the technical effects of the space station in-station device interaction device 310 can refer to the technical effects of the space station in-station device interaction method based on scene semantic segmentation described in the above method embodiment, and will not be repeated here.

[0128] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0129] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0130] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer, or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0131] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0132] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0133] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0134] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0135] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0136] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of the system or unit, which can be electrical, mechanical or other forms.

[0137] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0138] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0139] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0140] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for device interaction within a space station based on scene semantic segmentation, characterized in that: The method comprises: Collect images of equipment inside the space station and obtain training data; Build a semantic segmentation model for the image to be trained based on the model structure of the DeepLabv3+ network; Based on the image segmentation loss function, the training data is used to train the image semantic segmentation model to obtain the image semantic segmentation model; Obtain the current scene device image; input the current scene device image into the image semantic segmentation model to obtain the device segmentation image; Obtain interaction requirements; based on the space station equipment database, according to interaction requirements and equipment segmentation images, retrieve space station equipment information in the UI interface.

2. The method for space station in-station device interaction based on scene semantic segmentation according to claim 1, characterized in that: The image semantic segmentation model includes an encoder part and a decoder part; The backbone network of the encoder part is a lightweight convolutional neural network MobileNetV2; The multi-scale feature extraction module of the encoder part is a DenseASPP module of dense void spatial pyramid pooling; The image semantic segmentation model adopts the CBAM module that combines the channel attention mechanism and the spatial attention mechanism.

3. The method for space station in-station device interaction based on scene semantic segmentation according to claim 1, characterized in that: The image segmentation loss function is obtained by weighted calculation of multiple image semantic category loss functions; the category loss function is a cross entropy loss function.

4. The method for device interaction within a space station based on scene semantic segmentation according to claim 1, characterized in that: The step of inputting the current scene device image into the image semantic segmentation model to obtain the device segmentation image includes: Preprocess the current scene device image to obtain a processed device image; Based on the processed device image, preliminary feature extraction is performed through the backbone network to obtain the first feature map; According to the first feature map, the CBAM module is used to perform feature enhancement to obtain a third feature map; The third feature map is input into the DenseASPP module for multi-scale feature extraction to obtain a multi-scale dense feature map; Based on the first feature map, resolution recovery is performed according to the multi-scale dense feature map to obtain a device segmentation image.

5. The method for interacting with devices in a space station based on scene semantic segmentation according to claim 4, characterized in that: The method of performing feature enhancement using the CBAM module according to the first feature map to obtain a third feature map includes: Based on the multi-layer perceptron, channel attention calculation is performed according to the first feature map to obtain a channel attention map; Based on the channel attention map, the channel feature of the first feature map is enhanced to obtain the second feature map; Based on the 7×7 convolutional layer, spatial attention calculation is performed according to the second feature map to obtain the spatial attention map; Based on the channel attention map and the spatial attention map, feature enhancement is performed according to the first feature map to obtain a third feature map.

6. The method for space station in-station device interaction based on scene semantic segmentation according to claim 1, characterized in that: The space station device database is based on the interaction requirements and the device segmentation image, and the space station device information is retrieved in the UI interface, including: Extract information based on the device segmentation image to obtain device image information; Based on the space station equipment database, data query is performed according to the equipment image information to obtain detailed equipment information; Based on interaction needs, space station device information is retrieved through the UI interface according to device image information and device detailed information.

7. A space station in-station device interaction system based on scene semantic segmentation, wherein the space station in-station device interaction system based on scene semantic segmentation is used to implement the space station in-station device interaction method based on scene semantic segmentation according to any one of claims 1 to 6, characterized in that: The system includes a camera, an electronic device, and a touch screen, wherein: The camera is used to obtain the current scene device image; The electronic device is used to collect images of devices in the space station and obtain training data; construct a semantic segmentation model for the image to be trained based on the model structure of the DeepLabv3+ network; train the semantic segmentation model for the image to be trained using the training data based on the image segmentation loss function to obtain an image semantic segmentation model; input the current scene device image into the image semantic segmentation model to obtain a device segmentation image; The touch screen is used to obtain interaction requirements; based on the space station equipment database, according to the interaction requirements and the equipment segmentation image, the space station equipment information is retrieved in the UI interface.

8. The method for interacting with devices in a space station based on scene semantic segmentation according to claim 7, characterized in that: The electronic device is further used for: Preprocess the current scene device image to obtain a processed device image; Based on the processed device image, preliminary feature extraction is performed through the backbone network to obtain the first feature map; According to the first feature map, the CBAM module is used to perform feature enhancement to obtain a third feature map; The third feature map is input into the DenseASPP module for multi-scale feature extraction to obtain a multi-scale dense feature map; Based on the first feature map, resolution recovery is performed according to the multi-scale dense feature map to obtain a device segmentation image.

9. A space station device interaction device, characterized in that: The space station in-station device interaction device includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images