A semantic-driven based cross-modal disorder detection method

Abstract

The application relates to the technical field of obstacle detection, and is a cross-modal obstacle detection method based on semantic driving. First, a semantic driving transmission framework is proposed, the framework extracts task-related semantics, and the stability to channel noise is improved through source-channel joint coding; then a lightweight cross-modal semantic fusion method is designed, under the premise of maintaining the integrity of the intra-modal semantics, the learning of the complementary information of the multi-modal semantics is realized; finally, a dataset containing typical obstacles such as sudden pedestrians and irregular obstacles is constructed to evaluate the performance of the proposed method. The application solves the problems of large transmission delay and low detection precision existing in the traditional scheme; first, only the task-related semantics are transmitted, the data amount of transmission is effectively reduced, and the transmission delay is reduced; in addition, the semantic relationship of the multi-modal signals is mined through cross-modal semantic learning, the multi-modal signals can realize semantic complementation, and therefore the detection precision is improved.

Description

Technical Field

[0001] This invention relates to the field of obstacle detection technology, and more specifically, to a semantically driven cross-modal obstacle detection method. Background Technology

[0002] With the continuous innovation and development of computer technology, obstacle detection technology has been widely applied in various fields. For example, in the field of autonomous vehicles, obstacle detection technology is used to identify and track other vehicles, pedestrians, traffic signs, and road conditions in real time to ensure driving safety. In the manufacturing and industrial automation fields, robots and automated guided vehicles (AGVs) use obstacle detection technology to coordinate and control equipment to avoid collisions and improve production efficiency. Furthermore, obstacle detection technology is also applied in areas such as drones, agricultural machinery, and medical robots.

[0003] Taking Automated Guided Vehicles (AGVs) in the Industrial Internet of Things (IIoT) as an example, they are deployed in workplaces such as smart factories and logistics centers to facilitate intelligent material handling operations. As a new type of intelligent vehicle, AGVs are equipped with various types of sensors. These sensors enable them to intelligently transport materials according to predetermined routes and instructions, thus improving operational efficiency while reducing labor costs. However, due to the complexity and uncertainty of the real environment, there may be many obstacles in the AGV's predetermined route, such as irregularly placed boxes and fast-moving pedestrians. The presence of these obstacles can lead to reduced transportation efficiency. Therefore, a fast and accurate obstacle detection algorithm is needed to enable AGVs to perceive their surroundings and guide the path planning process.

[0004] Existing obstacle detection methods can be broadly categorized into two types based on their transmission architecture: device-side methods and device-edge collaborative methods. Firstly, device-side obstacle detection has been extensively researched and widely applied in fields such as autonomous driving. However, for devices with limited computing power, relying solely on their limited resources is insufficient for timely and reliable detection. To address this, a device-edge collaborative method has been proposed, which transmits the perceived data completely to an edge server, leveraging the server's powerful computing capabilities to perform the detection task. However, a significant portion of the image content is irrelevant to the final detection result; therefore, transmitting the complete data to the decision-maker is unnecessary. Furthermore, due to the large volume of data transmitted and limited computing resources, transmitting the complete data to the edge server cannot meet latency requirements.

[0005] Furthermore, a reliable obstacle detection scheme not only needs to meet the requirements of low latency but also needs to ensure detection accuracy. Currently, methods to ensure high reliability in obstacle detection can be broadly categorized into two types: single-modal obstacle detection and multi-modal fusion-based obstacle detection. J. Redmon et al., in their paper "YOLOv3: An incremental improvement," mentioned using single-modal data such as RGB, depth, and infrared images to perform detection tasks. However, using single-modal signals makes accurate detection difficult. For example, RGB images are not suitable for environments with insufficient lighting, depth images may lack complex details, and infrared images may not achieve accurate distance detection. Subsequently, in the context of multimodal learning, Q. Jiang et al., in their paper "Attention-based cross-modality feature complementation for multispectral pedestrian detection," proposed a multimodal fusion scheme for obstacle detection, employing static fusion strategies such as early fusion, mid-term fusion, and late-term fusion. These studies show that using infrared and depth images or RGB and depth images can achieve better detection results than single-modal methods. However, images of different modalities exhibit different characteristics, which can cause accuracy to vary when recognizing different types of objects. Therefore, G.Li et al. designed an adaptive weighting network in their paper "Cross-modal weighting network for RGB-D salient object detection" to enhance the system's stability and detection accuracy. However, while exploring intermodal complementarity, these methods neglect maintaining the semantic integrity within a modality. Furthermore, since signals from various modalities exhibit different characteristics under different conditions, existing techniques struggle to effectively handle the heterogeneity between different modalities. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a semantically driven cross-modal obstacle detection method for device-side applications. The specific technical solution is as follows:

[0007] A semantically driven cross-modal obstacle detection method is proposed. First, a semantically driven transmission framework is constructed, employing the mainstream target detection network YOLOv5, which is divided into two parts based on network structure and model parameter quantity. Second, data compression is considered to utilize the limited computing resources on the device, effectively reducing transmission latency by decreasing data volume and achieving rapid detection. Furthermore, cross-modal semantic learning is used to study the semantic relationships between multimodal signals, and a novel cross-modal semantic fusion method is proposed to achieve deep interaction and fusion of the semantics of the depth map and the infrared map, thereby improving detection accuracy.

[0008] Includes the following steps:

[0009] Step 1: On the device side, process the single-modal data of infrared and depth images to extract semantic information relevant to the task.

[0010] Step 2: A novel cross-modal semantic fusion model was designed to fuse semantic information from depth images and infrared images. The specific details of the cross-modal semantic fusion model are as follows:

[0011] First, using the single-modal semantic information obtained in step 1, intra-modal representation learning is performed to enhance the semantic information of depth images and infrared images;

[0012] Then, in order to make full use of information from different modalities, intermodal complementary learning was designed;

[0013] Finally, in order to obtain the fused semantic information, the intra-modal high-order semantics obtained by intra-modal representation learning and the complementary semantics obtained by inter-modal complementarity learning are fused together.

[0014] Step 3: A Joint Source-Channel Encoder (JSC encoder) was developed to compress the fused semantic data, thereby reducing transmission latency. The compressed fused semantics were directly mapped to the channel input symbols to achieve joint source-channel coding transmission. The JSC decoder was used on the edge server to decode and decompress the fused semantics.

[0015] Step 4: Perform semantic decoding on the edge server side to detect obstacles and provide support for subsequent path planning.

[0016] A further improvement to the present invention includes the following process in step 1:

[0017] Procedure 1-1: Using a local infrared camera and LiDAR, infrared and depth signals are acquired respectively for a large-scale dataset consisting of depth and infrared images. Where M is the number of pairs of depth maps and infrared maps. s jFor the j-th pair of depth map and infrared map, the j-th depth map... and the j-th infrared image Input the semantic extraction networks for depth maps and infrared maps respectively, and extract the corresponding semantic information f from the depth map and infrared map respectively. D and f I ;

[0018] In steps 1-2, after data acquisition, a deep neural network (DNN) is used as a semantic encoder to map the input depth image and infrared image to a high-dimensional semantic space to obtain semantic information related to the task. To ensure that the computing resources on the device are sufficient to support the operation, the number of parameters for the semantic encoding and fusion parts are 0.312Mb and 0.219Mb, respectively.

[0019] Specifically, a single-modal semantic encoder includes the following components:

[0020] The YOLOv5 object detection model utilizes the Focus, CBL, and CSP modules. The Focus network, a network structure in YOLOv5, aims to reduce the number of parameters to achieve dimensionality reduction while increasing the local receptive field. Through the Focus layer, information from the wh-plane is transformed to the channel dimension, and different features are extracted through convolution. Compared to using other convolutional or pooling layers, the Focus layer effectively reduces information loss due to downsampling and also reduces computation. The CBL is a standard convolutional module, including a regular convolutional layer (Conv), a batch normalization layer (BN), and a LeakyReLU activation function layer. The core idea of ​​the CSP structure is to divide the input feature map into two parts: one part is processed by a small convolutional network (called a sub-network), and the other part is directly processed by the next layer. The two feature maps are then concatenated as the input to the next layer.

[0021] By using the depth map X D and infrared image X I Two unimodal images are transmitted to the first half of YOLOv5 for unimodal semantic extraction, ultimately yielding two unimodal semantic f. D and f I .

[0022] A further improvement to the present invention includes the following process in step 2:

[0023] Step 2-1: An intra-modal representation learning module was constructed to enhance the semantic information of depth and infrared maps. This process is described as follows:

[0024]

[0025]

[0026] Where δ represents the nonlinear activation function SiLU, F D and F I This represents the convolution operation.

[0027] Step 2-2: To fully utilize the information from both depth and infrared images, intermodal complementarity learning is proposed. As mentioned above, depth and infrared images exhibit different characteristics under different conditions; therefore, an effective complementarity learning strategy is constructed to achieve semantic interaction between different modalities, facilitating the filling of information gaps in one modality with complementary information provided by the other. Specifically, a cross-view attention module is used to integrate these two different modalities, allowing depth semantics to focus on infrared semantics for a better understanding of obstacle types.

[0028] First, semantic segmentation is performed to obtain f'. I and f' D This is to ensure that the semantic format of the transformer block matches the input requirements.

[0029] Then, the semantic information of the depth map is used as queries in the transformer block, and the semantic information of the infrared map is used as key and value pairs in the transformer block. The cross-view attention module introduces multi-head attention mechanism and cross-attention operation successively in the depth semantic transformer block to generate f. cr The specific plan is as follows:

[0030]

[0031] in, and d represents different parameter matrices, and d represents the semantic dimension.

[0032] Finally, element-based addition and multilayer perceptrons are used to achieve deep semantic fusion, thereby obtaining f. iter .

[0033] Step 2-3: Integrate the intra-modal and inter-modal modules to obtain the final fused semantics. The intra-modal representation learning module learns the higher-order semantics of each modality to capture information that has a significant impact on the final result; the inter-modal semantic learning module achieves semantic information complementarity. The final fused semantics are obtained by combining the outputs of the two modules, which can be expressed by the following formula:

[0034]

[0035] in, This represents element-wise multiplication. This represents element-wise addition. Furthermore, to avoid directly affecting the depth and infrared semantics of the image, a residual structure is used to ensure f... D and f I The integrity of.

[0036] A further improvement to the present invention includes the following process in step 3:

[0037] Step 3-1: A JSC encoding scheme is designed to compress fused semantic information while reducing data transmission volume and mitigating the impact of noise. This scheme uses a JSC encoder constructed using a deep neural network (DNN) to compress the fused semantics. The role of the JSC encoder is to compress semantic information to reduce the number of symbols to be transmitted and improve robustness to channel noise during transmission.

[0038] Step 3-2: Directly map the compressed fused semantics to the channel input symbols to achieve joint source-channel coding transmission. An additive white Gaussian noise channel is used as the channel model. It is important to note that this assumes the channel model is known during training and remains unchanged during inference.

[0039] Step 3-3: On the edge server side, a JSC decoder is used for decoding to restore the fused semantics and simultaneously reduce channel distortion and interference between users. The role of the JSC decoder is to decompress the received symbols, restore the fused semantics, and minimize the distortion introduced by the channel, thereby achieving accurate data recovery.

[0040] A further improvement to the present invention, step 4 includes the following steps:

[0041] Step 4-1: Use a detection network to obtain the location and type information of the corresponding obstacles. The location information mainly consists of direction and distance information, which is obtained by analyzing depth images. The type information of the obstacles is obtained by feeding the fused semantic information into the latter half of the YOLOv5 model.

[0042] Three specific loss functions are introduced here to perform end-to-end optimization of the semantic fusion and obstacle detection networks. End-to-end optimization ensures that the semantic encoding stage captures task-relevant semantics, the semantic fusion stage obtains complementary semantic information, and ultimately improves the performance of obstacle detection. The specific loss functions are as follows:

[0043] L total =clc loss +α×obj loss +β×box loss ,

[0044] Among them, clc loss obj loss and box loss These represent the classification loss, confidence loss, and localization loss, respectively. α and β are coefficients used to adjust the proportions of the different loss terms.

[0045] Process 4-2: Obstacle detection has a wide range of applications, including obstacle avoidance for automated guided vehicles (AGVs) and industrial robots in the Industrial Internet of Things (IIoT). Taking AGVs in the IIoT as an example, mature robot path planning algorithms, such as those described by N. Peng et al. in the paper "Urban multiple routeplanning model using dynamic programming in reinforcement learning," are used to guide the device to a pre-set destination while effectively avoiding sudden obstacles along the way.

[0046] The beneficial effects of this invention are as follows: This invention designs a communication architecture based on semantic transmission, extracting and transmitting only the semantics relevant to the task, reducing the amount of data transmitted and achieving rapid detection; it utilizes deep learning strategies to achieve semantic mining of multimodal signals, mapping the signals to a high-dimensional semantic space, and designs an effective cross-modal learning strategy to enable complementary learning between multimodal signals, thereby improving detection accuracy; in addition, the constructed semantic-driven transmission architecture reduces end-to-end latency and mitigates the impact of channel noise. Attached Figure Description

[0047] Figure 1 This is a flowchart of a semantically driven cross-modal obstacle detection method according to the present invention.

[0048] Figure 2 This is a diagram of the semantic-driven transmission architecture of the present invention.

[0049] Figure 3 This is a diagram of the cross-modal semantic fusion method of the present invention.

[0050] Figure 4 This is a structural diagram of the JSC encoder and JSC decoder of the present invention.

[0051] Figure 5 This is a performance comparison chart of different schemes of the present invention under different signal-to-noise ratios. Detailed Implementation

[0052] To enhance understanding of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. These embodiments are only used to explain the invention and do not limit the scope of protection of the invention.

[0053] Example: A semantically driven cross-modal barrier detection method, such as... Figure 1-3 As shown, the method includes the following steps:

[0054] Step 1: On the device side, process the single-modal data of infrared and depth images to extract semantic information relevant to the task;

[0055] Procedure 1-1: Use a local infrared camera and LiDAR to acquire infrared and depth signals respectively. This is for large-scale datasets consisting of depth and infrared images. Where M is the number of pairs of depth maps and infrared maps. s j For the j-th pair of depth map and infrared map, the j-th depth map... and the j-th infrared image The data are passed to the depth map semantic extraction network and the infrared map semantic extraction network respectively, and the corresponding semantic information of the depth map and infrared map is extracted respectively.

[0056] The architecture of the entire network and the characteristics of the corresponding depth map are as follows:

[0057] Conv2d-1 [-1,32,320,320] 3,456 BatchNorm2d-2 [-1,32,320,320] 64 SiLU-3 [-1,32,320,320] 0 Conv-4 [-1,32,320,320] 0 Conv2d-5 [-1,64,160,160] 18,432 BatchNorm2d-6 [-1,64,160,160] 128 SiLU-7 [-1,64,160,160] 0 Conv-8 [-1,64,160,160] 0 Conv2d-9 [-1,32,160,160] 2,048 BatchNorm2d-10 [-1,32,160,160] 64 SiLU-11 [-1,32,160,160] 0 Conv-12 [-1,32,160,160] 0 Conv2d-13 [-1,32,160,160] 1,024 BatchNorm2d-14 [-1,32,160,160] 64 SiLU-15 [-1,32,160,160] 0 Conv-16 [-1,32,160,160] 0 Conv2d-17 [-1,32,160,160] 9,216 BatchNorm2d-18 [-1,32,160,160] 64 SiLU-19 [-1,32,160,160] 0 Conv-20 [-1,32,160,160] 0 Bottleneck-21 [-1,32,160,160] 0 Conv2d-22 [-1,32,160,160] 2,048 BatchNorm2d-23 [-1,32,160,160] 64 SiLU-24 [-1,32,160,160] 0 Conv-25 [-1,32,160,160] 0 Conv2d-26 [-1,64,160,160] 4,096 BatchNorm2d-27 [-1,64,160,160] 128 SiLU-28 [-1,64,160,160] 0 Conv-29 [-1,64,160,160] 0 C3-30 [-1,64,160,160] 0

[0058] The architecture of the entire network and the corresponding infrared image characteristics are as follows:

[0059]

[0060]

[0061] In steps 1-2, after data acquisition, a deep neural network (DNN) is used as a semantic encoder to map the input depth and infrared images into a high-dimensional semantic space to obtain task-related semantic information. To ensure sufficient computing resources on the device to support the operation, the parameter sizes for the semantic encoding and fusion parts are 0.312Mb and 0.219Mb, respectively.

[0062] Step 2: A novel cross-modal semantic fusion (CMSF) method is designed to fuse the semantic information of depth images and infrared images. The specific fusion model used in the cross-modal semantic fusion method is as follows:

[0063] First, using the single-modal semantic information obtained in the previous step, intramodal representation learning is performed to enhance the semantic information of depth images and infrared images;

[0064] Then, in order to make full use of information from different modalities, intermodal complementary learning was designed;

[0065] Finally, in order to obtain the fused semantic information, the intra-modal high-order semantics obtained by intra-modal representation learning and the complementary semantics obtained by inter-modal complementarity learning are fused together.

[0066] Step 2 specifically includes the following process:

[0067] Step 2-1: An intra-modal representation learning module was constructed to enhance the semantics of depth and infrared maps. This process is described as follows:

[0068]

[0069]

[0070] Where δ represents the nonlinear activation function SiLU, F D and F I This represents the convolution operation.

[0071] Step 2-2: To fully utilize the information from both depth and infrared images, intermodal complementarity learning is proposed. As mentioned above, depth and infrared images exhibit different characteristics under different conditions; therefore, an effective complementarity learning strategy is constructed to achieve semantic interaction between different modalities, facilitating the filling of information gaps in one modality with complementary information provided by the other. Specifically, a cross-view attention module is used to integrate these two different modalities, allowing depth semantics to focus on infrared semantics for a better understanding of obstacle types.

[0072] First, semantic segmentation is performed to obtain f'. I and f' D This is to ensure that the semantic format of the transformer block matches the input requirements.

[0073] Then, the semantic information of the depth map is used as queries in the transformer block, and the semantic information of the infrared map is used as key and value pairs in the transformer block. The cross-view attention module introduces multi-head attention mechanism and cross-attention operation successively in the depth semantic transformer block to generate f. cr The specific plan is as follows:

[0074]

[0075] in, and d represents different parameter matrices, and d represents the semantic dimension.

[0076] Finally, element-based addition and multilayer perceptrons are used to achieve deep semantic fusion, thereby obtaining f. iter .

[0077] Step 2-3: Integrate the intra-modal and inter-modal modules to obtain the final fused semantics. The intra-modal representation learning module learns the higher-order semantics of each modality to capture information that has a significant impact on the final result; the inter-modal semantic learning module achieves semantic information complementarity. The final fused semantics are obtained by combining the outputs of the two modules, which can be expressed by the following formula:

[0078]

[0079] in, This represents element-wise multiplication. This represents element-wise addition. Furthermore, to avoid directly affecting the depth and infrared semantics of the image, a residual structure is used to ensure f... D and f I The integrity of.

[0080] Step 3: A Joint Source-Channel Encoder (JSC encoder) was developed to compress the amount of data in the fused semantics, thereby reducing transmission latency; the compressed fused semantics were directly mapped to the channel input symbols to achieve joint source-channel coding transmission; and the JSC decoder was used on the edge server to decode and decompress the fused semantics.

[0081] Step 3-1: A JSC encoding scheme is designed to compress fused semantic information while reducing data transmission volume and mitigating the impact of noise. This scheme uses a JSC encoder constructed using a deep neural network (DNN) to compress the fused semantics. The role of the JSC encoder is to compress semantic information to reduce the number of symbols to be transmitted and improve robustness to channel noise during transmission.

[0082] Step 3-2: The compressed fused semantics are directly mapped to the channel input symbols to achieve joint source-channel coding transmission. An additive white Gaussian noise (AWGN) channel is used as the channel model, and the effects of static and slow-fading channels are considered. It is important to note that it is assumed that the channel model is known during training and remains unchanged during inference.

[0083] The characteristics of an AGWN channel are as follows: Given a channel input vector x∈C B and complex channel input signal x i The output y∈C is obtained from y=x+z. B Among them, z i ~CN(0,σ2 () is a noise source z∈C B Elements independently distributed in i = 1, ..., B are used to impose an average power constraint on the input vector, such that... In the case of a static AWGN channel, converted to maximum received SNR (dB),

[0084] Step 3-3: On the edge server side, a JSC decoder is used for decoding to restore the fused semantics and simultaneously reduce channel distortion and interference between users. The role of the JSC decoder is to decompress the received symbols, restore the fused semantics, and minimize the distortion introduced by the channel, thereby achieving accurate data recovery.

[0085] Step 4: Perform semantic decoding on the edge server side to detect obstacles and provide support for subsequent path planning.

[0086] Step 4-1: Use a detection network to obtain the location and type information of the corresponding obstacles. The location information mainly consists of direction and distance information, which is obtained by analyzing depth images. The type information of the obstacles is obtained by feeding the fused semantic information into the latter half of the YOLOv5 model.

[0087] Three specific loss functions are introduced here to perform end-to-end optimization of the semantic fusion and obstacle detection networks. End-to-end optimization ensures that the semantic encoding stage captures task-relevant semantics, the semantic fusion stage obtains complementary semantic information, and ultimately improves the performance of obstacle detection. The specific loss functions are as follows:

[0088] L total =clc loss +α×obj loss +β×box loss ,

[0089] Among them, clc loss obj loss and box loss These represent the classification loss, confidence loss, and localization loss, respectively. α and β are coefficients used to adjust the proportions of the different loss terms.

[0090] Process 4-2: Obstacle detection has a wide range of applications, including obstacle avoidance for automated guided vehicles (AGVs) and industrial robots in the Industrial Internet of Things (IIoT). Taking AGVs in the IIoT as an example, mature robot path planning algorithms, such as those described by N. Peng et al. in the paper "Urban multiple routeplanning model using dynamic programming in reinforcement learning," are used to guide the device to a pre-set destination while effectively avoiding sudden obstacles along the way.

[0091] The experimental results below show that, compared with existing methods, the obstacle detection method of this invention, which utilizes the complementary fusion of multimodal semantics, achieves better detection results.

[0092] This invention constructs a multimodal dataset of common obstacles encountered in the Industrial Internet of Things (IIoT) for experimental purposes. Specifically, it focuses on three types of obstacles: irregularly placed boxes, chairs, and randomly appearing pedestrians. A total of 1000 obstacle samples were collected, each containing a depth image and its corresponding infrared image. To meet the input requirements of YOLOv5, the sample size was adjusted to 640×640. 80% of the samples in the dataset were randomly selected for training, while the remaining 20% ​​were used for testing and performance evaluation.

[0093] In addition, some hyperparameters were set: mini-batch size of 32, epochs of 500, Ada optimizer for network optimization, and learning rate of 0.003. The hardware consisted of an Arduino Nano board configured for the AGV, an Nvidia GeForce RTX4090 GPU as the edge server, and a transmission bandwidth of 300Mb / s. The following four methods were tested for experimental comparison:

[0094] Existing methods include: Method 1, which inputs the depth image separately into the YOLOv5 network to test the single-modal obstacle detection performance; Method 2, which inputs the infrared image separately into the YOLOv5 network to test the single-modal obstacle detection performance; Method 3, which inputs the semantic concatenation fusion of the depth map and the infrared map into the YOLOv5 network to test the obstacle detection performance; and Method 4, which inputs the semantic data based on self-attention fusion into the YOLOv5 network to test the obstacle detection performance.

[0095] The experiment used widely used target detection metrics introduced by MS-COCO: Precision, Recall, mAP50, and mAP50:95, and compared the detection rates of each method by measuring end-to-end delay.

[0096] Table 1 shows the experimental results of this invention.

[0097] Existing Method 1 0.817 0.791 0.805 0.443 Existing Method 2 0.883 0.851 0.865 0.505 Existing Method 3 0.887 0.882 0.909 0.531 Existing Method 4 0.891 0.801 0.853 0.503 Existing Method 5 0.893 0.884 0.897 0.573

[0098] The results in Table 1 show that the proposed method has obvious advantages compared with the existing methods mentioned above. The reasons are as follows: (1) The semantic-driven transmission strategy reduces the amount of data transmitted, effectively reduces end-to-end latency, and thus improves the real-time performance of obstacle detection; (2) The cross-modal semantic fusion strategy promotes the interaction of semantic information, enabling complementary learning between different modalities.

[0099] After comparing the average detection performance of obstacles, the performance of the method of the present invention on different obstacles was tested.

[0100] Table 2 presents the quantitative results for different types of obstacles.

[0101] All obstacles 0.893 0.884 0.897 0.573 Randomly appearing pedestrians 0.938 0.85 0.879 0.549 irregularly arranged boxes 0.86 0.88 0.898 0.515 Chair 0.881 0.923 0.914 0.656

[0102] As can be seen from the results in Table 2, the method of the present invention performs well in different types of obstacles, highlighting the effectiveness and versatility of the method.

[0103] The following are end-to-end latency statistics for three different solutions: device-only, device-based, and device edge server:

[0104] Transmission latency of traditional solutions:

[0105] The data size is 640×640×3×2×8 (bits), the bandwidth is 300Mb, and the transmission delay is: 640×640×3×2×8 / 0.3×10 9 = 65.536ms, the total time is 65.5ms (transmission time) + 8ms (time before fusion) + 28ms (time after fusion) = 101.536ms;

[0106] Transmission time of semantic communication:

[0107] The fused semantics are 160×160×64×32 (float32), and the semantics compressed by the JSC encoder are 160×160×64×32 / 10. The transmission latency is 160×160×64×32 / 10 / 0.3×10. 9 =17.476ms, the total time is 32ms + 17.476ms + 28ms = 77.476ms.

[0108] Table 3 shows a comparison of end-to-end delays for different transmission schemes.

[0109] Devices only 165.284 Device edge (traditional communication) 101.536 Device edge (semantics-driven communication) 77.476

[0110] As can be clearly seen from Table 3, the transmission architecture designed in this invention can effectively reduce end-to-end latency, thereby improving the real-time performance of obstacle detection. The reasons are as follows: (1) The semantic-driven transmission strategy reduces the amount of data transmitted; (2) The edge server has sufficient computing resources, which accelerates the detection process.

[0111] from Figure 4 As can be seen, compared with traditional communication schemes, the semantic-driven transmission scheme proposed in this invention exhibits higher stability in the presence of channel noise. Specifically, semantic-driven communication is more stable under low signal-to-noise ratio conditions and can achieve ideal channel performance more quickly.

[0112] Figure 5 This is a performance comparison chart of different schemes of the present invention under different signal-to-noise ratios.

[0113] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A semantically driven cross-modal obstacle detection method, characterized in that, Includes the following steps: Step 1: On the device side, process the single-modal data of infrared and depth images to extract semantic information relevant to the task; Step 2: A cross-modal semantic fusion method is designed to fuse the semantic information of depth images and infrared images. The specific fusion model used in the cross-modal semantic fusion method is as follows: First, using the unimodal semantic information obtained in step 1, intramodal representation learning is performed. Then, intermodal complementary learning was designed. Finally, the intramodal high-order semantics obtained from intramodal representation learning and the complementary semantics obtained from intermodal complementarity learning are fused together. Step 3: A joint source-channel encoder was developed to directly map the compressed fused semantics to the channel input symbols. The joint source-channel encoder was used on the edge server to decode and decompress the fused semantics. Step 4: Perform semantic decoding on the edge server side to detect obstacles and provide support for subsequent path planning. Step 2 includes the following process: Step 2-1: An intra-modal representation learning module was constructed to enhance the semantic information of depth and infrared maps. This process is described as follows: ， in, SiLU represents the nonlinear activation function. and This represents the convolution operation; Procedure 2-2 proposes intermodal complementarity learning, which integrates two different modalities using a cross-view attention module: First, semantic segmentation is performed to obtain... and This is to ensure that the semantic format of the transformer block matches the input requirements; Then, the semantic information of the depth map is used as queries in the transformer block, and the semantic information of the infrared map is used as key-value pairs in the transformer block. The cross-view attention module introduces multi-head attention and cross-attention operations successively in the depth semantic transformer block to generate... The specific plan is as follows: ， in, and Representing different parameter matrices, The dimension representing semantics; Finally, element-based addition and multilayer perceptrons are used to achieve deep semantic fusion, thereby obtaining... ; Step 2-3: Integrate the intra-modal and inter-modal modules to obtain the final fused semantics. The intra-modal representation learning module learns the higher-order semantics of each modality to capture information that has a significant impact on the final result. The inter-modal semantic learning module achieves semantic information complementarity. The final fused semantics is obtained by combining the outputs of the two modules. This process is expressed by the following formula: ， in, This represents element-wise multiplication. This represents element-wise addition, using a residual structure to ensure... and The integrity of.

2. The semantically driven cross-modal obstacle detection method according to claim 1, characterized in that, Step 1 includes the following process: Procedure 1-1: Use local devices, namely infrared cameras and lidar, to collect infrared signals and depth signals respectively; In steps 1-2, after data acquisition is completed, a deep neural network is used as a semantic encoder to map the input depth image and infrared image to a high-dimensional semantic space to obtain semantic information related to the task.

3. The semantically driven cross-modal obstacle detection method according to claim 2, characterized in that, In process 1-1, for a large-scale dataset consisting of depth images and infrared images... ,in The number of paired depth maps and infrared maps. For the first For depth maps and infrared maps, the first Depth map and the Infrared image The semantic information of the depth map and the infrared map are respectively passed to the semantic extraction network and the semantic extraction network of the infrared map to extract the semantic information of the corresponding depth map and infrared map; in the process 1-2, the number of parameters of the semantic encoding and fusion parts are 0.312Mb and 0.219Mb, respectively.

4. The semantically driven cross-modal obstacle detection method according to claim 3, characterized in that, Step 3 includes the following process: Process 3-1: A JSC encoding scheme is designed, which uses a JSC encoder constructed through a deep neural network to compress and fuse semantics. Step 3-2: Directly map the compressed fused semantics to the channel input symbols to achieve joint coding and transmission of the source and channel, using additive white Gaussian noise as the channel model; Step 3-3: Use a JSC decoder on the edge server side to decode in order to restore the fused semantics and at the same time reduce channel distortion and interference between users.

5. The semantically driven cross-modal obstacle detection method according to claim 4, characterized in that, Step 4 includes the following process: Step 4-1: Use a detection network to obtain the location and type information of the corresponding obstacles. The location information mainly consists of direction and distance information, which is obtained by analyzing depth images. The type information of the obstacles is obtained by feeding the fused semantic information into the latter half of the YOLOv5 model. Process 4-2: Use mature robot path planning algorithms to guide the equipment to the pre-set destination.

6. The semantically driven cross-modal obstacle detection method according to claim 5, characterized in that, In process 4-1, three specific loss functions are introduced to perform end-to-end optimization of the semantic fusion and obstacle detection networks. The specific loss functions are as follows: , in, These represent classification loss, confidence loss, and localization loss, respectively. and This represents a coefficient used to adjust the proportions of different loss items.

Images