A semantic feature coding method and device under an intelligent connected vehicle detection task

Abstract

The application belongs to the technical field of intelligent networked vehicles, and provides a semantic feature coding method and device under intelligent networked vehicle detection tasks. The method comprises: using a weakly supervised semantic feature extraction model to perform semantic feature extraction on an input image; using a self-adaptive semantic coding model to code each semantic feature extracted to obtain semantic coding of the input image. The application can perform semantic feature extraction on product input images, and after channel coding, uses wireless channel transmission. Not only can the transmission data volume be greatly reduced, but also the low signal-to-noise ratio environment in the actual detection industrial scene can be adapted, the semantic feature mining precision of the input image is improved, and the adaptability of the semantic feature in the complex channel scene transmission is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent networked vehicles, in particular to a semantic feature coding method and device under intelligent networked vehicle detection tasks. BACKGROUND

[0002] Manufacturing is the backbone of the national economy, and its high-quality development is crucial to building a modern economic system. With the development of intelligent networked vehicle manufacturing and increasing competition, enterprises are increasingly demanding automation and efficiency of production lines. Traditional manual detection methods are time-consuming and labor-intensive, and are easily influenced by subjective factors. The introduction of intelligent networked vehicle product detection technology can achieve fast and accurate detection, improving the automation level of production lines. However, defects in vehicle products can directly affect product quality and user experience, so strengthening detection capabilities and improving manufacturing quality are of great significance to promoting the innovative development of China's intelligent networked vehicle manufacturing industry. Vehicle product surface quality inspection is an important part of quality inspection, and visual quality inspection of vehicle products focuses on key technologies such as high-quality sensors, high-stability light sources, high-precision recognition models, and low-latency recognition to improve the accuracy and real-time performance of vehicle product surface defect recognition. With the continuous advancement of computer technology, especially the development of image processing and some deep learning algorithms, computers can better process and analyze image data. This provides strong support for the development of intelligent networked vehicle product detection technology, enabling detection algorithms to more accurately identify and classify product defects.

[0003] Complex noise environments in industrial sites can cause severe distortion during vehicle product image acquisition and transmission, greatly affecting product defect detection accuracy, and inputting large amounts of pictures to recognition networks occupies a large amount of bandwidth resources.

[0004] Most existing task-oriented semantic communication transmission systems have adaptability problems under channel changes, and ignore the influence of product configuration, layout, and other features on defect location probability, leading to inaccurate semantic feature extraction and affecting product quality inspection accuracy. SUMMARY

[0005] To address the shortcomings of the prior art, the present application provides a semantic feature coding method and device under intelligent networked vehicle detection tasks to reduce data transmission volume, improve semantic feature mining accuracy of input images, and improve the adaptability of semantic features in complex channel scenarios.

[0006] A semantic feature coding method under intelligent networked vehicle detection tasks, characterized by comprising:

[0007] A weakly supervised semantic feature extraction model is used to extract semantic features from input images.

[0008] Encode the extracted semantic features using the adaptive semantic encoding model to obtain semantic encoding of the input image.

[0009] Further, the weakly supervised semantic feature extraction model is trained using weakly supervised training data.

[0010] Further, the weakly supervised training data is obtained in the following way:

[0011] using a pre-trained convolutional neural network to extract features from the input image to obtain a first feature map;

[0012] sampling the first feature map to obtain a second feature map using a self-attention mechanism to extract features from the sampled first feature map;

[0013] performing importance sorting on the second feature map to obtain a feature importance probability score for each channel;

[0014] multiplying the feature importance probability score of each channel with the first feature map element by element to obtain a third feature map;

[0015] performing channel summation on the third feature map to generate a feature importance map;

[0016] converting the feature importance map into a binary weak label map;

[0017] pairing the weak label map with the input image to form weakly supervised training data.

[0018] Further, the probability map sampling includes:

[0019] detecting the input image using an object detector to obtain a product category;

[0020] obtaining a posterior probability of defect site of the product category according to the product category and the prior probability of defect site of the pre-defined product category;

[0021] sampling the first feature map according to the posterior probability of defect site of the product category to obtain the sampled first feature map.

[0022] Further, the importance sorting on the second feature map to obtain a feature importance probability score for each channel includes:

[0023] performing global average pooling on the second feature map to obtain an average value of the feature map of each channel;

[0024] mapping the average value of the feature map of each channel to a non-negative space using an activation function ReLU to obtain a mapped channel feature map;

[0025] The mapped channel feature map is normalized to obtain a normalized channel feature map;

[0026] A probability distribution of pixel values of the normalized channel feature map is calculated using a softmax function to obtain a feature importance probability score of each channel.

[0027] Further, the semantic features include defect site semantic features, image global semantic features, and channel statistical semantic features.

[0028] Further, the adaptive semantic encoding model is obtained in the following manner:

[0029] The defect site semantic features, the image global semantic features, and the channel statistical semantic features are fused and reconstructed to obtain first multi-modal features;

[0030] The first multi-modal features are compressed by adding a signal-to-noise ratio condition to obtain second multi-modal features;

[0031] The second multi-modal features are encoded using a channel encoder to obtain semantic encoding of the input image.

[0032] Further, the defect site semantic features, the image global semantic features, and the channel statistical semantic features are fused and reconstructed to obtain first multi-modal features, including:

[0033] The defect site semantic features, the image global semantic features, and the channel statistical semantic features are weighted to obtain weighted RGB features;

[0034] The weighted RGB features are aligned with the image global semantic features, and convolution fusion is performed to obtain convolution fusion features;

[0035] The RGB features are fused and reconstructed with the convolution fusion features using a residual connection operation to obtain first multi-modal features.

[0036] Further, the first multi-modal features are compressed by adding a signal-to-noise ratio condition to obtain second multi-modal features, including:

[0037] The first multi-modal features are pooled using a global average pooling function;

[0038] The pooled first multi-modal features are added to different signal-to-noise ratio conditions to obtain context input information based on the signal-to-noise ratio;

[0039] The context information is fed into a fully connected neural network to generate a scaling factor, and the generated scaling factor is multiplied by the first multi-modal features to obtain first multi-modal features with different proportion weights;

[0040] The proportion weights are sorted to obtain second multi-modal features.

[0041] An adaptive semantic feature coding device under an intelligent networked vehicle detection task, comprising an extraction unit and a coding unit, wherein:

[0042] The extraction unit is configured to perform semantic feature extraction on the input image by using a weakly supervised semantic feature extraction model.

[0043] The coding unit is configured to code each semantic feature extracted by using an adaptive semantic coding model to obtain semantic coding of the input image.

[0044] The present application is directed to the problem of image transmission distortion affecting the accuracy of quality inspection caused by large image noise and poor channel quality in the scene of vehicle product visual quality inspection, and innovatively uses a semantic communication architecture, integrates quality inspection task features and product layout knowledge, proposes a weakly supervised semantic feature extraction method to estimate the parts of the input image prone to defects, and further designs a channel-adaptive semantic feature coding method that reduces the amount of transmitted data and improves the semantic feature mining accuracy of the input image and the adaptability of semantic features in complex channel scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the specific embodiments of the present application, the drawings required to be used in the specific embodiments or prior art description will be briefly introduced below. In all the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0046] Figure 1 A flow chart of a semantic feature coding method under an intelligent networked vehicle detection task provided by an embodiment of the present application;

[0047] Figure 2 A structural schematic diagram of a semantic feature coding device under an intelligent networked vehicle detection task provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] The embodiments of the technical solutions of the present application will be described in detail below with reference to the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0049] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present application should be understood as the usual meanings understood by the skilled in the art to which the present application belongs.

[0050] In one embodiment, the present application provides a semantic feature encoding method under the intelligent connected vehicle detection task, which can extract semantic features from product input images and transmit them through wireless channels after channel encoding. This not only greatly reduces the amount of transmission data, but also adapts to the low signal-to-noise ratio environment in actual detection industrial scenarios. Moreover, after semantic feature extraction and channel encoding transmission, the product input image can be input into the quality inspection model to determine whether the vehicle product image has defects.

[0051] A semantic feature encoding method under the intelligent connected vehicle detection task, comprising:

[0052] 1. Using a weakly supervised semantic feature extraction model to extract semantic features from the input image;

[0053] In one embodiment, the weakly supervised semantic feature extraction model is trained using weakly supervised training data.

[0054] Specifically, the weakly supervised training data is obtained in the following way:

[0055] (1) Using a pre-trained convolutional neural network to extract features from the input image to obtain a first feature map;

[0056] (2) Sampling the first feature map to obtain a second feature map using a self-attention mechanism to extract features from the sampled first feature map;

[0057] Using a self-attention mechanism to further extract features from the sampled first feature map, each pixel (or position) of the sampled first feature map will be compared with all other pixels to evaluate the degree of correlation between them, which can better capture global information. In this way, the weakly supervised semantic feature extraction model can identify which regions are most important for understanding the entire input image.

[0058] In one embodiment, the probability map sampling includes:

[0059] ① Using an object detector to detect the input image to obtain the product category;

[0060] ② Obtaining the posterior probability of the defect position of the product category according to the product category and the prior probability of the defect position of the pre-defined product category;

[0061] The defect position prior probability is the probability distribution of the defect position of the product category based on existing knowledge, experience or hypothesis, rather than relying on currently observed data or evidence. Specifically, in product defect detection, the likelihood of the occurrence of a specific defect category (including concave, scratch, crack, etc.), insertion position (referring to the specific position of the defect position in the input image) or scale (referring to the area occupied by the defect position in the entire input image).

[0062] For example, if it is known that a certain type of automotive product is more likely to have defects in a particular region, then the prior probability of defects in that particular region will be higher; the posterior probability is the probability distribution updated in combination with the actual observed data, and by observing the specific features in the input image, the initial prior probability can be updated to obtain a more accurate posterior probability. Using the Bayesian method, the prior probability is updated in combination with the actual observed data to obtain the posterior probability.

[0063] According to the input image I, the prior estimate of the defect position is the category c, the insertion position p and the scale s in the input image I.

[0064] The category refers to defects such as dents, scratches, cracks, etc.; the insertion position refers to the specific position of the defect position in the input image, usually represented by a coordinate, such as (x, y); the scale refers to the area occupied by the defect position in the entire input image.

[0065] The product prior probability o is defined as a vector o = {p, s, c} composed of the parameters of the defect position to be detected (category c, insertion position p and scale s).

[0066] The input image I and the product category label of the input image are c fg , initialize the defect position posterior probability P (o|I, c fg ) of the product category, and sample the image feature map of the prior probability for the first feature map, i.e. P (o|O′).

[0067] Assuming that the detected defect position and the defect position to be detected are independent of each other, the product category defect position posterior probability P (o|O′) of the input image can be decomposed as:

[0068] P (o|O′) = P (p, s, c|O′) = P (c|O′) P (p, s|O′, c) = P (c|O′) P (p|O′, c) P (s|O′, c, p)

[0069] ③ According to the defect position posterior probability of the product category, sample the first feature map to obtain the sampled first feature map.

[0070] ④ Using a self-attention mechanism to extract features from the sampled first feature map to obtain a second feature map

[0071] (3) Importance ranking of the second feature map to obtain the feature importance probability score of each channel;

[0072] In one embodiment, the feature importance probability score of each channel is obtained, including:

[0073] ①global average pooling is performed on the second feature map to obtain the average value of the feature map of each channel;

[0074] ②the average value of the feature map of each channel is mapped to a non-negative space using an activation function ReLU to obtain a mapped channel feature map;

[0075] In one of the embodiments, the position characteristics of the mixed Gaussian distribution can also be used to position constrain the c-type defect sites to be detected and the c-type defect sites detected. For the c-type defect sites to be detected Assuming that the center of the c-type defect site relative to the defect site to be detected satisfies the bimodal mixed Gaussian distribution, since the natural image is still true after being flipped left and right, the relative position probability distribution of the c-type defect site and the defect site to be detected usually has the characteristic of left-right symmetry with the reference (detected defect site) object as the axis, further constraining the mixed Gaussian distribution about the defect site to be detected left-right symmetry, expressed as:

[0076]

[0077] In the formula, F2 is a normalization factor, and are the means of the two Gaussian components of the left-right symmetry, is the corresponding covariance matrix, B is the image background area, N a is the Gaussian distribution.

[0078] ③normalization processing is performed on the mapped channel feature map to obtain a normalized channel feature map;

[0079] ④the probability distribution of the pixel value of the normalized channel feature map is calculated using a softmax function to obtain the feature importance probability score of each channel. It represents the importance degree of the channel in defect detection.

[0080] (4) multiply the feature importance probability score of each channel with the first feature map element by element to obtain a third feature map;

[0081] (5) perform channel summation on the third feature map to generate a feature importance map; according to the feature importance map, the region related to the defect in the input image can be located, so that the weakly supervised semantic feature extraction model pays more attention to the region where the defect site may appear in the input image.

[0082] (6) convert the feature importance map into a binary weak label map;

[0083] (7) Pairing the weak label map with the input image to form weakly supervised training data. According to the pixel value of the feature importance map, the defect position and the normal area are determined, and the feature importance map is converted into a binary weak label map. The generated weak label map is paired with the input image to form weakly supervised training data.

[0084] In one embodiment, the feature importance region can also be matched with the defect position in the input image by minimizing the cross-entropy loss function, and the feature importance region is associated with the defect position, thereby improving the accuracy of defect detection. Specifically, it is expressed as:

[0085]

[0086] where N is the number of pixels in the input image, y i is the true label of the i-th pixel (the label of the defect position), p i is the predicted probability of the i-th pixel (the value of the feature importance map).

[0087] In the initial stage, the pre-trained weakly supervised semantic feature extraction model parameters are used for training to improve the convergence speed and stability of the model.

[0088] Subsequently, the obtained weakly supervised training data is gradually introduced, and the weakly supervised semantic feature extraction model is continuously optimized through iterative training, thereby completing the training and optimization of the semantic feature extraction module of the input image.

[0089] 2, The extracted semantic features are encoded using an adaptive semantic encoding model to obtain the semantic encoding of the input image.

[0090] In one embodiment, the semantic features include defect position semantic features, image global semantic features, and channel statistical semantic features.

[0091] In one embodiment, the adaptive semantic encoding model is obtained in the following manner:

[0092] (1) Fusing and reconstructing the defect position semantic features, image global semantic features, and channel statistical semantic features to obtain a first multi-modal feature;

[0093] In one embodiment, the first multi-modal feature is obtained by:

[0094] ① Weighting the defect position semantic features, image global semantic features, and channel statistical semantic features to obtain a weighted RGB feature;

[0095] Channel statistical semantic features refer to the statistical properties of channels, which describe the behavior of channels when transmitting information.

[0096] defect site semantic feature F p image global semantic feature F g channel statistical semantic feature C a obtain structure attention score A t

[0097] image global semantic feature F g average the values of all spatial positions, thereby obtaining a statistical quantity of each channel, and convert the statistical information of the channel into a channel statistical semantic feature C a ; concatenate the channel statistical semantic feature C a with the image global semantic feature F g and the defect site semantic feature F p to obtain the attention score At.

[0098] image global semantic feature F g is a color feature extracted from the entire input image, which reflects the overall color distribution and attributes of the input image.

[0099] RGB feature F i refers to the RGB feature of each pixel point of the defect site semantic feature F p , the image global semantic feature F g .

[0100] RGB feature F i combined with attention A t to generate a weighted RGB feature F wi , which can be represented as:

[0101]

[0102] wherein, and represent element-level multiplication and concatenation operations, f E represents an expansion operation, so that F g has the same spatial dimension as F p .

[0103] The weighted RGB feature mainly suppresses the weight of unimportant features in the input image and enhances the weight of important features, and also makes the adaptive semantic encoding model more robust to local noise or occlusion.

[0104] ②Align the weighted RGB feature with the image global semantic feature and perform convolution fusion to obtain a convolution fusion feature;

[0105] ③Use residual connection operation, and then fuse and reconstruct details of the RGB feature and the convolution fusion feature to obtain a first multi-modal feature.

[0106] The weighted RGB feature F wi The defect site semantic feature F p After alignment, convolution fusion is performed; then, the weighted RGB feature is re-added for fusion and reconstruction of details; finally, after residual connection, the first multi-modal feature F m Can be expressed as:

[0107]

[0108] Where f RES is a residual connection operation.

[0109] In one of the embodiments, the first multi-modal feature F m And the support mask y s The basic foreground prototype p b Can be expressed as:

[0110]

[0111] Where i[·] represents an indicator function, (x, y) represents the spatial position of the pixel, and c represents the product defect category.

[0112] (2) Add a signal-to-noise ratio condition to compress the first multi-modal feature to obtain a second multi-modal feature;

[0113] In one of the embodiments, obtaining the second multi-modal feature includes:

[0114] ① Using a global average pooling function to perform a pooling operation on the first multi-modal feature;

[0115] ② Adding different signal-to-noise ratio conditions to the pooled first multi-modal feature to obtain signal-to-noise ratio-based context input information;

[0116] ③ Sending the context information into a fully connected neural network to generate a scaling factor, multiplying the generated scaling factor with the first multi-modal feature to obtain the first multi-modal feature with different proportion weights;

[0117] The fully connected neural network mainly consists of two fully connected layers.

[0118] ④ According to the proportion weight, the second multi-modal feature is obtained.

[0119] Selecting the first multi-modal feature with a high proportion weight, compressing the remaining weakly associated first multi-modal feature, and obtaining the second multi-modal feature.

[0120] To adapt to the complex channel environment of vehicle quality inspection site, and solve the problem of excessive consumption of computing resources caused by training multiple signal-to-noise ratio condition models, a soft attention mechanism is added to solve the cross-modal fusion problem of defect site semantic features, image global semantic features and channel statistical semantics under different signal-to-noise ratios.

[0121] (3) The second multi-modal feature is encoded by using a channel encoder to obtain the semantic encoding of the input image.

[0122] The second multi-modal feature is encoded by the channel encoder, and the additive white Gaussian noise (AWGN) channel is considered. The AWGN channel is an independent and identically distributed Gaussian white noise channel, and the samples of the noise are independent of each other. This independence makes channel analysis more convenient and reliable.

[0123] The compressed second multi-modal feature is transmitted through a wireless channel and is destroyed by additive white Gaussian noise. Then it is received by the receiver, and the cross-modal fusion of channel features and image semantic features is performed through the soft attention mechanism, so that the extracted semantic features can adapt to the dynamic actual channel conditions, and the reliability of the input image semantic feature transmission is improved.

[0124] After the receiver receives the signal, it is input into the fully connected classifier to map the signal to the predefined defect category. Then the Softmax function is used to convert the output of the fully connected layer to a probability distribution, representing the probability of each image being judged as a defect category. In this way, the intelligent networked vehicle product detection task is completed.

[0125] The present application aims at the problem of image transmission distortion affecting quality inspection precision caused by large image noise and poor channel quality in vehicle product visual quality inspection scene. A semantic communication architecture is innovatively used to fuse quality inspection task features and product layout knowledge. A weakly supervised semantic feature extraction method is proposed to estimate the parts of the input image prone to defects. Furthermore, a channel adaptive semantic feature encoding method is designed to solve the problems of low input image semantic feature mining precision and poor adaptability to complex channel scenes by fusing multiple channel model statistical distribution features and combining the perceived real-time signal-to-noise ratio state.

[0126] In one embodiment, as shown in Figure 2 An adaptive semantic feature encoding device for intelligent networked vehicle detection tasks is also provided, comprising an extraction unit and an encoding unit, wherein:

[0127] The extraction unit is used to extract semantic features of the input image by using a weakly supervised semantic feature extraction model.

[0128] The encoding unit is used to encode each semantic feature extracted by using an adaptive semantic encoding model to obtain the semantic encoding of the input image.

[0129] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0130] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should be covered in the scope of the claims and the specification of the present application.

Claims

1. A semantic feature coding method under an intelligent connected vehicle detection task, characterized in that, The method comprises the following steps: performing semantic feature extraction on an input image by using a weakly supervised semantic feature extraction model; encoding each of the extracted semantic features by using an adaptive semantic encoding model to obtain semantic encoding of the input image, wherein the semantic features include defect site semantic features, image global semantic features, and channel statistical semantic features; the adaptive semantic encoding model is obtained in the following manner: weighting the defect site semantic features, the image global semantic features, and the channel statistical semantic features to obtain weighted RGB features; aligning the weighted RGB features with the image global semantic features and then performing convolution fusion to obtain convolution fusion features; using a residual connection operation to fuse and reconstruct details of the RGB features and the convolution fusion features to obtain first multi-modal features; performing a global average pooling operation on the first multi-modal features; adding different signal-to-noise ratio conditions to the pooled first multi-modal features to obtain context input information based on signal-to-noise ratio; feeding the context input information into a fully connected neural network to generate a scaling factor, multiplying the generated scaling factor with the first multi-modal features to obtain first multi-modal features with different proportion weights; sorting the proportion weights to obtain second multi-modal features; encoding the second multi-modal features by using a channel encoder to obtain semantic encoding of the input image. 2.The semantic feature coding method under the intelligent vehicle detection task of claim 1, wherein, The weakly supervised semantic feature extraction model is trained by using weakly supervised training data. 3.The semantic feature coding method under the intelligent vehicle detection task of claim 2, wherein, The weakly supervised training data is obtained in the following manner: extracting features of an input image by using a pre-trained convolutional neural network to obtain a first feature map; sampling a probability map from the first feature map, extracting features of the sampled first feature map by using a self-attention mechanism to obtain a second feature map; sorting the importance of the second feature map to obtain a feature importance probability score of each channel; multiplying the feature importance probability score of each channel with the first feature map element by element to obtain a third feature map; performing channel summation on the third feature map to generate a feature importance map; converting the feature importance map into a binary weak label map; pairing the weak label map with the input image to form weakly supervised training data. 4.The semantic feature coding method under the intelligent vehicle detection task of claim 3, wherein, The probability map sampling comprises the following steps: detecting an input image by using an object detector to obtain a product category; obtaining a posterior probability of defect sites of the product category according to the product category and a prior probability of defect sites of the product category predefined in advance; sampling the first feature map according to the posterior probability of defect sites of the product category to obtain a sampled first feature map. 5.The semantic feature coding method under the intelligent vehicle detection task of claim 3, wherein, The sorting of the importance of the second feature map to obtain a feature importance probability score of each channel comprises the following steps: performing global average pooling on the second feature map to obtain an average value of a feature map of each channel; mapping the average value of the feature map of each channel to a non-negative space by using an activation function ReLU to obtain a mapped channel feature map; performing normalization processing on the mapped channel feature map to obtain a normalized channel feature map; A probability distribution of the normalized channel feature image pixel values is calculated using a softmax function to obtain a feature importance probability score for each channel.

6. An adaptive semantic feature coding device under intelligent connected vehicle detection tasks, characterized in that, The method comprises an extraction unit and an encoding unit, wherein: The extraction unit is configured to perform semantic feature extraction on the input image using a weakly supervised semantic feature extraction model. The encoding unit is configured to encode the extracted semantic features, including defect site semantic features, image global semantic features, and channel statistical semantic features, using an adaptive semantic encoding model to obtain semantic encoding of the input image. The adaptive semantic encoding model is obtained in the following manner: the defect site semantic features, image global semantic features, and channel statistical semantic features are weighted to obtain weighted RGB features; the weighted RGB features are aligned with the image global semantic features and then convolution fusion is performed to obtain convolution fusion features; a residual connection operation is used to further fuse and reconstruct details of the RGB features and the convolution fusion features to obtain first multi-modal features; a global average pooling function is used to perform a pooling operation on the first multi-modal features; the pooled first multi-modal features are added to different signal-to-noise ratio conditions to obtain context input information based on the signal-to-noise ratio; the context input information is input into a fully connected neural network to generate a scaling factor, the generated scaling factor is multiplied by the first multi-modal features to obtain first multi-modal features with different proportion weights; the proportion weights are sorted to obtain second multi-modal features; and a channel encoder is used to encode the second multi-modal features to obtain semantic encoding of the input image.

Images