Base station switching method, base station switching device and electronic equipment

Abstract

The invention provides a base station switching method and device and electronic equipment, and relates to the technical field of communication, and the base station switching method comprises the steps that wireless modal characteristics, visual modal characteristics and spatial modal characteristics are obtained, the wireless modal characteristics are used for representing channel information between a base station and a terminal, the visual modal characteristics are used for representing channel information between the base station and the terminal, and the spatial modal characteristics are used for representing channel information between the base station and the terminal; the visual modal features are used for representing information of pictures shot by the base station, and the spatial modal features are used for representing position information of the terminal in the pictures shot by the base station; base station prediction is carried out based on the wireless modal features, the visual modal features and the spatial modal features, and a base station prediction result is obtained; and carrying out base station switching based on the base station prediction result. According to the embodiment of the invention, the optimal service base station can be pre-judged through the wireless modal characteristics, the visual modal characteristics and the spatial modal characteristics, and base station switching is carried out, so that the base station switching effect can be improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a base station switching method, device and electronic equipment. Background Art

[0002] In the fifth generation mobile communication network, the millimeter wave massive multiple input multiple output (MIMO) system is used to increase the data transmission rate. However, due to the short wavelength of millimeter waves, the attenuation in the atmospheric environment is severe, especially in the rainy environment, so the millimeter wave transmission distance is short. In order to compensate for the path loss of millimeter wave transmission, beamforming algorithms are used in related technologies to focus signal energy on the target terminal to improve the transmission signal-to-noise ratio, thereby increasing the transmission rate. Due to the poor diffraction capability and narrow beam characteristics of millimeter wave beamforming, millimeter waves are sensitive to obstruction. In outdoor mobile scenarios, the communication link between the base station and the terminal is often obstructed. Tests have shown that once the communication link is blocked by a large vehicle, the energy of the received signal will be attenuated by more than 20dB. Such severe power attenuation will lead to link interruption and data transmission loss.

[0003] In the related art, base station switching is performed by judging obstruction and interruption based on the deterioration of communication indicators. When the communication indicators decay, obstruction has already occurred, which is not predictable. In addition, the target base station for user switching has a certain degree of randomness, and the optimal service base station at a future moment cannot be judged in advance, resulting in poor effect of base station switching. Summary of the invention

[0004] The embodiment of the present invention provides a base station switching method, a method, a device and an electronic device to solve the problem that the existing base station switching effect is poor.

[0005] To solve the above technical problems, the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present invention provides a base station switching method, the method comprising:

[0007] Acquire wireless modality features, visual modality features, and spatial modality features, wherein the wireless modality features are used to characterize channel information between a base station and a terminal, the visual modality features are used to characterize picture information taken by the base station, and the spatial modality features are used to characterize location information of the terminal in the picture taken by the base station;

[0008] Performing base station prediction based on the wireless modal features, visual modal features, and spatial modal features to obtain a base station prediction result;

[0009] Base station switching is performed based on the base station prediction result.

[0010] Optionally, the acquiring of wireless modality features, visual modality features, and spatial modality features includes:

[0011] Acquire multimodal information at multiple moments, the multimodal information including wireless modal information, visual modal information and spatial modal information, the wireless modal information is channel information between a base station and a terminal, the visual modal information is picture information taken by the base station, and the spatial modal information is location information of the terminal in the picture taken by the base station;

[0012] The multimodal information at the multiple moments is input into the LSTM network to obtain wireless modal features, visual modal features and spatial modal features.

[0013] Optionally, performing base station prediction based on the wireless modal features, the visual modal features, and the spatial modal features to obtain a base station prediction result includes:

[0014] Performing feature fusion processing on the wireless modality features, visual modality features, and spatial modality features to obtain fused features;

[0015] Base station prediction is performed based on the fused features to obtain base station prediction results.

[0016] Optionally, performing base station prediction based on the fused features to obtain a base station prediction result includes:

[0017] The fused features are input into a multi-task learning network to perform base station prediction and obtain a base station prediction result.

[0018] Optionally, the multi-task learning network includes at least two sub-task learning networks, the at least two sub-task learning networks include a sub-task learning network for base station prediction, the fused features include weighted features corresponding to each sub-task learning network in the at least two sub-task learning networks, and inputting the fused features into the multi-task learning network for base station prediction to obtain the base station prediction result includes:

[0019] Inputting the weighted features corresponding to each subtask learning network in the at least two subtask learning networks into the corresponding subtask learning network for prediction;

[0020] The base station prediction result is determined based on the output value of the subtask learning network for base station prediction.

[0021] Optionally, the at least two subtask learning networks further include at least one of a subtask learning network for region prediction and a subtask learning network for occlusion prediction;

[0022] The subtask learning network for region prediction is used to determine the optimal region among multiple regions divided according to the distance of the base station;

[0023] The subtask learning network for occlusion prediction is used to determine whether there is occlusion between the base station and the terminal.

[0024] Optionally, the loss function of the multi-task learning network is the sum of the product of the loss function of the sub-task learning network for base station prediction and a first weight parameter, the product of the loss function of the sub-task learning network for area prediction and a second weight parameter, and the product of the loss function of the sub-task learning network for occlusion prediction and a third weight parameter.

[0025] Optionally, the performing feature fusion processing on the wireless modality feature, the visual modality feature and the spatial modality feature to obtain the fused feature includes:

[0026] Inputting the wireless modality features, visual modality features and spatial modality features into an attention gate module to obtain a discrimination weight sequence;

[0027] Feature fusion processing is performed based on the discriminant weight sequence to obtain fused features, wherein the fused features include the product of the wireless modal feature and the corresponding discriminant weight in the discriminant weight sequence, the product of the visual modal feature and the corresponding discriminant weight in the discriminant weight sequence, and the product of the spatial modal feature and the corresponding discriminant weight in the discriminant weight sequence.

[0028] In a second aspect, an embodiment of the present invention provides a base station switching device, the device comprising:

[0029] An acquisition module, used to acquire wireless modal features, visual modal features and spatial modal features, wherein the wireless modal features are used to characterize channel information between a base station and a terminal, the visual modal features are used to characterize image information taken by the base station, and the spatial modal features are used to characterize location information of the terminal in the image taken by the base station;

[0030] A prediction module, used to perform base station prediction based on the wireless modal features, visual modal features and spatial modal features, and obtain a base station prediction result;

[0031] A switching module is used to perform base station switching based on the base station prediction result.

[0032] Optionally, the acquisition module is specifically used for:

[0033] Acquire multimodal information at multiple moments, the multimodal information including wireless modal information, visual modal information and spatial modal information, the wireless modal information is channel information between a base station and a terminal, the visual modal information is picture information taken by the base station, and the spatial modal information is location information of the terminal in the picture taken by the base station;

[0034] The multimodal information at the multiple moments is input into the LSTM network to obtain wireless modal features, visual modal features and spatial modal features.

[0035] Optionally, the prediction module includes:

[0036] A fusion unit, used for performing feature fusion processing on the wireless modality feature, the visual modality feature and the spatial modality feature to obtain fused features;

[0037] The prediction unit is used to perform base station prediction based on the fused features and obtain base station prediction results.

[0038] Optionally, the prediction unit is specifically used for:

[0039] The fused features are input into a multi-task learning network to perform base station prediction and obtain a base station prediction result.

[0040] Optionally, the multi-task learning network includes at least two sub-task learning networks, the at least two sub-task learning networks include a sub-task learning network for base station prediction, the fused features include weighted features corresponding to each sub-task learning network in the at least two sub-task learning networks, and the prediction unit is specifically used to:

[0041] Inputting the weighted features corresponding to each subtask learning network in the at least two subtask learning networks into the corresponding subtask learning network for prediction;

[0042] The base station prediction result is determined based on the output value of the subtask learning network for base station prediction.

[0043] Optionally, the at least two subtask learning networks further include at least one of a subtask learning network for region prediction and a subtask learning network for occlusion prediction;

[0044] The subtask learning network for region prediction is used to determine the optimal region among multiple regions divided according to the distance of the base station;

[0045] The subtask learning network for occlusion prediction is used to determine whether there is occlusion between the base station and the terminal.

[0046] Optionally, the loss function of the multi-task learning network is the sum of the product of the loss function of the sub-task learning network for base station prediction and a first weight parameter, the product of the loss function of the sub-task learning network for area prediction and a second weight parameter, and the product of the loss function of the sub-task learning network for occlusion prediction and a third weight parameter.

[0047] Optionally, the fusion unit is specifically used for:

[0048] Inputting the wireless modality features, visual modality features and spatial modality features into an attention gate module to obtain a discrimination weight sequence;

[0049] Feature fusion processing is performed based on the discriminant weight sequence to obtain fused features, wherein the fused features include the product of the wireless modal feature and the corresponding discriminant weight in the discriminant weight sequence, the product of the visual modal feature and the corresponding discriminant weight in the discriminant weight sequence, and the product of the spatial modal feature and the corresponding discriminant weight in the discriminant weight sequence.

[0050] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the base station switching method described in the first aspect.

[0051] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the base station switching method described in the first aspect are implemented.

[0052] In an embodiment of the present invention, wireless modal features, visual modal features, and spatial modal features are obtained, wherein the wireless modal features are used to characterize the channel information between the base station and the terminal, the visual modal features are used to characterize the image information taken by the base station, and the spatial modal features are used to characterize the location information of the terminal in the image taken by the base station; base station prediction is performed based on the wireless modal features, visual modal features, and spatial modal features to obtain base station prediction results; and base station switching is performed based on the base station prediction results. In this way, the optimal service base station is predicted by using wireless modal features, visual modal features, and spatial modal features, and base station switching is performed, which can improve the effect of base station switching. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0054] Figure 1 is a flow chart of a base station switching method provided by an embodiment of the present invention;

[0055] Figure 2 is a structural schematic diagram of a feature extraction network provided by an embodiment of the present invention;

[0056] Figure 3 is a structural diagram of a multi-task learning network provided by an embodiment of the present invention;

[0057] Figure 4 is a structural schematic diagram of a base station switching device provided by an embodiment of the present invention;

[0058] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In the embodiments of the present invention, a base station switching method, device and electronic device are proposed to solve the problem that the existing base station switching effect is poor.

[0061] See also Figure 1 , Figure 1 is a flow chart of a base station switching method provided by an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0062] Step 101: Acquire wireless modal features, visual modal features, and spatial modal features, wherein the wireless modal features are used to characterize channel information between a base station and a terminal, the visual modal features are used to characterize image information taken by the base station, and the spatial modal features are used to characterize location information of the terminal in the image taken by the base station.

[0063] The terminal may be a vehicle-mounted terminal, and the terminal may represent a user.

[0064] Among them, multimodal information at multiple moments can be obtained, and the multimodal information may include wireless modal information, visual modal information and spatial modal information, the wireless modal information is the channel information between the base station and the terminal, the visual modal information is the picture information taken by the base station, and the spatial modal information is the position information of the terminal in the picture taken by the base station; wireless modal features can be obtained based on the wireless modal information, visual modal features can be obtained based on the visual modal information, and spatial modal features can be obtained based on the spatial modal information.

[0065] In addition, the information required by the base station may include three modalities: wireless modality, visual modality, and spatial modality. Wireless modality information can be obtained by the base station sending a pilot signal and the terminal performing channel estimation. Visual modality information can be obtained by taking pictures with a camera installed at the base station. Spatial modality information can be calculated from multi-view visual images based on camera parameters.

[0066] It should be noted that the wireless modality information may be partial channel information between the base station and the terminal. The channel between the terminal and the base station b at time i is from Uniform sampling obtains some channels As wireless modality information. Visual modality information can be the RGB pictures taken by each base station. The picture taken by the camera of the bth base station at the i-th time is Downsample the image to reduce computational complexity, and get As visual modality information. Spatial modality information is the 3D coordinates of the terminal. Assume that the world coordinates of the terminal are (X w ,Y w ,Z w ), (u,v) is the pixel coordinate of the terminal in the camera image. The relationship between the world coordinates and the pixel coordinates is:

[0067]

[0068] Among them, f x and f y is the focal length of the camera on the x-axis and y-axis, u 0 and v 0 is the pixel coordinate of the center of the image, d x and d y is the length of the unit pixel in the x-axis and y-axis directions, P = [t x , t y , t z ] P is the relative displacement between the camera and the world coordinates, and R is the rotation matrix of the camera. The pixel coordinates of the terminal can be obtained from the image taken by the base station, and the world coordinates of the terminal can be calculated based on the pixel coordinates of multiple perspectives. Let the world coordinates of the terminal at time i be Loci , then Loc i is the spatial modal information.

[0069] Step 102: Perform base station prediction based on the wireless modal features, visual modal features, and spatial modal features to obtain base station prediction results.

[0070] Among them, the wireless modal features, visual modal features and spatial modal features can be subjected to feature fusion processing to obtain fused features; base station prediction can be performed based on the fused features to obtain base station prediction results.

[0071] Step 103: Perform base station switching based on the base station prediction result.

[0072] The base station prediction result may indicate a base station to be switched, and the base station switching based on the base station prediction result may be switching the base station to which the terminal is connected to the base station to be switched indicated by the base station prediction result.

[0073] In the related art, in order to solve the communication interruption caused by occlusion, a multi-point cooperative transmission method is adopted to reduce interference or convert interference into useful signals through cooperation between multiple base stations, so as to reduce the interruption probability and increase the transmission rate. However, the premise of multi-point cooperative transmission is to obtain the channel state information (CSI) of the user and each access point. In a large-scale MIMO system, the number of array antennas is huge, so the dimension of the channel matrix is ​​large, and estimating the CSI between the user and each access point will consume a lot of pilot resources and time. Another technical solution to avoid occlusion is base station switching, which determines whether occlusion occurs by monitoring communication indicators such as received signal strength (RSS). However, when the communication indicators decay, occlusion has already occurred, and base station switching is not predictable. And the target base station switched to is often the most adjacent base station, but the target base station is not necessarily the optimal base station, so it does not have optimality. The development of artificial intelligence technology has brought new opportunities to this problem. The embodiment of the present application is based on a base station pre-switching method based on multi-perspective visual information, which mines information from the unexpected visual dimension of spectrum resources, and uses deep learning technology to predict the optimal base station of the target user at a future moment.

[0074] It should be noted that the traditional multi-point coordination method requires the acquisition of the complete channel matrix between all users and each access point. In large-scale MIMO systems, the pilot overhead of channel estimation increases with the number of antennas. In addition, the coordination beamforming of multiple base stations and the coordination scheduling of multiple users based on channel state information are complex, and the time-frequency synchronization standards of multiple base stations are strict. The traditional base station switching method judges obstruction and interruption by the deterioration of communication indicators, which is not predictive. In addition, the target base station for user switching has a certain degree of randomness, and it is impossible to judge the optimal service base station in the future in advance.

[0075] The embodiment of the present application proposes to use multi-perspective visual information to predict the optimal service base station for vehicles on the road at future times in outdoor communication scenarios. By developing spectral resources, pilot overhead is saved. Long Short-Term Memory (LSTM) is used to extract scene features at historical times and predict the optimal base station at future times, thereby reducing the interruption probability of vehicle communications and achieving low latency and high transmission rates. While realizing base station prediction, the embodiment of the present application proposes a multi-task learning mechanism suitable for wireless communications, which improves the robustness and scenario generalization of vehicle base station prediction.

[0076] In an embodiment of the present invention, wireless modal features, visual modal features, and spatial modal features are obtained, wherein the wireless modal features are used to characterize the channel information between the base station and the terminal, the visual modal features are used to characterize the image information taken by the base station, and the spatial modal features are used to characterize the location information of the terminal in the image taken by the base station; base station prediction is performed based on the wireless modal features, visual modal features, and spatial modal features to obtain base station prediction results; and base station switching is performed based on the base station prediction results. In this way, the optimal service base station is predicted by using wireless modal features, visual modal features, and spatial modal features, and base station switching is performed, which can improve the effect of base station switching.

[0077] Optionally, the acquiring of wireless modality features, visual modality features, and spatial modality features includes:

[0078] Acquire multimodal information at multiple moments, the multimodal information including wireless modal information, visual modal information and spatial modal information, the wireless modal information is channel information between a base station and a terminal, the visual modal information is picture information taken by the base station, and the spatial modal information is location information of the terminal in the picture taken by the base station;

[0079] The multimodal information at the multiple moments is input into the LSTM network to obtain wireless modal features, visual modal features and spatial modal features.

[0080] The multiple moments may be a plurality of consecutive moments. For example, the structure of the network used for feature extraction may be as follows: Figure 2 As shown, CONV is a convolutional layer, FCN is a fully connected layer, the first module 11 may include a BatchNorm layer and a re-centering and rescaling function, the second module 12 may include a ReLu nonlinear activation function, and the third module 13 may include a maximum pooling and a sample-based discretization function (MaxPool, sample bas-sed discretization).

[0081] Taking the multiple moments as T moments as an example, multimodal information of consecutive T moments can be obtained:

[0082] CS i =[H i，sub , H i+1，sub , …, H i+T-1，sub ];

[0083]

[0084] LS i =[Loc i , Loc i+1 ,…,Loc i+T-1 ].

[0085] It should be noted that for visual modal information, a convolutional neural network can be used to encode each image to reduce the complexity of the network. An image coding network can be used to encode visual modal information. The image coding network can be a three-layer convolutional neural network. The convolution kernel of the first layer of the convolutional neural network can be set to 13, so that the network has a larger perception field of view. The convolution kernels of the second and third layers can be set to 9 and 5 respectively. Each convolution layer is followed by a normalization layer, an activation function layer, and a pooling layer. Finally, a fully connected layer can be used to encode the image into a one-dimensional vector. The encoded visual modal information is:

[0086]

[0087] Among them, CNN b (x) = MaxPool (ReLu (BatchNorm (Conv (x)))). Encode the visual modality information of T consecutive moments to obtain:

[0088]

[0089] In addition, the wireless modal information CS at T moments can be i , visual modality information Spatial modal information LS i Input the LSTM network to extract the features of each mode. For the historical information at time T, T LSTM units are needed to extract the historical features. The parameters of the i-th LSTM unit are x i 、v i and S i . Where x i is the input of the i-th LSTM unit, v i is the output of the i-th LSTM unit, S i is the state parameter of the i-th LSTM unit. The i-th LSTM unit partially discards the state of the previous unit through the forget gate, and the output of the forget gate is fi:

[0090] f i =σ(W f ·[v i-1 , x i ]+b f );

[0091] Where σ(·) is the sigmoid function The state retained by the i-1th LSTM unit is f i ⊙S i-1 , the input state of the i-th LSTM unit is: Then, the input gate calculates the parameter s i To determine how much information in the input state will be retained in the current cell's state:

[0092] s i =σ(W s ·[v i-1 , x i ]+b s )

[0093] Therefore, the updated status is: S i =f i ⊙S i-1 +s i ⊙S i Finally, the sigmoid layer is used to determine the output of LSTM:

[0094] v i =σ(W o ·[v i-1 , x i ]+b o )·tanh(S i);

[0095] Combine the output states of T LSTM units to obtain the features of the corresponding modality:

[0096]

[0097] The wireless mode information CS i , visual modality information Spatial modal information LS i Input LSTM network, the features extracted by the LSTM network are wireless modal features CF i , visual modality features Spatial modal characteristics LF i .

[0098] Optionally, performing base station prediction based on the wireless modal features, the visual modal features, and the spatial modal features to obtain a base station prediction result includes:

[0099] Performing feature fusion processing on the wireless modality features, visual modality features, and spatial modality features to obtain fused features;

[0100] Base station prediction is performed based on the fused features to obtain base station prediction results.

[0101] In one implementation, the performing feature fusion processing on the wireless modality feature, the visual modality feature, and the spatial modality feature to obtain the fused feature includes:

[0102] Inputting the wireless modality features, visual modality features and spatial modality features into an attention gate module to obtain a discrimination weight sequence;

[0103] Feature fusion processing is performed based on the discriminant weight sequence to obtain fused features, wherein the fused features include the product of the wireless modal feature and the corresponding discriminant weight in the discriminant weight sequence, the product of the visual modal feature and the corresponding discriminant weight in the discriminant weight sequence, and the product of the spatial modal feature and the corresponding discriminant weight in the discriminant weight sequence.

[0104] The wireless modality features, visual modality features and spatial modality features can be spliced ​​together and input into the attention gate module to obtain a discrimination weight sequence. For example, the wireless modality features, visual modality features and spatial modality features can be spliced ​​together as the input of the attention gate:

[0105]

[0106] In addition, the attention gate generates a discriminant weight sequence α=W according to the combined features gate ·F i，comb,in Where n img 、n cha 、n loc is the dimension of visual modality feature, wireless modality feature and spatial modality feature, N BS is the number of base stations. The elements in the discriminant weight sequence α (α 1 , α 2 , …, α NBS+2 ) is the weight parameter of visual modality feature, wireless modality feature and spatial modality feature, and the fused feature is

[0107] Optionally, performing base station prediction based on the fused features to obtain a base station prediction result includes:

[0108] The fused features are input into a multi-task learning network to perform base station prediction and obtain a base station prediction result.

[0109] Optionally, the multi-task learning network includes at least two sub-task learning networks, the at least two sub-task learning networks include a sub-task learning network for base station prediction, the fused features include weighted features corresponding to each sub-task learning network in the at least two sub-task learning networks, and inputting the fused features into the multi-task learning network for base station prediction to obtain the base station prediction result includes:

[0110] Inputting the weighted features corresponding to each subtask learning network in the at least two subtask learning networks into the corresponding subtask learning network for prediction;

[0111] The base station prediction result is determined based on the output value of the subtask learning network for base station prediction.

[0112] The output value of the subtask learning network for base station prediction may indicate the base station to be switched.

[0113] Optionally, the at least two subtask learning networks further include at least one of a subtask learning network for region prediction and a subtask learning network for occlusion prediction;

[0114] The subtask learning network for region prediction is used to determine the optimal region among multiple regions divided according to the distance of the base station;

[0115] The subtask learning network for occlusion prediction is used to determine whether there is occlusion between the base station and the terminal.

[0116] Among them, the structure of the multi-task learning network can be as follows Figure 3 For each subtask k, the discriminant weight sequence α can be calculated kDifferent weighted features can be obtained for different subtasks Each subtask learning network can contain L k Fully connected layer, the output is:

[0117]

[0118] Among them, Θ k = {W k , b k} is the parameter of the network learned for the kth subtask, Y k =[y k，1 ,y k，2 , …, y k，nk ] T , n k is the number of neurons in the output layer of the kth subtask, is the nonlinear activation function of the lth layer.

[0119] For the subtask for base station prediction, the probability that each base station will be the optimal base station in the future is calculated through the softmax layer:

[0120]

[0121] In addition, the loss function of the subtask learning network for base station prediction is:

[0122]

[0123] Among them, q 1 =[q 1，1 ,q 1，2 ,…,q 1,b ,…] T is a one-hot encoding label, the corresponding element of the optimal base station index is 1, and the rest of the elements are 0.

[0124] For the subtask used for regional prediction, we first divide it into N BS These regions are numbered in sequence and labeled q 2 =[q 2，1 ,q 2，2 ,…,q 2 , N BS ] T is a one-hot encoding vector, the element corresponding to the future region number is 1, and the rest are 0. The network output is:

[0125]

[0126] Among them, the subtask loss function of the subtask learning network for region prediction is:

[0127]

[0128] For the subtask learning network for occlusion prediction, the label is q 3 =[q 3，1 ,q 3，2 , …, q 3 ,N BS ] T , if there is an obstruction between the base station b and the terminal, then q 3,b =0, otherwise q 3，b =1.

[0129] Among them, the network output for the subtask of occlusion prediction is: p 3，i is the probability that there is no occlusion between the i-th base station and the terminal. The subtask loss function is the binary cross entropy (BCE) function:

[0130]

[0131] The subtasks for base station prediction, area prediction, and occlusion prediction are learned simultaneously. The loss function of the multi-task learning network is the superposition of each subtask:

[0132] Loss = σ 1 ·Loss 1 +σ 2 ·Loss 2 +σ 3 ·Loss 3 ;

[0133] where σ 1 , σ 2 , σ 3 is the adjustment parameter of each task weight.

[0134] Optionally, the loss function of the multi-task learning network is the sum of the product of the loss function of the sub-task learning network for base station prediction and a first weight parameter, the product of the loss function of the sub-task learning network for area prediction and a second weight parameter, and the product of the loss function of the sub-task learning network for occlusion prediction and a third weight parameter.

[0135] Among them, the first weight parameter, the second weight parameter and the third weight parameter may be preset values.

[0136] As a specific embodiment, the base station switching method includes the following process:

[0137] (1): The base station obtains multimodal information.

[0138] The information required by the base station may include three modalities: wireless modality, visual modality, and spatial modality. Wireless modality information can be obtained by the base station sending pilot signals and the terminal performing channel estimation. Visual modality information can be obtained by taking pictures with a camera installed at the base station. Spatial modality information can be calculated from multi-view visual images based on camera parameters.

[0139] It should be noted that the wireless modality information may be partial channel information between the base station and the terminal. The channel between the terminal and the base station b at time i is from Uniform sampling obtains some channels As wireless modality information. Visual modality information can be the RGB pictures taken by each base station. The picture taken by the camera of the bth base station at the i-th time is Downsample the image to reduce computational complexity, and get As visual modality information. Spatial modality information is the 3D coordinates of the terminal. Assume that the world coordinates of the terminal are (X w , Y w , Z w ), (u, v) are the pixel coordinates of the terminal in the camera image. The relationship between the world coordinates and the pixel coordinates is:

[0140]

[0141] Among them, f x and f y is the focal length of the camera on the x-axis and y-axis, u 0 and v 0 is the pixel coordinate of the center of the image, d x and d y is the length of the unit pixel in the x-axis and y-axis directions, P = [t x , t y , t z ] P is the relative displacement between the camera and the world coordinates, and R is the rotation matrix of the camera. The pixel coordinates of the terminal can be obtained from the image taken by the base station, and the world coordinates of the terminal can be calculated based on the pixel coordinates of multiple perspectives. Let the world coordinates of the terminal at time i be Loc i , then Loc i is the spatial modal information.

[0142] (2) Extract features from the multimodal information in step (1) through a feature extraction network.

[0143] The network structure of the feature extraction network is as follows Figure 2 By continuously tracking the user, we can obtain multimodal information for T consecutive moments:

[0144] CSi =[H i，sub , H i+1，sub , …, H i+T-1，sub ];

[0145]

[0146] LS i =[Loc i , Loc i+1 ,…,Loc i+T-1 ].

[0147] It should be noted that for visual modal information, a convolutional neural network can be used to encode each image to reduce the complexity of the network. An image coding network can be used to encode visual modal information. The image coding network can be a three-layer convolutional neural network. The convolution kernel of the first layer of the convolutional neural network can be set to 13, so that the network has a larger perception field of view. The convolution kernels of the second and third layers can be set to 9 and 5 respectively. Each convolution layer is followed by a normalization layer, an activation function layer, and a pooling layer. Finally, a fully connected layer can be used to encode the image into a one-dimensional vector. The encoded visual modal information is:

[0148]

[0149] Among them, CNN b (x) = MaxPool (ReLu (BatchNorm (Conv (x)))). Encode the visual modality information of T consecutive moments to obtain:

[0150]

[0151] In addition, the wireless modal information CS at T moments can be i , visual modality information Spatial modal information LS i Input the LSTM network to extract the features of each mode. For the historical information at time T, T LSTM units are needed to extract the historical features. The parameters of the i-th LSTM unit are x i 、v i and S i . Where x i is the input of the i-th LSTM unit, v i is the output of the i-th LSTM unit, S i is the state parameter of the i-th LSTM unit. The i-th LSTM unit partially discards the state of the previous unit through the forget gate, and the output of the forget gate is f i :

[0152] f i =σ(Wf ·[v i-1 , x i ]+b f );

[0153] Where σ(·) is the sigmoid function The state retained by the i-1th LSTM unit is f i ⊙S i-1 , the input state of the i-th LSTM unit is: Then, the input gate calculates the parameter s i To determine how much information in the input state will be retained in the current cell's state:

[0154] s i =σ(W s ·[v i-1 , x i ]+b s )

[0155] Therefore, the updated status is: Finally, the sigmoid layer is used to determine the output of LSTM:

[0156] v i =σ(W o ·[v i-1 , x i ]+b o )·tanh(S i );

[0157] Combine the output states of T LSTM units to obtain the features of the corresponding modality:

[0158]

[0159] The wireless mode information CS i , visual modality information Spatial modal information LS i Input LSTM network, the features extracted by the LSTM network are wireless modal features CF i , visual modality features Spatial modal characteristics LF i .

[0160] (3): The features of each modality are weighted and fused according to the importance of the features of different modalities to the task.

[0161] Since the features of each modality have different effects on the prediction task, judging their influence and weighting the features will help to perform subsequent tasks more accurately. First, concatenate the parameters of each modality as the input of the attention gate:

[0162]

[0163] The attention gate generates the discriminant weight α=W based on the combined features. gate ·F i,comb ,in where n img 、n cha 、n loc is the dimension of visual, wireless and spatial modality features, N BS is the number of base stations. The elements in α (α 1 , α 2 , …, α NBS+2 ) is the weight parameter of each mode, and the weighted feature is

[0164] (4): Using a multi-task learning mechanism for base station prediction can improve the prediction accuracy and the generalization of the network.

[0165] It should be noted that in order to learn a certain knowledge, it is necessary to learn a series of knowledge points related to it. This means that learning subtasks is of great help to learning the main task, and the learning of related subtasks is also of great significance for the generalization and promotion of knowledge. Subtasks closely related to predicting the optimal base station can be found to predict the area of ​​the terminal at future times and to predict the occlusion status between the terminal and each base station at future times. This is because, in a free space with no occlusion at all, the optimal base station for the terminal is the base station closest to the terminal. In addition, once there is occlusion between the terminal and base station b, base station b must not be the optimal base station. The multi-task learning network structure is as follows: Figure 3 shown.

[0166] For each subtask k, the weight parameter α is calculated according to the feature fusion scheme in step (3). k Different weighted features are obtained for different subtasks Each subtask learning network contains L k Fully connected layer, the output is:

[0167]

[0168] Among them, Θ k = {W k , b k} is the parameter of the network learned for the kth subtask, Y k =[y k，1 ,y k，2 , …, y k，nk ] T , n k is the number of neurons in the output layer of the kth subtask, is the nonlinear activation function of the first layer.

[0169] For the subtask for base station prediction, the probability that each base station will be the optimal base station in the future is calculated through the softmax layer:

[0170]

[0171] In addition, the loss function for the subtask of base station prediction is:

[0172]

[0173] Among them, q 1 =[q 1，1 ,q 1，2 , …, q 1,b ,…] T is a one-hot encoding label, the corresponding element of the optimal base station index is 1, and the rest of the elements are 0.

[0174] For the subtask used for regional prediction, we first divide it into N BS These areas are numbered in sequence, and the labels is a one-hot encoding vector, the element corresponding to the future region number is 1, and the rest are 0. The network output is:

[0175]

[0176] Among them, the subtask loss function for the subtask of region prediction is:

[0177]

[0178] For the subtask for occlusion prediction, the labels are If there is an obstruction between the base station b and the terminal, then q 3,b =0, otherwise q 3，b =1.

[0179] Among them, the network output for the subtask of occlusion prediction is: p 3，i is the probability that there is no occlusion between the i-th base station and the terminal. The subtask loss function is the binary cross entropy (BCE) function:

[0180]

[0181] The subtasks for base station prediction, area prediction, and occlusion prediction are learned simultaneously. The loss function of the multi-task learning network is the superposition of each subtask:

[0182] Loss = σ 1 ·Loss 1 +σ2 ·Loss 2 +σ 3 ·Loss 3 ;

[0183] where σ 1 , σ 2 , σ 3 is the adjustment parameter of each task weight.

[0184] The embodiment of the present invention proposes a base station switching method based on multi-view visual information, which uses information from multiple modes such as vision, space, and wireless to predict the optimal base station for users to switch to in the future based on a neural network. In addition, the embodiment of the present invention proposes a multi-task learning mechanism based on communication prior knowledge, which improves the robustness and generalization of base station prediction by learning subtasks related to the main task.

[0185] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a base station switching device provided by an embodiment of the present invention, such as Figure 4 As shown, the base station switching device 200 includes:

[0186] The acquisition module 201 is used to acquire wireless modal features, visual modal features and spatial modal features, wherein the wireless modal features are used to characterize the channel information between the base station and the terminal, the visual modal features are used to characterize the picture information taken by the base station, and the spatial modal features are used to characterize the position information of the terminal in the picture taken by the base station;

[0187] A prediction module 202, configured to perform base station prediction based on the wireless modality features, the visual modality features and the spatial modality features, and obtain a base station prediction result;

[0188] The switching module 203 is configured to perform base station switching based on the base station prediction result.

[0189] Optionally, the acquisition module is specifically used for:

[0190] Acquire multimodal information at multiple moments, the multimodal information including wireless modal information, visual modal information and spatial modal information, the wireless modal information is channel information between a base station and a terminal, the visual modal information is picture information taken by the base station, and the spatial modal information is location information of the terminal in the picture taken by the base station;

[0191] The multimodal information at the multiple moments is input into the LSTM network to obtain wireless modal features, visual modal features and spatial modal features.

[0192] Optionally, the prediction module includes:

[0193] A fusion unit, used for performing feature fusion processing on the wireless modality feature, the visual modality feature and the spatial modality feature to obtain fused features;

[0194] The prediction unit is used to perform base station prediction based on the fused features and obtain base station prediction results.

[0195] Optionally, the prediction unit is specifically used for:

[0196] The fused features are input into a multi-task learning network to perform base station prediction and obtain a base station prediction result.

[0197] Optionally, the multi-task learning network includes at least two sub-task learning networks, the at least two sub-task learning networks include a sub-task learning network for base station prediction, the fused features include weighted features corresponding to each sub-task learning network in the at least two sub-task learning networks, and the prediction unit is specifically used to:

[0198] Inputting the weighted features corresponding to each subtask learning network in the at least two subtask learning networks into the corresponding subtask learning network for prediction;

[0199] The base station prediction result is determined based on the output value of the subtask learning network for base station prediction.

[0200] Optionally, the at least two subtask learning networks further include at least one of a subtask learning network for region prediction and a subtask learning network for occlusion prediction;

[0201] The subtask learning network for region prediction is used to determine the optimal region among multiple regions divided according to the distance of the base station;

[0202] The subtask learning network for occlusion prediction is used to determine whether there is occlusion between the base station and the terminal.

[0203] Optionally, the loss function of the multi-task learning network is the sum of the product of the loss function of the sub-task learning network for base station prediction and a first weight parameter, the product of the loss function of the sub-task learning network for area prediction and a second weight parameter, and the product of the loss function of the sub-task learning network for occlusion prediction and a third weight parameter.

[0204] Optionally, the fusion unit is specifically used for:

[0205] Inputting the wireless modality features, visual modality features and spatial modality features into an attention gate module to obtain a discrimination weight sequence;

[0206] Feature fusion processing is performed based on the discriminant weight sequence to obtain fused features, wherein the fused features include the product of the wireless modal feature and the corresponding discriminant weight in the discriminant weight sequence, the product of the visual modal feature and the corresponding discriminant weight in the discriminant weight sequence, and the product of the spatial modal feature and the corresponding discriminant weight in the discriminant weight sequence.

[0207] The base station switching device can realize Figure 1 The various processes implemented in the method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described again here.

[0208] like Figure 5 As shown, an embodiment of the present invention further provides an electronic device 300, comprising: a processor 301, a memory 302, and a program stored in the memory 302 and executable on the processor 301. When the program is executed by the processor 301, the various processes of the above-mentioned base station switching method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0209] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned base station switching method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a ROM, RAM, a magnetic disk or an optical disk.

[0210] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0211] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0212] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A base station switching method, It is characterized in that The method comprises: Acquire wireless modality features, visual modality features, and spatial modality features, wherein the wireless modality features are used to characterize channel information between a base station and a terminal, the visual modality features are used to characterize picture information taken by the base station, and the spatial modality features are used to characterize location information of the terminal in the picture taken by the base station; Performing base station prediction based on the wireless modal features, visual modal features, and spatial modal features to obtain a base station prediction result; Base station switching is performed based on the base station prediction result.

2. The method according to claim 1, It is characterized in that The acquiring of wireless modality features, visual modality features and spatial modality features includes: Acquire multimodal information at multiple moments, the multimodal information including wireless modal information, visual modal information and spatial modal information, the wireless modal information is channel information between a base station and a terminal, the visual modal information is picture information taken by the base station, and the spatial modal information is location information of the terminal in the picture taken by the base station; The multimodal information at the multiple moments is input into the LSTM network to obtain wireless modal features, visual modal features and spatial modal features.

3. The method according to claim 1, It is characterized in that The performing base station prediction based on the wireless modal features, the visual modal features and the spatial modal features to obtain the base station prediction result includes: Performing feature fusion processing on the wireless modality features, visual modality features, and spatial modality features to obtain fused features; Base station prediction is performed based on the fused features to obtain base station prediction results.

4. The method according to claim 3, It is characterized in that The performing base station prediction based on the fused features to obtain a base station prediction result includes: The fused features are input into a multi-task learning network to perform base station prediction and obtain a base station prediction result.

5. The method according to claim 4, It is characterized in that The multi-task learning network includes at least two sub-task learning networks, the at least two sub-task learning networks include a sub-task learning network for base station prediction, the fused features include weighted features corresponding to each sub-task learning network in the at least two sub-task learning networks, and the inputting the fused features into the multi-task learning network for base station prediction to obtain a base station prediction result includes: Inputting the weighted features corresponding to each subtask learning network in the at least two subtask learning networks into the corresponding subtask learning network for prediction; The base station prediction result is determined based on the output value of the subtask learning network for base station prediction.

6. The method according to claim 5, It is characterized in that The at least two subtask learning networks further include at least one of a subtask learning network for region prediction and a subtask learning network for occlusion prediction; The subtask learning network for region prediction is used to determine the optimal region among multiple regions divided according to the distance of the base station; The subtask learning network for occlusion prediction is used to determine whether there is occlusion between the base station and the terminal.

7. The method according to claim 6, It is characterized in that The loss function of the multi-task learning network is the sum of the product of the loss function of the sub-task learning network for base station prediction and the first weight parameter, the product of the loss function of the sub-task learning network for area prediction and the second weight parameter, and the product of the loss function of the sub-task learning network for occlusion prediction and the third weight parameter.

8. The method according to claim 3, It is characterized in that The performing feature fusion processing on the wireless modality feature, the visual modality feature and the spatial modality feature to obtain the fused feature includes: Inputting the wireless modality features, visual modality features and spatial modality features into an attention gate module to obtain a discrimination weight sequence; Feature fusion processing is performed based on the discriminant weight sequence to obtain fused features, wherein the fused features include the product of the wireless modal feature and the corresponding discriminant weight in the discriminant weight sequence, the product of the visual modal feature and the corresponding discriminant weight in the discriminant weight sequence, and the product of the spatial modal feature and the corresponding discriminant weight in the discriminant weight sequence.

9. A base station switching device, It is characterized in that The device comprises: An acquisition module, used to acquire wireless modal features, visual modal features and spatial modal features, wherein the wireless modal features are used to characterize channel information between a base station and a terminal, the visual modal features are used to characterize image information taken by the base station, and the spatial modal features are used to characterize location information of the terminal in the image taken by the base station; A prediction module, used to perform base station prediction based on the wireless modal features, visual modal features and spatial modal features, and obtain a base station prediction result; A switching module is used to perform base station switching based on the base station prediction result.

10. An electronic device, It is characterized in that include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the base station switching method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the base station switching method according to any one of claims 1 to 8 are implemented.

Images