A deep learning-based communication and environment perception integration method and device, and a storage medium

Abstract

The application discloses a communication and environment perception integrated method based on deep learning, which comprises the following steps: obtaining signal receiving strength on different beam pairs and converting into RSRP images; detecting characteristic patterns generated by multipath and non-direct path in the RSRP images and estimating the arrival angle and departure angle of the detected multipath; tracking and correlating multipath angle parameters in the user movement process to obtain the change process of AOA and AOD of the same multipath at different positions in the user movement process; using the change process of the multipath AOA and AOD obtained by tracking, identifying the LOS and NLOS paths of each multipath; fitting the AOA and AOD motion trajectory points of each multipath in the user movement process, using the parameters obtained by fitting to represent different scatterers in the environment generating the multipath, extracting the environment and user position information contained in the AOA-AOD space, combining the sensor information of the user end, and completing the estimation of the position of the scatterer in the environment and the user position.

Description

TECHNICAL FIELD

[0001] The application provides a deep learning-based communication and environment perception integration method, device and storage medium, and belongs to the cross field of millimeter wave communication, deep learning and communication perception integration. BACKGROUND

[0002] Millimeter wave communication technology is one of the key technologies of the future sixth generation (6G) wireless communication. Compared with sub-6GHz communication systems, millimeter wave communication systems have larger bandwidth and higher data rate, and array antennas and beamforming technology are used to combat the high path loss of millimeter wave frequency bands. These characteristics bring higher time delay and angle resolution to millimeter wave communication systems, making it possible for them to have both communication and perception capabilities. In addition, wireless perception systems and millimeter wave communication systems are more similar in frequency bands, channel characteristics and hardware architecture, making it possible to realize communication and perception integration.

[0003] There are many problems and challenges in realizing communication and perception integration. First, the millimeter wave channel is highly related to the environment geometry, and the environment perception depends on the accurate estimation of the channel multipath parameters. Second, in the mobile scenario, the channel multipath parameters change over time, and tracking and correlating the angle parameters of the multipath are major challenges for beam management and environment perception. Finally, the LOS and NLOS paths in the millimeter wave channel contain the geometric relationship between the user, the base station and the environment, respectively, and identifying the LOS and NLOS paths can use the millimeter wave channel multipath parameters for environment perception more finely. SUMMARY

[0004] The purpose of the application is to provide a method for estimating, tracking, LOS / NLOS path identification and environment perception of channel multipath angle parameters with high reliability and high real-time based on the information obtained in the beam management process.

[0005] Technical scheme: In order to achieve the above purpose, the application provides a deep learning-based communication and environment perception integration method, which specifically comprises the following steps:

[0006] Step 1: In the beam scanning process, the base station and the user obtain the signal reception strength on different beam pairs and convert it into a signal reception strength RSRP image;

[0007] Step 2: Use a deep learning-based target detection method to detect the feature patterns produced by the multipath in the RSRP image and estimate the angle of arrival AOA and the angle of departure AOD of the detected multipath; the multipath includes the direct path LOS and the non-direct path NLOS;

[0008] Step 3: Using a multi-target tracking algorithm, based on the estimated multipath values ​​at different locations during the user's movement, the multipath angle parameters are tracked and correlated to obtain the changes in AOA and AOD of the same multipath at different locations during the user's movement.

[0009] Step 4: Using the tracked multipath AOA and AOD trajectories and their changing processes, LOS and NLOS identification is performed for each multipath according to the different changing patterns of AOA and AOD of the LOS and NLOS paths.

[0010] Step 5: Fit the AOA and AOD trajectory points of each multipath during the user's motion process. Use the fitted parameters to represent the different scatterers that generate multipath in the environment. Extract the environment and user location information contained in the RSRP image. Combined with the sensor information on the user end, complete the estimation of the azimuth angle between the virtual base station and the base station in the environment and the user's position.

[0011] Furthermore, in step 1, the beam scanning is any one of an exhaustive beam scanning scheme and a layered codebook beam scanning scheme; and the method of converting into an RSRP image is: using RSRP values ​​of different beam pairs to generate a color heat map or a grayscale image.

[0012] Furthermore, in step 2, the algorithm for estimating the arrival angle AOA and the departure angle AOD of the detected multipath is any one of the YOLO algorithm, MUISC, ESPRIT, and OMP algorithms based on deep learning.

[0013] Furthermore, in step 3, during the user's movement, the specific method of the multipath angle parameter tracking association algorithm based on the multi-target tracking algorithm ByteTrack is as follows:

[0014] (3.1) Using the tracking algorithm ByteTrack, based on the estimation of multipath AOA and AOD in RSRP images, the position changes of multipaths in continuous RSRP images during user movement are tracked as follows:

[0015] (1) The RSRP image I at a certain position in the user's motion is regarded as a frame. In the user's motion trajectory, the RSRP images at different positions arranged in chronological order constitute a continuous RSRP image sequence.

[0016] (2) For The kth frame I in k , the detection frame detected by YOLO is divided into a set of detection frames above the confidence threshold τ And the set of detection boxes below the threshold

[0017] (3) The set of trajectories tracked in the k-th frame is defined as

[0018] (4) In the first frame I1, the high-confidence bounding boxes are initialized as trajectories

[0019] (5) In the following frames, each multi-path trajectory is associated with the new location of the trajectory in the current frame I k+1 The new location of the trajectory in the current frame I is firstly predicted by the Kalman filter based on

[0020] (6) For the construction of the k+1-th frame, the first step of trajectory association is performed by calculating the similarity between the predicted bounding boxes of the trajectories in and the bounding boxes in using the intersection over union IoU, and the Hungarian algorithm is used to associate the trajectories in with the bounding boxes in based on the similarity;

[0021] (7) After the first association, the trajectories that are successfully associated are added to , and the predicted trajectories and the bounding boxes that are not associated are recorded as and

[0022] (8) The second step of association is performed between and , where the similarity between the predicted bounding boxes of the trajectories in and the bounding boxes in is calculated using the intersection over union IoU, and the Hungarian algorithm is used to associate the trajectories in and based on the similarity;

[0023] (9) After the second association, the trajectories that are successfully associated are in the set , the bounding boxes that are not successfully associated in will no longer be retained, and the trajectories that are not associated in will be stored in ;

[0024] (10) will be associated in the following frames, and the trajectories in that cannot be re-tracked within a certain number of frames will be deleted;

[0025] (11) The high-confidence bounding boxes that are not matched will be initialized as new trajectories and stored in ​​​​In;

[0026] (12) For each path in the k+1th frame, based on the result of the current frame tracking Assign an identification ID to each path;

[0027] (3.2) Let and respectively be the AOA and AOD of the qth path in the pth frame RSRP image in the user motion trajectory, The AOA and AOD coordinates of the qth path in all frames are put into the set ,

[0028] Further, in step 4, the identification method of the LOS path and the NLOS path is as follows:

[0029] (3) Let The latest and the second new AOA-AOD coordinates in and where i>j, the LOS and NLOS identification of the qth path is carried out by calculating the following formula:

[0030]

[0031] If Z>0, the qth path is identified as the LOS path, and if Z<0, it is the NLOS path.

[0032] Further, the method of step 5 is as follows:

[0033] (1) The NLOS path moves along a straight line with a slope of-1, and the scatterers that produce different NLOS paths are located at the same azimuth angle α of the base station. Different NLOS path trajectories are represented by straight lines with a slope of-1 and different intercepts, so as to distinguish the scatterers in the environment. Assuming that the qth path is a NLOS path, its AOA and AOD motion trajectory is represented as:

[0034] Φ=-Θ+Δ q

[0035] where Θ, Φ and Δ q are respectively the AOA, AOD and intercept of the qth NLOS path;

[0036] (2) The least squares method is applied to all AOA-AOD coordinates in q to estimate Δ NLOS ;

[0037] (3) Based on Θ NLOS =2α-Φ q +kπ, it is derived that Δ q =2αkπ, thereby estimating the azimuth angle a of the q-th NLOS path with respect to the virtual base station on the scatterer q ; F NLOS is the angle of departure AOD of the NLOS path; k is a constant;

[0038] (4) The positions of the base station and the environmental scatterers are fixed, and the trajectories of the LOS path and the NLOS path generated by the scatterer have characteristics in the RSRP image. A mapping between the positions of the scatterers and the user is constructed. According to the AOA and the AOD of each multipath, the position information of the user and the environment is reflected. In combination with the motion state information of the user output by the inertial measurement unit (IMU), the position of the user is estimated.

[0039] (5) In the RSRP image, the trajectory of the LOS path is located on Θ LOS = F LOS +kπ, and the trajectory of the NLOS path is located on Θ LOS ∈[0, 2π), F LOS ∈[0, 2π), and the trajectory of the LOS path in the RSRP image is divided into two segments of Θ LOS = F LOS -π, Θ LOS ∈[0, π), and Θ LOS = F LOS +π, Θ LOs ∈[π, 2π); F LOS is the angle of departure AOD of the LOS path.

[0040] (6) In the RSRP image, if the virtual base station generated by a certain scatterer is located at an azimuth angle a with respect to the base station, the trajectory of the NLOS path is located on Θ NLOS = 2a - F NLOS +π, , considering that the angle value range in the RSRP image is [0, 2π), when and , the part exceeding the angle value range [0, 2π) of the line segment is taken modulo;

[0041] (7) In combination with the trajectory expressions of the LOS and NLOS paths in steps (5) and (6), the RSRP image is regionally divided according to the azimuth of the virtual base station generated by the scatterer with respect to the base station. On this basis, a mapping codebook of different regions of the RSRP image and the positions of the scatterers in the environment is constructed.

[0042] (8) The azimuth angle of the virtual base station generated by the scatterer with respect to the base station is located according to the mapping codebook in step (7).

[0043] Furthermore, the application provides a device for integrated communication and environment perception based on deep learning, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the methods when executing the computer program.

[0044] Furthermore, the application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of any of the methods.

[0045] Beneficial effects, compared with the prior art, the technical scheme of the application has the following beneficial technical effects:

[0046] The technical scheme provided by the application can directly perform channel estimation and perception in the beam management process, does not depend on the LOS and NLOS identification of time delay parameters requiring high clock synchronization and large bandwidth, can further perform LOS and NLOS identification on each multipath compared with the existing LOS and NLOS scene identification, can utilize the mapping relationship between the motion characteristics of the multipath in the AOA-AOD space and the environment geometry to complete the perception of the scatterer in the environment, and the environment feature correlation calculation amount of the present scheme is small compared with the traditional SLAM algorithm (such as BP-SLAM, PHD-SLAM, etc.) which utilizes probability to perform data correlation calculation. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 FIG. 1 is a schematic diagram of the geometric relationship between the base station, the user and the reflector in the embodiment of the application;

[0048] Figure 2 FIG. 2 is a schematic diagram of the trajectory of the LOS path in the RSRP image in the embodiment of the application;

[0049] Figure 3 FIG. 3 is a schematic diagram of the trajectory of the NLOS path in the RSRP image in the embodiment of the application;

[0050] Figure 4 FIG. 4 is a schematic diagram of the region division of the RSRP image according to the relationship between the scatterer and the environment and the codebook in the embodiment of the application;

[0051] Figure 5 FIG. 5 is a flowchart of the method for integrated communication and environment perception based on deep learning provided by the embodiment of the application. DETAILED DESCRIPTION

[0052] In the following description, specific details are set forth in order to provide a thorough understanding of the application. However, persons skilled in the art will understand that the application can be practiced without these specific details.

[0053] In order to make the purposes, technical solutions and advantages of the present application clearer, specific embodiments will be described in detail below with reference to the drawings. The communication and environment perception integrated method based on deep learning provided by the embodiments of the present application, as shown in the accompanying drawings, comprises the following steps: Figure 5

[0054] Step 101: The base station and the user obtain the signal receiving strength RSRP on different beam pairs in the beam scanning process and convert it into an RSRP image.

[0055] It should be noted that:

[0056] (1) The base station and the user use their respective beam codebooks for beam scanning and measure the RSRP on different beam pairs during the scanning process. i,j The RSRP measured by the base station using the i-th beam and the user using the j-th beam is represented as Rij, and when all the base stations and users complete the RSRP measurement of all the beams, an RSRP matrix R can be obtained, which is represented as

[0057]

[0058] Wherein, M UE and M BS are the number of beams in the codebook used by the user and the base station for beam scanning, respectively, and the angles corresponding to different beam numbers in the codebook increase with the increase of the beam number. Different beam numbers correspond to different angle domain directions, so the RSRP matrix covers a discrete range of AOA and AOD.

[0059] According to the RSRP matrix measured by different base stations and users towards pairs at the same test point, an RSRP image with a larger coverage range of AOA and AOD is obtained.

[0060] 1) First, an initialized RSRP image I is created, and all elements of the image are set to zero, where the two dimensions of the image represent AOA and AOD, respectively.

[0061] 2) Secondly, in order to prevent beam pattern distortion caused by the boundary cycle of the RSRP image at 0° and 360°, the range of AOA and AOD of the RSRP image is extended according to the angle range of the beam scanning.

[0062] 3) Next, in order to align the AOA and AOD angle indexes of the RSRP matrix and the RSRP image I, the RSRP matrix measured by different base station-user towards pairs is interpolated. The row number and column number of the interpolated RSRP matrix remain unchanged in the AOA-AOD coverage range, and the row number and column number of the interpolated RSRP matrix are consistent with the row number and column number of the sub-matrix with the same AOA-AOD range in the RSRP image I. ​

[0063] 4) Finally, the interpolated RSRP matrix is mapped to the RSRP image I according to the respective AOA and AOD angular coverage, and the RSRP image I is normalized to a normalized RSRP matrix with value range 0-255 Specifically expressed as:

[0064]

[0065] where I max is the maximum value in I, is the down rounding operator. The normalized RSRP matrix is copied as the RGB three channels of the image as the generated RSRP image. The generated RSRP image has P rows and Q columns, and the minimum and maximum values of the AOA coverage are Θ min and Θ max , and the minimum and maximum values of the AOD coverage are Φ min and Φ max .

[0066] Step 102: using a deep learning-based target detection method, the feature pattern generated by the multipath in the RSRP image is detected, and the arrival angle AOA and the departure angle AOD of the detected multipath are estimated; the multipath includes the direct path LOS and the non-direct path NLOS.

[0067] It should be noted that:

[0068] (1) The beam pattern generated by the channel multipath is sparsely distributed in the RSRP image, which can be regarded as multiple targets in the image, and the channel multipath estimation is converted into a target detection problem.

[0069] (2) YOLO is a real-time target detection algorithm based on deep learning. When the RSRP image is input into the pre-trained YOLO network, the number of detected multipath beam patterns is Each detected target is represented by a detection box. The label of the detection box of the lth target is (class l , c x,l , c y,l , w l , h l ), where class l is the target type, the normalized horizontal and vertical coordinates of the target center point in the RSRP image coordinate system (the ratio of the center point pixel position to Q or P) is c x,l , c y,l , and the normalized width and height (the ratio of the two side lengths of the detection box to Q or P) is w l , h l .

[0070] (3) Estimate AOA and AOD based on bounding box label, find the maximum RSRP in the bounding box containing the lth beam pattern, i.e. the position of the maximum element (x l,max ,y l,max ), the AOA and AOD estimates of the lth beam are:

[0071]

[0072] (4) Put all the AOA and AOD estimates of the beams in the RSRP image into the set , the AOA-AOD coordinates of the lth beam are

[0073] Step 103: Use a multi-target tracking algorithm to track and associate the multi-path angle parameters in the user's movement based on the estimated values of the multi-paths at different positions in the user's movement, and obtain the change process of the AOA and AOD of the same multi-path at different positions in the user's movement.

[0074] It should be noted that:

[0075] (3.1) Use the tracking algorithm ByteTrack to track the position change process of the multi-paths in the continuous RSRP images in the user's movement based on the estimates of the AOA and AOD of the multi-paths in the RSRP images, and the method is as follows:

[0076] (1) Take the RSRP image I at a certain position in the user's movement as a frame, and the RSRP images at different positions in the user's movement trajectory are arranged in time sequence to form a continuous RSRP image sequence

[0077] (2) For the kth frame I k in , the bounding boxes detected by YOLO are divided into a bounding box set higher than the confidence threshold τ and a bounding box set lower than the confidence threshold τ

[0078] (3) The set consisting of the tracks tracked in the kth frame is defined as

[0079] (4) In the first frame I1, the high-confidence bounding boxes are initialized as the tracks

[0080] (5) In the subsequent frames, the new positions k+1 of each multi-path track in the current frame I are first predicted based on by the Kalman filter;

[0081] (6) For the composition of the k+1 frame , the first step of trajectory association is performed, and the similarity between the detection box of the trajectory prediction in and the lost tracking trajectory and the detection box in is calculated by the intersection over union IoU, and the Hungarian algorithm is used to associate and and the lost tracking trajectory based on the similarity;

[0082] (7) After the first association, the successfully associated trajectories are added to , and the prediction trajectories and detection boxes that are not associated are respectively recorded as and

[0083] (8) The second step of association is performed between and , wherein the similarity between the detection box of the trajectory prediction in and the detection box in is calculated by the intersection over union IoU, and the Hungarian algorithm is used to associate and based on the similarity;

[0084] (9) After the second association, the successfully associated trajectories are in the set , the detection boxes that are not associated in will not be retained, and the trajectories that are not associated in will be stored in ;

[0085] (10) will be associated in the subsequent frames, and the trajectories in that cannot be re-tracked within a certain number of frames thereafter will be deleted;

[0086] (11) The high-confidence detection boxes that are not matched will be initialized as new trajectories and stored in ;

[0087] (12) For each multi-path in the k+1 frame, based on the tracking result of the current frame assigns an identification ID to each path;

[0088] (3.2) Let and be the AOA and AOD of the path with ID q in the p-th frame RSRP image in the user motion trajectory, the AOA and AOD coordinates of the path with ID q in all frames are put into the set middle,

[0089] Step 104: Using the tracked multipath AOA and AOD trajectories and their changing processes, perform LOS / NLOS identification on each multipath according to the different changing patterns of AOA and AOD of the LOS path and the NLOS path.

[0090] It should be noted that:

[0091] (1) Note Θ NLOS and Φ NLOS are the AOA and AOD of the NLOS path, Θ LOS and Φ LOS The AOA and AOD of the LOS path are respectively, and the AOA and AOD of the NLOS path meet the following constraints

[0092] sin(2α-Θ NLOS -Φ NLOS )=0

[0093] Among them, such as Figure 1 As shown in Figure 2, α is the direction angle between the base station and the virtual base station generated by the NLOS path on the scatterer.

[0094] (2) The angular relationship between the LOS path and the NLOS path can be further expressed as

[0095] Θ NLOS =2α-Φ NLOS +kπΘ LOS =Φ LOS +kπ

[0096] Therefore, the beam patterns of the LOS path and the NLOS path move along the straight lines with slopes of 1 and -1 respectively in the RSRP image, that is, the AOA and AOD change rates of the LOS path and the NLOS path are

[0097]

[0098] (3) Record The latest and second latest AOA-AOD coordinates are and Where i>j. The LOS and NLOS identification of the qth path can be performed by calculating the following formula:

[0099]

[0100] If Z>0, the qth path is identified as a LOS path, and if Z<0, it is an NLOS path.

[0101] Step 105: fitting the AOA and AOD trajectory points of each multipath of the user motion process, using the fitted parameters to represent different scatterers in the environment that generate multipaths, extracting the environment and user position information contained in the RSRP image, combining the sensor information of the user terminal, and completing the estimation of the azimuth angle of the virtual base station and the base station in the environment and the user position.

[0102] It should be noted that:

[0103] (1) The NLOS path moves along a straight line with a slope of -1. Since the scatterers that generate different NLOS paths are located at different azimuth angles a of the base station, a straight line with a slope of -1 and different intercepts Δ q is used to represent the trajectories of different NLOS paths, that is, to distinguish the scatterers in the environment. It is assumed that the qth path is an NLOS path, and the AOA and AOD trajectories are represented as

[0104] Φ = - Θ + Δ q

[0105] (2) The least squares method can be applied to all AOA-AOD coordinates in to estimate Δ q .

[0106] (3) Based on Θ NLOS = 2a - Φ NLOs + kπ, it can be deduced that Δ q = 2a q + kπ, and thus the azimuth angle a q of the virtual base station at the base station on the qth NLOS path scatterer can be estimated.

[0107] (4) Since the position of the base station in the environment and the position of the scatterer in the environment are fixed, the trajectories of the LOS path and the NLOS path generated by the scatterer have characteristics in the RSRP image, and a mapping between the RSRP image and the environmental information (scatterer position and user position, etc.) can be constructed. At the same time, the positions of the LOS and NLOS paths in the RSRP image, that is, the AOA and AOD of each multipath, reflect the position information of the user and the environment, and combined with the user motion state information output by the inertial measurement unit (IMU, Inertial Measurement Unit), the user position can be estimated.

[0108] (5) As shown in Figure 2 , in the RSRP image, the motion trajectory of the LOS path is located at Θ LOS = Φ LOS + kπ, considering that Θ LOS ∈ [0, 2π), Φ LOS ∈ [0, 2π), the trajectory of the LOS path is divided into Θ LOS= Φ LOS - π, Θ LOS ∈ [0, π) and Θ LOS = Φ LOS + π, Θ LOS ∈ [π, 2π) ; Φ LOS is the angle of departure AOD of the LOS path;

[0109] (6) In the RSRP image, if the virtual base station generated by a scatterer is located at the direction angle of the base station is α, without considering the range of angle values, the motion trajectory of the NLOS path is located at Θ NLOS = 2α - Φ NLOS + π, . Considering that the range of angle values in the RSRP image is [0, 2π), when and , since part of the line segment exceeds the range of angle values [0, 2π) of the RSRP image, the part exceeding the range of [0, 2π) can be taken modulo, and thus the final NLOS path can present a multi-segment trajectory in the RSRP image. Figure 3 (a) and (b) respectively show the motion trajectories of the NLOS paths generated by the scatterers of and in the RSRP image.

[0110] (7) Combining the motion trajectory expressions of the LOS and NLOS paths in steps (5) and (6), the RSRP image is divided into regions according to the azimuth of the virtual base station generated by the scatterer relative to the base station, and on this basis, a mapping codebook of different regions of the RSRP image and the positions of the scatterers in the environment is constructed, such as Figure 4 (a) shows the multi-path AOA / AOD region that can be generated by the scatterer of 0≤α≤π, Figure 4 (b) shows the multi-path AOA / AOD region that can be generated by the scatterer of π≤α≤2π. On this basis, the divided regions shown in the figure can be further divided to construct a mapping codebook of different regions of the RSRP image and the positions of the scatterers in the environment.

[0111] (8) According to the mapping codebook in step (7), the azimuth angle of the virtual base station generated by the scatterer relative to the base station is located.

[0112] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A deep learning-based integrated communication and environment perception method, characterized in that, The method specifically comprises the following steps: Step 1, the base station BS and the user obtain the signal receiving strength on different beam pairs in the beam scanning process, and convert it into a signal receiving strength RSRP image; Step 2, using a deep learning-based target detection method, the characteristic patterns generated by multipath in the RSRP image are detected, and the angles of arrival AOA and angles of departure AOD of the detected multipath are estimated; the multipath includes the direct path LOS and the non-direct path NLOS; Step 3, using a multi-target tracking algorithm, based on the estimated values of the multipath at different positions during the user motion, the angle parameters of the multipath during the user motion are tracked and associated, and the change process of the AOA and AOD of the same multipath at different positions during the user motion is obtained; Step 4, using the tracking obtained multipath AOA and AOD trajectories and change processes, according to the different change rules of the AOA and AOD of the LOS path and the NLOS path, the LOS and NLOS of each multipath are identified; Step 5, fitting the AOA and AOD motion trajectory points of each multipath during the user motion, using the parameters obtained by fitting to represent different scatterers in the environment that generate multipath, extracting the environment and user position information contained in the RSRP image, combining the sensor information of the user terminal, and completing the estimation of the azimuth of the virtual base station vBS and the base station and the position of the user in the environment.

2. The method of claim 1, wherein, In step 1, the beam scanning method is any one of the exhaustive beam scanning scheme and the hierarchical codebook beam scanning scheme; the method of converting into an RSRP image is: using the RSRP values of different beam pairs to generate a color heat map or a gray scale image. 3.The method of claim 1, wherein, In step 2, the algorithm for estimating the angles of arrival AOA and angles of departure AOD of the detected multipath is any one of the deep learning-based YOLO algorithm, MUISC, ESPRIT, and OMP algorithm.

4. The method of claim 1, wherein, The specific method in step 3 is as follows: (3.1) using the tracking algorithm ByteTrack, based on the estimation of the AOA and AOD of the multipath in the RSRP image, tracking the position change process of the multipath in the continuous RSRP image during the user motion, the method is as follows: (1) take the RSRP image I of a certain position in user motion as a frame, and the RSRP images of different positions arranged in time sequence in user motion trajectory form a continuous RSRP image sequence (2) For the k-th frame I k , the bounding boxes detected by YOLO are divided into two sets according to the confidence threshold τ, the set of bounding boxes higher than the threshold and the set of bounding boxes lower than the threshold (3) Set defined as the set of trajectories tracked by the kth frame (4) In the first frame I1, the high-confidence detection box is initialized to the trajectory (5) In subsequent frames, each multipath trace is predicted in the new position of the current frame I k+1 based on the previous position of the previous frame I by the Kalman filter ; (6) For the k+1 frame , the trajectory association of the first step is performed, and the similarity between the detection box in and the trajectory prediction detection box and the lost tracking trajectory is calculated by the intersection over union IoU , and the Hungarian algorithm is used to associate and and the lost tracking trajectory based on the similarity; (7) After the first association, the trajectories that are successfully associated are added to In this case, the unassociated predicted trajectories and the detection boxes are denoted as and (8) The second step of association is performed between and The similarity between the detection box of the trajectory prediction in and the detection box in is calculated by the intersection over union IoU, and the Hungarian algorithm is used to associate and based on the similarity. (9) After the second association, the trajectories successfully associated in the set , are no longer retained in the set of detection boxes that have not been successfully associated, the trajectories that have not been associated will be stored in the set (10) will be associated in the subsequent frames, for trajectories that cannot be re-tracked within a certain number of frames thereafter will be deleted; (11) high confidence bounding boxes that are not matched will be initialized to the new track and stored in ; (12) For each path in the k+1 frame, based on the result of the current frame tracking Assign an identification ID to each path; (3.2) record and AOA and AOD of the ID q radial in the pth frame of RSRP images in the user motion trajectory, respectively, The AOA and AOD coordinates of the ID q radial in all frames are put into the set , 5. The method of claim 4, wherein the method further comprises: In step 4, the identification method of the LOS path and the NLOS path is as follows: (3) record The newest and the second newest AOA-AOD coordinates are respectively and where i>j, the LOS and NLOS identification of the qth path is performed by calculating the following formula: If Z>0, the qth path is identified as a LOS path, and if Z<0, it is a NLOS path.

6. The method of claim 5, wherein the method further comprises: The method of step 5 is as follows: (1) the NLOS path moves along a straight line with a slope of -1, and the scatterers generating different NLOS paths are located at the same azimuth angle α of the base station, using straight lines with different intercepts and a slope of -1 to represent the trajectories of different NLOS paths, to distinguish the scatterers in the environment, assuming that the qth path is a NLOS path, its AOA and AOD motion trajectory is represented as: Φ = -Θ + Δ q where Θ, Φ and Δ q are the AOA, AOD and intercept of the qth NLOS path, respectively; (2) applying least squares to all the AOA-AOD coordinates in to estimate Δ q ; (3) based on Θ NLOS = 2a - f NLOS + k p, derive D q = 2a q + k p, thereby estimate the azimuth angle a q of the virtual base station on the scatterer relative to the base station for the q-th NLOS path NLOS ; f is the angle of departure AOD of the NLOS path; k is a constant; (4) the position of the base station in the environment and the position of the environmental scatterer are fixed, the trajectories of the LOS path and the NLOS path generated by the scatterer have characteristics in the RSRP image, a mapping between the scatterer position and the user position is constructed, according to the position information of the user and the environment reflected by the AOA and AOD of each multipath, combined with the user motion state information output by the inertial measurement unit IMU, the user position is estimated; (5) In the RSRP image, the motion trajectory of the LOS path lies on Θ LOS = Φ LOS + kπ, considering Θ LOS ∈ [0, 2π), Φ LOS ∈ [0, 2π), the trajectory of the LOS path is divided into two segments in the RSRP image, Θ LOS = Φ LOS - π, Θ LOS ∈ [0, π) and Θ LOS = Φ LOS + π, Θ LOS ∈ [π, 2π); the Φ LOS is the angle of departure AOD of the LOS path; (6) In the RSRP image, if a virtual base station generated by a certain scatterer is located at the azimuth angle of the base station as α, the NLOS motion trajectory is located at Θ NLOS = 2α - Φ NLOS + π, Above, considering that the angle value range in the RSRP image is [0, 2π), when and , because part of the line segment exceeds the angle value range [0, 2π) of the RSRP image, the part exceeding the range [0, 2π) is taken modulo. (7) Combine the LOS and NLOS path trajectory expressions in steps (5) and (6), and divide the RSRP image into regions according to the position of the virtual base station generated by the scatterer relative to the base station, and on this basis, construct a mapping codebook of different regions of the RSRP image and the positions of the scatterers in the environment; (8) Locate the azimuth angle of the virtual base station generated by the scatterer relative to the base station according to the mapping codebook in step (7).

7. A communication and environment perception integrated device based on deep learning, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: The processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

Images