A 5G beamforming method based on target detection and three-dimensional measurement assistance

Abstract

The application discloses a 5G beam forming method based on target detection and three-dimensional measurement assistance, which comprises the following steps: step S10: an optical RGB camera is mounted above each 5G base station, and the 5G base station and the optical RGB camera are combined to form a visual radio reference anchor point; step S20: the absolute space position coordinates of the optical RGB camera are obtained by using a high-precision surveying instrument, the attitude rotation matrix of the optical RGB camera relative to a world coordinate system is obtained by using a structured vanishing point direction estimation method, and the relative pose of each optical RGB camera is obtained; the plane parameters of the real ground in the camera coordinate system are calculated in combination with the gravity direction vector and the optical RGB camera pose; the target personnel is accurately positioned and tracked through the cross-view multi-optical RGB camera target detection and three-dimensional measurement of computer vision, and the three-dimensional spherical coordinates of the user relative to the 5G base station are solved in real time and accurately.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of computer vision, artificial intelligence and mobile communication technology, in particular to a 5G beamforming method based on target detection and three-dimensional measurement assistance. BACKGROUND

[0002] Beamforming is one of the key technologies in 5G NR communication system, through the weighting of antenna array weight, a narrow beam in a specific direction can be formed, which can improve the signal transmission quality, improve the service range of base station and improve the energy utilization rate, and can effectively support the high speed, low latency and large number of connections of 5G mobile communication system. According to different weight strategies, the traditional 5G beamforming technology is divided into static beamforming based on precoding and dynamic beamforming based on time-sharing beam scanning. The former will form a fixed number, width and direction of beam, and when the beam direction is misaligned, the channel gain will decrease. The latter needs to go through SSB scanning, user random access, CSI-RS scanning, user reporting SSB measurement results and other processes. The complex handshake between 5G base station and user leads to transmission delay, which is difficult to adapt to the high dynamic application scenario of pedestrians in a small range.

[0003] Target detection technology based on computer vision can identify and detect and track specific objects in video or images. Due to the development of deep learning technology, the accuracy, real-time performance and reliability of visual target detection methods have been significantly improved, and have been widely used in intelligent tasks such as video monitoring and automatic control. The camera pre-deployed in the environment usually maintains a fixed pose, and the field of view is limited, which cannot realize large-scale monitoring and identification. Cross-view multi-camera target detection technology can realize the feature correlation of specific objects in the field of view of multiple cameras, which has important significance for real-time monitoring of complex application scenarios. However, the target detection technology based on multiple cameras can only obtain the two-dimensional plane coordinates of the target, and has scale uncertainty.

[0004] Three-dimensional measurement technology based on computer vision can use geometric constraint information such as epipolar geometry and homography transformation to obtain the three-dimensional spatial position coordinates of specific objects through camera plane pixel coordinates, realize three-dimensional measurement, and has been widely used in intelligent tasks such as three-dimensional reconstruction, robot navigation, simultaneous localization and mapping. When there is an overlapping range of multi-camera field of view in the environment, the scene depth information and three-dimensional position information of the corresponding objects in the image can be obtained according to the binocular vision triangulation reconstruction or least squares, but it is still the relative coordinates of real space elements in the camera coordinate system; when the environment is only captured by a single camera field of view, the corresponding transformation between camera plane coordinates and real ground coordinates can be calculated through the homography matrix, but the real pose of the camera relative to the world coordinate system needs to be obtained in advance. SUMMARY

[0005] The application aims to provide a 5G beam forming method based on target detection and three-dimensional measurement assistance to solve the problems of 5G beam forming direction misalignment, channel gain decline and communication resource waste in complex dynamic pedestrian scenes.

[0006] To achieve the above-mentioned purpose, the application provides the following technical scheme: a 5G beam forming method based on target detection and three-dimensional measurement assistance, comprising the following steps:

[0007] Step S10: An optical RGB camera is mounted above each 5G base station, and the 5G base station and the optical RGB camera are combined to form a visual radio reference anchor point;

[0008] Step S20: The absolute space position coordinates of the optical RGB camera are obtained by using a high-precision surveying instrument, the attitude rotation matrix of the optical RGB camera relative to the world coordinate system is obtained by using a structured vanishing point direction estimation method, and the relative pose of each optical RGB camera is obtained; the plane parameters of the real ground in the camera coordinate system are calculated in combination with the gravity direction vector and the optical RGB camera pose, and the homography matrix of the pixel plane coordinate system of each optical RGB camera relative to the ground is obtained;

[0009] Step S30: Each 5G base station performs omnidirectional scanning of a synchronization signal block (SSB) in time, a user uses a wide beam scanning, and the user preliminarily determines the narrow beam range of the base station after random access, thereby realizing coarse alignment of the beam direction, and simultaneously activating the optical RGB camera to take images, performing two-dimensional positioning on the user and the mobile terminal device used by the target detection algorithm, and obtaining the candidate box and the two-dimensional coordinates of the mass center corresponding to the mobile terminal;

[0010] Step S40: The users detected by multiple optical RGB cameras are associated and matched, and it is judged whether the user appears in the visual range of multiple optical RGB cameras at the same time;

[0011] Step S50: The three-dimensional space position coordinates of the mobile terminal are used to calculate the depth, azimuth angle and pitch angle relative to each reference anchor point, calculate the array steering vector and antenna weight of the planar antenna array, and the 5G base station performs vector addition on the weight and the to-be-transmitted downlink signal, thereby adjusting the beam width, direction and power, and realizing 5G intelligent beam forming.

[0012] Further, in step S10, the optical RGB camera of each reference anchor point is initially aligned in the direction of the 5G base station antenna array, and the association mapping between the camera plane coordinates and the 5G base station three-dimensional spherical coordinates is constructed, so that the visual perception of the optical RGB camera can assist the 5G signal beam direction adjustment.

[0013] Further, in step S20, the three-dimensional spatial position coordinates of the reference anchor points are obtained by using high-precision surveying instruments, and according to the characteristics that the vanishing point coordinates in a structured environment are only related to the camera pose in the camera coordinate system, the pose rotation matrix of each optical RGB camera relative to the real world is obtained by using the vanishing point detection method of computer vision, so as to complete the external parameter calibration of all optical RGB cameras.

[0014] Further, the three-dimensional rectangular camera coordinate system and the three-dimensional spherical 5G base station coordinate system are established respectively, so as to associate the camera plane two-dimensional coordinates, the real world user three-dimensional coordinates and the antenna array beam direction.

[0015] Further, the 5G base station beam scanning is simultaneously applied with the optical RGB camera target detection and three-dimensional measurement, the communication link with the user is determined in the time-sharing scanning, the two-dimensional positioning and tracking of the user and the terminal equipment used by the user are performed within the field of view of the optical RGB camera, the three-dimensional spatial position relationship is solved, the three-dimensional spherical coordinates relative to the 5G base station are further calculated, and the calculation of the antenna array steering vector and the array weight is assisted, so that the beamforming of the downlink signal direction, width and power is performed.

[0016] Further, at the beam management level, only the coarse alignment of the 5G base station and the user mobile terminal equipment is needed, and then the beam direction is fine-adjusted by target detection and three-dimensional measurement, without the need of CSI-RS scanning and user SSB measurement feedback reporting.

[0017] Further, in step S40, if the user can be captured by two or more optical RGB cameras and establish a communication link with the 5G base station, the three-dimensional spatial position coordinates of the mobile terminal are calculated by binocular triangulation or least squares; if the user can be captured by only one optical RGB camera and establish a communication link with the corresponding 5G base station, the position coordinates of the user on the ground are solved by using the homography transformation matrix.

[0018] Further, the three-dimensional spatial position of the user and the mobile terminal equipment obtained by computer vision can be converted into three-dimensional spherical coordinates relative to the base station antenna array, and then the 5G base station antenna array steering vector can be obtained by Kronecker product, so as to realize the weighting of the array antenna channel and the accurate alignment of the radio downlink signal.

[0019] The beneficial effects of the present application are that in the above technical solution, the present application accurately locates and tracks the target personnel through cross-view multi-optical RGB camera target detection and three-dimensional measurement of computer vision, solves the three-dimensional spherical coordinates of the user relative to the 5G base station in real time and accurately, calculates the antenna array weight matrix, thereby realizing the rapid and accurate beam forming of the 5G downlink signal, wherein the cross-view multi-optical RGB camera target detection solves the problem of insufficient field of view coverage of a single optical RGB camera, the combination of multi-optical RGB camera three-dimensional measurement and homographic transformation of a single optical RGB camera solves the three-dimensional monitoring and positioning problem of dynamic pedestrians in the case that the depth information of a complex scene is unknown, the conversion of visual three-dimensional rectangular coordinates to base station three-dimensional spherical coordinates realizes efficient solution of the antenna array weight matrix, and solves the problems of time delay, misalignment and power waste caused by the traditional 5G beam scanning handshake process.

[0020] The present application uses computer vision target detection and three-dimensional measurement to assist the beam forming of the 5G base station downlink signal, and realizes the signal quality improvement and reliability enhancement of wireless link transmission in specific areas such as office buildings, railway stations and airports. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0022] Figure 1 The method step diagram provided for the embodiments of the present application;

[0023] Figure 2 The method concept schematic diagram provided for the embodiments of the present application;

[0024] Figure 3 The computer vision assisted 5G beam forming method flowchart provided for the embodiments of the present application;

[0025] Figure 4 The 5G base station antenna array weight calculation principle diagram provided for the embodiments of the present application. DETAILED DESCRIPTION

[0026] In order to make those skilled in the art better understand the technical solutions of the present application, the present application will be further described in detail with reference to the drawings.

[0027] In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more; the orientation or positional relationship indicated by the terms "upper", "lower", "left", "right", "inner", "outer", "front end", "rear end", "head", "tail" and the like is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In the description of the present application, it should be noted that, unless otherwise specified and limited, the terms "connected", "connected" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0028] As shown in Figures 1-4 The embodiment of the present application provides a 5G beam forming method based on target detection and three-dimensional measurement assistance, which comprises the following steps:

[0029] Step S10: An optical RGB camera is mounted above each 5G base station, and the 5G base station and the optical RGB camera are combined to form a visual radio reference anchor point; each 5G base station is provided with an optical RGB camera, a plurality of optical RGB cameras are used for visual field intersection in a space where pedestrians can move stereoscopically, to ensure that there is an overlapping area; for a space where pedestrians can only move in a plane, only one optical RGB camera can be arranged, the direction of the optical RGB camera of each reference anchor point is initially aligned with the direction of the 5G base station antenna array, a three-dimensional rectangular camera coordinate system and a three-dimensional spherical 5G base station coordinate system are established respectively, the camera plane two-dimensional coordinate, the real world user three-dimensional coordinate and the antenna array beam direction are associated, and the association mapping of the camera plane coordinate and the 5G base station three-dimensional spherical coordinate is constructed, so that the visual perception of the optical RGB camera can assist the adjustment of the 5G signal beam direction;

[0030] Step S20: Obtain the absolute spatial position coordinates of the optical RGB camera using high-precision surveying instruments, obtain the attitude rotation matrix of the optical RGB camera relative to the world coordinate system using a structured vanishing point direction estimation method, and obtain the relative pose of each optical RGB camera; combine the gravity direction vector and the optical RGB camera pose to calculate the plane parameters of the real ground in the camera coordinate system, obtain the homography transformation matrix of the pixel plane coordinate system of each optical RGB camera relative to the ground, obtain the three-dimensional spatial position coordinates of the reference anchor point using high-precision surveying instruments, and obtain the attitude rotation matrix of each optical RGB camera relative to the real world using the vanishing point detection method of computer vision according to the characteristics that the vanishing point coordinates in the camera coordinate system are only related to the camera pose in a usually structured environment, thereby completing the external parameter calibration of all optical RGB cameras;

[0031] Step S30: Each 5G base station performs omnidirectional scanning of the synchronization signal block (SSB), the user uses wide beam scanning, and the user performs random access to preliminarily determine the narrow beam range of the base station, thereby realizing coarse alignment of the beam direction, and simultaneously activating the optical RGB camera to take images, performing two-dimensional positioning on the user and the mobile terminal device used by the target detection algorithm, and obtaining the candidate box corresponding to the mobile terminal and the two-dimensional coordinates of the centroid thereof;

[0032] Step S40: Associate and match the users detected by multiple optical RGB cameras, determine whether the user appears in the field of view range of multiple optical RGB cameras at the same time, if the user can be captured by two or more optical RGB cameras and establish a communication link with the 5G base station, then the three-dimensional spatial position coordinates of the mobile terminal are calculated by binocular triangulation or least squares; if the user can only be captured by one optical RGB camera and establish a communication link with the corresponding 5G base station, then the position coordinates of the user on the ground are solved using the homography transformation matrix;

[0033] Step S50: Calculate the depth, azimuth angle, and elevation angle relative to each reference anchor point using the three-dimensional spatial position coordinates of the mobile terminal, calculate the array steering vector and antenna weight of the planar antenna array, and the 5G base station adds the weight to the to-be-transmitted downlink signal to adjust the beam width, direction, and power, thereby realizing 5G intelligent beamforming.

[0034] Specifically, the 5G base station is matched with the optical RGB camera as a reference anchor point, the number of reference anchor points is increased in the area where the user can move stereoscopically, and the number of reference anchor points can be reduced (minimum of 1) in the area where the user only moves in a plane.

[0035] For each reference anchor point, the 5G base station is taken as the reference origin with the azimuth angle φ, the elevation angle θ, and the distance r being 0, a three-dimensional spherical coordinate system is constructed, the camera optical center is taken as the three-dimensional coordinates X c , Yc and Z c a three-dimensional rectangular coordinate system is constructed with the reference origin point where both X c axis points to the front direction by the camera optical center, at the same time, aligns the initial direction of the base station antenna array, and makes the baseline vectors of the optical RGB camera and the 5G antenna array approximately 0.

[0036] The three-dimensional spatial geographic coordinates of the reference anchor network are measured by using high-precision RTK receivers and total stations, and are converted into local northeast celestial coordinates (n is the number of reference anchors).

[0037] For each optical RGB camera of the reference anchor, a one-time shooting of the field of view environment image is performed to obtain the environmental structure vanishing point coordinate vector representation in the nth camera coordinate system Since the vanishing point coordinate vector and the optical RGB camera pose have the following relationship:

[0038]

[0039] where is the pose of the optical RGB camera relative to the local Manhattan world coordinate system, (d0 d1 d2) T is the unit direction vector of the three-axis coordinate system. Since the local Manhattan world coordinate system is usually aligned with the three-axis direction of the world coordinate system, the pose rotation matrix of each optical RGB camera relative to the world coordinate system can be obtained

[0040] For each optical RGB camera of the reference anchor, the real ground plane in the camera coordinate system is obtained T p+d=0, where n is the camera coordinate system representation of the ground plane normal vector, which is parallel to the gravity direction vector, so it can also be obtained by solving the vertical vanishing point coordinates, p is the spatial point coordinate on the ground plane, and d is the distance from the camera optical center to the ground plane, which is directly obtained by The homography transformation matrix of the pixel coordinates and the real ground coordinates is as follows:

[0041]

[0042] Thus, the initialization configuration of each reference anchor is completed.

[0043] The 5G base station of each reference anchor transmits SSB beams in a time-sharing scanning manner on the network side, and the user scans with a wide beam. After the user and the base station scan once, the preliminary narrow beam range of the base station and the wide beam range of the user can be determined, and the user sends PRACH for random access.

[0044] After detecting the user base station access, the reference anchor activates the optical RGB camera to take pictures. A multi-label Markov random method is used to detect and identify users under multiple optical RGB cameras with different angles, considering the different light, brightness and appearance changes of users at different angles, extracting key features for data correlation and user identification, achieving multi-angle multi-user target detection and tracking, and obtaining user candidate box coordinates.

[0045] Further, for a specific user identified by the candidate box, a Yolo target detection framework is used to further detect mobile terminal devices such as smartphones, tablets and laptops, etc., to obtain the candidate box of the mobile terminal and calculate the two-dimensional centroid coordinates

[0046] If the user appears in the field of view of multiple optical RGB cameras at the same time, a least squares solution of the three-dimensional space position is used by triangulation, as follows:

[0047]

[0048] wherein are the 1st, 2nd and 3rd row vectors of the optical RGB camera pose matrix (R|t).

[0049] If the user only appears in the field of view of one optical RGB camera and moves on the ground plane, a linear solution of the three-dimensional space position is used by a pre-calibrated homography matrix, as follows:

[0050]

[0051] Based on the three-dimensional space position P W =(X Y Z) of the user and the space position The azimuth, elevation and depth of the user relative to the 5G base station antenna array can be solved, as follows:

[0052]

[0053] As shown in Figure 4 , according to the azimuth and elevation, the Kronecker product is used to calculate the 5G base station antenna array steering vector, as follows:

[0054]

[0055] wherein is the Kronecker product, and the depth r is used to control the beam power.

[0056] Further, after the array steering vector is calculated, the array antenna weight matrix w(i) can be obtained, assuming that the antenna channel sequence is i, the channel input signal is x(i), the noise introduced by the channel H is N, and the channel output signal is y(i), then the following can be obtained

[0057] y(i) = Hx(i) + N#(7)

[0058] After being weighted by the complex vector w(i), the following can be obtained

[0059] y(i) = Hw(i)x(i) + N#(8)

[0060] By changing the signal amplitude and phase, beamforming is realized.

[0061] The simultaneous application of 5G base station beam scanning and optical RGB camera target detection and three-dimensional measurement determines the establishment of a communication link with the user in time-sharing scanning, and performs two-dimensional positioning and tracking on the user and the terminal equipment used by the user within the field of view of the optical RGB camera. After solving the three-dimensional spatial position relationship, the three-dimensional spherical coordinates relative to the 5G base station are further calculated, which are used to assist the calculation of the array steering vector and the array weight of the antenna array, so as to perform beamforming on the direction, width and power of the downlink signal.

[0062] At the beam management level, only the coarse alignment of the 5G base station and the user mobile terminal equipment is needed, and then the beam direction is fine-tuned through target detection and three-dimensional measurement, without the need for CSI-RS scanning and user SSB measurement feedback reporting.

[0063] The three-dimensional spatial position of the user and the mobile terminal equipment obtained by computer vision can be converted into three-dimensional spherical coordinates relative to the base station antenna array, and then the 5G base station antenna array steering vector can be obtained through the Kronecker product, so as to realize the weighting of the array antenna channel and the accurate alignment of the radio downlink signal.

[0064] The above only describes certain exemplary embodiments of the present application in a descriptive manner, and it is needless to say that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present application. Therefore, the above figures and descriptions are illustrative in nature and should not be understood as limiting the scope of protection of the claims of the present application.

Claims

1. A 5G beamforming method based on target detection and 3D measurement assistance, characterized in that: The steps include: Step S10: Each 5G base station is also equipped with an optical RGB camera. The 5G base station and the optical RGB camera together form a visual radio reference anchor point. Step S20: using a high-precision surveying instrument to obtain the absolute spatial position coordinates of the optical RGB camera, using a structured vanishing point direction estimation method to obtain the posture rotation matrix of the optical RGB camera relative to the world coordinate system, and obtaining the relative position and posture of each optical RGB camera; Combining the gravity direction vector and the optical RGB camera pose, the plane parameters of the real ground in the camera coordinate system are calculated, and the homography transformation matrix of the pixel plane coordinate system of each optical RGB camera relative to the ground is obtained; Step S30: Each 5G base station performs omnidirectional scanning of the synchronization signal block (SSB) in a time-sharing manner. The user uses a wide beam to scan. After the user performs random access, the narrow beam range of the base station is preliminarily determined to achieve coarse alignment of the beam direction. At the same time, the optical RGB camera is activated to capture images, and the user is detected and identified under the cross-view multi-optical RGB camera. Key features are extracted for data association and user identification, and the coordinates of the user candidate frame are obtained. For the specific user after the candidate frame is identified, the target detection algorithm is used to perform two-dimensional positioning of the mobile terminal device used, and the candidate frame corresponding to the mobile terminal and the two-dimensional coordinates of its center of mass are obtained; Step S40: If the user appears in the field of view of multiple optical RGB cameras at the same time, the three-dimensional spatial coordinates of the mobile terminal are calculated using binocular triangulation and least squares. If the user appears in the field of view of only one optical RGB camera and is moving on the ground plane, the three-dimensional spatial coordinates of the mobile terminal are solved using the homography transformation matrix; Step S50: Using the three-dimensional spatial position coordinates of the mobile terminal, calculate the depth, azimuth, and elevation angle relative to each reference anchor point, and calculate the array steering vector and antenna weights of the planar antenna array. The 5G base station vector-adds the weights to the downlink signal to be transmitted, adjusts the beam width, direction, and power, and realizes 5G intelligent beamforming.

2. A 5G beamforming method based on target detection and 3D measurement assistance according to claim 1, characterized in that: In step S10, the optical RGB camera of each reference anchor point is initially aligned with the direction of the 5G base station antenna array, and an associated mapping between the camera plane coordinates and the 5G base station three-dimensional spherical coordinates is constructed, so that the optical RGB camera visual perception can assist in adjusting the 5G signal beam direction.

3. The 5G beamforming method based on target detection and 3D measurement assistance according to claim 1, characterized in that: In step S20, a high-precision surveying instrument is used to obtain the three-dimensional spatial coordinates of the reference anchor point. Based on the characteristic that the vanishing point coordinates in a structured environment are only related to the camera pose in the camera coordinate system, a vanishing point detection method based on computer vision is used to obtain the pose rotation matrix of each optical RGB camera relative to the real world, thereby completing the extrinsic calibration of all optical RGB cameras.

4. The 5G beamforming method based on target detection and 3D measurement assistance according to claim 2, characterized in that: A three-dimensional rectangular camera coordinate system and a three-dimensional spherical 5G base station coordinate system are established respectively, so that the two-dimensional coordinates of the camera plane, the three-dimensional coordinates of the real-world user and the antenna array beam direction are associated.

5. The 5G beamforming method based on target detection and 3D measurement assistance according to claim 1, characterized in that: During time-sharing scanning, a communication link is established with the user. Within the field of view of the optical RGB camera, two-dimensional positioning and tracking of the user and the terminal device used are performed. After solving the three-dimensional spatial position relationship, the three-dimensional spherical coordinates relative to the 5G base station are further calculated to assist in the calculation of the antenna array steering vector and array weights, thereby performing targeted beamforming of the downlink signal direction, width and power.

6. The 5G beamforming method based on target detection and 3D measurement assistance according to claim 1, characterized in that: At the beam management level, only rough alignment of the 5G base station and the user's mobile terminal device is required, and then the beam direction is fine-tuned through target detection and three-dimensional measurement. There is no need for CSI-RS scanning and user SSB measurement feedback reporting.

7. The 5G beamforming method based on target detection and 3D measurement assistance according to claim 1, characterized in that: The three-dimensional spatial positions of users and mobile terminal devices obtained through computer vision are converted into three-dimensional spherical coordinates relative to the base station antenna array, and then the 5G base station antenna array steering vector is obtained through the Kronecker product to achieve weighted array antenna channels and precise alignment of radio downlink signals.

Images