5G beam forming method based on target detection and three-dimensional measurement assistance

Abstract

The invention discloses a 5G beam forming method based on target detection and three-dimensional measurement assistance, and the method comprises the following steps: S10, carrying an optical RGB camera above each 5G base station, and combining the 5G base stations and the optical RGB cameras to form a visual radio reference anchor point; s20, acquiring absolute spatial position coordinates of the optical RGB cameras by using a high-precision surveying and mapping instrument, acquiring attitude rotation matrixes of the optical RGB cameras relative to a world coordinate system by using a structured vanishing point direction estimation method, and acquiring relative poses of the optical RGB cameras; calculating plane parameters of the real ground in a camera coordinate system in combination with the gravity direction vector and the optical RGB camera pose; according to the invention, the target person is accurately positioned and tracked through cross-view multi-optical RGB camera target detection and three-dimensional measurement of computer vision, and the three-dimensional spherical coordinate of the user relative to the 5G base station is accurately solved in real time.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, artificial intelligence and mobile communication technology, and specifically to a 5G beamforming method based on target detection and three-dimensional measurement assistance. Background Art

[0002] Beamforming is one of the key technologies in the 5GNR communication system. By weighting the antenna array weights, a narrow beam in a specific direction can be formed. It has the functions of improving signal transmission quality, improving base station service range, and improving energy utilization, and can effectively support the high rate, low latency and large number of connections of the 5G mobile communication system. According to different weight strategies, traditional 5G beamforming technology is divided into static beamforming based on precoding and dynamic beamforming based on time-sharing beam scanning. The former will form beams with a fixed number, width and direction. When the beam direction is misaligned, there is a problem of decreased channel gain. The latter requires SSB scanning, user random access, CSI-RS scanning, and user reporting of SSB measurement results. The complex handshake between the 5G base station and the user causes transmission delay, which is difficult to adapt to the application scenario of high dynamics of pedestrians in a small range.

[0003] Computer vision-based object detection technology can identify, detect and track specific objects in videos or images. Due to the development of deep learning technology, the accuracy, real-time performance and reliability of visual object detection methods have been significantly improved, and they have been widely used in intelligent tasks such as video surveillance and automatic control. The cameras pre-deployed in the environment usually maintain a fixed posture and have a limited field of view, which makes it impossible to achieve large-scale monitoring and recognition. Cross-view multi-camera object detection technology can realize the feature association of specific objects within the field of view of multiple cameras, which is of great significance for real-time monitoring of complex application scenarios. However, multi-camera-based object detection technology can usually only obtain the two-dimensional plane coordinates of the target, and has scale uncertainty.

[0004] The three-dimensional measurement technology based on computer vision can use geometric constraint information such as epipolar geometry and homography transformation to obtain the three-dimensional spatial position coordinates of a specific object through the camera plane pixel coordinates to achieve three-dimensional measurement. It has been widely used in intelligent tasks such as three-dimensional reconstruction, robot navigation, simultaneous positioning and mapping. When there are overlapping ranges of multiple camera fields of view in the environment, the scene depth information and the three-dimensional position information of the corresponding object in the image can be restored by triangulation reconstruction based on binocular vision or least squares recovery, but it is still the relative coordinates of the real space elements in the camera coordinate system; when the environment is only captured by a single camera field of view, the homography matrix can be used to calculate the associated transformation between the camera plane coordinates and the real ground coordinates, but the real position of the camera relative to the world coordinate system needs to be obtained in advance. Summary of the invention

[0005] The purpose of the present invention is to provide a 5G beamforming method based on target detection and three-dimensional measurement assistance to solve the practical application problems of 5G beamforming direction misalignment, channel gain reduction and communication resource waste in complex dynamic pedestrian scenes.

[0006] In order to achieve the above object, the present invention provides the following technical solution: a 5G beamforming method based on target detection and three-dimensional measurement assistance, comprising the following steps:

[0007] Step S10: Each 5G base station is also equipped with an optical RGB camera, and the 5G base station and the optical RGB camera form a visual radio reference anchor point;

[0008] Step S20: using a high-precision surveying and mapping instrument to obtain the absolute spatial position coordinates of the optical RGB camera, using a structured vanishing point direction estimation method to obtain the attitude rotation matrix of the optical RGB camera relative to the world coordinate system, and obtaining the relative posture of each optical RGB camera; combining the gravity direction vector and the optical RGB camera posture, calculating the plane parameters of the real ground in the camera coordinate system, and obtaining the homography transformation matrix of the pixel plane coordinate system of each optical RGB camera relative to the ground;

[0009] Step S30: Each 5G base station performs omnidirectional scanning of the synchronization signal block SSB in a time-sharing manner. The user uses a wide beam to scan. After the user performs random access, the narrow beam range of the base station is preliminarily determined to achieve rough alignment of the beam direction. At the same time, the optical RGB camera is activated to capture images. The user and the mobile terminal device used are two-dimensionally positioned through the target detection algorithm to obtain the candidate box corresponding to the mobile terminal and the two-dimensional coordinates of its center of mass;

[0010] Step S40: performing association matching on the users detected by the multiple optical RGB cameras to determine whether the user appears in the field of view of the multiple optical RGB cameras at the same time;

[0011] Step S50: Using the three-dimensional spatial position coordinates of the mobile terminal, calculate the depth, azimuth and elevation angle relative to each reference anchor point, calculate the array steering vector and antenna weights of the planar antenna array, and the 5G base station vector-adds the weights to the downlink signal to be transmitted, adjusts the beam width, direction and power, and realizes 5G intelligent beamforming.

[0012] Furthermore, in step S10, the optical RGB camera orientation of each reference anchor point is initially aligned with the direction of the 5G base station antenna array, and an associated mapping between the camera plane coordinates and the 5G base station three-dimensional spherical coordinates is constructed, so that the optical RGB camera visual perception can assist in adjusting the 5G signal beam direction.

[0013] Furthermore, in step S20, a high-precision surveying and mapping instrument is used to obtain the three-dimensional spatial position coordinates of the reference anchor point, and based on the characteristic that the vanishing point coordinates in a usual structured environment are only related to the camera posture in the camera coordinate system, a vanishing point detection method of computer vision is used to obtain the posture rotation matrix of each optical RGB camera relative to the real world, thereby completing the external parameter calibration of all optical RGB cameras.

[0014] Furthermore, a three-dimensional rectangular camera coordinate system and a three-dimensional spherical 5G base station coordinate system are established respectively, so as to associate the two-dimensional coordinates of the camera plane, the three-dimensional coordinates of the real-world user, and the antenna array beam direction.

[0015] Furthermore, the 5G base station beam scanning is applied simultaneously with the optical RGB camera target detection and three-dimensional measurement. In the time-sharing scanning, the communication link with the user is determined. In the field of view of the optical RGB camera, the user and the terminal device used are two-dimensionally tracked. After solving the three-dimensional spatial position relationship, the three-dimensional spherical coordinates relative to the 5G base station are further calculated to assist in the calculation of the antenna array steering vector and array weights, thereby performing targeted beamforming of the downlink signal direction, width and power.

[0016] Furthermore, at the beam management level, only a rough alignment between the 5G base station and the user's mobile terminal device is required, and then the beam direction is fine-tuned through target detection and three-dimensional measurement. There is no need for CSI-RS scanning and user SSB measurement feedback reporting.

[0017] Furthermore, in step S40, if the user can be captured by two or more optical RGB cameras and a communication link is established with a 5G base station, the three-dimensional spatial position coordinates of the mobile terminal are calculated by binocular triangulation or least squares; if the user can only be captured by one optical RGB camera and a communication link is established with the corresponding 5G base station, the homography transformation matrix is ​​used to solve the user's position coordinates on the ground.

[0018] Furthermore, the three-dimensional spatial positions of users and mobile terminal devices obtained through computer vision can be converted into three-dimensional spherical coordinates relative to the base station antenna array, and then the 5G base station antenna array steering vector can be obtained through the Kronecker product, thereby realizing the weighting of the array antenna channel and the precise alignment of the radio downlink signal.

[0019] The beneficial effects of the present invention are as follows: in the above technical scheme, the present invention accurately locates and tracks the target personnel through cross-view multi-optical RGB camera target detection and three-dimensional measurement of computer vision, solves the three-dimensional spherical coordinates of the user relative to the 5G base station in real time and accurately, and calculates the antenna array weight matrix, thereby realizing fast and accurate beamforming of 5G downlink signals, wherein the cross-view multi-optical RGB camera target detection solves the problem of insufficient field of view coverage of a single optical RGB camera, the combination of multi-optical RGB camera three-dimensional measurement and single optical RGB camera homography transformation solves the problem of three-dimensional monitoring and positioning of dynamic pedestrians in complex scenes with unknown depth information, and the conversion of visual three-dimensional rectangular coordinates to base station three-dimensional spherical coordinates realizes the efficient solution of the antenna array weight matrix, and solves the problems of delay, shaping misalignment and power waste caused by the traditional 5G beam scanning handshake process;

[0020] The present invention uses computer vision target detection and three-dimensional measurement to assist in the beamforming of 5G base station downlink signals, thereby improving the signal quality and enhancing the reliability of wireless link transmission in specific areas such as office buildings, railway stations, and airports. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0022] Figure 1 A method step diagram provided for an embodiment of the present invention;

[0023] Figure 2 A conceptual diagram of a method provided by an embodiment of the present invention;

[0024] Figure 3 A flow chart of a computer vision-assisted 5G beamforming method provided in an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of the 5G base station antenna array weight calculation principle provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0027] In the description of the present invention, unless otherwise specified, "multiple" means two or more; the orientation or position relationship indicated by the terms "upper", "lower", "left", "right", "inner", "outer", "front end", "rear end", "head", "tail", etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0028] like Figure 1-4 As shown, an embodiment of the present invention provides a 5G beamforming method based on target detection and three-dimensional measurement assistance, comprising the following steps:

[0029] Step S10: Each 5G base station is also equipped with an optical RGB camera. The 5G base station and the optical RGB camera together form a visual radio reference anchor point. Each 5G base station is equipped with an optical RGB camera. For spaces where pedestrians can move in three dimensions, multiple optical RGB cameras are used to cross the field of view to ensure that there are overlapping areas. For spaces where pedestrians can only move in two dimensions, only one optical RGB camera can be arranged. The direction of the optical RGB camera of each reference anchor point is initially aligned with the direction of the 5G base station antenna array. A three-dimensional rectangular camera coordinate system and a three-dimensional spherical 5G base station coordinate system are established respectively, so that the camera plane two-dimensional coordinates, the real-world user three-dimensional coordinates and the antenna array beam direction are associated, and an associated mapping between the camera plane coordinates and the 5G base station three-dimensional spherical coordinates is constructed, so that the optical RGB camera visual perception can assist in adjusting the 5G signal beam direction.

[0030] Step S20: using a high-precision surveying and mapping instrument to obtain the absolute spatial position coordinates of the optical RGB camera, using a structured vanishing point direction estimation method to obtain the attitude rotation matrix of the optical RGB camera relative to the world coordinate system, and obtaining the relative position and posture of each optical RGB camera; combining the gravity direction vector and the optical RGB camera posture, calculating the plane parameters of the real ground in the camera coordinate system, obtaining the homography transformation matrix of the pixel plane coordinate system of each optical RGB camera relative to the ground, using a high-precision surveying and mapping instrument to obtain the three-dimensional spatial position coordinates of the reference anchor point, according to the characteristics that the vanishing point coordinates in a usually structured environment are only related to the camera posture in the camera coordinate system, using a vanishing point detection method of computer vision to obtain the attitude rotation matrix of each optical RGB camera relative to the real world, thereby completing the external parameter calibration of all optical RGB cameras;

[0031] Step S30: Each 5G base station performs omnidirectional scanning of the synchronization signal block SSB in a time-sharing manner. The user uses a wide beam to scan. After the user performs random access, the narrow beam range of the base station is preliminarily determined to achieve rough alignment of the beam direction. At the same time, the optical RGB camera is activated to capture images. The user and the mobile terminal device used are two-dimensionally positioned through the target detection algorithm to obtain the candidate box corresponding to the mobile terminal and the two-dimensional coordinates of its center of mass;

[0032] Step S40: performing association matching on the users detected by multiple optical RGB cameras to determine whether the user appears in the field of view of multiple optical RGB cameras at the same time; if the user can be captured by two or more optical RGB cameras and establish a communication link with the 5G base station, the three-dimensional spatial position coordinates of the mobile terminal are calculated by binocular triangulation or least squares; if the user can only be captured by one optical RGB camera and establish a communication link with the corresponding 5G base station, the homography transformation matrix is ​​used to solve the user's position coordinates on the ground;

[0033] Step S50: Using the three-dimensional spatial position coordinates of the mobile terminal, calculate the depth, azimuth and elevation angle relative to each reference anchor point, calculate the array steering vector and antenna weights of the planar antenna array, and the 5G base station vector-adds the weights to the downlink signal to be transmitted, adjusts the beam width, direction and power, and realizes 5G intelligent beamforming.

[0034] Specifically, the 5G base station and the optical RGB camera are used as a reference anchor point. The number of reference anchor points is increased in the area where the user can move in three dimensions, and the number of reference anchor points can be reduced (the minimum is 1) in the area where the user can only move in two dimensions.

[0035] For each reference anchor point, a 3D spherical coordinate system is constructed with the 5G base station as the reference origin with azimuth angle φ, elevation angle θ and distance r all equal to 0; the optical center of the camera is used as the 3D coordinate X c , Yc and Z c The reference origin is 0, and a three-dimensional rectangular coordinate system is constructed. c The axis points forward from the camera's optical center, aligning the initial direction of the base station antenna array and making the baseline vector of the optical RGB camera and 5G antenna array approximately 0.

[0036] Use high-precision RTK receivers and total stations to measure the three-dimensional geographic coordinates of the reference anchor network and convert them into local northeast celestial coordinates. (n is the number of reference anchor points).

[0037] For each reference anchor point, the optical RGB camera takes a one-time shot of the field of view environment image to obtain the coordinate vector representation of the vanishing point of the environment structure in the nth camera coordinate system. Since the vanishing point coordinate vector has the following relationship with the optical RGB camera posture:

[0038]

[0039] in is the pose of the optical RGB camera relative to the local Manhattan world coordinate system, (d0 d1 d2) T is the unit direction vector corresponding to the three-axis coordinate system. Since the local Manhattan world coordinate system is usually aligned with the three-axis direction of the world coordinate system, the attitude rotation matrix of each optical RGB camera relative to the world coordinate system can be obtained

[0040] For each reference anchor point’s optical RGB camera, obtain the plane parameters n of the real ground plane in the camera coordinate system. T p+d=0, where n is the normal vector of the ground plane in the camera coordinate system, which is parallel to the gravity direction vector, so it can also be solved by the vanishing point coordinates in the vertical direction. p is the spatial point coordinate on the ground plane, and d is the distance from the camera optical center to the ground plane. Directly obtain. Solve the homography transformation matrix between its pixel coordinates and the real ground coordinates as follows:

[0041]

[0042] This completes the initialization configuration of each reference anchor point.

[0043] The 5G base station at each reference anchor point uses time-sharing scanning to transmit SSB beams on the network side, and users scan with wide beams. After both the user and the base station have scanned once, the preliminary narrow beam range of the base station and the wide beam range of the user can be determined, and the user sends PRACH for random access.

[0044] After detecting the user base station access, the reference anchor activates the optical RGB camera to capture images. A multi-label Markov random method is used to detect and identify users under multiple optical RGB cameras across perspectives, while taking into account the different lighting, brightness and appearance changes of users under different perspectives, extracting key features for data association and user identification, realizing multi-perspective multi-user target detection and tracking, and obtaining the coordinates of user candidate boxes.

[0045] Furthermore, for specific users after candidate box recognition, target detection frameworks such as Yolo are used to further detect mobile terminal devices such as smartphones, tablets, and laptops, obtain the candidate box of the mobile terminal, and calculate the two-dimensional centroid coordinates.

[0046] If the user appears in the field of view of multiple optical RGB cameras at the same time, triangulation is used to perform the least squares solution of the three-dimensional space position, as follows:

[0047]

[0048] in They are the 1st, 2nd and 3rd row vectors of the optical RGB camera pose matrix (R|t) respectively.

[0049] If the user only appears in the field of view of one optical RGB camera and moves on the ground plane, the pre-calibrated homography change matrix is ​​used to perform a linear solution for the three-dimensional spatial position, as follows:

[0050]

[0051] Based on the user's three-dimensional spatial position P W = (XYZ) and the spatial position of the reference anchor point The azimuth, elevation and depth of the user relative to the 5G base station antenna array can be solved as follows:

[0052]

[0053] like Figure 4 As shown, according to the azimuth and elevation angles, the Kronecker product is used to calculate the 5G base station antenna array steering vector as follows:

[0054]

[0055] in is the Kronecker product, and the depth r is used to control the beam power.

[0056] Furthermore, after calculating the array steering vector, the array antenna weight matrix w(i) can be obtained. Assuming that the antenna channel sequence is i, the channel input signal is x(i), the noise introduced when passing through the channel H is N, and the channel output signal is y(i), we can get

[0057] y(i)=Hx(i)+N#(7)

[0058] After weighting with the complex vector w(i), we can get

[0059] y(i)=Hw(i)x(i)+N#(8)

[0060] Beamforming is achieved by changing the signal amplitude and phase.

[0061] The 5G base station beam scanning is applied simultaneously with the optical RGB camera target detection and three-dimensional measurement. During the time-sharing scanning, a communication link is established with the user. Within the field of view of the optical RGB camera, two-dimensional positioning and tracking of the user and the terminal device used by the user are performed. After solving the three-dimensional spatial position relationship, the three-dimensional spherical coordinates relative to the 5G base station are further calculated to assist in the calculation of the antenna array steering vector and array weights, thereby performing targeted beamforming of the downlink signal direction, width and power.

[0062] At the beam management level, only rough alignment of the 5G base station and the user's mobile terminal device is required, and then the beam direction is fine-tuned through target detection and three-dimensional measurement. There is no need for CSI-RS scanning and user SSB measurement feedback reporting.

[0063] The three-dimensional spatial position of users and mobile terminal devices obtained through computer vision can be converted into three-dimensional spherical coordinates relative to the base station antenna array, and then the 5G base station antenna array steering vector can be obtained through the Kronecker product to achieve weighted array antenna channels and precise alignment of radio downlink signals.

[0064] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A 5G beamforming method based on target detection and three-dimensional measurement assistance, characterized in that: The steps include: Step S10: Each 5G base station is also equipped with an optical RGB camera, and the 5G base station and the optical RGB camera form a visual radio reference anchor point; Step S20: using a high-precision surveying and mapping instrument to obtain the absolute spatial position coordinates of the optical RGB camera, using a structured vanishing point direction estimation method to obtain the posture rotation matrix of the optical RGB camera relative to the world coordinate system, and obtaining the relative posture of each optical RGB camera; Combine the gravity direction vector and the optical RGB camera pose to calculate the plane parameters of the real ground in the camera coordinate system, and obtain the homography transformation matrix of the pixel plane coordinate system of each optical RGB camera relative to the ground. Step S30: Each 5G base station performs omnidirectional scanning of the synchronization signal block SSB in a time-sharing manner. The user uses a wide beam to scan. After the user performs random access, the narrow beam range of the base station is preliminarily determined to achieve rough alignment of the beam direction. At the same time, the optical RGB camera is activated to capture images. The user and the mobile terminal device used are two-dimensionally positioned through the target detection algorithm to obtain the candidate box corresponding to the mobile terminal and the two-dimensional coordinates of its center of mass; Step S40: performing association matching on the users detected by the multiple optical RGB cameras to determine whether the user appears in the field of view of the multiple optical RGB cameras at the same time; Step S50: Using the three-dimensional spatial position coordinates of the mobile terminal, calculate the depth, azimuth and elevation angle relative to each reference anchor point, calculate the array steering vector and antenna weights of the planar antenna array, and the 5G base station vector-adds the weights to the downlink signal to be transmitted, adjusts the beam width, direction and power, and realizes 5G intelligent beamforming.

2. According to claim 1, a 5G beamforming method based on target detection and three-dimensional measurement assistance is characterized in that: In step S10, the optical RGB camera orientation of each reference anchor point is initially aligned with the direction of the 5G base station antenna array, and an associated mapping between the camera plane coordinates and the 5G base station three-dimensional spherical coordinates is constructed, so that the optical RGB camera visual perception can assist in adjusting the 5G signal beam direction.

3. The 5G beamforming method based on target detection and three-dimensional measurement assistance according to claim 1, characterized in that: In step S20, a high-precision surveying and mapping instrument is used to obtain the three-dimensional spatial position coordinates of the reference anchor point. According to the characteristic that the vanishing point coordinates in a usual structured environment are only related to the camera posture in the camera coordinate system, a vanishing point detection method of computer vision is used to obtain the posture rotation matrix of each optical RGB camera relative to the real world, thereby completing the external parameter calibration of all optical RGB cameras.

4. The 5G beamforming method based on target detection and 3D measurement assistance according to claim 2, characterized in that: A three-dimensional rectangular camera coordinate system and a three-dimensional spherical 5G base station coordinate system are established respectively to associate the camera plane two-dimensional coordinates, the real-world user three-dimensional coordinates, and the antenna array beam direction.

5. The 5G beamforming method based on target detection and three-dimensional measurement assistance according to claim 1, characterized in that: The 5G base station beam scanning is applied simultaneously with the optical RGB camera target detection and three-dimensional measurement. During the time-sharing scanning, a communication link is established with the user. Within the field of view of the optical RGB camera, two-dimensional positioning and tracking of the user and the terminal device used by the user are performed. After solving the three-dimensional spatial position relationship, the three-dimensional spherical coordinates relative to the 5G base station are further calculated to assist in the calculation of the antenna array steering vector and array weights, thereby performing targeted beamforming of the downlink signal direction, width and power.

6. The 5G beamforming method based on target detection and three-dimensional measurement assistance according to claim 1, characterized in that: At the beam management level, only rough alignment of the 5G base station and the user's mobile terminal device is required, and then the beam direction is fine-tuned through target detection and three-dimensional measurement. There is no need for CSI-RS scanning and user SSB measurement feedback reporting.

7. The 5G beamforming method based on target detection and 3D measurement assistance according to claim 1, characterized in that: In step S40, if the user can be captured by two or more optical RGB cameras and establish a communication link with the 5G base station, the three-dimensional spatial position coordinates of the mobile terminal are calculated by binocular triangulation or least squares; if the user can only be captured by one optical RGB camera and establish a communication link with the corresponding 5G base station, the homography transformation matrix is ​​used to solve the user's position coordinates on the ground.

8. The 5G beamforming method based on target detection and 3D measurement assistance according to claim 1, characterized in that: The three-dimensional spatial position of users and mobile terminal devices obtained through computer vision can be converted into three-dimensional spherical coordinates relative to the base station antenna array, and then the 5G base station antenna array steering vector can be obtained through the Kronecker product to achieve weighted array antenna channels and precise alignment of radio downlink signals.

Images