Beam alignment method and apparatus based on terminal visual perception

Abstract

The application provides a terminal visual perception-based beam alignment method and device, wherein the method comprises the following steps: acquiring multi-view pictures of the environment within the visual perception range of the terminal; acquiring scattering object distribution characteristics corresponding to each scattering object within the visual perception range of the terminal based on the multi-view pictures; and determining the optimal receiving and transmitting beam pair corresponding to the target terminal based on the scattering object distribution characteristics and the position information of the target terminal. The method represents the electromagnetic propagation characteristics of the channel between the terminal and the base station by using the scattering object distribution characteristics and the terminal position information, thereby determining the optimal receiving and transmitting beam pair corresponding to the target terminal, overcoming the defect that the existing beam alignment method has high time-frequency resource overhead, and realizing the rapid alignment of the beam with less pilot resource overhead.

Description

Technical Field

[0001] This invention relates to the field of wireless communication technology, and in particular to a beam alignment method and apparatus based on terminal visual perception. Background Technology

[0002] Beamforming based on large-scale antenna arrays is one of the key technologies for overcoming the high path attenuation problem in high-frequency, high-bandwidth communications such as millimeter waves or terahertz waves. It has become an important foundation for supporting 5G and even the next generation of mobile communication systems to achieve ultra-high-speed, low-latency communication.

[0003] Traditional beam alignment methods require traversing and scanning the entire beam codebook or estimating the channel matrix through pilots. However, in large-scale antenna array scenarios, the number of beams in the codebook is large and the dimension of the system channel matrix is ​​high, resulting in high time and frequency resource overhead for traditional beam alignment methods. Summary of the Invention

[0004] This invention provides a beam alignment method and apparatus based on terminal visual perception, which overcomes the shortcomings of existing beam alignment methods with high time and frequency resource overhead, and achieves beam alignment with lower time delay and time and frequency resource overhead.

[0005] On one hand, the present invention provides a beam alignment method based on terminal visual perception, comprising: acquiring multi-view images of the environment within the terminal's visual perception range; acquiring the scattering object distribution characteristics corresponding to each scattering object within the terminal's visual perception range based on the multi-view images; and determining the optimal transmit / receive beam pair corresponding to the target terminal based on the scattering object distribution characteristics and the location information of the target terminal.

[0006] Furthermore, the step of obtaining the scattering object distribution features corresponding to the scattering objects within the terminal's visual perception range based on the multi-view images includes: extracting scattering object feature information corresponding to various types of scattering objects in the multi-view images using a target detection algorithm; performing feature design based on the scattering object feature information to obtain the scattering object distribution features; wherein, the scattering object feature information includes the relative position information, size information, and attitude information of each scattering object within the terminal's visual perception range in the camera coordinate system.

[0007] Further, the step of designing features based on the scattering object feature information to obtain the scattering object distribution features includes: converting the relative position and attitude information of each scattering object in the camera coordinate system into position and attitude information in the base station coordinate system; dividing the coordinate axis plane where the base station coordinate system is located into multiple grids of equal size; based on the scattering object feature information, determining the normalized maximum length, width, and height, as well as the average azimuth angle, of all scattering objects contained in a single grid within the base station coverage area to obtain the four-dimensional feature vector corresponding to the grid; concatenating the four-dimensional feature vectors corresponding to all grids within the base station coverage area to obtain the scattering object distribution features; wherein, when it is determined that a grid does not contain any scattering objects, the four-dimensional feature vector corresponding to the grid is a zero vector.

[0008] Further, determining the optimal transmit / receive beam pair corresponding to the target terminal based on the distribution characteristics of the scattering object and the location information of the target terminal includes: inputting the distribution characteristics of the scattering object and the location information of the target terminal into a pre-trained beam alignment neural network model to obtain the sequence number of the optimal transmit / receive beam pair; wherein, the location information of the target terminal is the location information of the target terminal in the base station coordinate system, and the beam alignment neural network model includes a first sub-neural network, a second sub-neural network, and a third sub-neural network, wherein the first sub-neural network is used to process the distribution characteristics of the scattering object, the second sub-neural network is used to process the location information of the target terminal, and the third sub-neural network is used to process the fusion features of the distribution characteristics of the scattering object and the location information of the target terminal.

[0009] Further, the step of inputting the scattering object distribution features and the target terminal's location information into a pre-trained beam alignment neural network model to obtain the sequence number of the optimal transmit / receive beam pair includes: inputting the scattering object distribution features into the first sub-neural network to obtain a first output feature; inputting the target terminal's location information in the base station coordinate system into the second sub-neural network to obtain a second output feature; fusing the first output feature and the second output feature to obtain a corresponding fused feature; and inputting the fused feature into the third sub-neural network to obtain the sequence number of the optimal transmit / receive beam pair.

[0010] Furthermore, the first sub-neural network, the second sub-neural network, and the third sub-neural network all adopt a fully connected residual network structure.

[0011] Further, training the beam alignment neural network model specifically includes: acquiring a training set of scattering object distribution features corresponding to scattering objects within the visual perception range of the terminal based on multi-view images of the environment acquired within the terminal's visual perception range; acquiring the current position information of the terminal and the optimal transmit / receive beam pair corresponding to the terminal; using the scattering object distribution feature training set and the current position information of the terminal as sample inputs, and using the sequence number of the optimal transmit / receive beam pair corresponding to the terminal as the real sample label, training the beam alignment neural network model until convergence.

[0012] Secondly, the present invention also provides a beam alignment device based on terminal visual perception, comprising: a multi-view image acquisition module for acquiring multi-view images of the environment within the terminal's visual perception range; a scattering object distribution feature acquisition module for acquiring scattering object distribution features corresponding to each scattering object within the terminal's visual perception range based on the multi-view images; and an optimal transmit / receive beam determination module for determining the optimal transmit / receive beam pair corresponding to the target terminal based on the scattering object distribution features and the location information of the target terminal.

[0013] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the beam alignment method based on terminal visual perception as described above.

[0014] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the beam alignment method based on terminal visual perception as described above.

[0015] The beam alignment method based on terminal visual perception provided by this invention acquires multi-view images of the environment within the terminal's visual perception range, and obtains the distribution characteristics of scattering objects corresponding to each scattering object within the terminal's visual perception range based on the multi-view images. Then, based on the scattering object distribution characteristics and the target terminal's location information, the optimal transmit / receive beam pair corresponding to the target terminal is determined. This method characterizes the electromagnetic propagation characteristics of the channel between the terminal and the base station by using scattering object distribution characteristics and terminal location information, thereby determining the optimal transmit / receive beam pair corresponding to the target terminal. This overcomes the drawback of existing beam alignment methods having high time-frequency resource overhead, and can achieve rapid beam alignment with less pilot resource overhead. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the beam alignment method based on terminal visual perception provided by the present invention.

[0018] Figure 2 A schematic diagram of a terminal perception scenario for the beam alignment method based on terminal visual perception provided by the present invention;

[0019] Figure 3 A flowchart illustrating the design of the scattering object distribution characteristics for the beam alignment method based on terminal visual perception provided by this invention.

[0020] Figure 4 A schematic diagram illustrating the inference process of the beam alignment neural network model provided by this invention;

[0021] Figure 5 A schematic diagram of the beam alignment device based on terminal visual perception provided by the present invention;

[0022] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Figure 1 A schematic flowchart of the beam alignment method based on terminal visual perception provided by this invention is shown. Figure 1 As shown, the method includes:

[0025] S110: Acquire multi-view images of the environment within the terminal's visual perception range.

[0026] It should be noted that different coordinate systems are defined in this embodiment, specifically including the Base Coordinate System (BCS), the Mobile Coordinate System (MCS), and the Camera Coordinate System (CCS). The BCS has a coordinate axis X. B -Y B -Z B MCS has a coordinate axis X M -Y M -Z M Considering that the terminal is equipped with C cameras with different horizontal orientations, the coordinate system CCS of the i-th camera is defined with coordinate axis X. C,i -Y C,i -Z C,i , i = 1, 2, ..., C.

[0027] It is understandable that acquiring multi-view images of the environment within the visual perception range of the terminal involves using the C horizontally oriented RGB cameras configured on the aforementioned terminal to capture images of different viewing angles from multiple terminals, ensuring that the viewing angles of all cameras configured on each terminal can fully cover the 360-degree viewing angle range.

[0028] The terminal's visual perception range is the same as the visual perception range of the configured camera.

[0029] Figure 2 This diagram illustrates a terminal perception scenario using the beam alignment method based on terminal visual perception provided by the present invention. Figure 2 As shown, the base station coordinate system BCS has the base station BS as its origin, and its coordinate axes are represented by X. B -Y B -Z B Z B This refers to the vertical top-down view, which is not shown in the diagram.

[0030] The terminal coordinate system MCS takes the vehicle-mounted terminal MS as its origin, and its coordinate axis is X. M -Y M -Z M Z M This refers to the vertical top-down view, which is not shown in the diagram. The vehicle-mounted terminal is equipped with multiple RGB cameras. The camera coordinate system (CCS) has the i-th camera as its origin, and its coordinate axis is X. C,i -Y C,i -Z C,i Similarly, Z C,i This refers to the vertical top-down view, which is not shown in the diagram.

[0031] according to Figure 2 It can also be seen that the azimuth angle of the i-th camera in the terminal coordinate system BCS is θ. i M .

[0032] S120: Based on multi-view images, obtain the scattering object distribution characteristics corresponding to scattering objects within the terminal's visual perception range.

[0033] It is understandable that the multi-view images obtained in step S110 include various scattering objects, including the user. Based on this, target detection algorithms, segmentation algorithms, or tracking algorithms are used to extract the scattering object distribution features corresponding to various scattering objects around the terminal from each multi-view image.

[0034] The distribution characteristics of scattering objects include, but are not limited to, the relative position information, size information, and attitude information of various scattering objects.

[0035] Relative position information refers to the position information of various scattering objects relative to multiple coordinate systems. In this embodiment, the relative position information of various scattering objects is the position information in the base station coordinate system, which can be obtained by converting the position information of various scattering objects in the camera coordinate system; size information refers to the length, width, height, and shape information of the corresponding scattering object; attitude information refers to the azimuth information.

[0036] S130, based on the distribution characteristics of scattering objects and the location information of the target terminal, determines the optimal transmit / receive beam pair corresponding to the target terminal.

[0037] It is understandable that, based on the distribution characteristics of each scattering object within the visual perception range of the terminal obtained in step S120 above, the optimal transmit / receive beam pair corresponding to the target terminal can be determined based on the distribution characteristics of the scattering objects and the location information of the target terminal.

[0038] Preferably, this step can be implemented by a pre-trained beam alignment neural network model, specifically, using the partial features of the scattering object and the position information of the target terminal as input, and the optimal transmit / receive beam pair corresponding to the target terminal as output.

[0039] The location information of the target terminal can be obtained through devices such as GPS.

[0040] It should be noted that the distribution characteristics of scattering objects and the location information of the target terminal can jointly characterize the electromagnetic propagation characteristics of the channel between the target terminal and the base station, thereby characterizing the optimal transmit and receive beam pair and achieving rapid beam alignment.

[0041] In this embodiment, multi-view images of the environment within the terminal's visual perception range are acquired, and the distribution characteristics of scattering objects corresponding to each scattering object within the terminal's visual perception range are obtained based on these multi-view images. Then, based on the scattering object distribution characteristics and the target terminal's location information, the optimal transmit / receive beam pair corresponding to the target terminal is determined. This method characterizes the electromagnetic propagation characteristics of the channel between the terminal and the base station by using scattering object distribution characteristics and terminal location information, thereby determining the optimal transmit / receive beam pair corresponding to the target terminal. This overcomes the drawback of existing beam alignment methods having high time-frequency resource overhead, and can achieve rapid beam alignment with less pilot resource overhead.

[0042] Based on the above embodiments, further, based on multi-view images, the distribution features of scattering objects corresponding to scattering objects within the terminal's visual perception range are obtained, including: extracting scattering object feature information corresponding to various types of scattering objects in multi-view images using a target detection algorithm; designing features based on the scattering object feature information to obtain scattering object distribution features; wherein, the scattering object feature information includes the relative position information, size information, and attitude information of each scattering object within the terminal's visual perception range in the camera coordinate system.

[0043] Understandably, this involves acquiring the distribution features of scattering objects within the terminal's visual perception range based on multi-view images. Specifically, firstly, using a 3D target detection algorithm, O can be detected from the image captured by the i-th camera. i There are scattering objects. Let the position coordinates and azimuth angle of the j-th object in the k-th image, in the i-th camera coordinate system (CCS), be respectively... and For ease of description, the j-th scattering object in the k-th image will be denoted as the (i,j)-th scattering object.

[0044] Based on the detection of scattering objects in multi-view images, further, the feature information of scattering objects corresponding to various types of scattering objects in multi-view images is extracted. The feature information of scattering objects includes, but is not limited to, the relative position information, length, width and height information, shape information and azimuth information of each scattering object.

[0045] Then, feature design is performed based on the extracted scattering object feature information to obtain the scattering object distribution features. Specifically, the relative position and attitude information of each scattering object in the camera coordinate system are converted into position and attitude information in the base station coordinate system; the coordinate axis plane where the base station coordinate system is located is divided into multiple grids of equal size; based on the scattering object feature information, the normalized maximum length, width, and height, as well as the average azimuth angle, of all scattering objects contained in a single grid within the base station coverage area are determined to obtain the four-dimensional feature vector corresponding to the grid; the four-dimensional feature vectors corresponding to all grids within the base station coverage area are concatenated to obtain the scattering object distribution features.

[0046] It is understandable that multi-view images are acquired by multiple cameras configured on the target terminal, and the position information of various scattering objects extracted from the multi-view images is the position information under the camera coordinate system (CCS), while the distribution characteristics of scattering objects are under the base station coordinate system (BCS). Therefore, it is necessary to first convert the relative position information of each scattering object under the camera coordinate system (CCS) into the position information under the base station coordinate system (BCS).

[0047] Assume the position coordinates and azimuth angle of the i-th camera in the terminal coordinate system MCS are respectively and The obtained position coordinates of the target terminal in the terminal coordinate system BCS are:

[0048] To transform the position coordinates and azimuth angles of various scattering objects in the camera coordinate system (CCS) to the base station coordinate system (BCS), we can first perform a transformation from the CCS to the terminal coordinate system (MCS). Specifically, through coordinate transformation, we can obtain the position coordinates and azimuth angles of the (i,j)th object in the terminal MCS.

[0049]

[0050]

[0051] in, Let be the azimuth angle of the i-th camera in the terminal coordinate system MCS. Let be the position coordinates of the j-th object in the k-th image, in the i-th camera coordinate system (CCS). Let be the position coordinates of the i-th camera in the terminal coordinate system MCS. Let be the azimuth angle of the i-th camera in the terminal coordinate system MCS.

[0052] Furthermore, the position coordinates of the (i,j)th object in the base station coordinate system BCS are:

[0053]

[0054] The azimuth angle of the (i,j)th object in the base station coordinate system BCS is:

[0055]

[0056] Then, the coordinate plane X where the base station coordinate system BCS is located... B -Y B Divide into multiple equal-sized grids, each with a length and width of L. G With W G .

[0057] Assuming the number of grids intersecting with the base station coverage area is G, the g-th grid can be represented as... and are the column number and row number of the g-th grid in all grids of the plane, respectively, and both are integers.

[0058] Then, the coverage area of ​​the base station is included in the g-th grid. All scattering objects in the set, the set of these scattering objects' indices, can be denoted as:

[0059]

[0060] According to v g From all the scattering objects, the length, width, and height of each scattering object can be extracted, thus obtaining v. g The maximum length l of all scattering objects in the middle max,g Width w max,g With height h max,g .

[0061] Let L be the maximum length, width, and height of all scattering objects within the base station's coverage area. max W max With H max Combined with v g The maximum length l of all scattering objects in the middle max,g Width w max,g With height h max,g We can get v g The normalized maximum length, width, and height corresponding to all scattering objects in the equation.

[0062] Among them, the normalized maximum length is Normalized maximum width is Normalized maximum height is

[0063] At the same time, according to v gThe azimuth information of each scattering object in the middle is used to calculate v. g The average azimuth angle corresponding to all scattering objects in the middle, specifically, the average azimuth angle is expressed as follows:

[0064]

[0065] Among them, Card(v g ) represents set v g The number of elements in the text.

[0066] Based on the normalized maximum length, width, and height obtained from the above calculations, as well as the average azimuth angle, the mesh can be obtained. The corresponding four-dimensional feature vector is represented as follows:

[0067]

[0068] Furthermore, the four-dimensional feature vectors corresponding to all grids within the coverage area of ​​the base station are stitched together. Specifically, the four-dimensional feature vectors of all G grids are stitched together row by row into a G×4 matrix F. This matrix F can describe the characteristics of the distribution of scattering objects in the scene within the visual perception range of the terminal, that is, the corresponding scattering object distribution characteristics.

[0069] It should be noted that, if a certain grid does not contain any scattering objects, the four-dimensional eigenvector corresponding to that grid is denoted as the zero vector.

[0070] Figure 3 A flowchart illustrating the design process of the scattering object distribution characteristics in the beam alignment method based on terminal visual perception provided by this invention is shown. Figure 3 As shown on the left, based on multi-view images acquired by multiple cameras configured on the vehicle terminal, such as front view, side view and rear view images, the feature information of various scattering objects corresponding to the scattering objects is extracted from the multi-view images.

[0071] like Figure 3 As shown on the right, feature design is performed based on the extracted scattering object feature information. Specifically, the position coordinates and azimuth angles of various scattering objects in the camera coordinate system (CCS) are first transformed to the base station coordinate system (BCS). The coordinate plane of the base station coordinate system is then divided into multiple grids of equal size, with the grid length and width being L. G With W G As can be seen from the image, different scattering objects are covered in multiple grids of equal size.

[0072] The specific process for obtaining the distribution characteristics of scattering objects has been detailed above and will not be elaborated upon here.

[0073] In this embodiment, by using a target detection algorithm to extract the scattering object feature information corresponding to various scattering objects in multi-view images, and by designing features based on the scattering object feature information, the corresponding scattering object distribution features can be obtained. Then, by using the scattering object distribution features and terminal location information to characterize the electromagnetic propagation characteristics of the channel between the terminal and the base station, the optimal transmit / receive beam pair corresponding to the target terminal can be determined. This overcomes the defect of existing beam alignment methods having high time and frequency resource overhead, and can achieve fast beam alignment with less pilot resource overhead.

[0074] Based on the above embodiments, further, based on the distribution characteristics of the scattering object and the location information of the target terminal, the optimal transmit / receive beam pair corresponding to the target terminal is determined, including: inputting the distribution characteristics of the scattering object and the location information of the target terminal into a pre-trained beam alignment neural network model to obtain the sequence number of the optimal transmit / receive beam pair.

[0075] Understandably, based on the distribution characteristics of scattering objects and the location information of the target terminal, the optimal transmit / receive beam pair corresponding to the target terminal can be determined. Specifically, the distribution characteristics of scattering objects and the location information of the target terminal can be used as inputs to a pre-trained beam alignment neural network model, which can then output the sequence number of the optimal transmit / receive beam pair corresponding to the target terminal.

[0076] It should be noted that the location information of the target terminal refers to the location information of the target terminal in the base station coordinate system. The beam alignment neural network model includes a first sub-neural network, a second sub-neural network, and a third sub-neural network. The first sub-neural network is used to process the distribution characteristics of the scattering object, the second sub-neural network is used to process the location information of the target terminal, and the third sub-neural network is used to process the fusion characteristics of the distribution characteristics of the scattering object and the location information of the target terminal.

[0077] The distribution characteristics of scattering objects and the location information of the target terminal are input into a pre-trained beam alignment neural network model to obtain the optimal transmit / receive beam pair. Specifically, the distribution characteristics of scattering objects are input into the first sub-neural network to obtain the first output feature; the location information of the target terminal in the base station coordinate system is input into the second sub-neural network to obtain the second output feature; the first output feature and the second output feature are fused to obtain the corresponding fused feature; the fused feature is input into the third sub-neural network to obtain the sequence number of the optimal transmit / receive beam pair.

[0078] It is understandable that the distribution characteristics F of the scattering object are compared with the position coordinates of the target terminal in the terminal coordinate system BCS. As input to the beam alignment neural network model, the beam alignment neural network model uses the first sub-neural network to extract features of the scattering object distribution characteristics to obtain the first output feature, and uses the second sub-neural network to extract features of the target terminal's position information to obtain the second output feature.

[0079] Then, the first output feature of the first sub-neural network is added and fused with the second output feature of the second sub-neural network, and the fused feature is input into the third sub-neural network to obtain the sequence number of the optimal transmit / receive beam pair, and thus obtain the corresponding optimal transmit / receive beam pair.

[0080] In one specific embodiment, the first, second, and third sub-neural networks all employ a fully connected residual network structure. Of course, other network structures can also be used for these sub-neural networks, and no specific limitations are imposed here.

[0081] Figure 4 A schematic diagram illustrating the inference process of the beam alignment neural network model provided by this invention is shown. Figure 4 As shown, the beam alignment neural network model includes two inputs: one is the distribution characteristics of the scattering object, and the other is the location information of the target terminal.

[0082] The first sub-neural network extracts features of the scattering object distribution characteristics, while the second sub-neural network extracts features of the target terminal's location information. The output features of the first and second sub-neural networks are added and fused together, and the fused features are then input into the third sub-neural network to output the sequence number of the optimal transmit / receive beam pair.

[0083] according to Figure 4 It can also be seen that the first sub-neural network, the second sub-neural network, and the third sub-neural network can use the same network structure.

[0084] It should be noted that before using the beam alignment neural network model for inference, it is necessary to train the beam alignment neural network model. Specifically, based on the multi-view images of the environment within the terminal's visual perception range, a training set of scattering object distribution features corresponding to scattering objects within the terminal's visual perception range is obtained; the terminal's current location information and the optimal transmit / receive beam pair corresponding to the terminal are obtained; the beam alignment neural network model is trained until convergence using the scattering object distribution feature training set and the terminal's current location information as sample inputs, and the optimal transmit / receive beam pair number corresponding to the terminal as the real sample label.

[0085] Understandably, at multiple locations within the base station's coverage area, multiple cameras equipped on the terminal collect multi-view images and record the terminal's current location information, which can be obtained via GPS. At the same time, the optimal transmit / receive beam pair at that location is measured.

[0086] Based on the multi-view images collected above, the distribution features of scattering objects corresponding to scattering objects within the visual perception range of the terminal are obtained, and a corresponding training set of scattering object distribution features is formed.

[0087] Then, using the training set of scattering object distribution features and the current location information of the acquired terminal, the beam alignment neural network model is trained. Specifically, the scattering object distribution features and the current location information of the terminal are used as sample inputs, and the sequence number of the optimal transmit / receive beam pair corresponding to the terminal is used as the real sample label. The beam alignment neural network model is trained until convergence.

[0088] In this embodiment, by inputting the distribution characteristics of scattering objects and the location information of the target terminal into a pre-trained beam alignment neural network model, the index of the optimal transmit / receive beam pair is obtained, thus yielding the corresponding optimal transmit / receive beam pair. This method characterizes the electromagnetic propagation characteristics of the channel between the terminal and the base station by using the distribution characteristics of scattering objects and the terminal location information, thereby determining the optimal transmit / receive beam pair corresponding to the target terminal. This overcomes the drawback of existing beam alignment methods having high time-frequency resource overhead, and can achieve rapid beam alignment with less pilot resource overhead.

[0089] Figure 5 A schematic diagram of the beam alignment device based on terminal visual perception provided by the present invention is shown. Figure 5 As shown, the device includes: a multi-view image acquisition module 510, used to acquire multi-view images of the environment within the visual perception range of the terminal; a scattering object distribution feature acquisition module 520, used to acquire the scattering object distribution features corresponding to scattering objects within the visual perception range of the terminal based on the multi-view images; and an optimal transmit / receive beam determination module 530, used to determine the optimal transmit / receive beam pair corresponding to the target terminal based on the scattering object distribution features and the location information of the target terminal.

[0090] In this embodiment, the multi-view image acquisition module 510 acquires multi-view images of the environment within the terminal's visual perception range. The scattering object distribution feature acquisition module 520 acquires the scattering object distribution features corresponding to each scattering object within the terminal's visual perception range based on the multi-view images. Then, the optimal transmit / receive beam determination module 530 determines the optimal transmit / receive beam pair corresponding to the target terminal based on the scattering object distribution features and the target terminal's location information. This device characterizes the electromagnetic propagation characteristics of the channel between the terminal and the base station by using scattering object distribution features and terminal location information, thereby determining the optimal transmit / receive beam pair corresponding to the target terminal. This overcomes the drawback of existing beam alignment methods having high time-frequency resource overhead, enabling rapid beam alignment with less pilot resource overhead.

[0091] The beam alignment device based on terminal visual perception provided in this embodiment of the invention can be referred to in correspondence with the beam alignment method based on terminal visual perception described above, and will not be repeated here.

[0092] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640. The processor 610, communication interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a beam alignment method based on terminal visual perception. This method includes: acquiring multi-view images of the environment within the terminal's visual perception range; acquiring the scattering object distribution characteristics corresponding to each scattering object within the terminal's visual perception range based on the multi-view images; and determining the optimal transmit / receive beam pair corresponding to the target terminal based on the scattering object distribution characteristics and the target terminal's location information.

[0093] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the beam alignment method based on terminal visual perception provided by the above methods. The method includes: acquiring multi-view images of the environment within the terminal's visual perception range; acquiring the scattering object distribution characteristics corresponding to each scattering object within the terminal's visual perception range based on the multi-view images; and determining the optimal transmit / receive beam pair corresponding to the target terminal based on the scattering object distribution characteristics and the location information of the target terminal.

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A beam alignment method based on terminal visual perception, characterized in that, include: Acquire multi-view images of the environment within the terminal's visual perception range; Based on the multi-view images, the distribution features of scattering objects corresponding to each scattering object within the terminal's visual perception range are obtained, including: extracting scattering object feature information corresponding to various scattering objects in the multi-view images using a target detection algorithm; and designing features based on the scattering object feature information to obtain the scattering object distribution features. The distribution characteristics of the scattering object and the location information of the target terminal are simultaneously input into a pre-trained beam alignment neural network model, and the optimal transmit / receive beam pair corresponding to the target terminal is output by the beam alignment neural network model.

2. The beam alignment method based on terminal visual perception according to claim 1, characterized in that, The scattering object feature information includes the relative position information, size information, and attitude information of each scattering object within the terminal's visual perception range in the camera coordinate system.

3. The beam alignment method based on terminal visual perception according to claim 2, characterized in that, The step of designing features based on the scattering object's feature information to obtain the scattering object's distribution features includes: The relative position and attitude information of each scattering object in the camera coordinate system are converted into position and attitude information in the base station coordinate system; The coordinate plane containing the base station coordinate system is divided into multiple grids of equal size; Based on the scattering object feature information, the normalized maximum length, width, and height, as well as the average azimuth angle, of all scattering objects contained in a single grid within the base station coverage area are determined, and the four-dimensional feature vector corresponding to the grid is obtained. By concatenating the four-dimensional feature vectors corresponding to all grids within the coverage area of ​​the base station, the distribution characteristics of the scattering object are obtained. In cases where the grid does not contain any scattering objects, the four-dimensional eigenvector corresponding to the grid is a zero vector.

4. The beam alignment method based on terminal visual perception according to claim 1, characterized in that, The location information of the target terminal is the location information of the target terminal in the base station coordinate system. The beam alignment neural network model includes a first sub-neural network, a second sub-neural network, and a third sub-neural network. The first sub-neural network is used to process the distribution characteristics of the scattering object, the second sub-neural network is used to process the location information of the target terminal, and the third sub-neural network is used to process the fusion characteristics of the distribution characteristics of the scattering object and the location information of the target terminal.

5. The beam alignment method based on terminal visual perception according to claim 4, characterized in that, The step of inputting the distribution characteristics of the scattering object and the location information of the target terminal into a pre-trained beam alignment neural network model to obtain the sequence number of the optimal transmit / receive beam pair includes: The distribution characteristics of the scattering object are input into the first sub-neural network to obtain the first output feature; The position information of the target terminal in the base station coordinate system is input into the second sub-neural network to obtain the second output feature; The first output feature and the second output feature are fused together to obtain the corresponding fused feature. The fused features are input into the third sub-neural network to obtain the sequence number of the optimal transmit / receive beam pair.

6. The beam alignment method based on terminal visual perception according to claim 5, characterized in that, The first sub-neural network, the second sub-neural network, and the third sub-neural network all adopt a fully connected residual network structure.

7. The beam alignment method based on terminal visual perception according to any one of claims 1-6, characterized in that, Training the beam alignment neural network model specifically includes: Based on the multi-view images of the environment within the visual perception range of the terminal, a training set of scattering object distribution features corresponding to scattering objects within the visual perception range of the terminal is obtained. Obtain the current location information of the terminal, as well as the optimal transmit / receive beam pair corresponding to the terminal; The beam alignment neural network model is trained until convergence using the training set of the scattering object distribution features and the current location information of the terminal as sample inputs, and the sequence number of the optimal transmit / receive beam pair corresponding to the terminal as the real sample label.

8. A beam alignment device based on terminal visual perception, characterized in that, include: The multi-view image acquisition module is used to acquire multi-view images of the environment within the visual perception range of the terminal. The scattering object distribution feature acquisition module is used to acquire the scattering object distribution features corresponding to each scattering object within the terminal's visual perception range based on the multi-view image, including: extracting scattering object feature information corresponding to various types of scattering objects in the multi-view image using a target detection algorithm; and designing features based on the scattering object feature information to obtain the scattering object distribution features. The distribution characteristics of the scattering object and the location information of the target terminal are simultaneously input into a pre-trained beam alignment neural network model, and the optimal transmit / receive beam pair corresponding to the target terminal is output by the beam alignment neural network model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the beam alignment method based on terminal visual perception as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the beam alignment method based on terminal visual perception as described in any one of claims 1 to 7.

Images