Information sensing method based on electromagnetic waves, and device and storage medium

Abstract

Provided in the present application are an information sensing method based on electromagnetic waves, and a device and a storage medium. The method is applied to a terminal device, the terminal device is configured with a transmitting antenna array and a receiving antenna array of electromagnetic waves, and the electromagnetic waves are millimeter waves or terahertz waves. The method comprises: transmitting a sensing detection sequence signal on the basis of a transmitting antenna array; on the basis of a receiving antenna array, receiving, in segments, a sensing echo sequence signal which is formed by means of reflection after the sensing detection sequence signal reaches a target object; performing division to obtain a plurality of virtual imaging areas on the basis of the sensing echo sequence signal, which is received in segments; and on the basis of signal parameters, corresponding to pixel grid units in each virtual imaging area, of the sensing echo sequence signal, which is received in segments, generating a near-field image sequence of the target object relative to each virtual imaging area.

Description

Information perception method, device and storage medium based on electromagnetic waves

[0001] Cross-references

[0002] This invention claims priority to a Chinese patent application filed with the Patent Office of China on June 14, 2023, with application number 202310705921.7 and title “Information perception method, device and storage medium based on electromagnetic waves”. The entire contents of this application are incorporated herein by reference. Technical Field

[0003] The present application relates to the field of terminal application technology, and in particular to an information perception method, device and storage medium based on electromagnetic waves. Background Art

[0004] With the development of artificial intelligence technology, more and more artificial intelligence-based applications are being implemented on terminal devices, such as face recognition, gesture recognition, and liveness detection.

[0005] The vast majority of AI applications on terminal devices rely on user information perception. Currently, this is primarily accomplished through the terminal device's camera and under-screen fingerprint sensor, which can affect the screen display to varying degrees.

[0006] Therefore, how to enable the terminal device to perceive the target object information without affecting the screen display is the technical problem to be solved by this application.

[0007] Summary of the Invention

[0008] The purpose of this application is to provide an information perception method, device and storage medium based on electromagnetic waves, which can enable a terminal device to perceive target object information without affecting the screen display.

[0009] In order to achieve the above objectives, the embodiments of the present application are implemented as follows:

[0010] In a first aspect, an electromagnetic wave-based information perception method is provided, which is applied to a terminal device, wherein the terminal device is configured with an electromagnetic wave transmitting antenna array and a receiving antenna array, and the electromagnetic wave is a millimeter wave or a terahertz wave. The method includes: transmitting a perception detection sequence signal based on the transmitting antenna array; receiving a perception echo sequence signal formed by reflection of the perception detection sequence signal after reaching a target object based on segmented reception by the receiving antenna array; dividing a plurality of virtual imaging areas based on the segmented received perception echo sequence signal, wherein each virtual imaging area is composed of a pixel grid; generating a near-field image sequence of the target object relative to each virtual imaging area based on signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received perception echo sequence signal.

[0011] In a second aspect, a terminal device is provided, comprising a processor and a transmitting antenna array and a receiving antenna array configured with electromagnetic waves, wherein the electromagnetic waves are millimeter waves or terahertz waves, and the transmitting antenna array and the receiving antenna array are arranged in a non-operable area of ​​the screen of the terminal device, wherein: the transmitting antenna array is used to transmit a perception detection sequence signal; the receiving antenna array is used to segmentally receive a perception echo sequence signal formed by reflection of the perception detection sequence signal after reaching the target object; the processor is used to divide the plurality of virtual imaging areas based on the segmented received perception echo sequence signal, each of the plurality of virtual imaging areas being composed of a pixel grid; and, based on the signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received perception echo sequence signal, a near-field image sequence of the target object relative to each virtual imaging area is generated.

[0012] In a third aspect, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is used by the processor to execute the method of the first aspect.

[0013] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and wherein the computer program implements the method of the first aspect when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0015] FIG1 is a flow chart of an electromagnetic wave-based information perception method according to an embodiment of the present application.

[0016] FIG2 is a schematic diagram of a first deployment of an electromagnetic wave antenna array of a terminal device according to an embodiment of the present application.

[0017] FIG3 is a schematic diagram of a second deployment of the electromagnetic wave antenna array of the terminal device in an embodiment of the present application.

[0018] FIG4 is a schematic diagram of dividing a virtual imaging area in an information perception method according to an embodiment of the present application.

[0019] FIG5 is a schematic diagram of a third deployment of the electromagnetic wave antenna array of the terminal device according to an embodiment of the present application.

[0020] FIG6 is a schematic diagram of a fourth deployment of the electromagnetic wave antenna array of the terminal device according to an embodiment of the present application.

[0021] FIG7 is a schematic structural diagram of a user information sensing device based on electromagnetic waves according to an embodiment of the present application.

[0022] FIG8 is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] As mentioned above, with the development of artificial intelligence technology, more and more artificial intelligence-based applications are being implemented on terminal devices, such as face recognition, gesture recognition, and liveness detection.

[0024] The vast majority of AI applications on mobile devices rely on user information perception. Currently, this is primarily accomplished through the device's camera and under-screen fingerprint sensor, which can affect the screen display to varying degrees. For example, when the camera is activated for face recognition, the screen will enter the shooting interface. Alternatively, when a user places their finger on the screen for fingerprint recognition, a portion of the screen display area will be obscured.

[0025] In view of this, the present application proposes a new user information perception solution for terminal devices. Specifically, the present application deploys an antenna array (including a transmitting antenna array and a receiving antenna array) of high-frequency electromagnetic waves such as millimeter waves and terahertz in the non-operable area of ​​the terminal device screen. The target object (such as the user) can be scanned by the electromagnetic wave antenna array and then two-dimensional / or three-dimensional modeling is performed, thereby modeling the near-field image of the target object to perform related artificial intelligence applications such as gesture recognition, face recognition and liveness detection. The entire process does not require the user to start the camera or touch the fingerprint sensor of the screen, thereby reducing the impact on the screen display to a certain extent.

[0026] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this specification.

[0027] On the one hand, an embodiment of the present application provides an electromagnetic wave-based information perception method, which is applied to a terminal device, wherein the terminal device is equipped with an electromagnetic wave transmitting antenna array and a receiving antenna array. Figure 1 is a flow chart of the information perception method, which includes the following steps:

[0028] Step S102: transmitting a sensing detection sequence signal based on a transmitting antenna array.

[0029] It should be understood that the sensing detection sequence signal is composed of electromagnetic wave signals in a continuous time sequence. For example, if the time sequence granularity is seconds, a sensing detection sequence signal with a length of 10 seconds is composed of the electromagnetic wave signal from the 1st second to the 10th second.

[0030] In one embodiment, the transmitting antenna array can be set in the non-operable area (Active Area, AA) of the screen of the terminal device, and / or the gap area between the screen and the frame, so as not to affect the normal display of the screen.

[0031] In an exemplary embodiment, in order to reduce hardware costs, the transmitting antenna array can be reused as the communication antenna of the terminal device, thereby achieving the effect of synaesthesia (fusion of communication technology and perception technology).

[0032] Step S104: receiving, based on the receiving antenna array, segmented reception of the sensing detection sequence signal, which is formed by the reflection of the sensing detection sequence signal reaching the target object.

[0033] It should be understood that the perceived echo sequence signal is composed of echo signals in a continuous time sequence. For example, with seconds as the time sequence granularity, a perceived echo sequence signal of 5 seconds in length is composed of the echo signal from the 1st second to the 5th second.

[0034] In one embodiment, the receiving antenna array is arranged in a non-AA area of ​​the screen of the terminal device, and / or a gap area between the screen and the frame, so as not to affect the normal display of the screen.

[0035] In an exemplary embodiment, in order to reduce hardware costs, the receiving antenna array can be reused as the communication antenna of the terminal device, thereby achieving a synaesthesia effect.

[0036] Step S106 : dividing the segmented received sensed echo sequence signals into multiple virtual imaging areas, wherein each virtual imaging area is composed of a pixel grid.

[0037] It should be understood that the virtual imaging area described herein refers to an imaging area independent of the terminal device, virtually demarcated based on the deployment positions of the transmitting and receiving antenna arrays, as well as the electromagnetic wave transmission and reception patterns, rather than the display area of ​​the screen. Each virtual imaging area is divided into a grid of pixels, referred to herein as a pixel grid.

[0038] The division of the virtual imaging area depends on the deployment positions of the transmitting antenna array and the receiving antenna array, as well as the transmission and reception modes of electromagnetic waves, which are not specifically limited in this article.

[0039] In an exemplary embodiment:

[0040] There are no less than two transmitting antenna arrays and receiving antenna arrays. Correspondingly, the terminal device is configured with a first mode, a second mode, and a third mode for transmitting and receiving electromagnetic waves according to the deployed transmitting antenna arrays and receiving antenna arrays.

[0041] In the first mode, each transmitting antenna array transmits a sensing detection sequence signal in a time-sequential manner, and each receiving antenna array receives the sensing echo sequence signal corresponding to the sensing detection sequence signal of each transmitting antenna array transmitted in a time-sequential manner; for example, the first transmitting antenna array transmits the sensing detection sequence signal first, and after the first transmitting antenna array completes transmission, the second transmitting antenna array completes transmission of the sensing detection sequence signal, ..., in this process, the sensing echo sequence signals received by each receiving antenna array originating from different transmitting antenna arrays are also time-sequentially connected. In the second mode, each transmitting antenna array transmits a sensing detection sequence signal independently, and each receiving antenna array receives the sensing echo sequence signal originating from each transmitting antenna array respectively. That is, under the same timing, each receiving antenna array can receive sensing echo sequence signals originating from different transmitting antenna arrays.

[0042] In the third mode, the transmit antenna arrays and receive antenna arrays are grouped together. Each transmit antenna array independently transmits a sensing detection sequence signal, and each receive antenna array receives the sensing echo sequence signal originating from the transmit antenna array in its group. For example, if the first transmit antenna array and the first receive antenna array are grouped together, the first receive antenna array will only receive the sensing echo sequence signal originating from the first transmit antenna array.

[0043] For ease of understanding, the first transmitting antenna array N1 and the second transmitting antenna array N2, as well as the first receiving antenna array M1 and the second receiving antenna array M2 are taken as examples for introduction.

[0044] In the first mode, this embodiment divides the virtual imaging area by the perception echo sequence signal corresponding to the perception detection sequence signal N (N is composed of the N1 perception detection sequence signal and the N2 perception detection sequence signal that are connected in time sequence) received by the first receiving antenna array M1 and the perception echo sequence signal corresponding to the perception detection sequence signal that is connected in time sequence received by the second receiving antenna array M2, that is, obtaining two virtual imaging areas N*M1 and N*M2.

[0045] In the second mode, this embodiment divides the virtual imaging area by the perception echo sequence signal of the first transmitting antenna array N1 received by the first receiving antenna array M1, the perception echo sequence signal originating from the second transmitting antenna array N2 received by the first receiving antenna array M1, the perception echo sequence signal originating from the first transmitting antenna array N1 received by the second receiving antenna array M2, and the perception echo sequence signal originating from the second transmitting antenna array N2 received by the second receiving antenna array M2, that is, obtaining four virtual imaging areas of N1*M1, N1*M2, N2*M1 and N2*M2.

[0046] In the third mode, this embodiment divides the virtual imaging area by the perception echo sequence signal originating from the first transmitting antenna array N1 and received by the first receiving antenna array M1 and the perception echo sequence signal originating from the second transmitting antenna array N2 and received by the second receiving antenna array M2, that is, obtaining two virtual imaging areas N1*M1 and N2*M2.

[0047] It should be understood that the divided virtual imaging area is used to present a near-field image sequence of a portion of the target object, that is, a near-field image sequence.

[0048] Step S108 : generating a near-field image sequence of the target object relative to each virtual imaging area based on signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received perception echo sequence signals.

[0049] In one embodiment, a heterodyne mixing signal of a pixel grid in each virtual imaging area can be first determined based on the near-field spherical wavefront of the segmented received echo signal sequence. Next, a virtual array compensation vector is determined based on the time delay of an array element in a receiving antenna array relative to the pixel grid in each virtual imaging area, and the heterodyne mixing signal of the pixel grid in each virtual imaging area is compensated based on the virtual array compensation vector. Thereafter, the heterodyne mixing signal of the pixel grid in each virtual imaging area after compensation is coherently superimposed to determine the signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received perceived echo sequence signal. The signal parameters include signal amplitude and signal phase.

[0050] After obtaining the signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received perception echo sequence signal, the near-field image sequence corresponding to each virtual imaging area can be generated based on the signal parameters of the pixel grid. As an example:

[0051] In one embodiment, the signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received perception echo sequence signal can be mapped to a spatial coordinate system (three-dimensional or two-dimensional) for image scanning modeling, thereby generating a near-field image sequence corresponding to each virtual imaging area.

[0052] In another embodiment, a near-field image sequence corresponding to each virtual imaging region can also be generated based on artificial intelligence. Specifically, a Fourier transform is performed on the signal parameters of the pixel grid in each virtual imaging region corresponding to the segmented received perceived echo sequence signals to obtain frequency domain feature data. The frequency domain feature data is then input into a convolutional neural network model, which predicts the near-field image sequence corresponding to each virtual imaging region based on the frequency domain feature data. The convolutional neural network model is trained based on the frequency domain feature data corresponding to the sample perceived echo sequence signals and the near-field image sequence labels annotated with the sample perceived echo sequence signals. During training, when the frequency domain feature data corresponding to the sample perceived echo sequence signals is input into the convolutional neural network model, the model provides a predicted value for the near-field image sequence. The model then calculates the error between the predicted value and the true value of the labeled near-field image sequence, and iteratively optimizes the parameters of the convolutional neural network model over multiple rounds to reduce the error. This allows the model's predicted value to approach the true value, essentially learning to convert the frequency domain feature data into a near-field image representation.

[0053] In one embodiment, artificial intelligence applications such as gesture recognition, face recognition, and liveness detection can be performed on the target object based on the near-field image sequence of the target object.

[0054] Taking face recognition as an example, a target image with standard clarity from a near-field image sequence of a three-dimensional target object is first fed into a Faster Region Convolutional Neural Network (FRCN) for face recognition to attempt to encode the face region in the target image (the FRCN can use the face region in the image as the region of interest for encoding). The resulting encoding is then fed into a classifier for face recognition to complete face recognition.

[0055] In one embodiment, a fast Fourier transform can also be performed on the segmented received perception echo sequence signals to obtain a frequency domain signal sequence that shows the relationship between the chest distance of the target object and the pulse peak; then, based on the pre-set quantitative relationship between the phase change and the chest displacement in the frequency domain signal sequence, the heartbeat signal and / or breathing signal of the target object can be determined, that is, the health information of the perceived user can be determined.

[0056] In another embodiment, the near-field image sequences corresponding to the multiple virtual imaging areas can be stitched together to obtain a stitched near-field image sequence of the target object; then, artificial intelligence applications such as gesture recognition, face recognition, and liveness detection can be performed on the target object based on the stitched near-field image sequence.

[0057] Specifically, the stitching process requires registering the near-field image sequences of multiple virtual imaging areas. Registration involves matching the spatial positions of the near-field image sequences of different images. Once registration is complete, the near-field image sequences of the multiple virtual imaging areas can be stitched together in the correct spatial positions.

[0058] In an exemplary embodiment, the registration method may include the following two methods:

[0059] Method 1: Registration based on translation vector;

[0060] That is, the near-field image sequence corresponding to at least part of the virtual imaging area is mapped to the frequency domain, and the translation vector relative to the frequency domain between the near-field image sequences corresponding to at least part of the virtual imaging area is calculated based on the cross-power spectral density function; then, the near-field image sequences corresponding to at least part of the virtual imaging area are aligned based on the translation vector.

[0061] Method 2: Registration based on common feature points;

[0062] That is, common feature points between near-field image sequences corresponding to at least part of the virtual imaging area are found in the time domain; then, the near-field image sequences corresponding to at least part of the virtual imaging area are registered with reference to the common feature points.

[0063] After the registration is completed, image stitching and fusion are required to stitch the near-field image sequences of multiple virtual imaging areas into a stitched near-field image of the target object. Optionally, multiple stitched near-field image sequences corresponding to multiple virtual imaging areas generated multiple times can be stitched to generate multiple stitched near-field image sequences of the target object.

[0064] Afterwards, artificial intelligence applications such as gesture recognition, face recognition, and liveness detection are performed on the target object based on the received near-field image.

[0065] Taking face recognition as an example, a near-field image sequence of a three-dimensional target object is first fed into a Faster Region Convolutional Neural Network (FRCN) for face recognition to attempt to encode the facial region in the near-field image sequence (the FRCN can use the facial region in the image as the region of interest for encoding). The resulting encoding is then fed into a classifier for face recognition to complete face recognition.

[0066] In one embodiment, a fast Fourier transform can be performed on the segmented received induced echo sequence signals to obtain a frequency domain signal sequence showing the relationship between the chest distance of the target object and the pulse peak; then, based on the quantitative relationship between the phase change and the chest displacement in the preset frequency domain signal sequence, the heartbeat signal and / or breathing signal of the target object can be determined, that is, the health information of the user can be sensed.

[0067] The following describes the information perception method of this embodiment in conjunction with application scenarios.

[0068] Application Scenario 1

[0069] In this application scenario, electromagnetic wave transmitting and receiving antenna arrays are deployed in the non-AA area on the front of the terminal device's screen. The transmitting antenna array uses a uniform linear array (ULA) and is specifically set up in the non-operable areas on the upper and lower halves of the screen and on both sides of the lower half. The receiving antenna array uses a uniform planar array (UPA) and is specifically set up in the non-operable areas at the top and bottom of the screen.

[0070] Referring to Figure 2, the transmitting antenna array specifically includes a first transmitting antenna array N1 (N1 is reused in this article to represent the number of array elements of the first transmitting antenna array) and a second transmitting antenna array N2 (N2 is reused in this article to represent the number of array elements of the second transmitting antenna array); the receiving antenna array specifically includes a first receiving antenna array M1 (M1 is reused in this article to represent the number of array elements of the first receiving antenna array) and a second receiving antenna array M2 (M2 is reused in this article to represent the number of array elements of the second receiving antenna array). In this case, the first transmitting antenna array N1 is set in the non-AA area on both sides of the upper half of the screen, the second transmitting antenna array N2 is set in the non-operable area on both sides of the lower half of the screen, the first receiving antenna array M1 is set in the non-AA area at the top of the screen, and the second receiving antenna array M2 is set in the non-operable area at the bottom of the screen.

[0071] In addition, as shown in Figure 3, if the area of ​​the non-AA area of ​​the screen is limited, some array elements of the transmitting antenna array (N1, N2) and the antenna array (M1, M2) can be deployed at the edge of the AA area to minimize the impact on the screen display while ensuring sufficient array elements.

[0072] Based on the above, the transmit and receive antenna arrays in this application scenario 1 utilize a multiple-in, multiple-out (MIMO) array configuration. For example, the horizontal array elements are spaced evenly apart at half a wavelength (or 1 / 4 wavelength, but spacing too small is not recommended to prevent significant mutual interference). The vertical array elements are spaced evenly apart at 0.5 wavelengths. This means that while the imaging resolution requirements can be met under normal conditions, the spacing between the vertical elements is appropriately increased to reduce power consumption.

[0073] Correspondingly, based on the deployment structure shown in FIG2 or FIG3 , the process of constructing a near-field image sequence includes:

[0074] Step S1: The transmitting antenna arrays on both sides of the screen transmit electromagnetic wave sequences, and the receiving antenna arrays at the top and bottom of the screen receive corresponding echo signal sequences in segments.

[0075] As mentioned above, there are three modes for transmitting and receiving electromagnetic waves:

[0076] In the first mode, a total of N (N=N1+N2) array elements on both sides of the vertical screen transmit electromagnetic wave signal sequences; correspondingly, the M1 array elements horizontally at the top of the screen first perform segmented reception; then, the M2 array elements horizontally at the bottom of the screen perform segmented reception of the echoes.

[0077] In the second mode, a total of N1 array elements in the left and right columns on both sides of the upper vertical part of the screen transmit an electromagnetic wave signal sequence, which is received by the M1 horizontal array elements at the top and M2 horizontal array elements at the bottom of the screen. At the same time, a total of N2 array elements in the left and right columns on both sides of the lower vertical part of the screen transmit an electromagnetic wave signal sequence, and the echoes are received in segments by the M1 horizontal array elements at the top and M2 horizontal array elements at the bottom of the screen.

[0078] In the third mode, the electromagnetic wave signal sequence of the left and right columns of the upper half of the screen on both sides vertically, totaling N1 array elements, is received in segments by the M1 horizontal array elements at the top of the screen. At the same time, the electromagnetic wave signal sequence of the left and right columns of the lower half of the screen on both sides vertically, totaling N2 array elements, is transmitted, and the echo is received in segments by the M2 horizontal array elements at the bottom of the screen.

[0079] Step S2: dividing the segmented echo information array formed by the segmented reception of the receiving array into a number of virtual imaging areas.

[0080] It should be noted that, considering the large distance between M1 and M2, this application scenario adopts a method of dividing multiple virtual imaging areas and then stitching them together to determine the near-field image sequence of the target object.

[0081] Based on the above principles, the three modes in step S1 are respectively divided into the following virtual imaging areas:

[0082] The first mode division obtains two virtual imaging areas, N*M1 and N*M2;

[0083] The second mode division results in four virtual imaging areas: N1*M1, N1*M2, N2*M1, and N2*M2;

[0084] The third mode division obtains two virtual imaging areas, N1*M1 and N2*M2.

[0085] Step S3: Each virtual imaging area is further divided into a number of pixel grid points, and the signal amplitude and phase value of each pixel grid point is obtained by heterodyne mixing.

[0086] Electromagnetic wave imaging relies on heterodyne mixing technology to measure complex signals containing amplitude and phase information. Specifically, an antenna transmits an electromagnetic wave signal, which is then scanned onto the imaging target. The receiving antenna then receives the echo signal reflected from the target. The amplitude and phase of the reflected signal are then determined by heterodyne mixing of the received echo signal and the transmitted signal.

[0087] Based on the virtual imaging area divided in step S2, the echo signals received by the receiving array within the area need to be mixed with the transmitted signals of the transmitting array to generate a heterodyne mixed signal, and then the amplitude and phase values ​​of the signal are extracted. This detailed process has been described above and will not be repeated here.

[0088] Step S4: traverse each virtual imaging area and complete image reconstruction of each virtual imaging area.

[0089] Based on the three modes of step S1, each virtual imaging sub-area is traversed, namely, phase mixing compensation and sidelobe suppression are performed on the two virtual imaging areas N*M1 and N*M2, the four virtual imaging areas N1*M1, N1*M2, N2*M1, and N2*M2, and the two virtual imaging areas N1*M1 and N2*M2. After traversal, the x-axis, y-axis, and z-axis values ​​of the imaging grid points are obtained. Then, the plane scanning results within a limited range after the selected array element reference point (the 2D image data x and y carry the scattering coefficient, with the scattering coefficient as the z-axis) are arranged layer by layer according to the 3D coordinate axis, or cylindrically scanned, to form a complete 3D coordinate point cloud data. Then, through feature extraction, image registration, stitching, and fusion, the 3D image reconstruction is completed. The 2D image reconstruction focuses only on the x and y axis processing, omitting the z axis processing.

[0090] Step S5: performing image stitching and synthesis on each virtual imaging area to form an overall target image reconstruction.

[0091] Corresponding to the first two modes of step S1, the first mode requires stitching and synthesizing the 3D reconstructed images of the two imaging areas N*M1 and N*M2, and the second mode requires stitching and synthesizing the 3D reconstructed images of the four imaging areas N1*M1, N1*M2, N2*M1, and N2*M2. This stitching and synthesis is possible because correlation points can be found in the distances between N*M1 and N*M2, between N1*M1 and N2*M1, and between N1*M2 and N2*M2. Overall, large-aperture virtual imaging of N*M array elements is ultimately achieved through N+M array elements. At the same time, the imaging resolution will be improved due to the higher the electromagnetic wave frequency band, the smaller the array element wavelength, and the greater the number of array elements per unit area.

[0092] Since the top and bottom of the screen are two independent receiving arrays, image stitching for the first two modes in step S1 is essential in step S5. Since the two receiving arrays M1 and M2 have a purely two-dimensional translation relationship, the images received by both are subjected to a fast Fourier transform based on the phase correlation method. The two images to be registered are transformed into the frequency domain, and then the translation vector between the two images is directly calculated from their cross-power spectrum, which enables rapid image registration. However, compared to the first mode's splicing of N*M1 and N*M2, the second mode's splicing of N1*M1, N1*M2, N2*M1, and N2*M2 is more complicated. N1*M1 and N2*M2 must be spliced ​​into N*M1, while N1*M2 and N2*M2 must be spliced ​​into N*M2 before N*M1 and N*M2 are spliced ​​together. (Alternatively, N1*M1 and N2*M1 can be spliced ​​into N1*M while N1*M1 and N2*M2 are spliced ​​into N2*M before N1*M and N2*M are spliced ​​together.) This means that the time saved in the area scan phase by the second mode does not translate into time saved in the subsequent image stitching phase. If the terminal device's screen area is too large, the phase difference between M1 and M2 in the near field using the first mode will be too large, making phase compensation difficult. In this case, the second mode will be necessary. The phase correlation method relies on the need for a large overlap ratio of the registered images, so it is also possible to first find the feature points (such as boundary points and inflection points) in the two images based on the time domain, and determine the correspondence between the feature points between the images. Then, this correspondence is used to find the transformation relationship between the two images for image registration and complete the stitching.

[0093] The third mode is suitable for processing two independent imaging regions because there is little phase correlation or feature correlation between N1*M1 and N2 / M2. For example, the N1*M1 virtual imaging region is suitable for gesture recognition, while the N2*M2 virtual imaging region is suitable for face recognition.

[0094] This application scenario realizes large-aperture virtual imaging of N*M array elements for N+M array elements, which can be compared and explained from the perspective of real aperture imaging.

[0095] Among them, the real aperture imaging process is: the echo signal is mixed to obtain a heterodyne mixing signal, and the heterodyne mixing signal is subjected to FFT (fast Fourier transform) to obtain the distance dimension information. The corresponding phase compensation vector is obtained by the time delay from different array elements to the same pixel grid point, and the compensation vector is used to compensate and sum the data on the same distance unit of each channel to obtain the pixel point signal, and the image reconstruction is completed by traversing all related pixel points. The premise of real aperture imaging is that the path difference between any two array elements cannot be greater than the distance resolution unit, otherwise it will cause different phase plane times from different array elements to the array, resulting in an aperture transition effect. The so-called aperture transition effect means that when the distance r from the target to the center of the array is less than or equal to 2L 2 / λ, the target is in the near-field imaging area of ​​the array. Under near-field conditions, the phase difference between array elements is jointly affected by the spatial angle and distance of the target relative to the array. The signal received by the array is no longer a plane wavefront, but a spherical wavefront, and a phase compensation vector is required.

[0096] To solve the aperture transition effect, it is necessary to obtain the corresponding phase compensation vector by delaying the array element at the same pixel grid point at each sampling moment, then compensate and sum the heterodyne mixing signals of different channels, and then perform FFT on the compensated signals to obtain the target distance dimension information.

[0097] On the basis of the above, if the application scenario has the problem of aperture transit effect, it can be processed accordingly in the same way as real aperture imaging to solve the aperture transit effect.

[0098] Similarly, compared to real-aperture imaging, where angular resolution is proportional to the array aperture size, if the front-facing antennas of a terminal device are arranged as under-screen antennas with half-wavelength equidistant planes based on Antenna-on-Display (AoD) technology, this would result in an excessive number of array elements and channels, posing a serious cost and power challenge to the terminal device. Therefore, N+M array elements can be used to achieve N*M element MIMO large-aperture array virtual imaging, thereby improving resolution while avoiding cost and power issues.

[0099] Figure 4 is a schematic diagram of determining the virtual imaging area in a MIMO array configuration. In Figure 4, MIMO employs a 2-transmitter, 6-receiver uniform linear array (bold crosses represent array elements), with a receiving element spacing of three wavelengths. In the far field, the distance difference between R1 and R2 and R7 and R8 is considered equal, resulting in an equivalent 1-transmitter, 12-receiver uniform virtual linear array. However, in the near field, the distance difference between the two is different, preventing the equivalent uniform virtual linear array. Therefore, a more accurate virtual array steering vector is determined based on the phase relationship between the two.

[0100] When the TX1 array element transmits, the receiving array element steering vector is:

[0101] Wherein, Δt2 to Δt6 represent the time delay difference of the 2nd to 6th array elements relative to the reference receiving array element, and include the angle information θ of the grid point in the imaging area relative to the array.

[0102] When the Tx2 array element transmits, the receiving array element steering vector is:

[0103] Where ΔT2 represents the delay difference between the second transmitting element and the reference transmitting element. Accordingly, the receiving steering vector array is: A = [a1, a2] T (3)

[0104] In the case of N electromagnetic wave array elements on both sides of the screen in the vertical direction, the corresponding receiving steering vector is: A=[a1,a2,...,a N ] T (4)

[0105] The FMCW (Frequency Modulated Continuous Wave) signal emitted by each electromagnetic wave array element on both sides of the screen in the vertical direction is:

[0106] in, is the snapshot time, rect(t) is the rectangular window function, fc is the carrier frequency, γ is the FM slope, and Tp is the FM period.

[0107] According to the top and bottom area partition imaging technology proposed by the present invention, the transmitting array element a i Emitted, passing through the pixel grid point n of the virtual imaging area l , return to the top to receive element a rtop and bottom receiving element a rbot The signals can be expressed as:

[0108] in, is the echo delay, c is the speed of light, is the echo amplitude, which is affected by the target distance, target scattering cross-sectional area, reflection coefficient and single element pattern function.

[0109] are the distances between the transmitting and receiving array elements and the grid points respectively:

[0110] are pixel grid points n l , transmitting element a i a of the transmitting and receiving array elements rtop 、a rBot coordinate.

[0111] The heterodyne mixed signals obtained by mixing the echo signals received by the receiving array elements in the horizontal areas at the top and bottom of the screen with the transmitted signals are:

[0112] Among them, N nettop 、N netbot are the total number of pixel grids in the virtual imaging area at the top / bottom of the screen, and NTX is the total number of transmitting array elements. For the virtual imaging area, the phase values ​​to be compensated for the total number of transmitting array elements are:

[0113] Because the distance between M1 and M2 is too large, a rtop 、a rBot There is a large jump in the coordinates of the virtual imaging areas at the top and bottom of the screen. If compensation is performed at the beginning, the compensation may not be in place due to the large span of the top and bottom areas, resulting in an unsatisfactory original image. Therefore, separate compensation is used for the top and bottom areas to facilitate a better foundation for image processing in the later stage.

[0114] The echo signals of equations (4), (11) and (12) are used to perform near-field beamforming of the MIMO array. At this time, the received signals of the arrays at the top and bottom of the screen are:

[0115] The received signals of each array element are multiplied by the corresponding receiving channel weight and then summed. The received signals at the top and bottom of the screen to achieve beamforming are:

[0116] Among them, w top =[w1,w2,...,w M1 ] T 、w bot =[w1,w2,...,w M2 ] T are the weight vectors of the virtual imaging areas at the top and bottom of the screen respectively, M1 and M2 are the number of receiving array elements in the two virtual imaging areas, and A top (θ), A bot (θ) are the receiving steering vector matrices of the two virtual imaging areas, is the echo signal of the imaging grid points in the two virtual imaging areas, and n(t) is the noise. m =a(θ n ) can compensate for the arrival of the transmitted signal of each array element at the receiving near field area θ m The phase difference caused by the different paths in the direction makes the signal θ m The coherent addition in the near field region of the direction forms a direction θ m The transmitted beam in the direction of the received signal The signal is windowed to suppress side lobes and transformed into a frequency domain signal through Fourier transform (FFT). The frequency domain signal is then trained through a convolutional neural network (CNN) model to obtain a sub-near-field image.

[0117] Among them, the time domain signal of the virtual imaging area is transformed by FFT and trained through the convolutional neural network model to obtain the sub-near-field image. The specific process is: the frequency domain signal after FFT transformation is used as the input layer, and after several convolution layers and flattening, it is output through the fully connected layer. The convolution layer can be subdivided into convolution and pooling. The convolution process is to perform convolution through the convolution kernel and activation through the activation function. This process is essentially equivalent to feature extraction by the filter, which is equivalent to the convolutional neural network being able to "see" how the pixel group forms a straight line or curve. The deep neural network contains multiple layers. In the next layer, the convolutional neural network "sees" how the straight lines or curves form certain shapes, step by step, until a complete near-field image is formed. This process requires the use of frequency domain signals (frequency domain feature data) of multiple consecutive labeled near-field image sequence labels to train the convolutional neural network, which will not be repeated here.

[0118] With the convolutional neural network model, a near-field image of the target object can be constructed. The signals of multiple virtual imaging areas can be Fourier transformed and the convolutional neural network model can be used to construct multiple sub-near-field images, forming a near-field image sequence.

[0119] Taking a 60GHz carrier frequency as an example, with a wavelength of λ = 5mm, the transmit array elements in the terminal device are arranged with a horizontal spacing of 77.5mm and a vertical spacing of 15mm. For a screen length of 150mm, there are N = 20 (2 x 10) transmit array elements, located on the left and right sides of the screen. The receive array elements are spaced λ / 2 = 2.5mm horizontally and vertically, respectively, and arranged in a planar array at the top and bottom of the screen. The total number of receive elements at the top is M1 = 60 (2 x 30), and the total number of receive elements at the bottom is M2 = 90 (3 x 30). Therefore, the number of virtual MIMO array elements is N * M = 20 * 150. The total element area for the virtual full array, based on the top and bottom receivers, is 150mm * 150mm (vertical array length 150cm, horizontal array length 75mm * 2). Therefore, the array aperture L is 150mm, and the near-far field boundary is r ≤ 2L. 2 / λ = 45m. A reference array element is selected as the center point, and a planar scan (cylindrical scan is also possible) is performed on the top and bottom imaging pixel grids of the screen, all within a range of ±5m from the center point along the x, y, and z axes. Phase compensation is used to eliminate aperture transition effects. Three-dimensional near-field imaging is then performed using Fourier transform and convolutional neural networks, ultimately forming three-dimensional sub-near-field images of the two virtual imaging areas at the top and bottom of the screen.

[0120] Because the two 3D sub-near-field images capture signals transmitted by the transmitting elements on both sides of the screen and reflected by the target, the overlap is relatively large. The aforementioned phase correlation method allows for rapid image stitching. Feature extraction can also be used to identify matching point sets, and then image registration, stitching, and fusion are performed based on their geometric relationships. The resulting fused overall 3D image offers higher resolution and more comprehensive information for applications such as gesture and face recognition. To achieve more accurate angle recognition, multiple overall 3D images are required, forming multiple 3D image sequences. Using a faster region convolutional neural network (Faster Region CNN) on single or multiple 3D image sequences, regions of interest, such as faces and gestures, can be identified. A further convolutional neural network is then applied to these regions of interest for face and gesture recognition. Compared to implementing a convolutional neural network on a virtual imaging area to construct sub-near-field images, the Faster Region CNN adds a region candidate module. This is followed by a further convolutional neural network applied to the regions of interest. Image classifiers are then constructed using datasets such as CFAR-10 for refined face and gesture recognition.

[0121] Based on the 60GHz carrier frequency, the number of virtual array elements is generally too small, necessitating the addition of additional elements to the AA area around the screen. Furthermore, low-frequency electromagnetic waves require larger screens, such as tablets, to accommodate more elements. However, adding too many elements is generally not feasible, as power consumption must be balanced. Ultimately, electromagnetic wave virtual array imaging on terminal devices can be used for covert target detection and in low-light or no-light environments, providing a degree of privacy protection (preventing the misuse of profile pictures captured by explicit cameras, for example).

[0122] While gesture and facial recognition can be achieved through MIMO virtual array 3D imaging, breathing and heartbeat detection do not require 3D imaging. Instead, they can be achieved by using an electromagnetic array to detect phase changes in a FNCW frequency-modulated continuous wave (FMCW) signal within a specific range caused by minute target vibrations. This requirement can be met with a minimum of two electromagnetic antenna elements, one transmitting and one receiving. By performing Doppler FFT on the acquired velocity spectrum, the various components can be resolved. A 60GHz electromagnetic element can detect movements as small as less than 1mm. The reflected signal from a person's chest movement is phase-modulated. The electromagnetic element transmits multiple linear frequency-modulated pulses at predetermined intervals. For each pulse, a distance FFT is performed at the distance corresponding to the chest position, from which the phase and, therefore, the velocity are calculated. The heart rate of an adult ranges from 0.8Hz to 2Hz, and the respiratory rate from 0.1Hz to 0.5Hz. By selecting the velocity components of the heart and respiratory rates in the FFT and plotting their changes over time, an electrocardiogram and respiratory spectrum are obtained. The number of peaks per minute for each frequency represents the heart and respiratory rate.

[0123] The relationship between phase change and displacement of the heart or chest cavity is as follows:

[0124] Δφ b is the phase change, and ΔR is the displacement change of the heart or chest cavity. The heartbeat or breathing signal can be regarded as a sinusoidal signal b(t) with frequency and phase:

[0125] Where B is the scanning bandwidth, T is the scanning time period, and b(t) is essentially the result of heterodyne mixing of the FMCW wave s(t) emitted by the electromagnetic wave antenna transmitting element and the echo r(t) received by the receiving element after being reflected by the target, that is: b(t)≈s(t)·r(t) (23)

[0126] Measure small vibrations and perform FFT on b(t) at chest distance to obtain Δφ b , and then calculate the velocity component, and then separate the breathing and heartbeat signals at one time through bandpass filtering.

[0127] In addition to breathing and heartbeat perception, auxiliary positioning perception can also be performed based on the above technology.

[0128] The electromagnetic wave antenna array on the front of the screen is used for communication, but it is not necessary to use all electromagnetic wave imaging array elements. Only a small number of elements at the top, bottom, and sides of the screen are sufficient. Because electromagnetic wave antenna elements in a single direction are easily disturbed due to rapid propagation attenuation, placing elements around the four directions of the screen will achieve more ideal performance. More importantly, electromagnetic wave communication in this situation can effectively improve communication quality on the front of the screen (typically, electromagnetic wave antennas on the back of terminals are severely weakened due to the obstruction of the screen).

[0129] Application Scenario 2

[0130] As shown in Figure 5, in this application scenario 2, the transmitting antenna array includes a first transmitting antenna array N1 and a second transmitting antenna array N2; the receiving antenna array includes a first receiving antenna array M1 and a second receiving antenna array M2. The first transmitting antenna array N1 can be set only in the non-operable area on one side of the upper half of the screen (the right side in the figure), and the second transmitting antenna array N2 can be set only in the non-operable area on the opposite side of the lower half of the screen corresponding to N1 (the left side in the figure, placed on the opposite side due to the poor penetration of millimeter wave signals and easy to be blocked, and the millimeter wave antenna array is placed in different directions to improve the anti-interference capability), and the first transmitting antenna array N1 and the second transmitting antenna array N2 are both composed of only a single row of array elements; the first receiving antenna array M1 and the second receiving antenna array M2 are located in the non-operable areas at the top and bottom of the screen respectively, and are both composed of only a single row of array elements. The first receiving antenna array M1 receives the induction echo sequence signals corresponding to the induction detection sequence signals transmitted by the first transmitting antenna array N1, and the second receiving antenna array M2 receives the induction echo sequence signals corresponding to the induction detection sequence signals transmitted by the first transmitting antenna array N2. Based on the induction echo sequence signals received in the upper and lower segments of the screen, two virtual imaging areas corresponding to the upper and lower halves of the screen are formed.

[0131] A small number of vertical rows of micro-horn antenna elements can be placed in the gaps between the sides of the screen and the frame as the transmitting antenna array. The number of elements should be arranged at a full wavelength spacing of four on each side to meet communication requirements. AoD electromagnetic wave antenna elements can be used as replacements in the non-AA areas on the vertical sides of the screen.

[0132] The gaps between the top and bottom of the screen and the frame, or the non-AA areas nearby, can accommodate a larger number of micro-speakers or electromagnetic wave antenna elements for the receiving antenna array compared to the vertical layout. When the horizontal screen size is 80mm and the transmitting carrier frequency is 60GHz, 20 to 30 elements can be arranged at half-wavelength intervals at the top and bottom of the screen.

[0133] It can be seen that the number of array elements in the transmitting antenna array and the receiving antenna array at this time is far lower than that in application scenario 1, and does not meet the segmented reception imaging accuracy requirements when the terminal device is static. In other words, the number of array elements in the transmitting antenna array and the receiving antenna array in application scenario 2 is limited.

[0134] With further reference to FIG6 , the transmitting antenna array (N1, N2) and / or the receiving antenna array (M1, M2) include a plurality of miniature horn antennas whose main structures extend along the inner surface of the frame. FIG6 (A) and FIG4 (B) respectively provide schematic diagrams of the miniature horn antenna between the screen and the frame; FIG6 (C) provides a schematic diagram of the feed port of the miniature horn antenna being connected to the main circuit board of the terminal device via the package chip module board and the flexible printed circuit board (FPC). It can be seen that the miniature horn antenna in FIG6 is optically bonded to the glass cover in parallel with the liquid crystal display (LCD). It should be understood that the miniature horn antenna in this application scenario 2 extends along the inner surface of the frame, so the thickness of the frame will not be increased too much, that is, the miniature horn antenna is radiated on the screen toward the user without affecting the AA area of ​​the screen.

[0135] It should be noted that compared to application scenario 1, this application scenario 2 deploys a smaller number of electromagnetic wave array elements for imaging, which can be improved using existing virtual aperture compressed sensing technology to enhance imaging accuracy and range. By swinging the terminal device's scanning method, a large-scale virtual aperture antenna array with greater degrees of freedom is formed, approaching the imaging effect of a real aperture antenna with thousands of antenna elements. Because the trajectory of the user's hand-swinging terminal device is usually sparse and not completely regular, algorithms such as compressed sensing, tomography, and sparse aperture are required to process the sparsely sampled signal waveform to obtain millimeter-level high-resolution images.

[0136] Correspondingly, taking the deployment design shown in FIG5 as an example, the process of constructing a near-field image sequence includes:

[0137] Step S'1: The transmitting antenna arrays on both sides of the screen transmit electromagnetic wave sequences, and the receiving antenna arrays at the top and bottom of the screen receive corresponding echo signal sequences in segments.

[0138] Different from the first application scenario, the segmented reception here requires the user to manually swing the terminal device, such as in a Z-shape.

[0139] Step S'2: dividing the segmented echo information array formed by the segmented reception of the receiving array into a number of virtual imaging areas.

[0140] The initial division of the virtual imaging area is similar to the third mode of application scenario one, but the difference is that when the terminal device is swinging, multiple virtual imaging areas with partial overlap will be divided, so the overlapping parts will be used as a reference to stitch the near-field image sequence of the target object.

[0141] Step S'3: further divide each virtual imaging area into a number of pixel grid points, and obtain the signal amplitude and phase value of each pixel grid point by heterodyne mixing technology.

[0142] Similar to the first scenario, heterodyne mixing can be used to obtain the signal amplitude and phase values ​​of each pixel grid point. However, considering that the acquisition is dynamic, the processing speed of heterodyne mixing is relatively fast, requiring a more powerful processor.

[0143] Step S'4: traverse each virtual imaging area and complete image reconstruction of each virtual imaging area.

[0144] This scenario is similar to the first application scenario, but the traversal method is much more complex. After heterodyne mixing and sidelobe suppression, the sparsely sampled signal waveform must be processed through algorithms such as compressed sensing, tomography, and sparse aperture. Compressed sensing, tomography, and sparse aperture are primarily used to combat the instability of manual Z-shaped dynamic acquisition. To this end, the signal collected at each dynamic moment is first mapped onto a two-dimensional plane using compressed sensing for image analysis. The resolved two-dimensional image is then transversely accumulated to obtain a three-dimensional image, similar to medical MRI tomography.

[0145] Step S'5: performing image stitching and synthesis on each virtual imaging area to form an overall target image reconstruction.

[0146] As described in step S'1, although the third mode similar to the first application scenario is adopted to divide the virtual imaging area, since there will be repeated parts during the continuous dynamic acquisition process, the near-field image sequences of each virtual imaging area can be spliced ​​according to the repeated parts of each virtual imaging area to obtain a complete near-field image sequence of the target object.

[0147] The above is an exemplary introduction to the application of the information perception method of this embodiment. The characteristics of application scenario 1 and application scenario 2 are compared as follows:

[0148] In addition, considering the scenarios in which users typically use terminal devices, terminal communication and non-centralized functions such as breathing, heartbeat, and imaging perception overlap. Therefore, breathing, heartbeat detection, and imaging detection can be performed in a time-sharing manner. Similarly, electromagnetic wave communication can also be performed in a time-sharing manner with these perception functions, which relatively avoids the complexity of simultaneous communication and perception. Since perception, including imaging, breathing, heartbeat, and positioning, is transmitted and received between different electromagnetic wave arrays on the terminal screen, and communication is carried out through interaction between the terminal electromagnetic wave array and the base station, the former is mainly based on frequency modulated continuous wave (FMCW) signals, and the latter is mainly based on orthogonal frequency division multiplexing (OFDM) signals. Perception and communication use different signal forms in the same electromagnetic wave array element or array to work in a time-sharing integrated manner, so that both functions can be perfectly realized without interfering with each other.

[0149] Furthermore, because fewer array elements are needed for communication, breathing, and heartbeats, several elements within the imaging array can be dedicated to non-imaging purposes. The missing pixel grid points are then filled with interpolated values ​​from adjacent pixel grid points. In this case, different electromagnetic wave array elements are operating simultaneously for FMCW and OFDM signals. Due to the differences in signal mechanisms, mutual interference is minimal, and since the array elements are arranged together, there is no substantial impact on imaging.

[0150] Alternatively, based on the integrated communication and perception technology, when a certain mode of perception or communication continues to work, in order to avoid causing the other function to be unable to work for a long time, the electromagnetic wave antenna array can be used to assist with communication in the perception state and assist with perception in the communication state.

[0151] In addition, since OFDM has a low distance resolution capability due to its high peak-to-average ratio only at long distances, in the close-range application scenario in front of the terminal screen, the performance of OFDM and FMCW is similar, and OFDM can be used for perception. In this way, communication and perception can be carried out in the same signal form, so communication and perception can be carried out simultaneously. However, since communication is the interaction between the terminal electromagnetic wave array and the base station array, and perception is the interaction between the screen electromagnetic wave arrays, the electromagnetic wave receiving array simultaneously receives the far-field echo of the base station signal and the near-field echo of the screen electromagnetic wave transmitting array, and then separates the signals according to the time shift, Doppler frequency shift, and phase shift of the near-field and far-field echoes. The separated signals use the aforementioned perception method to achieve simultaneous communication and perception.

[0152] On the other hand, corresponding to the information perception method shown in Figure 1, an embodiment of the present application also provides a terminal device. Figure 7 is a schematic structural diagram of the terminal device, including: an electromagnetic wave transmitting antenna array and a receiving antenna array, and a processor. The transmitting antenna array and the receiving antenna array are set in the non-operable area of ​​the screen of the terminal device, and / or the gap between the screen and the frame.

[0153] The transmitting antenna array is used to transmit a sensing detection sequence signal.

[0154] The receiving antenna array is used to segmentally receive the sensing echo sequence signals formed by the sensing detection sequence signals being reflected from the target object.

[0155] The processor is configured to divide the plurality of virtual imaging areas into a plurality of virtual imaging areas based on the segmented received perception echo sequence signals, each of the plurality of virtual imaging areas being composed of a pixel grid; and to generate a near-field image sequence corresponding to each virtual imaging area based on signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received perception echo sequence signals; and to splice the near-field image sequences corresponding to the plurality of virtual imaging areas to obtain a target near-field image sequence corresponding to the target object.

[0156] Specifically, the transmitting antenna array of this embodiment includes a first transmitting antenna array N1 and a second transmitting antenna array N2; the receiving antenna array includes a first receiving antenna array M1 and a second receiving antenna array M2.

[0157] Referring to Figure 2 or Figure 3, the first transmitting antenna array N1 is set in the non-operable area on both sides of the upper half of the screen, and the second transmitting antenna array N2 is set in the non-operable area on both sides of the lower half of the screen; the first receiving antenna array M1 is set in the non-operable area at the top of the screen, and the second receiving antenna array M2 is set in the non-operable area at the bottom of the screen.

[0158] Considering energy consumption, the number of antenna array elements should not be too large. In one embodiment, the number of array elements in the first transmit antenna array N1 and the second transmit antenna array N2 in the column direction is less than or equal to 10; and the number of array elements in the first receive antenna array M1 and the second receive antenna array M2 in the row direction is less than or equal to 20.

[0159] In one embodiment, in order to further reduce the number of elements in the antenna array, as shown in Figure 5, the first transmitting antenna array N1 can be set only in the non-operable area on the right or left side of the upper half of the screen, and the second transmitting antenna array N2 can be set only in the non-operable area on the left or right side of the lower half of the screen, and the first transmitting antenna array N1 and the second transmitting antenna array N2 are both composed of only a single row of elements; the first receiving antenna array M1 and the second receiving antenna array M2 are respectively located in the non-operable areas at the top and bottom of the screen, and are both composed of only a single row of elements. This layout corresponds to the second application scenario. The first receiving antenna array M1 receives the first transmitting antenna array N1. In an exemplary embodiment, the transmitting antenna array (N1, N2) and the receiving antenna array (M1, M2) can be designed using under-screen antenna technology, or, as shown in Figure 6, at least part of the array elements of the transmitting antenna array (N1, N2) and the receiving antenna array (M1, M2) are set in the gap between the frame of the terminal device and the screen, and the transmitting antenna array (N1, N2) and the receiving antenna array (M1 and M2) both include a plurality of miniature horn antennas whose main structures extend along the inner surface of the frame.

[0160] In one embodiment, the number of array elements in the transmitting antenna arrays (N1, N2) and receiving antenna arrays (M1, M2) is determined by the wavelength of the electromagnetic wave and the width of the screen. For example, if the electromagnetic wave frequency is 81 GHz and the screen width is 80 mm, if under-screen antenna technology is used, the number of array elements at half-wavelength spacing in the first receiving antenna array M1 and the second receiving antenna array M2 can be 30, and the number of array elements in the first transmitting antenna array N1 and the second transmitting antenna array N2 can be 4. Alternatively, if micro-horn antennas are used, the number of array elements at half-wavelength spacing in the first receiving antenna array M1 and the second receiving antenna array M2 can be less than or equal to 16, and the number of array elements in the first transmitting antenna array N1 and the second transmitting antenna array N2 can also be 4.

[0161] In one embodiment, based on the fact that both the millimeter wave sequence signal and the echo sequence signal are used to perceive user information, the transmitting antenna array and the receiving antenna array of the terminal device can also be multiplexed as communication antennas to achieve synaesthesia integration.

[0162] For example, the transmitting antenna array and the receiving antenna array are time-division multiplexed as the communication antennas of the terminal device; wherein: the transmitting antenna array adopts the orthogonal frequency division multiplexing (OFDM) mode to send communication signals for a period of time, and adopts the frequency modulated continuous wave (FMCW) mode to send the sensing detection sequence signal for another period of time; the receiving antenna array adopts the orthogonal frequency division multiplexing mode to receive the communication signal for a period of time, and adopts the frequency modulated continuous wave mode to receive the sensing echo sequence signal for another period of time.

[0163] For another example, the transmitting antenna array and the receiving antenna array are multiplexed as the communication antennas of the terminal device at the same time; the transmitting antenna array uses orthogonal frequency division multiplexing mode to send communication signals, and uses frequency modulated continuous wave mode to send perception detection sequence signals; the receiving antenna array uses orthogonal frequency division multiplexing mode to receive communication signals, and uses frequency modulated continuous wave mode to receive perception echo sequence signals.

[0164] For another example, the transmitting antenna array uses an orthogonal frequency division multiplexing mode to send communication signals and sensing detection sequence signals, and the receiving antenna array also uses an orthogonal frequency division multiplexing mode to receive communication signals and sensing echo sequence signals.

[0165] In particular, when the terminal size is small, in order to achieve ideal imaging and other perception purposes, regardless of whether it corresponds to the application scenario one or the application scenario two, or the antenna array is reused as a communication antenna to achieve integrated communication perception, due to the relatively long wavelength of millimeter waves, the size of a single antenna array element arranged in the non-operating area of ​​the screen will be large, resulting in an insufficient number of antenna arrays composed of multiple antenna array elements, which will ultimately affect the perceived imaging quality or even make effective imaging impossible. At this time, the millimeter wave signal can be replaced by a terahertz signal with a smaller wavelength to meet the requirements.

[0166] It should be understood that the terminal device of this embodiment can serve as the execution subject of the method shown in Figure 1, and thus can implement the steps and functions in the method shown in Figure 1, which will not be repeated here.

[0167] FIG8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Referring to FIG8 , at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage, etc. Of course, the electronic device may also include hardware required for other services.

[0168] The processor, network interface, and memory can be interconnected via an internal bus, such as an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. These buses can be classified as address buses, data buses, and control buses. For ease of illustration, FIG8 shows only one bidirectional arrow, but this does not imply that there is only one bus or only one type of bus.

[0169] The memory is used to store the computer program. Specifically, the computer program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides the computer program to the processor.

[0170] Optionally, the processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming the terminal device shown in Figure 7 above at the logical level. Correspondingly, the processor executes the program stored in the memory and is specifically used to perform the following operations:

[0171] A sensing detection sequence signal is transmitted based on the transmitting antenna array.

[0172] The receiving antenna array segments receive the sensing echo sequence signal formed by the sensing detection sequence signal being reflected from the target object.

[0173] Based on the segmented received sensed echo sequence signal, a plurality of virtual imaging areas are divided, wherein each virtual imaging area is composed of a pixel grid.

[0174] Based on the signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received perception echo sequence signals, a near-field image sequence of the target object relative to each virtual imaging area is generated.

[0175] The electronic device of this embodiment is equipped with an electromagnetic wave transmitting antenna array and a receiving antenna array. The transmitting antenna array transmits a sensing detection sequence signal, and the receiving antenna array receives the sensing detection sequence signal in segments, which is reflected by the target object (user) and forms a sensing echo sequence signal. Next, the sensing echo sequence signal is divided into multiple virtual imaging areas according to the sensing echo sequence signal received in segments, so as to determine the signal parameters of the pixel grid in each virtual imaging area corresponding to the sensing echo sequence signal received in segments. Then, image modeling is performed based on the signal parameters of the pixel grid, and a near-field image sequence corresponding to each virtual imaging area is generated. After that, the near-field image sequences corresponding to these virtual imaging areas are further spliced ​​to obtain a complete near-field image sequence of the target object, and then related artificial intelligence applications such as face recognition, gesture recognition, and liveness detection are performed based on the near-field image sequence of the target object. It can be seen that the electronic device of this embodiment senses user information by transmitting and receiving electromagnetic waves by an antenna array. Compared with the traditional sensing method based on cameras and fingerprint sensors, it does not affect the normal display of the screen too much.

[0176] The information perception method disclosed in the embodiments shown in this specification can be applied to a processor and implemented by the processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0177] Of course, in addition to software implementation, the electronic device in this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0178] In addition, an embodiment of the present application further provides a computer-readable storage medium that stores one or more computer programs, each of which includes instructions. Optionally, when the instructions are executed by a portable electronic device that includes multiple application programs, the portable electronic device can perform the steps of the method shown in Figure 1, including:

[0179] A sensing detection sequence signal is transmitted based on the transmitting antenna array.

[0180] The receiving antenna array segments receive the sensing echo sequence signal formed by the sensing detection sequence signal being reflected from the target object.

[0181] Based on the segmented received sensed echo sequence signal, a plurality of virtual imaging areas are divided, wherein each virtual imaging area is composed of a pixel grid.

[0182] Based on the signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received perception echo sequence signals, a near-field image sequence of the target object relative to each virtual imaging area is generated.

[0183] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0184] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0185] The above are merely examples of the present invention and are not intended to limit this specification. For those skilled in the art, various modifications and variations of this specification are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification shall be included within the scope of the claims of this specification. In addition, all other embodiments obtained by those of ordinary skill in the art without creative effort shall fall within the scope of protection of this document.

Claims

1. An information perception method based on electromagnetic waves, applied to terminal equipment, in, The terminal device is configured with a transmitting antenna array and a receiving antenna array of electromagnetic waves, the electromagnetic waves are millimeter waves or terahertz waves, and the method includes: Transmitting a sensing detection sequence signal based on the transmitting antenna array; Based on the receiving antenna array, segmented reception of the sensing detection sequence signal is formed by reflection of the sensing echo sequence signal after reaching the target object; Dividing a plurality of virtual imaging regions based on the segmented received sensed echo sequence signal, wherein each of the plurality of virtual imaging regions is composed of a pixel grid; Based on the signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received perception echo sequence signals, a near-field image sequence of the target object relative to each virtual imaging area is generated.

2. The method according to claim 1, in, The near-field image sequences corresponding to the multiple virtual imaging areas are spliced ​​to obtain a spliced ​​near-field image sequence of the target object.

3. The method according to claim 1, wherein: The transmitting antenna array and the receiving antenna array are no less than two; the terminal device is configured with at least one of a first mode, a second mode and a third mode; in: In the first mode, each transmitting antenna array transmits a sensing detection sequence signal in a time sequence, and each receiving antenna array respectively receives a sensing echo sequence signal corresponding to the sensing detection sequence signal of each transmitting antenna array transmitted in a time sequence; In the second mode, each of the transmitting antenna arrays transmits a sensing detection sequence signal independently, and each of the receiving antenna arrays receives a sensing echo sequence signal from each of the transmitting antenna arrays respectively; In the third mode, the transmitting antenna arrays correspond to the receiving antenna arrays in groups, each transmitting antenna array transmits a sensing detection sequence signal independently, and each receiving antenna array receives a sensing echo sequence signal from the transmitting antenna array of the group to which it belongs.

4. The method according to claim 3, in, The transmitting antenna array includes a first transmitting antenna array and a second transmitting antenna array, and the receiving antenna array of the terminal device includes a first receiving antenna array and a second receiving antenna array; The step of dividing a plurality of virtual imaging areas based on the segmented received sensed echo sequence signal comprises: When the terminal device adopts the first mode, a virtual imaging area is divided according to the sensing echo sequence signals corresponding to the sensing detection sequence signals transmitted in a time sequence and received by the first receiving antenna array and the sensing echo sequence signals corresponding to the sensing detection sequence signals transmitted in a time sequence and received by the second receiving antenna array; When the terminal device adopts the second mode, the first receiving antenna array receives the sensing echo sequence signal of the first transmitting antenna array, the first receiving antenna array receives the sensing echo sequence signal of the first transmitting antenna array, and the first receiving antenna array receives the sensing echo sequence signal of the first transmitting antenna array. The virtual imaging area is divided according to the sensed echo sequence signal of the first transmitting antenna array, the sensed echo sequence signal originated from the first transmitting antenna array and received by the second receiving antenna array, and the sensed echo sequence signal originated from the second transmitting antenna array and received by the second receiving antenna array; When the terminal device adopts the third mode, the virtual imaging area is divided according to the perceived echo sequence signal originated from the first transmitting antenna array and received by the first receiving antenna array and the perceived echo sequence signal originated from the second transmitting antenna array and received by the second receiving antenna array.

5. The method according to claim 3, in, In the case where the number of array elements in the transmitting antenna array and the receiving antenna array is limited, the perceived echo sequence signal in the third mode is received by the receiving antenna array segment when the terminal device is swinging, wherein the limited number of array elements in the transmitting antenna array and the receiving antenna array includes that the number of array elements in the transmitting antenna array and the receiving antenna array does not meet the segmented reception imaging accuracy requirement when the terminal device is static.

6. The method according to any one of claims 1 to 5, in, Before generating a near-field image sequence corresponding to each virtual imaging area based on the signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received perceived echo sequence signal, the method further includes: Determine a heterodyne mixing signal of a pixel grid in each virtual imaging area based on a near-field spherical wavefront of the segmented received echo signal sequence; Determine a virtual array compensation vector based on a time delay of an array element in the receiving antenna array relative to a pixel grid in each virtual imaging area, and compensate a heterodyne mixing signal of the pixel grid in each virtual imaging area based on the virtual array compensation vector; The compensated heterodyne mixing signals of the pixel grid in each virtual imaging area are coherently superimposed to determine the signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received perception echo sequence signals.

7. The method according to any one of claims 1 to 5, in, The generating of a near-field image sequence corresponding to each virtual imaging area based on the signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received sensed echo sequence signal comprises: Mapping the signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received sensed echo sequence signal to a spatial coordinate system for image scanning modeling, and generating a near-field image sequence corresponding to each virtual imaging area; and / or, Performing Fourier transform on the signal parameters of the pixel grid in each virtual imaging area corresponding to the segmented received perceived echo sequence signal to obtain frequency domain feature data, and inputting the frequency domain feature data into a convolutional neural network model to obtain a near-field image sequence corresponding to each virtual imaging area; wherein the convolutional neural network model is trained based on the frequency domain feature data corresponding to the sample perceived echo sequence signal and the near-field image sequence label annotated with the sample perceived echo sequence signal.

8. The method according to claim 2, in, The near-field image sequences corresponding to the multiple virtual imaging areas are spliced ​​to obtain the spliced ​​image sequence of the target object. After the near-field image sequence, the method includes: Registering near-field image sequences corresponding to at least some of the multiple virtual imaging areas; wherein the registration method includes at least one of the following: Mapping the near-field image sequences corresponding to the at least part of the virtual imaging area to the frequency domain, and calculating the translation vectors of the near-field image sequences corresponding to the at least part of the virtual imaging area relative to the frequency domain based on the cross-power spectral density function, and registering the near-field image sequences corresponding to the at least part of the virtual imaging area based on the translation vectors; Determine common feature points between near-field image sequences corresponding to at least part of the virtual imaging area based on the time domain, and register the near-field image sequences corresponding to at least part of the virtual imaging area according to the common feature points.

9. The method according to any one of claims 1 to 5, in, Also includes: The sensed echo sequence signals received in segments are subjected to fast Fourier transform to obtain a frequency domain signal sequence showing the relationship between the chest distance of the target object and the pulse peak, and the heartbeat signal and / or breathing signal of the target object are determined based on a preset quantitative relationship between the phase change and the chest displacement in the frequency domain signal sequence.

10. A terminal device, in, A processor and an array of transmitting antennas and an array of receiving antennas for electromagnetic waves, wherein the electromagnetic waves are millimeter waves or terahertz waves, wherein: The transmitting antenna array is used to transmit the sensing detection sequence signal; The receiving antenna array is used to segmentally receive the sensing echo sequence signal formed by the sensing detection sequence signal being reflected after reaching the target object; The processor is used to divide a plurality of virtual imaging areas based on the segmented received perceived echo sequence signals, each of the plurality of virtual imaging areas being composed of a pixel grid; and to generate a near-field image sequence of the target object relative to each virtual imaging area based on signal parameters of the segmented received perceived echo sequence signals corresponding to the pixel grid in each virtual imaging area.

11. The terminal device according to claim 10, wherein: The transmitting antenna array includes a first transmitting antenna array and a second transmitting antenna array. in, The first transmitting antenna array is arranged in the non-operable areas on both sides of the upper half of the screen, and the second transmitting antenna array is arranged in the non-operable areas on both sides of the lower half of the screen; The receiving antenna array includes a first receiving antenna array and a second receiving antenna array, wherein the first receiving antenna array is arranged in a non-operable area at the top of the screen, and the second receiving antenna array is arranged in a non-operable area at the bottom of the screen.

12. The method according to claim 10, wherein: At least part of the array elements of the transmitting antenna array and the receiving antenna array are arranged below the screen using display screen antenna technology; or, At least some array elements of the transmitting antenna array and the receiving antenna array are arranged in the gap between the frame of the terminal device and the screen, in, The transmitting antenna array and the receiving antenna array are arranged in the terminal In the case of a gap between the frame of the device and the screen, the transmitting antenna array and the receiving antenna array both include a plurality of miniature horn antennas whose main structures extend along the inner surface of the frame.

13. The terminal device according to claim 11, in, The transmitting antenna array and the receiving antenna array are time-division multiplexed as the communication antennas of the terminal device; wherein: the transmitting antenna array uses an orthogonal frequency division multiplexing mode to send communication signals, and uses a frequency modulated continuous wave mode to send the sensing detection sequence signal; the receiving antenna array uses an orthogonal frequency division multiplexing mode to receive communication signals, and uses a frequency modulated continuous wave mode to receive the sensing echo sequence signal; or, The transmitting antenna array and the receiving antenna array are simultaneously multiplexed as the communication antennas of the terminal device; wherein: the terminal device is provided with a perception mode and a communication mode; wherein: the transmitting antenna array uses an orthogonal frequency division multiplexing mode to send communication signals and the perception detection sequence signals, and the receiving antenna array uses an orthogonal frequency division multiplexing mode to receive communication signals and the perception echo sequence signals.

14. An electronic device, in, include: A memory for storing one or more computer programs; A processor, configured to load the one or more computer programs to execute the method according to any one of claims 1 to 9.

15. A computer-readable storage medium having one or more computer programs stored thereon. in, When the one or more computer programs are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images