Sensing method based on wi-fi signal, and electronic device and storage medium

Abstract

Provided in the present disclosure are a sensing method based on a Wi-Fi signal, and an electronic device and a storage medium. The sensing method based on a Wi-Fi signal is applied to an onboard device, wherein the onboard device comprises an antenna and a reconfigurable intelligent surface. The sensing method based on a Wi-Fi signal comprises: controlling a reconfigurable intelligent surface to reflect a Wi-Fi signal to a region-of-interest of a target object in a vehicle cabin; collecting first CSI data of the region-of-interest by means of an antenna; and on the basis of the first CSI data, acquiring sensing information of the target object in the region-of-interest.

Description

Wi-Fi signal-based sensing method, electronic device, and storage medium

[0001] This application claims priority to Chinese patent application No. 202410350492.0, filed on March 22, 2024, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of communication technologies, and in particular to a Wi-Fi signal-based sensing method, electronic device, and storage medium. Background Art

[0003] Currently, in a motor vehicle, a mobile phone can sense people or objects inside the vehicle by receiving in-vehicle Wi-Fi (wireless fidelity) signals scattered by people or objects inside the vehicle. Summary of the Invention

[0004] On the one hand, a Wi-Fi signal-based perception method is provided, which is applied to an in-vehicle device, wherein the in-vehicle device includes an antenna and a reconfigurable intelligent surface (RIS). The Wi-Fi signal-based perception method includes: controlling the intelligent metasurface to reflect the Wi-Fi signal to a region of interest (ROI) of a target object in a vehicle cabin; collecting first CSI (channel state information) data of the ROI through the antenna; and obtaining perception information of the target object in the ROI based on the first CSI data.

[0005] In another aspect, a Wi-Fi signal-based sensing device is provided for use in an in-vehicle device comprising an antenna and a smart metasurface. The Wi-Fi signal-based sensing device comprises a control module, a collection module, and an acquisition module. The control module is configured to control the smart metasurface to reflect Wi-Fi signals toward an area of ​​interest (ROI) of a target object within the vehicle cabin. The acquisition module is configured to collect first CSI data of the ROI via the antenna. The acquisition module is configured to acquire, based on the first CSI data, sensing information of the target object within the ROI.

[0006] In another aspect, an electronic device is provided, comprising: a memory and a processor. The memory is coupled to the processor; the memory is used to store a computer program; and the processor implements the above-mentioned sensing method when executing the computer program.

[0007] On the other hand, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above-mentioned perception method is implemented.

[0008] On the other hand, a computer program product is provided, which includes computer program instructions, and when the computer program instructions are executed by a processor, the above-mentioned perception method is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] To more clearly illustrate the technical solutions of the present disclosure, the following briefly introduces the drawings required for use in some embodiments of the present disclosure. Obviously, the drawings described below are only drawings of some embodiments of the present disclosure, and those skilled in the art can also derive other drawings based on these drawings.

[0010] FIG1 is a schematic diagram of the architecture of a perception system according to some embodiments of the present disclosure.

[0011] FIG2 is a schematic diagram of the internal structure of an electronic device according to some embodiments of the present disclosure.

[0012] FIG3 is a flowchart of a Wi-Fi signal-based sensing method according to some embodiments of the present disclosure.

[0013] FIG4 is a flowchart of another Wi-Fi signal-based sensing method according to some embodiments of the present disclosure.

[0014] FIG5 is a schematic diagram of channel distribution according to some embodiments of the present disclosure.

[0015] FIG6 is a flowchart of another Wi-Fi signal-based sensing method according to some embodiments of the present disclosure.

[0016] FIG7 is a flowchart of another Wi-Fi signal-based sensing method according to some embodiments of the present disclosure.

[0017] FIG8 is a flowchart of another Wi-Fi signal-based sensing method according to some embodiments of the present disclosure.

[0018] FIG9 is a flowchart of another Wi-Fi signal-based sensing method according to some embodiments of the present disclosure.

[0019] FIG10 is a flowchart of another Wi-Fi signal-based sensing method according to some embodiments of the present disclosure.

[0020] FIG11 is a flowchart of another Wi-Fi signal-based sensing method according to some embodiments of the present disclosure.

[0021] FIG12 is a schematic diagram of the internal structure of a Wi-Fi signal-based sensing device according to some embodiments of the present disclosure.

[0022] FIG13 is a schematic structural diagram of another electronic device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions of this disclosure in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this disclosure, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0024] It should be noted that, in this disclosure, words such as "exemplary" or "for example" are used to describe examples, illustrations, or explanations. Any embodiment or design described in this disclosure using words such as "exemplary" or "for example" should not be interpreted as being more preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0025] In the following, the terms "first," "second," etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the quantity of the technical features indicated. Thus, a feature defined by the terms "first," "second," etc. may explicitly or implicitly include one or more of the features.

[0026] In this disclosure, unless otherwise specified, " / " means "or." For example, A / B can mean either A or B. "And / or" is used herein solely to describe an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: only A, A and B, or only B. Furthermore, "at least one" means one or more, and "a plurality" means two or more.

[0027] Currently, in the field of Wi-Fi sensing inside vehicles, if there is a Wi-Fi device inside the vehicle that transmits Wi-Fi signals, a terminal with Wi-Fi sensing capabilities can collect Wi-Fi signals scattered by objects inside the vehicle to obtain object perception information. However, the signal strength of the Wi-Fi signals emitted by the Wi-Fi device is not high enough, and the Wi-Fi signals scattered by objects collected by the terminal may not be able to obtain more accurate perception information. In other words, the terminal's ability to perceive people or objects inside the vehicle through Wi-Fi signals alone is insufficient, resulting in low perception information accuracy and poor perception effects. Directly capturing images inside the vehicle using devices such as cameras to obtain high-precision information about objects inside the vehicle also poses the risk of leaking personal privacy.

[0028] To address the aforementioned issues, embodiments of the present disclosure propose a Wi-Fi signal-based sensing method for use in an in-vehicle device. The in-vehicle device includes an antenna and an intelligent metasurface. The sensing method includes: controlling the intelligent metasurface to reflect Wi-Fi signals toward an area of ​​interest (ROI) of a target object within the vehicle cabin; using the antenna to collect first CSI data of the ROI; and obtaining sensing information of the target object within the ROI based on the first CSI data. The intelligent metasurface has beamforming capabilities, which can reflect Wi-Fi signals in space toward the ROI where the target object is located, thereby enhancing the signal strength within the ROI. Target objects within the ROI can scatter Wi-Fi signals within the ROI. The antenna can collect first CSI signals from the ROI. This first CSI data is used to represent channel state information corresponding to the Wi-Fi signals scattered by the target object. Because the target object has characteristics such as shape and motion, the first CSI data corresponding to the Wi-Fi signals scattered by the target object contains information related to the shape, motion, and other characteristics of the target object. Consequently, the in-vehicle device can obtain sensing information of the target object within the ROI based on the first CSI data. Furthermore, because the intelligent metasurface enhances the signal strength within the region of interest, the first CSI data contains information related to the characteristics of the target object with greater accuracy. Thus, by enhancing the signal strength within the region of interest using the intelligent metasurface, sensory information about the target object within the region of interest can be acquired more accurately and reliably, and with higher precision.

[0029] The Wi-Fi signal-based perception method provided by the embodiment of the present disclosure can be applied to the perception system shown in FIG1 . As shown in FIG1 , the perception system includes: a Wi-Fi device 101 , a vehicle-mounted device 102 , and a target object 103 .

[0030] The Wi-Fi device 101 is used to transmit Wi-Fi signals. The Wi-Fi device 101 can be a vehicle-mounted Wi-Fi device fixedly installed in a vehicle, or a portable Wi-Fi device.

[0031] The vehicle-mounted device 102 includes a smart metasurface 1021 and an antenna 1022. The smart metasurface 1021 is configured to reflect Wi-Fi signals into an area of ​​interest (ROI) where the target object 103 is located. The antenna 1022 is configured to receive Wi-Fi signals scattered by the target object 103 and collect CSI data corresponding to the Wi-Fi signals scattered by the target object 103.

[0032] The target object 103 may be a person or object in the vehicle, which may scatter the Wi-Fi signal in the area where the target object 103 is located.

[0033] It should be noted that Figure 1 is only an exemplary architecture diagram, and the number and type of devices, target objects, etc. included in Figure 1 are not limited. For example, in addition to the devices and target objects shown in Figure 1, the perception system may also include the human body as a target object, or may include two or more antennas to improve the CSI data collection capability, or may include two or more smart metasurfaces to improve the ability to reflect Wi-Fi signals to the area of ​​interest, or may include a user terminal to transmit perception information of the target object to the user terminal.

[0034] As shown in Figure 2, an embodiment of the present disclosure provides a schematic diagram of the internal structure of a vehicle-mounted device 102, which includes: an intelligent metasurface 201, a processor 202, a memory 203, an antenna 204, a radio frequency front-end module 205 and a power management module 206.

[0035] The smart metasurface 201 is used to reflect Wi-Fi signals to the area where the target object is located.

[0036] The processor 202 is used to control the smart metasurface 201 to reflect the Wi-Fi signal and process the Wi-Fi signal received by the antenna to obtain the above-mentioned first CSI signal.

[0037] The memory 203 is used for the processor 202 to obtain or store data.

[0038] Antenna 204 is used to receive Wi-Fi signals.

[0039] The RF front-end module 205 is used to perform signal processing operations such as noise reduction and signal conversion on the Wi-Fi signal received by the antenna 204 .

[0040] The power management module 206 is used to provide power to various modules in the device 102 that require power.

[0041] In one implementation, the vehicle-mounted device 102 may further include a backup power supply 207. The backup power supply 207 is used to supply power to various modules in the vehicle-mounted device 102 that require power when the vehicle is not started.

[0042] In one implementation, the in-vehicle device 102 may further include a vehicle-machine interface 208, which is used to connect the in-vehicle device and other electronic devices or systems deployed on the vehicle, such as an in-car monitoring system (IMS), an in-vehicle safety control system, an in-vehicle infotainment system, and a telematics box (T-BOX). The in-car monitoring system also includes a driver monitoring system (DMS) and an object monitoring system (OMS). The telematics box can control the LTE (Long Term Evolution) / 5G (5th generation mobile communication technology) antenna and interact with the roadside unit (RSU), telematics service provider (TSP), or mobile Internet through V2X (Vehicle-to-Everything) or PC5 port (Sidelink interface in LTE-V2X technology), or interact with user terminals.

[0043] The perception method provided by the embodiment of the present disclosure is applied to the object perception scenario inside the vehicle. For example, the biometric information and motion feature information of the driver or passengers inside the vehicle can be obtained through the perception method provided by the embodiment of the present disclosure, as well as the location information and material information of the objects inside the vehicle. Furthermore, based on the above-mentioned perception information, it is also possible to interact with other systems deployed on the terminal or vehicle. The other systems may include one or more of the following: an in-cabin monitoring system, an on-board safety control system, an on-board infotainment system, and a vehicle remote information sending system. Furthermore, based on the perception method provided by the embodiment of the present disclosure, applications in scenarios such as safety and entertainment can be realized.

[0044] The sensing method provided by the embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0045] The perception method provided by the embodiment of the present disclosure can be applied to the vehicle-mounted device 102 in the perception system shown in Figure 1. Figure 3 shows a flow chart of a perception method. As shown in Figure 3, the perception method includes S301 to S303.

[0046] In S301 , the smart metasurface is controlled to reflect the Wi-Fi signal to an area of ​​interest of a target object in a vehicle cabin.

[0047] The target objects in the vehicle cabin can be the driver, passengers, pets or objects.

[0048] The smart metasurface is a signal-reflecting device with beamforming capabilities. It can include multiple reflective units (or multiple meta-atomic materials). The reflective units can receive Wi-Fi signals and reflect them based on their configuration.

[0049] For example, the multiple reflective units in the smart metasurface can reflect Wi-Fi signals with phases and amplitudes corresponding to their respective configurations. The Wi-Fi signals reflected by these multiple reflective units are superimposed on each other to achieve a strong signal strength in a specific direction (which can be the direction of the region of interest). In other words, the smart metasurface implements beamforming (or the ability to control the direction of the electromagnetic beam corresponding to the Wi-Fi signal).

[0050] In some embodiments, the collection of multiple reflective units in the smart metasurface is a uniform platform array (UPA).

[0051] In some embodiments, the spacing between adjacent reflective units in the smart metasurface is less than or equal to half the wavelength of the electromagnetic wave corresponding to the Wi-Fi signal. This allows more reflective units to be deployed within a fixed area, increasing the density of reflective units in space, thereby enhancing the smart metasurface's ability to control radio waves and improving beamforming effects.

[0052] In some embodiments, there may be multiple areas of interest in the vehicle cabin, and the on-board equipment can control the smart metasurface to reflect Wi-Fi signals to the multiple areas of interest.

[0053] In S302, first CSI data of a region of interest is collected through an antenna.

[0054] It should be understood that the antenna can receive Wi-Fi signals in space. The Wi-Fi signals include Wi-Fi signals scattered by target objects within the region of interest and other interference signals. The vehicle-mounted device receives the Wi-Fi signals through the antenna and processes the Wi-Fi signals to obtain the first CSI data of the region of interest.

[0055] It should be understood that the first CSI data for the ROI includes channel state information of the channel between the ROI and the antenna (i.e., the electromagnetic waves corresponding to the Wi-Fi signal scattered by the target object). This first CSI data includes information about changes in the channel state between the ROI and the antenna caused by characteristics such as the target object's shape or movement.

[0056] In some embodiments, a Wi-Fi signal is composed of multiple mutually orthogonal subcarriers (multi-input and multi-output orthogonal frequency division multiplexing, MIMO-OFDM). In this case, the subcarrier with the smallest inter-packet variance among the multiple subcarriers can be determined as a channel to be processed based on principal component analysis (PCA) or other statistical estimation methods. The channel state information data of the channel to be processed is the first CSI data. The subcarrier with the smallest variance has the lowest noise signal strength and the highest useful signal strength (Wi-Fi signal carrying target object characteristic information).

[0057] In some embodiments, the antenna may include multiple antenna units, each of which can receive Wi-Fi signals. The vehicle-mounted device may perform conjugate multiplication or conjugate division on the Wi-Fi signals received by each of the multiple antenna units to eliminate baseband effects.

[0058] In S303, perception information of a target object within the region of interest is acquired based on the first CSI data.

[0059] In an exemplary implementation, when the target object is an object, the target object's perception information includes one or more of the following: whether the object exists, the object's location information, the object's material, and the object's type (e.g., car keys, handbags, luggage, etc.); when the target object is a driver, the target object's perception information includes one or more of the following: the driver's facial expression, the driver's motion characteristics (e.g., body movements), and the driver's biometric information (e.g., breathing, heartbeat, etc.); when the target object is a passenger, the target object's perception information includes one or more of the following: the passenger's motion characteristics, the passenger's biometric information, the passenger's facial expression, etc. It is understandable that because the first CSI data includes information about changes in the channel state between the region of interest and the antenna due to characteristics such as the shape or motion of the target object, the on-board device can obtain perception information of the target object within the region of interest based on the first CSI data.

[0060] It should be understood that based on the shape characteristics of the target object, perceptual information such as the size, outline or existence of the target object can be determined; based on the shape and motion characteristics of the target object, perceptual information of the motion characteristics of the large-scale target object can be determined, and biometric information of the small-scale target object, such as the breathing, heartbeat and other perceptual information of the person can also be determined.

[0061] In an exemplary embodiment, the vehicle-mounted device can determine the distance and direction information between multiple points on the target object and the vehicle-mounted device based on the first CSI data, thereby obtaining spatial distribution information of the multiple points on the target object. The spatial distribution of the multiple points is the contour information of the target object. The more points selected on the target object, the higher the resolution of the obtained contour information of the target object. The stronger the signal strength of the Wi-Fi signal corresponding to the first CSI data, the more accurate the distance and direction information between the multiple points on the target object and the vehicle-mounted device.

[0062] Furthermore, the vehicle-mounted device can also determine velocity information between the multiple points on the target object and the vehicle-mounted device based on the distance information and direction information between the multiple points on the target object and the vehicle-mounted device at each of the multiple time points. The velocity information of a point on the target object includes the direction and velocity of movement of the point relative to the vehicle-mounted device. The shorter the intervals between the multiple time points, the more accurate the velocity information of the target object.

[0063] In an exemplary embodiment, the on-board equipment can obtain parameters such as angle of arrival (AoA), time of flight (ToF), and Doppler frequency shift (DFS) based on the first CSI data, and process the parameters such as angle of arrival, time of flight, and Doppler frequency shift based on MUSIC (multiple signal classification) to obtain perception information of the target object.

[0064] For example, in the case of the i-th time packet, the j-th subcarrier, the l-th antenna, and the k-th path, the perception parameter satisfies the following formula (1): τk (i,j,l)≈f c τ k +Δf j τ k +f c Δd l φ k -f Dk Δt i (1)

[0065] Among them, τ k is the flight time of the kth path, φ k is the arrival angle of the kth path, is the Doppler shift of the kth path, f c is the center frequency of the subcarrier, Δt i represents the time difference between the i-th time packet and the reference time, Δf j represents the frequency difference between the jth subcarrier and the reference carrier, Δd l Represents the antenna distance difference between the lth antenna and the reference antenna.

[0066] In an exemplary implementation, a mapping relationship between Doppler frequency shift features and perception information can be obtained based on a convolutional neural network model, so as to identify the perception information based on the trained neural network model.

[0067] It's understood that Doppler shift is the change in path length of a signal reflected from a moving object, causing the observed signal frequency to shift. When a wireless signal reflects off a moving person or animal and reaches the receiver, the Doppler effect produces a Doppler shift of approximately 10 Hz. This Doppler shift can be used to determine the direction of the signal's incidence.

[0068] In one implementation, the antenna and the smart metasurface are integrated. The antenna is a uniform linear array (ULA), which has an angle difference with the uniform antenna array of the smart metasurface. In this way, when the Wi-Fi device is determined, three reference points in space (i.e., the position of the smart metasurface, the position of the antenna, and the position of the Wi-Fi device) can be obtained. Then, the position of the target object relative to these three reference points can be accurately determined through the Wi-Fi signal channel from the Wi-Fi device to the smart metasurface, the channel from the smart metasurface to the target object, and the channel from the target object to the antenna.

[0069] In some embodiments, if the vehicle is detected to have been parked in an abnormal state based on the perception information of the target object, a warning message is issued. This abnormal state may be caused by a passenger's luggage or a driver's belongings, such as car keys, being left behind in the vehicle while it is parked. It should be understood that if the vehicle is detected to have been parked in an abnormal state based on the perception information of the target object, the warning message can be issued to prevent passengers or drivers from leaving valuables behind, thereby ensuring a good driving or riding experience.

[0070] In some embodiments, when it is recognized based on the perception information of the personnel that a person has engaged in dangerous behavior, at least one of the following operations is performed: issuing a warning message, activating a personal protection device, instructing the vehicle control system to perform automatic driving, and instructing the vehicle control system to brake. Dangerous behaviors may include at least one of the following: a passenger's behavior that interferes with the driver's safe driving, the driver's dangerous driving behavior such as playing with a mobile phone, fatigue driving behavior such as an increased frequency of blinking or yawning (or the frequency reaches a frequency threshold), an abnormal expression of the driver or passenger (such as a painful expression), an abnormal biometric feature of the driver or passenger (such as increased breathing or heartbeat). In the embodiment of the present disclosure, based on the perception information, it is recognized that a person has engaged in dangerous behavior, and one or more operations such as information prompts, automatic driving personal protection, and vehicle braking are performed, which can safely control the vehicle driving or protect the personnel, and ensure the driving safety of the vehicle and the personal safety of the driver, passengers, and other personnel.

[0071] In some embodiments, when an item is identified as a dangerous item based on the sensing information of the item, at least one of the following operations is performed: issuing an alarm message, activating a personal protective device, instructing the vehicle control system to perform automatic driving, and instructing the vehicle control system to brake. Dangerous items include chemically corrosive items, flammable and explosive items, and other items that may cause damage to the vehicle or pose a safety hazard to the vehicle's driving. In the embodiment of the present disclosure, based on the alarm message, it is possible to remind the passenger or driver that the item cannot be brought on board; or activate a personal protective device to cooperate with the driver's emergency brake operation; or instruct the vehicle control system to perform automatic driving so that the driver can perform emergency treatment on flammable and explosive items; or instruct the vehicle control system to brake to stop the vehicle when the vehicle is carrying dangerous items. In this way, the driving safety of the vehicle and the safety of the people in the vehicle can be guaranteed.

[0072] In some embodiments, the vehicle-mounted device can send perception information or warning information corresponding to the perception information to the user terminal based on the air interface in the vehicle system.

[0073] Users can also send perception requests to the vehicle-mounted device based on the terminal to perceive the target object again to avoid perception information errors.

[0074] In some embodiments, an application (APP) for interacting with the vehicle-mounted device may be deployed on the terminal. The application may be used to receive information sent by the vehicle-mounted device, or send a perception request to the vehicle-mounted device to enable the vehicle-mounted device to perform a perception operation on the target object, etc.

[0075] In one implementation, the vehicle-mounted device can delay operation after parking to determine whether an abnormal stay event has occurred. The abnormal stay event includes one or more of the following: a passenger staying in the vehicle, a pet staying in the vehicle, or an item left in the vehicle. Furthermore, in the event of an abnormal stay event, the vehicle-mounted device can sense whether the driver or passenger is near the vehicle based on the external sensing device; issue a voice prompt based on the presence of a driver or passenger near the vehicle; and send a prompt message to the driver's or passenger's terminal based on the absence of a driver or passenger near the vehicle to inform the driver or passenger of the abnormal stay event.

[0076] In the disclosed embodiment, an onboard device controls the intelligent metasurface to reflect Wi-Fi signals to an area of ​​interest of a target object within the vehicle cabin, thereby enhancing the signal strength of the Wi-Fi signals scattered by the target object. The antenna then collects first CSI data of the area of ​​interest, which contains characteristic information of the target object. Consequently, the onboard device can obtain perceptual information of the target object within the area of ​​interest based on the first CSI data. Furthermore, because the signal strength of the Wi-Fi signals scattered by the target object is relatively strong, the characteristic information of the target object contained in the first CSI data is more accurate, enabling the onboard device to more accurately obtain the perceptual information of the target object and to obtain higher-precision perceptual information of the target object.

[0077] In combination with FIG3 , as shown in FIG4 , in one implementation, in the above S301 , controlling the smart metasurface to reflect the Wi-Fi signal to the area of ​​interest of the target object in the vehicle cabin includes S401 and S402 .

[0078] In S401, position information of a region of interest is determined, and first beam parameters of the smart metasurface are determined based on the position information of the region of interest.

[0079] In an exemplary implementation, the location information of the region of interest may be location information of the region of interest relative to the Wi-Fi device, the smart metasurface, and the antenna.

[0080] The first beam parameter is used to make the reflected beam of the smart metasurface cover the area of ​​interest.

[0081] In S402 , based on the first beam parameter of the smart metasurface, the smart metasurface is controlled to reflect a signal to an area of ​​interest of a target object in a vehicle cabin.

[0082] In an exemplary embodiment, the first beam parameter (or any other beam parameter) of the smart metasurface is a beamforming matrix. The beamforming matrix includes M×M elements, each element corresponding to a reflective unit of the smart metasurface, each element representing a phase offset of the reflective unit, and M and N are positive integers.

[0083] As shown in Figure 5, it is a schematic diagram of the channel distribution between an electronic device, a Wi-Fi device, and an area of ​​interest according to an embodiment of the present disclosure. h represents the channel between the smart metasurface in the electronic device and the area of ​​interest, r represents the channel between the area of ​​interest and the antenna in the electronic device, g represents the channel between the Wi-Fi device and the smart metasurface, P represents the channel between the Wi-Fi device and the antenna, q represents the channel between the Wi-Fi device and the area of ​​interest, and v represents the channel between the Wi-Fi device and the area of ​​interest. i 、 They are used to represent the incident azimuth and incident elevation angles of the channel between the Wi-Fi device and the smart metasurface, v o 、 They are used to represent the initial azimuth and elevation angle of the channel between the smart metasurface and the area of ​​interest, b represents the distance between the antenna and the smart metasurface (when the antenna and the smart metasurface are fixedly connected), α is the rotation angle of the Wi-Fi device’s transmitting antenna, and h a is the wire vector of the signal reflected by the smart metasurface received by the antenna in the electronic device, δ is the outgoing azimuth angle of the signal transmitted by the Wi-Fi device to the smart metasurface, and X is the coordinate axis.

[0084] In one implementation, the region of interest includes region of interest 1 and region of interest 2, and the channel h includes h1 and h2, the channel q includes q1 and q2, and the channel r includes r1 and r2.

[0085] In some embodiments, the beamforming matrix is ​​a diagonal matrix Elements are j is the imaginary unit, θ m is the adjustable angle of the antenna element, M and m are positive integers, 1≤m≤M.

[0086] The Wi-Fi signal received by the antenna satisfies the following formula (2): y = ζr H h H Φgus+z (2)

[0087] Where y represents the Wi-Fi signal received by the antenna, ζ represents the loss factor of the Wi-Fi signal during the round trip (used to indicate the degree of signal attenuation), h represents the channel (i.e., electromagnetic wave) between the smart metasurface and the area of ​​interest, and h H represents the transpose of h, r represents the channel between the region of interest and the antenna, r H represents the transpose of r, g is the channel between the Wi-Fi device and the smart metasurface, u is the beamforming of the Wi-Fi device, s is the multiple carrier signals transmitted by the Wi-Fi device, and z is Gaussian white noise.

[0088] The Wi-Fi device includes M×N antenna units, the channel g between the Wi-Fi device and the smart metasurface includes M×N subcarriers, the smart metasurface includes M×M reflection units, the channel h between the smart metasurface and the area of ​​interest includes M×M subcarriers, the antenna includes M×1 antenna units, and the channel r between the area of ​​interest and the antenna includes M×1 subcarriers.

[0089] When there are multiple areas of interest that need to be sensed, the antenna maximizes the weighted sum-rate (WSR) of the received Wi-Fi signal to ensure the minimum SNR (signal-to-noise ratio) of Wi-Fi perception. The intelligent metasurface beamforming problem model satisfies the following formula (3):

[0090] Where L represents the number of regions of interest, K represents the number of Wi-Fi signal paths from a region of interest to the antenna, and ρ l Indicates the weight of the channel between the antenna and each area of ​​interest. P is the channel between Wi-Fi and the antenna. Represents the square of the modulus of w.

[0091] In one implementation, the antenna also receives Wi-Fi signals scattered by target objects within the region of interest and transmitted directly to the region of interest from Wi-Fi devices. At this time, the intelligent metasurface beamforming problem model that maximizes the weighted sum-rate (WSR) of communication and ensures the minimum SNR (signal-to-noise ratio) perceived by Wi-Fi satisfies the following formula (4):

[0092] Where q is the channel from the Wi-Fi device to the area of ​​interest, q l H is the channel q from the Wi-Fi device to the area of ​​interest l l The transpose of .

[0093] For the NP-hard (Non-deterministic Polynomial-time hard) problem of solving the above formula (3) or formula (4), the alternating optimization method can be used to solve the problem of r l , Φ, u are optimized, that is, fixed Φ and u are optimized r l , fixed r l and Φ optimize u, fix r l and u optimizes Φ.

[0094] Based on the singular value decomposition method, for semi-positive matrices, rank 1 relaxation is used to solve the maximum eigenvector when necessary to obtain the suboptimal solution, and r is optimized. lFor optimizing Φ, we use the augmented Lagrange transform and quadratic transformation to transform it into a second-order constrained second-order programming problem, and then use semidefinite relaxation to transform it into a convex semidefinite programming problem. When optimizing u, we first normalize it and constrain the normalized 2-norm to 1. We then use the maximum ratio combining ratio to transform it into a weighted minimum mean square error problem, which is then transformed into a non-convex second-order constrained second-order programming problem. Auxiliary variables are then introduced to transform it into a convex semidefinite matrix, and rank-1 relaxation is performed when necessary to obtain a suboptimal solution.

[0095] In some embodiments, the channel between the Wi-Fi device and the smart metasurface includes the outgoing azimuth and outgoing pitch angles of the Wi-Fi signal transmitted by the Wi-Fi device, and also includes the incident azimuth and incident pitch angles of the Wi-Fi signal received by the smart metasurface. Similarly, each of the above channels includes the outgoing azimuth and outgoing pitch angles of the channel transmitting end and the incident azimuth and incident pitch angles of the channel receiving end. In this way, the spatial position of the target object can be accurately determined through these outgoing azimuth, outgoing pitch angles, incident azimuth and incident pitch angles.

[0096] In the embodiment of the present disclosure, the vehicle-mounted device determines a first beam parameter that enables the reflection beam of the intelligent metasurface to cover the area of ​​interest based on the position information of the area of ​​interest, thereby controlling the intelligent metasurface to reflect the signal to the area of ​​interest of the target object in the vehicle cabin, thereby enhancing the signal strength in the area of ​​interest and improving the perception accuracy of the target object.

[0097] In some embodiments, a sensing system consisting of a Wi-Fi device, a smart metasurface, an antenna, and a target object includes a channel between the Wi-Fi device and the smart metasurface, a signal between the Wi-Fi device and the target object, a channel between the smart metasurface and the target object, a channel between the target object and the antenna, and a channel between the Wi-Fi device and the antenna. The electromagnetic waves emitted by the Wi-Fi device include M×N subcarriers, the electromagnetic waves emitted by the smart metasurface include M×M subcarriers, and the electromagnetic waves received by the antenna include M subcarriers.

[0098] 4 , as shown in FIG6 , in one implementation, in the above S401 , determining the location information of the region of interest includes S601 to S603 .

[0099] In S601 , based on the second beam parameter of the smart metasurface, the smart metasurface is controlled to reflect the Wi-Fi signal to each seating area in the vehicle cabin.

[0100] The second beam parameter is used to make the reflected beam of the smart metasurface cover all seating areas in the vehicle cabin.

[0101] In an exemplary embodiment, an onboard device controls the smart metasurface to reflect Wi-Fi signals to various seating areas within the vehicle cabin based on multiple beam parameters. The device collects CSI data corresponding to each beam parameter via an antenna and determines the vehicle cabin imaging information corresponding to each CSI data. The onboard device determines the beam parameter corresponding to the most accurate vehicle cabin imaging information as the second beam parameter. The vehicle cabin imaging information is the outline information of the vehicle cabin determined by the onboard device based on the CSI data. Based on the beam parameter corresponding to the most accurate vehicle cabin information, the smart metasurface can be effectively controlled to reflect Wi-Fi signals to various seating areas within the vehicle cabin.

[0102] It should be understood that the target object may be located in any seating area in the vehicle cabin area. Therefore, the on-board device controls the smart metasurface to reflect the Wi-Fi signal to each seating area in the vehicle cabin.

[0103] In S602 , second CSI data in the vehicle cabin is collected through an antenna.

[0104] It should be understood that the second CSI data includes feature information of the seat area where the target object exists and feature information of the seat area where the target object does not exist.

[0105] In S603, target object detection is performed based on the second CSI data to determine a seat area where the target object exists, and the position information of the seat area where the target object exists is used as the position information of the region of interest.

[0106] It should be understood that since the area within the vehicle is divided into multiple seating areas, the multiple seating areas are used as references, and the position information of the seating area where the target object exists is used as the position information of the area of ​​interest, the area of ​​interest of the target object can be determined more quickly.

[0107] In combination with FIG6 , as shown in FIG7 , in one implementation, in the above S603 , target object detection is performed based on the second CSI data to determine the seat area where the target object exists, including S701 to S703 .

[0108] In S701, second frequency domain feature information is obtained based on second CSI data.

[0109] The second frequency domain characteristic information is used to characterize the channel frequency domain characteristics of the vehicle cabin in the current scenario.

[0110] In an exemplary implementation, the frequency response of the channel through which the Wi-Fi device transmits the Wi-Fi signal to the smart metasurface satisfies the following formula (5):

[0111] Where f is the frequency of the subcarrier, t is the arrival time of the subcarrier, N is the number of paths in the channel, and a n is the signal attenuation on the nth path, τ n (t) is the propagation delay on the nth path, and n is a positive integer less than or equal to N.

[0112] The frequency response of the channel where the Wi-Fi device transmits the Wi-Fi signal to the smart metasurface is divided into the empty cabin environment path and the path with the target object, then formula (5) can be written as:

[0113] Among them, H s (f,t) is the sum of the frequency responses of the empty cabin environment path, H d (f, t) is the sum of the frequency responses of the path where the target object exists, N d is the number of paths where the target object exists.

[0114] The sum of the frequency responses of the paths with target objects can be divided into dynamic paths such as human bodies and animals and static paths corresponding to stationary objects, so H d (f,t) can be written as:

[0115] Among them, a n,h is a dynamic path, a n,a is a static path, τ n,h (t) is the static path delay, τ n,a (t) is the static path delay.

[0116] In some embodiments, the frequency response of the channel through which the Wi-Fi signal transmitted by the Wi-Fi device reaches the smart metasurface can be processed based on principal component analysis (PCA) to perform data dimensionality reduction and obtain lower-dimensional data such as static paths and dynamic paths.

[0117] In S702 , a target area is determined based on the second frequency domain feature information and the third frequency domain feature information.

[0118] The third frequency domain feature information is used to characterize the channel frequency domain characteristics of the vehicle cabin in an empty cabin environment. The target area is the area where the channel frequency domain characteristics in the current scenario and the channel frequency domain characteristics in the empty cabin environment are different.

[0119] It should be understood that the second frequency domain feature information includes the frequency domain feature information of the seat area where the target object exists and the frequency domain feature information of the seat area where the target object does not exist, and the third frequency domain feature information includes the frequency domain feature information of each seat area. Therefore, the frequency domain feature information of the seat area where the target object exists and the frequency domain feature information of the seat area where the target object does not exist are obviously different. Therefore, the target area can be determined quickly and accurately.

[0120] In some embodiments, the antenna receives signals corresponding to scattered paths of multiple target objects. If the distance and angle corresponding to the signal of at least one scattered path are within the distance interval and angle interval corresponding to one or more seating areas, the in-vehicle device determines the one or more seating areas as target areas.

[0121] In S703 , the seating area where the target area is located is determined as the seating area where the target object exists.

[0122] In combination with FIG6 , as shown in FIG8 , in one implementation, the perception method provided by the example of the present disclosure further includes S801 to S803 .

[0123] In S801 , when the vehicle cabin is empty, third CSI data in the vehicle cabin is collected through an antenna.

[0124] In S802 , based on the third CSI data, the location information of each seating area in the vehicle cabin and the location information of the Wi-Fi device are determined.

[0125] In one implementation, determining the location information of the Wi-Fi device includes: the vehicle-mounted device determining an angle parameter and a distance parameter of the Wi-Fi device relative to the antenna based on a signal received by the antenna; and the vehicle-mounted device determining the location information of the Wi-Fi device based on the angle parameter and the distance parameter.

[0126] In some embodiments, the signal received by the antenna can be processed based on FFT (fast Fourier transform) and DBF (digital beam forming) to obtain the angle parameters and distance parameters of the Wi-Fi device relative to the antenna. DBF methods can include DML (deterministic maximum likelihood estimation), MUSIC (multiple signal classification), MVDR (minimum variance distortionless response), ESPRIT (rstimating signal parameters via rotational invariance techniques), OMP (orthogonal matching pursuit), and other methods.

[0127] In an exemplary implementation, the vehicle-mounted device determines the angle parameter and distance parameter of the Wi-Fi device relative to the antenna based on the signal received by the antenna. The signal received by the antenna satisfies the following formula (8) after FFT: y = y p +y r +z (8)

[0128] Among them, y p Direct signal for Wi-Fi devices, y r is the signal scattered by the empty cabin environment, and z is Gaussian white noise.

[0129] y p and y r The following formulas (9) and (10) are satisfied respectively: y r =β r D(τ r )·Γ(Φ) (10)

[0130] Among them, β p is the channel gain of channel p between the Wi-Fi device and the antenna, D(τ p ) and D(τ r ) is the delay matrix, τ p is the delay of channel p, τ r is the delay of channel r, β ris the channel gain of channel r between the cabin environment and the antenna, Γ(Φ)∈C M×M Phase modulation matrix of smart metasurface.

[0131] The phase modulation matrix of the i-th row and j-th column in the smart metasurface satisfies the following formula (11):

[0132] Among them, ξ j is the RIS phase response vector corresponding to the jth transmission symbol, h a (ε) is the response vector of the antenna to the smart metasurface, g(ω) is the response vector of the smart metasurface to the Wi-Fi device, ε∈(υ o ,φ o ),ω∈(υ i ,φ i ),υ o ,φ o are the emission azimuth and emission pitch angles of the smart metasurface, υ i ,φ i are the incident azimuth and input pitch angle of the smart metasurface.

[0133] The outgoing azimuth angle, outgoing pitch angle, incident azimuth angle, and input pitch angle of the smart metasurface can be expressed as formula (12):

[0134] in, is the wavelength of the nth subcarrier, f c is the frequency of the subcarrier.

[0135] Assume that the coordinates of the mth reflection unit of the smart metasurface are (r x,m ,r y,m ,r z,m ), the center coordinates of the transmitting antenna of the Wi-Fi device are (w x ,w y ,w z ), the antenna rotation angle of the transmitting antenna of the Wi-Fi device is α, then the incident angle of the smart metasurface receiving the signal transmitted by the Wi-Fi device is formula (13):

[0136] Assume that the center coordinate of the array of reflective units of the smart metasurface is (a x ,a y ,a z ), represents the incident azimuth of the signal received by the antenna, then υ i 、φ i 、 The following formulas (14), (15), and (16) are satisfied respectively:

[0137] The above τ p , τ r Expressed as follows using formulas (17) and (18): τ p =||aw||2 / c+τ Δ (17) τ r =||ar||2 / c+||rw||2 / c+τ' Δ (18)

[0138] Where a is the receiving antenna steering vector, r is the RIS receiving steering vector, w is the antenna steering vector of the Wi-Fi device, and c is the speed of light.

[0139] Based on the channel delay between the smart metasurface and the Wi-Fi device, we can get the expression τ' p Formula (19): τ' p =||rw||2 / c+τ' Δ (19)

[0140] When the antenna and the smart metasurface are fixedly connected, the distance b between the center of the antenna and the smart metasurface satisfies formula (20): b≈(||aw||2cosθ-||rw||2)(sinυ i cosφ i ,sinυ i sinφ i ,cosυ i ) / c(20)

[0141] Since the antenna and the smart metasurface are fixedly connected, b is a known quantity. Based on the above formulas (8) to (20), the parameters required to calculate the location information of the Wi-Fi device can be determined as the vector in, is the delay of channel p, is the transmission tilt angle of the transmitting antenna of the Wi-Fi device, is the incident azimuth angle of the signal received by the smart metasurface, is the incident pitch angle of the signal received by the smart metasurface.

[0142] In one implementation, the vector can be obtained by iteratively interleaving the maximum likelihood estimation method. Each parameter in .

[0143] Based on the estimated parameter vector The rotation angle α and position of the transmitting antenna of the Wi-Fi device can be obtained

[0144] The solutions of formula (21) and formula (22) can be based on the maximum likelihood method and multiple groups of υ i and φ i (The smart metasurface changes the phase of the reflected signal corresponding to υ i and φ i ) is used for estimation. At the same time, the error of the estimated result can be judged based on the Cramerrow lower bound, and the result closest to the Cramerrow lower bound among the multiple estimation results is taken as the final estimation result. When estimating, the corresponding mean square error formula is:

[0145] Among them, E represents expectation, J β is the Fisher information matrix containing β. For example, J β The expression of the element in row i and column j of satisfies the following formula (24):

[0146] Where x is the observation vector and p is the channel between the Wi-Fi device and the antenna.

[0147] At this point, it can be determined The Cramer-Rao lower bound of is:

[0148] Among them, CRLB(β i ) represents the Cramer-Rao lower bound of the i-th β, and M is a positive integer.

[0149] Based on formulas (21), (22), (23), (24), and (25), the rotation angle α and position of the transmitting antenna of the Wi-Fi device can be determined.

[0150] The following is an exemplary implementation method for determining the location of the seating area of ​​a vehicle when the vehicle cabin is empty. The Wi-Fi signal received by the antenna includes K scattered paths. Based on s representing the vector pointing from the region of interest to the antenna, it can be determined that the signal delay from the Wi-Fi device to the antenna satisfies formula (26): τ wifi-Ant =||as k ||2 / c+||rs k ||2 / c+||wr||2 / c (26)

[0151] Among them, τwifi-Ant Indicates the signal delay from the Wi-Fi device to the antenna, s k represents the steering vector from the region of interest to the antenna in the kth scattering path, a represents the steering vector of the antenna, r represents the steering vector of the smart metasurface, w represents the steering vector of the transmitting antenna of the Wi-Fi device, c is the speed of light, 1≤k≤K, and K is a positive integer.

[0152] When the delay from the transmitting antenna to the antenna of the Wi-Fi device can be determined, based on formula (26), it can be determined that the delay from the area of ​​interest to the receiving antenna satisfies formula (27): τ ROI-Ant =||as k ||2 / c+||rs k ||2 / c (27)

[0153] Among them, τ ROI-Ant represents the time delay of the signal from the region of interest to the receiving antenna. In this way, based on the time delay expressed in formula (27), the distance between the region of interest and the antenna can be determined, that is, the distance between the vehicle's seating area and the antenna can be determined.

[0154] For the angle between the area of ​​interest and the antenna (or the angle between the seating area and the antenna), the angle can be calculated based on the incident azimuth angle of the antenna receiving the area of ​​interest. And the output pitch angle φ of the smart metasurface o and the exit azimuth υ o , perform parameter estimation. Let the coordinates of the perception point of the kth scattering path in the region of interest be but φ o 、υ o The following formulas (28)(29)(30) are satisfied:

[0155] Where a is the coordinate of the intelligent metasurface at the center of the antenna (a x ,a y ,a z ), r is the coordinate of the reflection unit of the smart metasurface in the kth scattering path (r x ,r y ,r z ), based on formulas (28)(29)(30) and the joint probability density function of the channel parameters, maximum likelihood estimation is performed to obtain the angle between the area of ​​interest and the antenna (or the angle between the seating area and the antenna).

[0156] In one implementation, the angle between the area of ​​interest and the antenna (or the angle between the seating area and the antenna) closest to the Cramero lower bound can be determined based on the joint probability density function of the channel parameters corresponding to the reflected signals of multiple smart metasurfaces with different phases.

[0157] In some embodiments, there may be multiple Wi-Fi devices transmitting signals. In this case, the Wi-Fi device with the strongest signal may be selected for parameter estimation.

[0158] In S803 , second beam parameters are determined based on the position information of each seating area in the vehicle cabin and the position information of the Wi-Fi device.

[0159] In one implementation, the second beam parameter is a second codebook corresponding to the smart metasurface. The smart metasurface can modulate the reflected beam based on the second codebook so that the modulated beam can cover all seating areas of the vehicle.

[0160] In one implementation of the disclosed embodiment, the vehicle-mounted device includes multiple smart metasurfaces. In conjunction with FIG3 , as shown in FIG9 , in S301 above, controlling the smart metasurface to reflect Wi-Fi signals to an area of ​​interest of a target object within the vehicle cabin includes S901 to S903 .

[0161] In S901 , the matching degree between each of the plurality of smart metasurfaces and the region of interest is determined.

[0162] The matching degree is used to characterize the coverage effect of the reflected beam of the smart metasurface on the area of ​​interest.

[0163] In S902 , based on the matching degree between each of the multiple smart metasurfaces and the region of interest, a target smart metasurface that meets a preset matching degree condition is determined from the multiple smart metasurfaces.

[0164] In some embodiments, the preset matching condition includes any one of the following: the highest matching degree, greater than or equal to a matching degree threshold. In some embodiments, when there are multiple smart metasurfaces with the highest matching degree, the vehicle-mounted device randomly selects one as the target smart metasurface.

[0165] In S903 , the target intelligent metasurface is controlled to reflect the Wi-Fi signal to an area of ​​interest of a target object in the vehicle cabin.

[0166] In one implementation, the on-board device determines the delay of the channel corresponding to the reflection of the Wi-Fi signal from each smart metasurface to the area of ​​interest, and determines the smart metasurface with the lowest channel delay as the target smart metasurface.

[0167] In one implementation, multiple smart metasurfaces serve as backups for each other. If one smart metasurface fails to work, the on-board equipment uses other smart metasurfaces as a replacement.

[0168] In one implementation of the disclosed embodiment, the vehicle-mounted device may include a first smart metasurface, a first antenna, a second smart metasurface, and a second antenna. The first smart metasurface, the first antenna, the second smart metasurface, and the second antenna are all fixedly installed inside the vehicle.

[0169] It should be understood that when the on-board device includes only one smart metasurface and one antenna, the on-board device can directly determine the incident azimuth angle of the scattering channel (the channel corresponding to the Wi-Fi signal scattered by the target object) based on the CSI data corresponding to the scattering channel between the antenna and the target object (or seating area). The incident elevation angle of the scattering channel is obtained by parameter estimation based on the outgoing azimuth angle and outgoing elevation angle of the reflection channel between the smart metasurface and the target object (the channel corresponding to the Wi-Fi signal reflected by the smart metasurface).

[0170] In the case of the first smart metasurface, the first antenna, the second smart metasurface, and the second antenna of the vehicle-mounted device, the vehicle-mounted device can directly determine the incident azimuth and incident elevation angle of the scattering channels between the first antenna and the second antenna and the target object based on the CSI data corresponding to the scattering channels between the first antenna and the second antenna and the target object. In this way, the uncertainty of parameter estimation can be avoided and more accurate perception information of the target object can be obtained.

[0171] It is understandable that compared to the process of changing the phase of the smart metasurface at least once to perform parameter estimation and obtain the incident azimuth and incident elevation angles of the channel between the target object and the antenna when there is only one smart metasurface, when the first smart metasurface and the second smart metasurface reflect Wi-Fi signals to the area of ​​interest, the on-board device does not need to change the phase of the smart metasurface to perform parameter estimation on the incident azimuth and incident elevation angles of the channel between the target object and the antenna. In this way, the perception information of the target object can be determined more conveniently and quickly.

[0172] In some embodiments, the first smart metasurface and the second smart metasurface can be used to enhance the signal strength in the driver's area and the passenger's area, respectively.

[0173] In some embodiments, the first smart metasurface is installed in front of (or directly above) the area where the driver's face is facing, and the second smart metasurface is installed above the area where the passenger sits. In this way, the smart metasurface can better beamform the driver's and passenger's faces, thereby obtaining more accurate perception information.

[0174] In the disclosed embodiment, by determining the target intelligent metasurface with the highest matching degree to reflect the Wi-Fi signal to the target object in the vehicle cabin, the region of interest can have the best signal strength distribution, thereby enabling more accurate and reliable acquisition of perception information of the target object.

[0175] In one implementation, the target object is an object, and the perception information of the target object includes material. In conjunction with FIG3 , as shown in FIG10 , in the above S303 , based on the first CSI data, the perception information of the target object in the region of interest is obtained, including S1001 to S1003 .

[0176] In S1001, based on first CSI data collected by the two antennas respectively, a signal phase difference and a signal amplitude entropy between the two antennas are determined.

[0177] It should be understood that of the two antennas, the signal received by one antenna passes through the target object, while the signal received by the other antenna passes through the air. The signals passing through the target object and the air have different phase and amplitude changes. The process of determining the signal phase difference and signal amplitude entropy based on the phase and amplitude changes satisfies the following formula: ΔΨ=ΔA1 / ΔA2=exp(-(L1-L2)(κ t -κ f )) (32)

[0178] Where ΔΞ is the phase difference of the signals received by the two antennas, is the phase difference between the subcarrier passing through the target and the air, are the phase test values ​​of the two antennas on the target object, L1 and L2 are the distances that the signal travels inside the target object, and η t is the signal phase constant of the signal passing through the target object, η f is the signal phase constant when the signal passes through the air, ΔΨ is the amplitude entropy of the two antennas, ΔA1, ΔA2 are the amplitude test values ​​of the signal passing through the target object, and κ t ,κ f They are the amplitude attenuation constant related to the target object material and the amplitude attenuation constant passing through the air.

[0179] In S1002 , material characteristic parameters are determined based on the signal phase difference and the signal amplitude entropy.

[0180] Material characteristic parameters are not affected by the size of the item.

[0181] In one implementation, the signal phase difference and the signal amplitude entropy are input into a material feature construction model to obtain material feature parameters.

[0182] An exemplary method of eliminating the influence of the size of the object is to eliminate L1 and L2 based on formulas (31) and (32) to obtain material characteristic parameters that satisfy formula (33):

[0183] in, It is the material characteristic parameter.

[0184] In S1003 , the material of the object is identified based on the material feature parameters.

[0185] In one implementation, the material parameters and the material of the object may be learned and trained based on a support vector machine (SVM) classifier.

[0186] In the disclosed embodiment, material characteristic parameters that are not affected by the size of the object are determined based on the first CSI data collected by each of the two antennas. Since the material characteristic parameters can characterize the correlation between material characteristics and signal amplitude and signal phase changes, the material of the object can be accurately and reliably determined based on the material characteristic parameters.

[0187] In another implementation, in combination with FIG3 , as shown in FIG11 , in the above S303 , acquiring perception information of a target object in the region of interest based on the first CSI data includes S1101 and S1102 .

[0188] In S1101 , Doppler frequency shift characteristic information of a target object is determined based on a first CSI.

[0189] In some embodiments, the on-board device extracts Doppler shift characteristic information of the target object from the time series samples of the first CSI. For example, the on-board device performs eigendecomposition on the covariance matrix corresponding to the time series samples of the first CSI to obtain a signal subspace corresponding to the maximum eigenvalue and a noise subspace corresponding to the minimum eigenvalue. The on-board device then performs parameter estimation based on the orthogonality between the signal subspace and the noise subspace to obtain the Doppler shift characteristic information.

[0190] In S1102, the Doppler frequency shift feature information of the target object is input into the perception information determination model to obtain the perception information of the target object.

[0191] In one implementation, the perception information determination model is obtained based on transfer learning or reinforcement learning.

[0192] Reinforcement learning can be used to update the perception information determination model each time it determines perception information. Through the Markov decision process, the perception information determination model can quickly identify the perception information of frequently identified target objects.

[0193] Based on the transfer learning and the trained perception information determination model, the trained perception information determination model can determine the perception information of the target object that has not been identified.

[0194] In some embodiments, a perception information determination model applied to a new task can be generated based on transfer learning and the perception information determination model. The perception information determination model applied to the new task consists of the feature extraction part of the original perception information determination model and a fully connected layer suitable for the new task.

[0195] In an exemplary implementation, a perceptual information determination model applied to a new task may be generated based on transfer learning and a perceptual information determination model based on a deep Q network (DQN).

[0196] Based on the deep Q network, the loss function used in the model for generating perceptual information applied to the new task satisfies the following formula (34):

[0197] in, is the Q function, are the parameters carried by the Q function itself, is the parameter that is continuously updated during the learning process, i is the number of model iterations during the learning process, and L i (θ i ) is the loss amount.

[0198] In the embodiment of the present disclosure, based on the Doppler frequency shift characteristic information of the target object and the perception information determination model, more accurate and reliable perception information of the target object can be obtained.

[0199] In one implementation of the embodiment of the present disclosure, the sensing frequency band of the above-mentioned Wi-Fi signal can be a centimeter wave band (below 30 GHz, such as 2.4 GHz, 5 GHz) or a millimeter wave band (above 30 GHz).

[0200] When the sensing frequency band is the millimeter wave band (such as 40GHz), no antenna is required and the target object can be located only by the smart metasurface.

[0201] It should be understood that when the sensing frequency band is the millimeter wave band, the electromagnetic waves between the smart metasurface and the target object satisfy the propagation laws in the near field. In this case, the connection between the propagation angle of the electromagnetic wave and the propagation distance is closer (there is a mapping relationship between the propagation distance and the propagation angle), that is, the distance between the target object and the smart metasurface can be obtained based on the propagation direction of the electromagnetic wave between the smart metasurface and the target object, thereby obtaining the location information of the target object.

[0202] In some embodiments, the onboard device divides the reflective area of ​​the smart metasurface into zones, and the smart metasurface then reflects based on these zones. This allows the onboard device to estimate parameters related to the smart metasurface, such as azimuth and pitch angle, based solely on a single reflection, rather than relying on time-varying phase reflections. In some embodiments, the onboard device can be located on the interior of the vehicle cabin, or on the roof of the cabin.

[0203] In some embodiments, the intelligent metasurface of the vehicle-mounted equipment is implemented using a transparent material and can be installed on the windshield.

[0204] In some embodiments, the smart metasurface can be made of flexible material and can be installed by applying or the like.

[0205] In one implementation method provided in an embodiment of the present disclosure, the on-board device controls the intelligent metasurface to reflect the Wi-Fi signal to the area of ​​interest. There are multiple reference points in the area of ​​interest, and the phase and intensity of the Wi-Fi signal reflected by each reflection unit in the intelligent metasurface are different. Therefore, the received signal strength indication (RSSI) received by each test point in the area of ​​interest is different. The on-board device collects the received signal strength indication received by each test point in the area of ​​interest through an antenna as a fingerprint data set of the area of ​​interest. In addition, the on-board device queries the fingerprint database based on the fingerprint data set input of the area of ​​interest to obtain the perception information of the target object in the area of ​​interest. The fingerprint database is generated based on the fingerprint recognition model, and the fingerprint recognition model is obtained based on the neural network model.

[0206] In some embodiments, the vehicle-mounted device also inputs the fingerprint dataset of the region of interest into a reference point determination model (the similarity determination model can be a deep learning model) to obtain the reference point of each target point in the region of interest. The similarity between the reference point of a target point and the target point is relatively high. The vehicle-mounted device processes the reference point of each target point based on the K-Nearest neighbor (KNN) algorithm to obtain the location of the target object in the region of interest.

[0207] In one implementation of the disclosed embodiment, the vehicle-mounted device employs different sensing strategies during different vehicle trial phases. While driving, the device monitors the driver and detects any threats to driving safety. While parked, the device detects whether anyone or any items have been left behind in the vehicle. While parked, the device continues operating for a preset delay and completes the sensing strategy based on backup power.

[0208] In some embodiments, the user can control the preset duration of the delayed operation of the vehicle-mounted device based on the terminal.

[0209] It is understandable that, in order to realize the above functions, the sensing device includes hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the algorithm steps of each example described in the embodiments of the present disclosure, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present disclosure.

[0210] The embodiment of the present disclosure can divide the functional modules of the sensing device according to the above-mentioned method embodiment. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one functional module. The above-mentioned integrated module can be implemented in the form of hardware or software. It should be noted that the division of modules in the embodiment of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation. The following is an example of dividing each functional module corresponding to each function.

[0211] Figure 12 is a schematic diagram of the structure of a Wi-Fi signal sensing device according to an embodiment of the present disclosure. The sensing device can execute the sensing method provided in the above method embodiment. As shown in Figure 12, the sensing device 120 includes: a control module 1201, a collection module 1202, and an acquisition module 1203.

[0212] The control module 1201 is used to control the smart metasurface to reflect Wi-Fi signals to the area of ​​interest of the target object in the vehicle cabin.

[0213] The acquisition module 1202 is configured to acquire first CSI data of the region of interest through the antenna.

[0214] The acquisition module 1203 is configured to acquire perception information of the target object in the region of interest based on the first CSI data.

[0215] In the case of implementing the functions of the above-mentioned integrated modules in hardware, the embodiments of the present disclosure provide another structure of the electronic device involved in the above-mentioned embodiments. As shown in Figure 13, the electronic device 130 includes: a processor 1302 and a bus 1304. In some embodiments, the electronic device may also include a memory 1301. In some embodiments, the electronic device may also include a communication interface 1303.

[0216] The processor 1302 may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of the present disclosure. The processor 1302 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof, and may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of the present disclosure. The processor 1302 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP (digital signal processor) and a microprocessor, and the like.

[0217] The communication interface 1303 is used to connect to other devices via a communication network, such as Ethernet, wireless access network, or wireless local area network (WLAN).

[0218] The memory 1301 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0219] As an implementation, memory 1301 may exist independently of processor 1302. Memory 1301 may be connected to processor 1302 via bus 1304 to store instructions or program codes. When processor 1302 calls and executes the instructions or program codes stored in memory 1301, the perception method provided in the embodiments of the present disclosure can be implemented.

[0220] In another implementation, the memory 1301 may also be integrated with the processor 1302 .

[0221] Bus 1304 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 1304 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, FIG13 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0222] Some embodiments of the present disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) having computer program instructions stored therein. When the computer program instructions are executed on a computer, the computer executes the perception method described in any of the above embodiments.

[0223] Exemplarily, the above-mentioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memories (EPROMs), cards, sticks, or key drives, etc.). The various computer-readable storage media described in the present disclosure may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0224] The embodiments of the present disclosure provide a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the perception method described in any one of the above embodiments.

[0225] In the disclosed embodiment, an on-board device controls the intelligent metasurface to reflect Wi-Fi signals to an area of ​​interest of a target object within the vehicle cabin, thereby enhancing the signal strength of the Wi-Fi signal scattered by the target object. The antenna then collects first CSI data of the area of ​​interest, which contains characteristic information of the target object. Thus, the on-board device can obtain perceptual information of the target object within the area of ​​interest based on the first CSI data. Furthermore, because the signal strength of the Wi-Fi signal scattered by the target object is relatively strong, the characteristic information of the target object contained in the first CSI data is more accurate, enabling the on-board device to more accurately obtain the perceptual information of the target object and obtain higher-precision perceptual information of the target object.

[0226] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present disclosure shall be covered by the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. A Wi-Fi signal sensing method, applied to an in-vehicle device, wherein the in-vehicle device includes an antenna and a smart metasurface; the method comprises: controlling the smart metasurface to reflect Wi-Fi signals to an area of ​​interest of a target object within a vehicle cabin; collecting first channel state information (CSI) data of the region of interest through the antenna; Acquire perception information of a target object within the region of interest based on the first CSI data.

2. The method according to claim 1, wherein The controlling the smart metasurface to reflect the Wi-Fi signal to the region of interest of the target object in the vehicle cabin includes: Determining location information of the region of interest, and determining first beam parameters of the smart metasurface based on the location information of the region of interest, wherein the first beam parameters are used to enable the reflected beam of the smart metasurface to cover the region of interest; Based on the first beam parameter of the smart metasurface, the smart metasurface is controlled to reflect the Wi-Fi signal to the area of ​​interest of the target object in the vehicle cabin.

3. The method according to claim 2, wherein: The determining the location information of the area of ​​interest includes: controlling the smart metasurface to reflect Wi-Fi signals to each seating area in the vehicle cabin based on a second beam parameter of the smart metasurface, wherein the second beam parameter is used to enable the reflected beam of the smart metasurface to cover each seating area in the vehicle cabin; collecting second CSI data in the vehicle cabin through the antenna; Target object detection is performed based on the second CSI data to determine a seat area where the target object exists, and the position information of the seat area where the target object exists is used as the position information of the region of interest.

4. The method according to claim 3, wherein: The detecting the target object based on the second CSI data to determine the seat area where the target object exists includes: obtaining, based on the second CSI data, second frequency domain characteristic information, where the second frequency domain characteristic information is used to characterize a channel frequency domain characteristic of the vehicle cabin in a current scenario; Determining a target area based on the second frequency domain characteristic information and the third frequency domain characteristic information; wherein the third frequency domain characteristic information is used to characterize the channel frequency domain characteristics of the vehicle cabin in an empty cabin environment; and the target area is an area where the channel frequency domain characteristics in the current scenario and the channel frequency domain characteristics in the empty cabin environment differ; The seating area where the target area is located is determined as the seating area where the target object exists.

5. The method according to claim 3, further comprising: When the vehicle cabin is empty, collecting third CSI data in the vehicle cabin through the antenna; determining, based on the third CSI data, location information of each seating area in the vehicle cabin and location information of a Wi-Fi device; The second beam parameters are determined based on the position information of each seating area in the vehicle cabin and the position information of the Wi-Fi device.

6. The method according to claim 1, wherein The smart metasurface comprises a plurality of smart metasurfaces, and controlling the smart metasurface to reflect the Wi-Fi signal to the region of interest of the target object in the vehicle cabin includes: Determining a degree of matching between each of the plurality of smart metasurfaces and the region of interest, the degree of matching being used to characterize a coverage effect of a reflected beam of the smart metasurface on the region of interest; Based on the matching degree between each of the multiple smart metasurfaces and the region of interest, determining a target smart metasurface that meets a preset matching condition from the multiple smart metasurfaces; The target intelligent metasurface is controlled to reflect the Wi-Fi signal to the region of interest of the target object in the vehicle cabin.

7. The method according to claim 1, wherein The target object is an object, and the perception information includes a material. Acquiring the perception information of the target object within the region of interest based on the first CSI data includes: Determine, based on first CSI data collected by each of the two antennas, a signal phase difference and a signal amplitude entropy between the two antennas; determining a material characteristic parameter based on the signal phase difference and the signal amplitude entropy, wherein the material characteristic parameter is not affected by the size of the object; Based on the material characteristic parameters, the material of the object is identified.

8. The method according to claim 1, wherein The acquiring, based on the first CSI data, the perception information of the target object within the region of interest includes: determining Doppler frequency shift characteristic information of the target object based on the first CSI data; The Doppler frequency shift characteristic information of the target object is input into a perception information determination model to obtain the perception information of the target object.

9. The method according to claim 1, further comprising: When it is identified based on the perception information of the target object that the vehicle has an abnormal parking event after parking, an alarm message is issued.

10. The method according to claim 1, wherein The target object includes a person, and the method further includes: When it is identified based on the perception information of the person that the person is engaging in dangerous behavior, at least one of the following operations is performed: issuing a warning message, activating a personal protection device, instructing the vehicle control system to perform automatic driving, and instructing the vehicle control system to brake.

11. The method according to claim 1, wherein The target object includes an item, and the method further includes: When the object is identified as a dangerous object based on the perception information of the object, at least one of the following operations is performed: issuing an alarm message, activating a personal protection device, instructing the vehicle control system to perform automatic driving, and instructing the vehicle control system to brake.

12. The method according to claim 1, further comprising: Sending the perception information of the target object to a vehicle-mounted system or terminal; the vehicle-mounted device is communicatively connected to the vehicle-mounted system or the terminal; wherein the vehicle-mounted system includes one or more of the following: a vehicle-mounted monitoring system, a vehicle-mounted safety control system, a vehicle-mounted infotainment system, and a remote information sending system.

13. The method according to claim 1, wherein The Wi-Fi signal reflected by the smart metasurface is sent by a Wi-Fi device, and the Wi-Fi device is a Wi-Fi device fixedly installed in the vehicle cabin or a portable Wi-Fi device.

14. The method according to claim 1, wherein The sensing frequency band of the Wi-Fi signal is a centimeter wave band or a millimeter wave band.

15. An electronic device comprising: A memory and a processor; wherein the memory is coupled to the processor; the memory is used to store instructions executable by the processor; and when the processor executes the instructions, the method according to any one of claims 1-14 is performed.

16. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, which, when executed on a computer, enable the computer to perform the method according to any one of claims 1 to 14.

17. A computer program product, wherein: The computer program product comprises computer program instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 14.

Images