Signal generation method, equipment, cluster, medium, program product and device

Abstract

The invention discloses a signal generation method, equipment, a cluster, a medium, a program product and a device, and belongs to the technical field of Internet of Things sensing and signal generation, and the method specifically comprises the following steps: designing a general millimeter wave sensing signal generation model based on a physical propagation path; respectively simulating a human reflection signal, an environment reflection signal and a multipath reflection signal by using a general millimeter wave sensing signal generation model, and constructing a full-scene signal synthesis model; and inputting environment data and an RGB image into the full-scene signal synthesis model, and generating a millimeter wave signal by using the generated 3D human body grid model, the environment grid data and the environment reflection characteristic information. According to the method, physical modeling and deep learning are combined, high efficiency, flexibility and accuracy of millimeter wave signal generation are guaranteed, the method has better performance than the prior art, and on the premise that generalization ability of a signal generation framework is guaranteed, quality and reliability of signal generation and application are greatly improved.

Description

Background Art

[0002] In recent years, due to its high-resolution characteristics, millimeter-wave technology has been widely used in fields such as intelligent transportation, security monitoring, smart home, and industrial Internet of Things. However, most of the existing millimeter-wave signal generation methods rely on expensive hardware devices and cannot meet the requirements of low cost, high flexibility, and high efficiency. With the rapid development of artificial intelligence and deep learning technologies, software-based millimeter-wave signal generation methods have become a research hotspot. Especially in complex scenarios, the transmission and reflection of millimeter-wave signals are affected by factors such as the environment, human movements, and obstacles. How to accurately simulate and generate these signals is a key technical problem faced by millimeter-wave radar systems.

[0003] Currently, there are some millimeter-wave signal generation methods based on deep learning, which usually require a large amount of labeled data for training. However, the acquisition cost of labeled data is high, and there are problems of large differences in data distribution, which limits the generalization ability of the model. In addition, most of the existing technologies are difficult to balance the diversity of signal generation and the high quality of the generated signals. In order to effectively improve the accuracy and robustness of signal generation, how to construct an efficient millimeter-wave signal generation framework using environmental information and multipath propagation models has become an important research direction. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a signal generation method, device, cluster, medium, program product, and apparatus for efficiently generating high-quality millimeter-wave signals and applying them to intelligent sensing and activity recognition tasks, so as to solve the technical problems that millimeter-wave signal generation depends on hardware devices, lacks an efficient software model, cannot fully simulate complex environmental factors, and cannot fully utilize environmental and reflection information for signal optimization.

[0005] The present invention adopts the following technical solutions: In the first aspect, a signal generation method is provided, including the following steps: Design a general millimeter-wave sensing signal generation model based on physical propagation paths; Use the general millimeter-wave sensing signal generation model to simulate human reflection signals, environmental reflection signals, and multipath reflection signals respectively, and construct a full-scene signal synthesis model; Input environmental data and RGB images into the full-scene signal synthesis model, and generate millimeter-wave signals using the generated 3D human body mesh model and environmental mesh data.

[0006] Preferably, the reflection signal of the full-scene signal synthesis model is:

[0007] Among them, is the human body reflection signal, is the environmental reflection signal, is the multipath reflection signal.

[0008] Preferably, the human body reflection signal is:

[0009] Among them, is the number of reflecting surfaces, is the direct reflection signal of the i-th reflecting surface, is the amplitude coefficient of the i-th reflecting surface, is the phase of the reflection signal of the i-th reflecting surface, is the imaginary unit, is the natural logarithm.

[0010] Preferably, the amplitude coefficient of the environmental reflection signal is as follows:

[0011] Among them, is the simulated transmit antenna gain, is the simulated receive antenna gain, is the gain function related to the azimuth angle of the simulated transceiver antenna pair, is the azimuth angle where the i-th reflecting surface is located, is the gain function related to the elevation angle of the simulated transceiver antenna pair, is the elevation angle where the i-th reflecting surface is located, is the area of the i-th reflecting surface, is the amplitude coefficient of the outgoing wave in the direction of the receive antenna of the i-th reflecting surface, is the gain coefficient of the surface material of the i-th reflecting surface for the amplitude, is the square of the distance between the i-th reflecting surface and the simulated transceiver antenna pair.

[0012] Preferably, the amplitude of the multipath reflection signal is as follows:

[0013] Among them, are all the products of the corresponding coefficients of the two reflecting surfaces, is the wavelength, is the transmit power, is the square of the distance between the i-th reflecting surface and the simulated transceiver antenna pair.

[0014] In a second aspect, an embodiment of the present invention provides a signal generation device, including: A construction module that designs a general millimeter-wave sensing signal generation model based on the physical propagation path; A simulation module that uses a general millimeter-wave sensing signal generation model to simulate human reflection, environmental reflection, and multipath signals respectively, and constructs a complete scene signal synthesis model; A generation module that inputs environmental data and RGB images into the scene signal synthesis model, and uses the generated 3D human body mesh model and environmental mesh data to generate millimeter-wave signals.

[0015] In a third aspect, an electronic device is provided. The electronic device includes a processor and a memory, and computer instructions are stored on the memory. When the computer instructions are executed by the processor, the electronic device performs the actions of the method according to the first aspect or any of its embodiments above.

[0016] In a fourth aspect, a computing device cluster is provided. The computing device cluster includes at least one computing device, and each computing device includes a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the operations of the method according to the first aspect or any of its embodiments above.

[0017] In a fifth aspect, a computer-readable storage medium is provided. Computer-executable instructions are stored on the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, the operations of the method according to the first aspect or any of its embodiments above are implemented.

[0018] In a sixth aspect, a computer program or a computer program product is provided. The computer program or the computer program product is tangibly stored on a computer-readable medium and includes computer-executable instructions, and when the computer-executable instructions are executed, the operations of the method according to the first aspect or any of its embodiments above are implemented.

[0019] Compared with the prior art, the present invention has at least the following beneficial effects: A signal generation method generates signals by modeling the scenario (only through snapshots of the environment), without relying on complex real-time environment acquisition and annotation. It solves the adaptation problems of traditional methods in multi-scenario and dynamic environments and can adaptively adjust the signal generation method in different scenarios. This feature enables this method to provide a more flexible millimeter-wave signal generation scheme in various practical application scenarios; the generated signals can more accurately restore the signal propagation characteristics in complex environments through multi-path propagation such as simulating human body reflection and environmental reflection. Traditional methods usually only model a single reflection path, while this method can integrate multiple reflection paths and environmental factors to generate high-quality millimeter-wave signals closer to the actual application requirements; by fully softwareizing the process to simulate human body reflection, environmental reflection, and multi-path reflection signals, it significantly reduces hardware dependence, lowers costs, and improves the flexibility and efficiency of signal generation; it can generate millimeter-wave signals that conform to the actual situation in various complex environments and can adapt to the signal characteristics in different scenarios. Existing methods often cannot meet the adaptation requirements of multiple scenarios simultaneously, while the present invention can handle different scenarios and environments through a unified generation framework, greatly enhancing the universality and flexibility of the system.

[0020] Furthermore, it eliminates the dependence on a large amount of labeled data in the deep learning-based approach. By combining environmental snapshots and a signal generation model, it can still obtain high-quality signal generation results with a small amount of labeled data or no labeled data at all. In contrast, existing deep learning-based signal generation methods usually require a large amount of manually labeled data to train deep learning models, thus increasing the data annotation cost and making it difficult to generalize to different environments.

[0021] Furthermore, it can flexibly handle different dynamically changing environments. Traditional hardware devices usually can only work in a predetermined environment, while this method can simulate the propagation of millimeter-wave signals in any environment based on a simplified physical model and propagation model, with stronger adaptability and can provide more accurate signal generation in dynamically changing environments.

[0022] It can be understood that the beneficial effects of the second to sixth aspects above can refer to the relevant descriptions in the first aspect above and will not be elaborated here.

[0023] In summary, the present invention combines physical modeling and deep learning to achieve the high efficiency, flexibility, and accuracy of millimeter-wave signal generation, has more excellent performance than the prior art, and greatly improves the quality and reliability of signal generation and application on the premise of ensuring the generalization ability of the signal generation framework.

[0024] The technical solution of the present invention will be further described in detail below through the accompanying drawings and embodiments. Description of the Drawings

[0025] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments of the present application. Obviously, the following described drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0026] Figure 1 Shows the overall structure diagram of the full-dimensional definable millimeter-wave sensing signal generation method; Figure 2 Shows the schematic diagram of the chirp-type millimeter-wave signal and the calculation dimension schematic diagrams of Range-FFT, Doppler-FFT, and Angle-FFT of the original millimeter-wave signal; Figure 3 Shows the schematic diagram of the signal modeling module calculating the influence of the reflection plane orientation on the signal intensity; Figure 4 Shows the schematic diagram of the signal modeling module calculating the signal secondary reflection and multi-path tracking; Figure 5 Shows the qualitative comparison result diagram of the millimeter-wave data captured in real time when the experimenter walks and the generated data in a complex environment; Figure 6 Shows the quantitative comparison result diagram of the millimeter-wave data generated and the millimeter-wave data characteristics spectrogram captured in real time in each scenario and action, where (a) is the RA signature and (b) is the MD signature; Figure 7 Shows the experimental result diagram of using the generated millimeter-wave data for DNN training in each scenario and action and performing action classification tests on the data captured in real time; Figure 8 Shows the schematic block diagram of an example device that can be used to implement the embodiments of the present disclosure; Figure 9 Shows the schematic block diagram of an example computing device cluster that can be used to implement the embodiments of the present disclosure; Figure 10 Shows the schematic block diagram of an example implementation manner of a computing device cluster that can be used to implement the embodiments of the present disclosure. Detailed implementation manners

[0027] Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the protection scope of the present application.

[0028] In the description of the present invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0029] It should also be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0030] It should be further understood that the term "and / or" used in the specification of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the present invention, the character " / " generally represents an "or" relationship between the preceding and following related objects.

[0031] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present invention to describe preset ranges, etc., these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range can also be referred to as the second preset range, and similarly, the second preset range can also be referred to as the first preset range.

[0032] Depending on the context, the word "if" as used herein can be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detecting (stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)".

[0033] Various structural schematic diagrams according to the disclosed embodiments of the present invention are shown in the accompanying drawings. These figures are not drawn to scale, where certain details are enlarged for the purpose of clear expression, and certain details may be omitted. The shapes of various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary. In practice, there may be deviations due to manufacturing tolerances or technical limitations, and those skilled in the art can design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0034] The present invention provides a signal generation method. By constructing a comprehensive physical signal transmission model, it can efficiently generate millimeter-wave signals that conform to the actual environment without the need for labeled data, and has good cross-scene adaptability, significantly improving the authenticity and quality of signal generation and the adaptability of the system; through the software-based method of physical modeling of signal transmission, high-quality millimeter-wave signals are generated and more extensive applications are realized; it not only reduces the dependence on hardware, but also can accurately simulate the transmission and reflection characteristics of millimeter-wave signals in various environments, and has broad application prospects.

[0035] Embodiment 1 Please refer to Figure 1 , a signal generation method of the present invention generates high-quality millimeter-wave sensing signals in a software form by modeling the physical propagation path of radio frequency signals; comprehensively considers the advantages and disadvantages of deep learning methods and traditional physical modeling methods. While using depth camera snapshots and pre-trained object recognition networks to provide scene information, according to the high directivity and strong attenuation characteristics of millimeter-wave sensing signals, the multi-path tracking algorithm in traditional simulation algorithms is simplified to achieve a balance between high efficiency and high quality of signal generation; the signal generation in the scene is divided into three modules: human reflection synthesis, environmental reflection synthesis, and multi-path reflection synthesis. Each module considers the influencing factors of the material, orientation, and angle of the reflecting surface; finally, an action classification network with practical use significance is trained with the generated data, and it shows good classification effects on the actually captured sensing data set. The specific implementation details include the following steps: S1. Design a general millimeter-wave sensing signal generation model based on the physical propagation path; Please refer to Figure 1 , Figure 1 The left side is a schematic diagram of the overall signal generation framework, and the right side is a schematic diagram of two methods for providing human body mesh information for the signal generation framework and a schematic diagram of the signal quality evaluation method.

[0036] The signal generation framework is divided into three modules: the human signal reflection module, the environmental signal reflection module, and the multipath reflection module. Each module is based on the input environmental grid and other auxiliary information, and simulates its propagation results by simulating the signal propagation method and path. Considering different influences during the signal propagation process, such as the material of the reflection surface, the orientation of the reflection surface, and the azimuth of the reflection surface, etc., a high-fidelity millimeter-wave signal result is generated. Two schemes for preparing signal generation are shown on the right.

[0037] The present invention is a general signal generation structure, so it allows users to select the human body grid obtained by various methods such as using a depth camera or text generation as the input of the signal generation model. The right lower part shows the quantitative analysis of the quality of the generated signal by the present invention, and uses the result that it can ensure the ability of the action classification model in the downstream task to prove the characteristic of high quality of the generated data.

[0038] S2. Through physical modeling, simulate human reflection, environmental reflection, and multipath signals to construct a complete scene signal synthesis model; Please refer to Figure 2 , the left side is a schematic diagram of the chirp of the intermediate-frequency millimeter-wave signal. Transmitted signal:

[0039] Received signal after reflection:

[0040] Obtained through simulated mixing:

[0041] That is, a sine signal with a fixed frequency, and the frequency and phase are both related to the reflection surface is obtained.

[0042] The common purpose of the subsequent three modules is to simulate the intermediate-frequency millimeter-wave signal in the virtual space that starts from the radar antenna and is reflected back to the receiving antenna by any reflection surface.

[0043] S201. Model the human reflection signal, and denote the direct reflection signal of the reflection surface i as , where is the amplitude coefficient, is related to the position of the reflection surface from the virtual radar antenna:

[0044] where is the starting frequency, B is the bandwidth, is the duration of the chirp, is the simulated flight time of the millimeter-wave signal from transmission to reception, D is the distance between the virtual radar antenna and the reflection surface, and c is the speed of light.

[0045] For the amplitude coefficient we have:

[0046] where is the transmitting and receiving gain of the antenna, is the wavelength, P is the transmitting power, is the area of the reflector.

[0047] Furthermore, please refer to Figure 3 , for , considering the influence of the reflector orientation on the amplitude of the reflected signal according to the principle of quasi-specular reflection, that is, the intensity of the outgoing wave approximately conforms to a Gaussian distribution related to the angle. After the incident wave undergoes quasi-specular reflection, the outgoing direction vector with the strongest amplitude is , and the included angle between the receiving antenna direction vector and will be used as a parameter to calculate the amplitude of the outgoing wave towards the receiving antenna:

[0048] where η is an empirical value, is the unit normal vector of the reflector, is the incident vector. Based on the above formula, the model can calculate the signal amplitude caused by the orientation of any reflector, thereby enhancing the authenticity of the finally generated signal.

[0049] Furthermore, for , to consider the influence of different reflector materials on the millimeter-wave reflection, the basic principle of millimeter-wave reflection on different reflector materials conforms to the Fresnel reflection principle. Considering the influence of horizontal and vertical polarization, the horizontal and vertical polarization coefficients are modeled as follows:

[0050] where is the incident angle, is the complex dielectric constant, and its modeling method is , is the relative dielectric constant, is the conductivity. The final model is as follows:

[0051] where and are unit vectors in the parallel and perpendicular polarization directions. The reflection surface of different materials has a great influence on the amplitude of millimeter waves, especially when the material crosses metal and non-metal, so it is very important to model the reflection surface material to optimize the generation of millimeter wave signals.

[0052] When counting the reflection surfaces in the whole scene, the hidden point removal algorithm is used to calculate the set of reflection surfaces that can reflect millimeter waves, and the number of reflection surfaces is denoted as , then the final modeled human body reflection signal is:

[0053] S202. Model the environmental reflection signal in the sensing scene; The basic modeling of the environmental reflection signal is the same as that of the human body reflection signal, and additional modeling is carried out on . Denote and as the azimuth angle and elevation angle of the environmental reflection surface relative to the radar antenna. According to the millimeter wave radar user guide, we model as , that is, a Gaussian distribution related to the angle of the reflection surface, so as to enhance the accuracy of the generated signal.

[0054] The amplitude coefficient of the final environmental reflection signal is as follows:

[0055] S203. Further, please refer to Figure 4 , which shows the reflection path when simulating multipath signals. Different from the traditional multipath tracking algorithm, considering the dual characteristics of strong directivity and fast attenuation of millimeter waves, the present invention models the reflection planes in the environment that have a high probability of generating secondary reflections, and calculates the secondary reflection signals generated by these planes in the primary signal generation, so as to replace the omnidirectional multipath tracking reflection signals, balance the accuracy and time cost of signal generation, and accurately generate millimeter wave sensing signals in dynamic scenes.

[0056] In the figure, is the direction vector of the signal directly reflected back to the receiving antenna by the human body reflection surface (the center point is ) at the current moment, is the strongest reflection direction vector of the human body reflection surface relative to the incident signal, and this direction is modeled as the incident direction of the secondary reflection. Considering any other reflection surface at the current moment, denote the center of this surface as . If the following equation has a solution, it means that the secondary incident direction can be incident on the sphere with as the center and a radius of :

[0057] That is, it is considered that the secondary reflected wave can pass through reflection and return to the receiving antenna Rx. Among them, is selected as the inradius of the reflecting surface. The present invention records this propagation path as a high-probability reflection path and saves it as a list in the memory, see Figure 4 on the right. As the relative positions of the human body and the environment in the scene change dynamically, the high-probability reflection path list can be modified to flexibly generate high-fidelity multipath signals.

[0058] For the amplitude of the multipath reflection signal, as shown in the formula:

[0059] where, are all the products of the corresponding coefficients of the two reflecting surfaces.

[0060] Finally, the reflection signals of the entire scene are modeled as the superposition of the human body reflection signal, the environment reflection signal, and the multipath reflection signal:

[0061] In addition, in order to simulate real commercial millimeter-wave radar signals, the present invention simultaneously models and generates the sensing signals of multiple virtual transceiver antenna pairs to enhance the practicality of the generated signals.

[0062] S3. By inputting the environmental data and RGB images collected by Kinect, using the generated 3D human body mesh model and environmental mesh data, generate millimeter-wave signals; To improve the versatility and generalization of the present invention, a mesh model is used as the input modality of the present invention. In the method of the present invention, a Kinect depth camera is used to take a single snapshot of the experimental scene, and the environmental point cloud is obtained using its depth information and further converted into a mesh model; combined with the pre-trained deep learning model VoteNet, 3D object recognition is performed on the environmental point cloud. After identifying and judging the material of each object, as described in step S2, the mesh information of the scene together with the material information is input into the signal generation model of the present invention to generate environmental reflection signals.

[0063] Furthermore, in order to expand the usage scenario of the present invention, two methods for inputting the human body mesh model are provided.

[0064] One is to generate the human body mesh by using the RGB image of Kinect and the pre-trained deep learning model Hand4Whole; The other is to learn the characteristics of the human body mesh from the text description by using the pre-trained diffusion model MDM and generate a continuously changing sequence of human body meshes.

[0065] The results of both of these two modes are 3D human body meshes, which are the same as the environmental information input into the present invention, thus ensuring the feasibility of calculating multi-path reflected signals.

[0066] S4. By processing and matching features such as Range-FFT, Range-Doppler, and Range-Angle of the millimeter-wave signals generated in step S3, quantitatively evaluate the quality of the generated signals; S401. Please refer to Figure 2 the right side. Perform FFT transformation on the three dimensions of the millimeter-wave signal respectively, and time-frequency characteristic spectrograms corresponding to the actual distance, speed, and angle can be obtained. These three-dimensional spectrograms are used to visually represent the human activity state in the current scene. Please refer to Figure 5 , which is the comparison result between the generated signal and the real signal when the experimenter walks in a complex scene. The high spectrogram similarity proves the adaptability of the present invention to dynamic scenes. The upper part is the three-dimensional spectrogram features of the millimeter-wave data actually collected at the current moment, and the rightmost part is the RGB image of the real perception scene. The lower part is the three-dimensional spectrogram features of the generated signal at the current moment, and the rightmost part is the schematic diagram of the environment, human body mesh, etc. input into the signal generation module of the present invention in the virtual space.

[0067] S402. In order to quantitatively evaluate the verisimilitude between the generated signal and the real signal, perform multi-scale similarity analysis (MS-SSIM) on the spectrograms at the same moment. Among them, in order to unify the Range-Doppler signals in a dynamic scene onto one spectrogram for comparison, the method of matrix compression is used to obtain the micro-Doppler spectrogram that maps the change in human body speed. Please refer to Figure 6 , in multiple actions and scenes, the generated data and the real data always maintain a high similarity. Even in extremely complex dynamic scenes such as "jumping", the present invention still maintains a multi-scale similarity higher than 0.8, ensuring the usability of the generated data.

[0068] S5. Use the generated signals for downstream tasks, such as activity recognition, and perform training and optimization through the DNN network to verify the effectiveness of signal generation.

[0069] During verification and evaluation, a commercially available millimeter-wave radar TI-IWR6843isk and a millimeter-wave development evaluation board DCA1000 are used to collect the actually captured millimeter-wave data set D1.

[0070] Based on the signal generation method of the present invention, two generated signal datasets D2 and D3 are constructed. The environmental grids of D2 and D3 are provided by the snapshots of Kinect, and then millimeter-wave signals are generated through the generation method of the present invention. The human body grid information of D2 is derived from the RGB information of Kinect, and is generated and input into the signal generation module with the help of the pre-trained network as described in step S3. The human body grid information of D3 is derived from the pre-trained diffusion model, generating a diverse sequence of continuous human body grids, which is input into the generation module of the present invention to obtain a rich millimeter-wave signal dataset.

[0071] Furthermore, the signals that are temporally aligned in D2 and D1 are used to verify the accuracy of the present invention, and the performance of D1 and D3 in the training of the deep learning model is used to verify the practicality of the present invention.

[0072] S501. Construct a human body grid library with 6 different dynamic actions, and then randomly initialize the morphological parameters of the human body grid to enhance the richness of the final dataset D3. Each action lasts for 3 seconds, with 15 frames per second, generating a total of 48.6k frames of millimeter-wave data. These data will be used for the training of the classification network.

[0073] S502. Build a deep learning network for action classification evaluation. Considering that the final millimeter-wave signal will be converted into a two-dimensional spectrogram, a convolutional layer is selected as the feature extraction part, combined with a long short-term memory network for temporal information aggregation and analysis, and finally a multi-layer perceptron is used to complete the action classification. The loss function during training is set to cross-entropy loss, and the training dataset does not contain any real captured millimeter-wave data at all.

[0074] S503. When testing the classification network, input the real captured dataset in D1, report and evaluate its classification results, and verify that the millimeter-wave data generated by the present invention contains rich scene information and has high practicality.

[0075] Please refer to Figure 7 , on the left is the recognition accuracy of each of the 6 actions. In multiple experimental scenarios, the average recognition accuracy of each action is higher than 90%, and the accuracy of special actions can reach 100%. On the right, the three confusion matrices are for the recognition accuracy and confusion degree of each action in three experimental environments. Except for fluctuations in special actions, the accuracy of all actions basically reaches 95%, fully demonstrating the accuracy and practicality of the millimeter-wave signals generated by the present invention.

[0076] Those skilled in the art can understand that various aspects of the present invention can be implemented as a system, a method, or a program product. Therefore, various aspects of the present invention can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "platform" here.

[0077] Embodiment 2 The present invention provides a signal generation device, which can be used to implement the above-mentioned signal generation method. Specifically, the signal generation device includes a construction module, a simulation module, and a generation module.

[0078] Among them, the construction module designs a general millimeter-wave sensing signal generation model based on the physical propagation path; The simulation module uses the general millimeter-wave sensing signal generation model to simulate human reflection, environmental reflection, and multipath signals respectively, and constructs a complete scene signal synthesis model; The generation module inputs environmental data and RGB images into the scene signal synthesis model, and generates millimeter-wave signals by using the generated 3D human body mesh model and environmental mesh data.

[0079] Embodiment 3 An embodiment of the present disclosure also provides a computing device 900. As Figure 8 shown, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other through the bus 902. The computing device 900 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 900.

[0080] The bus 902 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 8 only one line is shown here, but it does not mean that there is only one bus or one type of bus. The bus 904 can include a path for transmitting information between various components (such as the memory 906, the processor 904, and the communication interface 908) of the computing device 900.

[0081] The processor 904 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a Tensor Processing Unit (TPU), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), a micro processor (MP), or a digital signal processor (DSP).

[0082] The memory 906 may include volatile memory, such as random access memory (RAM). The processor 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0083] The memory 906 stores executable program code, and the processor 904 executes the executable program code to implement the foregoing signal generation methods respectively. That is, the memory 906 may store instructions for the methods and functions related to the computing device 110 in any of the foregoing embodiments.

[0084] The communication interface 908 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or a communication network.

[0085] Embodiment 4 An embodiment of the present disclosure further provides a computing device cluster 1000. The computing device cluster includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0086] As Figure 9As shown, the computing device cluster includes at least one computing device 900. Instructions for performing the methods and functions of the computing device 110 involved in any of the foregoing embodiments may be stored in the memory 906 of one or more of the computing devices 900 in the computing device cluster.

[0087] In some possible implementation manners, partial instructions for performing the methods and functions of the computing device 110 involved in any of the foregoing embodiments may also be stored separately in the memory 906 of one or more of the computing devices 900 in the computing device cluster. In other words, a combination of one or more computing devices 900 may jointly execute the instructions for performing the methods and functions of the computing device 110.

[0088] It should be noted that the memories 906 in different computing devices 900 in the computing device cluster may store different instructions for respectively performing partial functions of the apparatus 800. That is, the instructions stored in the memories 906 of different computing devices 900 may implement the foregoing signal generation method.

[0089] In some possible implementation manners, one or more computing devices in the computing device cluster may be connected via a network. Among them, the network may be a wide area network or a local area network, etc. Figure 10 A possible implementation manner 1100 is shown. As Figure 10 shown, two computing devices 900A and 900B are connected via a network 1110. Specifically, they are connected to the network through the communication interfaces in each computing device. In this type of possible implementation manner, for example, instructions for performing the functions of the first generation module 810 and the second generation module 820 are stored in the memory 906 of the computing device 900A. At the same time, instructions for performing the function of the adjustment module 830 are stored in the memory 906 of the computing device 900B.

[0090] Figure 10 The connection manner between the computing device clusters shown may be considered in view of the fact that the method for the computing device 110 provided in this application may require storing a large amount of data and performing intensive calculations. Therefore, it is considered to hand over the function implemented by the adjustment module 830 to the computing device 900B for execution.

[0091] It should be understood that Figure 10 the functions of the computing device 900A shown in

[0092] Embodiment 5 Embodiments of the present disclosure also provide a computer-readable storage medium, on which computer instructions are stored. When the processor runs the instructions, the processor is caused to execute the methods and functions of the computing device 110 in any of the above embodiments.

[0093] In general, the various embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software, which may be executed by a controller, a microprocessor, or other computing devices. Although the various aspects of the embodiments of the present disclosure are shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented as, by way of non-limiting example, hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0094] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which are executed in a device on a target real or virtual processor to perform the processes / methods as referred to the accompanying drawings above. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functions of program modules may be combined or divided as needed. The machine-executable instructions for program modules may be executed within a local or distributed device. In a distributed device, program modules may be located in local and remote storage media.

[0095] The computer program code for implementing the methods of the present disclosure may be written in one or more programming languages. These computer program codes may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing devices, such that when the program codes are executed by the computer or other programmable data processing devices, the functions / operations specified in the flowcharts and / or block diagrams are caused to be implemented. The program codes may be executed entirely on the computer, partially on the computer, as a stand-alone software package, partially on the computer and partially on a remote computer, or entirely on a remote computer or server.

[0096] In the context of the present disclosure, the computer program code or related data may be carried by any suitable carrier, so that the device, apparatus, or processor can perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, acoustic, or other forms of propagated signals, such as carrier waves, infrared signals, etc.

[0097] A computer-readable medium can be any tangible medium that contains or stores a program for use in or related to an instruction execution system, apparatus, or device, or a data storage device such as a data center that contains one or more available media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More specific examples of computer-readable storage media include electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0098] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Generally, the components described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0099] Please refer to Figure 1 , which is a schematic diagram of the overall signal generation structure of the present invention. The present invention decouples the signal generation module, uses a single depth camera snapshot to capture scene information, combines the input human mesh model, and efficiently generates millimeter-wave sensing raw data that is close to reality and rich in information for the training of downstream deep learning networks, alleviating the problems of scarce quantity and poor quality of datasets in the field of wireless sensing.

[0100] Please refer to Figure 2 , which is a schematic diagram of a millimeter-wave signal representing the chirp type and a schematic diagram of the calculation dimensions of Range-FFT, Doppler-FFT, and Angle-FFT for the original millimeter-wave signal. The present invention models the signal generation module and constructs the original millimeter-wave signal starting from the generation model of the chirp signal.

[0101] Please refer to Figure 3, which shows a schematic diagram when the signal modeling module calculates the influence of the reflection plane orientation on the signal strength. In the present invention, the amplitude of the reflected millimeter wave is modeled as a quasi-specular reflection, and the amplitude coefficient and the angle of the orientation are modeled as an approximate Gaussian distribution. Therefore, the amplitude coefficient brought by the orientation angle belonging to the reflection surface is calculated according to each reflection plane. In addition, the present invention also models a plurality of coefficients affecting the amplitude gain, such as the material coefficient of the reflection surface, the angle gain coefficient of the reflection surface, and the gain coefficient of the transceiver antenna.

[0102] Please refer to Figure 4 , which shows a quadratic reflection path tracking method designed by the present invention for calculating multipath reflection and balancing accuracy and time consumption. Due to the high attenuation and high directivity of millimeter waves, when the present invention models the multipath reflection of millimeter waves, only the results of two reflections are considered. At the same time, for the selection of the two reflection paths, a path tracking method with a high probability of reflection is proposed to record the reflection surface information under a quadratic reflection path and adjust and maintain it in a dynamic scenario.

[0103] Please refer to Figure 5 and Figure 6 , the scene information converted from the depth camera information, and then millimeter wave data is generated, which can be considered theoretically consistent with the synchronously collected original millimeter wave data. By performing the same data processing, qualitative and quantitative analysis on the real millimeter wave data and the generated data, it can be seen that the generated data and the real data under this method have extremely high similarity, verifying the correctness and accuracy of this generation method.

[0104] Generally speaking, compared with the method based on deep learning, the signal generation based on physical modeling does not require too much preparatory work and is easier to generalize to different human bodies and scenarios after the modeling is completed, but the generation accuracy is not high. In order to overcome this shortcoming, the present invention considers a plurality of gain coefficients including the material of the reflection surface, the orientation of the reflection surface, the position of the reflection surface, and the gain of the transceiver antenna during the modeling process, making up for the disadvantages of the signal generation method based on physical modeling. At the same time, in order to further enhance the practicality of the present invention, the calculation idea of multipath reflection is extended in combination with the special properties of the generated signal, effectively reducing the computational overhead brought by multipath reflection tracking.

[0105] Please refer to Figure 7, through model evaluation, we can find that the signals generated by the present invention have high practical value. Using a deep learning model designed with classical convolutional neural networks and recurrent neural networks, trained only on generated data and tested on real data, the average classification accuracy can reach over 90% in various test environments, which means that the generated signals are not simple data imitations, but have the mapping relationship information with the physical world just like real data. This conclusion further shows that the present invention is of great significance for the problem of scarce data sets in the field of wireless sensing, and is expected to promote the further development of the millimeter-wave sensing field.

[0106] In summary, a signal generation method, device, cluster, medium, program product and apparatus of the present invention obtain the position, material information and mesh of any object in the scene, and the mesh information of any human action in the scene through a single depth camera snapshot and a pre-trained neural network; while modeling the signal reflection model of a single reflecting surface, designing and modeling multi-path reflection signals to ensure the practicality of the generated signals; fully utilizing the material, size, orientation and other information of each reflecting surface to model the amplitude gain of the reflection signal to ensure the accuracy of the generated signals; evaluating the multi-scale similarity between the generated signals and the real captured signals, training a classification model with the generated signals and testing on real signals, realizing the leap of using the generated signals for end-to-end model training, and finally realizing the generation of realistic millimeter-wave signals in any dynamic scene, providing new ideas for the data generation direction in the field of wireless sensing.

[0107] In addition, although the operations of the method of the present disclosure are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the illustrated operations must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart may be changed in the order of execution. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution. It should also be noted that the features and functions of two or more devices according to the present disclosure may be embodied in one device. Conversely, the features and functions of one device described above may be further divided and embodied by multiple devices.

[0108] The various implementations of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed implementations. Many modifications and variations are obvious to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described implementations. The selection of the terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technologies in the market, or to enable other ordinary skilled persons in the technical field to understand the various implementation manners disclosed herein.

Claims

1. A signal generation method, characterized in that: The following steps are involved: Design a general mmWave sensing signal generation model based on the physical propagation path; The universal millimeter wave perception signal generation model is used to simulate human reflection signals, environmental reflection signals, and multipath reflection signals, respectively, to build a full-scenario signal synthesis model. The environmental data and RGB images are input into the full-scene signal synthesis model, and the millimeter wave signal is generated using the generated 3D human mesh model and environmental mesh data.

2. The signal generation method according to claim 1, characterized in that: Reflected signal of the full-scene signal synthesis model for: in, The human body reflects the signal. is the environmental reflection signal, It is a multipath reflected signal.

3. The signal generating method according to claim 2, characterized in that: Human body reflection signal for: in, is the number of reflective surfaces, is the direct reflection signal of reflection surface i, is the amplitude coefficient of the reflection surface i, is the phase of the reflected signal from reflection surface i, is an imaginary unit, is the natural logarithm.

4. The signal generation method according to claim 2, characterized in that: The amplitude coefficient of the environmental reflection signal is as follows: in, To simulate the transmit antenna gain, To simulate the receive antenna gain, To simulate the gain function of the transmit and receive antenna pair related to the azimuth angle, is the azimuth of the reflecting surface i, To simulate the gain function of the transmit and receive antenna pair related to the elevation angle, is the elevation angle of the reflecting surface i, is the area of ​​the reflecting surface i, is the amplitude coefficient of the outgoing wave from the reflecting surface i toward the receiving antenna, is the gain coefficient of the surface material of the reflecting surface i to the amplitude, is the square of the distance between the reflecting surface i and the simulated transmitting and receiving antenna pair.

5. The signal generating method according to claim 2, characterized in that: Amplitude of multipath reflection signal as follows: in, are the products of the corresponding coefficients of the two reflection surfaces. is the wavelength, is the transmission power, is the square of the distance between the reflecting surface i and the simulated transmitting and receiving antenna pair.

6. A signal generating device, characterized in that: include: Building modules to design a general mmWave sensing signal generation model based on physical propagation paths; The simulation module uses a universal millimeter wave perception signal generation model to simulate human reflection, environmental reflection, and multipath signals to build a complete scene signal synthesis model. The generation module inputs the environmental data and RGB image into the scene signal synthesis model, and generates the millimeter wave signal using the generated 3D human mesh model and environmental mesh data.

7. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 5.

8. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 5.

9. A computer program product, characterized in that The computer program product contains computer executable instructions, which implement the method according to any one of claims 1 to 5 when executed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images