A method and apparatus for generating millimeter-wave radar data based on video.

Abstract

This invention discloses a method and apparatus for generating millimeter-wave radar data based on video. First, a variant twin network is used to select key data segments from the video. Then, a human region indexing algorithm is used to achieve a one-to-one matching between human mesh data and human position information. Next, a human mesh model and a depth prediction model are used to generate corresponding depth information, radar cross-section, and radial velocity. Subsequently, a multi-person reflection model is used to simulate multipath reflection and attenuation of radar signals, outputting convertible coarse radar data. Finally, a Transformer model is used to generate realistic radar data. This method solves the imbalance problem between different action categories, ensures the stability of the machine learning model used for human perception, resolves the problem of incorrect matching between depth information and multi-person mesh data, and addresses the multipath reflection and attenuation problems of radar signals during transmission and reception. It can be applied to downstream tasks related to human perception, such as human action recognition and target detection.

Description

Technical Field

[0001] This invention relates to the field of millimeter-wave radar sensing technology, and in particular to a method and apparatus for generating millimeter-wave radar data based on video. Background Technology

[0002] Millimeter-wave radar, as an emerging sensor, has been widely used in systems such as autonomous driving, intelligent transportation, and smart homes, providing robust and privacy-preserving human perception. The development of millimeter-wave radar technology has spawned a range of off-the-shelf products, characterized by low cost, small size, and ease of integration. Common applications of these products focus on two main areas: first, motion recognition, such as (i) smart gyms that adjust air conditioning (e.g., temperature, fan speed) based on the user's workout; and (ii) AR / VR (e.g., Ready Player One, Oasis), enabling multi-person interaction by recognizing human movements. The second area is target detection, such as (i) smart nighttime return detection, which identifies targets and executes intelligent do-not-disturb control or alarm operations; and (ii) air conditioning venting to avoid people, which adjusts the air vents by detecting the positions of multiple people.

[0003] However, existing millimeter-wave radar datasets are typically task-specific and limited in size, which severely restricts the potential of deep learning models to achieve high generalization and robustness. Furthermore, the acquisition and annotation of large-scale millimeter-wave radar data is a labor-intensive and time-consuming process, significantly limiting the expansion of millimeter-wave radar datasets.

[0004] To address the current shortage of millimeter-wave radar datasets, existing work has utilized various data sources to synthesize millimeter-wave radar data with promising results. Motion capture-based radar data generation methods have low computational complexity, but their data is typically sparse (containing only a dozen or so key joints), providing only a coarse radar signal. Depth camera-based radar data generation methods can provide detailed 3D point cloud information of the human body, but their datasets lack information on some common movements. The latest work (Ahuja K, Jiang Y, Goel M, et al. Vid2Doppler: synthesizing Doppler radar data from videos for training privacy-preserving activity recognition[C] / / Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems.2021:1-10) utilizes rich 2D video data to generate large-scale millimeter-wave radar training data, but it has the following limitations: (1) It is only applicable to single-person scenes and cannot handle multipath reflection and attenuation in multi-person scenes; (2) The generated data is not transferable and can only be applied to specific application scenarios; (3) Publicly available video datasets often have class imbalance problems, which leads to poor model stability. Therefore, it is difficult to generate realistic and transferable millimeter-wave radar data from videos. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by proposing a method for generating millimeter-wave radar data using abundant video data. This method solves the problem of a lack of relevant millimeter-wave radar datasets and can generate realistic and convertible millimeter-wave radar data from videos, which can then be applied to downstream tasks based on millimeter-wave radar data.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a method for generating millimeter-wave radar data based on video, comprising the following steps:

[0008] S1. Use a variant twin network to extract key data segments from the video;

[0009] S2. Use the human body region indexing algorithm to achieve one-to-one matching between human body grid data and human body location information;

[0010] S3. Generate corresponding depth information, radar cross-section, and radial velocity using the human body mesh model and depth prediction model;

[0011] S4. Employ a multi-person reflection model to simulate the multipath reflection and attenuation of radar signals, and output convertible coarse radar data.

[0012] S5. Use the Transformer model to generate realistic radar data.

[0013] Furthermore, in step S1, the variant Siamese network includes two sub-networks. The first sub-network sequentially passes the selected keyframes through convolutional layers to obtain the corresponding feature vector sets. The second sub-network calculates the Euclidean distance between different feature vectors to obtain a distance vector set, and passes the obtained distance vector set through a fully connected layer to obtain a quantized scalar set between keyframes. Subsequently, the scalar set is used as the input and output of the sigmoid function to produce the corresponding contrast matrix. Finally, the breadth-first search algorithm is used to cluster the contrast matrix to output the key data segments in the video.

[0014] Furthermore, the specific process of extracting key data segments from the video using a variant twin network in step S1 is as follows:

[0015] S11. For an input video, divide it into n segments and extract one frame from each segment as a keyframe to obtain the n keyframes of the video.

[0016] S12. The first sub-network uses 16 convolutional layers to extract a set of feature vectors {f1, f2, ..., fn} from n keyframes. n};

[0017] S12. Using the Euclidean distance between different feature vectors, calculate the distance vector set {z} using formula (1). 1，2 , ..., z 1，n , z 2，3 , ..., z 2，n , ..., z n-1，n};

[0018]

[0019] S13. Calculate the similarity between keyframes using the contrastive loss function, as shown in formulas (2) and (3), to obtain a scalar set {l 1，2 , ..., l 1，n , l 2，3 , ..., l 2，n , ..., l n-1，n};

[0020]

[0021] L(W,(Y,x n-1 x n )i )=(1-Y)L S (z n-1，n (x n-1 x n ) i )+YL D (z n-1，n (x n-1 x n ) i (3)

[0022] Where (Y, x n-1 x n ) i Represents the i-th sample, which includes an input pair x. n-1 x n A label Y representing whether the two inputs belong to the same category, W representing model parameters, and L... S L represents the loss function when the two inputs belong to the same category. D Loss functions representing different categories;

[0023] S14. Use the sigmoid function to normalize the scalar set to obtain an upper triangular contrast matrix with zeros on the diagonal.

[0024] S15. Use the breadth-first search algorithm to cluster the comparison matrix to obtain key video segments.

[0025] Furthermore, the specific process of achieving one-to-one matching between human body grid data and human body location information using the human body region indexing algorithm in step S2 is as follows:

[0026] S21. Use the object detection model to identify the bounding box of each human body in the input video frame, and assign a unique index to each human body bounding box starting from the first video frame.

[0027] S22, in each upcoming frame F i In the middle, calculate the Fth... i Frame and F i-1 The position offset and region offset of the bounding box in the frame are used to determine the Fth frame. i Each bounding box in the frame matches the corresponding index previously assigned; if the Fth bounding box... i If a previously unseen target appears in a frame, a new index will be automatically assigned to that target;

[0028] S23. Output the Fth index in index order. i All human bounding boxes in the frame.

[0029] Furthermore, the process of generating the corresponding depth information, radar cross-section, and radial velocity using the human body mesh model and depth prediction model in step S3 is as follows:

[0030] S31. Based on the bounding boxes output by the human region indexing algorithm in step S2, crop and extract all human targets {P1, P2, ..., P} in the current frame. n};

[0031] S32. Using a single-person human body mesh fitting model, for each target P... i Generate the corresponding human body mesh M i Simultaneously, using a depth prediction model, the center point depths {d1, d2, ..., d} of all targets are predicted. n};

[0032] S33, based on depth d i The corresponding human body mesh M i Transform to the world coordinate system to obtain the human body mesh set corresponding to the current frame;

[0033] S34. Based on the human body mesh set, calculate the radar cross area of ​​the mesh surface in the current frame, and calculate the radial velocity corresponding to each vertex based on the motion history of each mesh vertex in the previous frame.

[0034] Furthermore, before using a multi-person reflection model to simulate the multipath reflection and attenuation of radar signals in step S4, virtual physical objects are added to the output of the single-person human body mesh fitting model, and multiple rays are emitted from the radar coordinate point into space, denoted as the ray set R, before the reflection simulation is performed.

[0035] Furthermore, the specific process of simulating multipath reflection and attenuation of radar signals using a multi-person reflection model in step S4 is as follows:

[0036] S41. Calculate the i-th ray r in the ray set R. i The first intersection point P with the human body mesh set and the mesh face where P is located;

[0037] S42. Calculate the direction vector of the reflected ray according to formula (4). At the same time, the attenuated signal strength T is calculated according to formula (5). s Where γ is the attenuation coefficient, and the signal strength T is... s Reflected rays r above the threshold s Add to set R;

[0038]

[0039] T s =T i ·γ (5)

[0040] in, For the i-th ray r i The direction vector, T i For the i-th ray r i signal strength, Let P be the normal vector of the grid surface where the first intersection point P is located;

[0041] S43. Repeat the above steps until set R is empty;

[0042] S44. Based on the results of the reflection simulation, output the signal intensity map in the space.

[0043] Furthermore, in step S5, the Transformer model consists of two parts: an encoder and a decoder. The encoder and decoder each consist of 6 identical layers. Each layer of the encoder has two sub-layers: the first sub-layer is a multi-head attention mechanism, and the second sub-layer is a fully connected feedforward network. Each layer of the decoder has three sub-layers: the first sub-layer is a multi-head attention mechanism, the second sub-layer is a fully connected feedforward network, and the third sub-layer performs a multi-head attention mechanism on the output of the encoder.

[0044] Furthermore, residual connections and layer normalization are applied to each multi-head attention.

[0045] Secondly, the present invention also provides an apparatus for generating millimeter-wave radar data based on video, the apparatus comprising a key segment extraction module and a data generation module, to implement the steps of the method for generating millimeter-wave radar data based on video as described in any of the preceding claims, wherein:

[0046] The key segment extraction module uses a variant Siamese network to output key data segments from the video.

[0047] The data generation module first uses a human body region indexing algorithm to achieve a one-to-one matching of human body grid data and human body position information. Then, it uses a human body grid model and a depth prediction model to generate corresponding depth information, radar cross section and radial velocity. Subsequently, it uses a multi-person reflection model to simulate multipath reflection and attenuation of radar signals and outputs convertible coarse radar data. Finally, it uses a Transformer model to generate realistic radar data.

[0048] Thirdly, the present invention also provides an electronic device, characterized in that it includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0049] Memory, used to store computer programs;

[0050] When a processor executes a program stored in memory, it implements the steps of any of the above methods for generating millimeter-wave radar data based on video.

[0051] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described methods for generating millimeter-wave radar data based on video.

[0052] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0053] The method and apparatus for generating millimeter-wave radar data based on video proposed in this invention select key segments from redundant video data, solving the imbalance problem between different action categories. At the same time, it ensures the stability of the machine learning model used for human perception, makes the generated data more realistic by using data fitting, solves the problem of mismatch between depth information and multi-person grid data, and solves the multipath reflection and attenuation problems of radar signals during transmission and reception. It can convert nearly unlimited video data into realistic and convertible millimeter-wave radar data, which can be applied to downstream tasks related to human perception such as human action recognition and target detection. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0055] Figure 1 This is a schematic diagram of the system architecture for a method of generating millimeter-wave radar data based on video, provided in an embodiment of the present invention.

[0056] Figure 2 This is a schematic diagram of an electronic device structure for implementing a method for generating millimeter-wave radar data based on video, as provided in an embodiment of the present invention. Detailed Implementation

[0057] To better understand this technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described examples are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0059] The overall system architecture of the method for generating millimeter-wave radar data based on video proposed in this invention is as follows: Figure 1 As shown, it includes two parts: a data generation network and a variant Siamese network. (1) The variant Siamese network divides the traditional Siamese network into two sub-networks for data selection. The first sub-network sequentially passes the selected keyframes through convolutional layers to obtain the corresponding feature vector sets. The second sub-network calculates the Euclidean distance between different feature vectors to obtain a distance vector set, and passes the obtained distance vector set through a fully connected layer to obtain a quantized scalar set between keyframes. Subsequently, the scalar set is used as the input and output of the sigmoid function to output the corresponding contrast matrix. Finally, the breadth-first search algorithm (BFS) is used to cluster the contrast matrix to output the key data segments in the video. (2) The data generation network consists of four key modules: a human region indexing algorithm, a depth prediction and human mesh fitting model, a multi-person reflection model, and a Transformer-based domain transformation. First, a human body region indexing algorithm is used to achieve a one-to-one matching of human body mesh data and human body location information. Then, the human body mesh model and depth prediction model are used to generate corresponding depth information, radar cross section (RCS), and radial velocity. Subsequently, a multi-person reflection model is used to simulate the multipath reflection and attenuation of radar signals, and convertible coarse radar data is output. Finally, a Transformer model is used to generate realistic radar data.

[0060] (1) Variant twin network

[0061] Because class imbalance in videos can transfer to generated radar data, deep learning models trained on generated data suffer from low stability. Therefore, eliminating video redundancy and resolving class imbalance is crucial. This invention designs a variant of the Siamese network to eliminate video redundancy and address class imbalance. The specific process is as follows:

[0062] First, given an input video, it is divided into n segments, and one frame is extracted from each segment as a keyframe, thus obtaining the n keyframes of the video. Then, the first sub-network uses 16 convolutional layers to extract a set of feature vectors {f1, f2, ..., fn} from these n keyframes. n}; Subsequently, the second sub-network uses the Euclidean distance between different feature vectors to calculate the distance vector set {z} using formula (1). 1，2 , ..., z 1，n , z 2，3 , ..., z 2，n , ..., z n-1，n}; Then, the contrastive loss function is used to calculate the similarity between keyframes, as shown in formulas (2) and (3), to obtain a scalar set {l 1，2 , ..., l 1，n , l2，3 , ..., l 2，n , ..., l n-1，n};

[0063]

[0064]

[0065] L(W(Y,x n-1 x n ) i )=(1-Y)L S (z n-1，n (x n-1 x n ) i )+YL D (z n-1，n (x n-1 x n ) i (3)

[0066] Where (Y, x n-1 x n ) i Represents the i-th sample, which includes an input pair x. n-1 x n A label Y representing whether the two inputs belong to the same category, W representing model parameters, and L... S L represents the loss function when the two inputs belong to the same category. D Loss functions representing different categories.

[0067] Subsequently, the sigmoid function is used to normalize the scalar set, resulting in an upper triangular contrast matrix with zeros on the diagonal. Finally, the BFS algorithm is used to cluster the contrast matrix to obtain key video segments.

[0068] (2) Data Generation Network

[0069] To convert 2D video data into convertible millimeter-wave radar data, this invention proposes a data generation network, specifically comprising the following four modules:

[0070] 1) Human Region Indexing Algorithm. First, an object detection model is used to identify the bounding boxes of each human body in the input video frames. Starting from the first frame of the video, a unique index is assigned to each appearing human bounding box. Then, in each upcoming frame F... i In this algorithm, the Fth... i Frame and F i-1 The position offset and region offset of the bounding box in the frame are used to determine the Fth frame. iEach bounding box in the frame matches the corresponding index previously assigned; simultaneously, if the F-th bounding box... i If a previously unseen target appears in a frame, the algorithm will automatically assign a new index to that target; finally, the F-th target will be output according to the index order. i All human bounding boxes in the frame.

[0071] 2) Depth Prediction and Human Mesh Fitting Module. First, based on the bounding boxes output by the human region indexing algorithm, all human targets {P1, P2, ..., P} in the current frame are cropped and extracted. n Then, using a single-person human body mesh fitting model, for each target P... i Generate the corresponding human body mesh M i Simultaneously, using a depth prediction model, the center point depths {d1, d2, ..., d} of all targets are predicted. n}; then, based on depth d i The corresponding M i Transform to the world coordinate system to obtain the human body mesh set corresponding to the current frame; finally, based on the human body mesh set, calculate the radar reflection area of ​​the mesh surface in the current frame, and calculate the radial velocity corresponding to each vertex based on the motion history of each mesh vertex in the previous frame.

[0072] 3) Multi-person Reflection Model. Inspired by ray tracing in computer graphics, this invention proposes a multi-person reflection model to simulate the multipath reflection and attenuation of radar signals during transmission and reception, in order to simulate the real-world data acquisition environment. To simulate the data acquisition environment in the real world, virtual physical objects, such as walls and floors, need to be added to the output of the human body mesh fitting module, and multiple rays, denoted as ray set R, are emitted from the radar coordinates into space. Then, reflection simulation is performed. First, the i-th ray r in R is calculated. i (direction vector) Signal strength T i The first intersection point P with the above human body mesh set, and the mesh surface where P is located (normal vector is...) Then, the direction vector of the reflected ray is calculated according to formula (4). At the same time, the attenuated signal strength T is calculated according to formula (5). s Where γ is the attenuation coefficient, and the signal strength T is... s Reflected rays r above the threshold s Add it to set R; then repeat the above steps until set R is empty; finally, based on the results of the reflection simulation, output the signal intensity map in space.

[0073]

[0074] T s =T i ·γ (5)

[0075] 4) Transformer-based Domain Transformation Module. Directly using the signal intensity map output by the aforementioned multi-person reflection model results in coarse data lacking characteristic noise and continuity. To address this issue, this invention introduces a Transformer model to fit the data and generate realistic radar data. This module consists of an encoder and a decoder, each composed of six identical layers, with two sub-layers in each layer. The first sub-layer is a multi-head attention mechanism, and the second sub-layer is a fully connected feedforward network. Furthermore, an additional sub-layer is inserted into the decoder, which performs the multi-head attention mechanism on the encoder's output. Residual connections and layer normalization are applied to each multi-head attention. The Transformer is trained using pre-paired real and generated data, and then the trained Transformer is used to fit the generated data.

[0076] Practical testing has shown that the video-based millimeter-wave radar data generation algorithm proposed in this invention can generate high-quality, highly realistic millimeter-wave radar data, and has significant advantages in downstream applications such as action recognition and target detection. Experimental results show that the mean absolute error (MAE) of the data generated by this invention is 0.06 (standard deviation 0.02) compared to real data. In action recognition, the accuracy of the model trained using the generated radar data (84.67% / 80.65%) is roughly equivalent to that trained using real-world radar data (90.7% / 86.5%). If we augment the generated radar dataset with a small amount of real radar data, the recognition accuracy jumps to 97.75% / 93.3% in single / multi-person scenarios. Furthermore, the target detection task exhibits a similar performance trend. Simultaneously, in single / multi-person scenarios, this invention improves the recognition accuracy by 7.59% / 8.25% compared to not using a variant Siamese network, while reducing the latency of key segment selection by 242 times.

[0077] Corresponding to the method for generating millimeter-wave radar data based on video provided in the above embodiments of the present invention, the present invention also provides an apparatus for generating millimeter-wave radar data based on video, the apparatus comprising:

[0078] The key segment extraction module uses a variant twin network to output key data segments from the video. The data generation module first uses a human region indexing algorithm to achieve a one-to-one matching of human grid data and human position information. Then, it uses a human grid model and a depth prediction model to generate corresponding depth information, radar cross section and radial velocity. Subsequently, it uses a multi-person reflection model to simulate the multipath reflection and attenuation of radar signals and outputs convertible coarse radar data. Finally, it uses a Transformer model to generate realistic radar data.

[0079] Corresponding to the device for generating millimeter-wave radar data based on video provided in the above embodiments of the present invention, the present invention also provides an electronic device.

[0080] like Figure 2 As shown, the electronic device includes a processor 201, a communication interface 202, a memory 203, and a communication bus 204. The processor 201, communication interface 202, and memory 203 communicate with each other via the communication bus 204.

[0081] Memory 203 is used to store computer programs;

[0082] When the processor 201 executes the program stored in the memory 203, it implements the steps of any of the methods for generating millimeter-wave radar data based on video provided in the above embodiments of the present invention.

[0083] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0084] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0085] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0086] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0087] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the methods for generating millimeter-wave radar data based on video provided in the embodiments of the present invention.

[0088] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of any of the methods for generating millimeter-wave radar data based on video provided in the embodiments of the present invention.

[0089] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital terminal equipment line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0090] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0091] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments, electronic device embodiments, computer-readable storage medium embodiments, and computer program product embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0092] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for generating millimeter wave radar data based on video, the method comprising: Comprising the following steps: S1, using a variant twin network to extract key data segments in the video; The variant twin network in step S1 includes two sub-networks, the first sub-network sequentially passes the selected key frames through the convolution layer to obtain the corresponding feature vector set; the second sub-network calculates the Euclidean distance between different feature vectors to obtain a distance vector set, and passes the obtained distance vector set through a fully connected layer to obtain a quantized scalar set between the key frames; then, the scalar set is input into the sigmoid function to output a corresponding comparison matrix; finally, the breadth-first search algorithm is used to cluster the comparison matrix to output the key data segments in the video; The specific process of step S1 for extracting key data segments in the video using a variant twin network is as follows: S11, for an input video, divide it into n segments, and extract one frame from each segment as a key frame, so as to obtain n key frames of the video; S12, the first sub-network uses 16-layer convolution to extract a feature vector set {f1, f2,..., fn} from n key frames n}; S 12. Using the Euclidean distance between different feature vectors, calculate the distance vector set {z} using formula (1). 1,2 ,...,z 1，n , z 2，3 , ..., z 2，n , ..., z n-1，n }; S13, use a contrast loss function to calculate the similarity between the key frames, as shown in formulas (2) and (3) as shown, a set of scalars {l 1，2 ,...,l 1，n ,...,l 2，3 ,...,l 2，n ,...,l n-1，n} is obtained; L(W, (Y, x n-1 , x n ) i ) = (1 - Y)L S (z n-1，n (x n-1 , x n ) i )+ YL D (z n-1，n (x n-1 , x n ) i ) (3) where (Y, x n-1 , x n ) i represents the ith sample, which includes an input pair x n-1 , x n and a label Y representing whether the two inputs belong to the same category, W represents the model parameters, L S represents the loss function when the two inputs belong to the same category, and L D represents the loss function when they belong to different categories. S14, use the sigmoid function to normalize the scalar set to obtain an upper triangular comparison matrix with a zero diagonal; S15, use the breadth-first search algorithm to cluster the comparison matrix to obtain the key video segments; S2, use a human region index algorithm to realize one-to-one matching of human grid data and human position information; S3, use a human grid model and a depth prediction model to generate corresponding depth information, radar cross section and radial velocity; S4, use a multi-person reflection model to simulate the multipath reflection and attenuation of radar signals, and output convertible rough radar data; S5, use a Transformer model to generate realistic radar data.

2. The method for generating millimeter wave radar data based on video according to claim 1, wherein, The specific process of step S2 for realizing one-to-one matching of human grid data and human position information using a human region index algorithm is as follows: S21, use a target detection model to identify the boundary box of each human body for the input video frame, and assign a unique index to each human body boundary box appearing from the first video frame; S22, in each upcoming frame F i , the position offset and area offset of the bounding box in the F i th frame and the F i-1 th frame are calculated, so that each bounding box in the F i th frame matches the corresponding index assigned previously; if a target appears in the F i th frame that has not been seen before, a new index will be automatically assigned to the target; S23, output the Fth according to the index order i All human body bounding boxes in the frame.

3. The method for generating millimeter wave radar data based on video of claim 2, wherein, The process of step S3 for generating corresponding depth information, radar cross section and radial velocity using a human grid model and a depth prediction model is as follows: S31, according to the boundary frame output by the human region index algorithm in step S2, all human targets {P1, P2,..., Pn} in the current frame are extracted by cutting. n} S32, using a single-person human mesh fitting model, respectively fitting each target P i Generate the corresponding human mesh M i At the same time, using a depth prediction model, predict the center point depth {d1, d2,..., d n} of all targets; S33、according to the depth d i The corresponding human body mesh M i Convert to the world coordinate system to obtain a human body mesh set corresponding to the current frame; S34, according to the human grid set, calculate the radar cross-sectional area of the grid surface in the current frame, and according to the motion history of each grid vertex in the previous frame, calculate the corresponding radial velocity of each vertex.

4. The method for generating millimeter wave radar data based on video of claim 3, wherein, Before step S4 simulates the multipath reflection and attenuation of radar signals using a multi-person reflection model, a virtual physical object is added to the output results of the single-person grid fitting model, and a plurality of rays are emitted from the radar coordinate point to the space, denoted as a ray set R, and then the reflection simulation is performed.

5. The method for generating millimeter wave radar data based on video of claim 4, wherein, The specific process of step S4 for simulating the multipath reflection and attenuation of radar signals using a multi-person reflection model is as follows: S41, calculate the i-th ray r in the ray set R i The first intersection point P of the human body grid set and the grid surface where P is located; S42, calculate the direction vector of the reflected ray according to formula (4) Meanwhile, calculate the signal intensity T after attenuation according to formula (5) s where γ is the attenuation coefficient, and the signal intensity T s The reflected ray r s is added to the set R; T s = T i · γ (5) in, For the i-th ray r i The direction vector, T i For the i-th ray r i signal strength, Let P be the normal vector of the grid surface where the first intersection point P is located; S43, repeatedly repeat the above steps until the set R is empty; S44, according to the results of the reflection simulation, output the signal intensity map in the space.

6. The method for generating millimeter wave radar data based on video of claim 1, wherein, The Transformer model in step S5 is composed of an encoder and a decoder, the encoder and the decoder are each composed of 6 identical layers, each layer of the encoder has two sub-layers, the first sub-layer is a multi-head attention mechanism, and the second sub-layer is a fully connected feedforward network, each layer of the decoder has three sub-layers, the first sub-layer is a multi-head attention mechanism, the second sub-layer is a fully connected feedforward network, and the third sub-layer performs a multi-head attention mechanism on the output of the encoder.

7. The method for generating millimeter wave radar data based on video of claim 6, wherein, Residual connections and layer normalization are used for each multi-head attention.

8. An apparatus for generating millimeter wave radar data based on video, the apparatus comprising: The device comprises a key segment extraction module and a data generation module to implement the method of any one of claims 1-7, wherein: The key segment extraction module extracts key data segments in the video using a variant twin network. The data generation module first realizes one-to-one matching of human grid data and human position information using a human region indexing algorithm, then generates corresponding depth information, radar cross-section and radial velocity using a human grid model and a depth prediction model, subsequently simulates multi-path reflection and attenuation of radar signals using a multi-person reflection model to output convertible coarse radar data, and finally generates realistic radar data using a Transformer model.

Images