Video-based millimeter wave data automatic annotation method and system

Abstract

The present invention discloses a method and system for automatic millimeter wave data annotation based on video, wherein the method comprises: using multi-threading to synchronously collect millimeter wave data of a to-be-identified area at each moment and each frame of image data in a corresponding video; inputting each frame of image data into a trained behavior recognition neural network model for identification, so as to obtain a human motion recognition result in the video at each moment; processing the millimeter wave data to obtain micro-Doppler feature data with time dimension information; dividing and annotating the micro-Doppler feature data according to the human motion recognition result, so as to obtain annotated millimeter wave data; thereby, by synchronously collecting video data and millimeter wave data, the millimeter wave radar data with weaker interpretability can be annotated through the video data, thereby not only saving the data annotation cost but also improving the accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic data annotation, and in particular to a video-based millimeter wave data automatic annotation method and a video-based millimeter wave data automatic annotation system. Background Art

[0002] In related technologies, with the decline in the birth rate, the proportion of the elderly population continues to rise. my country is facing the problem of an aging population. The needs of a large number of elderly people for life care, travel, etc. are increasing, among which the demand for safety protection is particularly prominent; falls by elderly people over 65 years old are very likely to cause fractures, and if they cannot receive timely treatment, it can lead to life-threatening situations; traditional methods of detecting fall behavior usually use video data collected by cameras for identification. This detection method cannot protect the privacy of the elderly in a home care environment.

[0003] Compared with video data, millimeter wave data can better protect personal privacy. However, in order to realize millimeter wave-based fall detection, millimeter wave data needs to be labeled according to different actions. Because this data has natural privacy protection properties, it is difficult to label. In the process of manual labeling of millimeter wave data, it is usually labeled based on other visual data, generally using video data. In the manual labeling process, there are problems of high time cost and inconsistent labeling standards.

[0004] Video data can be used as a good visual reference for millimeter-wave data. Video data can be used to annotate millimeter-wave data by action. However, if millimeter-wave radar data and camera data are collected separately, it is difficult to synchronize the millimeter-wave radar data and camera data during the collection process, and there is a time difference. At the same time, the frame rate of the collected data cannot be guaranteed to be stable. Data asynchrony and frame rate instability will lead to unpredictable errors in the subsequent data labeling process. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems in the above-mentioned technology to a certain extent. To this end, one purpose of the present invention is to propose a method for automatic annotation of millimeter wave data based on video, by synchronously collecting video data and millimeter wave data, so as to annotate the millimeter wave radar data with poor interpretability through video data, thereby not only saving data annotation costs but also improving accuracy.

[0006] The second objective of the present invention is to provide a video-based millimeter wave data automatic annotation system.

[0007] To achieve the above-mentioned purpose, the first aspect of the present invention proposes a method for automatic millimeter wave data annotation based on video, comprising the following steps: using multi-threaded synchronous acquisition of millimeter wave data of the area to be identified at each moment and each frame of image data in the corresponding video; inputting each frame of image data into a trained behavior recognition neural network model for identification to obtain the human motion recognition result in the video at each moment; processing the millimeter wave data to obtain micro-Doppler feature data with time dimension information; dividing and annotating the micro-Doppler feature data according to the human motion recognition result to obtain annotated millimeter wave data.

[0008] According to the video-based millimeter wave data automatic labeling method of an embodiment of the present invention, first, multi-threaded synchronous acquisition of the millimeter wave data of the area to be identified at each moment and each frame of image data in the corresponding video is adopted; then, each frame of image data is input into the trained behavior recognition neural network model for recognition, so as to obtain the human action recognition result in the video at each moment; then, the millimeter wave data is processed to obtain micro-Doppler feature data with time dimension information; then, the micro-Doppler feature data is divided and labeled according to the human action recognition result, so as to obtain the labeled millimeter wave data; thus, by synchronously acquiring video data and millimeter wave data, the millimeter wave radar data with weaker interpretability can be labeled through the video data, which not only saves the data labeling cost but also improves the accuracy.

[0009] In addition, the video-based millimeter wave data automatic annotation method proposed in the above embodiment of the present invention may also have the following additional technical features:

[0010] Optionally, multi-threading is used to synchronously collect millimeter wave data of the area to be identified at each moment and each frame of image data in the corresponding video, including: a synchronous control thread issues an acquisition instruction to a millimeter wave acquisition thread and a camera acquisition thread, so as to control the millimeter wave acquisition thread and the camera acquisition thread to start collecting data at the same time; the synchronous control thread falls into sleep mode, so as to wait for the millimeter wave acquisition thread and the camera acquisition thread to complete the collection of data once; the millimeter wave acquisition thread and the camera acquisition thread start collecting data, and after the collection is completed, the synchronous control thread is awakened by the longest thread to start the collection of the next frame of data, and the cycle is repeated until the collection is completed.

[0011] Optionally, the training process of the behavior recognition neural network model includes: establishing a behavior recognition neural network model; acquiring image data of different actions that need to be recognized in different scenarios to form a data set; inputting the data set into the behavior recognition neural network model so as to train the behavior recognition neural network model to obtain a trained behavior recognition neural network model.

[0012] Optionally, the millimeter wave data is processed to obtain micro-Doppler feature data with time dimension information, including: performing short-time Fourier transform on the millimeter wave data according to the front and rear frame signals of the millimeter wave data to obtain a range Doppler map; projecting the range Doppler information of each signal period of the range Doppler map onto the distance dimension, and accumulating each frame to obtain micro-Doppler feature data with time dimension information.

[0013] Optionally, the millimeter wave data is further divided on the time axis according to the start and end of the action in the behavior recognition result to obtain labels with the start and end of the action, so as to complete the automatic labeling of the millimeter wave data.

[0014] To achieve the above-mentioned purpose, the second aspect of the present invention proposes a video-based millimeter wave data automatic labeling system, including a data synchronization acquisition module, which is used to use multi-threaded synchronous acquisition of millimeter wave data of the area to be identified at each moment and each frame of image data in the corresponding video; a behavior recognition module, which is used to input each frame of image data into a trained behavior recognition neural network model for recognition, so as to obtain the human motion recognition result in the video at each moment; a millimeter wave data processing module, which is used to process the millimeter wave data to obtain micro-Doppler feature data with time dimension information; a data division and labeling module, which is used to divide and label the micro-Doppler feature data according to the human motion recognition result, so as to obtain the labeled millimeter wave data.

[0015] According to the video-based millimeter wave data automatic annotation system of the embodiment of the present invention, the data synchronization acquisition module adopts multi-threaded synchronous acquisition of the millimeter wave data of the area to be identified at each moment and each frame of image data in the corresponding video; the behavior recognition module inputs each frame of image data into the trained behavior recognition neural network model for recognition to obtain the human action recognition result in the video at each moment; the millimeter wave data processing module processes the millimeter wave data to obtain micro-Doppler feature data with time dimension information; the data division and annotation module divides and annotates the micro-Doppler feature data according to the human action recognition result to obtain the annotated millimeter wave data; thus, by synchronously acquiring video data and millimeter wave data, the millimeter wave radar data with weaker interpretability can be annotated through the video data, which not only saves the data annotation cost but also improves the accuracy.

[0016] In addition, the video-based millimeter wave data automatic annotation system proposed in the above embodiment of the present invention may also have the following additional technical features:

[0017] Optionally, the data synchronization acquisition module is also used for the synchronization control thread to issue acquisition instructions to the millimeter wave acquisition thread and the camera acquisition thread so as to control the millimeter wave acquisition thread and the camera acquisition thread to start collecting data at the same time; the synchronization control thread falls into sleep mode so as to wait for the millimeter wave acquisition thread and the camera acquisition thread to complete the acquisition of data once; the millimeter wave acquisition thread and the camera acquisition thread start to collect data, and after the acquisition is completed, wake up the synchronization control thread through the thread that takes the longest time so as to start the acquisition of the next frame of data, and repeat the cycle until the acquisition is completed.

[0018] Optionally, the behavior recognition module is also used to establish a behavior recognition neural network model; obtain image data of different actions that need to be recognized in different scenarios to form a data set; input the data set into the behavior recognition neural network model so as to train the behavior recognition neural network model to obtain a trained behavior recognition neural network model.

[0019] Optionally, the millimeter wave data processing module is also used to perform short-time Fourier transform on the millimeter wave data according to the previous and next frame signals of the millimeter wave data to obtain a range Doppler map; project the range Doppler information of each signal period of the range Doppler map onto the distance dimension, and accumulate each frame to obtain micro-Doppler feature data with time dimension information.

[0020] Optionally, the data division and labeling module is also used to further divide the millimeter wave data on the time axis according to the start and end of the action in the behavior recognition result, and obtain labels with the start and end of the action to complete the automatic labeling of the millimeter wave data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of a flow chart of a method for automatic annotation of millimeter wave data based on video according to an embodiment of the present invention;

[0022] Figure 2 A diagram of a data collection environment according to an embodiment of the present invention;

[0023] Figure 3 4 is a block diagram of a system for automatically annotating millimeter-wave data based on video according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0025] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0026] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0027] Figure 1 FIG. 1 is a flow chart of a method for automatically labeling millimeter wave data based on video according to an embodiment of the present invention. Figure 1 As shown, the video-based millimeter wave data automatic annotation method includes the following steps:

[0028] S101, using multi-threading to synchronously collect millimeter wave data of the area to be identified at each moment and each frame of image data in the corresponding video.

[0029] It should be noted that the millimeter wave data and video data are collected by the collection device, wherein, Figure 2 As shown, the acquisition device includes a millimeter-wave radar and two cameras. The millimeter-wave radar is placed directly above the acquisition area, and the two cameras are respectively located directly above and directly in front of the acquisition area. The collected video data is used to identify human behavior, and the millimeter-wave data is the original data that needs to be labeled.

[0030] As an embodiment, millimeter wave data is acquired through millimeter wave radar, and data is transmitted by connecting to the host via a network cable; camera data is acquired through a camera, and data is transmitted by connecting to the host via a USB2.0 / 3.0 interface; the host acts as a processing terminal and processes the data synchronously.

[0031] That is to say, cameras and millimeter-wave radars are used for data collection. The camera collects video data, and the millimeter-wave radar collects the object's movement speed, point cloud information, and position information. The collected millimeter-wave data is transmitted to the host through various connection methods such as network cables and USB; the collected image data is transmitted to the host through a USB connection, and the data of the two are aligned through data frame synchronization technology to eliminate time errors.

[0032] As an embodiment, multi-threaded synchronous acquisition of millimeter wave data of the area to be identified at each moment and each frame of image data in the corresponding video includes:

[0033] The synchronous control thread issues acquisition instructions to the millimeter wave acquisition thread and the camera acquisition thread, so as to control the millimeter wave acquisition thread and the camera acquisition thread to start collecting data at the same time;

[0034] The synchronization control thread falls into sleep mode to wait for the millimeter wave acquisition thread and the camera acquisition thread to complete data acquisition;

[0035] The millimeter wave acquisition thread and the camera acquisition thread start to collect data, and after the acquisition is completed, the synchronization control thread is awakened by the longest thread to start the acquisition of the next frame of data, and the cycle repeats until the acquisition is completed.

[0036] In other words, millimeter wave data and video data use multi-threaded synchronization technology, which includes three threads: control thread, millimeter wave acquisition thread and camera acquisition thread, to accurately collect each frame of data; during the acquisition process, only when all acquisition threads complete the acquisition of the current frame of data, the scheduling thread will generate a signal to enable all acquisition threads to collect the next frame of data.

[0037] S102, inputting each frame of image data into the trained behavior recognition neural network model for recognition, so as to obtain the human action recognition result in the video at each moment.

[0038] As an embodiment, the training process of the behavior recognition neural network model includes:

[0039] Establish a SlowFast-based behavior recognition neural network model;

[0040] Obtain image data of different actions that need to be recognized in different scenes to form a data set;

[0041] The data set is input into the behavior recognition neural network model so as to train the behavior recognition neural network model to obtain a trained behavior recognition neural network model.

[0042] That is to say, according to different scenarios, decide the actions that need to be recognized, create corresponding data sets, and perform transfer learning on the pre-trained model; the neural network model trained with the dedicated action data set is used to recognize human behavior in the scenario, and the actions of the human body in the video at each moment are obtained.

[0043] It should be noted that data labels are generated based on the results of human motion recognition to distinguish different types of motions, such as walking, squatting, running and other common motions.

[0044] S103, processing the millimeter wave data to obtain micro-Doppler characteristic data with time dimension information.

[0045] As an embodiment, millimeter wave data is processed to obtain micro-Doppler characteristic data with time dimension information, including:

[0046] Performing short-time Fourier transform on the millimeter wave data according to the front and rear frame signals of the millimeter wave data to obtain a range Doppler map;

[0047] The range Doppler information of each signal cycle of the range Doppler image is projected on the range dimension and accumulated in each frame to obtain micro-Doppler feature data with time dimension information.

[0048] That is to say, the millimeter wave data is processed, and a short-time Fourier transform is performed based on the previous and next frame signals to obtain a range-Doppler diagram. The range Doppler information of each signal cycle is projected on the distance dimension, and each frame is accumulated to obtain micro-Doppler feature data with information in the time dimension.

[0049] S104, dividing and labeling the micro-Doppler feature data according to the human motion recognition result to obtain labeled millimeter wave data.

[0050] As an embodiment, the millimeter wave data is further divided on the time axis according to the start and end of the action in the behavior recognition result, and labels with the start and end of the action are obtained to complete the automatic labeling of the millimeter wave data.

[0051] It should be noted that when using the visual neural network algorithm to perform behavior recognition on the video, the first step is to identify the position of the human body, and the second step is to identify the human body's movements, completing the segmentation and labeling of the millimeter wave data, avoiding the problem of the inability to unify the standards for manually labeling millimeter wave data, and saving data labeling costs; using retrained neural networks for behavior recognition, the recognition results are more accurate for different environments, reducing unnecessary expenses; and synchronizing the data during the collection process allows for more accurate subsequent processing, so as to accurately complete the data labeling task.

[0052] In summary, according to the video-based millimeter wave data automatic labeling method of an embodiment of the present invention, first, multi-threaded synchronous acquisition is used to collect the millimeter wave data of the area to be identified at each moment and each frame of image data in the corresponding video; then, each frame of image data is input into the trained behavior recognition neural network model for recognition to obtain the human motion recognition result in the video at each moment; then, the millimeter wave data is processed to obtain micro-Doppler feature data with time dimension information; then, the micro-Doppler feature data is divided and labeled according to the human motion recognition result to obtain the labeled millimeter wave data; thus, by synchronously acquiring video data and millimeter wave data, the millimeter wave radar data with weaker interpretability can be labeled through video data, which not only saves the data labeling cost but also improves the accuracy.

[0053] Figure 3 is a block diagram of a system for automatically annotating millimeter wave data based on video according to an embodiment of the present invention; Figure 3 As shown, the video-based millimeter wave data automatic annotation system includes: a data synchronization acquisition module 201, a behavior recognition module 202, a millimeter wave data processing module 203 and a data division and annotation module 204.

[0054] Among them, the data synchronization acquisition module 201 is used to use multi-threaded synchronous acquisition of the millimeter wave data of the area to be identified at each moment and each frame of image data in the corresponding video; the behavior recognition module 202 is used to input each frame of image data into the trained behavior recognition neural network model for recognition, so as to obtain the human motion recognition result in the video at each moment; the millimeter wave data processing module 203 is used to process the millimeter wave data to obtain micro-Doppler feature data with time dimension information; the data division and labeling module 204 is used to divide and label the micro-Doppler feature data according to the human motion recognition result, so as to obtain the labeled millimeter wave data.

[0055] As an embodiment, the data synchronization acquisition module 201 is also used to issue acquisition instructions to the millimeter wave acquisition thread and the camera acquisition thread through the synchronization control thread, so as to control the millimeter wave acquisition thread and the camera acquisition thread to start collecting data at the same time; the synchronization control thread falls into sleep mode to wait for the millimeter wave acquisition thread and the camera acquisition thread to complete the collection of data once; the millimeter wave acquisition thread and the camera acquisition thread start collecting data, and after the collection is completed, wake up the synchronization control thread through the thread that takes the longest time to enter the collection of the next frame of data, and repeat the cycle until the collection is completed.

[0056] As an embodiment, the behavior recognition module 202 is also used to establish a behavior recognition neural network model based on SlowFast; obtain image data of different actions that need to be recognized in different scenarios to form a data set; input the data set into the behavior recognition neural network model so as to train the behavior recognition neural network model to obtain a trained behavior recognition neural network model.

[0057] As an embodiment, the millimeter wave data processing module 203 is also used to perform short-time Fourier transform on the millimeter wave data according to the previous and next frame signals of the millimeter wave data to obtain a range Doppler map; project the range Doppler information of each signal period of the range Doppler map onto the distance dimension, and accumulate each frame to obtain micro-Doppler feature data with time dimension information.

[0058] As an embodiment, the data division and labeling module 204 is also used to further divide the millimeter wave data on the time axis according to the start and end of the action in the behavior recognition result, and obtain labels with the start and end of the action to complete the automatic labeling of the millimeter wave data.

[0059] It should be noted that the above explanations of the embodiment of the video-based millimeter wave data automatic labeling method are also applicable to the video-based millimeter wave data automatic labeling system of this embodiment, and will not be repeated here.

[0060] In summary, according to the video-based millimeter wave data automatic annotation system of the embodiment of the present invention, the data synchronization acquisition module adopts multi-threaded synchronous acquisition of the millimeter wave data of the area to be identified at each moment and each frame of image data in the corresponding video; the behavior recognition module inputs each frame of image data into the trained behavior recognition neural network model for recognition to obtain the human motion recognition result in the video at each moment; the millimeter wave data processing module processes the millimeter wave data to obtain micro-Doppler feature data with time dimension information; the data division and annotation module divides and annotates the micro-Doppler feature data according to the human motion recognition result to obtain the annotated millimeter wave data; thus, by synchronously acquiring video data and millimeter wave data, the millimeter wave radar data with weaker interpretability can be annotated through the video data, which not only saves the data annotation cost but also improves the accuracy.

[0061] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0062] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0063] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0064] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0065] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0066] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0067] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0068] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0069] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0070] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0071] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0072] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for automatic annotation of millimeter wave data based on video, characterized in that: The following steps are involved: Multi-threading is used to synchronously collect the millimeter wave data of the area to be identified at each moment and each frame of image data in the corresponding video; Input each frame of image data into a trained behavior recognition neural network model for recognition, so as to obtain a human action recognition result in the video at each moment; Processing the millimeter wave data to obtain micro-Doppler characteristic data with time dimension information; dividing and labeling the micro-Doppler feature data according to the human motion recognition result to obtain labeled millimeter wave data; The training process of the behavior recognition neural network model includes: Establish a neural network model for behavior recognition; Obtain image data of different actions that need to be recognized in different scenes to form a data set; Inputting the data set into a behavior recognition neural network model so as to train the behavior recognition neural network model to obtain a trained behavior recognition neural network model; The millimeter wave data is processed to obtain micro-Doppler characteristic data with time dimension information, including: Performing short-time Fourier transform on the millimeter wave data according to the preceding and following frame signals of the millimeter wave data to obtain a range Doppler map; Projecting the range Doppler information of each signal cycle of the range Doppler image on the range dimension and accumulating each frame to obtain micro-Doppler feature data with time dimension information; The millimeter wave data is further divided on the time axis according to the start and end of the action in the behavior recognition result, and labels with the start and end of the action are obtained to complete the automatic labeling of the millimeter wave data.

2. The video-based millimeter wave data automatic annotation method according to claim 1, characterized in that: Multi-threaded synchronous acquisition of the millimeter wave data of the area to be identified at each moment and each frame of image data in the corresponding video, including: The synchronous control thread issues an acquisition instruction to the millimeter wave acquisition thread and the camera acquisition thread, so as to control the millimeter wave acquisition thread and the camera acquisition thread to start collecting data at the same time; The synchronization control thread falls into sleep mode so as to wait for the millimeter wave acquisition thread and the camera acquisition thread to complete one data acquisition; The millimeter wave acquisition thread and the camera acquisition thread start to collect data, and after the acquisition is completed, the synchronization control thread is awakened by the longest thread to start the acquisition of the next frame of data, and the cycle is repeated until the acquisition is completed.

3. A video-based millimeter wave data automatic annotation system, characterized in that: include: A data synchronization acquisition module, wherein the data synchronization acquisition module is used to synchronously acquire the millimeter wave data of the area to be identified at each moment and each frame of image data in the corresponding video using multi-threading; A behavior recognition module, wherein the behavior recognition module is used to input each frame of image data into a trained behavior recognition neural network model for recognition, so as to obtain a human action recognition result in the video at each moment; A millimeter wave data processing module, the millimeter wave data processing module is used to process the millimeter wave data to obtain micro-Doppler characteristic data with time dimension information; A data division and annotation module, wherein the data division and annotation module is used to divide and annotate the micro-Doppler feature data according to the human motion recognition result to obtain annotated millimeter wave data; Wherein, the behavior recognition module is also used for: Establish a neural network model for behavior recognition; Obtain image data of different actions that need to be recognized in different scenes to form a data set; Inputting the data set into a behavior recognition neural network model so as to train the behavior recognition neural network model to obtain a trained behavior recognition neural network model; Wherein, the millimeter wave data processing module is also used for: Performing short-time Fourier transform on the millimeter wave data according to the preceding and following frame signals of the millimeter wave data to obtain a range Doppler map; Projecting the range Doppler information of each signal cycle of the range Doppler image on the range dimension and accumulating each frame to obtain micro-Doppler feature data with time dimension information; The data division and labeling module is also used to further divide the millimeter wave data on the time axis according to the start and end of the action in the behavior recognition result, and obtain labels with the start and end of the action to complete the automatic labeling of the millimeter wave data.

4. The video-based millimeter wave data automatic annotation system according to claim 3, characterized in that: The data synchronization acquisition module is also used to: The synchronous control thread issues an acquisition instruction to the millimeter wave acquisition thread and the camera acquisition thread, so as to control the millimeter wave acquisition thread and the camera acquisition thread to start collecting data at the same time; The synchronization control thread falls into sleep mode so as to wait for the millimeter wave acquisition thread and the camera acquisition thread to complete one data acquisition; The millimeter wave acquisition thread and the camera acquisition thread start to collect data, and after the acquisition is completed, the synchronization control thread is awakened by the longest thread to start the acquisition of the next frame of data, and the cycle is repeated until the acquisition is completed.

Images