Data generation method and device, electronic equipment and computer readable storage medium

Abstract

Embodiments of the present application disclose a data generation method and device, electronic equipment and computer readable storage medium, and relate to the technical field of data processing. The method comprises: obtaining gesture motion data, identifying at least one target peak value from the gesture motion data, extracting at least one gesture segment data from the gesture motion data according to each target peak value, labeling each gesture segment data to obtain a target gesture data set. By extracting the target peak value from the gesture motion data and extracting the gesture segment data from the gesture motion data based on the target peak value, the effectiveness and accuracy of the gesture segment data are improved, and then after labeling based on the gesture segment data to obtain the target gesture data set, the precision of the target gesture data set is also improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data generation method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Gesture recognition technology, as a natural and intuitive human-computer interaction method, shows broad application prospects in consumer electronics, smart homes, medical rehabilitation, virtual reality / augmented reality (VR / AR), and vehicle control. Especially in wearable devices (such as smartwatches and smart bracelets), where screen size is limited, gesture recognition provides a convenient and efficient supplementary interaction solution. For example, it can be used to quickly launch applications, control music playback, answer / hang up calls, view notifications, and operate devices in special scenarios (such as during exercise or with wet hands), thus enhancing the user's interactive experience.

[0003] Inertial Measurement Unit (IMU)-based gesture recognition has garnered significant attention in wearable devices due to its advantages such as low power consumption, small size, and insensitivity to ambient light. Current gesture recognition technologies primarily rely on gesture recognition models, and high-quality, high-precision gesture recognition models are fundamental to achieving high-quality interaction in the aforementioned application scenarios. However, training gesture recognition models requires accurately labeled gesture datasets.

[0004] Currently, there are schemes designed to collect data on specific gestures using motion capture systems to obtain gesture datasets. However, this method of continuously executing and collecting data on each specific gesture is inefficient. Schemes have also been designed to extract multiple gesture segments from natural, continuous gesture motion data to improve collection efficiency. However, current gesture segment data extraction is coarse, making it difficult to extract complete and effective gesture segment data, resulting in limited accuracy of the collected gesture datasets. Summary of the Invention

[0005] This application provides a data generation method, apparatus, electronic device, and computer-readable storage medium, which can improve the effectiveness and completeness of the extracted gesture fragment data, thereby improving the accuracy of the collected gesture dataset.

[0006] In a first aspect, embodiments of this application provide a data generation method, the method comprising:

[0007] Acquire gesture motion data;

[0008] Identify at least one target peak from the gesture motion data;

[0009] At least one gesture segment data is extracted from the gesture motion data based on each of the target peak values;

[0010] The gesture fragment data are labeled to obtain the target gesture dataset.

[0011] Secondly, embodiments of this application also provide a data generation apparatus, the apparatus comprising:

[0012] The acquisition module is used to acquire gesture motion data;

[0013] The recognition module is used to identify at least one target peak from the gesture motion data;

[0014] The extraction module is used to extract at least one gesture segment data from the gesture motion data based on each of the target peak values;

[0015] The annotation module is used to annotate each of the gesture fragment data to obtain the target gesture dataset.

[0016] Optionally, in some embodiments of this application, identifying at least one target peak from the gesture motion data includes:

[0017] Identify local peaks in the gesture motion data within at least one time window;

[0018] At least one target peak is selected from each of the local peaks according to the peak threshold.

[0019] Optionally, in some embodiments of this application, the gesture motion data includes acceleration data;

[0020] The identification of local peaks in the gesture motion data within at least one time window includes:

[0021] For each time window, the derivative of the magnitude data corresponding to the acceleration data in that time window is calculated to obtain a derivative dataset, which includes the derivative data corresponding to the magnitude data of at least one timestamp in the time window.

[0022] The derivative data points with values ​​greater than zero in the derivative dataset are selected as candidate peak values.

[0023] The maximum value is selected from the candidate peaks to obtain the local peak.

[0024] Optionally, in some embodiments of this application, the step of filtering out the maximum value from each of the candidate peaks to obtain the local peak includes:

[0025] Adjacent candidate peaks are compared in the order of timestamps, and the larger candidate peak is compared with the next candidate peak until the comparison of the last candidate peak in the time window is completed, thus obtaining the largest candidate peak.

[0026] Set the largest candidate peak as the local peak.

[0027] Optionally, in some embodiments of this application, the step of filtering out the maximum value from each of the candidate peaks to obtain the local peak includes:

[0028] The candidate peaks are added to the peak stack sequentially according to their timestamps, and...

[0029] If the current candidate peak value is greater than the previous candidate peak value, then the previous candidate peak value is removed from the peak value stack, and the current candidate peak value is retained.

[0030] If the current candidate peak value is less than the previous candidate peak value, then the current candidate peak value and the previous candidate peak value are retained. If the next candidate peak value is greater than the previous candidate peak value and the next candidate peak value is greater than the current candidate peak value, then the current candidate peak value and the previous candidate peak value are removed, and the next candidate peak value is retained.

[0031] After the last candidate peak in the time window is added to the peak stack and its size is compared, the candidate peak retained in the peak stack or the candidate peak at the top of the peak stack is set as the local peak, wherein the size of the candidate peaks in the peak stack decreases sequentially from the top to the bottom.

[0032] Optionally, in some embodiments of this application, the step of extracting at least one gesture segment data from the gesture motion data based on each of the target peak values ​​includes:

[0033] For each target peak, gesture segment data with a preset duration centered on the target peak is extracted from the gesture motion data.

[0034] Optionally, in some embodiments of this application, the method is applied to an extended reality device;

[0035] The step of labeling each of the gesture fragment data to obtain the target gesture dataset includes:

[0036] The original annotation results of each gesture segment data are obtained by annotating each gesture segment data using a pre-deployed model.

[0037] The original annotation results are displayed on a virtual screen.

[0038] Upon receiving the first modification information input for the original annotation result, a preliminary verification result is obtained;

[0039] The gesture segment data, gesture video data, and the preliminary verification result are displayed in an aligned manner, wherein the gesture video data is captured synchronously with the gesture motion data;

[0040] In response to receiving the second modification information input for the preliminary verification result, the target verification result is obtained;

[0041] The target gesture dataset is generated based on the target verification results.

[0042] Thirdly, embodiments of this application also provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps in the data generation method described above.

[0043] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the data generation method described above.

[0044] Fifthly, embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in embodiments of this application.

[0045] In summary, the embodiments of this application acquire gesture motion data, identify at least one target peak from the gesture motion data, extract at least one gesture segment data from the gesture motion data based on each target peak, and label each gesture segment data to obtain a target gesture dataset.

[0046] Specifically, by extracting target peaks from gesture motion data and then extracting gesture fragment data from the gesture motion data based on these target peaks, the effectiveness and accuracy of gesture fragment data are improved. Consequently, after annotating the gesture fragment data to obtain the target gesture dataset, the accuracy of the target gesture dataset is also improved. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of a scenario in which an extended reality device, as provided in an embodiment of this application, performs the data generation method;

[0049] Figure 2 This is a flowchart illustrating the data generation method provided in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of the structure of the data generation device provided in the embodiments of this application;

[0051] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0052] Explanation of icon numbers:

[0053] 101-Extended Reality Device; 102-Smartwatch; 301-Acquisition Module; 302-Recognition Module; 303-Extraction Module; 304-Annotation Module; 401-Processor; 402-Memory; 403-Power Supply; 404-Input Unit. Detailed Implementation

[0054] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] This application provides a data generation method, apparatus, electronic device, and computer-readable storage medium. Specifically, this application provides a data generation apparatus suitable for electronic devices, which include terminal devices or servers. The terminal devices include, but are not limited to, mobile phones, tablets, laptops, smartwatches, smart bracelets, or extended reality devices. The extended reality devices include virtual reality (VR) devices, augmented reality (AR) devices, and mixed reality (MR) devices. In this application embodiment, the extended reality device includes at least wearable electronic devices in the form of glasses or head-mounted displays. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The server can be directly or indirectly connected via wired or wireless communication.

[0056] For example, please see Figure 1 Taking extended reality devices as an example to implement this data generation method, Figure 1This is a schematic diagram illustrating a scenario where an extended reality device, according to an embodiment of this application, executes the data generation method. Specifically, the execution process of the data generation method by the extended reality device is as follows:

[0057] The extended reality device 101 acquires gesture motion data, which is collected by the smartwatch 102 through its built-in IMU sensor. The smartwatch 102 sends the collected gesture motion data to the extended reality device 101. Subsequently, after receiving the gesture motion data, the extended reality device 101 identifies at least one target peak from the gesture motion data, extracts at least one gesture segment data from the gesture motion data based on the target peak, and annotates the gesture segment data to obtain a target gesture dataset.

[0058] For example, by focusing on the peak acceleration, peak velocity, and peak angular velocity of the gesture movement, gesture fragment data can be extracted from the gesture movement data to obtain relatively complete and effective gesture fragment data.

[0059] In summary, the embodiments of this application improve the effectiveness and accuracy of gesture fragment data by extracting target peaks from gesture motion data and extracting gesture fragment data from gesture motion data based on target peaks. Furthermore, after annotating the gesture fragment data to obtain the target gesture dataset, the accuracy of the target gesture dataset is also improved.

[0060] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0061] Please see Figure 2 , Figure 2 This is a flowchart illustrating the data generation method provided in this application embodiment. Although the flowchart shows a logical order, in some cases, the steps shown or described can be performed in a different order than that shown in the flowchart. Specifically, the specific flow of this data generation method is as follows:

[0062] 201. Obtain gesture motion data.

[0063] The gesture motion data is obtained by collecting data from an IMU sensor, such as a smartwatch or smart bracelet with a built-in IMU sensor, which collects data on the user's hand movements.

[0064] Accordingly, in this embodiment of the application, the gesture motion data can be obtained by receiving a smartwatch or smart bracelet.

[0065] 202. Identify at least one target peak from the gesture motion data.

[0066] In this context, a peak value refers to a local maximum (also called a maximum point) of a physical quantity in time or space. For example, for gesture motion data, the corresponding peak values ​​include peak values ​​based on acceleration, velocity, angular velocity, etc. Target peak values ​​are specific peak values ​​selected from multiple peak values. In essence, target peak values ​​are the result of filtering and selecting from the existing peak values. For instance, by filtering out invalid peak values ​​based on a peak threshold, several target peak values ​​that can characterize the gesture motion are obtained. These target peak values ​​might include the peak values ​​of acceleration, velocity, or angular velocity at their highest points during the gesture motion. Each target peak value corresponds to a valid gesture motion.

[0067] 203. Extract at least one gesture segment data from the gesture motion data based on each of the target peak values.

[0068] Understandably, since the target peak is a filtered and effective peak, the validity and accuracy of the gesture fragment data extracted based on the target peak are guaranteed. For example, gesture fragment data containing each target peak is extracted from gesture motion data, so that each gesture fragment data corresponds to a valid gesture motion.

[0069] Understandably, using the target peak value as a reference for gesture segment data extraction improves the accuracy of gesture segment data extraction and helps to obtain complete and effective gesture segment data.

[0070] 204. Label each of the gesture fragment data to obtain the target gesture dataset.

[0071] Among them, annotation refers to the annotation of the gesture type corresponding to each gesture fragment data. For example, the gesture type to which each gesture fragment data belongs is annotated, such as finger click, finger movement, palm probing or palm extension.

[0072] It is understandable that after labeling each gesture fragment data, gesture fragment data labeled with gesture types is obtained. This data can be used as the target gesture dataset for training the gesture recognition model, so that the trained gesture recognition model can learn the relationship between gesture fragment data and gesture types. In turn, it can identify and classify various gesture fragment data, analyze their corresponding gesture types, and realize the function of gesture recognition.

[0073] In summary, the embodiments of this application improve the effectiveness and accuracy of gesture fragment data by extracting target peaks from gesture motion data and extracting gesture fragment data from gesture motion data based on target peaks. Furthermore, after annotating the gesture fragment data to obtain the target gesture dataset, the accuracy of the target gesture dataset is also improved.

[0074] Optionally, in this embodiment of the application, in order to further improve the effectiveness and accuracy of the collected gesture segment data, this embodiment of the application uses the target peak as the center and controls the extraction of gesture segment data according to a preset duration. That is, optionally, in some embodiments of this application, the step "extracting at least one gesture segment data from the gesture motion data according to each of the target peaks" includes:

[0075] For each target peak, gesture segment data with a preset duration centered on the target peak is extracted from the gesture motion data.

[0076] For example, for each target peak, data segments 500ms before and 500ms after the target peak are extracted from the gesture motion data, centered on the target peak, and combined to obtain gesture segment data with a duration of approximately 1 second.

[0077] It is understandable that by taking the target peak as the center and combining the data segments before and after it, the gesture segment data can be obtained, which helps to obtain gesture segment data centered on the gesture action, thus improving the effectiveness and accuracy of the gesture segment data.

[0078] Optionally, in this embodiment, local peak values ​​of each time window can be extracted separately according to the time window, and effective target peak values ​​can be obtained by peak threshold filtering. That is, optionally, in some embodiments of this application, the step "identifying at least one target peak value from the gesture motion data" includes:

[0079] Identify local peaks in the gesture motion data within at least one time window;

[0080] At least one target peak is selected from each of the local peaks according to the peak threshold.

[0081] The local peak is the largest peak among multiple timestamps corresponding to each time window, belonging to a local maximum value. The length of this time window can be set based on the general duration of the gesture; for example, the size of the time window can be set according to the general duration of the gesture.

[0082] The peak threshold is set based on the peak values ​​of the gestures. For example, by collecting the peak values ​​of multiple gestures, the minimum peak value corresponding to each gesture is set as the peak threshold. The purpose of setting the peak threshold is to filter out invalid local peak values ​​and obtain the valid target peak value for a given gesture. For instance, filtering out local peak values ​​corresponding to common, general gestures that do not contain specific instructions can prevent the acquisition of invalid gesture fragment data based on these invalid local peak values, thus improving the effectiveness and accuracy of the gesture fragment data collected by this solution.

[0083] For example, taking peak values ​​relative to acceleration as an example, i.e., the gesture motion data includes acceleration data, the step "identifying local peak values ​​of the gesture motion data within at least one time window" includes:

[0084] For each time window, the derivative of the magnitude data corresponding to the acceleration data in that time window is calculated to obtain a derivative dataset, which includes the derivative data corresponding to the magnitude data of at least one timestamp in the time window.

[0085] The derivative data points with values ​​greater than zero in the derivative dataset are selected as candidate peak values.

[0086] The maximum value is selected from the candidate peaks to obtain the local peak.

[0087] For example, the derivative of the acceleration data or the corresponding magnitude signal is calculated according to the time window, and the derivative data at zero-crossing points are set as candidate peak values. Furthermore, the maximum value among multiple candidate peak values ​​is selected as the maximum peak value corresponding to that time window, i.e., the local peak value.

[0088] Optionally, in this embodiment of the application, the local peak with the largest value can be selected by comparing candidate peaks. That is, the step "selecting the maximum value from each candidate peak to obtain the local peak" includes:

[0089] Adjacent candidate peaks are compared in the order of timestamps, and the larger candidate peak is compared with the next candidate peak until the comparison of the last candidate peak in the time window is completed, thus obtaining the largest candidate peak.

[0090] Set the largest candidate peak as the local peak.

[0091] For example, the candidate peaks are compared in order of timestamps, and the larger candidate peak is compared with the next candidate peak in order to obtain the maximum value among the multiple candidate peaks.

[0092] Optionally, in this embodiment, the principle of a monotonic stack can also be used to filter out the local peak with the largest value from multiple candidate peaks. That is, optionally, in some embodiments of this application, the step "filtering out the maximum value from each of the candidate peaks to obtain the local peak" includes:

[0093] The candidate peaks are added to the peak stack sequentially according to their timestamps, and...

[0094] If the current candidate peak value is greater than the previous candidate peak value, then the previous candidate peak value is removed from the peak value stack, and the current candidate peak value is retained.

[0095] If the current candidate peak value is less than the previous candidate peak value, then the current candidate peak value and the previous candidate peak value are retained. If the next candidate peak value is greater than the previous candidate peak value and the next candidate peak value is greater than the current candidate peak value, then the current candidate peak value and the previous candidate peak value are removed, and the next candidate peak value is retained.

[0096] After the last candidate peak in the time window is added to the peak stack and its size is compared, the candidate peak retained in the peak stack or the candidate peak at the top of the peak stack is set as the local peak, wherein the size of the candidate peaks in the peak stack decreases sequentially from the top to the bottom.

[0097] It should be noted that the peak stack has a monotonic property, meaning that the elements in the stack always remain monotonically increasing or monotonically decreasing.

[0098] For example, in this embodiment of the application, each candidate peak is added to the peak stack in the order of timestamps, and the added candidate peak is compared with the previously added and retained candidate peak using the peak stack. By removing the smaller one and retaining the largest one, the stack always retains the larger candidate peak. Then, after each candidate peak is added to the peak stack and compared, the local peak with the largest value is obtained.

[0099] For example, taking local peak values ​​of 0.10, 0.15, 0.20, 0.18, and 0.22, with corresponding timestamps of 1.0s, 1.2s, 1.3s, 1.5s, and 1.7s, the specific processing steps for the peak stack include:

[0100] Step 1: Receive the candidate peak value of 0.10 and push it onto the stack;

[0101] The second step is to receive the candidate peak value of 0.15, compare the candidate peak value of 0.15 with the candidate peak value of 0.10, and remove the candidate peak value of 0.10 and keep the candidate peak value of 0.15, or push the candidate peak value of 0.15 onto the stack.

[0102] The third step is to receive the candidate peak value of 0.20, compare the candidate peak value of 0.20 with the candidate peak value of 0.15, and remove the candidate peak value of 0.15 and keep the candidate peak value of 0.20, or push the candidate peak value of 0.20 onto the stack.

[0103] Step 4: Receive candidate peak value 0.18, compare candidate peak value 0.20 and candidate peak value 0.18. Since 0.18 < 0.20, and candidate peak value 0.18 is the newly added candidate peak value, push candidate peak value 0.18 onto the stack. At this time, the stack retains candidate peak value 0.20 and candidate peak value 0.18.

[0104] Step 5: Receive candidate peak value 0.22, compare candidate peak value 0.22 with candidate peak values ​​0.20 and 0.18. Since 0.22 > 0.20 and 0.22 > 0.18, remove candidate peak value 0.20 and remove candidate peak value 0.18, and retain candidate peak value 0.22, or push candidate peak value 0.22 onto the stack.

[0105] After processing, only the largest candidate peak value of 0.22 is retained in the peak stack, and this candidate peak value of 0.22 is taken as the current local peak value. Correspondingly, its timestamp is 1.7s.

[0106] For another example, if filtering by peak threshold yields target peak values ​​of 0.22 (corresponding to a timestamp of 1.7s), 0.25 (corresponding to a timestamp of 3.2s), 0.26 (corresponding to a timestamp of 5.4s), and 0.27 (corresponding to a timestamp of 6.9s), then four gesture segments of approximately 1 second each can be extracted from the gesture motion data, centered at timestamps of 1.7s, 3.2s, 5.4s, and 6.9s. After labeling each gesture segment, a four-part labeled gesture dataset is obtained. This dataset can then be used to train a gesture recognition model.

[0107] Optionally, in this embodiment, after obtaining the gesture fragment data, a local lightweight model can be used first to perform gesture prediction and inference to obtain the original annotation results of the gesture fragment data. Then, manual verification is combined to obtain manually verified annotation data. That is, optionally, in some embodiments of this application, the step "annotating each of the gesture fragment data to obtain the target gesture dataset" includes:

[0108] The original annotation results of each gesture segment data are obtained by annotating each gesture segment data using a pre-deployed model.

[0109] The original annotation results are displayed on a virtual screen.

[0110] Upon receiving the first modification information input for the original annotation result, a preliminary verification result is obtained;

[0111] The gesture segment data, gesture video data, and the preliminary verification result are displayed in an aligned manner, wherein the gesture video data is captured synchronously with the gesture motion data;

[0112] In response to receiving the second modification information input for the preliminary verification result, the target verification result is obtained;

[0113] The target gesture dataset is generated based on the target verification results.

[0114] It should be noted that the pre-deployed model includes a local lightweight model, which is a pre-deployed gesture recognition model. This pre-deployed model can be the same as or different from the gesture recognition model to be trained on the target gesture dataset. For example, if the pre-deployed model is different from the gesture recognition model to be trained, inference can be performed using another model, combined with manual verification to speed up the annotation of gesture data. After manually calibrating and obtaining the annotated gesture data, the gesture recognition model to be trained can then be used. Alternatively, if the pre-deployed model is the gesture recognition model to be trained, preliminary inference can be performed based on its general inference capabilities, combined with manual annotation. This also reduces the workload of manual annotation. After obtaining the annotated gesture data, the pre-deployed model can be trained to improve its inference capabilities.

[0115] It is understood that, in the embodiments of this application, the annotation results can also be calibrated by combining video data of hand gestures. For example, when hand gestures generate hand motion data, hand gesture video data can be captured simultaneously using a camera. Correspondingly, the hand gesture video data and hand gesture segment data can be displayed in alignment so that users can calibrate the hand gesture segment data and the annotation results corresponding to the hand gesture segment data by displaying the hand gesture video data.

[0116] For example, by displaying the original annotation results of a pre-deployed model on a virtual display screen of an augmented reality device, users can modify and confirm these results through gestures, a matching ring or other tools, or eye-tracking data to obtain preliminary verification results. Furthermore, the virtual display screen of the augmented reality device can also be used to align and display gesture video data and gesture segment data, allowing users to review the gesture video data and recalibrate the gesture segment data and the corresponding preliminary verification results to obtain more accurate target verification results.

[0117] It is understood that the data generation method of this application embodiment is applicable to watches, mobile phones, computers, extended reality devices, or data generation systems composed of watches, mobile phones and computers, or watches, extended reality devices and computers.

[0118] For example, taking a data generation system consisting of a watch, a mobile phone, and a computer as an example, the mobile phone and the watch are connected via BLE. The user sends a command to start collecting data via the mobile phone. The mobile phone then controls the watch to collect the user's hand gesture motion data based on the IMU sensor via BLE. The mobile phone also captures hand gesture video data at the same time. The watch can then directly obtain the local peak values ​​of each time window through peak detection, and further obtain the target peak value through peak stack. According to each target peak value, it obtains multiple gesture segment data from the gesture motion data. Then, the watch calls the pre-deployed model to identify the original annotation results corresponding to each gesture segment data. Finally, the watch sends the gesture segment data, gesture motion data, and original annotation results to the mobile phone.

[0119] After receiving the gesture fragment data, gesture motion data, and original annotation results sent by the watch, the mobile phone displays the original annotation results corresponding to the gesture fragment data on the mobile phone interface. Users can confirm or modify the original annotation results through the mobile phone interface to obtain the preliminary verification results of the first manual confirmation.

[0120] Then, the mobile phone can send the gesture segment data, gesture motion data, original annotation results, and gesture video data to the computer (or it can first upload to the cloud and then download it from the cloud). The computer uses the ELAN tool to align and display the gesture video data and gesture segment data. The user can use the ELAN tool and the gesture video data to further verify the preliminary verification results of the gesture segment data to obtain the target verification result. Then, based on the target verification result, the annotated target gesture dataset is obtained.

[0121] In summary, the embodiments of this application improve the effectiveness and accuracy of gesture fragment data by extracting target peaks from gesture motion data and extracting gesture fragment data from gesture motion data based on target peaks. Furthermore, after annotating the gesture fragment data to obtain the target gesture dataset, the accuracy of the target gesture dataset is also improved.

[0122] Furthermore, the efficiency of target peak selection is improved by filtering target peaks based on a monotonic peak stack.

[0123] Furthermore, by extracting gesture fragment data centered on the target peak and according to a preset duration, the extracted gesture fragment data becomes more effective and complete.

[0124] To facilitate better implementation of the data generation method of this application, this application also provides a data generation apparatus based on the above-described data generation method. The meanings of the terms used are the same as in the data generation method described above, and specific implementation details can be found in the descriptions of the method embodiments.

[0125] Please see Figure 3 , Figure 3This is a schematic diagram of the structure of the data generation device provided in the embodiments of this application, wherein the data generation device may specifically be as follows:

[0126] The acquisition module 301 is used to acquire gesture motion data;

[0127] The recognition module 302 is used to identify at least one target peak from the gesture motion data;

[0128] Extraction module 303 is used to extract at least one gesture segment data from the gesture motion data based on each of the target peak values;

[0129] The annotation module 304 is used to annotate each of the gesture fragment data to obtain the target gesture dataset.

[0130] Optionally, in some embodiments of this application, identifying at least one target peak from the gesture motion data includes:

[0131] Identify local peaks in the gesture motion data within at least one time window;

[0132] At least one target peak is selected from each of the local peaks according to the peak threshold.

[0133] Optionally, in some embodiments of this application, the gesture motion data includes acceleration data;

[0134] The identification of local peaks in the gesture motion data within at least one time window includes:

[0135] For each time window, the derivative of the magnitude data corresponding to the acceleration data in that time window is calculated to obtain a derivative dataset, which includes the derivative data corresponding to the magnitude data of at least one timestamp in the time window.

[0136] The derivative data points with values ​​greater than zero in the derivative dataset are selected as candidate peak values.

[0137] The maximum value is selected from the candidate peaks to obtain the local peak.

[0138] Optionally, in some embodiments of this application, the step of filtering out the maximum value from each of the candidate peaks to obtain the local peak includes:

[0139] Adjacent candidate peaks are compared in the order of timestamps, and the larger candidate peak is compared with the next candidate peak until the comparison of the last candidate peak in the time window is completed, thus obtaining the largest candidate peak.

[0140] Set the largest candidate peak as the local peak.

[0141] Optionally, in some embodiments of this application, the step of filtering out the maximum value from each of the candidate peaks to obtain the local peak includes:

[0142] The candidate peaks are added to the peak stack sequentially according to their timestamps, and...

[0143] If the current candidate peak value is greater than the previous candidate peak value, then the previous candidate peak value is removed from the peak value stack, and the current candidate peak value is retained.

[0144] If the current candidate peak value is less than the previous candidate peak value, then the current candidate peak value and the previous candidate peak value are retained. If the next candidate peak value is greater than the previous candidate peak value and the next candidate peak value is greater than the current candidate peak value, then the current candidate peak value and the previous candidate peak value are removed, and the next candidate peak value is retained.

[0145] After the last candidate peak in the time window is added to the peak stack and its size is compared, the candidate peak retained in the peak stack or the candidate peak at the top of the peak stack is set as the local peak, wherein the size of the candidate peaks in the peak stack decreases sequentially from the top to the bottom.

[0146] Optionally, in some embodiments of this application, the step of extracting at least one gesture segment data from the gesture motion data based on each of the target peak values ​​includes:

[0147] For each target peak, gesture segment data with a preset duration centered on the target peak is extracted from the gesture motion data.

[0148] Optionally, in some embodiments of this application, the method is applied to an extended reality device;

[0149] The step of labeling each of the gesture fragment data to obtain the target gesture dataset includes:

[0150] The original annotation results of each gesture segment data are obtained by annotating each gesture segment data using a pre-deployed model.

[0151] The original annotation results are displayed on a virtual screen.

[0152] Upon receiving the first modification information input for the original annotation result, a preliminary verification result is obtained;

[0153] The gesture segment data, gesture video data, and the preliminary verification result are displayed in an aligned manner, wherein the gesture video data is captured synchronously with the gesture motion data;

[0154] In response to receiving the second modification information input for the preliminary verification result, the target verification result is obtained;

[0155] The target gesture dataset is generated based on the target verification results.

[0156] In this embodiment, the acquisition module 301 first acquires gesture motion data, the recognition module 302 identifies at least one target peak from the gesture motion data, the extraction module 303 extracts at least one gesture segment data from the gesture motion data based on each target peak, and the annotation module 304 annotates each gesture segment data to obtain a target gesture dataset.

[0157] In this embodiment of the application, by extracting a target peak from the gesture motion data and extracting gesture fragment data from the gesture motion data based on the target peak, the effectiveness and accuracy of the gesture fragment data are improved. In turn, after the target gesture dataset is obtained by labeling based on the gesture fragment data, the accuracy of the target gesture dataset is also improved.

[0158] In addition, this application also provides an electronic device, such as Figure 4 As shown, it illustrates a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically:

[0159] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0160] The processor 401 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0161] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0162] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power equipment debugging circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0163] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0164] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402, thereby implementing the steps in any of the data generation methods provided in the embodiments of this application.

[0165] This application embodiment acquires gesture motion data, identifies at least one target peak from the gesture motion data, extracts at least one gesture segment data from the gesture motion data based on each target peak, and annotates each gesture segment data to obtain a target gesture dataset.

[0166] Specifically, by extracting target peaks from gesture motion data and then extracting gesture fragment data from the gesture motion data based on these target peaks, the effectiveness and accuracy of gesture fragment data are improved. Consequently, after annotating the gesture fragment data to obtain the target gesture dataset, the accuracy of the target gesture dataset is also improved.

[0167] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0168] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0169] Therefore, this application provides a computer-readable storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the data generation methods provided in this application.

[0170] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0171] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0172] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the data generation methods provided in this application, the beneficial effects that any of the data generation methods provided in this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0173] The foregoing has provided a detailed description of a data generation method, apparatus, electronic device, and computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data generation method, characterized in that, The method includes: Acquire gesture motion data; Identify at least one target peak from the gesture motion data; At least one gesture segment data is extracted from the gesture motion data based on each of the target peak values; The gesture fragment data are labeled to obtain the target gesture dataset; The gesture motion data includes acceleration data, and identifying at least one target peak from the gesture motion data includes: For each time window, the derivative of the magnitude data corresponding to the acceleration data in that time window is calculated to obtain a derivative dataset, which includes the derivative data corresponding to the magnitude data of at least one timestamp in the time window. The derivative data points with values ​​greater than zero in the derivative dataset are selected as candidate peak values. The maximum value is selected from all the candidate peak values ​​to obtain the local peak value; At least one target peak is selected from each of the local peaks according to the peak threshold; The step of selecting the maximum value from each of the candidate peaks to obtain the local peak includes: The local peak with the largest value is selected from multiple candidate peaks using the principle of a monotonic stack.

2. The data generation method according to claim 1, characterized in that, The step of selecting the maximum value from each of the candidate peaks to obtain the local peak includes: The candidate peaks are added to the peak stack sequentially according to their timestamps, and... If the current candidate peak value is greater than the previous candidate peak value, then the previous candidate peak value is removed from the peak value stack, and the current candidate peak value is retained. If the current candidate peak value is less than the previous candidate peak value, then the current candidate peak value and the previous candidate peak value are retained. If the next candidate peak value is greater than the previous candidate peak value and the next candidate peak value is greater than the current candidate peak value, then the current candidate peak value and the previous candidate peak value are removed, and the next candidate peak value is retained. After the last candidate peak in the time window is added to the peak stack and its size is compared, the candidate peak retained in the peak stack or the candidate peak at the top of the peak stack is set as the local peak, wherein the size of the candidate peaks in the peak stack decreases sequentially from the top to the bottom.

3. The data generation method according to claim 1, characterized in that, The step of extracting at least one gesture segment data from the gesture motion data based on each of the target peak values ​​includes: For each target peak, gesture segment data with a preset duration centered on the target peak is extracted from the gesture motion data.

4. The data generation method according to claim 1, characterized in that, The method is applied to augmented reality devices; The step of labeling each of the gesture fragment data to obtain the target gesture dataset includes: The original annotation results of each gesture segment data are obtained by annotating each gesture segment data using a pre-deployed model. The original annotation results are displayed on a virtual screen. Upon receiving the first modification information input for the original annotation result, a preliminary verification result is obtained; The gesture segment data, gesture video data, and the preliminary verification result are displayed in an aligned manner, wherein the gesture video data is captured synchronously with the gesture motion data; In response to receiving the second modification information input for the preliminary verification result, the target verification result is obtained; The target gesture dataset is generated based on the target verification results.

5. A data generation device, characterized in that, The device includes: The acquisition module is used to acquire gesture motion data; The recognition module is used to identify at least one target peak from the gesture motion data; The extraction module is used to extract at least one gesture segment data from the gesture motion data based on each of the target peak values; The annotation module is used to annotate each of the gesture fragment data to obtain the target gesture dataset; The gesture motion data includes acceleration data, and identifying at least one target peak from the gesture motion data includes: For each time window, the derivative of the magnitude data corresponding to the acceleration data in that time window is calculated to obtain a derivative dataset, which includes the derivative data corresponding to the magnitude data of at least one timestamp in the time window. The derivative data points with values ​​greater than zero in the derivative dataset are selected as candidate peak values. The maximum value is selected from all the candidate peak values ​​to obtain the local peak value; At least one target peak is selected from each of the local peaks according to the peak threshold; The step of selecting the maximum value from each of the candidate peaks to obtain the local peak includes: The local peak with the largest value is selected from multiple candidate peaks using the principle of a monotonic stack.

6. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the data generation method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data generation method as described in any one of claims 1-4.

Images