Virtual reality motion capture method and system, computer equipment and medium

Abstract

The invention provides a virtual reality motion capture method and system, computer equipment and a medium. The method comprises the steps of collecting virtual reality motion feature time sequence data, audio time sequence data and image time sequence data of a preset duration; extracting hand action feature data according to the virtual reality action feature time sequence data, identifying a current action type according to the hand action feature data, and generating a corresponding manuscript generation instruction according to the current action type; performing identification processing on the image time sequence data, and generating a corresponding virtual manuscript according to the obtained handwriting content and the manuscript generation instruction; and performing similarity comparison on the audio time sequence data and the virtual text, editing and updating the virtual text according to an obtained similarity comparison result, and loading the updated virtual text. According to the method, the action capture accuracy in the VR conference scene can be improved, meanwhile, accurate generation and effective editing of the conference record manuscript can be guaranteed, and the application value of the VR conference scene is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality motion capture technology, and in particular to a virtual reality motion capture method, system, computer equipment and medium. Background Art

[0002] With the continuous development and progress of virtual reality (VR) technology, motion capture and VR scene applications have also shown great potential in conference scenarios. However, due to the technical design limitations of traditional VR motion capture systems that rely on gaze tracking technology to ensure that the captured movements match the user's gaze direction, it is not only difficult to apply to conference scenarios where users need to frequently check meeting materials, notebooks or other participants instead of always looking at the VR interface, but also greatly limits the freedom and flexibility of users' conference communication, directly reducing the user's meeting experience.

[0003] Although existing VR motion capture devices are capable of capturing user hand and body movements, they are difficult to ensure motion capture accuracy and real-time performance in highly dynamic or complex action scenarios, so their application in existing VR conference scenarios is mainly focused on simple demonstrations and interactions. No corresponding solutions are provided for the organization of meeting minutes and manuscripts, resulting in the inability to meet the application requirements of high-end VR conference scenarios such as meeting minutes and remote collaboration without line of sight, reducing the use value of VR conference scenarios. Summary of the invention

[0004] The purpose of the present invention is to provide a virtual reality motion capture method, which improves the accuracy of motion recognition from the aspect of identifying detailed hand movements by comprehensively analyzing hand feature data such as wrist motion trajectory, pen grip point pressure and hand movement speed. At the same time, by combining motion feature data, audio data and image data to generate a virtual text image in a VR conference scene, it avoids the errors and limitations of relying on a single data to obtain virtual text, significantly improves the generation quality of virtual text for conference scenes, and effectively enhances the application value of VR conference scenes.

[0005] In order to achieve the above objectives, it is necessary to provide a virtual reality motion capture method, system, computer equipment and medium in response to the above technical problems.

[0006] In a first aspect, an embodiment of the present invention provides a virtual reality motion capture method, the method comprising the following steps: Collecting virtual reality action feature time series data, audio time series data, and image time series data of preset duration; Extracting hand motion feature data according to the virtual reality motion feature time series data, identifying the current motion type according to the hand motion feature data, and generating corresponding document generation instructions according to the current motion type; the document generation instructions include text generation instructions and picture generation instructions; Recognize and process the image time series data, and generate a corresponding virtual document according to the obtained handwriting content and the document generation instruction; the virtual document includes virtual text and picture annotations; The audio time sequence data is compared with the virtual text for similarity, the virtual text is edited and updated according to the obtained similarity comparison result, and the updated virtual text is loaded.

[0007] Furthermore, the hand motion feature data includes wrist motion trajectory features, pen grip point pressure change features and hand motion speed change features; The step of extracting hand motion feature data according to the virtual reality motion feature data comprises: Extracting wrist motion features from the virtual reality action feature time series data to obtain wrist motion time series features, and generating the wrist motion trajectory features based on the wrist motion time series features; Extracting the pen holding point pressure feature in the virtual reality action feature time series data to obtain the pen holding point pressure time series feature, and generating the pen holding point pressure change feature according to the pen holding point pressure time series feature; The hand movement features in the virtual reality action feature time series data are extracted to obtain the hand movement speed time series features, and the hand movement speed change features are generated according to the hand movement speed time series features.

[0008] Furthermore, the step of identifying the current action type according to the hand action feature data, and generating a corresponding document generation instruction according to the current action type includes: Constructing a wrist motion trajectory continuity function according to the wrist motion trajectory characteristics; According to the pen holding point pressure change characteristics, construct a pen holding point pressure change frequency function; Constructing a hand movement speed change weighting function according to the hand movement speed change characteristics; Obtaining a corresponding action recognition factor according to the wrist motion trajectory continuity function, the pen grip point pressure change frequency function and the hand motion speed change function; Determine whether the action recognition factor is greater than a preset factor threshold. If so, determine the current action type as a drawing action and generate the drawing generation instruction. Otherwise, determine the current action type as a writing action and generate the text generation instruction.

[0009] Furthermore, the action recognition factor is expressed as: In the formula, in, represents the action recognition factor at time t; represents the continuity value of the wrist motion trajectory at time t, k represents a constant, represents the displacement change of the wrist at time t; represents the frequency of the pressure change at the pen holding point at time t, A represents the amplitude, represents the i-th pressure peak moment, σ represents the standard deviation of the Gaussian distribution of pressure change, and n represents the number of pressure peaks; represents the weighted value of the hand movement speed change at time t, Indicates the change in hand movement speed at time t; , and represents a constant coefficient.

[0010] Furthermore, the step of performing a similarity comparison between the audio time sequence data and the virtual text and editing and updating the virtual text according to the obtained similarity comparison result includes: Differentiate the audio time sequence data by timbre to obtain a plurality of audio feature data; Extracting preset keywords from the virtual text to obtain a number of text feature data; Perform text similarity analysis based on all audio feature data, all text feature data and a preset text similarity calculation model to obtain corresponding similarity comparison results; If the similarity comparison result is less than a preset similarity threshold, the audio time sequence data is converted into corresponding audio text data, and the virtual text is updated and edited according to the audio text data.

[0011] Furthermore, the preset text similarity calculation model is expressed as: in, represents text similarity; J represents the number of audio feature data; M represents the number of text feature data; , and Respectively represent the frequency change rate, duration and length of the extracted text content of the j-th audio feature data; , and Represents the weight coefficient of audio feature data; and They represent the text word length and occurrence frequency of the mth text feature data respectively; δ, θ and ε represent the weight coefficients of the text feature data.

[0012] Furthermore, the method further comprises: The virtual reality action feature timing data is compared with the preset indication action standard feature data, and corresponding action instructions are generated according to the matched preset indication action standard feature data, so as to perform virtual interaction according to the action instructions; the action instructions include document transfer instructions and document modification instructions.

[0013] In a second aspect, an embodiment of the present invention provides a virtual reality motion capture system, the system comprising: A data collection module, used to collect virtual reality action feature time series data, audio time series data and image time series data of a preset duration; an action recognition module, configured to extract hand action feature data according to the virtual reality action feature time series data, identify a current action type according to the hand action feature data, and generate a corresponding document generation instruction according to the current action type; the document generation instruction includes a text generation instruction and a picture generation instruction; A document generation module, used to perform recognition processing on the image time series data, and generate a corresponding virtual document according to the obtained handwriting content and the document generation instruction; the virtual document includes virtual text and picture annotations; The manuscript updating module is used to perform a similarity comparison between the audio timing data and the virtual text, edit and update the virtual text according to the similarity comparison result, and load the updated virtual text.

[0014] Furthermore, the system further comprises: The interactive action recognition module is used to compare the virtual reality action feature timing data with the preset indication action standard feature data, and generate corresponding action instructions according to the matching preset indication action standard feature data, so as to perform virtual interaction according to the action instructions; the action instructions include document transfer instructions and document modification instructions.

[0015] In a third aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0017] The present invention provides a virtual reality motion capture method, system, computer device and storage medium. The method realizes the collection of virtual reality motion feature time series data, audio time series data and image time series data of preset duration, extracts hand motion feature data according to the virtual reality motion feature time series data, identifies the current motion type according to the hand motion feature data, generates a document generation instruction corresponding to the current motion type including text generation instruction and picture generation instruction, and performs recognition processing on the image time series data, generates a virtual document corresponding to virtual text and picture annotation according to the obtained handwriting content and the document generation instruction, and compares the audio time series data with the virtual text for similarity, edits and updates the virtual text according to the obtained similarity comparison result, and loads the updated virtual text. Compared with the prior art, the virtual reality motion capture method can improve the accuracy of motion capture in VR conference scenes, and can also ensure the accurate generation and effective editing of conference record documents, thereby effectively improving the application value of VR conference scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of a virtual reality motion capture method according to an embodiment of the present invention; Figure 2 is another flowchart of the virtual reality motion capture method according to an embodiment of the present invention; Figure 3 is a structural schematic diagram of a virtual reality motion capture system in an embodiment of the present invention; Figure 4 is another structural schematic diagram of the virtual reality motion capture system in an embodiment of the present invention; Figure 5 It is a diagram of the internal structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and beneficial effects of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Obviously, the embodiments described below are part of the embodiments of the present invention and are only used to illustrate the present invention, but are not used to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] In one embodiment, Figure 1 As shown, a virtual reality motion capture method is provided, comprising the following steps: S11. Collect virtual reality action feature timing data, audio timing data and image timing data of preset duration; wherein, the preset duration can be determined according to the data duration required for action recognition, virtual document record generation and virtual interactive action analysis in the actual application scenario, and is not limited here. Virtual reality action feature timing data can be collected through the existing VR motion capture equipment deployed (such as optical or inertial motion capture equipment) to collect the action data of the participants, and the timing feature extraction of the action data, including the timing data of relevant motion parameters such as the position, speed and acceleration of each action part of interest at different time points. For example, the hand movements of the participants need to be analyzed later to facilitate the generation of virtual documents and the realization of flexible virtual interactions. Here, it is necessary to obtain the relevant timing data of the hand movement characteristics for subsequent use. Audio timing data can be understood as the voice information of participants during the meeting obtained through VR recording equipment or audio collectors and other related audio collection devices deployed according to the needs of VR conference scenarios, and image timing data can be understood as the image information of participants during the meeting (including meeting record images made by participants during the meeting) collected by using VR headsets, panoramic cameras or VR digital acquisition and detection systems and other equipment. There are no specific limitations here.

[0021] This embodiment can ensure the accuracy of VR conference image generation by simultaneously collecting three types of data, namely, motion feature data, audio data, and image data, while also providing effective data support for subsequent motion command recognition, virtual document generation, and virtual interaction.

[0022] S12, extracting hand movement feature data according to the virtual reality movement feature time series data, identifying the current movement type according to the hand movement feature data, and generating a corresponding document generation instruction according to the current movement type; the document generation instruction includes a text generation instruction and a picture generation instruction; wherein the hand movement feature data can be understood as relevant feature data that facilitates reliable analysis of the hand movement behavior of the participants, and considering that the hand movement type is related to the hand movement speed, the finger holding point pressure condition and the wrist movement condition, in order to more accurately identify the hand movement type, this embodiment optimizes the hand movement feature data to include wrist movement trajectory characteristics, pen holding point pressure change characteristics and hand movement speed change characteristics; specifically, the step of extracting hand movement feature data according to the virtual reality movement feature data includes: The wrist motion features in the virtual reality action feature timing data are extracted to obtain the wrist motion timing features, and the wrist motion trajectory features are generated based on the wrist motion timing features; wherein the wrist motion features can be understood as the wrist motion data captured by using a high-precision sensor, and the corresponding wrist motion trajectory features can be understood as the features obtained by screening the wrist motion timing features, which can accurately describe the motion state and motion trajectory of the wrist in different time periods, and may include the wrist position and wrist motion speed that change with time. The specific screening method can be implemented by correlation analysis and other technologies, which will not be described in detail here.

[0023] The pen grip point pressure feature in the virtual reality action feature time series data is extracted to obtain the pen grip point pressure time series feature, and the pen grip point pressure change feature is generated according to the pen grip point pressure time series feature; wherein, the pen grip point pressure feature can be collected by using a VR stylus and VR gloves to obtain each pen grip point pressure data; the change of the collected pen grip point pressure features at different times is analyzed to obtain the corresponding pen grip point pressure change feature.

[0024] The hand motion features in the virtual reality action feature time series data are extracted to obtain the hand motion speed time series features, and the hand motion speed change features are generated according to the hand motion speed time series features; wherein the hand motion features can be understood as the hand motion speed data collected by wearing an optical data glove or an inertial data glove; and the changes of the collected hand motion features at different times are analyzed to obtain the corresponding hand motion speed change features.

[0025] After the hand motion feature data is obtained through the above method steps, it is used to effectively identify the writing or drawing action made by the hand, so as to generate accurate manuscript generation instructions according to the action type recognition result; specifically, the steps of identifying the current action type according to the hand motion feature data and generating the corresponding manuscript generation instruction according to the current action type include: According to the wrist motion trajectory characteristics, a wrist motion trajectory continuity function is constructed; wherein the wrist motion trajectory continuity function can be understood as a data model for evaluating whether the wrist motion is smooth and coherent, which can be constructed based on simulation analysis of the wrist motion trajectory characteristics, and is preferably expressed as: in, represents the continuity value of the wrist motion trajectory at time t; k represents a constant; Represents the displacement change of the wrist at time t.

[0026] According to the pen holding point pressure change characteristics, a pen holding point pressure change frequency function is constructed; wherein the pen holding point pressure change frequency function can be understood as a mathematical model constructed based on the fitting analysis of the pen holding point pressure change characteristics, which is expressed as: in, represents the frequency of the pressure change at the pen holding point at time t; A represents the amplitude; represents the i-th pressure peak moment; σ represents the standard deviation of the Gaussian distribution of pressure variation; n represents the number of pressure peaks.

[0027] According to the hand movement speed change characteristics, a hand movement speed change weighting function is constructed; wherein the hand movement speed change weighting function can be understood as a mathematical model for quantifying the slight changes in hand movement speed, which can be expressed as: in, represents the weighted value of the hand movement speed change at time t; Indicates the change in hand movement speed at time t.

[0028] According to the wrist motion trajectory continuity function, the pen grip point pressure change frequency function and the hand motion speed change function, a corresponding action recognition factor is obtained; wherein the action recognition factor can be understood as a judgment basis obtained by comprehensively analyzing the wrist motion continuity, the pen grip point pressure change and the hand motion speed change, which can be used to effectively recognize writing actions and drawing actions; specifically, the action recognition factor is expressed as: in, represents the action recognition factor at time t; , and Indicates the constant coefficient, which can be set according to actual needs; It represents the coherence value of the wrist motion trajectory at time t, reflecting the smoothness of the wrist during the motion process. A higher value means more coherent wrist movement and a more drawing-like hand movement. The growth rate of the total value obtained by integration gradually increases; It represents the frequency value of the pressure change of the pen holding point at time t. The frequency value of the pressure change of the pen holding point reflects the frequency of breakpoints in the writing or drawing process. The lower the value, the more coherent the wrist movement is, and the hand movement is more inclined to drawing. The growth rate of the total value obtained by integration gradually decreases; represents the weighted value of the hand movement speed change at time t. Since writing is usually a fast hand movement, while drawing is often a slow hand movement, the hand movement speed change is used to distinguish between writing and drawing. The lower the value, the more coherent the wrist movement is, and the hand movement is more inclined to drawing. The growth rate of the total value obtained by integration gradually decreases; therefore, although , and The total value obtained by the integral calculation is gradually increasing, but its growth rate is gradually decreasing. It can be considered that when F(t) is lower than the preset factor threshold, it means that the hand movement is a writing movement.

[0029] Determine whether the action recognition factor is greater than a preset factor threshold. If so, determine the current action type as a drawing action, and generate the drawing generation instruction. Otherwise, determine the current action type as a writing action, and generate the text generation instruction. The preset factor threshold can be determined according to actual application needs. Taking into account the characteristics of high wrist movement continuity requirements, large changes in pen grip pressure, and large changes in hand movement speed corresponding to drawing actions, it is preferred to determine the action type greater than the preset factor threshold as a drawing action, and the action type less than or equal to the preset factor threshold as a writing action. This can efficiently and accurately identify the hand action types of participants and improve the accuracy of action recognition.

[0030] This embodiment introduces motion recognition factors to comprehensively consider hand feature data such as wrist motion trajectory, pen grip pressure and hand movement speed, thereby improving the accuracy of motion recognition in terms of identifying detailed hand movements, effectively distinguishing between text and pictorial representations, and thereby providing reliable guarantees for the accurate generation of meeting minutes (virtual documents).

[0031] S13, recognize and process the image time series data, and generate a corresponding virtual document according to the obtained handwriting content and the document generation instruction; the virtual document includes virtual text and picture annotations; wherein, the handwriting content can be understood as the note information of the participants identified and extracted from the image time series data, and in practical applications, it can be extracted from the collected image time series data through OCR (Optical Character Recognition) optical character recognition technology, machine learning algorithm or recurrent neural network model and other technologies, which will not be described in detail here. After obtaining the handwriting content, a virtual document (conference record document) can be generated according to the corresponding document generation instruction, that is, if the above action type is identified as a writing action, then it is only necessary to generate virtual text and perform corresponding text type marking, and if the above action type is identified as a drawing action, it is necessary to generate a picture document and perform corresponding drawing type marking.

[0032] S14. Compare the audio timing data with the virtual text, edit and update the virtual text according to the obtained similarity comparison result, and load the updated virtual text. Among them, the similarity comparison can be understood as a similarity analysis process between the information content involved in the audio timing data and the text information in the virtual text generated by the image timing data. Based on the similarity comparison result, the accuracy of the virtual text generation can be judged, and the accuracy of the virtual text can be corrected accordingly. Specifically, the steps of comparing the audio timing data with the virtual text and editing and updating the virtual text according to the obtained similarity comparison result include: Distinguish the timbres of the audio timing data to obtain several audio feature data. The specific steps include: performing noise reduction processing on the collected audio timing data to remove background noise and other interference signals, and dividing the continuous audio signal after noise reduction into short frame segments for subsequent feature extraction; performing a fast Fourier transform on each audio frame, and then obtaining the MFCC feature vector of the audio through a Mel filter bank and a discrete cosine transform. Classify the extracted MFCC feature vectors through a clustering algorithm, and divide the MFCC feature vectors into human voices from different speakers and environmental sounds according to the timbre similarity in the vectors, and obtain several audio feature data storing different timbre information.

[0033] Extract preset keywords from the virtual text to obtain several text feature data. The specific steps include: performing word segmentation on the virtual text to decompose it into individual words, and loading a preset vocabulary library and text library, and removing common meaningless words in the virtual text, such as "de", "shi", "zai", etc. Use the TF-IDF algorithm to judge the importance of each word, and identify the words that appear frequently in the virtual text and rarely appear in common texts as keywords, that is, according to a preset threshold, select the words with higher TF-IDF values as keywords, and convert the extracted keywords into vector form, with each keyword corresponding to a dimension, to form a high-dimensional vector to represent the content of the entire text, and obtain several text feature data; Perform text similarity analysis according to all the audio feature data, all the text feature data and a preset text similarity calculation model to obtain the corresponding similarity comparison result. Among them, the preset text similarity calculation model is expressed as: Among them, represents the text similarity; J represents the number of audio feature data; M represents the number of text feature data; 、 and Respectively represent the frequency change rate, duration and length of the extracted text content of the j-th audio feature data; , and Represents the weight coefficient of audio feature data, which can be set according to actual needs; and Respectively represent the length and frequency of the text word of the mth text feature data; δ, θ and ε represent the weight coefficients of the text feature data, which can be set according to actual needs and considered by using integral calculation. , and The total value obtained by integration indicates that the text parameter can fuse the audio feature data and the text feature data to obtain multi-dimensional feature fusion data, which can more comprehensively capture the similarity between text and audio. The higher the SIM value, the higher the similarity between audio and text.

[0034] In practical applications, the process of obtaining similarity comparison results can be understood as follows: first, the frequency change of each audio feature data can be calculated by analyzing the spectrum of continuous frames. In the preprocessing stage, the audio is divided into multiple frames, and the length of each frame represents the feature duration. Then, the audio feature is converted into text using automatic speech recognition technology (such as ASR model). The text length output by the automatic speech recognition technology is the length of the text content that can be converted by the audio feature, and finally the frequency change rate, duration and extracted text content length corresponding to each audio feature data are obtained; at the same time, each text feature data is analyzed and processed in terms of text length and word frequency, and the text length is obtained by counting the total number of text feature data after word segmentation, and then the text after word segmentation is traversed to count the number of occurrences of each word, and normalization is performed to obtain the occurrence frequency information corresponding to each text feature data; then, the analysis results of all audio feature data and all text feature data are input into the preset text similarity calculation model to obtain the required text similarity comparison results.

[0035] If the similarity comparison result is less than a preset similarity threshold, the audio time series data is converted into corresponding audio text data, and the virtual text is updated and edited according to the audio text data; wherein, the preset similarity threshold can be determined according to actual application requirements. When the similarity comparison result is lower than the preset similarity threshold, it is considered that the virtual text contains certain error information, and it needs to be corrected in combination with the keywords and semantics in the audio data.

[0036] The audio text data in this embodiment can be understood as the text data obtained by converting the audio time series data through the relevant text conversion technology. Considering the authenticity and reliability of the audio time series data, the audio text data corresponding to the audio time series data is used as the reference text content to update and calibrate the virtual text. After the audio text data is segmented, the keywords of the audio text data are extracted by the above method, and the corresponding keywords and keyword word order information are checked by traversing the reference text content to determine whether there are errors in the reference text. When there are errors, the audio text data is introduced into the reference text content for coverage calibration, which can not only reduce the error rate of virtual text generation, but also actively correct the virtual text based on the keywords and semantics in the audio data, effectively improving the flexibility of virtual text editing, and improving the readability and accuracy of the virtual text. It should be noted that in actual applications, in order to verify the accuracy of the virtual text after editing and updating, the edited virtual text data and the audio time series data can be compared and analyzed for text similarity based on the above method steps, and whether a secondary editing is required is determined based on the comparison results.

[0037] The embodiment of the present invention provides a technical solution for collecting virtual reality action feature time series data, audio time series data and image time series data of preset duration, identifying the current action type according to the hand action feature data extracted from the virtual reality action feature time series data, generating a document generation instruction corresponding to the text generation instruction and the picture generation instruction according to the current action type, then obtaining the handwriting content by processing the image time series data recognition and generating a virtual document corresponding to the virtual text and the picture annotation in combination with the document generation instruction, editing and updating the virtual text based on the similarity comparison result between the audio time series data and the virtual text, and loading the updated virtual text. This solution can not only improve the accuracy of action capture in VR conference scenarios, but also avoid the errors and limitations of generating virtual text based on single data, ensure the accurate generation and effective editing of conference record documents, significantly improve the accuracy and readability of virtual document content, ensure the generation quality of virtual documents, and better meet the application requirements of high-end VR conference scenarios such as conference records and remote collaboration when the line of sight is not following.

[0038] In addition, considering that there will be efficient transmission and manually specified modification requirements of conference record manuscripts in actual applications, in order to ensure the transmission and modification efficiency of virtual manuscripts in virtual interaction scenarios, this embodiment preferably also uses standard directive action feature data such as preset manuscript transmission actions and modification actions to facilitate real-time perception of virtual interaction requirements of manuscripts; specifically, Figure 2 As shown, the method also includes: S15, comparing the virtual reality action feature time series data with the preset indication action standard feature data, and generating corresponding action instructions according to the matched preset indication action standard feature data, so as to perform virtual interaction according to the action instructions; the action instructions include a document transfer instruction and a document modification instruction; Among them, the preset indication action standard feature data can be understood as the relevant feature data extracted according to the preset standard hand movements for executing manuscript transmission and manuscript modification, which can be pre-stored in the preset action matching database. In practical applications, when one or some action features in the collected virtual reality action feature timing data match the standard feature data of the Chinese manuscript transmission action or the standard feature data of the manuscript modification action in the preset action matching database, it can be considered that the participants need to execute the manuscript transmission instruction or the manuscript modification instruction, and then generate action instructions that match the indication action data in the virtual reality action feature timing data, so as to improve the timeliness of modification and transmission efficiency of virtual manuscripts in VR conference scenes.

[0039] The embodiment of the present invention collects virtual reality action feature time series data, audio time series data and image time series data of preset duration, identifies the current action type according to the hand action feature data extracted from the virtual reality action feature time series data, generates a document generation instruction including a text generation instruction and a picture generation instruction according to the current action type, then processes the image time series data to obtain the handwriting content and generates a virtual document including virtual text and picture annotations in combination with the document generation instruction, edits and updates the virtual text based on the similarity comparison result between the audio time series data and the virtual text, loads the updated virtual text, and compares the virtual reality action feature time series data with the preset indication action standard feature data to generate a document transfer instruction or a document modification instruction and other action instruction indicating virtual interaction solutions, thereby effectively solving the problem that the existing VR conference scene cannot guarantee the accuracy of motion capture and cannot realize the effective organization and synchronization of conference record documents without line of sight following. The application defects of step transmission can not only improve the accuracy of motion capture in VR conference scenarios from the aspect of identifying detailed hand movements by comprehensively considering hand feature data such as wrist movement trajectory, pen holding point pressure and hand movement speed, but also generate virtual text images in VR by combining action, audio and image data and correcting the virtual text according to keywords and semantics in audio data. This technology avoids the errors and limitations of virtual text generation based on single data, ensures the accurate generation and effective editing of meeting minutes, significantly improves the accuracy and readability of virtual document content, ensures the generation quality of virtual documents, and can perceive virtual document interactive action instructions in real time to realize the transmission and secondary editing of virtual documents, improve the transmission efficiency and modification flexibility of virtual documents, so as to better meet the application needs of high-end VR conference scenarios such as meeting records and remote collaboration without line of sight, thereby enhancing the application value of VR conference scenarios.

[0040] It should be noted that although the steps in the above flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders.

[0041] In one embodiment, Figure 3 As shown, a virtual reality motion capture system is provided, the system comprising: Data collection module 1, used to collect virtual reality action feature time series data, audio time series data and image time series data of preset duration; The action recognition module 2 is used to extract hand action feature data according to the virtual reality action feature time series data, identify the current action type according to the hand action feature data, and generate a corresponding document generation instruction according to the current action type; the document generation instruction includes a text generation instruction and a picture generation instruction; The document generation module 3 is used to perform recognition processing on the image time series data, and generate a corresponding virtual document according to the obtained handwriting content and the document generation instruction; the virtual document includes virtual text and picture annotations; The manuscript updating module 4 is used to perform a similarity comparison between the audio time sequence data and the virtual text, edit and update the virtual text according to the similarity comparison result, and load the updated virtual text.

[0042] In one embodiment, Figure 4 As shown, a virtual reality motion capture system is provided, and the system also includes: The interactive action recognition module 5 is used to compare the virtual reality action feature timing data with the preset indication action standard feature data, and generate corresponding action instructions according to the matching preset indication action standard feature data, so as to perform virtual interaction according to the action instructions; the action instructions include document transfer instructions and document modification instructions.

[0043] It should be noted that in order to ensure the smoothness of virtual reality interaction in VR conference scenes and improve the user experience as much as possible, this embodiment preferably sets a 5G communication module, and each module in the above system uses the 5G communication module for data interaction and transmission to effectively reduce the data transmission delay in the conference scene.

[0044] For the specific definition of the virtual reality motion capture system, please refer to the definition of the virtual reality motion capture method above, and the corresponding technical effects can also be obtained equivalently, which will not be repeated here. Each module in the above virtual reality motion capture system can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0045] Figure 5 FIG. 1 shows an internal structure diagram of a computer device in an embodiment, and the computer device may specifically be a terminal or a server. Figure 5As shown, the computer device includes a processor, a memory, a network interface, a display, a camera and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a virtual reality motion capture method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0046] It can be understood by those skilled in the art that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computing device may include more or less components than those shown in the figure, or combine certain components, or have the same component arrangement.

[0047] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.

[0048] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0049] In summary, the embodiments of the present invention provide a virtual reality motion capture method and system, wherein the virtual reality motion capture method realizes the collection of virtual reality motion feature time series data, audio time series data and image time series data of a preset time length, identifies the current motion type according to the hand motion feature data extracted from the virtual reality motion feature time series data, generates a document generation instruction corresponding to the text generation instruction and the picture generation instruction according to the current motion type, then processes the image time series data to obtain the handwriting content and generates a virtual document corresponding to the virtual text and the picture annotation in combination with the document generation instruction, edits and updates the virtual text based on the similarity comparison result between the audio time series data and the virtual text, loads the updated virtual text, and compares the virtual reality motion feature time series data with the preset indication motion standard feature data to generate a document transfer instruction or a document modification instruction and other motion instruction indicating the technical solution of virtual interaction, and the method does not Only by comprehensively considering hand feature data such as wrist movement trajectory, pen grip pressure and hand movement speed, can the accuracy of motion capture in VR conference scenarios be improved from the perspective of identifying detailed hand movements. Moreover, the technology can generate virtual text images in VR by combining motion, audio and image data and correct the virtual text according to keywords and semantics in the audio data, thereby avoiding the errors and limitations of generating virtual text based on single data, ensuring the accurate generation and effective editing of meeting minutes, significantly improving the accuracy and readability of virtual document content, ensuring the generation quality of virtual documents, and sensing virtual document interactive action instructions in real time to realize the transmission and secondary editing of virtual documents, thereby improving the transmission efficiency and modification flexibility of virtual documents, so as to better meet the application needs of high-end VR conference scenarios such as meeting minutes and remote collaboration without line of sight, thereby enhancing the application value of VR conference scenarios.

[0050] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0051] The above-mentioned embodiments only express several preferred implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principle of the present invention, and these improvements and substitutions should also be regarded as the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the protection scope of the claims.

Claims

1. A virtual reality motion capture method, characterized in that: The method comprises the following steps: Collecting virtual reality action feature time series data, audio time series data, and image time series data of preset duration; Extracting hand motion feature data according to the virtual reality motion feature time series data, identifying a current motion type according to the hand motion feature data, and generating a corresponding document generation instruction according to the current motion type; the document generation instruction includes a text generation instruction and a picture generation instruction; Recognize and process the image time series data, and generate a corresponding virtual document according to the obtained handwriting content and the document generation instruction; the virtual document includes virtual text and picture annotations; The audio time sequence data is compared with the virtual text for similarity, the virtual text is edited and updated according to the obtained similarity comparison result, and the updated virtual text is loaded.

2. The virtual reality motion capture method according to claim 1, characterized in that: The hand motion feature data includes wrist motion trajectory features, pen grip point pressure change features and hand motion speed change features; The step of extracting hand motion feature data according to the virtual reality motion feature data comprises: Extracting wrist motion features from the virtual reality action feature time series data to obtain wrist motion time series features, and generating the wrist motion trajectory features based on the wrist motion time series features; Extracting the pen holding point pressure feature in the virtual reality action feature time series data to obtain the pen holding point pressure time series feature, and generating the pen holding point pressure change feature according to the pen holding point pressure time series feature; The hand movement features in the virtual reality action feature time series data are extracted to obtain the hand movement speed time series features, and the hand movement speed change features are generated according to the hand movement speed time series features.

3. The virtual reality motion capture method according to claim 2, characterized in that: The steps of identifying the current action type according to the hand action feature data, and generating a corresponding document generation instruction according to the current action type include: Constructing a wrist motion trajectory continuity function according to the wrist motion trajectory characteristics; According to the pen holding point pressure change characteristics, construct a pen holding point pressure change frequency function; Constructing a hand movement speed change weighting function according to the hand movement speed change characteristics; Obtaining a corresponding action recognition factor according to the wrist motion trajectory continuity function, the pen grip point pressure change frequency function and the hand motion speed change function; Determine whether the action recognition factor is greater than a preset factor threshold. If so, determine the current action type as a drawing action and generate the drawing generation instruction. Otherwise, determine the current action type as a writing action and generate the text generation instruction.

4. The virtual reality motion capture method according to claim 3, characterized in that: The action recognition factor is expressed as: In the formula, in, represents the action recognition factor at time t; represents the continuity value of the wrist motion trajectory at time t, k represents a constant, represents the displacement change of the wrist at time t; represents the frequency of the pressure change at the pen holding point at time t, A represents the amplitude, represents the i-th pressure peak moment, σ represents the standard deviation of the Gaussian distribution of pressure change, and n represents the number of pressure peaks; represents the weighted value of the hand movement speed change at time t, Indicates the change in hand movement speed at time t; , and represents a constant coefficient.

5. The virtual reality motion capture method according to claim 1, wherein: The step of performing a similarity comparison between the audio time sequence data and the virtual text and editing and updating the virtual text according to the obtained similarity comparison result comprises: Differentiate the audio time sequence data by timbre to obtain a plurality of audio feature data; Extracting preset keywords from the virtual text to obtain a number of text feature data; Perform text similarity analysis based on all audio feature data, all text feature data and a preset text similarity calculation model to obtain corresponding similarity comparison results; If the similarity comparison result is less than a preset similarity threshold, the audio time sequence data is converted into corresponding audio text data, and the virtual text is updated and edited according to the audio text data.

6. The virtual reality motion capture method according to claim 5, characterized in that: The preset text similarity calculation model is expressed as: in, represents text similarity; J represents the number of audio feature data; M represents the number of text feature data; , and Respectively represent the frequency change rate, duration and length of the extracted text content of the j-th audio feature data; , and Represents the weight coefficient of audio feature data; and They represent the text word length and occurrence frequency of the mth text feature data respectively; δ, θ and ε represent the weight coefficients of the text feature data.

7. The virtual reality motion capture method according to claim 1, wherein: The method further comprises: The virtual reality action feature timing data is compared with the preset indication action standard feature data, and corresponding action instructions are generated according to the matched preset indication action standard feature data, so as to perform virtual interaction according to the action instructions; the action instructions include document transfer instructions and document modification instructions.

8. A virtual reality motion capture system, characterized in that: The system comprises: A data collection module, used to collect virtual reality action feature time series data, audio time series data and image time series data of a preset duration; an action recognition module, configured to extract hand action feature data according to the virtual reality action feature time series data, identify a current action type according to the hand action feature data, and generate a corresponding document generation instruction according to the current action type; the document generation instruction includes a text generation instruction and a picture generation instruction; A document generation module, used to perform recognition processing on the image time series data, and generate a corresponding virtual document according to the obtained handwriting content and the document generation instruction; the virtual document includes virtual text and picture annotations; The manuscript updating module is used to perform a similarity comparison between the audio timing data and the virtual text, edit and update the virtual text according to the similarity comparison result, and load the updated virtual text.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images