Action recognition method, system, device and storage medium based on VR technology

Abstract

The present application relates to the field of virtual reality interaction technology, and in particular to a method, system, device and storage medium for action recognition based on VR technology, wherein the method comprises obtaining a user video to be recognized and dividing it into several continuous video frames, collecting user head angle data corresponding to the video frames; determining the center point of the video frame, judging one by one based on the center point whether the current video frames other than the first frame have an overall offset relative to the first frame, if it is judged that an overall offset has occurred, introducing the user head angle data corresponding to the current video frame to adjust it, inputting the judged several continuous video frames into the action recognition model, and the action recognition model outputs the actions of several continuous video frames; using an action splicing algorithm to splice the actions of the recognized several continuous video frames to generate the final recognized user action. The present application can solve the problem of reduced recognition accuracy of traditional action recognition technology when the user and camera perspectives are offset by adjusting the image perspective.

Description

Technical Field

[0001] The present application relates to the field of virtual reality interaction technology, and in particular to a motion recognition method, system, device and storage medium based on VR technology. Background Art

[0002] Motion recognition is one of the important research directions in the field of virtual reality (VR) technology and computer vision, and is widely used in education and training, medical rehabilitation, entertainment interaction, etc. Especially in the VR environment, the analysis of user motion behavior is crucial to improving the interactive experience and the accuracy of system feedback.

[0003] Existing motion recognition technology usually relies on cameras combined with computer vision algorithms. The specific implementation method includes: first, using a camera to capture the user's motion video, decomposing the video into several frames, and then extracting features from each frame. Through these features, computer vision algorithms can perform motion recognition and classification. In this process, the system mainly relies on the visual information in the video frame to identify the user's motion, and analyzes and judges it through the algorithm model.

[0004] However, the existing technology has some problems in processing user actions. In particular, when the user's head or body is offset relative to the camera, the image captured in the video may not accurately reflect the user's actual action. Specifically, when the user's head or body deviates from the camera's direct view direction, the image captured by the camera will be offset in angle, which causes the motion trajectory and posture changes in the image to be inconsistent with the actual situation. This offset makes the action recognition based on these image features prone to errors, thereby affecting the accuracy of action recognition. Summary of the invention

[0005] The present application provides a method, system, device and storage medium for action recognition based on VR technology, which can solve the problem of reduced recognition accuracy of traditional action recognition technology when the user and camera perspectives are offset by adjusting the image perspective. The present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a method for motion recognition based on VR technology, the method comprising:

[0007] Obtain a user video to be identified, decompose the user video into several consecutive video frames, and collect user head angle data corresponding to each video frame;

[0008] Determine the center points of several consecutive video frames, and determine whether the current video frames other than the first frame have an overall offset relative to the first frame based on the center points. If it is determined that an overall offset has occurred, introduce the user head angle data corresponding to the current video frame to adjust it. If it is determined that no overall offset has occurred, no adjustment is performed.

[0009] Inputting the determined consecutive video frames into a pre-built action recognition model, wherein the action recognition model outputs actions of the consecutive video frames;

[0010] The action stitching algorithm based on dynamic time warping is used to stitch the actions of several consecutive video frames to generate the final recognized user action.

[0011] In a specific implementation scheme, obtaining a user video to be identified, decomposing the user video into a plurality of continuous video frames, and collecting user head angle data corresponding to each video frame includes:

[0012] The user's head angle data corresponding to each video frame is collected by wearing a VR helmet on the user's head. The user's head angle data includes roll angle data, yaw angle data and pitch angle data, wherein the roll angle is used to represent the rotation of the user's head around the front-to-back axis, the yaw angle is used to represent the rotation of the user's head around the vertical axis, and the pitch angle is used to represent the rotation of the user's head around the left-right axis.

[0013] In a specific possible implementation manner, determining the center points of a plurality of consecutive video frames, and judging one by one based on the center points whether the current video frames other than the first frame have an overall offset relative to the first frame comprises:

[0014] Use edge detection algorithm to extract the outline of the user's current frame, identify key parts of the human body based on the detected outline and use geometric analysis method to calculate the coordinates of the key parts, determine the center point of the human body and record the coordinates of the center point;

[0015] Starting from the second frame, after determining the center point position and coordinates of the current frame, use the following formula to calculate the overall offset of the current frame:

[0016] in, is the offset distance of the center point of the current frame relative to the center point of the previous frame, , is the center point coordinate of the current frame, , is the center point coordinate of the first frame;

[0017] The calculated offset distance is compared with a preset offset threshold. If the offset exceeds the threshold, it is considered that the current video frame is offset as a whole relative to the first video frame. Otherwise, it is considered that the current video frame is not offset as a whole relative to the first video frame.

[0018] In a specific possible implementation scheme, if it is determined that an overall offset occurs, introducing the user head angle data corresponding to the current video frame to adjust it includes:

[0019] The difference between the user head angle data of the current video frame and the first frame is calculated as follows:

[0020] ;

[0021] ;

[0022] ;

[0023] in, is the roll angle data difference between the current video frame and the first frame, is the yaw angle data difference between the current video frame and the first frame, is the pitch angle data difference between the current video frame and the first frame;

[0024] Construct the roll angle rotation matrix separately , yaw angle rotation matrix , pitch angle rotation matrix , based on the roll angle rotation matrix , yaw angle rotation matrix , pitch angle rotation matrix Combined rotation matrix as follows:

[0025] Construct the rotation matrix Afterwards, use the rotation matrix Rotate a point in 3D space using the perspective projection matrix Map the rotated 3D points onto the 2D image plane to get the new coordinates in the image , use the nearest neighbor interpolation method to determine the The pixel values ​​obtained by interpolation calculation are filled into the corresponding positions of the new image to complete the image adjustment.

[0026] In a specific embodiment, the use of a rotation matrix Rotate a point in 3D space using the perspective projection matrix Map the rotated 3D points onto the 2D image plane to get the new coordinates in the image include:

[0027] For each point in the image , through the rotation matrix Calculate the rotated coordinates :

[0028] The rotated coordinates are then passed through the perspective projection matrix Project it onto the two-dimensional image plane and get the new coordinates in the image :

[0029] .

[0030] In a specific implementation scheme, the inputting of the determined several consecutive video frames into a pre-built action recognition model, wherein the action recognition model outputs the actions of the several consecutive video frames includes:

[0031] Identify the current frame key points, denoted as , calculate each pair of key points The Euclidean distance between :

[0032] in, and The key points are and After calculating the Euclidean distance of each pair, we get a The distance matrix ; For the distance matrix Normalize and get the distance ratio matrix :

[0033] in, is the distance matrix The maximum Euclidean distance in ;

[0034] Use the mean square error method to calculate the distance ratio matrix of the current video frame The mean square error value of the distance ratio matrix corresponding to each action template in the action template library preset in the action recognition model:

[0035] in, and They are the current video frame and the action template. The key points and After calculating the mean square error between the current video frame and all action templates, the action template with the lowest mean square error is selected as the action recognition result of the current video frame.

[0036] In a specific implementation scheme, the step of inputting the determined consecutive video frames into a pre-built action recognition model, wherein the action recognition model outputs the consecutive video frames, further comprises:

[0037] If the calculated mean square error value is higher than the preset threshold, it is determined that there is no action matching the current video frame in the action template library, and several consecutive video frames are continuously analyzed backward, and the mean square error values ​​of several subsequent consecutive video frames are calculated, until it is detected that the mean square error value of the next frame is less than or equal to the preset threshold, and the actions of all consecutive video frames that previously exceeded the threshold are determined to be new actions, and the actions of all consecutive video frames that previously exceeded the threshold are identified, and the identified actions and the corresponding distance ratio matrix calculated previously are included in the action template library to update the action template library.

[0038] In the second aspect, the present application provides a motion recognition system based on VR technology, which adopts the following technical solution:

[0039] A motion recognition system based on VR technology, comprising:

[0040] A data acquisition module is used to obtain a user video to be identified, decompose the user video into a number of continuous video frames, and collect user head angle data corresponding to each video frame;

[0041] A video frame adjustment module, used to determine the center point of several consecutive video frames, and determine one by one based on the center point whether the current video frames other than the first frame have an overall offset relative to the first frame, if it is determined that an overall offset has occurred, introduce the user head angle data corresponding to the current video frame to adjust it, and if it is determined that no overall offset has occurred, no adjustment is performed;

[0042] An action recognition module, used to input the determined consecutive video frames into a pre-built action recognition model, and the action recognition model outputs the actions of the consecutive video frames;

[0043] The action stitching module is used to stitch the actions of several consecutive video frames identified using an action stitching algorithm based on dynamic time warping to generate the final identified user action.

[0044] In a third aspect, the present application provides an electronic device, comprising a processor and a memory; the memory stores a program, and the program is loaded and executed by the processor to implement a motion recognition method based on VR technology as described in the first aspect.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium, wherein a program is stored in the storage medium, and when the program is executed by a processor, it is used to implement a motion recognition method based on VR technology as described in the first aspect.

[0046] In summary, the beneficial effects of this application include at least:

[0047] 1) By combining the user's head angle data to adjust the perspective, the video frames are analyzed at a relatively consistent perspective. This method can reduce the perspective difference caused by the user's head deviation, thereby improving the accuracy of action recognition and ensuring that the action recognition model can accurately capture the user's real actions.

[0048] 2) By collecting the angle data of the head and adjusting the current frame image based on this data, the image can be corrected to a similar perspective as the first frame. This adjustment ensures the consistency of the image between frames, provides a more standardized perspective for subsequent action recognition, and thus significantly improves the accuracy of action recognition. This method effectively solves the problem of image deviation caused by posture changes in traditional technologies, and ensures the stability and accuracy of the action recognition system under various operating conditions.

[0049] 3) By analyzing the mean square error of consecutive video frames, the system can more accurately distinguish stable new actions from occasional abnormal or erroneous actions. The error of a single frame may lead to misjudgment due to instantaneous offset or occasional action, so the system uses data from consecutive frames to verify the stability of the action and ensure the accuracy of the recognition results.

[0050] First, the user video is obtained and decomposed into several continuous video frames, and the head angle data corresponding to each video frame is collected; then, according to the center point of each frame, it is determined whether an overall offset has occurred, and the offset video frame is adjusted based on the head angle data; finally, the adjusted continuous video frames are input into the pre-built action recognition model for action recognition, and the dynamic time warping algorithm is used to splice the recognized actions to generate the final user action. This method solves the problem of reduced recognition accuracy of traditional action recognition technology when the user and camera perspectives are offset by adjusting the image perspective, and improves the accuracy of action recognition in practical applications.

[0051] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of the action recognition method based on VR technology in an embodiment of the present application.

[0053] Figure 2 It is a schematic diagram of the overall process of the action recognition method based on VR technology in an embodiment of the present application.

[0054] Figure 3 It is a structural block diagram of the action recognition system based on VR technology in an embodiment of the present application.

[0055] Figure 4 It is a block diagram of an electronic device for motion recognition based on VR technology in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The specific implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

[0057] Optionally, the present application takes the action recognition method based on VR technology provided in each embodiment as an example used in an electronic device, where the electronic device is a terminal or a server. The terminal may be a computer, a tablet computer, etc. This embodiment does not limit the type of electronic device.

[0058] Reference Figure 1 , is a flow chart of a method for motion recognition based on VR technology provided by an embodiment of the present application, the method comprising at least the following steps:

[0059] Step S101: Obtain a user video to be identified, decompose the user video into several continuous video frames, and collect user head angle data corresponding to each video frame.

[0060] Step S102, determine the center points of several consecutive video frames, and determine one by one based on the center points whether the current video frames other than the first frame have an overall offset relative to the first frame. If it is determined that an overall offset has occurred, introduce the user head angle data corresponding to the current video frame to adjust it. If it is determined that no overall offset has occurred, no adjustment is made.

[0061] Step S103: input the determined consecutive video frames into a pre-built action recognition model, and the action recognition model outputs actions of the consecutive video frames.

[0062] Step S104: Use a dynamic time warping-based action stitching algorithm to stitch the actions of several consecutive video frames to generate a final recognized user action.

[0063] In step S101, firstly, the user's action video is captured in real time by a camera as the user video to be identified. In implementation, the camera is usually set in front of the user and the coverage of the camera can cover the user's whole body and surroundings. Then the user video is decomposed into several continuous video frames, and the video frame rate is usually set to 30-60 frames per second to ensure the smoothness of the action and the capture of details. Each video frame represents the action state of the user at a certain moment. Finally, the user's head angle data corresponding to each video frame is collected by a VR helmet worn on the user's head. The angle data includes roll angle data, yaw angle data and pitch angle data, wherein the roll angle is used to represent the rotation of the user's head around the front-back axis, the yaw angle is used to represent the rotation of the user's head around the vertical axis, and the pitch angle is used to represent the rotation of the user's head around the left-right axis. In implementation, ensure that the timestamp of each video frame is consistent with the corresponding head angle data timestamp, so as to accurately pair the angle data with the video frame.

[0064] Optionally, the present application collects user head angle data in real time by building a gyroscope into the VR helmet. The present application does not limit the method of collecting user head angle data.

[0065] In addition, preferably, the refresh rate (sampling frequency) of the gyroscope is usually high. Common gyroscopes can reach a sampling frequency of 100 Hz or even higher, which means that head angle data can be collected 100 times per second. However, the frame rate of the camera is usually 30-60 frames per second, which results in the number of angle data collected being much larger than the number of video frames. Therefore, if multiple angle data are collected in a certain video frame, the angle data in this time period is averaged or weighted averaged to obtain an angle value representing the video frame to ensure the stability of the angle data.

[0066] In step S102, the center points of several consecutive video frames are determined, and based on the center points, it is determined one by one whether the current video frames other than the first frame have an overall offset relative to the first video frame. If it is determined that an overall offset has occurred, the user head angle data corresponding to the current video frame is introduced to adjust it. If it is determined that no overall offset has occurred, no processing is performed.

[0067] Specifically, first use an edge detection algorithm to extract the outline of the user's current frame and separate the background. Then, based on the detected outline, identify the key parts of the human body, such as shoulders, waist, knees, etc., and use a geometric analysis method to calculate the coordinates of the key parts. Then, determine the center point of the human body and record the coordinates of the center point. In this application, the center point between the shoulder and the waist is defined as the center point of the human body, which serves as the representative point of the overall posture.

[0068] Optionally, the present application uses the Canny edge detection algorithm in the prior art to extract the outline of the user in the current video frame, and the present application does not limit the specific type of the edge detection algorithm.

[0069] For each frame, when processing the first frame, only the user's initial posture is recorded and the initial center point position and coordinates are determined. This center point will be used as a reference for subsequent frames. Starting from the second frame, after determining the center point position and coordinates of the current frame, the overall offset of the current frame is calculated using the following formula:

[0070] in, is the offset distance of the center point of the current frame relative to the center point of the previous frame, , is the center point coordinate of the current frame, , is the coordinate of the center point of the first frame. Finally, the calculated offset distance is compared with the preset offset threshold. If the offset exceeds the threshold, it is considered that the current video frame is offset relative to the first video frame as a whole, otherwise it is considered that the current video frame is not offset relative to the first video frame as a whole.

[0071] Then, the current frame image except the first frame is adjusted according to the result of whether there is an overall offset. If it is judged that there is an overall offset, the user's head angle data corresponding to the current frame is introduced to adjust it. If it is judged that there is no overall offset, no adjustment is required. Based on the judgment of whether the overall offset occurs, the current frame image is adjusted to present the most suitable perspective in the action recognition model. This is because when the body as a whole is offset, the change in the head angle can help adjust the image to a more standard perspective, thereby improving the accuracy of subsequent action recognition.

[0072] Specifically, when the user head angle data corresponding to the current frame needs to be introduced to adjust it, the difference between the user head angle data of the current video frame and the first frame is first calculated as follows:

[0073] ;

[0074] ;

[0075] ;

[0076] in, is the roll angle data difference between the current video frame and the first frame, is the yaw angle data difference between the current video frame and the first frame, It is the pitch angle data difference between the current video frame and the first frame.

[0077] A three-dimensional rotation matrix is ​​then constructed, applying the above three-dimensional angle data to the three-dimensional space to represent the rotation of roll, yaw and pitch angles respectively.

[0078] Roll angle rotation matrix as follows:

[0079] Yaw rotation matrix as follows:

[0080] Pitch rotation matrix as follows:

[0081] Then the matrix is ​​rotated based on the roll angle , yaw angle rotation matrix , pitch angle rotation matrix Combined rotation matrix as follows:

[0082] Construct the rotation matrix Afterwards, use the rotation matrix , rotate the point in 3D space, and then use the perspective projection matrix The rotated 3D points are mapped to the 2D image plane. The specific process is as follows: First, for each point in the image , through the rotation matrix Calculate the rotated coordinates :

[0083] The rotated coordinates are then passed through the perspective projection matrix Project it onto the two-dimensional image plane and get the new coordinates in the image :

[0084] Among them, after applying the rotation matrix, the new position of the object in three-dimensional space is obtained , followed by the perspective projection matrix The point in three-dimensional space Convert to points on a two-dimensional image , the perspective projection matrix According to the depth of the point in three-dimensional space Adjust its size and position in the image - the farther away the object is from the viewpoint, the smaller it appears. This is the perspective effect. According to the depth Adjustment and , and finally get the new coordinates on the image Specifically, perspective projection is actually scaling the three-dimensional coordinates to a two-dimensional plane by dividing by To achieve:

[0085] ;

[0086] ;

[0087] It should be noted that the perspective projection matrix It is an existing standard computer graphics tool for projecting points in three-dimensional space onto a two-dimensional image plane. In the embodiment of the present application, the glm::perspective function of OpenGL is selected. This perspective projection matrix makes the three-dimensional points in the image conform to the perspective effect, so that the user's perspective can be accurately simulated when adjusting the image. By calculating the head angle difference and applying the three-dimensional rotation matrix, the image of the current frame can be adjusted to the perspective that is most suitable for recognizing the action. This adjustment helps the system maintain a consistent perspective, so that the action recognition of subsequent frames can be better connected with the initial action, improving the coherence of action recognition.

[0088] In addition, in the above formula is a homogeneous coordinate form, representing a three-dimensional point Coordinates in four-dimensional space, perspective projection matrix The rotated 3D points are mapped onto the 2D image plane to obtain the final 2D image coordinates. The last component in the homogeneous coordinates is 1 in this application to ensure the correctness of the perspective transformation.

[0089] The new coordinates of each pixel in the video frame at the adjusted viewing angle are calculated using the previous method. ,because may be non-integer coordinates, so the nearest neighbor interpolation method is used to determine the coordinates in the new image. The pixel value of The value of is taken directly from the closest original image. That is, find The closest original pixel coordinate and assign its pixel value to the new image Position, and finally fill the pixel value obtained by interpolation calculation into the corresponding position of the new image to complete the image adjustment.

[0090] In step S102, by detecting whether the video frame has an overall offset relative to the initial frame, and introducing the user's head angle data for adjustment, each frame of the image can be adjusted to a position consistent with the initial frame's perspective. This adjustment ensures that even when the user moves or turns his head, the action in the image can still accurately reflect the user's real action. Adjusting the image of the current frame to a perspective consistent with the initial frame helps ensure that the action of the subsequent frame can be naturally connected with the initial action. This perspective consistency makes the actions between different frames in the action sequence more coherent, avoiding the problem of action integration caused by perspective differences, thereby improving the effect of the overall action recognition system. In addition, the system allows a certain amount of deviation, which means that the image will not be adjusted due to every tiny offset, which can avoid misjudgment caused by overly strict offset judgment. This design makes the system more robust, able to handle small offsets of the user in natural movements, and reduces unnecessary adjustments caused by slight offsets.

[0091] It should be noted that in actual operation, when the overall offset occurs, the angle of the head will inevitably be different from the initial state. This is because in actual operation, the user's body may move significantly, and the head will usually adjust accordingly. Therefore, using the angle data of the head to adjust the current frame image can effectively correct the image to a similar perspective as the first frame. This adjustment ensures the consistency of the image with the initial perspective, provides a more standardized perspective for subsequent action recognition, and improves the accuracy of recognition.

[0092] In step S103, the continuous video frames processed in step S102 are input into the pre-built action recognition model for action recognition. Specifically, the action recognition model first recognizes the current frame by calling the MediaPipe library provided by Google. key points, denoted as , such as the tip of the nose, shoulder, wrist, knee or toe. Then calculate each pair of key points The Euclidean distance between:

[0093] in, and The key points are and After calculating the Euclidean distance of each pair, we get a The distance matrix , the distance matrix Every element of Indicates The key points and The distance between key points, .

[0094] Then the distance matrix Normalize and get the distance ratio matrix :

[0095] in, is the distance matrix The maximum Euclidean distance in . Using the maximum Euclidean distance for normalization can eliminate the scale change caused by different distances between different frames or different users. This process makes all distance scale matrices have a unified scale, which is convenient for comparison between different data.

[0096] In implementation, a dynamically updated action template library is also embedded in the action recognition model. The action template library includes a large number of collected action templates and their corresponding distance ratio matrices. The calculation method of the distance ratio matrix corresponding to the action template is the same as the calculation method of the current video frame, so it will not be described again. In actual implementation, the distance ratio matrices corresponding to the action templates are all pre-calculated, so they will not take up too many computing resources in subsequent matching actions.

[0097] Calculate the distance ratio matrix of the current video frame Afterwards, the distance ratio matrix of the current video frame is calculated using the mean square error method The mean square error value of the distance ratio matrix corresponding to each action template in the action template library:

[0098] in, and They are the current video frame and the action template. The key points and After calculating the mean square error between the current video frame and all action templates, the action template with the lowest mean square error is selected as the action recognition result of the current video frame.

[0099] In step S103, the distance ratio matrix is ​​normalized to standardize the distances between all key points into a proportional relationship, which is not affected by the body size differences of specific individuals. This means that no matter what the user's body size is, the proportional relationship in the matrix remains consistent, so that action recognition is not interfered by individual differences. In addition, by converting the actual distance into a proportional relationship, the distance ratio matrix reduces the absolute distance involved in the calculation and reduces the complexity of the calculation. This not only improves the calculation efficiency, but also reduces the amount of data that needs to be processed during the recognition process. Since the distance ratio matrix converts all action templates and the current frame into a proportional relationship, this makes the template matching process more consistent and simple. By comparing the mean square error value in the proportional matrix, the matching action template can be found more accurately, improving the recognition accuracy.

[0100] In addition, as a preferred embodiment, in another feasible embodiment, if the calculated mean square error value is higher than the preset threshold, it is determined that there is no action matching the current video frame in the action template library. In order to ensure that the recognized action is a new action rather than an occasional abnormality, several consecutive video frames are analyzed backward, and the mean square error value of the subsequent several consecutive video frames is continued to be calculated until the mean square error value of the next frame is detected to be less than or equal to the preset threshold, indicating that the action has returned to the known mode. At this time, the action of all consecutive video frames that previously exceeded the threshold is determined to be a new action, and the MediaPipe library provided by Google is called again to identify the action of all consecutive video frames that previously exceeded the threshold. The recognized action and the corresponding distance ratio matrix calculated previously are included in the action template library to update the action template library. By analyzing continuous video frames instead of a single frame, it is possible to effectively avoid misjudging occasional abnormal or erroneous actions as new actions. For example, occasional instantaneous actions or short-term posture changes may not represent actual new actions, so multiple frames of data are required to confirm. The above preferred embodiment ensures that the recognized new action is stable and consistent, rather than occasional abnormal actions, by analyzing the mean square error of continuous video frames. This can improve the accuracy of new action recognition and the robustness of the system, while dynamically updating the action template library to keep it up to date, improving the system's adaptability and performance in practical applications.

[0101] In another feasible embodiment, in addition to the above method for calculating the mean square error value, the following formula can also be used to calculate the key point similarity between the current video frame and each action template in the action template library: :

[0102] in, and They are the current video frame and the action template. The key points and The distance ratio between key points, is the total number of key points in the current video frame and action template. Is the first The key points and After all calculations are completed, the action template with the highest similarity to the key points of the current video frame is calculated as the action recognition result of the current video frame.

[0103] In the design of the above formula, the introduction of the exponential function makes the treatment of distance differences more flexible. Smaller distance differences will lead to a significant increase in the similarity score, while larger distance differences will quickly reduce the similarity score. This nonlinear processing can more accurately capture the slight differences between the template and the video frame. Compared with the mean square error method, mean square error (MSE) is a standard method for linear error processing, while exponential similarity processes errors through a nonlinear function, making it more sensitive to smaller errors and rapidly decaying to large errors. This can more accurately capture subtle matching degrees and improve recognition accuracy. In addition, since the exponential function has a higher responsiveness to smaller distance differences, this method can better identify actions that are similar to the template but have subtle differences. Compared with MSE, this method is more sensitive to capturing details and can improve the ability to recognize fine movements.

[0104] In implementation, the action recognition model in the present application is constructed based on a convolutional neural network model, that is, the above two matching methods are used as hidden layer processing algorithms of the convolutional neural network to process the input video frames.

[0105] In step S104, the dynamic time warping (DTW) algorithm is a classic time series alignment algorithm, which is used to calculate the similarity between two time series and is particularly suitable for processing nonlinear alignment problems in time series data. Specifically, the key features of the action, such as key point coordinates, are extracted from each video frame and converted into time series data. The DTW algorithm is used to align the time series features of multiple consecutive video frames. The DTW algorithm finds the optimal alignment path by calculating the minimum distance between time series. Finally, the aligned time series are spliced ​​into a complete user action sequence to generate the final recognition result. Since step S104 applies the existing technology, it will not be described in detail.

[0106] Optionally, the present application selects the DTW algorithm in the prior art to splice the actions, and other algorithms for splicing actions may also be selected. The present application does not limit the type of algorithm for splicing actions.

[0107] In summary, combined with Figure 2First, obtain the user video to be identified and decompose it into several continuous video frames, and collect the user head angle data corresponding to each video frame; secondly, determine the center point of several continuous video frames, and judge whether the current video frame has an overall offset relative to the first frame. If an offset occurs, adjust the viewing angle based on the head angle data; then, input the adjusted continuous video frames into the pre-built action recognition model for action recognition; finally, use the dynamic time warping algorithm to splice the identified actions to generate the final user action. This application solves the problem of inaccurate action recognition caused by the angle offset between the user and the camera in traditional action recognition technology by introducing the user's head angle data to adjust the viewing angle of the video frame. Through this adjustment, it is ensured that the video frames are analyzed at the same viewing angle, thereby improving the accuracy and consistency of action recognition.

[0108] Figure 3 This is a structural block diagram of a motion recognition system based on VR technology provided by an embodiment of the present application. The device includes at least the following modules:

[0109] The data acquisition module is used to obtain the user video to be identified, decompose the user video into several continuous video frames, and collect the user head angle data corresponding to each video frame;

[0110] A video frame adjustment module is used to determine the center point of several consecutive video frames, and determine whether the current video frames other than the first frame have an overall offset relative to the first frame based on the center point. If it is determined that an overall offset has occurred, the user head angle data corresponding to the current video frame is introduced to adjust it. If it is determined that no overall offset has occurred, no adjustment is performed;

[0111] An action recognition module is used to input the determined continuous video frames into a pre-built action recognition model, and the action recognition model outputs the actions of the continuous video frames;

[0112] The action stitching module is used to stitch the actions of several consecutive video frames identified using an action stitching algorithm based on dynamic time warping to generate the final identified user action.

[0113] For relevant details, refer to the above method embodiment.

[0114] Figure 4 4 is a block diagram of an electronic device provided by an embodiment of the present application. The device at least includes a processor 401 and a memory 402.

[0115] The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0116] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction, which is used to be executed by the processor 401 to implement the action recognition method based on VR technology provided in the method embodiment of the present application.

[0117] In some embodiments, the electronic device may further optionally include: a peripheral device interface and at least one peripheral device. The processor 401, the memory 402 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0118] Of course, the electronic device may also include fewer or more components, which is not limited in this embodiment.

[0119] Optionally, the present application also provides a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the motion recognition method based on VR technology of the above method embodiment.

[0120] Optionally, the present application also provides a computer product, which includes a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the motion recognition method based on VR technology of the above method embodiment.

[0121] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A motion recognition method based on VR technology, characterized in that: The method comprises: Obtain a user video to be identified, decompose the user video into continuous video frames, and collect user head angle data corresponding to each video frame; Determine the center point of the continuous video frames, and determine whether the current video frames other than the first frame have an overall offset relative to the first frame based on the center point, if it is determined that an overall offset has occurred, introduce the user head angle data corresponding to the current video frame to adjust it, and if it is determined that no overall offset has occurred, do not adjust it; Inputting the judged continuous video frames into a pre-built action recognition model, the action recognition model outputs the actions of the continuous video frames; The action stitching algorithm based on dynamic time warping is used to stitch the actions of the identified continuous video frames to generate the final recognized user action.

2. The action recognition method based on VR technology according to claim 1, characterized in that: The acquiring of the user video to be identified, decomposing the user video into continuous video frames, and collecting user head angle data corresponding to each video frame comprises: The user's head angle data corresponding to each video frame is collected by wearing a VR helmet on the user's head. The user's head angle data includes roll angle data, yaw angle data and pitch angle data, wherein the roll angle is used to represent the rotation of the user's head around the front-to-back axis, the yaw angle is used to represent the rotation of the user's head around the vertical axis, and the pitch angle is used to represent the rotation of the user's head around the left-right axis.

3. The action recognition method based on VR technology according to claim 1, characterized in that: The determining of the center points of the continuous video frames and judging one by one based on the center points whether the current video frames other than the first frame have an overall offset relative to the first frame include: Use edge detection algorithm to extract the outline of the user's current frame, identify key parts of the human body based on the detected outline and use geometric analysis method to calculate the coordinates of the key parts, determine the center point of the human body and record the coordinates of the center point; Starting from the second frame, after determining the center point position and coordinates of the current frame, use the following formula to calculate the overall offset of the current frame: in, is the offset distance of the center point of the current frame relative to the center point of the previous frame, is the center point coordinate of the current frame, is the center point coordinate of the first frame; The calculated offset distance is compared with a preset offset threshold. If the offset exceeds the threshold, it is considered that the current video frame is offset as a whole relative to the first video frame. Otherwise, it is considered that the current video frame is not offset as a whole relative to the first video frame.

4. The action recognition method based on VR technology according to claim 2, characterized in that: If it is determined that an overall offset occurs, introducing the user head angle data corresponding to the current video frame to adjust it includes: The difference between the user head angle data of the current video frame and the first frame is calculated as follows: in, Roll angle data of the current video frame Roll angle data of the first frame The difference, The yaw angle data of the current video frame The yaw angle data of the first frame The difference, The pitch angle data of the current video frame The pitch angle data of the first frame The difference between Construct the roll angle rotation matrix separately , yaw angle rotation matrix , pitch angle rotation matrix , based on the roll angle rotation matrix , yaw angle rotation matrix , pitch angle rotation matrix Combined rotation matrix as follows: Construct the rotation matrix Afterwards, use the rotation matrix Rotate a point in 3D space using the perspective projection matrix Map the rotated 3D points onto the 2D image plane to get the new coordinates in the image , use the nearest neighbor interpolation method to determine the The pixel values ​​obtained by interpolation calculation are filled into the corresponding positions of the new image to complete the image adjustment.

5. The action recognition method based on VR technology according to claim 4 is characterized in that: The use of rotation matrix Rotate a point in 3D space using the perspective projection matrix Map the rotated 3D points onto the 2D image plane to get the new coordinates in the image include: For each point in the image , through the rotation matrix Calculate the rotated coordinates : The rotated coordinates are then passed through the perspective projection matrix Project it onto the two-dimensional image plane and get the new coordinates in the image : .

6. The action recognition method based on VR technology according to claim 1, characterized in that: The action of inputting the determined continuous video frames into a pre-built action recognition model, wherein the action recognition model outputs the continuous video frames, comprises: Identify the current frame key points, denoted as , calculate each pair of key points The Euclidean distance between : in, and The key points are and After calculating the Euclidean distance of each pair, we get a The distance matrix ; For the distance matrix Normalize and get the distance ratio matrix : in, is the distance matrix The maximum Euclidean distance in ; Use the mean square error method to calculate the distance ratio matrix of the current video frame The mean square error value of the distance ratio matrix corresponding to each action template in the action template library preset in the action recognition model: in, and They are the current video frame and the action template. The key points and After calculating the mean square error between the current video frame and all action templates, the action template with the lowest mean square error is selected as the action recognition result of the current video frame.

7. The action recognition method based on VR technology according to claim 6, characterized in that: The step of inputting the determined continuous video frames into a pre-built action recognition model, wherein the action recognition model outputs the continuous video frames, further comprises: If the calculated mean square error value is higher than the preset threshold, it is determined that there is no action matching the current video frame in the action template library, and the continuous video frames are continued to be analyzed backward, and the mean square error values ​​of subsequent continuous video frames are calculated until the mean square error value of the next frame is detected to be less than or equal to the preset threshold. The actions of all continuous video frames that previously exceeded the threshold are determined to be new actions, and the actions of all continuous video frames that previously exceeded the threshold are identified. The identified actions and the corresponding distance ratio matrix calculated previously are included in the action template library to update the action template library.

8. A motion recognition system based on VR technology, characterized in that: include: A data acquisition module is used to obtain a user video to be identified, decompose the user video into continuous video frames, and collect user head angle data corresponding to each video frame; A video frame adjustment module, used to determine the center point of continuous video frames, and determine one by one based on the center point whether the current video frames other than the first frame have an overall offset relative to the first frame, if it is determined that an overall offset has occurred, introduce the user head angle data corresponding to the current video frame to adjust it, and if it is determined that no overall offset has occurred, no adjustment is performed; An action recognition module, used to input the determined continuous video frames into a pre-built action recognition model, and the action recognition model outputs the actions of the continuous video frames; The action stitching module is used to stitch the actions of the identified continuous video frames using an action stitching algorithm based on dynamic time warping to generate the final identified user action.

9. An electronic device, characterized in that: The device includes a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement a motion recognition method based on VR technology as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and when the program is executed by the processor, it is used to implement the action recognition method based on VR technology as described in any one of claims 1 to 7.

Images