Motion capture method and apparatus

Abstract

The application provides a motion capture method and device, wherein the motion capture method comprises: acquiring a collected user operation video, and acquiring motion information corresponding to the user operation video; performing motion recognition on each target video frame in a listening time period to determine a target motion performed in the listening time period, wherein each target video frame is a video frame in the user operation video that meets a motion capture analysis condition, and the listening time period is determined based on the motion information; determining whether the target motion matches the motion information, and if so, determining that the target motion is captured, and performing a corresponding set operation according to the target motion. Through the frame extraction and intermittent detection mode, the consumption of performance by motion capture is reduced, and the specified long motion can be analyzed, which is more flexible in various business scenarios, saves the resources consumed by motion capture, and improves the efficiency of motion capture.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a motion capture method. This application also relates to a motion capture device, a computing device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of computer and internet technologies, the video game industry has grown rapidly, having a huge impact on the entertainment and even lifestyles of more and more people. Users can perform corresponding actions to control the game process.

[0003] In existing technologies, motion analysis of user actions is mostly static motion analysis, which means identifying the current posture through sensors or analyzing a single frame of static image to obtain the current action posture. There is no analysis of long motions, which requires real-time monitoring of every user posture, greatly wasting motion capture resources and reducing the efficiency of motion capture. Summary of the Invention

[0004] In view of this, embodiments of this application provide a motion capture method. This application also relates to a motion capture device, a computing device, and a computer-readable storage medium, to solve the technical problems of high resource consumption and low efficiency in existing motion capture technologies.

[0005] According to a first aspect of the embodiments of this application, a motion capture method is provided, comprising:

[0006] Acquire the collected user operation video and obtain the corresponding action information of the user operation video;

[0007] Action recognition is performed on each target video frame within the monitoring time period to determine the target action executed within the monitoring time period. Each target video frame is a video frame in the user operation video that meets the motion capture analysis conditions, and the monitoring time period is determined based on action information.

[0008] Determine if the target action matches the action information. If so, confirm that the target action has been captured and execute the corresponding set operation based on the target action.

[0009] According to a second aspect of the embodiments of this application, a motion capture device is provided, comprising:

[0010] The acquisition module is configured to acquire the collected user operation video and obtain the action information corresponding to the user operation video;

[0011] The recognition module is configured to perform action recognition on each target video frame within the monitoring time period to determine the target action performed within the monitoring time period. Each target video frame is a video frame in the user operation video that meets the motion capture analysis conditions, and the monitoring time period is determined based on action information.

[0012] The determination module is configured to determine whether the target action matches the action information. If so, it determines the target action to be executed within the listening time period.

[0013] According to a third aspect of the embodiments of this application, a computing device is provided, comprising:

[0014] Memory and processor;

[0015] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the operation steps of the motion capture method described above.

[0016] According to a fourth aspect of the present application, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the operation steps of the motion capture method described above.

[0017] The motion capture method provided in this application acquires user operation videos and obtains the corresponding motion information of the user operation videos; performs motion recognition on each target video frame within the monitoring time period through a motion capture analysis component to determine the target action executed within the monitoring time period, wherein each target video frame is a video frame in the user operation video that meets the motion capture analysis conditions, and the monitoring time period is determined based on the motion information; determines whether the target action matches the motion information, and if so, determines that the target action has been captured, and executes the corresponding set operation according to the target action.

[0018] In this scenario, video frames within the monitoring time period can be extracted from the captured user operation video. Motion analysis is performed only on target video frames that meet the motion capture analysis conditions within the monitoring time period to identify long operations performed by the user within the monitoring time period. By using frame extraction and intermittent detection, the performance consumption of motion capture is reduced. Furthermore, specified long actions can be analyzed to determine whether they meet the expected action information. This allows for more flexible application in various game business scenarios, saving resources consumed by motion capture and improving the efficiency of motion capture. Attached Figure Description

[0019] Figure 1 This is a flowchart of a motion capture method provided in an embodiment of this application;

[0020] Figure 2a This is a schematic diagram of a user-side playback interface provided in one embodiment of this application;

[0021] Figure 2b This is a schematic diagram of a user interface provided in one embodiment of this application;

[0022] Figure 3a This is a flowchart illustrating a motion capture method applied to a rhythm game scenario, provided by an embodiment of this application.

[0023] Figure 3b This is a flowchart of a process for ending motion capture monitoring according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the structure of a motion capture device provided in one embodiment of this application;

[0025] Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this application. Detailed Implementation

[0026] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0027] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a” and “the” as used in one or more embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this application refers to and includes any or all possible combinations of one or more associated listed items.

[0028] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0029] First, the terms and concepts involved in one or more embodiments of this application will be explained.

[0030] Motion capture, also known as motion capture technology, refers to the technique of recording and processing the movements of people or other objects. It is widely used in entertainment, sports, medical applications, computer vision, and robotics. In filmmaking and video game development, it typically records the movements of human actors, converts them into digital model movements, and generates two-dimensional or three-dimensional computer animation. Capturing subtle facial or finger movements is often referred to as performance capture. In many fields, motion capture is sometimes called motion tracking, but in filmmaking and game development, motion tracking usually refers to matchmoving.

[0031] Mediapipe: An open-source motion capture solution that supports common cross-platform solutions and many common AI functions, such as face detection; FaceMesh (using machine learning to create dynamic 3D facial contours, a facial geometry solution that can estimate 468 3D facial landmarks in real time, even on mobile devices): Reconstructs 3D meshes of faces from images / videos, which can be used for AR rendering; Human segmentation: Segments people from images / videos, which can be used for video conferencing; Gesture tracking: Can mark the 3D coordinates of 21 key points; Human pose estimation: Can provide the 3D coordinates of 33 key points; Hair coloring: Can detect hair and color it.

[0032] SEI: Supplemental Enhancement Information, belonging to the bitstream category, provides a method for adding extra information to the video bitstream and is one of the features of the H.264 / H.265 video compression standard.

[0033] SDK (Software Development Kit): Generally, these are collections of development tools used by software engineers to create application software for specific software packages, frameworks, hardware platforms, operating systems, etc. In a broader sense, software development tools refer to a collection of related documentation, examples, and tools that assist in developing a particular type of software. Generally, an SDK is the application development tool used by an application on a specific platform. It facilitates application creation through compilers, debuggers, software frameworks, etc. It can be as simple as some documentation providing an application programming interface (API) for a programming language, but it may also include complex hardware that can communicate with an embedded system. Common tools include utilities for debugging and other purposes. SDKs also often include sample code, supporting technical annotations, or other supporting documentation to clarify ambiguities for basic reference materials.

[0034] It's important to note that most game motion capture technologies rely on motion capture tools to calculate 3D real-world points, which is prohibitively expensive for ordinary users. Secondly, current motion capture for planar cameras (non-3D cameras) is mostly implemented on desktop clients, with no mature and readily available solutions for web browsers. Furthermore, while the open-source MediaPipe package can be used for point analysis of images and video streams in a browser, it lacks engineering-level encapsulation, meaning it's not integrated into a specific application scenario. Moreover, current motion analysis is mostly static motion analysis, lacking analysis of long movements such as certain fitness exercises, and is client-side implemented without web-based solutions. Therefore, current market solutions cannot be maturely applied to web-based game scenarios.

[0035] Therefore, to address the aforementioned issues, this application provides a motion capture method with a complete game scene SDK, applicable to most devices, enabling rapid access and ensuring accurate motion analysis data. Specifically, frame skipping and intermittent detection are used to analyze specified long movements to determine if they meet expected behavior, reducing the performance overhead of motion capture and allowing for more flexible application in various game scenarios.

[0036] This application provides a motion capture method, and also relates to a motion capture device, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0037] Figure 1 A flowchart of a motion capture method according to an embodiment of this application is shown, which specifically includes the following steps:

[0038] Step 102: Acquire the collected user operation video and obtain the action information corresponding to the user operation video.

[0039] It should be noted that motion capture methods can be applied to various interactive scenarios, such as interactive games and live streaming. Taking a game scenario as an example, users can control the game process by performing certain actions (with or without contact with the screen), such as waving their limbs, swiping or tapping on the screen. Therefore, user operation videos captured by camera components can be obtained. These user operation videos can then be analyzed to identify the specific actions performed by the user and reflected in the game interface.

[0040] In addition, action information is pre-set information about the user's actions to be performed, such as which operation the user should perform within a certain time period. In other words, action information can include the execution time period and the corresponding information about the operation to be performed. In practice, action information can specify only the type of action to be performed, or it can specify both the type of action and the specific execution location.

[0041] In one optional implementation of this embodiment, obtaining the action information corresponding to the user's operation video includes:

[0042] Receive and parse the video to be played, and obtain supplementary and enhanced information about the video to be played;

[0043] Obtain action information from supplementary and enhanced information.

[0044] The action information includes the time period to be executed and the corresponding operation to be executed. The time period to be executed is determined based on the audio in the video to be played, and the operation to be executed is the operation set for the time period to be executed.

[0045] In practical applications, this action information can be pre-set by staff based on the process of interaction, or it can be obtained through analysis of information related to the interactive operation process. Taking an audio game scenario as an example, the drum beat data of the selected audio can be analyzed to determine the action type in each time period. The set of action types in each time period can then be used as the action information for subsequent analysis.

[0046] When analyzing the drum beat data of the selected audio, the rhythm or other attributes of the audio can be analyzed manually to set the action type for each time period, i.e., manually determine the action information; in addition, the action type for each time period can be automatically determined by recognizing elements such as audio volume, beat, vocals, and instruments, i.e., automatically determine the action information.

[0047] For example, the action information could be: perform a swipe operation within 5-10 seconds, and perform a tap operation within 12-15 seconds; or, the action information could also be: perform a swipe operation from the upper left to the lower right within 5-10 seconds, and perform a tap operation at the upper left position within 12-15 seconds.

[0048] It should be noted that when the video pusher pushes a video to be played to the user, it can carry pre-analyzed motion information based on the audio of the video in the supplementary enhancement information (SEI) of the video to be played. Therefore, after receiving the video to be played, the user can parse the video and obtain the corresponding motion information from the supplementary enhancement information of the video. This motion information can instruct the user to perform actions while watching the video.

[0049] In addition, the user-side interface can display action prompts before the scheduled execution time or the end time of the action, based on the acquired action information. These prompts include a graphic indicating the action to be performed, instructing the user to execute the action before its end time. Furthermore, the prompts can also include remaining time, indicating how much time is left before the action can be executed. Moreover, if the action has a specific execution location requirement, the prompts can also include that location.

[0050] Example, Figure 2a This is a schematic diagram of a user-side playback interface provided in one embodiment of this application, as shown below. Figure 2a As shown, assuming the user needs to perform a swipe from top to bottom on the left side within 5 to 10 seconds, before the 10th second, a swipe icon (symbolizing a finger swiping from top to bottom) can be displayed in the upper left corner of the playback screen to indicate the upcoming swipe action and that the action will be performed on the left side. If the current time is 5 seconds, the upper left corner of the playback screen can also display "5 seconds remaining," meaning there are 5 seconds left to perform the swipe action.

[0051] Step 104: Perform motion recognition on each target video frame within the monitoring time period to determine the target action executed within the monitoring time period. Each target video frame is a video frame in the user operation video that meets the motion capture analysis conditions, and the monitoring time period is determined based on the motion information.

[0052] It's important to note that action information indicates which operation needs to be performed within a specific time period. Therefore, based on this action information, we can determine which time periods require user actions and necessitate monitoring of those actions. Actions performed during other time periods are ineffective and can therefore be ignored, allowing for intermittent monitoring. Thus, we can determine the monitoring time periods based on the action information, and then perform action recognition on each target video frame within those periods to identify the target action to be performed.

[0053] In practical applications, in order to save motion capture analysis resources, not all video frames within the monitoring time period need to be analyzed. Instead, target video frames that meet the motion capture analysis conditions can be extracted from the video frames within the monitoring time period, and motion recognition can be performed on the target video frames. By extracting frames, the recognition and analysis of long motions within the monitoring time period can be achieved, which greatly reduces the resources consumed by motion capture.

[0054] Among them, motion capture analysis conditions are pre-set conditions for determining whether to perform motion recognition. For example, the motion capture analysis conditions can be that the time interval for performing motion recognition must be greater than the set interval threshold.

[0055] In one optional implementation of this embodiment, action recognition is performed on each target video frame within the monitoring time period to determine the target action performed within the monitoring time period, including:

[0056] The motion capture analysis component is used to perform motion recognition on each target video frame within the monitoring time period to obtain the set of motion points for each target video frame.

[0057] Based on the set of action points, determine the target action to be performed within the monitoring time period.

[0058] Specifically, motion points refer to the identified positional coordinates related to the performed action, such as the coordinates of fingers, palms, elbows, and arms. A motion point set refers to the collection of motion points identified from various target video frames. Additionally, a motion capture analysis component refers to a tool capable of image analysis to identify motion points; for example, a motion capture analysis component could refer to Mediapipe.

[0059] It should be noted that when performing motion recognition on each target video frame within the monitoring period to determine the target action executed within that period, motion recognition can be performed on each target video frame first to obtain a set of motion points for each frame. Then, based on this set of motion points, the target action executed within the monitoring period can be determined. Specifically, motion capture analysis components can be used to identify the motion points during motion recognition on each target video frame within the monitoring period.

[0060] In an optional implementation of this embodiment, after obtaining the motion points of the target video frame by performing motion recognition on the target video frame through the motion capture analysis component, the method further includes:

[0061] Record the start time of the motion capture analysis component's processing.

[0062] It should be noted that when the motion capture analysis component performs motion recognition on the target video frame and obtains the motion points of the target video frame, the processing start time of the motion capture analysis component can be recorded. This records the time period during which the motion capture analysis component starts to perform motion recognition on the target video frame, so that the actual listening duration can be determined based on the processing start time.

[0063] In one optional implementation of this embodiment, motion capture analysis components are used to perform motion recognition on each target video frame within the monitoring time period to obtain a set of motion points for each target video frame, including:

[0064] If the time interval between the current video frame and the previous target video frame exceeds the interval threshold, determine whether the motion capture monitoring function is enabled.

[0065] If the motion capture monitoring function is enabled, the current video frame will be used as the target video frame. The motion capture analysis component will be used to perform motion recognition on the target video frame to obtain the motion points of the target video frame.

[0066] Based on the current set of motion points and the set listening conditions, determine whether to end the listening. If not, take the current video frame as the target video frame, obtain the next video frame in the user operation video as the current video frame, and continue to execute the operation steps of determining whether the motion capture listening function is enabled when the time interval between the current video frame and the previous target video frame exceeds the interval threshold, until the listening ends and the set of motion points of each target video frame within the listening time period is obtained.

[0067] It should be noted that the current video frame refers to the video frame that needs to be analyzed. It starts with the first video frame in the user operation video and then proceeds sequentially to process each subsequent video frame in the user operation video.

[0068] In practical applications, motion capture analysis conditions can be that the time interval for motion recognition is greater than a set interval threshold. The previous target video frame is the previously selected video frame that underwent motion recognition. Therefore, the time interval between the current video frame and the previous target video frame can be determined. If the time interval is not greater than the preset interval threshold, it means that the current video frame does not need to undergo motion recognition and is not considered as a target video frame.

[0069] Additionally, if the time interval is greater than a preset interval threshold, it indicates that the distance from the target video frame for motion recognition in the previous frame has exceeded the preset interval threshold. It is preliminarily determined that motion recognition can be performed on the current video frame. At this point, it can be further determined whether the motion capture monitoring function is enabled. This motion capture monitoring function can be controlled to be enabled or disabled based on the time period to be monitored. Therefore, if the motion capture monitoring function is enabled, it means that the current video frame is within the monitoring time period. At this time, the current video frame can be used as the target video frame for motion recognition. If the motion capture monitoring function is not enabled, it means that the current video frame is not within the monitoring time period. At this time, the current video frame can not be used as the target video frame, and motion recognition will not be performed.

[0070] In practice, if the current video frame is the first video frame in the user's operation video, or if there is no target video frame before the current video frame, it can be assumed that the time interval between it and the previous target video frame exceeds the interval threshold. Further, it can be determined whether the motion capture monitoring function is enabled. If it is enabled, the current video frame is used as the target video frame for action recognition. If it is not enabled, the current video frame is ignored and no recognition is performed.

[0071] The interval threshold can be determined based on the device's display refresh rate, that is, the time interval at which a video frame is extracted is determined based on the display refresh rate. For example, for a 60Hz display, it is executed once every 16.67ms (derived from 1000ms / 60), that is, a video frame is extracted every 16.67ms.

[0072] It should be noted that there is no direct correlation between the display refresh rate of a user device and its performance. An excessively fast refresh rate can lead to severe performance degradation. Therefore, it is necessary to limit the frequency of video frame extraction to reduce the number of executions. For example, setting 1 second to 20 frames per second for motion capture analysis can reduce CPU consumption from about 70% to about 30% on a Macbook Pro with an M1 chip.

[0073] In addition, by using the current video frame as the target video frame and performing motion recognition on the target video frame through the motion capture analysis component to obtain the motion points of the target video frame, the timestamp of the target video frame in the user's operation video can also be recorded, which makes it easier to determine the time interval between subsequent video frames and it.

[0074] Furthermore, using the current video frame as the target video frame, the motion capture analysis component performs motion recognition on the target video frame to obtain the motion points of the target video frame. After obtaining the motion point set composed of the currently obtained motion points (i.e., the current motion point set) and the set listening conditions, it can be determined whether to end the listening. The set listening conditions are based on the motion information and are the conditions for starting and ending the listening. For example, the set listening conditions can be whether all target video frames within the listening time period have been identified, that is, whether the end time of the listening time period has been reached; or, the set listening conditions can also be whether all motion points corresponding to the type of action to be executed indicated by the motion information have been collected, that is, whether the target action has been successfully identified. If yes, the listening ends; if no, the listening continues.

[0075] If, based on the already acquired set of motion points for each target video and the set monitoring conditions, it is determined that monitoring cannot be terminated at this time, the current video frame can be used as the target video frame, and the next video frame in the user operation video can be obtained as the current video frame. The process continues to loop back and execute the operation steps of determining whether the motion capture monitoring function is enabled when the time interval between the current video frame and the previous target video frame exceeds the interval threshold. That is, it continues to identify other target video frames that meet the motion capture analysis conditions within the monitoring time period until it is determined that monitoring can be terminated, that is, the required motion points have been monitored. At this time, the set of motion points for each target video frame within the monitoring time period can be obtained.

[0076] In practical applications, besides determining whether to end monitoring based on motion points and set monitoring conditions, an interrupt signal can also be manually triggered to end monitoring when a terminal signal is detected. It should be noted that each time the motion capture analysis component performs motion capture point analysis, it consumes CPU resources and blocks the main thread. Therefore, considering the game scenario, motion capture analysis only needs to be performed for certain periods. Thus, this embodiment uses conditional interrupts and forced interrupts to flexibly control whether to end monitoring, effectively reducing the average CPU performance consumption.

[0077] In this embodiment, video frames within a listening time period can be extracted from the collected user operation video, and motion analysis is performed only on target video frames that meet the motion capture analysis conditions within the listening time period to identify long operations performed by the user within the listening time period. By using frame extraction and intermittent detection, the performance consumption of motion capture is reduced, thereby analyzing the specified long actions, saving the resources consumed by motion capture, and improving the efficiency of motion capture.

[0078] In one optional implementation of this embodiment, the current video frame is used as the target video frame, and motion recognition is performed on the target video frame using a motion capture analysis component to obtain the motion points of the target video frame, including:

[0079] Determine whether the current state of the motion capture analysis component is idle;

[0080] If the device is in an idle state, the current video frame will be used as the target video frame.

[0081] The target video frame is fed into the motion capture analysis component, which performs motion recognition on the target video frame to obtain the motion points of the target video frame.

[0082] It should be noted that if the motion capture monitoring function is confirmed to be enabled, it is necessary to further determine whether the current state of the motion capture analysis component is idle. If it is not idle, it means that the motion capture analysis component is currently processing motion recognition for other video frames, and in this case, the current video frame can be ignored and no recognition or analysis can be performed. If it is idle, the current video frame can be used as the target video frame and passed to the motion capture analysis component. The motion capture analysis component then performs motion recognition on the target video frame to obtain the motion points of the target video frame.

[0083] In practical applications, each time a target video frame is passed to the motion capture analysis component, the running status of the motion capture analysis component can be set to running. When the motion point returned by the motion capture analysis component is obtained, it means that the motion capture analysis component has finished recognizing and processing the target video frame. At this time, the running status of the motion capture analysis component can be updated to idle.

[0084] In addition, the time when the target video frame is passed to the motion capture analysis component can be recorded, and this time can be used as the start time of the motion capture analysis component's processing.

[0085] In this embodiment, it can be determined whether the current state of the motion capture analysis component is idle. If it is idle, the target video frame is then passed in for recognition and analysis, which avoids a large number of recognition tasks queuing up and causing congestion, thus improving motion capture efficiency.

[0086] In an optional implementation of this embodiment, the action information includes the execution time period and the corresponding action type to be executed. Before determining whether the motion capture monitoring function is enabled, the following steps are also included:

[0087] Determine the corresponding listening conditions based on the execution time period and the corresponding type of action to be executed;

[0088] Control the activation and deactivation of motion capture monitoring function according to the set monitoring conditions.

[0089] It should be noted that the action information includes the execution time period and the corresponding action type to be executed, specifying which action should be executed during which time period. Therefore, based on the execution time period and the corresponding action type to be executed, the corresponding setting listening conditions can be determined, and then the motion capture listening function can be turned on and off according to the setting listening conditions.

[0090] In practical applications, the setting of the monitoring conditions can refer to the conditions that the video frames to be monitored must meet, that is, under what circumstances the motion capture monitoring function is activated and monitored, and under what circumstances the motion capture monitoring function is paused and monitored.

[0091] Using the previous example, the motion information could be: a sliding action is performed within 5-10 seconds, and a tapping action is performed within 12-15 seconds. In this case, the monitoring conditions can be set to start motion capture monitoring at the 5th second, pause motion capture monitoring at the 10th second, start motion capture monitoring at the 12th second, and pause motion capture monitoring at the 15th second; or, the monitoring conditions can also be set to start motion capture monitoring at the 5th second, pause motion capture monitoring when a complete sliding action is captured, start motion capture monitoring at the 12th second, and pause motion capture monitoring when a complete tapping action is captured.

[0092] In this embodiment, the activation and deactivation of motion capture monitoring function can be flexibly controlled by setting monitoring conditions. Motion capture analysis only needs to be performed at certain times, which effectively reduces the average performance consumption of the CPU.

[0093] In one optional implementation of this embodiment, after controlling the activation and deactivation of the motion capture monitoring function according to the set monitoring conditions, the method further includes:

[0094] Record the start time of the motion capture monitoring function.

[0095] It should be noted that after controlling the activation and deactivation of the motion capture monitoring function according to the set monitoring conditions, the activation time of the motion capture monitoring function can also be recorded, thereby recording when the motion capture monitoring function starts monitoring video frames, so as to determine the actual monitoring duration based on the processing start time.

[0096] In an optional implementation of this embodiment, before performing motion recognition on each target video frame within the monitoring time period using the motion capture analysis component, the motion capture analysis component can be initialized. That is, before performing motion recognition on each target video frame within the monitoring time period using the motion capture analysis component to obtain the motion point set of each target video frame, the following steps are also included:

[0097] Obtain the initialization file corresponding to the motion capture analysis component, and set the configuration parameters corresponding to the initialization file;

[0098] Obtain permissions for the camera component and use it to capture video of user actions.

[0099] The target video frame is selected from the user's operation video and sent to the motion capture analysis component. The motion capture analysis component performs image analysis on the target video frame to obtain the analysis results.

[0100] If the analysis result is successful, the motion capture analysis component initialization is considered complete.

[0101] It should be noted that the initialization files required by the motion capture analysis component can be precisely packaged in advance using webpack (a code compilation tool). Therefore, the initialization files corresponding to the motion capture analysis component can be obtained directly, ensuring that the user's device does not download unnecessary modules.

[0102] In practical applications, after obtaining the initialization file corresponding to the motion capture analysis component, you can also set relevant configuration parameters of the initialization file based on the actual application scenario, such as model complexity, smooth landmarks, minimum detection confidence, minimum tracking confidence, etc.

[0103] In addition, after setting the configuration parameters corresponding to the initialization file, the motion capture analysis component is not yet fully initialized. It can still obtain permissions from the camera component. If camera permissions cannot be obtained, an exception warning will be thrown. If camera permissions are obtained, the camera component can capture user operation video, select target video frames from the user operation video and pass them to the motion capture analysis component. The motion capture analysis component performs image analysis on the target video frames to obtain analysis results. If the analysis results include the identified motion points, the analysis result is considered successful, and the motion capture analysis component is considered to have completed initialization. If the analysis results do not include the identified motion points, the analysis result is considered to have failed, and the motion capture analysis component initialization is considered to have failed. Then, another video frame is passed to the motion capture analysis component to continue testing whether the initialization is complete.

[0104] In this embodiment of the application, after obtaining the permission of the camera component, the camera component can always be turned on and continuously capture user operation videos. At this time, any video frame in the captured user operation video can be input into the motion capture analysis component. The result callback after parsing the video frame ensures that the complete logic initialization of the motion capture analysis component is successful. Otherwise, in formal use, frame skipping will occur due to the incomplete initialization of the motion capture analysis component.

[0105] In one optional implementation of this embodiment, the target action to be performed within the monitoring time period is determined based on the action point set, including:

[0106] Based on the pre-initialized and generated listening task queue, determine the action type corresponding to the action point set;

[0107] By using the target judgment logic corresponding to the action type, the target action to be executed within the monitoring time period is determined based on the action point set.

[0108] In practical applications, action information includes the execution time period and the corresponding action type to be executed. Therefore, the action information can be initialized based on the execution time period and the corresponding action type to be executed in the action information to generate a corresponding listening task queue. This listening task queue includes listening tasks corresponding to the execution time period and the action type.

[0109] Therefore, during the recursive motion capture frame process, that is, when the motion capture analysis component performs motion recognition on each target video frame within the monitoring time period to obtain the motion point set of each target video frame, for each current video frame, before determining whether the time interval between the current video frame and the previous target video frame exceeds the interval threshold, the monitoring task queue is used to determine whether the current video frame is within the monitoring time period. If it is within the monitoring time period, the time interval between it and the previous target video frame is determined to further determine whether motion capture recognition should be performed; if it is not within the monitoring time period, it is determined that the current video frame does not need to be motion captured, and the current video frame is skipped directly, and the monitoring of the next video frame continues until the motion point set of each target video frame within the monitoring time period is obtained.

[0110] In practice, the current video frame is pre-determined to be within the listening time period by using the listening task queue. The timestamp of the current video frame can be determined. Based on the timestamp, the listening task queue is checked to see if a corresponding listening task exists. If it exists, it means that the current video frame is within the listening time period. If it does not exist, it means that the current video frame is not within the listening time period and no motion capture listening is required.

[0111] Continuing with the previous example, the action information is: a sliding action is performed within 5-10 seconds, and a tapping action is performed within 12-15 seconds. Therefore, the monitoring task queue includes monitoring task 1: 5-10 seconds, sliding action; and monitoring task 2: 12-15 seconds, tapping action. Assuming the timestamp of the current video frame is 5 seconds, querying the monitoring task queue reveals a corresponding monitoring task 1. At this point, it can be determined that the current video frame is within the monitoring time period. Further analysis based on the time interval between the current frame and the previous target video frame can then determine whether motion capture recognition is necessary. Alternatively, assuming the timestamp of the current video frame is 11 seconds, querying the monitoring task queue reveals no corresponding monitoring task. In this case, it can be determined that the current video frame is not within the monitoring time period, and motion capture recognition is unnecessary; the monitoring can continue to the next video frame.

[0112] Furthermore, after initializing the action information and generating a monitoring task queue, this queue includes the action types to be monitored for each execution time period. Therefore, based on this monitoring task queue, the action type corresponding to the action point set can be determined. Specifically, the timestamp range of the target video frames corresponding to the action point set can be determined, and based on this timestamp range, the corresponding action type can be determined by querying the monitoring task queue.

[0113] Using the previous example, assuming the timestamp range of the target video frame corresponding to the action point set is 5 seconds to 10 seconds, querying the monitoring task queue reveals that monitoring task 1 needs to monitor sliding operations within 5 seconds to 10 seconds. At this time, it can be determined that the action type corresponding to the action point set identified within this monitoring time period (i.e., within the timestamp range of the target video frame corresponding to the action point set) should be a sliding operation.

[0114] It should be noted that after determining the action type corresponding to the action point set based on the pre-initialized monitoring task queue, different action types have different action judgment logic. Therefore, the action point set can be analyzed using the target judgment logic corresponding to the action type of the action point set to determine the target action to be executed within the monitoring time period. In this way, different types of actions can be identified and judged using different judgment logics, improving the accuracy of motion capture.

[0115] Additionally, the possible execution locations of each action type included in the action information can be enumerated and assigned corresponding numbers to obtain an action parameter mapping table. Subsequently, after identifying the target action and execution location within the monitoring period, the corresponding action identifier can be returned. This action identifier indicates to the user the target action to be executed within the monitoring period, as well as the specific execution location, which can then be drawn and displayed on the operation interface.

[0116] For example, assuming the action type to be executed in the action information is 5 seconds to 10 seconds, a sliding operation; 12 seconds to 15 seconds, a tapping action, then enumerating each execution position of each action can yield the action parameter mapping table shown in Table 1 below.

[0117] Table 1 Action Parameter Mapping Table

[0118] 1 Top left tap - Knock.LT 2 Tap the top right corner - Knock.RT 3 Bottom left tap - Knock.LB 4 Tap the bottom right corner - Knock.RB 4 Knock.T with both hands tapping above - 6 Two-handed downward strike - Knock.B 7 Slide from top to bottom on the left - Slide.LT2LB 8 Slide from top to bottom on the right - Slide.RT2RB 9 Slide your hands down from top to bottom - Slide.T2B …… ……

[0119] In one optional implementation of this embodiment, the action type is a sliding action; based on the target judgment logic corresponding to the action type and the set of action points, the target action to be executed within the monitoring time period is determined, including:

[0120] Determine whether the longest displacement distance between any two hand coordinates in the action point set is greater than a distance threshold.

[0121] If so, based on the coordinates of any two palms, determine the starting and ending positions of the sliding motion, and thus determine the sliding position of the target sliding action executed within the monitoring time period.

[0122] It should be noted that when a user performs a swipe action, the palm needs to move from one position to another. Therefore, there must be two palm coordinates whose displacement distance is greater than the distance threshold. In other words, it is necessary to determine whether the longest displacement distance between any two palm coordinates in the action point set is greater than the distance threshold. If it is, it means that the palm has moved a relatively long distance, and the target action is a swipe action. If not, it means that the target action is a non-swipe action.

[0123] In this embodiment of the application, if the target action is determined to be a sliding action, the starting point and ending point of the sliding action can be determined based on the coordinates of any two palms, and the sliding position of the target sliding action executed within the monitoring time period can be determined, so as to subsequently control the playback of the corresponding sliding action animation in the operation interface.

[0124] In one optional implementation of this embodiment, the action type is a tapping action; based on the target judgment logic corresponding to the action type and the set of action points, the target action to be executed within the monitoring time period is determined, including:

[0125] Determine the coordinates of the shoulder, elbow, and palm in the set of action points, and determine the angles formed between the shoulder and elbow, and between the palm and elbow;

[0126] Determine if the included angle is greater than the set angle;

[0127] If so, the striking position of the target striking action executed within the monitoring period is determined based on the coordinates of the palm.

[0128] It should be noted that when a user performs a tapping action, they need to bend their arm. The angle formed between the shoulder and elbow, and between the palm and elbow should be greater than the set angle. Therefore, the angle formed between the shoulder and elbow, and between the palm and elbow can be determined. If the angle is greater than the set angle, it means that the target action is a tapping action; otherwise, it means that the target action is a non-tapping action.

[0129] In this embodiment of the application, if the target action is determined to be a tapping action, the tapping position of the target tapping action executed within the monitoring time period can be determined according to the coordinates of the palm, so as to subsequently control the playback of the corresponding tapping action animation in the operation interface.

[0130] Step 106: Determine whether the target action matches the action information. If so, determine that the target action has been captured and execute the corresponding set operation according to the target action.

[0131] Specifically, the setting operation corresponding to the target action can be to draw and display a virtual image of the user performing the target action in the user's interface; or, the setting operation corresponding to the target action can also be a control operation, which can control the current process, such as the video playback process or the current game process. As an example, when a swipe operation is captured, the game can be controlled to enter the next chapter.

[0132] It should be noted that the action point set is the location coordinates of each target video frame that meets the motion capture analysis conditions within the monitoring time period, which are related to the executed action. Therefore, based on the action point set, the target action performed by the user within the monitoring time period can be determined. Then, the target action and action information are matched. If they match, it means that the long action performed by the user within the monitoring time period is as expected. At this time, the virtual image of the user performing the target action can be drawn and displayed in the operation interface. For example, the corresponding target action can be played in the user's Live2D image. Alternatively, the control operation corresponding to the monitored target action can be determined, and the game or video process in the user's terminal can be controlled based on the control operation.

[0133] Example, Figure 2b This is a schematic diagram of a user interface provided in one embodiment of this application, such as... Figure 2b As shown, the identified target action is a swipe from top to bottom, and this target action matches the pre-set action information. That is, the swipe operation performed by the user during the listening period is as expected. Assuming the user's virtual image is a finger, a finger swiping from top to bottom can be drawn on the operation interface.

[0134] In one optional implementation of this embodiment, determining whether the target action matches the action information includes:

[0135] Based on the action information, determine the type of action to be executed corresponding to the monitoring time period;

[0136] Determine if the target action belongs to the type of action to be executed. If so, determine if the target action and action information match.

[0137] It should be noted that action information indicates what type of action needs to be performed within a specific time period. Therefore, based on the action information, the type of action to be performed corresponding to the listening time period can be determined. Then, it can be determined whether the target action performed by the user within the identified listening time period is the type of action to be performed. If so, it means that the user performed the expected target action within the listening time period, and it can be determined that the target action and action information match. Subsequently, the corresponding animation can be played in the operation interface. If not, it means that the target action performed by the user within the listening time period does not meet expectations, and it can be determined that the target action and action information do not match. Subsequently, the corresponding animation will not be played in the operation interface.

[0138] In this embodiment, a specified long motion is analyzed by frame extraction and intermittent detection to determine whether it meets the expected motion. If it does, the animation is rendered; otherwise, it is ignored and no animation is rendered. This achieves motion capture, reduces the performance consumption of motion capture, and can be applied more flexibly to various business scenarios.

[0139] In one optional implementation of this embodiment, after the motion capture analysis component performs motion recognition on the target video frame to obtain the motion points of the target video frame, the processing start time of the motion capture analysis component is recorded. After controlling the activation and deactivation of the motion capture monitoring function according to set monitoring conditions, the activation time of the motion capture monitoring function is recorded. Therefore, after determining the target action executed within the monitoring time period based on the set of motion points, the method further includes:

[0140] The actual monitoring duration of the target action is calculated based on the current time, the start time of the motion capture monitoring function, and the start time of the processing.

[0141] It should be noted that the processing start time represents the point in time when the motion capture analysis component begins to identify and process the target video frames, i.e., the time when it prepares to start parsing the target video frames. Furthermore, after obtaining the motion point set of each target video frame within the monitoring time period, the time when the motion point set of the target video frames was acquired can also be recorded, i.e., the acquisition time of the motion point set. This acquisition time of the motion point set can represent the point in time when the motion capture analysis component has finished parsing each target video frame and obtained the corresponding coordinates. Therefore, subtracting the processing start time from the acquisition time of the motion point set yields the actual processing time of the motion capture analysis component.

[0142] In practical applications, after the motion capture monitoring function is activated based on the set monitoring conditions, the activation time of the motion capture monitoring function can be recorded. The monitoring conditions are determined based on the action information; therefore, the activation time of the motion capture monitoring function refers to the point in time indicated by the action information that motion capture monitoring needs to be activated. Furthermore, since action judgment itself involves program execution time—that is, the judgment logic of the target action requires time—the current time is recorded after confirming that the target action and action information match, i.e., confirming that the target action meets the expected requirements. This current time is the time point recorded after determining that the target action meets the expected requirements. Therefore, subtracting the activation time of the motion capture monitoring function from the current time yields the initial monitoring duration of the motion capture monitoring function. Subtracting the actual processing time of the motion capture analysis component from this initial monitoring duration yields the actual monitoring duration of the target action.

[0143] For example, assuming the processing start time is 5 seconds and the time to acquire the action point set is 6.2 seconds, the actual processing time of the motion capture analysis component can be obtained as 1.2 seconds from 6.2 seconds to 5 seconds. Assuming the current time is 9 seconds and the motion capture listening function starts at 4.9 seconds, the initial listening duration can be obtained as 4.1 seconds from 9 seconds to 4.9 seconds. The actual listening duration of the target action can be obtained as 2.9 seconds from the initial listening duration of 4.1 seconds to 1.2 seconds.

[0144] In this embodiment, although the action information indicates which operation should be performed within a specific time period, such as a swipe operation between 5 and 10 seconds, the actual time users perform these actions varies. Furthermore, the time it takes to identify and determine that the target action meets expectations is not the actual listening time of the target action. Program execution may cause timeline deviations. Therefore, the actual listening duration of the target action can be calculated based on the current time, the start time of the motion capture listening function, and the processing start time. This actual listening duration deducts the analysis time of the video frames by the motion capture analysis component and the program execution time of the judgment logic, among other errors. Thus, for scenarios requiring millisecond-level data, the motion capture analysis component blocking the main thread can cause overall timeline deviations. Therefore, by recording the start time of the motion capture listening function and the processing start time as auxiliary timing methods, the inaccuracy of data caused by the actual calculation time in different device models can be eliminated.

[0145] In one optional implementation of this embodiment, the corresponding set operation is performed according to the target action, including:

[0146] Draw and display the user's virtual avatar that performs the target action in the user interface;

[0147] The target actions are scored based on the actual monitoring duration.

[0148] It should be noted that the actual listening time is the actual time the user takes to execute the action. In other words, the actual listening time can represent the execution speed of the target action. Therefore, the target action can be scored based on the actual listening time. For example, the shorter the actual listening time, the higher the score; or, the longer the actual listening time, the higher the score.

[0149] In practical applications, if the target action identified during the monitoring period is determined to be in line with expectations, the corresponding target action will be played in the user's interface, and the score corresponding to the target action will be calculated to control the game process.

[0150] It should be noted that the motion capture method provided in this application embodiment can be encapsulated into a motion capture SDK, thereby efficiently and quickly connecting to browsers on most devices to achieve motion capture. In other words, this application embodiment provides a complete motion capture SDK for game scenes, applicable to most devices, with fast integration and guaranteed accuracy of the obtained motion capture recognition data.

[0151] The motion capture method provided in this application can extract video frames within a listening time period from the collected user operation video, and only perform motion analysis on target video frames that meet the motion capture analysis conditions within the listening time period to identify long operations performed by the user within the listening time period. By using frame extraction and intermittent detection, the performance consumption of motion capture is reduced, and specified long actions can be analyzed to determine whether they meet the expected action information. This allows for more flexible application to various game business scenarios, saves resources consumed by motion capture, and improves the efficiency of motion capture.

[0152] The following is in conjunction with the appendix Figure 3a Taking the application of the motion capture method provided in this application in a rhythm game scenario as an example, the motion capture method will be further explained. Figure 3a This application provides a flowchart illustrating a motion capture method for rhythm game scenarios, which includes the following steps:

[0153] After the streamer selects and starts the music, they can pre-analyze the drum beat data (i.e., motion information, including time periods and the corresponding set of actions to be performed). This drum beat data can be added to the user operation video captured by the camera via SEI. When the user watches the live stream, they can obtain the relevant drum beat data through the captured user operation video. After the user learns that the streamer has started the rhythm game through a broadcast message, the user will launch the motion capture SDK obtained by the above motion capture method. This motion capture SDK can identify the target action performed by the user within the listening time period to determine whether it is the expected action type. If it meets the expectation, the corresponding action will be played in the user's Live2D avatar and the corresponding score will be calculated.

[0154] The entire motion capture method is divided into three main modules: initialization (initialization of the motion capture analysis component and acquisition of camera component permissions), motion capture frame recursion, and motion capture monitoring.

[0155] In practice, after the game starts, the motion capture SDK can be initialized, and then motion capture frame recursion can be initiated. After starting motion capture frame recursion, drum beat data (time period and corresponding action type to be executed) can be obtained through SEI, and the action parameters can be initialized according to the action type to be executed in the drum beat data. Then, the listening conditions are set according to the drum beat data, and the set listening conditions are passed to the motion capture listener so that the motion capture listener can determine whether to end the listening and start the motion capture listener.

[0156] The motion capture SDK initialization process specifically includes: starting initialization, recording the initialization start time, initializing the Mediapipe core logic by passing in a pre-packaged initialization file, setting configuration parameters (which may include model complexity, smoothing landmarks, minimum detection confidence, and minimum tracking confidence), obtaining camera permissions (if acquisition fails, an exception is thrown; if acquisition is successful, a video frame (any frame from the user operation video captured by the camera) is passed to the Mediapipe), and the Mediapipe performs image analysis using machine learning. If the analysis result is successful, the motion capture SDK initialization is completed; if the analysis result fails, it returns and continues to pass a video frame to the Mediapipe.

[0157] The motion capture frame recursion process can specifically include: acquiring the current video frame, determining the time interval between the current video frame and the previous target video frame (i.e., the time of the last motion recognition), determining whether the time interval exceeds an interval threshold; if not, the current video frame ends; if so, the callback start time of the current video frame is recorded. Then, determining whether to start motion capture monitoring; if not, the current video frame ends; if so, determining whether the Mediapipe is idle; if not, the current video frame ends; if so, the current video frame is passed to the Mediapipe as the target video frame, and motion recognition is performed on the target video frame through the Mediapipe to obtain the motion points of the target video frame. Finally, the processing start time of the Mediapipe is recorded, and the current video frame ends.

[0158] Motion capture monitoring can specifically include: starting motion capture monitoring based on set monitoring conditions, recording the start time, waiting for Mediapipe callback (i.e., waiting for target video frames to be passed in from Mediapipe for motion recognition), and obtaining the motion points returned by Mediapipe. The motion points corresponding to each target video frame obtained so far can be merged into a motion point set. Then, based on the currently obtained motion point set and the pre-set monitoring conditions, it is determined whether to end motion capture monitoring. If not, it continues to wait for Mediapipe callback to obtain the motion points of the next video frame; if yes, it obtains the motion point set corresponding to each target video frame, calculates the actual monitoring duration based on the motion capture start time, processing start time, and current time, ends motion capture monitoring, and returns information such as the obtained motion point set and the actual monitoring duration.

[0159] It should be noted that it is also possible to determine whether the listening needs to be terminated, i.e., whether the user manually triggers the terminal listening. If so, the motion capture listening will be terminated immediately, and the motion capture listening module's listening logic for video frames will be terminated.

[0160] If it is not necessary to terminate the monitoring, information such as the set of motion points returned by the motion capture monitoring and the actual monitoring duration can be obtained. Based on the set of motion points, the target actions within the monitoring time period can be identified and matched to control the game process.

[0161] In practical applications, Figure 3b This application provides a flowchart of a process for ending motion capture monitoring according to an embodiment. The specific implementation steps for determining whether to end monitoring based on the set of motion points and pre-input preset monitoring conditions are as follows: Figure 3b As shown, the currently acquired set of motion points is determined and passed to the motion conversion module. The motion conversion module then returns the motion recognition result. Next, motion matching is performed based on the motion recognition result and the type of motion to be executed in the drum beat data. If a match is found, motion capture monitoring ends, and the identified target motion and execution position are returned; otherwise, motion capture monitoring continues.

[0162] Among them, such as Figure 3b As shown, the action conversion module performs action recognition and returns the action recognition result. Specifically, this can include: obtaining a set of action points; determining the action type corresponding to the set of action points based on the initialized action parameters; if the action type is a tapping action, then entering the tapping action judgment logic; if the action type is a sliding action, then entering the sliding action judgment logic. The action recognition result is obtained and returned through either the tapping action judgment logic or the sliding action judgment logic.

[0163] The specific logic for judging the tapping action may include: determining the coordinates of the shoulder, elbow, and palm in the action point set, and determining the angle formed between the shoulder and elbow, and between the palm and elbow; judging whether the angle is greater than a set angle. If not, it is judged as an invalid action and the result is returned; if so, the tapping position of the target tapping action executed within the monitoring time period is determined based on the coordinates of the palm, it is determined as a tapping action, and the tapping position is returned.

[0164] The sliding action judgment logic can specifically include: determining whether the longest displacement distance between any two hand coordinates in the action point set is greater than a distance threshold. If not, it is judged as an invalid action and the result is returned; if so, based on any two hand coordinates, the starting position and ending position of the sliding action are determined, the sliding position of the target sliding action executed within the monitoring time period is determined, it is determined as a sliding action, and the sliding position is returned.

[0165] In addition, such as Figure 3b As shown, drum beat data can also be initialized in advance, which can include: initialization, mapping the execution time and corresponding action type in the drum beat data to the listening task corresponding to the time period-action type, establishing the corresponding listening task queue (time period-action type), and completing the initialization.

[0166] The motion capture method provided in this application can extract video frames within a listening time period from the collected user operation video, and only perform motion analysis on target video frames that meet the motion capture analysis conditions within the listening time period to identify long operations performed by the user within the listening time period. By using frame extraction and intermittent detection, the performance consumption of motion capture is reduced, and specified long actions can be analyzed to determine whether they meet the expected action information. This allows for more flexible application in various rhythm game scenarios, saves resources consumed by motion capture, and improves the efficiency of motion capture.

[0167] Corresponding to the above method embodiments, this application also provides embodiments of motion capture devices. Figure 4 A schematic diagram of the structure of a motion capture device according to an embodiment of this application is shown. Figure 4 As shown, the device includes:

[0168] The acquisition module 402 is configured to acquire the collected user operation video and acquire the action information corresponding to the user operation video;

[0169] The recognition module 404 is configured to perform action recognition on each target video frame within the monitoring time period to determine the target action executed within the monitoring time period. Each target video frame is a video frame in the user operation video that meets the motion capture analysis conditions, and the monitoring time period is determined based on the action information.

[0170] The determination module 406 is configured to determine whether the target action matches the action information. If so, it determines that the target action has been captured and performs the corresponding set operation based on the target action.

[0171] Optionally, the acquisition module 402 is further configured as follows:

[0172] Receive and parse the video to be played, and obtain supplementary and enhanced information about the video to be played;

[0173] Obtain action information from supplementary and enhanced information.

[0174] Optionally, the action information includes a time period to be executed and a corresponding operation to be executed. The time period to be executed is determined based on the audio in the video to be played, and the operation to be executed is an operation set for the time period to be executed.

[0175] Optionally, the identification module 404 is further configured as follows:

[0176] The motion capture analysis component is used to perform motion recognition on each target video frame within the monitoring time period to obtain the set of motion points for each target video frame.

[0177] Based on the set of action points, determine the target action to be performed within the monitoring time period.

[0178] Optionally, the identification module 404 is further configured as follows:

[0179] If the time interval between the current video frame and the previous target video frame exceeds the interval threshold, determine whether the motion capture monitoring function is enabled.

[0180] If the motion capture monitoring function is enabled, the current video frame will be used as the target video frame. The motion capture analysis component will be used to perform motion recognition on the target video frame to obtain the motion points of the target video frame.

[0181] Based on the current set of motion points and the set listening conditions, determine whether to end the listening. If not, take the current video frame as the target video frame, obtain the next video frame in the user operation video as the current video frame, and continue to execute the operation steps of determining whether the motion capture listening function is enabled when the time interval between the current video frame and the previous target video frame exceeds the interval threshold, until the listening ends and the set of motion points of each target video frame within the listening time period is obtained.

[0182] Optionally, the identification module 404 is further configured as follows:

[0183] Determine whether the current state of the motion capture analysis component is idle;

[0184] If the device is in an idle state, the current video frame will be used as the target video frame.

[0185] The target video frame is fed into the motion capture analysis component, which performs motion recognition on the target video frame to obtain the motion points of the target video frame.

[0186] Optionally, the action information includes the execution time period and the corresponding action type to be executed; the identification module 404 is further configured to:

[0187] Determine the corresponding listening conditions based on the execution time period and the corresponding type of action to be executed;

[0188] Control the activation and deactivation of motion capture monitoring function according to the set monitoring conditions.

[0189] Optionally, the device further includes an initialization module configured to:

[0190] Obtain the initialization file corresponding to the motion capture analysis component, and set the configuration parameters corresponding to the initialization file;

[0191] Obtain permissions for the camera component and use it to capture video of user actions.

[0192] The target video frame is selected from the user's operation video and sent to the motion capture analysis component. The motion capture analysis component performs image analysis on the target video frame to obtain the analysis results.

[0193] If the analysis result is successful, the motion capture analysis component initialization is considered complete.

[0194] Optionally, module 406 is further configured as follows:

[0195] Based on the pre-initialized and generated listening task queue, determine the action type corresponding to the action point set;

[0196] By using the target judgment logic corresponding to the action type, the target action to be executed within the monitoring time period is determined based on the action point set.

[0197] Optionally, the action type is a sliding action; the determination module 406 is further configured as follows:

[0198] Determine whether the longest displacement distance between any two hand coordinates in the action point set is greater than a distance threshold.

[0199] If so, based on the coordinates of any two palms, determine the starting and ending positions of the sliding motion, and thus determine the sliding position of the target sliding action executed within the monitoring time period.

[0200] Optionally, the action type is a tapping action; the determination module 406 is further configured as follows:

[0201] Determine the coordinates of the shoulder, elbow, and palm in the set of action points, and determine the angles formed between the shoulder and elbow, and between the palm and elbow;

[0202] Determine if the included angle is greater than the set angle;

[0203] If so, the striking position of the target striking action executed within the monitoring period is determined based on the coordinates of the palm.

[0204] Optionally, module 406 is further configured as follows:

[0205] Based on the action information, determine the type of action to be executed corresponding to the monitoring time period;

[0206] Determine if the target action belongs to the type of action to be executed. If so, determine if the target action and action information match.

[0207] Optionally, the identification module 404 is further configured as follows:

[0208] Record the start time of the motion capture monitoring function;

[0209] Record the start time of the motion capture analysis component's processing;

[0210] Accordingly, module 406 is further configured as follows:

[0211] Calculate the actual monitoring duration of the target action based on the current time, the start time of the motion capture monitoring function, and the start time of the processing.

[0212] Draw and display the user's virtual avatar that performs the target action in the user interface;

[0213] The target actions are scored based on the actual monitoring duration.

[0214] The motion capture device provided in this application can extract video frames within a listening time period from the captured user operation video, and only perform motion analysis on target video frames that meet the motion capture analysis conditions within the listening time period to identify long operations performed by the user within the listening time period. By using frame extraction and intermittent detection, the performance consumption of motion capture is reduced, and specified long actions can be analyzed to determine whether they meet the expected action information. It can be more flexibly applied to various game business scenarios, saves the resources consumed by motion capture, and improves the efficiency of motion capture.

[0215] The above is a schematic representation of a motion capture device according to this embodiment. It should be noted that the technical solution of this motion capture device and the technical solution of the motion capture method described above belong to the same concept. For details not described in detail in the technical solution of the motion capture device, please refer to the description of the technical solution of the motion capture method described above.

[0216] Figure 5 A structural block diagram of a computing device according to an embodiment of this application is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0217] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Controller (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0218] In one embodiment of this application, the aforementioned components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0219] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 500 can also be a mobile or stationary server.

[0220] The processor 520 is used to execute the following computer-executable instructions to implement the operation steps of the above-described motion capture method.

[0221] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the motion capture method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the motion capture method described above.

[0222] An embodiment of this application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the operation steps of the motion capture method described above.

[0223] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the motion capture method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the motion capture method described above.

[0224] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0225] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0226] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0227] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0228] The preferred embodiments disclosed above are merely illustrative of this application. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this application. These embodiments are selected and specifically described in this application to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A motion capture method, characterized in that, include: Acquire the collected user operation video and obtain the action information corresponding to the user operation video; Action recognition is performed on each target video frame within the monitoring time period to determine the target action executed within the monitoring time period. The target video frames are video frames in the user operation video that meet the motion capture analysis conditions. The monitoring time period is determined based on the action information. Determine whether the target action matches the action information. If so, determine that the target action has been captured and execute the corresponding set operation according to the target action. The step of performing action recognition on each target video frame within the monitoring time period to determine the target action performed within the monitoring time period includes: If the time interval between the current video frame and the previous target video frame exceeds the interval threshold, determine whether the motion capture monitoring function is enabled. If the motion capture monitoring function is enabled, the current video frame is used as the target video frame, and the motion capture analysis component is used to perform motion recognition on the target video frame to obtain the motion points of the target video frame. Based on the current set of action points and the set listening conditions, determine whether to end the listening. If not, take the current video frame as the target video frame, obtain the next video frame in the user operation video as the current video frame, and continue to execute the operation steps of determining whether the motion capture listening function is enabled when the time interval between the current video frame and the previous target video frame exceeds the interval threshold, until the listening ends and the set of action points of each target video frame within the listening time period is obtained. Based on the set of action points, the target action to be performed within the monitoring time period is determined.

2. The motion capture method according to claim 1, characterized in that, The step of obtaining the action information corresponding to the user operation video includes: Receive and parse the video to be played, and obtain supplementary enhancement information of the video to be played; The action information is obtained from the supplementary and enhanced information.

3. The motion capture method according to claim 1, characterized in that, The action information includes a time period to be executed and a corresponding operation to be executed. The time period to be executed is determined based on the audio in the video to be played, and the operation to be executed is an operation set for the time period to be executed.

4. The motion capture method according to claim 1, characterized in that, The step of using the current video frame as the target video frame and performing motion recognition on the target video frame through the motion capture analysis component to obtain the motion points of the target video frame includes: Determine whether the current state of the motion capture analysis component is idle; If the state is idle, the current video frame is used as the target video frame; The target video frame is transmitted to the motion capture analysis component, which performs motion recognition on the target video frame to obtain the motion points of the target video frame.

5. The motion capture method according to claim 1, characterized in that, The action information includes the execution time period and the corresponding action type to be executed; Before determining whether the motion capture monitoring function is enabled, the process also includes: Based on the execution time period and the corresponding type of action to be executed, determine the corresponding set listening conditions; Based on the set monitoring conditions, control the activation and deactivation of the motion capture monitoring function.

6. The motion capture method according to any one of claims 1-5, characterized in that, Before determining whether the motion capture monitoring function should be enabled when the time interval between the current video frame and the previous target video frame exceeds the interval threshold, the method further includes: Obtain the initialization file corresponding to the motion capture analysis component, and set the configuration parameters corresponding to the initialization file; Obtain permission from the camera component and use the camera component to capture video of user operations; The target video frame is selected from the user operation video and sent to the motion capture analysis component. The motion capture analysis component performs image analysis on the target video frame to obtain the analysis result. If the analysis result is successful, the initialization of the motion capture analysis component is considered complete.

7. The motion capture method according to any one of claims 1-5, characterized in that, The step of determining the target action to be performed within the monitoring time period based on the set of action points includes: Based on the pre-initialized and generated listening task queue, determine the action type corresponding to the action point set; Based on the target judgment logic corresponding to the action type, the target action to be executed within the monitoring time period is determined according to the action point set.

8. The motion capture method according to claim 7, characterized in that, The action type is a sliding action; the target judgment logic corresponding to the action type, based on the action point set, determines the target action to be executed within the monitoring time period, including: Determine whether the longest displacement distance between any two hand coordinates in the set of action points is greater than a distance threshold; If so, based on the coordinates of any two hands, determine the starting and ending positions of the sliding motion, and determine the sliding position of the target sliding action executed within the monitoring time period.

9. The motion capture method according to claim 7, characterized in that, The action type is a tapping action; the target judgment logic corresponding to the action type determines the target action to be executed within the monitoring time period based on the action point set, including: Determine the coordinates of the shoulder, elbow, and palm in the set of action points, and determine the included angles formed between the shoulder and the elbow, and between the palm and the elbow; Determine whether the included angle is greater than the set angle; If so, the striking position of the target striking action performed during the listening time period is determined based on the coordinates of the palm.

10. The motion capture method according to any one of claims 1-5, characterized in that, Determining whether the target action matches the action information includes: Based on the action information, determine the type of action to be executed corresponding to the monitoring time period; Determine whether the target action belongs to the type of action to be executed. If so, determine whether the target action matches the action information.

11. The motion capture method according to claim 5, characterized in that, After controlling the activation and deactivation of the motion capture monitoring function according to the set monitoring conditions, the method further includes: Record the activation time of the motion capture monitoring function; Accordingly, after obtaining the motion points of the target video frame by performing motion recognition on the target video frame through the motion capture analysis component, the method further includes: Record the processing start time of the motion capture analysis component; Accordingly, after determining the target action to be performed within the monitoring time period based on the set of action points, the method further includes: Calculate the actual monitoring duration of the target action based on the current time, the start time of the motion capture monitoring function, and the processing start time; Accordingly, the step of performing the corresponding set operation based on the target action includes: The user's virtual image performing the target action is drawn and displayed in the user interface; The target action is scored based on the actual monitoring duration.

12. A motion capture device, characterized in that, include: The acquisition module is configured to acquire the collected user operation video and acquire the action information corresponding to the user operation video; The recognition module is configured to perform action recognition on each target video frame within the monitoring time period to determine the target action performed within the monitoring time period, wherein each target video frame is a video frame in the user operation video that meets the motion capture analysis conditions, and the monitoring time period is determined based on the action information. The determination module is configured to determine whether the target action matches the action information. If so, it determines that the target action has been captured and performs the corresponding set operation according to the target action. The identification module is further configured as follows: If the time interval between the current video frame and the previous target video frame exceeds the interval threshold, determine whether the motion capture monitoring function is enabled. If the motion capture monitoring function is enabled, the current video frame is used as the target video frame, and the motion capture analysis component is used to perform motion recognition on the target video frame to obtain the motion points of the target video frame. Based on the current set of action points and the set listening conditions, determine whether to end the listening. If not, take the current video frame as the target video frame, obtain the next video frame in the user operation video as the current video frame, and continue to execute the operation steps of determining whether the motion capture listening function is enabled when the time interval between the current video frame and the previous target video frame exceeds the interval threshold, until the listening ends and the set of action points of each target video frame within the listening time period is obtained. Based on the set of action points, the target action to be performed within the monitoring time period is determined.

13. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the operation steps of the motion capture method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, It stores computer-executable instructions, which, when executed by a processor, implement the operational steps of the motion capture method according to any one of claims 1-11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the operational steps of the motion capture method according to any one of claims 1-11.

Images