Action state determination method, apparatus, device, medium, and program product

Abstract

This disclosure provides a method, apparatus, device, medium, and program product for determining motion states, relating to the field of artificial intelligence technology, and particularly to the field of computer vision technology. The specific implementation of the motion state determination method is as follows: extracting a first image sequence from a video to be processed; wherein the target object in the first image sequence is in a preparatory motion state; performing three-dimensional reconstruction of the joints of the target object in the first image sequence to obtain a sequence of differences in the flexion of the bilateral knee joints of the target object; determining the moment with the largest difference in the flexion of the bilateral knee joints of the target object in the sequence of differences as the motion preparation moment; and performing three-dimensional reconstruction of the joints of the target object in the first image related to the motion preparation moment to obtain the first motion state of the target object at the motion preparation moment.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the field of computer vision technology. Specifically, it relates to a method, apparatus, device, medium, and program product for determining an action state. Background Technology

[0002] With the development of big data technology, the field of sports biomechanics analysis has the ability to judge the standardization of athletes' technical movements by reviewing them, and then optimize and improve the movements. However, the manual review mode is difficult to accurately locate the core moment of key technical movements, which limits the optimization effect. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, medium, and program product for determining an action state.

[0004] According to one aspect of this disclosure, a method for determining an action state is provided, comprising: extracting a first image sequence from a video to be processed; wherein a target object in the first image sequence is in a preparatory motion state; performing three-dimensional reconstruction on the joints of the target object in the first image sequence to obtain a sequence of differences in the flexion of the bilateral knee joints of the target object; determining the moment when the difference in the flexion of the bilateral knee joints of the target object is the largest in the sequence of differences as the motion preparation moment; and performing three-dimensional reconstruction on the joints of the target object in the first image related to the motion preparation moment to obtain a first action state of the target object at the motion preparation moment.

[0005] According to another aspect of this disclosure, an action state determination apparatus is provided, comprising: an extraction module for extracting a first image sequence from a video to be processed; wherein a target object in the first image sequence is in a pre-motion state; a first reconstruction module for performing three-dimensional reconstruction of the joints of the target object in the first image sequence to obtain a sequence of differences in the flexion of the bilateral knee joints of the target object; a determination module for determining the moment when the difference in the flexion of the bilateral knee joints of the target object is the largest in the sequence of differences in the flexion of the bilateral knee joints of the target object as the motion preparation moment; and a second reconstruction module for performing three-dimensional reconstruction of the joints of the target object in the first image related to the motion preparation moment to obtain a first action state of the target object at the motion preparation moment.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described above.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This illustration schematically shows an exemplary system architecture to which the action state determination method and apparatus can be applied according to embodiments of the present disclosure;

[0012] Figure 2 A flowchart illustrating a method for determining an action state according to an embodiment of the present disclosure is shown schematically.

[0013] Figure 3 A flowchart illustrating the timing change sequence of determining a first position according to an embodiment of the present disclosure is shown schematically;

[0014] Figure 4 A flowchart illustrating the extraction of a first image sequence according to an embodiment of the present disclosure is shown schematically;

[0015] Figure 5 The diagram schematically illustrates an interface for motion detection using a multimodal large model according to an embodiment of the present disclosure;

[0016] Figure 6 This schematically illustrates a flowchart of motion detection in a multimodal large model of a video to be processed according to an embodiment of the present disclosure;

[0017] Figure 7 The illustration schematically shows the posture of a target object at a moment of motion preparation according to an embodiment of the present disclosure;

[0018] Figure 8 The illustration schematically shows the posture of a target object at the moment of motion initiation according to an embodiment of the present disclosure;

[0019] Figure 9 A block diagram of an operation state determination device according to an embodiment of the present disclosure is schematically shown; and

[0020] Figure 10A block diagram of an electronic device suitable for implementing an action state determination method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] In related technologies, key frames corresponding to technical actions are usually located by manually observing the playback of motion videos. However, due to the highly dynamic nature of motion scenes, it is difficult to accurately capture target frames.

[0023] In response, the proposed improvement involves having athletes wear sensors to accurately capture their movements. However, wearing these sensors can interfere with the athletes' normal movements, causing them to become distorted and resulting in the collected movement data lacking effective analytical value.

[0024] In view of this, the present disclosure provides a method for determining an action state, including: extracting a first image sequence from a video to be processed; wherein a target object in the first image sequence is in a preparatory motion state; performing three-dimensional reconstruction on the joints of the target object in the first image sequence to obtain a sequence of differences in the flexion of the bilateral knee joints of the target object; determining the moment when the difference in the flexion of the bilateral knee joints of the target object is the largest in the sequence of differences as the motion preparation moment; and performing three-dimensional reconstruction on the joints of the target object in the first image related to the motion preparation moment to obtain a first action state of the target object at the motion preparation moment.

[0025] Figure 1 The illustration schematically shows an exemplary system architecture for which action state determination methods and apparatus can be applied according to embodiments of the present disclosure.

[0026] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture to which the action state determination method and apparatus can be applied may include a terminal device, but the terminal device can implement the action state determination method and apparatus provided by the embodiments of this disclosure without interacting with the server.

[0027] like Figure 1As shown, the system architecture 100 according to this embodiment may include an image acquisition device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the image acquisition device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0028] Users can interact with server 105 via network 104 using image acquisition device 101, second terminal device 102, and third terminal device 103 to receive or send messages, etc. Image acquisition device 101 can be used to acquire video to be processed. Image acquisition device 101 can transmit the acquired video to second terminal device 102 and third terminal device 103 via network 104. Various communication client applications can be installed on second terminal device 102 and third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0029] The second terminal device 102 and the third terminal device 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0030] Server 105 can be a server that provides various services, such as a backend management server that supports the content browsed by users using image acquisition device 101, second terminal device 102, and third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0031] It should be noted that the action state determination method provided in this embodiment can generally be executed by the second terminal device 102 and the third terminal device 103. Correspondingly, the action state determination device provided in this embodiment can also be disposed in the second terminal device 102 and the third terminal device 103.

[0032] Alternatively, the action state determination method provided in this embodiment can generally be executed by server 105. Correspondingly, the action state determination device provided in this embodiment can generally be located in server 105. The action state determination method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with image acquisition device 101, second terminal device 102, third terminal device 103, and / or server 105. Correspondingly, the action state determination device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with image acquisition device 101, second terminal device 102, third terminal device 103, and / or server 105.

[0033] For example, when a user performs technical motion analysis, the video to be processed, captured by the image acquisition device 101, can be uploaded to the second terminal device 102 and the third terminal device 103. The second terminal device 102 and the third terminal device 103 then send the video to be processed to the server 105, which analyzes the video to determine the first motion state of the target object at the motion preparation moment and / or the second motion state of the target object at the motion initiation moment. Alternatively, a server or server cluster capable of communicating with the second terminal device 102, the third terminal device 103, and / or the server 105 can analyze the video to determine the first motion state of the target object at the motion preparation moment and / or the second motion state of the target object at the motion initiation moment.

[0034] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0035] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0036] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0037] Figure 2 A flowchart illustrating a method for determining an action state according to an embodiment of the present disclosure is shown schematically.

[0038] like Figure 2 As shown, the method includes operations S210~S240.

[0039] In operation S210, the first image sequence is extracted from the video to be processed.

[0040] In operation S220, the joints of the target object in the first image sequence are reconstructed in three dimensions to obtain the sequence of differences in the bending degree of the bilateral knee joints of the target object.

[0041] In operation S230, the moment when the difference in the flexion of the two knee joints of the target object is the largest in the difference change sequence is determined as the motion preparation moment.

[0042] In operation S240, the joints of the target object in the first image related to the motion preparation moment are reconstructed in three dimensions to obtain the first motion state of the target object at the motion preparation moment.

[0043] The video to be processed includes a target object that performs a specific movement within the video. This specific movement can include sports categories with clearly defined phases such as preparation, initiation, and subsequent continuous actions, such as running and swimming.

[0044] The target object in the first image sequence is in a state of readiness to move. The first image sequence includes multiple frames arranged in chronological order, where each frame is a video frame extracted from the video to be processed. Typically, when the target object is in a state of readiness to move, its limbs are basically still. Therefore, based on the range of motion of the target object in the video to be processed, images in which the target object remains basically still can be determined as the first image sequence.

[0045] For each frame in the first image sequence, the location of the joints of the target object can be determined. Then, based on the locations of these joints, 3D reconstruction is performed to determine the difference in the flexion of the bilateral knee joints of the target object in that frame. Based on the differences in the flexion of the bilateral knee joints of the target object in each frame of the first image sequence, a sequence of differences in flexion is obtained.

[0046] Although the target object in the first image sequence is in a state of preparation for movement and the target object is basically still, it is usually difficult for the target object to remain absolutely still. Therefore, the state of the target object is different in multiple frames of the first image sequence. The frame image that is closest to the standard state of preparation for movement can be selected from the multiple frames of the first image sequence as the first image of the target object's preparation moment for movement.

[0047] The criteria for determining the motion preparation moment can be determined based on the type of specific motion action performed by the target object in the video to be processed.

[0048] In the embodiments of this disclosure, taking swimming as a specific sport as an example, before the preparatory moment, the target subject may be in a waiting state, in a relaxed state, and the overall movement state is significantly different from the preparatory moment. During the preparatory moment, the target subject adjusts their body posture, making the front leg relatively straight and the back leg significantly bent due to accumulating power. Therefore, the moment with the greatest difference in the degree of flexion between the two knee joints can be considered the target subject's preparatory moment.

[0049] In this case, the moment when the difference between the two knee joints of the target object is the largest in the difference change sequence can be taken as the motion preparation moment. The joints of the target object in the first image related to the motion preparation moment can be reconstructed in three dimensions to determine the first action state of the target object at the motion preparation moment. The first action state can include the bending angle of each joint of the target object at the motion preparation moment, which is used to describe the body posture of the target object at the motion preparation moment.

[0050] According to embodiments of this disclosure, by extracting a first image sequence of the preparatory movement state, performing three-dimensional reconstruction of the joint points in the first image sequence, and capturing the maximum difference in knee flexion between the two sides, the preparatory moment of movement can be accurately located and the first action state of the corresponding target object at the preparatory moment of movement can be restored. This solves the problem that traditional action state detection is unable to accurately lock the key preparatory posture before the start of movement. Compared with schemes that rely on a single frame image or subjective judgment, analyzing and locating the joint points using three-dimensional data and quantified difference indicators improves the objectivity of the preparatory moment identification and the accuracy of the action state restoration, providing reliable data support for subsequent motion analysis.

[0051] According to embodiments of this disclosure, such as Figure 2 In the operation S210 shown, extracting the first image sequence from the video to be processed may include: performing object detection on the video to be processed to obtain a temporal change sequence of the first position of the target object; the first position represents the geometric center position of the detection box used to mark the target object in the video to be processed; and extracting the first image sequence from the video to be processed based on the positional difference between adjacent moments in the temporal change sequence of the first position.

[0052] Figure 3 A flowchart illustrating the timing change sequence of determining a first position according to an embodiment of the present disclosure is shown.

[0053] like Figure 3 As shown, the video to be processed 310 can be input into the target detection model M310 to obtain the target detection result 320 for the video to be processed 310. The target detection result 320 includes the detection boxes used to mark the position of the target object in each of the multiple frames of the video to be processed 310, as well as the geometric center position of each detection box.

[0054] The geometric center position of each detection box in the target detection result 310 can be used as the first position of that frame image. Multiple first positions are arranged according to the order of each frame image in the video to be processed, resulting in a temporal change sequence 330 of the first positions. In this example, the temporal change sequence 330 of the first positions includes the first position 331 at the first time step, the first position 332 at the second time step, the first position 333 at the third time step, and the first time step 334 at the fourth position. Based on the temporal change sequence 330 of the first positions, the change of the first positions over time can be determined. Since the first position is the geometric center position of the detection box used to mark the position of the target object, the first position can reflect the position of the target object; that is, the change of the target object's position over time can be determined based on the temporal change sequence of the first positions.

[0055] The positional differences between adjacent moments in the temporal change sequence 330 of the first position can be determined based on the differences between two adjacent elements in the temporal change sequence of the first position. Based on the positional differences between adjacent moments, the displacement of the target object within the time period corresponding to the aforementioned adjacent moments can be determined. Therefore, based on the judgment criteria for motion preparation moment, the displacement of the target object in each time period of the video to be processed can be filtered to extract multiple frames of images whose displacement meets the aforementioned judgment criteria from the video to be processed, thus obtaining the first image sequence.

[0056] According to embodiments of this disclosure, a first image sequence is extracted based on the temporal variation differences of the center position of the target object detection box, thereby achieving automated screening of images in the preparation motion state. Without the need for manual annotation or complex motion template matching, image segments of the target object in the preparation stage are quickly identified through positional stability judgment, reducing interference from invalid image data and improving the efficiency of subsequent joint reconstruction and motion analysis.

[0057] According to an embodiment of this disclosure, the temporal change sequence of the first position includes the first position at T time points; T is an integer greater than 1.

[0058] Based on the positional difference between adjacent moments in the temporal change sequence of the first position, extracting a first image sequence from the video to be processed includes: in response to determining that the positional difference between the first position at moment t+1 and the first position at moment t is less than a predetermined difference threshold, extracting the image at moment t+1 and the image at moment t from the video to be processed; t is greater than or equal to 1 and less than or equal to T-1; in response to determining that t is less than T-1, returning to perform a comparison operation between the positional difference and the predetermined difference threshold, and incrementing t; and in response to determining that t is equal to T-1, obtaining the first image sequence.

[0059] The two-dimensional coordinates of the first position at time t+1 and the first position at time t in the image at their respective times can be determined. The positional difference between the first position at time t+1 and the first position at time t can be determined based on the difference between these two-dimensional coordinates. Finally, the first image sequence can be determined based on the relationship between the positional difference and a preset difference threshold. The following section combines... Figure 4 This process will be explained. Figure 4 A flowchart illustrating the extraction of a first image sequence according to an embodiment of the present disclosure is shown.

[0060] like Figure 4 As shown, the process includes operations S401 to S407.

[0061] In operation S401, the first position at time t+1 and the first position at time t are obtained. Specifically, the first position 401 at time t+1 and the first position 402 at time t can be determined from the time sequence of the first position.

[0062] In operation S402, the position difference is calculated.

[0063] For example, the two-dimensional coordinates of the first position at time t+1 in the image are (x... t+1 y t+1 The first position at time t has two-dimensional coordinates in the image: (x...) t y t Then the difference between the above two-dimensional coordinates in the x-direction is x. t+1 - x t The difference in the y-direction is y t+1 - y t .

[0064] In one embodiment, since the determination criterion requires the target object to be in a basically static state, the differences in multiple directions can be accumulated as the position difference between the first position at time t+1 and the first position at time t.

[0065] In another embodiment, the differences in the x-direction and y-direction can be vector-summed, and the vector sum can be used to represent the positional difference. This vector sum can represent the difference from the coordinate point (x...y). t y t ) points to coordinate point (x t+1 y t+1 The vector is therefore used to represent the positional difference between the first position at time t+1 and the first position at time t.

[0066] In operation S403, determine whether the position difference is less than a predetermined difference threshold. If it is less, execute operation S404; otherwise, execute operation S407.

[0067] In operation S404, extract the image at time t+1 and the image at time t.

[0068] If the positional difference is less than a predetermined difference threshold, it means that the overall displacement of the target object from time t to time t+1 is less than the preset displacement. It can be determined that the target object is in a basically stationary state from time t to time t+1, that is, in a state of preparing to move. Therefore, the images at time t+1 and time t are extracted from the video to be processed.

[0069] In operation S405, determine if t is less than T-1. If it is less, execute operation S406; otherwise, execute operation S407.

[0070] In operation S406, increment t. Return to execute operation S401.

[0071] If t is less than T-1, it can be determined that there are still images in the current video to be processed that have not undergone the comparison operation between position difference and predetermined difference threshold. Therefore, after incrementing t, the above comparison operation is performed again, that is, the relationship between the position difference between the first position at time t+1 and the first position at time t+2 and the predetermined difference threshold is determined.

[0072] Operation S407, operation ended.

[0073] If the positional difference is greater than a predetermined difference threshold, it means that the overall displacement of the target object from time t+1 to time t+2 is greater than the preset displacement. It can be determined that the target object is not in a state of preparation for movement from time t+1 to time t+2. Therefore, it is not necessary to extract the images at time t+2 and time t+1 from the video to be processed.

[0074] Given that t equals T-1, it can be determined that each frame of the video to be processed has been processed. The currently extracted images can be arranged in chronological order to obtain the first image sequence.

[0075] According to embodiments of this disclosure, by cyclically comparing the positional differences between adjacent time points with a predetermined threshold, continuous extraction and integrity assurance of the first image sequence are achieved. Only consecutive frames with positional differences less than the threshold are retained, ensuring that the extracted image sequences all correspond to the relatively static preparation state of the target object. This avoids analysis errors caused by image mixing during motion and further improves the purity of the preparation motion state data.

[0076] Since the preparation and initiation phases have a strict sequential order, the target object will not be in the preparation phase first, then the initiation phase, and then back to the preparation phase. Therefore, in another example of this disclosure, in response to determining that the positional difference between the first position at time k+1 and the first position at time k is greater than a predetermined difference threshold, processing for multiple times after time k can be stopped, and the currently extracted images are arranged chronologically to obtain the first image sequence. By reducing the number of comparison operations, the extraction efficiency of the first image sequence is improved.

[0077] According to another embodiment of this disclosure, in addition to determining the motion of the target object at two adjacent moments by sequentially calculating the position difference of the first position at two adjacent moments, and then determining the first image sequence indicating that the target object is in a state of preparing for motion, a standard action in the state of preparing for motion can also be set, and the standard action can be matched with the action of the target object in the video to be processed, so as to extract the first image sequence of the target object in the state of preparing for motion in the video to be processed.

[0078] Specifically, extracting a first image sequence from the video to be processed includes: performing motion detection on the video to be processed to obtain the matching degree between the actual action of the target object in the video to be processed and the predetermined preparation action; and based on the matching degree, extracting a first image sequence from the video to be processed that matches the predetermined preparation action.

[0079] A pre-defined preparation action can represent a standardized action of an object in a preparatory state. Each frame of the video to be processed can be sequentially compared with the pre-defined preparation action to determine the degree of matching between the actual action of the target object corresponding to each frame and the pre-defined preparation action.

[0080] For example, standard images corresponding to predetermined preparatory actions can be used as labels to train an image processing model. Each frame of the video to be processed is then input into the trained image processing model, and the similarity between each frame and the standard image is output. The similarity is used as the matching degree between the actual action of the target object in each frame and the predetermined preparatory action.

[0081] The matching degree can be filtered according to a preset matching degree threshold, and images with a matching degree higher than the matching degree threshold can be extracted. The extracted images are then arranged in chronological order to obtain the first image sequence.

[0082] According to embodiments of this disclosure, a first image sequence is extracted based on the matching degree between the actual action and the predetermined preparatory action, thereby achieving accurate screening of the preparatory action. Compared to positional screening, focusing directly on the matching of the action itself can better fit the core definition of the preparatory motion state, avoiding the erroneous extraction of images when the target object is stationary but not in the preparatory action state, and improving the correlation between the image sequence and the analysis target.

[0083] According to embodiments of this disclosure, motion detection is performed on the video to be processed to obtain the matching degree between each frame of the video to be processed and a predetermined preparatory action, including: obtaining descriptive text for describing the predetermined preparatory action; and using a multimodal large model to perform motion detection on the video to be processed based on the descriptive text to obtain the matching degree.

[0084] In one example, an image corresponding to a predetermined preparatory action can be input into a large model, which will then generate descriptive text to describe that action. In another example, the motion category to which the predetermined preparatory action belongs can be used as a cue word input into the large model, which will then generate descriptive text.

[0085] The generated description text could be: The athlete stands facing the pool at the front of the starting block, with feet shoulder-width apart or slightly narrower, the balls of the feet firmly gripping the edge of the starting block, and heels slightly raised to maintain a forward lean; the upper body is naturally bent, the back is straight and the chest is not hunched, the arms are extended straight forward and upward and brought together, with palms facing each other or lightly touching, the head is kept neutral, and the gaze is focused on the water in front of the entry point; the core muscles are tightened, the hips are slightly lowered into a half-squat position, and the leg muscles are moderately pre-tensioned to store potential energy for power generation.

[0086] In one example of this disclosure, a multimodal large model can perform motion detection on each frame of the video to be processed according to the descriptive text to determine whether each frame matches the descriptive text, thereby determining the matching degree of each frame.

[0087] Figure 5 An interface for motion detection using a multimodal large model is illustrated schematically according to an embodiment of the present disclosure.

[0088] like Figure 5 As shown, the bottom of the interface contains the user input box, the right side displays the user-submitted content, and the left side displays the multimodal large model output content.

[0089] Users can input each frame of the video to be processed, along with descriptive text, into a multimodal large-scale model, and input prompts such as: "Determine whether the athlete's movements in the image are consistent with the movements described in the text, and output the degree of consistency, quantified as a percentage." The multimodal large-scale model can perform consistency comparisons of the target object's movements in the image based on the descriptive text and output a quantified matching degree, such as "In the image you provided, the athlete's movements are 95% consistent with the standard movements described in the text." According to embodiments of this disclosure, using a multimodal large-scale model combined with descriptive text for action detection overcomes the limitations of traditional action detection methods that rely on preset templates or labeled data. Pre-defined preparatory movements can be defined through natural language descriptions, eliminating the need for complex feature engineering or model retraining, thus improving the flexibility and adaptability of the method and enabling rapid response to preparatory movement detection needs in different scenarios.

[0090] In another example of this disclosure, using a multimodal large model to perform action detection on the video to be processed based on descriptive text and obtain a matching degree may include: using the multimodal large model to generate a reference image for displaying a predetermined preparatory action based on the descriptive text; and using the multimodal large model to perform matching analysis between the video to be processed and the reference image to generate a matching degree.

[0091] Figure 6 The flowchart illustrating motion detection in a multimodal large model of a video to be processed according to an embodiment of the present disclosure is shown schematically.

[0092] like Figure 6 As shown, inputting the descriptive text 601 into the multimodal large model M601 can generate a reference image 602 for displaying the predetermined preparatory action. Inputting the reference image 602 and the video to be processed 310 into the multimodal large model M601, and using the multimodal large model M601 to perform matching analysis between the video to be processed and the reference image 602, a matching degree 603 can be obtained.

[0093] According to embodiments of this disclosure, a reference image is generated through a multimodal large model and matched with video for analysis, transforming text descriptions into visual reference standards and improving the accuracy of action matching. The reference image provides a clear visual benchmark for action matching, avoiding the ambiguity that may exist in pure text descriptions, making the matching degree calculation more intuitive and reliable, and further ensuring the accuracy of the first image sequence extraction.

[0094] Figure 7 The illustration schematically shows the posture of a target object at a moment of motion preparation according to an embodiment of the present disclosure.

[0095] like Figure 7As shown, during the preparatory phase of the exercise, the subject stands on the exercise-assisting device with their legs staggered, the front leg positioned on the edge of the device and the back leg on the footplate. Because the footplate is higher than the platform, the back leg is bent more than the front leg. The upper body leans forward, the arms extend forward, and the palms lightly touch the front edge of the device. In swimming, the exercise-assisting device typically refers to the starting platform.

[0096] Knee flexion is typically used to represent the degree of leg flexion, where knee flexion represents the angle between the thigh and lower leg. Specifically, the thigh is determined by the position of the groin and the knee, and the lower leg is determined by the position of the knee and the ankle. Therefore, knee flexion characterizes the angle between the line segment between the groin and the knee and the line segment between the knee and the ankle.

[0097] According to embodiments of this disclosure, such as Figure 2 In the operation S220 shown, the joints of the target object in the first image sequence are reconstructed in three dimensions to obtain the difference change sequence of the bilateral knee joint flexion of the target object. This includes: reconstructing the joints of the target object in any image in the first image sequence in three dimensions to obtain the position of the first node of the target object in any image; the first node includes the first bilateral thigh root node, the first bilateral knee joint node and the first bilateral ankle node; processing the position of the first node of the target object in any image to obtain the difference of the bilateral knee joint flexion of the target object in any image; and sorting the differences of the bilateral knee joint flexion of the target object in each image according to the position of each image in the first image sequence to obtain the difference change sequence.

[0098] For any image in the first image sequence, determine the two-dimensional target object joints in the image, infer the three-dimensional coordinates of each target object joint based on the two-dimensional target object joints, thereby determining the three-dimensional position of the first node of the target object, and locate each part of the target object based on the position of the multiple nodes included in the first node in three-dimensional space to achieve three-dimensional reconstruction.

[0099] Based on the position of the first node in the 3D reconstruction results, the first line segment between the target object's groin and knee joint positions, and the second line segment between the target object's knee joint and ankle nodes are determined. The angle between the first and second line segments is used to determine the bilateral knee flexion of the target object. Then, the difference between the bilateral knee flexion is obtained.

[0100] Based on the differences in bilateral knee flexion, the images are sorted to obtain a difference change sequence. In the difference change sequence, the difference in bilateral knee flexion of the target object in the images that are earlier in the sequence is greater than the difference in bilateral knee flexion of the target object in the images that are later in the sequence.

[0101] According to embodiments of this disclosure, the quantification and temporal analysis of curvature differences are achieved by calculating the difference change sequence of key joints through three-dimensional reconstruction. Compared with angle calculation in two-dimensional images, three-dimensional reconstruction eliminates the influence of viewpoint deviation, the precise positioning of key joints ensures the accuracy of curvature calculation, and the difference change sequence provides a continuous and reliable quantitative basis for determining the motion preparation moment.

[0102] According to embodiments of this disclosure, the first bilateral thigh root nodes include a left thigh root node and a right thigh root node; the first bilateral knee joint nodes include a left knee joint node and a right knee joint node; and the first bilateral ankle nodes include a left ankle node and a right ankle node.

[0103] The position of the first node of the target object in any image is processed to obtain the difference in the bilateral knee flexion of the target object in any image. This includes: performing dot product processing on the positions of the left thigh root node, the left knee joint node, and the left ankle node to obtain the left knee flexion of the target object in any image; performing dot product processing on the positions of the right thigh root node, the right knee joint node, and the right ankle node to obtain the right knee flexion of the target object in any image; and obtaining the difference in the bilateral knee flexion of the target object in any image based on the difference between the left and right knee flexion.

[0104] by Figure 7 For example, Figure 7 The middle foreleg is the left leg, and the hind leg is the right leg. The left leg can be determined by the position of the left thigh root node 711, the position of the left knee joint node 712, and the position of the left ankle node 713. The right leg can be determined by the position of the right thigh root node 721, the position of the right knee joint node 722, and the position of the right ankle node 723.

[0105] Taking the calculation of the flexion of the left knee joint as an example, based on the position 711 of the left thigh root node and the position 712 of the left knee joint node, the vector 714 representing the left thigh can be determined. Based on the position 712 of the left knee joint node and the position 713 of the left ankle node, the vector 715 representing the left calf can be determined. By performing a dot product on the vector 714 of the left thigh and the vector 715 of the left calf, the dot product result of the left knee joint can be obtained.

[0106] Furthermore, based on the vector 714 of the left thigh and the vector 715 of the left calf, scalars representing the length of the left thigh and the length of the left calf can be determined respectively. By multiplying the above scalars, the scalar product of the left knee joint can be obtained.

[0107] According to the dot product formula, the dot product of two vectors is equal to the product of the scalar product of the two vectors and the cosine of the angle between the vectors. Therefore, after obtaining the dot product and scalar product of the left knee joint, the quotient of the dot product and scalar product of the left knee joint can be calculated to obtain the cosine of the left knee joint curvature α. The curvature of the left knee joint can be determined by calculating using inverse trigonometric functions.

[0108] Similarly, the right knee flexion β can be determined based on the position 721 of the right thigh root node, the position 722 of the right knee joint node, and the position 723 of the right ankle node.

[0109] The difference between the left and right knee flexion of the target object in any image can be determined by subtracting the left knee flexion from the right knee flexion. In this example, the difference between the two knee flexion of the target object is α-β.

[0110] According to embodiments of this disclosure, the flexion of both knee joints is calculated separately by dot product processing and the difference is calculated, thus achieving accurate quantification of bilateral differences. Dot product processing ensures the mathematical rigor of angle calculation, and separate calculation for both sides avoids interference from unilateral data, enabling the difference value to truly reflect the limb symmetry state of the target object when preparing for movement, providing a core quantitative indicator for determining the moment of movement preparation.

[0111] According to embodiments of this disclosure, such as Figure 2 In the operation S240 shown, the joints of the target object in the first image related to the motion preparation moment are reconstructed in three dimensions to obtain the first motion state of the target object at the motion preparation moment. This includes: extracting the first image related to the motion preparation moment from the first image sequence; reconstructing the joint positions of the target object in the first image in three dimensions to obtain the position of the second node of the target object in the first image; wherein the second node includes: the thigh root node, the knee joint node, and the torso node; and obtaining the first motion state of the target object at the motion preparation moment based on the position of the second node and the bilateral knee flexion of the target object at the motion preparation moment.

[0112] Since the difference in the flexion of the two knee joints of the target object is the greatest in the first image, meaning that the first image best represents the target object's posture at the moment of motion preparation, the joints of the target object in the first image can be reconstructed in three dimensions to further determine the target object's first motion state at the moment of motion preparation based on the reconstruction results.

[0113] It should be noted that, since the initial motion state of the target object needs to be determined, the joints of the target object no longer need to focus on the bilateral knee joints, but rather cover the entire target object to determine the overall posture of the target object at the moment of motion preparation. For example, the trunk node of the target object can be selected to represent the posture of the upper limbs, and the groin node and knee joint node can be selected to represent the posture of the lower limbs.

[0114] Specifically, in the first image, the nodes of the thigh root, knee joint, and torso of the target object can be determined respectively. Then, the three-dimensional coordinates of each joint point can be inferred based on the two-dimensional nodes, thereby determining the three-dimensional position of the second node of the target object and completing the three-dimensional reconstruction.

[0115] Based on the position of the second node and the bilateral knee flexion of the target object at the moment of motion preparation, the target object's movement and posture at the moment of motion preparation can be obtained, and this can be used as the first movement state.

[0116] According to embodiments of this disclosure, the scope of joint reconstruction is extended to the trunk nodes, and the first action state is reconstructed by combining the flexion of both knee joints with the positions of key nodes, thus achieving a comprehensive representation of the action state. Compared to analysis that only focuses on the knee joints, adding the trunk and groin node positions can more completely reflect the body posture of the target object, improving the comprehensiveness and realism of the analysis results.

[0117] According to an embodiment of this disclosure, the first motion state of the target object at the moment of motion preparation is obtained based on the position of the second node and the bilateral knee flexion of the target object at the moment of motion preparation. This includes: performing dot product processing on the positions of the thigh root node, the knee joint node, and the trunk node to obtain the hip joint angle of the target object at the moment of motion preparation; and obtaining the first motion state based on the bilateral knee flexion and hip joint angle of the target object at the moment of motion preparation.

[0118] Similar to the calculation of knee flexion, the vector representing the lower limb of the target object can be determined based on the position of the thigh root node and the position of the knee joint node. The vector representing the upper limb of the target object can be determined based on the position of the thigh root node and the position of the torso node. The dot product of the vectors of the lower limb and the upper limb can be used to obtain the dot product result of the hip joint.

[0119] Furthermore, based on the vectors of the upper limb and the lower limb, scalars representing the length of the upper limb and the length of the lower limb can be determined respectively. By performing scalar product on the above scalars, the scalar product result of the hip joint can be obtained.

[0120] The cosine of the hip joint angle can be obtained by quotient of the dot product and the scalar product of the hip joint. The hip joint angle can then be determined by inverse trigonometric functions.

[0121] It is understandable that by determining the hip joint angle and the flexion of both knee joints, the flexion angles of each joint related to the movement preparation moment can be determined, thereby determining the posture of the target object, i.e., the first movement state.

[0122] According to embodiments of this disclosure, the hip joint angle is calculated through dot product processing and combined with the knee joint flexion to construct a first action state, thereby achieving multi-joint angle collaborative representation of the action. The hip and knee joint angles together constitute the core joint parameters of the preparatory movement state. Compared with a single joint angle, this can more accurately describe the limb posture characteristics of the target object, improving the detail and scientific nature of the action state reconstruction.

[0123] After obtaining the first motion state, it can be analyzed to determine whether the target object's posture at the motion preparation moment is standard and scientific. Furthermore, based on the first motion state, the target object's posture at the motion preparation moment can be calibrated, generating a training strategy specifically for the motion preparation moment. Specifically, the first image at the motion preparation moment can be input into a multimodal large model, and based on the first motion state of the target object represented by the first image, a training strategy for improving the motion preparation moment can be generated.

[0124] In addition to the preparatory phase of exercise, the motion state determination method can be used to analyze the target object's movements during the initiation phase of exercise, so as to generate training strategies for the target object during the initiation phase of exercise.

[0125] Specifically, the motion state determination method may further include: extracting a second image sequence from the video to be processed; the acquisition time of the second image sequence is later than the motion preparation time; and performing motion state detection on the second image sequence to determine the motion start time.

[0126] Since the motion initiation phase is later than the motion preparation phase, and the motion preparation moment is a moment within the motion preparation phase, the images in the video to be processed after the motion preparation moment can be used as the second image sequence, and the target object in the second image sequence is in the motion initiation state.

[0127] By monitoring the motion state of the second image sequence, the frame image that is closest to the standard motion start state can be determined from the multiple frames of the second image sequence, and the time of that frame image is determined as the motion start time.

[0128] Specifically, a descriptive text describing the predetermined start action can be obtained. A multimodal large model can be used to perform motion detection on multiple frames of the second image sequence based on the descriptive text describing the predetermined start action, obtain the matching degree of each frame, and determine the moment of the image with the highest matching degree as the motion start moment.

[0129] According to embodiments of this disclosure, a second image sequence following the motion preparation moment is extracted, and the motion initiation moment is detected, enabling coherent analysis of the entire motion process's motion state. This overcomes the limitation of focusing only on a single moment of motion, constructing a complete temporal analysis chain from preparation to initiation, thereby achieving accurate identification of the motion initiation moment.

[0130] According to an embodiment of this disclosure, the second image sequence includes N images; N is an integer greater than 1.

[0131] The motion state detection of the second image sequence to determine the motion start time includes: performing three-dimensional reconstruction of the joints of the target object in the nth image to obtain the motion state of the supporting leg of the target object in the nth image; the supporting leg represents the leg whose contact state between the target object and the motion assist device changes at the motion start time; in response to determining that the motion state does not match the predetermined start motion state, returning to perform the three-dimensional reconstruction operation and incrementing n; n is an integer greater than or equal to 1 and less than or equal to N; and in response to determining that the motion state matches the predetermined start motion state, determining the acquisition time of the nth image as the motion start time.

[0132] In swimming, the start of the movement can be defined as the moment when a target standing on a motion-assist device jumps forward into the water by exerting force with their supporting leg. Therefore, the contact state between the target's supporting leg and the motion-assist device changes before and after the start of the movement. That is, before the start of the movement, the supporting leg is in contact with the motion-assist device, and after the start of the movement, the supporting leg is not in contact with the motion-assist device.

[0133] During the warm-up phase, the target athlete stands with their legs staggered, with the front leg bent less and the back leg bent more. During the subsequent initiation phase, the target athlete pushes off with their back leg, propelling them forward into the pool. Therefore, the leg with the higher degree of bending during the warm-up phase can be designated as the supporting leg.

[0134] Since the supporting leg needs to exert force during the start of the movement, its position at the start should be a straight leg. Therefore, the knee flexion of the supporting leg at the start of the movement is close to a horizontal angle. Thus, the state of the supporting leg described above can be taken as the predetermined starting position, and by comparing this predetermined starting position with the actual movement of the supporting leg, the start time of the movement can be determined.

[0135] Specifically, the joints of the target object in the nth image are obtained, and the coordinates of each joint in the three-dimensional space are inferred based on the two-dimensional position of each joint in the nth image, thereby determining the three-dimensional position of each joint and completing the three-dimensional reconstruction.

[0136] Based on the results of the 3D reconstruction, the motion state of the supporting leg of the target object in the nth image can be determined. The motion state is compared with the predetermined starting motion state. If a mismatch is found, the 3D reconstruction and subsequent judgment operations are re-executed after incrementing n. If a match is found, the current motion state of the supporting leg is determined to be consistent with the predetermined starting motion state that the supporting leg should have at the moment of motion initiation. Therefore, the acquisition time of the nth image can be determined as the moment of motion initiation.

[0137] According to embodiments of this disclosure, by reconstructing the joints of the supporting leg in three dimensions and matching them with a predetermined starting action state, the moment of motion initiation can be accurately located. Furthermore, by focusing on the supporting leg as a key component for motion initiation and using the predetermined starting action state as a criterion, interference from non-critical components is avoided, improving the accuracy of initiation moment recognition and adapting to scenarios with stringent requirements for initiation timing, such as those involving motion-assistance devices.

[0138] Figure 8 The illustration schematically shows the posture of a target object at the moment of motion initiation according to an embodiment of the present disclosure.

[0139] like Figure 8 As shown, at the moment of initiation of the movement, the supporting leg of the target object straightens due to force exertion and is in a state of about to separate from the motion assistance device, while the non-supporting leg has already left the motion assistance device and is therefore relatively relaxed. Therefore, the knee flexion of the supporting leg is less than that of the non-supporting leg. According to embodiments of this disclosure, the movement state characterizes the knee flexion of the supporting leg.

[0140] The joints of the target object in the nth image are reconstructed in three dimensions to obtain the motion state of the supporting leg of the target object in the nth image. This includes: reconstructing the joints of the target object in the nth image in three dimensions to obtain the position of the third node; wherein the third node includes the thigh root node of the supporting leg, the knee joint node of the supporting leg, and the ankle node of the supporting leg; and performing dot product processing on the positions of the thigh root node, the knee joint node, and the ankle node of the supporting leg to obtain the knee flexion of the supporting leg.

[0141] For the nth image, the key points of the two-dimensional target object in the image can be determined. Based on the two-dimensional key points of the target object, the three-dimensional coordinates of each key point of the target object can be inferred, thereby determining the three-dimensional position of the third node of the target object and completing the three-dimensional reconstruction.

[0142] Based on the third node, which includes the thigh root node and the knee joint node of the supporting leg, the vector representing the thigh of the supporting leg can be determined. Based on the knee joint node and the ankle node of the supporting leg, the vector representing the lower leg of the supporting leg can be determined. By performing a dot product on the vectors of the thigh and the lower leg of the supporting leg, the dot product result of the knee joint of the supporting leg can be obtained.

[0143] Furthermore, based on the vectors of the supporting leg thigh and the supporting leg calf, scalars representing the length of the supporting leg thigh and the supporting leg calf can be determined respectively. By multiplying these scalars, the scalar product of the supporting leg knee joint can be obtained.

[0144] After obtaining the dot product and scalar product results of the supporting leg knee joint, the quotient of the dot product and scalar product results of the supporting leg knee joint can be calculated to obtain the cosine value of the knee joint flexure. The knee joint flexure can then be determined by using inverse trigonometric functions.

[0145] According to embodiments of this disclosure, the knee flexure is calculated by three-dimensional reconstruction of key joints of the supporting leg and dot product processing, thereby achieving a quantitative representation of the supporting leg's movement state. The precise positioning of key joints and dot product processing ensure the accuracy of the flexure calculation, providing a reliable quantitative basis for matching with the predetermined starting movement state and guaranteeing the consistency of the starting moment determination.

[0146] According to another embodiment of this disclosure, motion state detection is performed on a second image sequence to determine the motion start time, including: performing target detection on the second image sequence to obtain a temporal change sequence of the second position of the target object; the second position represents the geometric center position of the detection box used to mark the target object in the second image sequence; and the moment with the largest horizontal position difference between adjacent moments in the temporal change sequence of the second position is determined as the motion start time.

[0147] It is understandable that the process of determining the temporal change sequence of the second position of the target object is similar to that of determining the temporal change sequence of the first position of the target object, except that the target detection object is different: target detection of the first image sequence can determine the temporal change sequence of the first position, and target detection of the second image sequence can determine the temporal change sequence of the second position.

[0148] Before the start of the movement, the target object waits on the motion-assisting device, so the target object moves only a small amount in the forward direction. Therefore, in the temporal change sequence of the second position, the difference in horizontal position between adjacent moments before the start of the movement is close to 0.

[0149] At the moment of motion initiation, the target object jumps forward into the water with the help of motion-assisting equipment, resulting in a large displacement in the forward direction. Therefore, in the temporal change sequence of the second position, the difference in horizontal position between adjacent moments of motion initiation is the greatest.

[0150] After the start of the movement, the target object enters the water and swims at a speed lower than its starting speed at the start of the movement. Therefore, in the temporal change sequence of the second position, the difference in horizontal position between adjacent moments after the start of the movement is smaller than that at the start of the movement.

[0151] Therefore, the moment with the greatest difference in horizontal position between adjacent moments in the temporal change sequence of the second position can be determined as the moment of motion initiation.

[0152] According to embodiments of this disclosure, the motion initiation time is determined based on the maximum difference in horizontal position between adjacent moments in the second position time-series change sequence, achieving rapid identification of the initiation time. No complex joint reconstruction and angle calculations are required; the initiation timing can be determined solely by the change in the target object's position, resulting in high computational efficiency and fast response speed.

[0153] according to Figure 8 It is known that at the start of the movement, the supporting leg of the target is in a straightened position, therefore the knee flexion of the supporting leg is smaller than that of the non-supporting leg. Therefore, the knee flexion of both sides can be determined separately, and then the supporting leg can be determined based on the knee flexion.

[0154] Similar to the method for calculating bilateral knee flexion in this disclosure, the joints of the target object in the nth image are reconstructed in three dimensions to obtain the motion state of the supporting leg of the target object in the nth image. The method further includes: reconstructing the joints of the target object in the nth image in three dimensions to obtain the position of a fourth node; the fourth node includes: a second bilateral thigh root node, a second bilateral knee joint node, and a second bilateral ankle node; performing dot product processing on the positions of the second bilateral thigh root node, the second bilateral knee joint node, and the second bilateral ankle node to obtain the bilateral knee flexion of the target object in the nth image; and determining the supporting leg based on the bilateral knee flexion.

[0155] After obtaining the bilateral knee flexion of the target object in the nth image, the leg corresponding to the knee with the smaller flexion can be identified as the supporting leg.

[0156] According to embodiments of this disclosure, the supporting leg is determined based on the flexion of both knee joints, achieving automated identification of the supporting leg. No manual annotation or additional sensor data is required; the supporting leg can be determined solely through 3D reconstruction of joint points and angle calculation. This enhances the autonomy and practicality of the method, avoids application scenario limitations caused by reliance on external devices, and improves the accuracy and efficiency of the supporting leg identification process.

[0157] according to Figure 8 It can be seen that a coordinate system can be established with the motion-assisting device as the origin, the target object's forward direction as the positive x-axis, and the vertical upward direction as the positive z-axis. Since the forward direction is the positive x-axis, and the non-supporting leg leaves the motion-assisting device before the supporting leg, it can be concluded that at the moment of motion initiation, the supporting leg of the target object is further back relative to the target object's forward direction compared to the non-supporting leg. Therefore, the supporting leg can be determined based on the position of the target object's ankles.

[0158] According to embodiments of this disclosure, the joints of the target object in the nth image are reconstructed in three dimensions to obtain the motion state of the supporting leg of the target object in the nth image. The method further includes: reconstructing the joints of the target object in the nth image in three dimensions to obtain the positions of the bilateral ankle nodes of the target object in the nth image; and determining the supporting leg based on the positions of the bilateral ankle nodes.

[0159] Since the supporting leg's ankle is positioned rearward relative to the target object's forward movement, the coordinates of both ankles in three-dimensional space can be determined based on their respective positions. Specifically, using the target object's forward movement direction as the x-axis, the ankle with the smaller x-component in its coordinates is the supporting leg's ankle.

[0160] According to embodiments of this disclosure, the ankle node position can directly reflect the anterior-posterior relationship of both ankles, thereby quickly determining the supporting leg, making the recognition logic simpler, more intuitive, and requiring less computation.

[0161] According to embodiments of this disclosure, the motion state determination method may further include: extracting a second image related to the motion initiation moment from a second image sequence; performing three-dimensional reconstruction on the joint positions of the target object in the second image to obtain the position of a fifth node of the target object in the second image; wherein the fifth node includes: the thigh root node of the non-supporting leg, the knee joint node of the non-supporting leg, and the ankle node of the non-supporting leg; performing dot product processing on the thigh root node of the non-supporting leg, the knee joint node of the non-supporting leg, and the ankle node of the non-supporting leg to obtain the knee flexion of the non-supporting leg; and determining the second motion state of the target object at the motion initiation moment based on the knee flexion of the non-supporting leg and the knee flexion of the supporting leg.

[0162] Similar to performing 3D reconstruction on the first image to obtain the first motion state of the target object at the moment of motion preparation, 3D reconstruction can be performed on the second image to obtain the position of the fifth node of the target object in the second image. The fifth node includes the joint point of the non-supporting leg.

[0163] By using the method of determining knee flexion, the knee flexion of the non-supporting leg can be determined based on the joint points of the non-supporting leg included in the fifth node.

[0164] Based on the knee flexion of the non-supporting leg and the knee flexion of the supporting leg, it is possible to determine

[0165] According to embodiments of this disclosure, by reconstructing key joints of the non-supporting leg in three dimensions and calculating its flexure, a second motion state is constructed by combining the flexure of the supporting leg, thus achieving a comprehensive restoration of the limb posture at the moment of motion initiation. Simultaneously, by characterizing the joint states of the supporting and non-supporting legs, the limb coordination relationship during the target object's initiation of motion is fully reflected, providing more comprehensive motion data for biomechanical analysis, assistive device control, and other applications.

[0166] After obtaining the second motion state, it can be analyzed to determine whether the target object's posture at the moment of motion initiation is standard and scientific. Furthermore, based on the second motion state, the target object's posture at the moment of motion initiation can be calibrated, generating a training strategy specifically for that moment. Specifically, the second image at the moment of motion initiation can be input into a multimodal large model. Based on the second motion state of the target object represented by the second image, a training strategy for improving the moment of motion initiation can be generated.

[0167] Figure 9 A block diagram of an operation state determination device according to an embodiment of the present disclosure is shown schematically.

[0168] like Figure 9 As shown, the action state determination device 900 may include an extraction module 910, a first reconstruction module 920, a determination module 930, and a second reconstruction module 940.

[0169] Extraction module 910 is used to extract a first image sequence from the video to be processed; wherein the target object in the first image sequence is in a state of preparing for motion;

[0170] The first reconstruction module 920 is used to perform three-dimensional reconstruction of the joints of the target object in the first image sequence to obtain the sequence of differences in the bending degree of the bilateral knee joints of the target object.

[0171] Module 930 is used to determine the moment when the difference in knee flexion between the two sides of the target object is greatest in the differential change sequence as the motion preparation moment; and

[0172] The second reconstruction module 940 is used to perform three-dimensional reconstruction of the joints of the target object in the first image related to the motion preparation moment, so as to obtain the first motion state of the target object at the motion preparation moment.

[0173] According to embodiments of this disclosure, the extraction module 910 includes a target detection submodule and a first extraction submodule.

[0174] The object detection submodule is used to perform object detection on the video to be processed and obtain the temporal change sequence of the first position of the object; the first position represents the geometric center position of the detection box used to mark the object in the video to be processed.

[0175] The first extraction submodule is used to extract the first image sequence from the video to be processed based on the positional differences between adjacent moments in the temporal change sequence of the first position.

[0176] According to embodiments of this disclosure, the extraction submodule includes an image extraction unit, a repetition comparison unit, and a sequence determination unit.

[0177] An image extraction unit is used to extract the image at time t+1 and the image at time t from the video to be processed in response to determining that the positional difference between the first position at time t+1 and the first position at time t is less than a predetermined difference threshold; t is greater than or equal to 1 and less than or equal to T-1.

[0178] The repeat comparison unit is used to return a comparison operation between the positional difference and a predetermined difference threshold in response to determining that t is less than T-1, and increment t.

[0179] A sequence determination unit is used to obtain a first image sequence in response to determining that t equals T-1.

[0180] According to embodiments of this disclosure, the extraction module 910 includes an action detection submodule and a second extraction submodule.

[0181] The motion detection submodule is used to perform motion detection on the video to be processed, and to obtain the matching degree between the actual motion of the target object in the video and the predetermined preparation motion.

[0182] The second extraction submodule is used to extract a first image sequence from the video to be processed that matches a predetermined preparation action, based on the matching degree.

[0183] According to embodiments of this disclosure, the motion detection submodule includes a text acquisition unit and a motion detection unit.

[0184] The text acquisition unit is used to acquire descriptive text that describes the predetermined preparatory actions.

[0185] The action detection unit is used to perform action detection on the video to be processed based on the descriptive text using a multimodal large model, and obtain the matching degree.

[0186] According to embodiments of this disclosure, the motion detection unit includes an image generation subunit and a matching subunit.

[0187] The image generation subunit is used to generate reference images based on descriptive text using a multimodal large model to demonstrate predetermined preparatory actions.

[0188] The matching subunit is used to perform matching analysis between the video to be processed and the reference image using a multimodal large model, and generate the matching degree.

[0189] According to embodiments of this disclosure, the first reconstruction module 920 includes a first reconstruction submodule, a difference determination submodule, and a sorting submodule.

[0190] The first reconstruction submodule is used to perform three-dimensional reconstruction of the joints of the target object in any image of the first image sequence to obtain the position of the first node of the target object in any image; the first node includes the first bilateral thigh root node, the first bilateral knee joint node and the first bilateral ankle node.

[0191] The difference determination submodule is used to process the position of the first node of the target object in any image to obtain the difference in the flexion of the two knee joints of the target object in any image.

[0192] The sorting submodule is used to sort the differences in the bilateral knee flexion of the target object in each image according to the position of each image in the first image sequence to obtain the difference change sequence.

[0193] According to embodiments of this disclosure, the difference determination submodule includes a first dot product unit, a second dot product unit, and a difference determination unit.

[0194] The first dot product unit is used to perform dot product processing on the positions of the left thigh root node, the left knee joint node, and the left ankle node to obtain the left knee joint flexure of the target object in any image.

[0195] The second dot product unit is used to perform dot product processing on the positions of the right thigh root node, the right knee joint node, and the right ankle node to obtain the right knee joint flexion of the target object in any image.

[0196] The difference determination unit is used to obtain the difference in the flexion of the two knee joints of the target object in any image based on the difference between the flexion of the left knee joint and the flexion of the right knee joint.

[0197] According to embodiments of this disclosure, the second reconstruction module 940 includes an image extraction submodule, a second reconstruction submodule, and a state determination submodule.

[0198] The image extraction submodule is used to extract the first image related to the motion preparation moment from the first image sequence.

[0199] The second reconstruction submodule is used to perform three-dimensional reconstruction of the joint positions of the target object in the first image to obtain the positions of the second nodes of the target object in the first image; wherein, the second nodes include: the thigh root node, the knee joint node, and the torso node.

[0200] The state determination submodule is used to obtain the first motion state of the target object at the moment of motion preparation based on the position of the second node and the bilateral knee flexion degree of the target object at the moment of motion preparation.

[0201] According to embodiments of this disclosure, the state determination submodule includes a third dot product unit and a state determination unit.

[0202] The third dot product unit is used to perform dot product processing on the positions of the thigh root node, the knee joint node, and the trunk node to obtain the hip joint angle of the target object at the moment of motion preparation.

[0203] The state determination unit is used to obtain the first motion state based on the bilateral knee flexion and hip angle of the target object at the moment of motion preparation.

[0204] According to embodiments of this disclosure, the action state determination device 900 further includes a second extraction module and a detection module.

[0205] The second extraction module is used to extract a second image sequence from the video to be processed; the acquisition time of the second image sequence is later than the motion preparation time.

[0206] The detection module is used to detect the motion state of the second image sequence and determine the motion start time.

[0207] According to embodiments of this disclosure, the detection module includes a third reconstruction submodule, a repeated reconstruction submodule, and a time determination submodule.

[0208] The third reconstruction submodule is used to perform three-dimensional reconstruction of the joints of the target object in the nth image to obtain the motion state of the supporting leg of the target object in the nth image; the supporting leg represents the leg whose contact state between the target object and the motion assist device changes at the moment of motion initiation.

[0209] The Repeat Reconstruction submodule is used to respond to a mismatch between the determined action state and the predetermined start action state, return to perform a 3D reconstruction operation, and increment n; n is an integer greater than or equal to 1 and less than or equal to N.

[0210] The timing determination submodule is used to determine the acquisition time of the nth image as the motion start time in response to the determination of the action state matching the predetermined start action state.

[0211] According to embodiments of this disclosure, the third reconstruction submodule includes a first reconstruction unit and a fourth dot product unit.

[0212] The first reconstruction unit is used to perform three-dimensional reconstruction of the joints of the target object in the nth image to obtain the position of the third node; wherein, the third node includes the thigh root node of the supporting leg, the knee joint node of the supporting leg, and the ankle node of the supporting leg.

[0213] The fourth dot product unit is used to perform dot product processing on the positions of the thigh root node, the knee joint node, and the ankle node of the supporting leg to obtain the knee flexion of the supporting leg.

[0214] According to embodiments of this disclosure, the third reconstruction submodule includes a second reconstruction unit, a fifth dot product unit, and a first support determination unit.

[0215] The second reconstruction unit is used to perform three-dimensional reconstruction of the joints of the target object in the nth image to obtain the position of the fourth node; the fourth node includes: the second bilateral thigh root node, the second bilateral knee joint node, and the second bilateral ankle node.

[0216] The fifth dot product unit is used to perform dot product processing on the positions of the second bilateral thigh root nodes, the second bilateral knee joint nodes, and the second bilateral ankle nodes, respectively, to obtain the bilateral knee joint flexion of the target object in the nth image.

[0217] The first support determination unit is used to determine the supporting leg based on the flexion of both knee joints.

[0218] According to embodiments of this disclosure, the third reconstruction submodule includes a third reconstruction unit and a second support determination unit.

[0219] The third reconstruction unit is used to perform three-dimensional reconstruction of the joints of the target object in the nth image, and obtain the positions of the bilateral ankle nodes of the target object in the nth image.

[0220] The second support determination unit is used to determine the supporting leg based on the position of the bilateral ankle nodes.

[0221] According to embodiments of this disclosure, the action state determination device 900 further includes an image extraction module, a third reconstruction module, a dot product module, and a state determination module.

[0222] The image extraction module is used to extract a second image related to the motion initiation time from the second image sequence.

[0223] The third reconstruction module is used to perform three-dimensional reconstruction of the joint positions of the target object in the second image to obtain the position of the fifth node of the target object in the second image; wherein, the fifth node includes: the thigh root node of the non-supporting leg, the knee joint node of the non-supporting leg, and the ankle node of the non-supporting leg.

[0224] The dot product module is used to perform dot product processing on the thigh root node, knee joint node, and ankle node of the non-supporting leg to obtain the knee flexion of the non-supporting leg.

[0225] The state determination module is used to determine the second motion state of the target object at the moment of motion initiation based on the knee flexion of the non-supporting leg and the knee flexion of the supporting leg.

[0226] According to embodiments of this disclosure, the detection module includes a detection submodule and a time determination submodule.

[0227] The detection submodule is used to perform target detection on the second image sequence to obtain the temporal change sequence of the second position of the target object; the second position represents the geometric center position of the detection box used to mark the target object in the second image sequence.

[0228] The time determination submodule is used to determine the moment with the greatest difference in horizontal position between adjacent moments in the temporal change sequence of the second position as the motion start moment.

[0229] It should be noted that the action state determination device part in the embodiments of this disclosure corresponds to the action state determination method part in the embodiments of this disclosure. For a detailed description of the action state determination device part, please refer to the action state determination method part, which will not be repeated here.

[0230] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0231] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0232] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0233] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0234] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0235] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded into random access memory (RAM) 1003 from storage unit 1008. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0236] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0237] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the action state determination method. For example, in some embodiments, the action state determination method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the action state determination method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform an action state determination method by any other suitable means (e.g., by means of firmware).

[0238] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0239] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0240] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0241] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0242] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0243] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0244] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0245] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for determining an action state, comprising: A reference image for displaying the predetermined preparatory action is generated based on descriptive text describing the predetermined preparatory action using a multimodal large model; as well as The multimodal large model is used to match and analyze the video to be processed with the reference image to generate the matching degree between the actual action of the target object in the video to be processed and the predetermined preparation action; Based on the matching degree, a first image sequence is extracted from the video to be processed; wherein the target object in the first image sequence is in a state of preparing for motion. Three-dimensional reconstruction is performed on the joints of the target object in any image of the first image sequence to obtain the position of the first node of the target object in any image; the first node includes the first bilateral thigh root node, the first bilateral knee joint node and the first bilateral ankle node; The position of the first node of the target object in any image is processed to obtain the difference in the bilateral knee flexion of the target object in any image. The differences in the bilateral knee flexion of the target object in each image are sorted according to the position of each image in the first image sequence to obtain the sequence of changes in the bilateral knee flexion of the target object; The moment when the difference in bilateral knee flexion of the target object is greatest in the differential change sequence is determined as the exercise preparation moment; and The joints of the target object in the first image related to the motion preparation moment are reconstructed in three dimensions to obtain the first motion state of the target object at the motion preparation moment.

2. The method of claim 1, wherein, The step of extracting the first image sequence from the video to be processed includes: Target detection is performed on the video to be processed to obtain a temporal change sequence of the first position of the target object; the first position represents the geometric center position of the detection box used to mark the target object in the video to be processed; and Based on the positional differences between adjacent moments in the temporal change sequence of the first position, the first image sequence is extracted from the video to be processed.

3. The method of claim 2, wherein, The temporal change sequence of the first position includes the first position at T time points; T is an integer greater than 1; Extracting the first image sequence from the video to be processed based on the positional differences between adjacent moments in the temporal change sequence of the first position includes: In response to determining that the positional difference between the first position at time t+1 and the first position at time t is less than a predetermined difference threshold, the images at time t+1 and time t are extracted from the video to be processed; t is greater than or equal to 1 and less than or equal to T-1; In response to determining that t is less than T-1, return to the comparison operation between the positional difference and the predetermined difference threshold, and increment t; and In response to determining that t equals T-1, the first image sequence is obtained.

4. The method of claim 1, wherein, The knee flexure is characterized by the angle of the line segment between the root of the thigh and the knee joint relative to the line segment between the knee joint and the ankle node.

5. The method of claim 4, wherein, The first bilateral thigh root nodes include a left thigh root node and a right thigh root node; the first bilateral knee joint nodes include a left knee joint node and a right knee joint node; the first bilateral ankle nodes include a left ankle node and a right ankle node. The step of processing the position of the first node of the target object in any image to obtain the difference in bilateral knee flexion of the target object in any image includes: The left knee joint flexure of the target object in any image is obtained by performing dot product processing on the positions of the left thigh root node, the left knee joint node, and the left ankle node. The dot product of the positions of the right thigh root node, the right knee joint node, and the right ankle node is performed to obtain the right knee joint flexion of the target object in any image. The difference in knee flexion between the left and right knees in any image is obtained based on the difference between the left and right knee flexion.

6. The method of claim 1, wherein, The step of performing three-dimensional reconstruction of the joints of the target object in the first image related to the motion preparation moment to obtain the first motion state of the target object at the motion preparation moment includes: Extract the first image related to the motion preparation moment from the first image sequence; The joint positions of the target object in the first image are reconstructed in three dimensions to obtain the positions of the second nodes of the target object in the first image; wherein, the second nodes include: the thigh root node, the knee joint node, and the torso node; Based on the position of the second node and the bilateral knee flexion of the target object at the moment of motion preparation, the first motion state of the target object at the moment of motion preparation is obtained.

7. The method of claim 6, wherein, The step of obtaining the first motion state of the target object at the moment of motion preparation based on the position of the second node and the bilateral knee flexion of the target object at the moment of motion preparation includes: The positions of the thigh root node, the knee joint node, and the trunk node are multiplied by a dot product to obtain the hip joint angle of the target object at the moment of motion preparation. The first motion state is obtained based on the bilateral knee flexion and hip angle of the target object at the moment of motion preparation.

8. The method according to claim 1, further comprising: Extract a second image sequence from the video to be processed; The acquisition time of the second image sequence is later than the motion preparation time; The motion state of the second image sequence is detected to determine the start time of the motion.

9. The method of claim 8, wherein, The second image sequence comprises N images; N is an integer greater than 1. Perform motion state detection on the second image sequence to determine the motion initiation time, including: The joints of the target object in the nth image are reconstructed in three dimensions to obtain the motion state of the supporting leg of the target object in the nth image; the supporting leg represents the leg whose contact state between the target object and the motion assist device changes at the moment of motion initiation; In response to determining that the action state does not match the predetermined start action state, the system returns to perform a 3D reconstruction operation and increments n; n is an integer greater than or equal to 1 and less than or equal to N; and In response to determining that the action state matches a predetermined start action state, the acquisition time of the nth image is determined as the motion start time.

10. The method of claim 9, wherein, The motion state characterizes the knee flexion of the supporting leg; The step of performing three-dimensional reconstruction of the joints of the target object in the nth image to obtain the motion state of the supporting leg of the target object in the nth image includes: The joints of the target object in the nth image are reconstructed in three dimensions to obtain the position of the third node; wherein, the third node includes the thigh root node of the supporting leg, the knee joint node of the supporting leg, and the ankle node of the supporting leg; and The knee flexure of the supporting leg is obtained by performing dot product on the positions of the thigh root node, the knee joint node, and the ankle node of the supporting leg.

11. The method of claim 9 or 10, wherein, The step of performing three-dimensional reconstruction of the joints of the target object in the nth image to obtain the motion state of the supporting leg of the target object in the nth image also includes: The joints of the target object in the nth image are reconstructed in three dimensions to obtain the position of the fourth node; the fourth node includes: the second bilateral thigh root node, the second bilateral knee joint node, and the second bilateral ankle node; Dot product processing is performed on the positions of the second bilateral thigh root nodes, the second bilateral knee joint nodes, and the second bilateral ankle nodes to obtain the bilateral knee flexion of the target object in the nth image; and The supporting leg is determined based on the bilateral knee flexion.

12. The method according to claim 9 or 10, wherein, The step of performing three-dimensional reconstruction of the joints of the target object in the nth image to obtain the motion state of the supporting leg of the target object in the nth image also includes: Perform 3D reconstruction of the joints of the target object in the nth image to obtain the positions of the bilateral ankle nodes of the target object in the nth image; and The supporting leg is determined based on the position of the bilateral ankle nodes.

13. The method of claim 12, further comprising: Extract a second image related to the motion initiation time from the second image sequence; The joint positions of the target object in the second image are reconstructed in three dimensions to obtain the position of the fifth node of the target object in the second image; wherein, the fifth node includes: the thigh root node of the non-supporting leg, the knee joint node of the non-supporting leg, and the ankle node of the non-supporting leg; The knee flexure of the non-supporting leg is obtained by performing dot product processing on the thigh root node, the knee joint node, and the ankle node of the non-supporting leg. The second motion state of the target object at the moment of motion initiation is determined based on the knee flexion of the non-supporting leg and the knee flexion of the supporting leg.

14. The method according to claim 8, wherein, The step of detecting the motion state of the second image sequence and determining the motion initiation time includes: Target detection is performed on the second image sequence to obtain a temporal change sequence of the second position of the target object; the second position represents the geometric center position of the detection box used to mark the target object in the second image sequence; and The moment with the largest difference in horizontal position between adjacent moments in the temporal change sequence of the second position is determined as the motion start moment.

15. An action state determination device, comprising: An extraction module is used to generate a reference image for displaying the predetermined preparatory action based on descriptive text describing the predetermined preparatory action using a multimodal large model; The multimodal large model is used to match and analyze the video to be processed with the reference image to generate the matching degree between the actual action of the target object in the video to be processed and the predetermined preparation action; Based on the matching degree, a first image sequence is extracted from the video to be processed; wherein the target object in the first image sequence is in a state of preparing for motion. The first reconstruction module is used to perform three-dimensional reconstruction of the joint points of the target object in any image of the first image sequence to obtain the position of the first node of the target object in any image; the first node includes a first bilateral thigh root node, a first bilateral knee joint node and a first bilateral ankle node. Processing the position of the first node of the target object in any image yields the difference in knee flexion between the two sides of the target object in any image; and The differences in the bilateral knee flexion of the target object in each image are sorted according to the position of each image in the first image sequence to obtain the sequence of changes in the bilateral knee flexion of the target object; The determination module is used to determine the moment when the difference in the bilateral knee flexion of the target object is greatest in the difference change sequence as the exercise preparation moment; and The second reconstruction module is used to perform three-dimensional reconstruction of the joints of the target object in the first image related to the motion preparation moment, so as to obtain the first motion state of the target object at the motion preparation moment.

16. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-14.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-14.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-14.

Images