An action scoring method, apparatus, device and storage medium

Abstract

The present disclosure relates to a motion scoring method, device, equipment and storage medium, and relates to the technical field of fitness mirrors. The method comprises the following steps: first, acquiring a human body key point sequence corresponding to a to-be-scored motion on a target image, a human body key point template and a human body joint angle template; determining a key point Pearson matching degree, a joint angle Pearson matching degree and a joint angle difference value similarity; and determining a motion score of the to-be-scored motion on the target image according to the key point Pearson matching degree, the joint angle Pearson matching degree and the joint angle difference value similarity. It can be seen that, by means of the key point Pearson matching degree, the joint angle Pearson matching degree and the joint angle difference value similarity, the motion score of the to-be-scored motion on the target image is determined, and the accuracy of the motion score is improved.

Description

Technical Field

[0001] This disclosure relates to the field of fitness mirror technology, and in particular to a motion scoring method, device, equipment, and storage medium. Background Technology

[0002] Currently, in the field of fitness mirror technology, users can use motion scores to reflect the effectiveness of exercising while following a fitness mirror.

[0003] However, when users exercise with a fitness mirror, the motion scores are usually evaluated based on data such as single angle differences or key point similarities, which leads to inaccurate motion scores. Therefore, how to improve the accuracy of motion scores has become a technical problem that urgently needs to be solved. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides a motion scoring method, apparatus, device, and storage medium.

[0005] The technical solution disclosed herein is as follows:

[0006] In a first aspect, this disclosure provides a motion scoring method, the method comprising:

[0007] Obtain the sequence of human key points corresponding to the action to be scored on the target image, as well as the human key point template and human joint angle template corresponding to the action to be scored; wherein, the human key point template includes a standard human key point sequence, and the human joint angle template includes a standard human joint angle sequence.

[0008] The Pearson matching degree between the standard human key point sequence in the human key point template and the human key point sequence corresponding to the action to be scored is determined as the key point Pearson matching degree.

[0009] In addition, the Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human key point sequence corresponding to the action to be scored is determined as the joint angle Pearson matching degree.

[0010] Based on the differences between the joint angles that correspond to each other in the standard human joint angle sequence and the human key point sequence, the joint angle difference similarity is determined.

[0011] The motion score of the action to be scored on the target image is determined based on the key point Pearson matching degree, the joint angle Pearson matching degree, and the joint angle difference similarity.

[0012] In one optional implementation, determining the Pearson match degree between the standard human keypoint sequence in the human keypoint template and the human keypoint sequence corresponding to the action to be scored, as the keypoint Pearson match degree, includes:

[0013] The weighted Pearson product moment of the standard human key point sequence in the normalized human key point template and the human key point sequence corresponding to the action to be scored is calculated as the key point Pearson matching degree; wherein, the weight values ​​of the standard human key points in the standard human key point sequence and the human key points in the human key point sequence are determined based on the weight values ​​of the preset human body regions to which they belong.

[0014] In one optional implementation, determining the Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human key point sequence corresponding to the action to be scored, as the joint angle Pearson matching degree, includes:

[0015] The weighted Pearson product moment of the joint angle formed by the standard human joint angle sequence in the normalized human joint angle template and the human key point sequence corresponding to the action to be scored is calculated as the joint angle Pearson matching degree; wherein, the weight value of the joint angle formed by the standard human joint angle in the standard human joint angle sequence and the human key point sequence is determined based on the weight value of the preset human region to which it belongs.

[0016] In one optional implementation, determining the action score of the action to be scored on the target image based on the keypoint Pearson matching degree, the joint angle Pearson matching degree, and the joint angle difference similarity includes:

[0017] Based on the similarity of the joint angle difference and the weight value of each joint angle, a first score is determined;

[0018] Furthermore, a second score is determined based on the Pearson matching degree of the key points and the Pearson matching degree of the joint angles;

[0019] Based on the first score and the second score, the action score of the action to be scored on the target image is determined.

[0020] In one optional implementation, determining the first score based on the similarity of the joint angle differences and the weight values ​​of each joint angle includes:

[0021] Determine the first fraction using formula (1):

[0022]

[0023] Where S1 is the first fraction, N+1 is the number of joint angles formed by the sequence of human key points, and Δx i W is the angle difference of the i-th joint angle among the corresponding joint angles formed by the standard human joint angle sequence and the human key point sequence. ai The weight value of the i-th joint angle is determined based on the weight value of the preset human body region to which the i-th joint angle belongs, where M is a preset first integer and P is a preset second integer.

[0024] In one optional implementation, determining the second score based on the keypoint Pearson matching degree and the joint angle Pearson matching degree includes:

[0025] Use formula (2) to determine the second fraction:

[0026] S2=min{[Peark*Wpk+min(Peara, Peak)*Wpa], 1}; (2)

[0027] Wherein, S2 is the second score, Peark is the key point Pearson matching degree, Wpk is the preset human key point Pearson product moment weight coefficient, Peara is the joint angle Pearson matching degree, and Wpa is the preset joint angle Pearson product moment weight coefficient.

[0028] In one optional implementation, determining the action score of the action to be scored on the target image based on the first score and the second score includes:

[0029] The action score of the action to be scored on the target image is determined using formula (3):

[0030] S=w*(w1*S1+(1-w1)*S2); (3)

[0031] Where S is the action score of the action to be scored on the target image, w is the preset scoring coefficient corresponding to the action to be scored, and w1 is the preset scoring coefficient.

[0032] In an optional implementation, before acquiring the sequence of human key points corresponding to the action to be scored on the target image, the method further includes:

[0033] Identify the human body detection bounding box on the first image and use it as the actual detection box;

[0034] Identify key points in the actual detection frame to obtain the matching sequence corresponding to the actual detection frame; wherein, the matching sequence includes key points with an ordered relationship;

[0035] The sequence to be matched is matched by a target sliding window to obtain the matching result corresponding to the sequence to be matched; wherein, the target sliding window is determined based on at least one preset standard action template;

[0036] Based on the matching results corresponding to the sequence to be matched, determine whether the action to be scored is matched on the first image;

[0037] If it is determined that the action to be scored is successfully matched on the first image, then the first image is determined as the target image.

[0038] In one optional implementation, the human detection box on the target image, as the actual detection box, includes:

[0039] The target image is input into a human detection network model, and after processing by the human detection network model, a human detection box on the target image is output; wherein, the human detection network model is trained using an image sample set labeled with detection boxes.

[0040] In one optional implementation, identifying human key points within the actual detection frame and obtaining the matching sequence corresponding to the actual detection frame includes:

[0041] The actual detection box is input into the human keypoint detection network model. After processing by the human keypoint detection network model, the human keypoints on the actual detection box are output. The human keypoint detection network is trained using training sample images and training supervised images.

[0042] Based on the human body key points on the actual detection box, a matching sequence corresponding to the actual detection box is generated.

[0043] In one optional implementation, the step of matching the sequence to be matched through a target sliding window to obtain the matching result corresponding to the sequence to be matched includes:

[0044] Weighted bipartite graph matching is performed based on the standard action template corresponding to the target sliding window, the key points in the sequence to be matched, and the weight values ​​corresponding to the preset human body regions to which the key points belong, to obtain the first Pearson product moment between the key points in the sequence to be matched and the standard action template corresponding to the target sliding window.

[0045] Based on the standard action template corresponding to the target sliding window, the joint angle formed by the key points in the sequence to be matched, and the weight value corresponding to the preset human body region to which the joint angle belongs, weighted bipartite graph matching is performed to obtain the second Pearson product moment between the joint angle formed by the key points in the sequence to be matched and the standard action template corresponding to the target sliding window.

[0046] The matching result corresponding to the sequence to be matched is determined based on the first Pearson product moment and the second Pearson product moment.

[0047] Secondly, this disclosure provides a motion scoring device, the device comprising:

[0048] The acquisition unit is used to acquire the sequence of human key points corresponding to the action to be scored on the target image, as well as the human key point template and human joint angle template corresponding to the action to be scored; wherein, the human key point template includes a standard human key point sequence, and the human joint angle template includes a standard human joint angle sequence.

[0049] The processing unit is used to determine the Pearson matching degree between the standard human key point sequence in the human key point template and the human key point sequence corresponding to the action to be scored, as the key point Pearson matching degree.

[0050] The processing unit is also used to determine the Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human key point sequence corresponding to the action to be scored, as the joint angle Pearson matching degree.

[0051] The processing unit is also used to determine the joint angle difference similarity based on the difference between the corresponding joint angles formed by the standard human joint angle sequence and the human key point sequence.

[0052] The processing unit is further configured to determine the action score of the action to be scored on the target image based on the key point Pearson matching degree, the joint angle Pearson matching degree, and the joint angle difference similarity.

[0053] In one optional implementation, the processing unit is specifically used to calculate the weighted Pearson product moment of the standard human key point sequence in the normalized human key point template and the human key point sequence corresponding to the action to be scored, as the key point Pearson matching degree; wherein, the weight values ​​of the standard human key points in the standard human key point sequence and the human key points in the human key point sequence are determined based on the weight values ​​of the preset human body regions to which they belong.

[0054] In one optional implementation, the processing unit is specifically used to calculate the weighted Pearson product moment of the joint angle formed by the standard human joint angle sequence in the normalized human joint angle template and the human key point sequence corresponding to the action to be scored, as the joint angle Pearson matching degree; wherein, the weight value of the joint angle formed by the standard human joint angle in the standard human joint angle sequence and the human key point sequence is determined based on the weight value of the preset human body region to which it belongs.

[0055] In one optional implementation, the processing unit is specifically configured to determine a first score based on the similarity of the joint angle difference and the weight values ​​of each joint angle;

[0056] The processing unit is specifically used to determine a second score based on the key point Pearson matching degree and the joint angle Pearson matching degree;

[0057] The processing unit is specifically configured to determine the action score of the action to be scored on the target image based on the first score and the second score.

[0058] In one alternative implementation, the processing unit is specifically configured to determine a first fraction using formula (1):

[0059]

[0060] Where S1 is the first fraction, N+1 is the number of joint angles formed by the sequence of human key points, and Δx i W is the angle difference of the i-th joint angle among the corresponding joint angles formed by the standard human joint angle sequence and the human key point sequence. ai The weight value of the i-th joint angle is determined based on the weight value of the preset human body region to which the i-th joint angle belongs, where M is a preset first integer and P is a preset second integer.

[0061] In one alternative implementation, the processing unit is specifically used to determine the second fraction using formula (2):

[0062] S2=min{[Peark*Wpk+min(Peara, Peak)*Wpa], 1}; (2)

[0063] Wherein, S2 is the second score, Peark is the key point Pearson matching degree, Wpk is the preset human key point Pearson product moment weight coefficient, Peara is the joint angle Pearson matching degree, and Wpa is the preset joint angle Pearson product moment weight coefficient.

[0064] In one optional implementation, the processing unit is specifically configured to determine the action score of the action to be scored on the target image using formula (3):

[0065] S=w*(w1*S1+(1-w1)*S2); (3)

[0066] Where S is the action score of the action to be scored on the target image, w is the preset scoring coefficient corresponding to the action to be scored, and w1 is the preset scoring coefficient.

[0067] In one optional implementation, the processing unit is specifically used to identify human body detection boxes on the first image as actual detection boxes;

[0068] The processing unit is specifically used to identify key points in the actual detection frame and obtain the matching sequence corresponding to the actual detection frame; wherein, the matching sequence includes key points with an ordered relationship;

[0069] The processing unit is specifically used to match the sequence to be matched through a target sliding window to obtain the matching result corresponding to the sequence to be matched; wherein, the target sliding window is determined based on at least one preset standard action template;

[0070] The processing unit is specifically used to determine whether the action to be scored is matched on the first image based on the matching result corresponding to the sequence to be matched;

[0071] The processing unit is specifically configured to determine the first image as the target image if it is determined that an action to be scored is successfully matched on the first image.

[0072] In one optional implementation, the processing unit is specifically configured to input the target image into a human detection network model, and after processing by the human detection network model, output human detection boxes on the target image; wherein, the human detection network model is trained using an image sample set labeled with detection boxes.

[0073] In one optional implementation, the processing unit is specifically used to input the actual detection box into the human keypoint detection network model, and after processing by the human keypoint detection network model, output the human keypoints on the actual detection box; wherein, the human keypoint detection network is trained using training sample images and training supervision images.

[0074] The processing unit is specifically used to generate a matching sequence corresponding to the actual detection frame based on the human body key points on the actual detection frame.

[0075] In one optional implementation, the processing unit is specifically used to perform weighted bipartite graph matching based on the standard action template corresponding to the target sliding window, the key points in the sequence to be matched, and the weight values ​​corresponding to the preset human body regions to which the key points belong, to obtain the first Pearson product moment between the key points in the sequence to be matched and the standard action template corresponding to the target sliding window.

[0076] The processing unit is specifically used to perform weighted bipartite graph matching based on the standard action template corresponding to the target sliding window, the joint angle formed by the key points in the sequence to be matched, and the weight value corresponding to the preset human body region to which the joint angle belongs, to obtain the second Pearson product moment between the joint angle formed by the key points in the sequence to be matched and the standard action template corresponding to the target sliding window.

[0077] The processing unit is specifically used to determine the matching result corresponding to the sequence to be matched based on the first Pearson product moment and the second Pearson product moment.

[0078] Thirdly, this disclosure provides an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to cause the electronic device to perform any of the action scoring methods provided in the first aspect above when executing the computer program.

[0079] Fourthly, this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement any of the action scoring methods provided in the first aspect above.

[0080] Fifthly, the present invention provides a computer program product that, when run on a computer, causes the computer to perform an action scoring method as provided in any of the first aspects.

[0081] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on the first computer-readable storage medium. The first computer-readable storage medium may be packaged together with the processor of the motion scoring device, or it may be packaged separately from the processor of the motion scoring device; this disclosure does not impose any limitations on this.

[0082] The descriptions of the second, third, fourth, and fifth aspects in this disclosure can be referenced to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second, third, fourth, and fifth aspects can be referenced to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0083] In this disclosure, the name of the aforementioned motion scoring device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear under other names. As long as the function of each device or functional module is similar to that of this disclosure, it falls within the scope of the claims of this disclosure and its equivalents.

[0084] These or other aspects of this disclosure will become more readily apparent in the following description.

[0085] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0086] This disclosure provides an action scoring method. First, it obtains the human keypoint sequence corresponding to the action to be scored on the target image, as well as the human keypoint template and human joint angle template corresponding to the action to be scored. The human keypoint template includes a standard human keypoint sequence, and the human joint angle template includes a standard human joint angle sequence. The Pearson matching degree between the standard human keypoint sequence in the human keypoint template and the human keypoint sequence corresponding to the action to be scored is determined as the keypoint Pearson matching degree. Similarly, the Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angles formed by the human keypoint sequence corresponding to the action to be scored is determined as the joint angle Pearson matching degree. Based on the differences between the corresponding joint angles formed by the standard human joint angle sequence and the human keypoint sequence, a joint angle difference similarity is determined. Finally, based on the keypoint Pearson matching degree, the joint angle Pearson matching degree, and the joint angle difference similarity, the action score for the action to be scored on the target image is determined. As can be seen, the embodiments of this disclosure determine the action score of the action to be scored on the target image by means of key point Pearson matching degree, joint angle Pearson matching degree, and joint angle difference similarity, thereby improving the accuracy of action scoring. Attached Figure Description

[0087] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0088] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0089] Figure 1 A scene architecture diagram of an action scoring method provided in this embodiment of the disclosure;

[0090] Figure 2 This is a schematic diagram of the display device in an action scoring method provided in an embodiment of the present disclosure;

[0091] Figure 3 This is a schematic diagram of the display device in another motion scoring method provided in an embodiment of the present disclosure;

[0092] Figure 4 This is a schematic diagram of an action scoring network architecture provided in an embodiment of the present disclosure;

[0093] Figure 5 A flowchart illustrating an action scoring method provided in an embodiment of this disclosure;

[0094] Figure 6 A flowchart illustrating another action scoring method provided in this embodiment of the disclosure;

[0095] Figure 7 This is a schematic diagram of the joint region in a motion scoring method provided in an embodiment of the present disclosure;

[0096] Figure 8 This is a schematic diagram of the double-chain Hungarian matching logic in an action scoring method provided in an embodiment of the present disclosure;

[0097] Figure 9 This is a schematic diagram of the structure of the display device 200 provided in an embodiment of the present disclosure;

[0098] Figure 10 This is a schematic diagram of a chip system provided in an embodiment of the present disclosure. Detailed Implementation

[0099] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0100] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0101] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0102] In this embodiment of the disclosure, YOLO stands for You Only Look Once, which is an object detection algorithm.

[0103] In this embodiment of the disclosure, the Single Shot MultiBox Detector refers to the Single Shot MultiBox Detector, abbreviated as SSD.

[0104] In this embodiment of the disclosure, Faster RCNN refers to Faster Region-Convolutional Neural Networks.

[0105] In this embodiment of the disclosure, the heatmap-based human skeleton point detection network HRNet refers to High-ResolutionNet.

[0106] In this embodiment of the disclosure, Simple Baseline refers to SBL, which provides a benchmark framework for human pose estimation.

[0107] In this embodiment of the disclosure, argmax is a function that evaluates the parameters (set) of a function.

[0108] The KM algorithm in this embodiment is a computer algorithm that is used to find the maximum weight matching under perfect matching.

[0109] Figure 1 A scene architecture diagram of an action scoring method provided in this disclosure embodiment is shown below. Figure 1 As shown, a user can operate the display device 200 via a mobile terminal 300 and a control device 100. The control device 100 can be a remote control, and communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired methods to control the display device 200. The user can input user commands through buttons on the remote control, voice input, control panel input, etc., to control the display device 200. In some embodiments, a mobile terminal, tablet computer, computer, laptop computer, and other smart devices can also be used to control the display device 200.

[0110] In some embodiments, the mobile terminal 300 can install software applications with the display device 200 to achieve connection and communication via network communication protocols, enabling one-to-one control operations and data communication. It can also transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronous display. The display device 200 can also communicate with the display device 200 via various communication methods. It can be allowed to communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The display device 200 can provide various content and interactive features. The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. In addition to providing broadcast television reception functions, the display device 200 can also be equipped with a smart network television function that provides computer support.

[0111] In some embodiments, the electronic device provided in this disclosure can be the display device 200 described above. When a user needs to exercise, they can turn on the display device 200 and instruct it to display a target video (such as a fitness video). The user can then perform corresponding fitness movements according to the prompts in the fitness video to achieve the desired fitness effect. During this process, an image acquisition device installed on the display device 200 will capture the user's fitness movements in real time, forming video frame data. The display device 200 processes the video frame data to determine at least one set of matching sequences corresponding to the human body. The display device 200 matches the sequences to be matched through a target sliding window, determines the matching result corresponding to each set of sequences to be matched, and determines whether the action to be scored is matched on the first image based on the matching result. If the action to be scored is successfully matched on the first image, the first image is determined as the target image, and the human key point sequence corresponding to the action to be scored on the target image is obtained. The display device 200 determines the Pearson matching degree between the standard human key point sequence in the human key point template and the human key point sequence corresponding to the action to be scored, as the key point Pearson matching degree, and determines the Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human key point sequence corresponding to the action to be scored, as the joint angle Pearson matching degree. Based on the difference between the joint angles that have a corresponding relationship between the standard human joint angle sequence and the human key point sequence, the joint angle difference similarity is determined. Based on the key point Pearson matching degree, the joint angle Pearson matching degree, and the joint angle difference similarity, the action score of the action to be scored on the target image is determined. In this way, the display device 200 can give an action score based on the action performed by the user according to the prompts, so that the user can determine the accuracy of the current action to be scored in a timely manner based on the action score.

[0112] Figure 2 This is a schematic diagram of the display device in an action scoring method provided in an embodiment of this disclosure. Figure 2The display device 200 shown includes at least one of the following: a tuner / demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to nth interface for input / output. The display 260 may be a touch-enabled display, such as a touch screen display. The tuner / demodulator 210 receives broadcast television signals via wired or wireless reception and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. The detector 230 is used to collect signals from the external environment or signals interacting with the external environment. The controller 250 and the tuner / demodulator 210 may be located in different separate devices; that is, the tuner / demodulator 210 may also be located in an external device of the main device containing the controller 250, such as an external set-top box.

[0113] In some embodiments, the image acquisition device may be a camera. The display device 200 may be equipped with at least one camera. The camera may be built into the display device 200, or it may be connected to the display device 200 via a wired or wireless connection. For example, the camera may be located at the lower edge of the display 260 of the display device 200. Of course, the location of the microphone on the display device 200 is not limited in this disclosure embodiment. Alternatively, the display device 200 may not include a camera, i.e., the aforementioned camera is not located within the display device 200. The display device 200 may connect an external camera via an interface (such as a USB interface 130). The external camera may be fixed to the display device 200 using an external fastener (such as a camera holder with a clip). For example, the external camera may be fixed to the edge of the display 260 of the display device 200, such as the upper edge, using an external fastener.

[0114] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0115] In some examples, the display device 200 of one or more embodiments is a television set 1, and the operating system of the television set 1 is the Android system, for example... Figure 3 As shown, TV 1 can be logically divided into an application layer (referred to as "application layer") 21, an application framework layer (referred to as "framework layer") 22, an Android runtime and system library layer (referred to as "system runtime library layer") 23, and a kernel layer 24.

[0116] The application layer 21 includes one or more applications. These applications can be system applications or third-party applications. For example, application layer 21 may include a first application that provides fitness video playback functionality. The framework layer 22 provides application programming interfaces (APIs) and programming frameworks for the applications in application layer 21. The system runtime library layer 23 provides support for the upper layer, namely the framework layer 22. When the framework layer 22 is used, the Android operating system runs the C / C++ libraries contained in the system runtime library layer 23 to implement the functions required by the framework layer 22. The kernel layer 24 acts as software middleware between the hardware layer and application layer 21, managing and controlling hardware and software resources.

[0117] In some examples, kernel layer 24 includes a first driver and a second driver. The first driver is used to send user operations collected by detector 230 to a first application, and the second driver is used to control display 260 to display the display information sent by display unit 213.

[0118] The first application on television set 1 is launched. Then, the first driver sends the user operation collected by detector 230 to the first application for recognition. Next, the processing unit 212 of the first application responds to the target operation received by acquisition unit 210 (e.g., selecting a target video in the first application), at which time the processing unit 212 controls the display unit 211 to display the target video. Then, the acquisition unit 210 acquires that the image acquisition device set on television set 1 will collect the user's fitness movements in real time, forming video frame data. Then, the processing unit 212 processes the video frame data acquired by acquisition unit 210 to determine at least one set of matching sequences corresponding to the human body. Then, the processing unit 212 processes the video frame data to determine at least one set of matching sequences corresponding to the human body. Processing unit 212 matches the sequences to be matched through a target sliding window, determines the matching result corresponding to each set of sequences to be matched, and determines whether the action to be scored is matched on the first image based on the matching result. If the action to be scored is successfully matched on the first image, the first image is determined as the target image, and the human key point sequence corresponding to the action to be scored on the target image is obtained. Processing unit 212 determines the Pearson matching degree between the standard human key point sequence in the human key point template and the human key point sequence corresponding to the action to be scored as the key point Pearson matching degree, and determines the Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human key point sequence corresponding to the action to be scored as the joint angle Pearson matching degree. Based on the difference between the joint angles that have a corresponding relationship between the standard human joint angle sequence and the human key point sequence, the joint angle difference similarity is determined. Based on the key point Pearson matching degree, the joint angle Pearson matching degree, and the joint angle difference similarity, the action score of the action to be scored on the target image is determined. In this way, the display device 200 can give an action score based on the action performed by the user according to the prompts, so that the user can determine the accuracy of the current action to be scored in a timely manner based on the action score.

[0119] Specifically, the electronic device provided in this embodiment may be the display 200 or the server 400 described above, and no limitation is made here.

[0120] Specifically, storage unit 213 is used to store the application program of the first application.

[0121] The video frame data involved in this application may be data authorized by the user or fully authorized by all parties.

[0122] In the following embodiments, the server 400 is used as the execution subject for the action scoring method provided in this disclosure embodiment to illustrate the method of this disclosure embodiment.

[0123] To clearly illustrate the embodiments of this disclosure, the following description is provided in conjunction with... Figure 4 A speech recognition network architecture provided by an embodiment of this disclosure is described.

[0124] See Figure 4 As shown, Figure 4 This is a schematic diagram of an action scoring network architecture provided in an embodiment of the present disclosure. Figure 4 In this system, the action scoring device receives input information and outputs the processing result of that information. The speech recognition module deploys a speech recognition service to convert audio into text; the semantic understanding module deploys a semantic understanding service to perform semantic parsing on the text; the business management module deploys a business instruction management service to provide business instructions; the language generation module deploys a language generation service (NLG) to convert instructions instructing the action scoring device into text; and the speech synthesis module deploys a text-to-speech (TTS) service to process the text corresponding to the instructions and send it to a speaker for playback. In one embodiment, Figure 4 The architecture shown can contain multiple entity service devices with different business services deployed, or one or more entity service devices can combine one or more functional services.

[0125] The motion scoring method provided in this disclosure can be applied to scenarios where users follow fitness videos displayed on a fitness mirror. Specifically, during the user's exercise, the image acquisition device captures the user's fitness movements in real time, forming video frame data. The video frame data is processed to determine at least one set of matching sequences corresponding to the human body. A target sliding window is used to match these sequences, determining the matching result for each set. Based on the matching result, it is determined whether a motion to be scored is matched on a first image. If a motion to be scored is successfully matched on the first image, the first image is designated as the target image. This allows the acquisition of the human keypoint sequence corresponding to the motion to be scored on the target image. Furthermore, the standard human keypoint sequence in the human keypoint template is compared with the motion to be scored. The Pearson matching degree between the corresponding human keypoint sequences is used as the keypoint Pearson matching degree. The Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human keypoint sequence corresponding to the action to be scored is used as the joint angle Pearson matching degree. The Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human keypoint sequence corresponding to the action to be scored is used as the joint angle Pearson matching degree. Then, based on the difference between the corresponding joint angles formed by the standard human joint angle sequence and the human keypoint sequence, the joint angle difference similarity is determined. Based on the keypoint Pearson matching degree, joint angle Pearson matching degree, and joint angle difference similarity, the action score of the action to be scored on the target image is determined, thereby improving the accuracy of action scoring.

[0126] This disclosure provides an action scoring method, referring to... Figure 5 This is a flowchart illustrating an action scoring method provided in an embodiment of the present disclosure. The method includes:

[0127] S501: Obtain the sequence of human key points corresponding to the action to be scored on the target image, as well as the human key point template and human joint angle template corresponding to the action to be scored.

[0128] The human body key point template includes a standard human body key point sequence, and the human body joint angle template includes a standard human body joint angle sequence.

[0129] In this embodiment of the disclosure, the target image can be any image containing the action to be scored. Accordingly, the sequence of human key points corresponding to the action to be scored on the target image can be obtained based on the division of the human skeleton and the division of the human joints.

[0130] For example, the human skeleton can be divided into 5 skeletal regions, such as: trunk, left upper limb, right upper limb, left lower limb, and right lower limb. The human joints can be divided into 12 joint regions, such as: left trunk angle, right trunk angle, left elbow angle, left arm angle, right elbow angle, right arm angle, left knee angle, left hip angle, left foot angle, right knee angle, right hip angle, and right foot angle.

[0131] The torso includes: left and right eyes, left and right ears, nose, left and right shoulders, left and right hips; the left upper limb includes: left palm, left wrist, left elbow and left shoulder; the right upper limb includes: right palm, right wrist, right elbow and right shoulder; the left lower limb includes: left hip, left knee and left ankle; and the right lower limb includes: right hip, right knee and right ankle.

[0132] That is, the sequence of key points of the human body can be a vector consisting of left and right eyes, left and right ears, nose, left and right shoulders, left and right hips, left palm, left wrist, left elbow, left shoulder, right palm, right wrist, right elbow, right shoulder, left hip, left knee, left ankle, right hip, right knee, and right ankle.

[0133] To more accurately describe the movements to be scored, the concept of joint angles is introduced. Similarly, joint angles can also be divided into five skeletal regions based on the distribution of bones. The trunk includes the left trunk angle and the right trunk angle; the left upper limb includes the left elbow angle and the left arm angle; the right upper limb includes the right elbow angle and the right arm angle; the left lower limb includes the left knee angle, the left hip angle, and the left foot angle; and the right lower limb includes the right knee angle, the right hip angle, and the right foot angle.

[0134] Accordingly, a standard human joint angle sequence may include a vector consisting of the left trunk angle, right trunk angle, left elbow angle, left arm angle, right elbow angle, right arm angle, left knee angle, left hip angle, left foot angle, right knee angle, right hip angle, and right foot angle.

[0135] In this embodiment of the disclosure, the human keypoint template corresponding to the action to be scored can be a standard human keypoint sequence corresponding to the action to be scored, which is used to perform Pearson matching with the human keypoint sequence corresponding to the action to be scored; the human joint angle template can be a standard human joint angle sequence corresponding to the action to be scored, which is used to perform Pearson matching with the joint angle formed by the human keypoint sequence corresponding to the action to be scored.

[0136] S502: Determine the Pearson matching degree between the standard human keypoint sequence in the human keypoint template and the human keypoint sequence corresponding to the action to be scored, and use it as the keypoint Pearson matching degree.

[0137] Among them, Pearson matching degree is used to measure the similarity between the standard human key point sequence in the human key point template and the human key point sequence corresponding to the action to be scored.

[0138] In one optional implementation, the weighted Pearson product moment of the standard human keypoint sequence in the normalized human keypoint template and the human keypoint sequence corresponding to the action to be scored is calculated as the keypoint Pearson matching degree.

[0139] The weight values ​​of the standard human key points in the standard human key point sequence and the human key points in the human key point sequence are determined based on the weight values ​​of the preset human body regions to which they belong.

[0140] In this embodiment of the disclosure, the standard human keypoint sequence in the human keypoint template and the human keypoint sequence corresponding to the action to be scored are normalized respectively to obtain the standard human keypoint sequence in the normalized human keypoint template and the human keypoint sequence corresponding to the action to be scored. Then, a weighted Pearson product moment calculation is performed on the standard human keypoint sequence in the normalized human keypoint template and the human keypoint sequence corresponding to the action to be scored, and the calculation result is used as the keypoint Pearson matching degree.

[0141] For example, normalization processing is performed on the standard human key point sequence in the human key point template.

[0142] In some examples, the standard human keypoint sequence in the human keypoint template can be normalized based on the theoretical detection box and the actual detection box. The standard human keypoint sequence in the human keypoint template can be input into the normalization formula to obtain the normalized standard human keypoint sequence in the human keypoint template.

[0143] The normalization formulas include:

[0144]

[0145] Among them, the nth standard human body key point S k_n The coordinates are equal to (x k_n y k_n The actual detection box B0 equals [B 0_xmin B 0_ymin B 0_w B 0_h ], r b0 Equal to the actual detection frame width B 0_w / Actual detection frame height B 0_h The nth normalized standard human key point S Nk_n The coordinates are equal to (x Nk_n y Nk_n ), B 0_xmin and B 0_ymin Let x and y be the coordinates of the top-left corner of the actual detection box B0 when it is mapped onto the target image, and a be the coordinates of the top-left corner of the actual detection box B0.s It is the normalization coefficient, and the theoretical detection box B1 = [B 1_xmin B 1_ymin B 1_w B 1_h ], a s =(a sx a sy ), a sx = Width B of the theoretical detection frame 1_w The width B of the actual detection frame 0_w The ratio, a sy = The theoretical detection frame height B 1_h The height B of the actual detection frame 0_h The ratio of .

[0146] Specifically, for the normalization processing of the human key point sequence corresponding to the action to be scored, the above-mentioned normalization processing of the standard human key point sequence in the human key point template can be referred to, and will not be repeated here in the embodiments of this disclosure.

[0147] S503: Determine the Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human key point sequence corresponding to the action to be scored, and use it as the joint angle Pearson matching degree.

[0148] In one optional implementation, the weighted Pearson product moment of the joint angle formed by the standard human joint angle sequence in the normalized human joint angle template and the human key point sequence corresponding to the action to be scored is calculated as the joint angle Pearson matching degree.

[0149] Among them, the weight value of the joint angle formed by the standard human joint angle and the human key point sequence in the standard human joint angle sequence is determined based on the weight value of the preset human body region to which it belongs.

[0150] Specifically, the normalization processing of the joint angles formed by the standard human joint angle sequence in the human joint angle template and the human key point sequence corresponding to the action to be scored can continue to refer to the above-mentioned normalization processing of the standard human key point sequence in the human key point template. This embodiment of the present disclosure will not be repeated here.

[0151] S504: Determine the similarity of joint angle differences based on the differences between joint angles that correspond to each other, which are formed by standard human joint angle sequences and human key point sequences.

[0152] In this embodiment of the disclosure, the similarity of the differences between each joint angle in the standard human joint angle sequence and the corresponding joint angle in the human key point sequence is determined.

[0153] S505: Determine the motion score of the action to be scored on the target image based on the Pearson matching degree of key points, the Pearson matching degree of joint angles, and the similarity of joint angle differences.

[0154] In one alternative implementation, a first score can be determined based on the joint angle difference similarity and the weight value of each joint angle, and a second score can be determined based on the key point Pearson matching degree and the joint angle Pearson matching degree. Then, based on the first score and the second score, the action score of the action to be scored on the target image can be determined.

[0155] Specifically, the first fraction can be determined using formula (1):

[0156]

[0157] Where S1 is the first fraction, N+1 is the number of joint angles formed by the sequence of human key points, and Δx i W represents the angle difference of the i-th joint angle among the corresponding joint angles formed by the standard human joint angle sequence and the human key point sequence. ai Let M be the weight value of the i-th joint angle. The weight value is determined based on the weight value of the preset human body region to which the i-th joint angle belongs. M is a preset first integer and P is a preset second integer.

[0158] For example, P can be 10, M can be 1, and N can be 11.

[0159] Use formula (2) to determine the second fraction:

[0160] S2=min{[Peark*Wpk+min(Peara, Peak)*Wpa], 1}; (2)

[0161] Where S2 is the second score, Peark is the keypoint Pearson matching degree, Wpk is the preset human keypoint Pearson product moment weight coefficient, Peara is the joint angle Pearson matching degree, and Wpa is the preset joint angle Pearson product moment weight coefficient.

[0162] The action score of the action to be scored on the target image is determined using formula (3):

[0163] S=w*(w1*S1+(1-w1)*S2); (3)

[0164] Where S is the action score of the action to be scored on the target image, w is the preset scoring coefficient corresponding to the action to be scored, and w1 is the preset scoring coefficient.

[0165] The action scoring method provided in this embodiment first obtains the human keypoint sequence corresponding to the action to be scored on the target image, as well as the human keypoint template and human joint angle template corresponding to the action to be scored. The human keypoint template includes a standard human keypoint sequence, and the human joint angle template includes a standard human joint angle sequence. The Pearson matching degree between the standard human keypoint sequence in the human keypoint template and the human keypoint sequence corresponding to the action to be scored is determined as the keypoint Pearson matching degree. The Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angles formed by the human keypoint sequence corresponding to the action to be scored is determined as the joint angle Pearson matching degree. Based on the difference between the corresponding joint angles formed by the standard human joint angle sequence and the human keypoint sequence, the joint angle difference similarity is determined. Based on the keypoint Pearson matching degree, the joint angle Pearson matching degree, and the joint angle difference similarity, the action score of the action to be scored on the target image is determined. As can be seen, the embodiments of this disclosure determine the action score of the action to be scored on the target image by means of key point Pearson matching degree, joint angle Pearson matching degree, and joint angle difference similarity, thereby improving the accuracy of action scoring.

[0166] Based on the above embodiments, this disclosure also provides an action scoring method, referring to... Figure 6 This is a flowchart illustrating another action scoring method provided in this embodiment of the disclosure, the method comprising:

[0167] S601: Identify the human body detection box on the first image and use it as the actual detection box.

[0168] The first image can be any frame of the user's image captured in real time by the image acquisition device.

[0169] In one optional implementation, the television 1 can project the first image onto the theoretical detection frame of the fitness movement corresponding to the current frame image to determine the image of the first image within the theoretical detection frame. Then, foreground and background separation is performed on the image within the theoretical detection frame to obtain the corresponding foreground. Image recognition is performed on the foreground to determine the human body contained within it. Then, by recognizing the human body, the smallest rectangle inscribed within the human body is determined, and this smallest rectangle is used as the actual detection frame corresponding to the human body.

[0170] In another optional implementation, the foreground and background of the first image are separated to obtain the foreground in the first image. Then, image recognition is performed on the foreground to determine the human body contained in the foreground. By recognizing the human body, the smallest rectangle that is inscribed in the human body is determined, and the smallest rectangle is used as the actual detection box corresponding to the human body.

[0171] In another optional implementation, the first image is input into a human detection network model, and after processing by the human detection network model, a human detection box on the first image is output.

[0172] The human detection network model was trained using a set of image samples labeled with detection boxes.

[0173] In one optional implementation, the size of the images in the image sample set may differ from the size of the first image. Therefore, when inputting the first image into the human detection network model, it is necessary to determine whether the size of the first image is the same as the size of the input image of the human detection network model. If it is determined that the size of the first image is the same as the size of the input image of the human detection network model, the first image can be directly input into the human detection network model; if it is determined that the size of the first image is different from the size of the input image of the human detection network model, the size of the first image needs to be converted to the same size as the input image of the human detection network model before the converted first image is input into the human detection network model.

[0174] Specifically, the training process of the human detection network model is as follows:

[0175] Obtain the training images and their labeling results.

[0176] The training image is input into a first object detection network to obtain the prediction result of the first object detection network for the training image. The first object detection network may include at least one of the object detection algorithms YOLO, SSD (Single-Shot Multiple-Frame Detector), and Faster R-CNN.

[0177] When the labeling results differ from the prediction results, the network parameters of the first target detection network are repeatedly adjusted until the first target detection network converges, thus obtaining the human body detection network model.

[0178] S602: Identify key points in the actual detection box and obtain the matching sequence corresponding to the actual detection box.

[0179] The sequence to be matched includes key points that have an order relationship.

[0180] In one optional implementation, the actual detection box is input into the human key point detection network model. After processing by the human key point detection network model, the human key points on the actual detection box are output. Based on the human key points on the actual detection box, the matching sequence corresponding to the actual detection box is generated.

[0181] The human keypoint detection network was trained using training sample images and training supervised images.

[0182] In this embodiment of the disclosure, after obtaining the actual detection box, the actual detection box can be input into the human key point detection network model. After the actual detection box is processed by the human key point detection network model, each human key point on the actual detection box is output. Then, based on each human key point on the output actual detection box, the matching sequence corresponding to the actual detection box is generated.

[0183] In this embodiment, the size of the actual detection box may differ from the size of the input image of the human keypoint detection network model. Therefore, when inputting the actual detection box into the human keypoint detection network model, it is necessary to determine whether the size of the actual detection box is the same as the size of the input image of the human keypoint detection network model. If it is determined that the size of the actual detection box is the same as the size of the input image of the human keypoint detection network model, the actual detection box can be directly input into the human keypoint detection network model. If it is determined that the size of the actual detection box is different from the size of the input image of the human keypoint detection network model, the size of the actual detection box needs to be converted to the same size as the input image of the human keypoint detection network model before the converted actual detection box is input into the human first target detection network model.

[0184] Specifically, the training process of the human keypoint detection network model is as follows:

[0185] First, training sample images and training supervision images are obtained. Both training sample images and training supervision images include human bodies and their corresponding key points. Then, a second object detection network is trained based on the training sample images to obtain a pre-trained second object detection network. Finally, the network parameters of the pre-trained second object detection network are adjusted based on the training supervision images to make the pre-trained second object detection network converge, thereby obtaining the human body key point detection network.

[0186] The second object detection network includes at least one of the following: HRNet, a heatmap-based human skeleton point detection network; SimpleBaseline; or a non-heatmap-based network with argmax as its output.

[0187] S603: Match the sequence to be matched using the target sliding window to obtain the matching result corresponding to the sequence to be matched.

[0188] The target sliding window is determined based on at least one preset standard action template.

[0189] The preset standard action template can be set according to requirements, and no limitation is made here in this embodiment.

[0190] Specifically, the target sliding window can consist of 'a' (e.g., a = 2) matching sliding windows (e.g., matching window 0 and matching window 1), and each matching sliding window contains a number of preset standard action templates greater than or equal to 'b' (e.g., b = 4). When matching the target sliding window with the sequence to be matched, it is necessary to match matching window 0 with the sequence to be matched, and match matching window 1 with the sequence to be matched, respectively, thereby improving the error tolerance and robustness of the action scoring method.

[0191] The matching sliding window includes a sliding window state variable, a sliding window history state variable, a sliding window template vector, and a sliding window timer. The format of the sliding window state variable is [[0, scorek0, scorea0], [1, scorek1, scorea1], ..., [m-1, scorek0 ... m-1 scorea m-1 The first parameter, m, represents the template index, scorek. m Scorea represents the total score of keypoints matched by the m-th standard action template. m This represents the total score of the key angle matching the m-th preset standard action template.

[0192] Where a and b are both integers greater than or equal to 0, and m is an integer greater than or equal to 1.

[0193] In one optional implementation, weighted bipartite graph matching can be performed based on the standard motion template corresponding to the target sliding window, the key points in the sequence to be matched, and the weight values ​​corresponding to the preset human body regions to which the key points belong, to obtain the first Pearson product moment between the key points in the sequence to be matched and the standard motion template corresponding to the target sliding window. Then, weighted bipartite graph matching can be performed based on the standard motion template corresponding to the target sliding window, the joint angles formed by the key points in the sequence to be matched, and the weight values ​​corresponding to the preset human body regions to which the joint angles belong, to obtain the second Pearson product moment between the joint angles formed by the key points in the sequence to be matched and the standard motion template corresponding to the target sliding window. The matching result corresponding to the sequence to be matched is determined based on the first and second Pearson product moments.

[0194] In this embodiment, a standard action template can be used as the left vertex X, and the sequence to be matched can be used as the right vertex Y. Weighted bipartite graph matching is then performed using the weight values ​​corresponding to the preset human body regions to which the key points belong, so that each group of left and right links X i Y j The sum of the Pearson product moments is maximized. Finally, edges with matching edge weights less than the template matching threshold are removed. The first Pearson product moment of the key points in the sequence to be matched and the standard action template corresponding to the target sliding window is obtained.

[0195] The preset human body area can be obtained by dividing the human body into skeletal and joint regions.

[0196] Specifically, the distribution of the human skeleton can be divided into at least one skeletal region, and the distribution of the human joints can be divided into at least one joint region.

[0197] For example, the human skeleton can be divided into 5 skeletal regions, such as: trunk, left upper limb, right upper limb, left lower limb, and right lower limb. The human joints can be divided into 12 joint regions, such as: left trunk angle, right trunk angle, left elbow angle, left arm angle, right elbow angle, right arm angle, left knee angle, left hip angle, left foot angle, right knee angle, right hip angle, and right foot angle.

[0198] The torso includes: left and right eyes, left and right ears, nose, left and right shoulders, left and right hips; the left upper limb includes: left palm, left wrist, left elbow and left shoulder; the right upper limb includes: right palm, right wrist, right elbow and right shoulder; the left lower limb includes: left hip, left knee and left ankle; and the right lower limb includes: right hip, right knee and right ankle.

[0199] Based on the different body parts worked by different fitness movements in the standard movement template, different weight values ​​were assigned to the five skeletal regions mentioned above. Therefore, when matching key points in the sequence to be matched with theoretical key points in the standard movement template, the weight value corresponding to the key point in the sequence to be matched can be determined to be equal to the weight value of the skeletal region to which the key point in the sequence to be matched belongs, based on the skeletal region to which the key point in the sequence to be matched belongs.

[0200] In addition, to more accurately describe the fitness movements of the standard movement template, the concept of joint angles is introduced. There are 12 joint angles in total, including A0 left trunk angle, A1 right trunk angle, A2 left elbow angle, A3 left arm angle, A4 right elbow angle, A5 right arm angle, A6 left knee angle, A7 left hip angle, A8 left foot angle, A9 right knee angle, A10 right hip angle, and A11 right foot angle.

[0201] Similarly, the joint angles can also be divided into five skeletal regions based on the distribution of bones. The trunk includes: A0 (left trunk angle) and A1 (right trunk angle); the left upper limb includes: A2 (left elbow angle) and A3 (left arm angle); the right upper limb includes: A4 (right elbow angle) and A5 (right arm angle); the left lower limb includes: A6 (left knee angle), A7 (left hip angle), and A8 (left foot angle); and the right lower limb includes: A9 (right knee angle), A10 (right hip angle), and A11 (right foot angle). Specific angles are as follows: Figure 7 As shown. Therefore, when matching key points in the sequence to be matched with theoretical key points in the standard motion template, the weight value of the key point in the sequence to be matched can be determined to be equal to the weight value of the bone region corresponding to the joint region to which the key point in the sequence to be matched belongs, based on the bone region to which the key point in the sequence to be matched belongs.

[0202] In this embodiment, a standard motion template can be used as the left vertex X, and the sequence to be matched can be used as the right vertex Y. Weighted bipartite graph matching is then performed using the weight values ​​corresponding to the preset human body regions to which the joint angles belong, so that each group of left and right links X i Y j The sum of the Pearson product moments is maximized. Finally, edges with matching edge weights less than the template matching threshold are removed to obtain the second Pearson product moment of the joint angles formed by the key points in the sequence to be matched and the standard action template corresponding to the target sliding window.

[0203] Specifically, after obtaining the first Pearson product moment and the second Pearson product moment, the key point score of the sequence to be matched can be determined based on the first Pearson product moment, the joint angle score formed by the key points in the sequence to be matched can be determined based on the second Pearson product moment, and then the matching result corresponding to the sequence to be matched can be determined based on the key point score of the sequence to be matched and the joint angle score formed by the key points in the sequence to be matched.

[0204] In this embodiment of the disclosure, the first average value of the first Pearson product moment corresponding to the key point in the sequence to be matched is determined based on the first Pearson product moment of the standard action template corresponding to the key point in the sequence to be matched and the target sliding window. Then, the second average value corresponding to the sequence to be matched is determined based on the first average value of the first Pearson product moment corresponding to the key point in the sequence to be matched. The second average value corresponding to the sequence to be matched is used as the key point score of the sequence to be matched.

[0205] In this embodiment of the disclosure, the third average value of the second Pearson product moment corresponding to the joint angle formed by the key points in the sequence to be matched is determined based on the second Pearson product moment of the standard action template corresponding to the target sliding window. Then, the fourth average value corresponding to the sequence to be matched is determined based on the third average value of the second Pearson product moment corresponding to the joint angle formed by the key points in the sequence to be matched. The fourth average value corresponding to the sequence to be matched is used as the joint angle score of the joint angle formed by the key points in the sequence to be matched.

[0206] In one optional implementation, the matching result corresponding to the sequence to be matched also includes the matching degree.

[0207] The matching degree can be characterized by the Pearson product moment of the total score of key points in the sequence to be matched and the total score of joint angles formed by key points in the sequence to be matched.

[0208] Specifically, when matching sequences based on a target sliding window, it is necessary to statistically analyze the matching results for each set of sequences. Since the differences in matching results between adjacent sequences are small, update logic can be set for the matching results. For example, if, when the current sequence is being matched with the matching window, the matching degree of any preset standard action template in the matching window is greater than the matching degree of that preset standard action template with the previous consecutive adjacent sequence, and the matching degree of all preset standard action templates in the matching window before that preset standard action template is greater than the template matching threshold (e.g., 0.5), then the matching degree, key point score, and key angle score of the sequence being matched are updated. If the matching degree of the first c (e.g., c equals 2) preset standard action templates in the matching window is greater than the template matching threshold, then the matching window is updated, the first two standard action templates in the matching window are popped, and d (e.g., d equals 2) preset standard action templates other than those included in the matching window are added to the end of the queue. If there are only two preset standard action templates in the matching slide window, and the matching degree of each preset standard action template is greater than the template matching threshold, then the slide window matching is successful.

[0209] Where c and d are both integers greater than or equal to 0.

[0210] Specifically, when matching window 0 with the sequence to be matched, and when matching window 1 with the sequence to be matched, the corresponding doubly-chained Hungarian matching logic is as follows: Figure 8 As shown.

[0211] Step 1: First, at time t0, obtain the key point SNk_t0 of the current sequence to be matched, and push the key point SNk_t0 of the current sequence to be matched onto the key point buffer.

[0212] Step 2: Compare the number of cached sequences to be matched with the matching window length. When the number of cached sequences to be matched is greater than or equal to the matching window length nlen (e.g., 6), retrieve the keypoints from the 6 sequences to be matched from the keypoint buffers to form the sequence Seqs to be matched, i.e., Seqs = [SNk_t0, SNk_t1, SNk_t2, SNk_3, SNk_4, SNk_5]. Also, retrieve n (e.g., 6) preset standard action templates from the Action templates according to the order of the sequences to be matched, forming matching window 0 and matching window 1.

[0213] Wherein, matching sliding window 0 (SeqT0) = matching sliding window 1 (SeqT1) = [SNTk_0,SNTk_1,SNTk_2,SNTk_3,SNTk_4,SNTk_5]), where SNTk_0 represents the preset standard action template corresponding to the 0th sequence to be matched.

[0214] Step 3: Perform a timeout check. If the timer 0 of the matching sliding window 0 has not timed out, proceed to (Step 4). If the sliding window timer 0 times out, check if the sliding window timer 1 is enabled. If the sliding window timer 1 is enabled, assign all the values ​​of the sliding window template SeqT1 to the sliding window template SeqT0, reset the sliding window template SeqT1, and clear the sliding window timer 1. If the sliding window timer 1 is not enabled, reset the sliding window template SeqT0 and clear the sliding window timer 0.

[0215] Step 4: When the 0th template in SeqT0 is equal to SNTk_0, the sequence to be matched Seqs and SeqT0 are matched based on the KM algorithm, and SeqT0 is updated after the match is obtained.

[0216] Step 5: After SeqT0 is updated, if the state of the matching sliding window 0 changes, the sliding window timer 0 is cleared and the timer is started.

[0217] Step 6: When the 0th template in SeqT0 is not equal to SNTk_0, first check the state of sliding window timer 1. If sliding window timer 1 times out, reset the matching sliding window 1 and clear sliding window timer 1. If sliding window timer 1 does not time out, use the KM algorithm to match the sequences to be matched, Seqs, SeqT0, and SeqT1 respectively. After matching, update SeqT0 and SeqT1 using the same update strategy as in (Step 4).

[0218] Step 7: After SeqT0 and SeqT1 are updated, if the state of matching window 0 or matching window 1 changes, then clear window timer 0 and window timer 1 respectively and start timing. If the number of remaining preset standard templates in matching window 0 or matching window 1 is 0, it means that the matching window has matched successfully. Therefore, it is necessary to first check matching window 0. If it matches successfully, then pass the value of matching window 1 to 0 and give a prompt: "Matching window 0 matched successfully." If matching window 0 does not match successfully, but matching window 1 matches successfully, then reset matching window 0, matching window 1, and related status registers and timers, and send a prompt: "Matching window 1 matched successfully."

[0219] In one optional implementation, after obtaining the keypoint scores of the sequence to be matched and the joint angle scores formed by the keypoints in the sequence to be matched, the total keypoint score of the sequence to be matched and the total joint angle score formed by the keypoints in the sequence to be matched can be determined based on the keypoint scores of the sequence to be matched and the joint angle scores formed by the keypoints in the sequence to be matched.

[0220] In this embodiment of the disclosure, the total score of the key points of the sequence to be matched can be the average score of each key point in the sequence to be matched; the total score of the key points of the sequence to be matched can also be the maximum score of each key point in the sequence to be matched.

[0221] In this embodiment of the disclosure, the total score of the joint angle formed by the key points in the sequence to be matched can be the average score of each joint angle formed by the key points in the sequence to be matched; the total score of the joint angle formed by the key points in the sequence to be matched can also be the maximum score of the key points in the joint angle formed by the key points in the sequence to be matched.

[0222] After determining the total score of key points in the sequence to be matched and the total score of joint angles formed by key points in the sequence to be matched, the scoring result of the sequence to be matched is determined based on the total score of key points in the sequence to be matched and the total score of joint angles formed by key points in the sequence to be matched, and then the matching result corresponding to the sequence to be matched is determined.

[0223] In one optional implementation, the scoring result of the sequence to be matched can be determined based on the correspondence between the total score of key points, the total score of joint angles formed by key points in the sequence to be matched, and the scoring result of the sequence to be matched. This leads to the determination of the matching result corresponding to the sequence to be matched.

[0224] In this embodiment of the disclosure, if the score result of the sequence to be matched is greater than or equal to a preset threshold, the matching result corresponding to the sequence to be matched is determined to be a successful match; if the score result of the sequence to be matched is less than the preset threshold, the matching result corresponding to the sequence to be matched is determined to be a failed match.

[0225] S604: Based on the matching results corresponding to the sequence to be matched, determine whether the action to be scored is matched on the first image.

[0226] In this embodiment of the disclosure, if the matching result corresponding to the sequence to be matched is a successful match, it is determined that the action to be scored is matched on the first image; if the matching result corresponding to the sequence to be matched is a failed match, it is determined that the action to be scored is not matched on the first image.

[0227] S605: If it is determined that the action to be scored is successfully matched on the first image, then the first image is determined as the target image.

[0228] In this embodiment of the disclosure, if the matching result corresponding to the sequence to be matched is a successful match, then it is determined that the action to be scored has been successfully matched on the first image, and the first image is determined as the target image.

[0229] S606: Obtain the sequence of human key points corresponding to the action to be scored on the target image, as well as the human key point template and human joint angle template corresponding to the action to be scored.

[0230] The human body key point template includes a standard human body key point sequence, and the human body joint angle template includes a standard human body joint angle sequence.

[0231] It should be noted that step S606 is the same as step S501, and the specific details can be found in the description of step S501.

[0232] S607: Determine the Pearson matching degree between the standard human keypoint sequence in the human keypoint template and the human keypoint sequence corresponding to the action to be scored, and use it as the keypoint Pearson matching degree.

[0233] It should be noted that step S607 is the same as step S502, and the specific details are described in the description of step S502.

[0234] S608: Determine the Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human key point sequence corresponding to the action to be scored, and use it as the joint angle Pearson matching degree.

[0235] It should be noted that step S608 is the same as step S503, and the specific details can be found in the description of step S503.

[0236] S609: Determine the similarity of joint angle differences based on the differences between joint angles that correspond to each other, which are formed by standard human joint angle sequences and human key point sequences.

[0237] It should be noted that step S609 is the same as step S504, and the specific details are described in the description of step S504.

[0238] S610: Determine the action score of the action to be scored on the target image based on the Pearson matching degree of key points, the Pearson matching degree of joint angles, and the similarity of joint angle differences.

[0239] It should be noted that step S610 is the same as step S505, and the specific details are described in the description of step S505.

[0240] In the action scoring method provided in this embodiment, firstly, a human body detection box on a first image is identified as the actual detection box. Key points within the actual detection box are then identified to obtain the matching sequence corresponding to the actual detection box. The matching sequence includes key points with an ordered relationship. A target sliding window is used to match the matching sequence to obtain the matching result. The target sliding window is determined based on at least one preset standard action template. Based on the matching result corresponding to the matching sequence, it is determined whether the action to be scored is matched on the first image. If the action to be scored is successfully matched on the first image, the first image is determined as the target image. The sequence of human key points corresponding to the action to be scored on the target image, as well as the human key point template and human joint angle template corresponding to the action to be scored, are obtained. In the process, the human keypoint template includes a standard human keypoint sequence, and the human joint angle template includes a standard human joint angle sequence. The Pearson matching degree between the standard human keypoint sequence in the human keypoint template and the human keypoint sequence corresponding to the action to be scored is determined as the keypoint Pearson matching degree. The Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human keypoint sequence corresponding to the action to be scored is determined as the joint angle Pearson matching degree. Based on the difference between the corresponding joint angles formed by the standard human joint angle sequence and the human keypoint sequence, the joint angle difference similarity is determined. Based on the keypoint Pearson matching degree, the joint angle Pearson matching degree, and the joint angle difference similarity, the action score of the action to be scored on the target image is determined. As can be seen, after successfully matching the action to be scored on the first image in this embodiment, the first image is determined as the target image, the sequence of human key points corresponding to the action to be scored on the target image is obtained, and the action score of the action to be scored on the target image is determined by means of key point Pearson matching degree, joint angle Pearson matching degree and joint angle difference similarity, thereby improving the accuracy of action scoring.

[0241] It should be noted that the above example uses the television set 1 as the executing entity for the action scoring method provided in this embodiment of the present disclosure. In other examples, the executing entity for the action scoring method provided in this embodiment of the present disclosure may also be the server 400. For example, when a user needs to exercise, they can turn on the display device 200 and instruct it to display a target video (such as a fitness video). The user can then perform the corresponding fitness movements according to the prompts in the fitness video to achieve the desired fitness effect. During this period, the image acquisition device installed on the display device 200 will collect the user's fitness movements in real time, forming video frame data. The television set 1 sends this video frame data to the server 400. The server 400 processes the video frame data and determines at least one set of matching sequences corresponding to the human body. Server 400 uses a target sliding window to match the sequences to be matched, and determines the matching result for each set of sequences to be matched. Based on the matching result, Server 400 determines whether the action to be scored is matched on the first image. If the action to be scored is successfully matched on the first image, the first image is determined as the target image. Server 400 obtains the human keypoint sequence corresponding to the action to be scored on the target image. Server 400 determines the Pearson matching degree between the standard human keypoint sequence in the human keypoint template and the human keypoint sequence corresponding to the action to be scored, as the keypoint Pearson matching degree. Server 400 also determines the Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human keypoint sequence corresponding to the action to be scored, as the joint angle Pearson matching degree. Based on the difference between the joint angles that have a corresponding relationship between the standard human joint angle sequence and the human keypoint sequence, the joint angle difference similarity is determined. Based on the keypoint Pearson matching degree, the joint angle Pearson matching degree, and the joint angle difference similarity, the action score of the action to be scored on the target image is determined. Server 400 sends display information carrying the action score to TV 1, so that TV 1 can display the action score in the display information after receiving the display information sent by server 400.

[0242] The foregoing primarily describes the solutions provided by the embodiments of this disclosure from a methodological perspective. To achieve the aforementioned functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0243] This disclosure embodiment can divide the action scoring device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing unit. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this disclosure embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0244] like Figure 9 The diagram shown is a structural schematic of a display device 200 provided in an embodiment of this disclosure. It includes a communicator 101, a display 102, and a processor 103.

[0245] The communicator 101 is used to acquire the sequence of human key points corresponding to the action to be scored on the target image captured by the image acquisition device when the display 102 plays the target video, as well as the human key point template and human joint angle template corresponding to the action to be scored; wherein, the human key point template includes a standard human key point sequence, and the human joint angle template includes a standard human joint angle sequence.

[0246] Processor 103 is used to determine the Pearson matching degree between the standard human key point sequence in the human key point template and the human key point sequence corresponding to the action to be scored, as the key point Pearson matching degree.

[0247] The processor 103 is further configured to determine the Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human key point sequence corresponding to the action to be scored, as the joint angle Pearson matching degree.

[0248] The processor 103 is also used to determine the joint angle difference similarity based on the difference between the corresponding joint angles formed by the standard human joint angle sequence and the human key point sequence.

[0249] The processor 103 is further configured to determine the action score of the action to be scored on the target image based on the key point Pearson matching degree, the joint angle Pearson matching degree, and the joint angle difference similarity.

[0250] In one optional implementation, the processor 103 is specifically used to calculate the weighted Pearson product moment of the standard human keypoint sequence in the normalized human keypoint template and the human keypoint sequence corresponding to the action to be scored, as the keypoint Pearson matching degree; wherein, the weight values ​​of the standard human keypoint in the standard human keypoint sequence and the human keypoint in the human keypoint sequence are determined based on the weight values ​​of the preset human body regions to which they belong.

[0251] In one optional implementation, the processor 103 is specifically used to calculate the weighted Pearson product moment of the joint angle formed by the standard human joint angle sequence in the normalized human joint angle template and the human key point sequence corresponding to the action to be scored, as the joint angle Pearson matching degree; wherein, the weight value of the joint angle formed by the standard human joint angle in the standard human joint angle sequence and the human key point sequence is determined based on the weight value of the preset human body region to which it belongs.

[0252] In one optional implementation, the processor 103 is specifically configured to determine a first score based on the joint angle difference similarity and the weight values ​​of each joint angle;

[0253] Processor 103 is specifically used to determine a second score based on the key point Pearson matching degree and the joint angle Pearson matching degree;

[0254] The processor 103 is specifically configured to determine the action score of the action to be scored on the target image based on the first score and the second score.

[0255] In one alternative implementation, the processing unit is specifically configured to determine a first fraction using formula (1):

[0256]

[0257] Where S1 is the first fraction, N+1 is the number of joint angles formed by the sequence of human key points, and Δx i W is the angle difference of the i-th joint angle among the corresponding joint angles formed by the standard human joint angle sequence and the human key point sequence. ai The weight value of the i-th joint angle is determined based on the weight value of the preset human body region to which the i-th joint angle belongs, where M is a preset first integer and P is a preset second integer.

[0258] In one alternative implementation, processor 103 is specifically configured to determine the second fraction using formula (2):

[0259] S2=min{[Peark*Wpk+min(Peara, Peak)*Wpa], 1}; (2)

[0260] Wherein, S2 is the second score, Peark is the key point Pearson matching degree, Wpk is the preset human key point Pearson product moment weight coefficient, Peara is the joint angle Pearson matching degree, and Wpa is the preset joint angle Pearson product moment weight coefficient.

[0261] In one optional implementation, the processor 103 is specifically configured to determine the action score of the action to be scored on the target image using formula (3):

[0262] S=w*(w1*S1+(1-w1)*S2); (3)

[0263] Where S is the action score of the action to be scored on the target image, w is the preset scoring coefficient corresponding to the action to be scored, and w1 is the preset scoring coefficient.

[0264] In one optional implementation, the processor 103 is specifically configured to identify human body detection boxes on the first image as actual detection boxes;

[0265] The processor 103 is specifically used to identify key points in the actual detection frame and obtain the matching sequence corresponding to the actual detection frame; wherein, the matching sequence includes key points with an ordered relationship;

[0266] The processor 103 is specifically configured to match the sequence to be matched through a target sliding window to obtain a matching result corresponding to the sequence to be matched; wherein the target sliding window is determined based on at least one preset standard action template;

[0267] The processor 103 is specifically used to determine whether the action to be scored is matched on the first image based on the matching result corresponding to the sequence to be matched;

[0268] The processor 103 is specifically configured to determine the first image as the target image if it is determined that an action to be scored is successfully matched on the first image.

[0269] In one optional implementation, the processor 103 is specifically configured to input the target image into a human detection network model, and after processing by the human detection network model, output human detection boxes on the target image; wherein, the human detection network model is trained using an image sample set labeled with detection boxes.

[0270] In one optional implementation, the processor 103 is specifically configured to input the actual detection box into the human keypoint detection network model, and after processing by the human keypoint detection network model, output the human keypoints on the actual detection box; wherein, the human keypoint detection network is trained using training sample images and training supervision images.

[0271] The processor 103 is specifically used to generate a matching sequence corresponding to the actual detection box based on the human body key points on the actual detection box.

[0272] In one optional implementation, the processor 103 is specifically used to perform weighted bipartite graph matching based on the standard action template corresponding to the target sliding window, the key points in the sequence to be matched, and the weight values ​​corresponding to the preset human body regions to which the key points belong, to obtain the first Pearson product moment between the key points in the sequence to be matched and the standard action template corresponding to the target sliding window.

[0273] The processor 103 is specifically used to perform weighted bipartite graph matching based on the standard action template corresponding to the target sliding window, the joint angle formed by the key points in the sequence to be matched, and the weight value corresponding to the preset human body region to which the joint angle belongs, to obtain the second Pearson product moment between the joint angle formed by the key points in the sequence to be matched and the standard action template corresponding to the target sliding window.

[0274] The processor 103 is specifically configured to determine the matching result corresponding to the sequence to be matched based on the first Pearson product moment and the second Pearson product moment.

[0275] All relevant content of each step involved in the above method embodiments can be applied to the functional description of the corresponding functional module, and its role will not be repeated here.

[0276] Of course, the display device 200 provided in this embodiment includes, but is not limited to, the modules described above. For example, the display device 200 may also include a memory 104. The memory 104 may be used to store the program code of the display device 200, and may also be used to store data generated by the display device 200 during operation, such as data in write requests.

[0277] As an example, combined Figure 3 The functions implemented by the acquisition unit 210 in the server 400 are the same as those implemented by the communicator 101; the functions implemented by the display unit 211 are the same as those implemented by the display 102; the functions implemented by the processing unit 212 are the same as those implemented by the processor 103; and the functions implemented by the storage unit 213 are the same as those implemented by the memory 104.

[0278] This disclosure also provides a chip system, with reference to... Figure 10This diagram illustrates a chip system provided in an embodiment of the present disclosure. This chip system can be applied to the display device 200 in the foregoing embodiments. The chip system includes at least one processor 1001 and at least one interface circuit 1002. The processor 1001 may be the processor in the aforementioned server 400. The processor 1001 and the interface circuit 1002 are interconnected via a line. The processor 1001 can receive and execute computer instructions from the memory of the display device 200 through the interface circuit 1002. When the computer instructions are executed by the processor 1001, the display device 200 can perform the various steps performed by the display device 200 in the foregoing embodiments. Of course, the chip system may also include other discrete components, which are not specifically limited in this embodiment.

[0279] This disclosure also provides a computer-readable storage medium for storing computer instructions executed by the display device 200.

[0280] This disclosure also provides a computer program product, including computer instructions executed by the above-described display device 200.

[0281] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of action scoring, characterized by, include: Obtain the sequence of human key points corresponding to the action to be scored on the target image, as well as the human key point template and human joint angle template corresponding to the action to be scored; wherein, the human key point template includes a standard human key point sequence, and the human joint angle template includes a standard human joint angle sequence. The Pearson matching degree between the standard human key point sequence in the human key point template and the human key point sequence corresponding to the action to be scored is determined as the key point Pearson matching degree. In addition, the Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human key point sequence corresponding to the action to be scored is determined as the joint angle Pearson matching degree. Based on the differences between the joint angles that correspond to each other in the standard human joint angle sequence and the human key point sequence, the joint angle difference similarity is determined. The motion score of the action to be scored on the target image is determined based on the key point Pearson matching degree, the joint angle Pearson matching degree, and the joint angle difference similarity. The step of determining the action score of the action to be scored on the target image based on the keypoint Pearson matching degree, the joint angle Pearson matching degree, and the joint angle difference similarity includes: Based on the similarity of the joint angle difference and the weight value of each joint angle, a first score is determined; And, the second fraction is determined using formula (2): ； （2） wherein, is the second score, is the key point Pearson match degree, is a preset human key point Pearson product moment weight coefficient, is the joint angle Pearson match degree; is a preset joint angle Pearson product moment weight coefficient; Based on the first score and the second score, the action score of the action to be scored on the target image is determined.

2. The method of claim 1, wherein, The step of determining the Pearson match degree between the standard human keypoint sequence in the human keypoint template and the human keypoint sequence corresponding to the action to be scored, as the keypoint Pearson match degree, includes: The weighted Pearson product moment of the standard human key point sequence in the normalized human key point template and the human key point sequence corresponding to the action to be scored is calculated as the key point Pearson matching degree; wherein, the weight values ​​of the standard human key points in the standard human key point sequence and the human key points in the human key point sequence are determined based on the weight values ​​of the preset human body regions to which they belong.

3. The method of claim 1, wherein, The step of determining the Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human key point sequence corresponding to the action to be scored, as the joint angle Pearson matching degree, includes: The weighted Pearson product moment of the joint angle formed by the standard human joint angle sequence in the normalized human joint angle template and the human key point sequence corresponding to the action to be scored is calculated as the joint angle Pearson matching degree; wherein, the weight value of the joint angle formed by the standard human joint angle in the standard human joint angle sequence and the human key point sequence is determined based on the weight value of the preset human region to which it belongs.

4. The method of claim 1, wherein, The determination of the first score based on the similarity of the joint angle differences and the weight values ​​of each joint angle includes: Determine the first fraction using formula (1): ； （1） in, The first score, +1 represents the number of joint angles formed by the sequence of key human body points. The joint angles that have a corresponding relationship between the standard human joint angle sequence and the human key point sequence are the first joint angles. The angle difference of each joint angle For the first The weight value of the joint angle, the weight value being based on the weight of the joint angle. The weight values ​​of the preset human body regions to which each joint angle belongs are determined, where M is a preset first integer and P is a preset second integer.

5. The method of claim 1, wherein, The step of determining the action score of the action to be scored on the target image based on the first score and the second score includes: The action score of the action to be scored on the target image is determined using formula (3): ； （3） wherein, a motion score of the to-be-scored motion on the target image, a preset scoring coefficient corresponding to the to-be-scored motion, a preset scoring coefficient.

6. The method of claim 1, wherein, Before obtaining the sequence of human key points corresponding to the action to be scored on the target image, the method further includes: Identify the human body detection bounding box on the first image and use it as the actual detection box; Identify key points in the actual detection frame to obtain the matching sequence corresponding to the actual detection frame; wherein, the matching sequence includes key points with an ordered relationship; The sequence to be matched is matched by a target sliding window to obtain the matching result corresponding to the sequence to be matched; wherein, the target sliding window is determined based on at least one preset standard action template; Based on the matching results corresponding to the sequence to be matched, determine whether the action to be scored is matched on the first image; If it is determined that the action to be scored is successfully matched on the first image, then the first image is determined as the target image.

7. The method of claim 6, wherein, The identification of human body detection boxes on the first image, as actual detection boxes, includes: The first image is input into a human detection network model, and after processing by the human detection network model, a human detection box on the first image is output; wherein, the human detection network model is trained using an image sample set marked with detection boxes.

8. The method of claim 6, wherein, The step of identifying key human body points within the actual detection frame and obtaining the matching sequence corresponding to the actual detection frame includes: The actual detection box is input into the human keypoint detection network model. After processing by the human keypoint detection network model, the human keypoints on the actual detection box are output. The human keypoint detection network is trained using training sample images and training supervised images. Based on the human body key points on the actual detection box, a matching sequence corresponding to the actual detection box is generated.

9. The method of claim 6, wherein, The step of matching the sequence to be matched through a target sliding window to obtain the matching result corresponding to the sequence to be matched includes: Weighted bipartite graph matching is performed based on the standard action template corresponding to the target sliding window, the key points in the sequence to be matched, and the weight values ​​corresponding to the preset human body regions to which the key points belong, to obtain the first Pearson product moment between the key points in the sequence to be matched and the standard action template corresponding to the target sliding window. Based on the standard action template corresponding to the target sliding window, the joint angle formed by the key points in the sequence to be matched, and the weight value corresponding to the preset human body region to which the joint angle belongs, weighted bipartite graph matching is performed to obtain the second Pearson product moment between the joint angle formed by the key points in the sequence to be matched and the standard action template corresponding to the target sliding window. The matching result corresponding to the sequence to be matched is determined based on the first Pearson product moment and the second Pearson product moment.

10. An action scoring apparatus, characterized by comprising: include: The acquisition unit is used to acquire the sequence of human key points corresponding to the action to be scored on the target image, as well as the human key point template and human joint angle template corresponding to the action to be scored; wherein, the human key point template includes a standard human key point sequence, and the human joint angle template includes a standard human joint angle sequence. The processing unit is used to determine the Pearson matching degree between the standard human key point sequence in the human key point template and the human key point sequence corresponding to the action to be scored, as the key point Pearson matching degree. The processing unit is also used to determine the Pearson matching degree between the standard human joint angle sequence in the human joint angle template and the joint angle formed by the human key point sequence corresponding to the action to be scored, as the joint angle Pearson matching degree. The processing unit is also used to determine the joint angle difference similarity based on the difference between the corresponding joint angles formed by the standard human joint angle sequence and the human key point sequence. The processing unit is further configured to determine the action score of the action to be scored on the target image based on the key point Pearson matching degree, the joint angle Pearson matching degree, and the joint angle difference similarity. The processing unit is specifically used for: Based on the similarity of the joint angle difference and the weight value of each joint angle, a first score is determined; And, the second fraction is determined using formula (2): ； （2） wherein, is the second score, is the key point Pearson match degree, is a preset human key point Pearson product moment weight coefficient, is the joint angle Pearson match degree; is a preset joint angle Pearson product moment weight coefficient; Based on the first score and the second score, the action score of the action to be scored on the target image is determined.

11. An electronic device, comprising: include: A memory and a processor, the memory being used to store a computer program; the processor being used to cause the electronic device to implement the action scoring method according to any one of claims 1-9 when executing the computer program.

12. A computer-readable storage medium, characterized in that, include: The computer-readable storage medium stores a computer program that, when executed by a computing device, causes the computing device to implement the action scoring method according to any one of claims 1-9.

Images