Posture suggestion method, electronic device and calculator readable storage medium

Abstract

The embodiment of the invention provides a posture suggestion method, an electronic device and a calculator readable storage medium. The method comprises the following steps: acquiring M first image frames of a first video stream in a specific time interval, and acquiring N second image frames of a second video stream in the specific time interval; k first joint points corresponding to K specified joints are judged in the first image frames, K second joint points corresponding to the K specified joints are judged in the second image frames, and the K first joint points correspond to the K second joint points respectively; obtaining an mth first image frame in the M first image frames and an nth second image frame in the N second image frames; and determining the difference between each first articulation point and each corresponding second articulation point in the mth first image frame and the nth second image frame, and providing a posture correction suggestion corresponding to the specific time interval according to the difference.

Description

Technical Field

[0001] The present invention relates to an image analysis technology, and more particularly to a posture suggestion method, an electronic device and a computer readable storage medium. Background Art

[0002] In modern life, learning sports through various sports teaching videos has become a very common learning method. However, for ordinary people, it is difficult to objectively judge whether the movements they perform are consistent with the demonstration movements presented in the sports teaching videos. Summary of the invention

[0003] In view of this, embodiments of the present invention provide a gesture suggestion method, an electronic device, and a computer-readable storage medium, which can be used to solve the above technical problems.

[0004] An embodiment of the present invention proposes a posture suggestion method, which is suitable for a posture suggestion method, characterized in that it includes: obtaining M first image frames of a first video stream within a specific time interval, and obtaining N second image frames of a second video stream within the specific time interval, wherein M and N are positive integers; determining K first joint points corresponding to K specified joints in each of the first image frames, and determining K second joint points corresponding to the K specified joints in each of the second image frames, wherein the K first joint points correspond to the K second joint points respectively, and K is a positive integer; obtaining the mth first image frame among the M first image frames and the nth second image frame among the N second image frames, wherein 1≤m≤M, and 1≤n≤N; and determining the differences between each of the first joint points and the corresponding each of the second joint points in the mth first image frame and the nth second image frame, and providing posture correction suggestions corresponding to the specific time interval accordingly.

[0005] An embodiment of the present invention proposes a posture suggestion method, characterized in that it includes a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the storage circuit, accesses the program code and executes: obtaining M first image frames of the first video stream within a specific time interval, and obtaining N second image frames of the second video stream within the specific time interval, wherein M and N are positive integers; determining K first joint points corresponding to K specified joints in each of the first image frames, and determining K second joint points corresponding to the K specified joints in each of the second image frames, wherein the K first joint points correspond to the K second joint points respectively, and K is a positive integer; obtaining the mth first image frame of the M first image frames and the nth second image frame of the N second image frames, wherein 1≤m≤M, and 1≤n≤N; and determining the difference between each of the first joint points and the corresponding second joint points in the mth first image frame and the nth second image frame, and providing posture correction suggestions corresponding to the specific time interval accordingly.

[0006] An embodiment of the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium records an executable computer program, and the executable computer program is loaded by a posture suggestion device to perform the following steps: obtain M first image frames of a first video stream within a specific time interval, and obtain N second image frames of a second video stream within the specific time interval, wherein M and N are positive integers; determine K first joint points corresponding to K specified joints in each of the first image frames, and determine K second joint points corresponding to the K specified joints in each of the second image frames, wherein the K first joint points correspond to the K second joint points respectively, and K is a positive integer; obtain the mth first image frame among the M first image frames and the nth second image frame among the N second image frames, wherein 1≤m≤M, and 1≤n≤N; and determine the difference between each of the first joint points and the corresponding each of the second joint points in the mth first image frame and the nth second image frame, and provide posture correction suggestions corresponding to the specific time interval accordingly. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the present invention and together with the description serve to explain the principles of the present invention.

[0008] Figure 1 is a schematic diagram of an electronic device according to an embodiment of the present invention.

[0009] Figure 2 is a flow chart of a posture comparison method according to the first embodiment of the present invention.

[0010] Figure 3 is a schematic diagram of a first image frame and a second image frame according to an embodiment of the present invention.

[0011] Figure 4 is a flow chart of determining the posture difference corresponding to a specific time interval T according to the first embodiment of the present invention.

[0012] Figure 5 It is a schematic diagram of a reference three-dimensional data structure according to the first embodiment of the present invention.

[0013] Figure 6 is a schematic diagram of determining a two-dimensional data structure according to an embodiment of the present invention.

[0014] Figure 7 is a flow chart of a posture suggestion method according to a second embodiment of the present invention.

[0015] Figure 8 FIG. 4 is a flow chart of determining a posture correction suggestion corresponding to a specific time interval T according to a second embodiment of the present invention.

[0016] Fig. 9 It is a flow chart of providing posture correction suggestions corresponding to a specific time interval based on a second two-dimensional data structure according to the second embodiment of the present invention.

[0017] Fig.10 is a schematic diagram of determining a specific path according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0018] Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0019] Please refer to Figure 1 , which is a schematic diagram of an electronic device according to an embodiment of the present invention. In different embodiments, the electronic device 100 can be implemented as various smart devices and / or computer devices, but is not limited thereto.

[0020] exist Figure 1 In the embodiment, the electronic device 100 includes a storage circuit 102 and a processor 104. The storage circuit 102 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk or other similar devices or a combination of these devices, and can be used to record multiple program codes or modules.

[0021] The processor 104 is coupled to the memory circuit 102 and may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), any other type of integrated circuit, a state machine, an Advanced RISC Machine (ARM) based processor, and the like.

[0022] In an embodiment of the present invention, the processor 104 can access the modules and program codes recorded in the storage circuit 102 to implement the various methods proposed by the present invention (such as the posture comparison method in the first embodiment and the posture suggestion method in the second embodiment), the details of which are described as follows.

[0023] Please refer to Figure 2 , which is a flow chart of a posture comparison method according to the first embodiment of the present invention. The method of this embodiment can be Figure 1 The electronic device 100 is executed as follows: Figure 1 Description of components shown Figure 2 Details of each step.

[0024] First, in step S210, the processor 104 obtains M first image frames of the first video stream in a specific time interval (hereinafter referred to as T), and obtains N second image frames of the second video stream in the specific time interval T, where M and N are positive integers.

[0025] Please refer to Figure 3 , which is a schematic diagram of a first image frame and a second image frame according to an embodiment of the present invention. Figure 3 In the video, the first video stream is a video of the first user 311 performing a specific action, the second video stream is a video of the second user 312 performing a specific action, and the first image frame 321 is, for example, one of the M first image frames, and the second image frame 322 is, for example, one of the N second image frames.

[0026] In this embodiment, the second user 312 is, for example, an action demonstrator, such as a coach, and the second video stream is, for example, a video of the second user 312 demonstrating the specific action (e.g., sit-ups). In addition, the first user 311 is, for example, a person who performs the specific action with reference to the second video stream, such as a trainee. In one embodiment, when the first user 311 watches the second video stream and imitates the second user 312 in performing the specific action, the relevant process can be captured by a camera or other imaging device as the first video stream and provided to the electronic device 100.

[0027] In one embodiment, the camera or imaging device may be built into the electronic device 100, or may be connected to the electronic device 100 by wire or wireless. In one embodiment, the first video stream and / or the second video stream of the electronic device 100 may be captured in real time, or may be obtained by the electronic device 100 from a related storage device, but is not limited thereto.

[0028] In one embodiment, M may be determined based on the frame rate of the first video stream and the length of the specific time interval T under consideration. For example, assuming that the frame rate of the first video stream is 30 frames per second and the length of the specific time interval is 1 second, M is, for example, 30 (i.e., 30*1). For another example, assuming that the frame rate of the first video stream is 30 frames per second and the length of the specific time interval is 1.5 seconds, M is, for example, 45 (i.e., 30*1.5).

[0029] Similarly, N may be determined according to the frame rate of the second video stream and the length of the specific time interval T under consideration. For example, assuming that the frame rate of the second video stream is 25 frames / second and the length of the specific time interval T is 1 second, N is, for example, 25 (i.e., 25*1). For another example, assuming that the frame rate of the second video stream is 25 frames / second and the length of the specific time interval T is 2 seconds, N is, for example, 25 (i.e., 25*2).

[0030] In one embodiment, the specific time interval T can be determined by the designer according to needs, and can be used as a time unit for the electronic device 100 to determine the difference in postures of the first user 311 and the second user 312 when performing the specific action.

[0031] For example, assuming that the specific time interval T is set to be from the Ath second to the Bth second (i.e., the length of the specific time interval T is (B-A) seconds), the processor 104 may obtain the image frames corresponding to the Ath second to the Bth second from the first video stream as the M first image frames in step S210, and obtain the image frames corresponding to the Ath second to the Bth second from the second video stream as the N second image frames, but is not limited to this.

[0032] In step S220 , the processor 104 determines K first joint points corresponding to K designated joints in each first image frame, and determines K second joint points corresponding to the K designated joints in each second image frame, where K is a positive integer.

[0033] In one embodiment, the K designated joints are, for example, K joints of interest that can be used to determine the difference in posture between the first user 311 and the second user 312. Figure 3 In an embodiment, the processor 104 may regard K joints that are more relevant to performing sit-ups as the above-mentioned joints of interest, such as shoulder joints, hip joints, knee joints, and ankle joints. In this case, joints that are not helpful in determining sit-ups may not be regarded as the above-mentioned joints of interest, such as joints on the face, finger joints, etc.

[0034] In one embodiment, the number of K can be determined by the designer according to the requirements and the specific action considered, such as 21, but is not limited thereto. In another embodiment, the processor 104 determines the same K joints for different specific actions.

[0035] In one embodiment, after obtaining the M first image frames, the processor 104 may execute a related joint point recognition algorithm (such as DeepPose) on each first image frame to determine the positions of each body joint of the first user 311 in each first image frame, and then find the joints corresponding to the K specified joints from the body joints in each first image frame as the K first joint points.

[0036] Similarly, after obtaining the N second image frames, the processor 104 may execute a relevant joint point recognition algorithm (such as DeepPose) on each second image frame to determine the positions of each body joint of the second user 312 in each second image frame, and then find the joints corresponding to the K specified joints from the body joints in each second image frame as the K second joint points, but is not limited to this.

[0037] In an embodiment of the present invention, the K first joint points correspond one-to-one to the K second joint points. In this case, the first and second joint points corresponding to each other will both correspond to the same designated joint. For example, assuming that the i-th first joint point of the K first joint points corresponds to the knee joint, the i-th second joint point of the K second joint points will also correspond to the knee joint. For another example, assuming that the j-th first joint point of the K first joint points corresponds to the hip joint, the j-th second joint point of the K second joint points will also correspond to the hip joint, but it is not limited to this (i and j are index values).

[0038] Afterwards, in step S230 , the processor 104 obtains the mth first image frame among the M first image frames and the nth second image frame among the N second image frames, wherein 1≤m≤M, and 1≤n≤N.

[0039] Next, in step S240 , the processor 104 determines the difference between each first joint point and each corresponding second joint point in the mth first image frame and the nth second image frame, and determines the posture difference corresponding to the specific time interval T accordingly.

[0040] In the first embodiment, the details of step S240 are supplemented by Figure 4 , 5 To illustrate, Figure 4 is a flowchart of determining the posture difference corresponding to a specific time interval T according to the first embodiment of the present invention, and Figure 5 It is a schematic diagram of a reference three-dimensional data structure according to the first embodiment of the present invention.

[0041] In step S410 , the processor 104 determines a reference three-dimensional data structure W0 based on the differences between each first joint point and each corresponding second joint point in the mth first image frame and the nth second image frame, wherein the reference three-dimensional data structure W0 has M×N×K reference data elements.

[0042] In the process of executing step S410 , the processor 104 may first initialize the reference three-dimensional data structure W0 . For example, the processor 104 may establish a data array with dimensions of M×N×K as the initialized reference three-dimensional data structure W0 .

[0043] Afterwards, the processor 104 may obtain the kth first joint point of the K first joint points in the mth first image frame as the first reference joint point, and obtain the kth second joint point of the K second joint points in the nth second image frame as the second reference joint point, where 1≤k≤K. Afterwards, the processor 104 may determine a position difference between the first reference joint point and the second reference joint point, and set a specific data element in the reference three-dimensional data structure W0 accordingly, wherein the specific data element corresponds to the mth first image frame, the nth second image frame, and the kth designated joint among the K designated joints.

[0044] In the first embodiment, the position of the specific data element in the reference three-dimensional data structure W0 is, for example, (m, n, k), and the specific data element can be correspondingly represented as W0(m, n, k), but is not limited thereto.

[0045] In one embodiment, when determining the position difference between the first reference joint point and the second reference joint point, the processor 104 can be configured to: obtain the first pixel coordinates of the first reference joint point in the mth first image frame; obtain the second pixel coordinates of the second reference joint point in the nth second image frame; and determine the position difference between the first reference joint point and the second joint point based on the distance between the first pixel coordinates and the second pixel coordinates.

[0046] For example, assuming that the kth specific joint (which corresponds to the kth first joint point and the kth second joint point) is a knee joint, and the mth first image frame and the nth second image frame are Figure 3 In this case, the processor 104 may find the knee joint of the first user 311 in the first image frame 321 as the first reference joint point, and find the knee joint of the second user 312 in the second image frame 322 as the second reference joint point. Afterwards, the processor 104 may determine the pixel coordinates of the knee joint of the first user 311 in the first image frame 321 as the first pixel coordinates, and determine the pixel coordinates of the knee joint of the second user 312 in the second image frame 322 as the second pixel coordinates. Afterwards, the processor 104 may determine the position difference between the first reference joint point and the second reference joint point based on the distance between the first pixel coordinates and the second pixel coordinates.

[0047] In some embodiments, the processor 104 may, for example, determine the (normalized) Euclidean distance between the first reference joint point and the second reference joint point as the position difference, but is not limited thereto.

[0048] After obtaining the position difference, the processor 104 can set the content of the reference data element at position (m, n, k) in the reference three-dimensional data structure W0 (i.e., W0(m, n, k)) to this position difference (e.g., the (normalized) Euclidean distance between the first reference joint point and the second reference joint point), but is not limited to this.

[0049] In an embodiment of the present invention, the processor 104 may determine the content of each reference data element in the reference three-dimensional data structure W0 based on the above mechanism.

[0050] exist Figure 5 In the embodiment, the reference three-dimensional data structure W0 includes K reference two-dimensional data structures W01 to W0K, which correspond to the K specified joints respectively. For example, the reference two-dimensional data structure W01 corresponds to the first specified joint among the K specified joints, the reference two-dimensional data structure W02 corresponds to the second specified joint among the K specified joints, and the reference two-dimensional data structure W0K corresponds to the Kth specified joint among the K specified joints.

[0051] In step S420 , the processor 104 determines a first two-dimensional data structure W1 by combining the K reference two-dimensional data structures, wherein the first two-dimensional data structure W1 has M×N first data elements.

[0052] In the process of determining the first two-dimensional data structure W1, the processor 104 may first initialize the first two-dimensional data structure W1. For example, the processor 104 may first establish a data array with a dimension of M×N as the initialized first two-dimensional data structure W1, but is not limited thereto. Afterwards, the processor 104 may find the reference data elements corresponding to the mth first image frame and the nth second image frame in each reference two-dimensional data structure W01-W0K, and determine the first data element corresponding to the mth first image frame and the nth second image frame in the first two-dimensional data structure W1 accordingly.

[0053] Please refer to Figure 6 , which is a schematic diagram of determining a two-dimensional data structure according to an embodiment of the present invention. In this embodiment, the reference data element corresponding to the mth first image frame and the nth second image frame in the reference two-dimensional data structure W01 is, for example, a reference data element located at (m, n), and this reference data element can be represented as W01(m, n). For example, the reference data element W01(1, 1) is, for example, a reference data element corresponding to the 1st first image frame and the 1st second image frame in the reference two-dimensional data structure W01; the reference data element W01(1, N) is, for example, a reference data element corresponding to the 1st first image frame and the Nth second image frame in the reference two-dimensional data structure W01; the reference data element W01(M, 1) is, for example, a reference data element corresponding to the Mth first image frame and the 1st second image frame in the reference two-dimensional data structure W01; the reference data element W01(M, N) is, for example, a reference data element corresponding to the Mth first image frame and the Nth second image frame in the reference two-dimensional data structure W01.

[0054] Similarly, the reference data element corresponding to the mth first image frame and the nth second image frame in the reference two-dimensional data structure W0K is, for example, a reference data element located at (m, n), and this reference data element can be represented as W0K(m, n). For example, the reference data element W0K(1, 1) is, for example, a reference data element corresponding to the 1st first image frame and the 1st second image frame in the reference two-dimensional data structure W0K; the reference data element W0K(1, N) is, for example, a reference data element corresponding to the 1st first image frame and the Nth second image frame in the reference two-dimensional data structure W0K; the reference data element W0K(M, 1) is, for example, a reference data element corresponding to the Mth first image frame and the 1st second image frame in the reference two-dimensional data structure W0K; and the reference data element W0K(M, N) is, for example, a reference data element corresponding to the Mth first image frame and the Nth second image frame in the reference two-dimensional data structure W0K.

[0055] In addition, the first data element corresponding to the mth first image frame and the nth second image frame in the first two-dimensional data structure W1 is, for example, the first data element located at (m, n), and this first data element can be represented as W1(m, n). For example, the first data element W1(1, 1) is, for example, the first data element corresponding to the 1st first image frame and the 1st second image frame in the first two-dimensional data structure W1; the first data element W1(1, N) is, for example, the first data element corresponding to the 1st first image frame and the Nth second image frame in the first two-dimensional data structure W1; the first data element W1(M, 1) is, for example, the first data element corresponding to the Mth first image frame and the 1st second image frame in the first two-dimensional data structure W1; the first data element W1(M, N) is, for example, the first data element corresponding to the Mth first image frame and the Nth second image frame in the first two-dimensional data structure W1.

[0056] Based on this, in the process of determining the first two-dimensional data structure W1, the processor 104 can respectively find the reference data elements W01(m,n)~W0K(m,n) in the reference two-dimensional data structures W01~W0K, and determine the first data element W1(m,n) in the first two-dimensional data structure W1 accordingly.

[0057] In one embodiment, the processor 104 may determine the first data element W1(m,n) based on the statistical characteristics (e.g., average value) and / or linear combination of the reference data elements W01(m,n)-W0K(m,n). In one embodiment, the processor 104 may also first normalize the reference data elements W01(m,n)-W0K(m,n) and then take their statistical characteristics and / or linear combination as the first data element W1(m,n), but is not limited thereto.

[0058] For example, the processor 104 may be configured to: determine the first data element W1(1,1) (eg, Figure 6 4 at the upper left corner of the first two-dimensional data structure W1); determining the first data element W1(M,1) (e.g. Figure 6 4 at the lower left corner of the first two-dimensional data structure W1); determining the first data element W1(1,N) (e.g. Figure 6 4 in the upper right corner of the first two-dimensional data structure W1); determining the first data element W1(M,N) (e.g. Figure 6 4) located at the lower right corner of the first two-dimensional data structure W1).

[0059] For other first data elements in the first two-dimensional data structure W1 , the processor 104 may make determinations based on similar principles, and the details are not further elaborated herein.

[0060] Next, in step S430 , the processor 104 finds surrounding elements corresponding to each first data element, and generates M×N second data elements accordingly, wherein the M×N second data elements form a second two-dimensional data structure M2 .

[0061] In the process of executing step S430, the processor 104 may first initialize the second two-dimensional data structure M2. For example, the processor 104 may first establish a data array with a dimension of M×N as the initialized second two-dimensional data structure M2. In this embodiment, the second data element corresponding to the mth first image frame and the nth second image frame is, for example, the second data element at position (m, n) in the second two-dimensional data structure W2, which may be represented as W2(m, n).

[0062] In one embodiment, when m and n are both 1, the processor 104 may determine that the second data element W2(m,n) is the first data element W1(m,n). That is, when determining the second data element W2(1,1) located at the upper left corner of the second two-dimensional data structure W2, the processor 104 may directly use the first data element W1(1,1) located at the upper left corner of the first two-dimensional data structure W1 as the second data element W2(1,1), but is not limited thereto.

[0063] In addition, when m or n is not 1, the processor 104 can obtain the surrounding elements corresponding to the first data element W1(m,n) in the second two-dimensional data structure W2, and add the first data element W1(m,n) and the corresponding surrounding elements to obtain the second data element W2(m,n).

[0064] In one embodiment, the surrounding elements corresponding to the first data element W1(m,n) in the second two-dimensional data structure W2 include at least one of the second data elements W2(m-1,n), W2(m,n-1) and W2(m-1,n-1). In addition, the surrounding elements corresponding to the first data element W1(m,n) in the second two-dimensional data structure W2 may include the smallest of the second data elements W2(m-1,n), W2(m,n-1) and W2(m-1,n-1).

[0065] For example, when determining the second data element W2(1,2), the processor 104 may find the surrounding elements corresponding to the first data element W1(1,2) in the second two-dimensional data structure W2, such as the second data element W2(1,1). Afterwards, the processor 104 may add the first data element W1(1,2) and the second data element W2(1,1) to obtain the second data element W2(1,2) (i.e., 1+4=5).

[0066] For example, when determining the second data element W2(2,1), the processor 104 may find the surrounding elements corresponding to the first data element W1(2,1) in the second two-dimensional data structure W2, such as the second data element W2(1,1). Afterwards, the processor 104 may add the first data element W1(2,1) and the second data element W2(1,1) to obtain the second data element W2(2,1) (i.e., 9+4=13).

[0067] For another example, when determining the second data element W2(2,2), the processor 104 may find the surrounding elements corresponding to the first data element W1(2,2) in the second two-dimensional data structure W2, for example, the smallest one among the second data elements W2(1,1), W2(1,2) and W2(2,1) (i.e., 4 corresponding to the second data element W2(1,1)). Afterwards, the processor 104 may add the first data element W1(2,2) and the second data element W2(1,1) to obtain the second data element W2(2,2) (i.e., 4+4=8).

[0068] In short, when m or n is not 1, the processor 104 may select one (e.g., the smallest one) from the second data elements W2(m-1,n), W2(m,n-1), and W2(m-1,n-1) and add it to the first data element W1(m,n) to obtain the second data element W2(m,n). From another perspective, the processor 104 may also be understood as selecting one from the second data elements to the left, upper left, and upper of the second data element W2(m,n) and adding it to the first data element W1(m,n) to determine the second data element W2(m,n), but is not limited thereto.

[0069] Based on the above mechanism, the processor 104 may correspondingly determine other second data elements in the second two-dimensional data structure W2 , and the details thereof are not further elaborated herein.

[0070] Then, in step S440, the processor 104 determines the posture difference corresponding to the specific time interval T based on the second two-dimensional data structure W2.

[0071] In one embodiment, the processor 104 may find a second data element corresponding to the Mth first image frame among the M first image frames and the Nth second image frame among the N second image frames among the M×N second data elements as a second specific data element (i.e., second data element W2(M,N)), and determine the second specific data element as a posture difference corresponding to the specific time interval T.

[0072] exist Figure 6 , since the second data element W2(M,N) is 34, the processor 104 can determine that the posture difference corresponding to the specific time interval T is 34. However, the above implementation is only used as an example and is not used to limit the possible implementation of the present invention. In an embodiment of the present invention, the posture difference represents the difference between the first user 311 and the second user 312 performing the specific action in the specific time interval T. The higher the posture difference, the less similar the actions of the first user 311 and the second user 312 are, and vice versa.

[0073] For other specific time intervals, the processor 104 can determine the corresponding posture difference based on the above mechanism, so that the first user 311 can know whether the action performed by the first user 311 is similar to that of the second user 312, and can perform corresponding action correction.

[0074] However, for users who have less relevant sports knowledge, just knowing whether the movements are similar may not be enough to make the correct movements. Based on this, the embodiment of the present invention further proposes a posture suggestion method, which can be used to provide users with relevant movement correction suggestions, and the relevant details are detailed as follows.

[0075] Please refer to Figure 7, which is a flow chart of a posture suggestion method according to a second embodiment of the present invention. The method of this embodiment can be Figure 1 The electronic device 100 is executed as follows: Figure 1 Description of components shown Figure 7 Details of each step.

[0076] In step S710, the processor 104 obtains M first image frames of the first video stream in a specific time interval T, and obtains N second image frames of the second video stream in the specific time interval T. In step S720, the processor 104 determines K first joint points corresponding to K specified joints in each first image frame, and determines K second joint points corresponding to the K specified joints in each second image frame. In step S730, the processor 104 obtains the mth first image frame of the M first image frames and the nth second image frame of the N second image frames.

[0077] In the second embodiment, the details of steps S710 to S730 can be referred to Figure 2 The relevant descriptions of steps S210 to S230 are not repeated here.

[0078] In step S740 , the processor 104 determines the difference between each first joint point and each corresponding second joint point in the mth first image frame and the nth second image frame, and provides a posture correction suggestion corresponding to the specific time interval T accordingly.

[0079] In the second embodiment, the details of step S740 are supplemented by Figure 8 To illustrate, Figure 8 FIG. 4 is a flow chart of determining a posture correction suggestion corresponding to a specific time interval T according to a second embodiment of the present invention.

[0080] In step S810, the processor 104 determines the reference three-dimensional data structure W0 based on the difference between each first joint point and the corresponding second joint point in the mth first image frame and the nth second image frame. In step S820, the processor 104 determines the first two-dimensional data structure W1 by combining the K reference two-dimensional data structures W01-W0K. In step S830, the processor 104 finds the surrounding elements corresponding to each first data element and generates M×N second data elements accordingly, wherein the M×N second data elements form a second two-dimensional data structure W2.

[0081] In the second embodiment, the details of steps S710 to S730 can be referred to Figure 4 The relevant descriptions of steps S410 to S430 are not repeated here.

[0082] In step S840 , the processor 104 determines a posture correction suggestion corresponding to the specific time interval T based on the second two-dimensional data structure W2 .

[0083] In the second embodiment, the details of step S840 are supplemented by Fig. 9 To illustrate, Fig. 9 It is a flow chart of providing posture correction suggestions corresponding to a specific time interval based on a second two-dimensional data structure according to the second embodiment of the present invention.

[0084] First, in step S910, the processor 104 may find a second data element corresponding to the Mth first image frame among the M first image frames and the Nth second image frame among the N second image frames among the M×N second data elements as the second specific data element (i.e., the second data element W2(M,N)).

[0085] In step S920 , the processor 104 may determine a specific path in each of the reference two-dimensional data structures W01 ˜ W0K based on the second specific data element.

[0086] Please refer to Fig.10 , which is a schematic diagram of determining a specific path according to the second embodiment of the present invention. Fig.10 In the scenario, assuming that M and N are 5 and 6 respectively, the processor 104 may find a plurality of third data elements involved in determining the second specific data element (ie, the second data element W2 (5, 6)) in the second two-dimensional data structure W2.

[0087] exist Fig.10 In the scenario, the determination process of the second data element W2(5,6) (i.e., the second specific data element) involves, for example, the operations in Table 1 below, wherein the details of determining the second data element W2(5,6) can be referred to. Figure 6 Related instructions for determining the second two-dimensional data structure W2.

[0088] Second data element Numeric W2(1,1)=W1(1,1) 4 W2(2,2)=W1(2,2)+W2(1,1) 8 W2(3,3)=W1(3,3)+W2(2,2) 17 W2(4,4)=W1(4,4)+W2(3,3) 21 W2(4,5)=W1(4,6)+W2(4,4) 30 W2(5,6)=W1(5,6)+W2(4,5) 34

[0089] Table 1

[0090] exist Fig.10 In the scenario, since the process of determining the second data element W2(5,6) involves the second data elements W2(1,1), W2(2,2), W2(3,3), W2(4,4), W2(4,5), the processor 104 may determine the second data elements W2(1,1), W2(2,2), W2(3,3), W2(4,4), W2(4,5) as the multiple third data elements, but is not limited to this.

[0091] Afterwards, the processor 104 may determine the reference path RP based on the multiple coordinates of the multiple third data elements in the second two-dimensional data structure W2. Fig.10 In the scenario, the coordinates of the third data elements in the second two-dimensional data structure W2 are, for example, (1,1), (2,2), (3,3), (4,4), and (4,5), respectively, and the processor 104 can determine the reference path RP accordingly.

[0092] Next, the processor 104 may find out a path corresponding to the reference path RP in each of the reference two-dimensional data structures W01 ˜ W0K as the corresponding specific path.

[0093] For example, in the reference two-dimensional data structure W01, the processor 104 may take the path corresponding to the reference path RP as the corresponding specific path P1 (which includes coordinates (1,1), (2,2), (3,3), (4,4), (4,5)); in the reference two-dimensional data structure W02, the processor 104 may take the path corresponding to the reference path RP as the corresponding specific path P2 (which includes coordinates (1,1), (2,2), (3,3), (4,4), (4,5)); in the reference two-dimensional data structure W0K, the processor 104 may take the path corresponding to the reference path RP as the corresponding specific path PK (which includes coordinates (1,1), (2,2), (3,3), (4,4), (4,5)).

[0094] Next, in step S930 , the process 104 may determine the reference values ​​corresponding to the reference two-dimensional data structures W01 ˜ W0K based on the specific paths P1 ˜PK corresponding to the reference two-dimensional data structures W01 ˜ W0K.

[0095] In one embodiment, the processor 104 may find a plurality of specific data elements corresponding to a specific path in a kth reference two-dimensional data structure among the reference two-dimensional data structures W01 - W0K, and sum the plurality of data elements into a reference value corresponding to the kth reference two-dimensional data structure.

[0096] For example, for the reference two-dimensional data structure W01 (i.e., the first reference two-dimensional data structure), the processor 104 may determine the reference data elements located on the specific path P1 as the specific data elements in the reference two-dimensional data structure W01. Afterwards, the processor 104 may sum the specific data elements in the reference two-dimensional data structure W01 as the reference value corresponding to the reference two-dimensional data structure W01.

[0097] For another example, for the reference two-dimensional data structure W02 (i.e., the second reference two-dimensional data structure), the processor 104 may determine the reference data elements located on the specific path P2 as the specific data elements in the reference two-dimensional data structure W02. Afterwards, the processor 104 may sum the specific data elements in the reference two-dimensional data structure W02 as the reference value corresponding to the reference two-dimensional data structure W02.

[0098] For another example, for the reference two-dimensional data structure W0K (i.e., the Kth reference two-dimensional data structure), the processor 104 may determine the reference data elements located on the specific path PK as the specific data elements in the reference two-dimensional data structure W0K. Afterwards, the processor 104 may sum up the specific data elements in the reference two-dimensional data structure W0K as the reference value corresponding to the reference two-dimensional data structure W0K.

[0099] For other reference two-dimensional data structures, the processor 104 may determine the corresponding reference values ​​based on the above mechanism, and the details are not further elaborated herein.

[0100] In some embodiments, the processor 104 may further normalize the reference values ​​corresponding to the reference two-dimensional data structures W01 ˜ W0K, but is not limited thereto.

[0101] Afterwards, in step S940 , the processor 104 provides a posture correction suggestion corresponding to the specific time interval T based on the reference values ​​corresponding to the reference two-dimensional data structures W01 ˜ W0K.

[0102] In the second embodiment, the reference two-dimensional data structures W01-W0K can be divided into different parts corresponding to the body, such as the head, torso, buttocks, arms, legs, upper body, lower body, etc. In one embodiment, the reference two-dimensional data structures W01-W0K can be divided into a first group and a second group corresponding to the first body part and the second body part, respectively. For ease of understanding, it is assumed below that the first body part and the second body part are the upper body and the lower body, respectively, but it is not limited to this. In this case, one or more of the reference two-dimensional data structures W01-W0K corresponding to the joints belonging to the upper body can be determined to belong to the first group, and one or more of the reference two-dimensional data structures W01-W0K corresponding to the joints belonging to the lower body can be determined to belong to the second group.

[0103] In one embodiment, the processor 104 may be configured to perform: determining a first difference value (hereinafter referred to as V1) based on the (normalized) reference values ​​corresponding to each reference two-dimensional data structure belonging to the first group; determining a second difference value (hereinafter referred to as V2) based on the (normalized) reference values ​​corresponding to each reference two-dimensional data structure belonging to the second group.

[0104] For ease of understanding, it is assumed below that K is 9, and the reference two-dimensional data structures W01-W06 belong to the first group, and the reference two-dimensional data structures W07-W09 belong to the second group. However, this is only used as an example and is not intended to limit possible implementations of the present invention.

[0105] Based on this, the processor 104 may determine the first difference value V1 based on the reference values ​​corresponding to the reference two-dimensional data structures W01-W06 belonging to the first group, and determine the second difference value V2 based on the reference values ​​corresponding to the reference two-dimensional data structures W07-W09 belonging to the second group. In one embodiment, the processor 104 may, for example, use the statistical characteristics and / or linear combination of the reference values ​​corresponding to the reference two-dimensional data structures W01-W06 as the first difference value V1, and use the statistical characteristics and / or linear combination of the reference values ​​corresponding to the reference two-dimensional data structures W07-W09 as the second difference value V2, but is not limited thereto.

[0106] Afterwards, the processor 104 may provide the posture correction suggestion corresponding to the specific time interval T based on the comparison result of the first difference value V1 and the second difference value V2.

[0107] In one embodiment, in response to determining that the first difference value V1 is greater than the second difference value V2, the processor 104 provides a first posture correction suggestion to correct a first body part (e.g., the upper body) as the posture correction suggestion. On the other hand, in response to determining that the first difference value V1 is not greater than the second difference value V2, the processor 104 provides a second posture correction suggestion to correct a second body part (e.g., the lower body) as the posture correction suggestion.

[0108] In one embodiment, the processor 104 may also first determine the posture difference according to the content of the first embodiment, and determine whether the posture difference is lower than the difference threshold. In response to determining that the posture difference is lower than the difference threshold, it means that the first user 311 better imitates the action of the second user 312. In this case, the processor 104 may provide a posture maintenance suggestion as the posture correction suggestion, or may not provide any suggestion.

[0109] On the other hand, in response to determining that the posture difference is not less than the difference threshold, it means that the first user 311 does not properly imitate the action of the second user 312. In this case, the processor 104 can provide the corresponding first posture correction suggestion and / or second posture correction suggestion as the posture correction suggestion according to the mechanism of the second embodiment, but is not limited thereto.

[0110] In another embodiment, the reference two-dimensional data structures W01~W0K can be divided into multiple different groups corresponding to multiple different body parts. The processor 104 can respectively determine the posture differences of the multiple different groups, select the group corresponding to the largest difference value, and determine whether the posture difference is not lower than the difference threshold, and then provide corresponding posture correction suggestions for the body part corresponding to the group with the largest difference.

[0111] In yet another embodiment, in addition to providing correction suggestions for postures of different body parts, the processor 104 may also provide a quantitative posture difference index or a posture difference interval according to the posture difference, for the user to refer to the degree of posture difference.

[0112] In summary, the method proposed in the embodiment of the present invention can compare the actions presented in different video streams and / or provide posture suggestions, so that the user can know whether the specific action has been performed correctly when referring to the video and which specific parts of the body need to be adjusted. In this way, the effect of learning actions through videos can be improved.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A gesture suggestion method, suitable for an electronic device, characterized in that: include: Obtain M first image frames of a first video stream within a specific time interval, and obtain N second image frames of a second video stream within the specific time interval, where M and N are positive integers; Determine K first joint points corresponding to K specified joints in each of the first image frames, and determine K second joint points corresponding to the K specified joints in each of the second image frames, wherein the K first joint points correspond to the K second joint points respectively, and K is a positive integer; Obtaining an m-th first image frame among the M first image frames and an n-th second image frame among the N second image frames, wherein 1≤m≤M, and 1≤n≤N; as well as The difference between each of the first joint points and the corresponding each of the second joint points in the mth first image frame and the nth second image frame is determined, and a posture correction suggestion corresponding to the specific time interval is provided accordingly.

2. The posture suggestion method according to claim 1, wherein the first video stream is a video of a first user performing a specific action, the second video stream is a video of a second user performing the specific action, and the posture correction suggestion is used to prompt the first user to adjust the way of performing the specific action.

3. The posture suggestion method according to claim 1, wherein the step of determining the difference between each of the first joint points and the corresponding each of the second joint points in the mth first image frame and the nth second image frame, and providing posture correction suggestions corresponding to the specific time interval accordingly comprises: Determine a reference three-dimensional data structure based on the difference between each of the first joint points and the corresponding each of the second joint points in the m-th first image frame and the n-th second image frame, wherein the reference three-dimensional data structure includes K reference two-dimensional data structures, and the reference three-dimensional data structure has M×N×K reference data elements; Determining a first two-dimensional data structure by combining the K reference two-dimensional data structures, wherein the first two-dimensional data structure has M×N first data elements; as well as Provide a posture correction suggestion corresponding to the specific time interval based on the M×N first data elements.

4. The posture suggestion method according to claim 3, wherein the step of determining the first two-dimensional data structure by combining the K reference two-dimensional data structures comprises: The reference data element corresponding to the m-th first image frame and the n-th second image frame is found in each of the reference two-dimensional data structures, and the first data element corresponding to the m-th first image frame and the n-th second image frame in the first two-dimensional data structure is determined accordingly.

5. The posture suggestion method according to claim 3, wherein the step of providing posture correction suggestions corresponding to the specific time interval based on the M×N first data elements comprises: Finding surrounding elements corresponding to each of the first data elements, and generating M×N second data elements accordingly, wherein the M×N second data elements form a second two-dimensional data structure; and Providing posture correction suggestions corresponding to the specific time interval based on the second two-dimensional data structure.

6. The posture suggestion method according to claim 5, wherein the first data element corresponding to the mth first image frame and the nth second image frame is represented by W1(m,n), the second data element corresponding to the mth first image frame and the nth second image frame is represented by W2(m,n), and the step of finding the surrounding elements corresponding to each of the first data elements and generating the M×N second data elements accordingly comprises: Initializing the second two-dimensional data structure; When m and n are both 1, W2(m,n) is determined to be W1(m,n); When m or n is not 1, the surrounding elements corresponding to W1(m,n) are obtained in the second two-dimensional data structure, and W1(m,n) and the corresponding surrounding elements are added together to obtain W2(m,n).

7. The posture suggestion method according to claim 6, wherein the step of providing posture correction suggestions corresponding to the specific time interval based on the second two-dimensional data structure comprises: Finding the second data element corresponding to the Mth first image frame among the M first image frames and the Nth second image frame among the N second image frames as the second specific data element from among the M×N second data elements; determining a specific path in each of the reference two-dimensional data structures based on the second specific data element; Determine the reference value corresponding to each of the reference two-dimensional data structures based on the specific path corresponding to each of the reference two-dimensional data structures; as well as The posture correction suggestion corresponding to the specific time interval is provided based on the reference numerical values ​​corresponding to each of the reference two-dimensional data structures.

8. The posture suggestion method according to claim 7, wherein the K reference two-dimensional data structures include a k-th reference two-dimensional data structure, 1≤k≤K, and the step of determining the reference value corresponding to each of the reference two-dimensional data structures based on the specific path comprises: Finding a plurality of specific data elements corresponding to the specific path in the kth reference two-dimensional data structure; The multiple data elements are summed up to obtain the reference value corresponding to the kth reference two-dimensional data structure.

9. The gesture suggestion method according to claim 7, wherein the step of determining the specific path in each of the reference two-dimensional data structures based on the second specific data element comprises: Finding, in the second two-dimensional data structure, a plurality of third data elements involved in determining the second specific data element; determining a reference path using a plurality of coordinates of the plurality of third data elements in the second two-dimensional data structure; as well as A path corresponding to the reference path is found in each of the reference two-dimensional data structures as the corresponding specific path.

10. The posture suggestion method according to claim 7, wherein the K reference two-dimensional data structures are divided into a first group and a second group corresponding to a first body part and a second body part respectively, and the step of providing the posture correction suggestion corresponding to the specific time interval based on the reference value corresponding to each of the reference two-dimensional data structures comprises: Determine a first difference value based on the reference value corresponding to each of the reference two-dimensional data structures belonging to the first group; Determine a second difference value based on the reference value corresponding to each of the reference two-dimensional data structures belonging to the second group; Providing the posture correction suggestion corresponding to the specific time interval based on the comparison result of the first difference value and the second difference value includes: In response to determining that the first difference value is greater than the second difference value, and the first difference value is not less than a difference threshold, providing a first posture correction suggestion for correcting the first body part as the posture correction suggestion; as well as In response to determining that the first difference value is not greater than the second difference value and the second difference value is not lower than the difference threshold, a second posture correction suggestion for correcting the second body part is provided as the posture correction suggestion.

11. The posture suggestion method according to claim 7, wherein the K reference two-dimensional data structures are divided into a plurality of different groups corresponding to a plurality of different body parts, and the step of providing the posture correction suggestion corresponding to the specific time interval based on the reference values ​​corresponding to each of the reference two-dimensional data structures comprises: Determining a plurality of different difference values ​​respectively based on the reference values ​​corresponding to the reference two-dimensional data structures belonging to the plurality of different groups; Based on the comparison results of the plurality of different difference values, selecting a third difference value having the largest difference value, wherein the third difference value corresponds to a third group, and the third group corresponds to a third body part; and In response to determining that the third difference value is not lower than the difference threshold, a second posture correction suggestion for correcting the third body part is provided as the posture correction suggestion.

12. The posture suggestion method according to claim 7, wherein the K reference two-dimensional data structures are divided into a plurality of different groups corresponding to a plurality of different body parts, and based on the reference values ​​corresponding to the reference two-dimensional data structures, the step of providing the posture correction suggestion corresponding to the specific time interval comprises: Determining a plurality of different difference values ​​respectively based on the reference values ​​corresponding to the reference two-dimensional data structures belonging to the plurality of different groups; Based on a plurality of different difference values, posture difference indicators corresponding to the difference values ​​are provided respectively.

13. The posture suggestion method according to claim 1, wherein the step of determining the difference between each of the first joint points and the corresponding each of the second joint points in the mth first image frame and the nth second image frame, and providing the posture correction suggestion corresponding to the specific time interval accordingly comprises: Determining a posture difference corresponding to the specific time interval based on the difference between each of the first joint points and the corresponding each of the second joint points in the mth first image frame and the nth second image frame; In response to determining that the posture difference is lower than the difference threshold, providing a posture maintenance suggestion as the posture correction suggestion.

14. An electronic device, characterized in that: include: A storage circuit for storing program codes; as well as A processor, coupled to the storage circuit, accesses the program code and executes: Obtain M first image frames of a first video stream within a specific time interval, and obtain N second image frames of a second video stream within the specific time interval, where M and N are positive integers; Determine K first joint points corresponding to K specified joints in each of the first image frames, and determine K second joint points corresponding to the K specified joints in each of the second image frames, wherein the K first joint points correspond to the K second joint points respectively, and K is a positive integer; Obtaining an m-th first image frame among the M first image frames and an n-th second image frame among the N second image frames, wherein 1≤m≤M, and 1≤n≤N; as well as The difference between each of the first joint points and the corresponding each of the second joint points in the mth first image frame and the nth second image frame is determined, and a posture correction suggestion corresponding to the specific time interval is provided accordingly.

15. A computer-readable storage medium, characterized in that: The computer readable storage medium records an executable computer program, and the executable computer program is loaded by the electronic device to perform the following steps: Obtain M first image frames of a first video stream within a specific time interval, and obtain N second image frames of a second video stream within the specific time interval, where M and N are positive integers; Determine K first joint points corresponding to K specified joints in each of the first image frames, and determine K second joint points corresponding to the K specified joints in each of the second image frames, wherein the K first joint points correspond to the K second joint points respectively, and K is a positive integer; Obtaining an m-th first image frame among the M first image frames and an n-th second image frame among the N second image frames, wherein 1≤m≤M, and 1≤n≤N; as well as The difference between each of the first joint points and the corresponding each of the second joint points in the mth first image frame and the nth second image frame is determined, and a posture correction suggestion corresponding to the specific time interval is provided accordingly.

Images