Motion detection method, device, computer equipment and computer readable storage medium

Abstract

The present application relates to an action detection method, device, computer equipment and computer-readable storage medium. The method includes: obtaining a first live broadcast image of a target anchor, and according to the first live broadcast image, obtaining the key point coordinates of each key point of the target anchor in the first live broadcast image; according to each key point coordinate, detecting whether the target anchor's action amplitude for a preset action is greater than a first amplitude threshold; if the action amplitude is greater than the first amplitude threshold, determining that the target anchor has started the preset action, and according to at least one second live broadcast image of the target anchor, detecting whether the target anchor has ended the preset action, the first live broadcast image and at least one second live broadcast image are adjacent in time sequence; if the target anchor has ended the preset action, determining that the target anchor has completed a preset action. This method can accurately detect the effective actions completed by the target anchor.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to an action detection method, device, computer equipment and computer-readable storage medium. Background Art

[0002] With the popularization of computer equipment and the development of the Internet, live broadcasting based on the Internet has become more and more common in people's daily lives. Live broadcasting is an important form of live broadcasting. Through live broadcasting, different anchors can interact with each other. The live broadcasting interface can simultaneously display the corresponding anchor room pictures of different anchors for the audience to watch.

[0003] During the live broadcast, the hosts can enhance the fun of the live broadcast through interactive methods such as talent PK. During the interactive process, the hosts may need to perform actions such as blinking, twisting the waist, etc. The hosts can be awarded corresponding PK scores based on the number of effective actions they complete.

[0004] Therefore, how to accurately detect the effective actions completed by the anchor has become a problem that needs to be solved urgently. Summary of the invention

[0005] Based on this, it is necessary to provide an action detection method, device, computer equipment and computer-readable storage medium that can accurately detect effective actions performed by the target anchor in response to the above technical problems.

[0006] In a first aspect, an embodiment of the present application provides an action detection method, the method comprising:

[0007] Acquire a first live broadcast image of a target anchor, and acquire key point coordinates of key points of the target anchor in the first live broadcast image according to the first live broadcast image;

[0008] According to the coordinates of each of the key points, detecting whether the target anchor's action amplitude for the preset action is greater than a first amplitude threshold;

[0009] If the action amplitude is greater than the first amplitude threshold, it is determined that the target anchor starts the preset action, and whether the target anchor ends the preset action is detected according to at least one second live broadcast image of the target anchor, and the first live broadcast image and the at least one second live broadcast image are sequentially adjacent in time sequence;

[0010] If the target anchor finishes the preset action, it is determined that the target anchor completes the preset action once.

[0011] In one embodiment, the detecting, based on at least one second live broadcast image of the target anchor, whether the target anchor has ended the preset action includes:

[0012] According to the time sequence of each of the second live broadcast images from front to back, it is detected in turn whether the action amplitude of the target anchor for the preset action in each of the second live broadcast images is less than a second amplitude threshold, and the second amplitude threshold is less than the first amplitude threshold;

[0013] If it is detected that in the target second live broadcast image, the action amplitude of the target anchor for the preset action is less than the second amplitude threshold, it is determined that the target anchor ends the preset action.

[0014] In one embodiment, detecting whether the target anchor's action amplitude for a preset action is greater than a first amplitude threshold according to the coordinates of each of the key points includes:

[0015] According to the coordinates of each of the key points, an angle value between the central axis of the body of the target anchor in the first live broadcast image and the vertical direction is obtained;

[0016] If the absolute value of the angle value is greater than the preset angle threshold, it is determined that the action amplitude of the target anchor for the preset action is greater than the first amplitude threshold.

[0017] In one embodiment, each of the key point coordinates includes a left shoulder key point coordinate, a right shoulder key point coordinate, a left waist key point coordinate, and a right waist key point coordinate, and obtaining the angle value between the body axis of the target anchor in the first live broadcast image and the vertical direction according to each of the key point coordinates includes:

[0018] Acquire the first midpoint coordinates according to the left shoulder key point coordinates and the right shoulder key point coordinates;

[0019] Acquire the second midpoint coordinates according to the left waist key point coordinates and the right waist key point coordinates;

[0020] According to the first midpoint coordinates and the second midpoint coordinates, the angle value between the central axis of the body of the target anchor in the first live broadcast image and the vertical direction is calculated.

[0021] In one embodiment, acquiring key point coordinates of key points of the target anchor in the first live broadcast image according to the first live broadcast image includes:

[0022] Acquire target position frame coordinates corresponding to the target portrait and the first live image;

[0023] According to the target position frame coordinates, intercepting a target image from the first live image;

[0024] Inputting the target image into a posture estimation model to obtain a heat map of each key point;

[0025] According to the heat map of each of the key points, the key point coordinates of each of the key points are obtained.

[0026] In one embodiment, obtaining the target position frame coordinates corresponding to the target portrait and the first live image includes:

[0027] Acquire portrait detection position frame coordinates, where the portrait detection position frame coordinates are position frame coordinates corresponding to the target portrait detected from a target live image before the first live image using a portrait detection model;

[0028] The portrait detection position frame coordinates are enlarged according to a preset enlargement ratio to obtain the target position frame coordinates.

[0029] In one embodiment, obtaining the key point coordinates of each key point according to the heat map of each key point includes:

[0030] For each of the key point's heat maps, obtaining the target coordinates of the key point in the target image according to the heat map;

[0031] According to the position of the target image in the first live image and the target coordinates of the key point in the target image, the key point coordinates of the key point in the first live image are acquired.

[0032] In one embodiment, the inputting the target image into the posture estimation model to obtain the heat map of each key point includes:

[0033] Inputting the target image into an affine transformation model to obtain affine transformation parameters corresponding to the target image;

[0034] Performing an affine transformation on the target image using the affine transformation parameters to obtain a corrected image;

[0035] The corrected image is input into the posture estimation model to obtain a heat map of each key point.

[0036] In one embodiment, obtaining the key point coordinates of each key point according to the heat map of each key point includes:

[0037] For each of the key point's heat maps, obtaining the corrected coordinates of the key point in the corrected image according to the heat map;

[0038] According to the affine transformation parameters, the corrected coordinates are inversely transformed to obtain the target coordinates of the key point in the target image;

[0039] According to the position of the target image in the first live image and the target coordinates of the key point in the target image, the key point coordinates of the key point in the first live image are acquired.

[0040] In one embodiment, the method further comprises:

[0041] detecting a similarity between the target image and a previous frame captured image, wherein the previous frame captured image is an image corresponding to the target portrait captured from a third live image before the first live image, and the third live image is adjacent to the first target image in time sequence;

[0042] If the similarity between the target image and the previous frame captured image is less than the similarity threshold, the step of inputting the target image into the posture estimation model to obtain the heat map of each key point is performed.

[0043] In one embodiment, detecting the similarity between the target image and a previous frame captured image includes:

[0044] Performing hash processing on the target image to obtain a first hash value;

[0045] Performing hash processing on the previous frame captured image to obtain a second hash value;

[0046] Calculate an absolute value of a difference between the first Hash value and the second Hash value;

[0047] If the absolute value of the difference is greater than the difference threshold, determining that the similarity between the target image and the previous frame captured image is less than the similarity threshold;

[0048] If the absolute value of the difference is less than or equal to the difference threshold, it is determined that the similarity between the target image and the previous frame captured image is greater than or equal to the similarity threshold.

[0049] In one embodiment, after detecting the similarity between the target image and the previous frame captured image, the method further includes:

[0050] If the similarity between the target image and the captured image of the previous frame is greater than or equal to the similarity threshold, the key point coordinates of each key point of the target anchor in the third live image are determined as the key point coordinates of each key point of the target anchor in the first live image.

[0051] In one embodiment, the method further comprises:

[0052] Get a sample target image;

[0053] Performing an affine transformation on the sample target image to obtain a dual sample image;

[0054] Training an initial pose estimation model according to the sample target image and the dual sample image to obtain the pose estimation model;

[0055] Among them, in an intermediate iteration process, the sample target image and the dual sample image are input into the intermediate posture estimation model to obtain the first predicted key point coordinates corresponding to the sample target image and the second predicted key point coordinates corresponding to the dual sample image; the first predicted key point coordinates are affine transformed to obtain the corrected predicted point coordinates; the loss value corresponding to the current iteration process is calculated based on the second predicted key point coordinates and the corrected predicted point coordinates; and the model parameters of the intermediate posture estimation model are adjusted based on the loss value.

[0056] In a second aspect, an embodiment of the present application provides an action detection device, the device comprising:

[0057] An acquisition module, used to acquire a first live broadcast image of a target anchor, and acquire key point coordinates of key points of the target anchor in the first live broadcast image according to the first live broadcast image;

[0058] A first detection module is used to detect whether the action amplitude of the target anchor for the preset action is greater than a first amplitude threshold according to the coordinates of each of the key points;

[0059] a second detection module, configured to determine that the target anchor has started the preset action if the action amplitude is greater than the first amplitude threshold, and detect whether the target anchor has ended the preset action according to at least one second live broadcast image of the target anchor, wherein the first live broadcast image and the at least one second live broadcast image are sequentially adjacent in time sequence;

[0060] The determination module is used to determine that the target anchor has completed the preset action once if the target anchor finishes the preset action.

[0061] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of the first aspect when executing the computer program.

[0062] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method of the first aspect described above.

[0063] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0064] By acquiring a first live broadcast image of a target anchor, and acquiring key point coordinates of key points of the target anchor in the first live broadcast image according to the first live broadcast image, then, according to the coordinates of each key point, detecting whether the target anchor's action amplitude for a preset action is greater than a first amplitude threshold, if the action amplitude is greater than the first amplitude threshold, determining that the target anchor has started the preset action, and then, according to at least one second live broadcast image of the target anchor, detecting whether the target anchor has ended the preset action; in this way, only when the action amplitude is large, is it determined that the target anchor has started the preset action, and when the target anchor has a small amplitude human body movement, it is not considered that the target anchor has started the preset action, and if the target anchor has ended the preset action, it is determined that the target anchor has completed a valid preset action, thereby achieving effective detection of the preset action of the target anchor. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a schematic diagram of an exemplary microphone connection interface;

[0066] Figure 2 is a flow chart of an action detection method in one embodiment;

[0067] Figure 3 is a schematic diagram of key points of an exemplary portrait;

[0068] Figure 4 is a flow chart of step 202 in one embodiment;

[0069] Figure 5 A schematic diagram of an exemplary waist-twisting action of a target anchor in a first live broadcast image and corresponding key points;

[0070] Figure 6 A schematic diagram of an exemplary waist-twisting action of a target anchor in a target second live broadcast image and corresponding key points;

[0071] Figure 7 It is a schematic diagram of a process in which a terminal obtains key point coordinates of each key point of a target anchor in a first live broadcast image according to a first live broadcast image in an embodiment;

[0072] Figure 8 is a schematic diagram of a network structure of an exemplary posture estimation model;

[0073] Fig. 9 is a flow chart of step 703 in one embodiment;

[0074] Fig.10 is a flow chart of step 704 in one embodiment;

[0075] Fig.11 It is a schematic diagram of a process in which a terminal obtains key point coordinates of key points of a target anchor in a first live broadcast image based on the similarity between the target image and a captured image of a previous frame in an embodiment;

[0076] Fig.12 Schematic diagram of a training process of a posture estimation model in one embodiment;

[0077] Fig.13 is a structural block diagram of a motion detection device in one embodiment;

[0078] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0080] With the popularization of computer equipment and the development of the Internet, live broadcasting based on the Internet has become more and more common in people's daily lives. Live broadcasting is an important form of live broadcasting. Through live broadcasting, different anchors can interact with each other. The live broadcasting interface of the anchor's terminal and the audience's terminal can simultaneously display the corresponding anchor room screens of different anchors.

[0081] For example, see Figure 1 , which is a schematic diagram of an exemplary microphone connection interface. Figure 1 The interface for connecting to the microphone shown includes two left and right anchor room screens. The anchor room screen on the left side of the interface for connecting to the microphone is the screen of the first anchor, and the anchor room screen on the right side of the interface for connecting to the microphone is the screen of the second anchor. During the process of connecting to the microphone with the first anchor and the second anchor, the first anchor and the second anchor can enhance the fun of the live broadcast through interactive methods such as talent PK. During the interactive process, the first anchor and the second anchor may need to perform interactive actions, for example, the first anchor and / or the second anchor may perform actions such as blinking, twisting the waist, etc.

[0082] Taking the first anchor's waist twisting action as an example, the terminal held by the first anchor needs to convert the corresponding PK score into the first anchor's PK score according to the number of effective waist twisting actions completed by the first anchor, which requires the terminal held by the first anchor to accurately detect the effective waist twisting actions completed by the first anchor. Therefore, how to accurately detect the effective actions completed by the anchor has become a problem that needs to be solved urgently.

[0083] In view of this, an embodiment of the present application provides an action detection method. In the action detection method, a first live broadcast image of a target anchor is obtained, and based on the first live broadcast image, key point coordinates of key points of the target anchor in the first live broadcast image are obtained. Then, based on the key point coordinates, whether the action amplitude of the target anchor for a preset action is greater than a first amplitude threshold is detected. If the action amplitude is greater than the first amplitude threshold, it is determined that the target anchor starts the preset action. Then, based on at least one second live broadcast image of the target anchor, it is detected whether the target anchor ends the preset action. In this way, when the action amplitude is large, it is determined that the target anchor starts the preset action. When the target anchor has a small human body movement, it is not considered that the target anchor starts the preset action. If the target anchor ends the preset action, it is determined that the target anchor completes a valid preset action, thereby achieving effective detection of the preset action of the target anchor.

[0084] The motion detection method provided in this application can be applied to the terminal of any anchor (hereinafter referred to as the target anchor). The terminal can be a personal computer, a laptop, a media player, a smart TV, a smart phone, a tablet computer, and a portable wearable device, etc. The embodiment of this application does not specifically limit the type of the terminal.

[0085] In one embodiment, Figure 2 As shown, a motion detection method is provided, comprising the following steps:

[0086] Step 201: obtain a first live broadcast image of a target anchor, and obtain key point coordinates of key points of the target anchor in the first live broadcast image according to the first live broadcast image.

[0087] In the embodiment of the present application, when the target anchor broadcasts live broadcast through the terminal, the terminal can shoot the target anchor to obtain the live broadcast video of the target anchor. When the terminal needs to detect the action of the target anchor, the video frame corresponding to the current moment is extracted from the live broadcast video, and the terminal obtains the first live broadcast image of the target anchor.

[0088] The timing for the terminal to perform action detection may be to receive an instruction input by the target anchor, such as an action counting instruction or an action scoring instruction, and then execute the action detection process. The timing for the terminal to perform action detection may also be to execute the action detection process after detecting a triggering event. For example, a triggering event may be: during the process of the target anchor being connected to the microphone or PKing with other anchors, if the terminal detects that the target anchor's PK score is lagging behind, then the action detection process is executed. In this application scenario, the terminal detects that the target anchor's PK score is lagging behind. The terminal may also display motion effects in the target anchor's anchor room screen to prompt the target anchor to perform a preset action, and then execute the action detection process. This embodiment does not limit the timing for the terminal to perform action detection.

[0089] The terminal obtains the key point coordinates of each key point of the target anchor in the first live broadcast image according to the first live broadcast image. Key points are also called joint points or skeleton points. A complete portrait includes 25 key points of the human skeleton. Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an exemplary key point of a complete portrait. Figure 3 As shown, the 25 key points include: nose 0, neck 1, right shoulder 2, right elbow 3, right wrist 4, left shoulder 5, left elbow 6, left wrist 7, sacrum 8, right waist 9, right knee 10, right ankle 11, left waist 12, left knee 13, left ankle 14, right eye 15, left eye 16, right ear 17, left ear 18, left toe one 19, left toe two 20, left heel 21, right toe one 22, right toe two 23, right heel 24.

[0090] In the embodiment of the present application, the key point coordinates of each key point of the target anchor in the first live broadcast image obtained by the terminal may be the coordinates of some or all of the above 25 key points, and no specific limitation is made here.

[0091] As an implementation mode, the terminal may extract features of the first live broadcast image and determine the key point coordinates of each key point through a posture estimation model based on a neural network. In another implementation mode, the terminal may also determine the key point coordinates of each key point in the live broadcast image adjacent to the first live broadcast image as the key point coordinates of each key point of the target anchor in the first live broadcast image, without specific limitation here.

[0092] Step 202: Detect, based on the coordinates of each key point, whether the target anchor's action amplitude for a preset action is greater than a first amplitude threshold.

[0093] The target anchor has different action amplitudes, and the key point coordinates of each key point of the target anchor in the first live broadcast image are also different. Therefore, the terminal can determine the size of the target anchor's action amplitude for the preset action according to the key point coordinates of each key point of the target anchor in the first live broadcast image.

[0094] Taking the preset action of twisting the waist as an example, the terminal calculates the angle between the target host's body axis and the vertical direction based on the coordinates of each key point. The larger the angle, the greater the amplitude of the target host's waist twisting action. Taking the squat action as an example, the terminal calculates the distance between the target host's knee key point and the ankle key point based on the coordinates of each key point. The closer the distance, the greater the amplitude of the target host's squat action, and so on. Based on parameters such as angle and distance, the terminal can quantify the size of the target host's action amplitude for the preset action.

[0095] The terminal detects whether the target anchor's action amplitude for the preset action is greater than a first amplitude threshold. For example, if the action amplitude is quantified by angle, the terminal detects whether the absolute value of the angle value is greater than the preset angle threshold, so as to determine whether the target anchor's action amplitude for the preset action is greater than the first amplitude threshold; if the action amplitude is quantified by the distance between key points, the terminal detects whether the absolute value of the calculated distance is greater than the preset distance threshold, so as to determine whether the target anchor's action amplitude for the preset action is greater than the first amplitude threshold.

[0096] Step 203: If the action amplitude is greater than the first amplitude threshold, it is determined that the target anchor starts the preset action, and based on at least one second live broadcast image of the target anchor, it is detected whether the target anchor ends the preset action.

[0097] If the terminal detects that the motion amplitude is greater than the first amplitude threshold, it is determined that the target anchor has started the preset motion. If the terminal detects that the motion amplitude is less than or equal to the first amplitude threshold, it is determined that the target anchor has not started the preset motion, and the terminal can obtain the adjacent live images after the first live image to continue to detect whether the target anchor has started the preset motion.

[0098] After the terminal detects that the target anchor starts the preset action, it detects whether the target anchor ends the preset action according to at least one second live broadcast image of the target anchor, and the first live broadcast image and the at least one second live broadcast image are adjacent in time sequence.

[0099] For example, at least one second live broadcast image includes second live broadcast image 1, second live broadcast image 2, second live broadcast image 3...second live broadcast image n, and the first live broadcast image, second live broadcast image 1, second live broadcast image 2, second live broadcast image 3...second live broadcast image n are continuous video frames in the live broadcast video of the target anchor. After the terminal detects from the first live broadcast image that the target anchor starts the preset action, it detects from the second live broadcast image 1 adjacent to the first live broadcast image whether the target anchor ends the preset action. If not, the terminal continues to detect from the second live broadcast image 2 adjacent to the second live broadcast image 1 whether the target anchor ends the preset action, and so on.

[0100] In a possible implementation manner, the terminal detects whether the target anchor has finished the preset action according to at least one second live broadcast image of the target anchor by executing the following steps A11 and A12:

[0101] Step A11: in accordance with the time sequence of the second live broadcast images from front to back, it is detected in turn whether the action amplitude of the target anchor for the preset action in each second live broadcast image is less than the second amplitude threshold.

[0102] In the embodiment of the present application, similar to the process in which the terminal detects whether the target anchor's action amplitude for a preset action is greater than a first amplitude threshold based on the coordinates of each key point, the terminal can quantify the size of the target anchor's action amplitude for the preset action in the second live broadcast image based on parameters such as the angle and distance corresponding to each key point.

[0103] After the terminal quantifies the magnitude of the target anchor's action amplitude for the preset action based on the second live broadcast image, the terminal detects whether the target anchor's action amplitude for the preset action in the second live broadcast image is less than a second amplitude threshold, and the second amplitude threshold is less than the first amplitude threshold.

[0104] Step A12: If it is detected that in the target second live broadcast image, the target anchor's action amplitude for the preset action is less than the second amplitude threshold, it is determined that the target anchor ends the preset action.

[0105] In the process of detecting the second live images in sequence, if the terminal detects from a second live image (i.e., the target second live image) that the target host's action amplitude for the preset action is less than the second amplitude threshold, it indicates that the target host's action amplitude has changed from a large amplitude in the first live image to a relatively small amplitude, thereby determining that the target host has ended the preset action.

[0106] Step 204: If the target anchor finishes the preset action, it is determined that the target anchor has completed the preset action once.

[0107] By executing step 203, if the terminal detects that the target anchor has finished the preset action in a second live broadcast image, the terminal determines that the target anchor has completed a preset action. In this way, the terminal completes the effective detection of a preset action of the target anchor.

[0108] In a possible implementation, the terminal may continue to detect whether the target anchor in the adjacent live image after the first live image starts the preset action, thereby realizing the counting of the preset action. Alternatively, the terminal may also calculate the cumulative number of preset actions according to the cumulative number of preset actions of the target anchor and convert the corresponding PK score, and so on.

[0109] For example, when the target anchor is conducting a live broadcast and PK with other anchors, if the terminal detects that the target anchor's PK score is lagging behind (for example, receiving a notification of the lagging score sent by the server), the terminal can display motion effects in the target anchor's anchor room screen to prompt the target anchor to perform preset actions. The terminal performs the above-mentioned action detection process, counts the valid preset actions completed by the target anchor, and converts the corresponding PK points according to the cumulative number of preset actions and accumulates them into the target anchor's PK score, thereby helping the target anchor to narrow or even surpass the PK score gap and improve the playability and fun of the live broadcast PK activity.

[0110] The above embodiment obtains a first live broadcast image of the target anchor, and obtains key point coordinates of key points of the target anchor in the first live broadcast image according to the first live broadcast image. Then, according to the coordinates of each key point, it is detected whether the target anchor's action amplitude for the preset action is greater than the first amplitude threshold. If the action amplitude is greater than the first amplitude threshold, it is determined that the target anchor starts the preset action. Then, according to at least one second live broadcast image of the target anchor, it is detected whether the target anchor ends the preset action. In this way, when the action amplitude is large, it is determined that the target anchor starts the preset action, and when the target anchor has a small human body movement, it is not considered that the target anchor starts the preset action. If the target anchor ends the preset action, it is determined that the target anchor completes a valid preset action, thereby achieving effective detection of the preset action of the target anchor.

[0111] In one embodiment, based on the above Figure 2 The embodiment shown, see Figure 4 This embodiment relates to a process of how the terminal detects whether the target anchor's action amplitude for a preset action is greater than a first amplitude threshold according to the coordinates of each key point. Figure 4 As shown, step 202 may include step 2021 and step 2022:

[0112] Step 2021, according to the coordinates of each key point, obtain the angle value between the body axis of the target anchor and the vertical direction in the first live broadcast image.

[0113] In this embodiment, taking the preset action of twisting the waist as an example, the terminal quantifies the magnitude of the target anchor's action for the preset action in the first live broadcast image according to the angle between the target anchor's body central axis and the vertical direction.

[0114] Based on the coordinates of each key point, the terminal obtains the angle value between the target host's body central axis and the vertical direction in the first live broadcast image. The angle value represents the degree to which the target host's body central axis deviates from the vertical direction. The larger the angle value, the greater the degree to which the target host's body central axis deviates from the vertical direction, that is, the greater the amplitude of the target host's waist twisting movement.

[0115] In a possible implementation of step 2021, the key point coordinates include left shoulder key point coordinates, right shoulder key point coordinates, left waist key point coordinates, and right waist key point coordinates. The terminal can implement the process of step 2021 by executing the following steps A21, A22, and A23:

[0116] Step A21, obtaining the first midpoint coordinates according to the left shoulder key point coordinates and the right shoulder key point coordinates.

[0117] The terminal calculates the coordinates of the midpoint between the left shoulder key point and the right shoulder key point, that is, the first midpoint coordinates, based on the coordinates of the left shoulder key point and the right shoulder key point.

[0118] Step A22, obtaining the second midpoint coordinates according to the left waist key point coordinates and the right waist key point coordinates.

[0119] The terminal calculates the coordinates of the midpoint between the left waist key point and the right waist key point, that is, the second midpoint coordinates, according to the coordinates of the left waist key point and the coordinates of the right waist key point.

[0120] Step A23, calculating the angle value between the body center axis of the target anchor in the first live broadcast image and the vertical direction according to the first midpoint coordinates and the second midpoint coordinates.

[0121] The line between the first midpoint coordinate and the second midpoint coordinate is the central axis of the target anchor's body. The terminal calculates the angle value between the central axis of the target anchor's body and the vertical direction in the first live broadcast image based on the first midpoint coordinate and the second midpoint coordinate.

[0122] See also Figure 5 , Figure 5 FIG. 1 is a schematic diagram of an exemplary waist-twisting action of a target anchor in a first live broadcast image and corresponding key points. Figure 5 As shown, area 1 is a schematic diagram of the target anchor's waist twisting action, and area 2 is a schematic diagram of the key points corresponding to area 1.

[0123] The coordinates of the left shoulder key point of the target anchor are The coordinates of the right shoulder key point are The coordinates of the midpoint between the left shoulder key point and the right shoulder key point, that is, the first midpoint coordinates are The coordinates of the left waist key points of the target anchor are The coordinates of the right waist key points are The coordinates of the midpoint between the left waist key point and the right waist key point, that is, the coordinates of the second midpoint are The terminal may use the following formula 1 to calculate the angle value of the angle θ between the body axis of the target anchor and the vertical direction in the first live broadcast image:

[0124]

[0125] in,

[0126] Thus, the terminal calculates the angle value between the central axis of the target anchor's body and the vertical direction in the first live broadcast image.

[0127] Step 2022: If the absolute value of the angle value is greater than the preset angle threshold, it is determined that the target anchor's action amplitude for the preset action is greater than the first amplitude threshold.

[0128] If the terminal detects that the absolute value of the angle value is greater than the preset angle threshold, it determines that the target anchor's action amplitude for the preset action is greater than the first amplitude threshold, that is, the target anchor's action amplitude for the preset action is large. The preset angle threshold can be set by itself during implementation. For example, the preset angle threshold has a value range of 8 to 15.

[0129] The terminal determines that the target anchor's action amplitude for the preset action is greater than the first amplitude threshold, and the terminal determines that the target anchor starts the preset action, and detects whether the target anchor ends the preset action according to at least one second live broadcast image of the target anchor.

[0130] In one possible implementation, see Figure 6 , Figure 6 FIG. 1 is a schematic diagram of an exemplary target host's waist twisting action and corresponding key points in the target second live broadcast image. Figure 5 Similar, such as Figure 6 As shown, area 1 is a schematic diagram of the target anchor's waist twisting action, and area 2 is a schematic diagram of the key points corresponding to area 1.

[0131] The terminal obtains the first midpoint coordinates according to the left shoulder key point coordinates and the right shoulder key point coordinates in the target second live image, and obtains the second midpoint coordinates according to the left waist key point coordinates and the right waist key point coordinates in the target second live image. Then, the terminal calculates the angle value between the target anchor's body axis and the vertical direction in the target second live image according to the first midpoint coordinates and the second midpoint coordinates in the target second live image. The process of the terminal calculating the angle value between the target anchor's body axis and the vertical direction in the target second live image is similar to the process of the above step 2021, and will not be repeated here.

[0132] Therefore, through the above steps, the terminal detects that the angle value between the target anchor's body axis and the vertical direction in the target second live broadcast image is less than or equal to the target angle threshold, then it is determined that the target anchor's action amplitude for the preset action is less than the second amplitude threshold, thereby determining that the target anchor has ended the preset action, and determining that the target anchor has completed a preset action. In this way, effective detection of the target anchor's preset action is achieved.

[0133] In one embodiment, in the above Figure 2 Based on the embodiment shown, see Figure 7 This embodiment relates to a process of how a terminal obtains the key point coordinates of each key point of a target anchor in a first live broadcast image according to a first live broadcast image. Figure 7 As shown, the process includes step 701, step 702, step 703 and step 704:

[0134] Step 701: Obtain target position frame coordinates corresponding to the target portrait and the first live broadcast image.

[0135] In the embodiment of the present application, the terminal obtains the target position frame coordinates corresponding to the target portrait and the first live image by inputting the first live image into a portrait detection model to obtain the target position frame coordinates corresponding to the target portrait and the first live image.

[0136] In another possible implementation of step 701, the terminal may implement the process of step 701 by executing the following steps A31 and A32:

[0137] A31, obtain the coordinates of the portrait detection position frame.

[0138] The portrait detection position frame coordinates may be position frame coordinates corresponding to a target portrait detected by the terminal from a target live image before the first live image using a portrait detection model.

[0139] A32, enlarging the portrait detection position frame coordinates according to a preset enlargement ratio to obtain the target position frame coordinates.

[0140] That is, the terminal uses the portrait detection model to perform portrait detection on the target live image, obtains the portrait detection position frame coordinates corresponding to the target portrait in the target live image, the terminal obtains the portrait detection position frame coordinates, and enlarges the portrait detection position frame coordinates according to a preset magnification ratio to obtain the target position frame coordinates. For example, the terminal enlarges the portrait detection position frame coordinates by 1.1 to 1.3 times to obtain the target position frame coordinates. As an implementation method, the number of live image frames between the target live image and the first live image can be between 5 and 15.

[0141] In the embodiment of the present application, when the terminal performs action detection, it is necessary to capture the portrait area of ​​the target anchor from the live image. When the terminal detects each frame of the live image, it can use the portrait detection model to detect the portrait detection position frame coordinates once every preset number of frames. The preset number of frames can be, for example, 5 to 15 frames. For example, the terminal can use the portrait detection model to detect the portrait detection position frame coordinates in the live image of the first frame, use the portrait detection model to detect the portrait detection position frame coordinates in the live image of the 11th frame, use the portrait detection model to detect the portrait detection position frame coordinates in the live image of the 21st frame, and so on. In this way, the terminal can enlarge the portrait detection position frame coordinates in the live image of the first frame according to the preset magnification ratio to obtain the portrait detection position frame coordinates of the second frame to the ninth frame, that is, the target position frame coordinates; similarly, the terminal can enlarge the portrait detection position frame coordinates in the live image of the 11th frame according to the preset magnification ratio to obtain the portrait detection position frame coordinates of the 12th frame to the 19th frame, and so on.

[0142] In this way, the terminal does not need to perform portrait detection on each frame of the live image, which greatly reduces the terminal's computational workload and improves the efficiency of terminal motion detection.

[0143] In an embodiment of the present application, the portrait detection model is generated by training a machine learning model with MnasNet as the backbone network and yolov3 as the detection framework after channel pruning. Channel pruning is beneficial to reducing the detection calculation amount of the portrait detection model and saving the computing resources of the terminal.

[0144] The length and width values ​​of the 9 anchors corresponding to the portrait detection model can be obtained by first scaling all training sample images of the portrait detection model according to the input size of the portrait detection model by the terminal to obtain multiple real portrait position frames, and then clustering the sizes of the multiple real portrait position frames by the terminal using the KMeans clustering algorithm, which is beneficial to improving the accuracy of the portrait detection model.

[0145] Step 702: capture a target image from the first live image according to the target position frame coordinates.

[0146] Step 703: input the target image into the posture estimation model to obtain the heat map of each key point.

[0147] The terminal captures the target image from the first live image according to the target position frame coordinates, and then inputs the target image into the posture estimation model to obtain a heat map of each key point.

[0148] In an embodiment of the present application, the posture estimation model can be a pruned neural network model based on Resnet18+FPN.

[0149] See also Figure 8 , Figure 8 A schematic diagram of the network structure of an exemplary posture estimation model.

[0150] like Figure 8 As shown, the exemplary posture estimation model is a two-stage model, each stage is a Resnet18+FPN framework, the network layer labeled ① is convolution layer 1 (conv1), the network layer labeled ② is convolution layer 2 (conv1), the network layer labeled ③ is convolution layer 3 (conv3), the network layer labeled ④ is an hourglass convolution layer (Bottleneck conv), the network layer labeled ⑤ is an upsample layer (Upsample), and the network layer labeled ⑥ is a posehead layer.

[0151] The terminal inputs the target image from Stage 1 into the posture estimation model. The posture estimation model maps the last layer of feature map of Stage 2 through a separable convolutional layer to obtain the heat map of each key point.

[0152] In other embodiments, the posture estimation model may also include more or less stages than two, and no specific limitation is made here.

[0153] Step 704, obtaining the key point coordinates of each key point according to the heat map of each key point.

[0154] The terminal inputs the target image into the posture estimation model. After obtaining the heat map of each key point, the terminal obtains the key point coordinates of each key point based on the heat map of each key point. Each key point corresponds to a heat map.

[0155] In a possible implementation of step 704, the terminal may perform the following steps A41 and A42 to implement the process of step 704:

[0156] Step A41, for each key point's heat map, obtain the target coordinates of the key point in the target image according to the heat map.

[0157] For a key point in the heat map, the terminal uses the soft-argmax algorithm to regress and obtain the normalized target coordinates of the key point. By using the soft-argmax algorithm, the accuracy of the target coordinates can be improved.

[0158] Step A42: acquiring the key point coordinates of the key point in the first live image according to the position of the target image in the first live image and the target coordinates of the key point in the target image.

[0159] The target coordinates are the coordinates of the key point in the target image. The terminal obtains the key point coordinates of the key point in the first live image according to the position of the target image in the first live image and the target coordinates of the key point in the target image.

[0160] As an implementation method, the terminal determines the position of the target image in the first live image, thereby obtaining the coordinates of the upper left corner of the target image in the first live image. The terminal accumulates the target coordinates of the key point in the target image and the coordinates of the upper left corner to obtain the coordinates of the key point in the first live image.

[0161] In the embodiment of the present application, the terminal obtains the target position frame coordinates corresponding to the target portrait and the first live image; according to the target position frame coordinates, the target image is captured from the first live image; the target image is input into the posture estimation model to obtain the heat map of each key point. Since the posture estimation model uses a neural network model cropped based on Resnet18+FPN, the terminal's computing resources can be reduced, the speed at which the terminal obtains the key point coordinates of each key point is increased, and the speed of the terminal's action detection is increased. In addition, since the demand for computing resources is small, even if the terminal's CPU performance is poor, fast action detection can be achieved, thereby improving the real-time performance of action detection.

[0162] In one embodiment, in another possible implementation of step 703, see Fig. 9 The terminal can also execute Fig. 9 Steps 7031, 7032 and 7033 shown implement the process of step 703:

[0163] Step 7031, input the target image into the affine transformation model to obtain the affine transformation parameters corresponding to the target image.

[0164] In the embodiment of the present application, in order to avoid errors in the coordinates of the target position frame, the terminal corrects the captured target image to improve the accuracy of the key point coordinates.

[0165] The terminal inputs the target image into the affine transformation model to obtain the affine transformation parameters corresponding to the target image. In an embodiment of the present application, the affine transformation model may include a plurality of ordinary convolutional layers connected in sequence, for example, 10 3x3 convolutional layers, each of which is sequentially connected to a normalized BN layer and a ReLU activation layer, and finally connected to two 1x1 convolutional layers, and the number of output channels of the last convolutional layer is 6. The terminal inputs the target image into the affine transformation model to obtain the affine transformation parameters corresponding to the target image. The affine transformation parameters are 6 parameters, representing parameters such as translation, rotation, scaling, shearing, and reflection.

[0166] The terminal can correct the target image according to the affine transformation parameters so that the portrait of the target anchor is located in the middle of the target image, thereby improving the accuracy of motion detection.

[0167] Step 7032, using affine transformation parameters to perform affine transformation on the target image to obtain a corrected image.

[0168] Step 7033, input the corrected image into the posture estimation model to obtain the heat map of each key point.

[0169] The terminal uses affine transformation parameters to perform affine transformation on the target image to obtain a corrected image. Then, the terminal inputs the corrected image into the posture estimation model to obtain the heat map of each key point.

[0170] Since the target image has undergone an affine transformation, the terminal needs to perform an inverse transformation corresponding to the affine transformation on the corrected coordinates of the key points in the corrected image obtained based on the heat map in step 7033 in order to convert the corrected coordinates of the key points in the corrected image into the key point coordinates of the key points in the first live image.

[0171] In one possible approach at step 704, Fig. 9 Based on the embodiment shown, see Fig.10 , step 704 may include: Fig.10 Step 7041, step 7042 and step 7043 shown:

[0172] Step 7041, for each key point's heat map, obtain the corrected coordinates of the key point in the corrected image according to the heat map.

[0173] For the heat map of each key point, the process of the terminal obtaining the corrected coordinates of the key point in the corrected image according to the heat map is similar to step A41. For the heat map of a key point, the terminal uses the soft-argmax algorithm to regress and obtain the normalized corrected coordinates of the key point in the corrected image.

[0174] Step 7042, performing an inverse transformation on the corrected coordinates according to the affine transformation parameters to obtain the target coordinates of the key points in the target image.

[0175] The terminal performs an inverse coordinate transformation on the correction coordinates according to the affine transformation parameters to obtain the target coordinates of the key points in the target image.

[0176] Step 7043: Acquire the key point coordinates of the key point in the first live image according to the position of the target image in the first live image and the target coordinates of the key point in the target image.

[0177] The terminal determines the position of the target image in the first live image, thereby obtaining the coordinates of the upper left corner of the target image in the first live image. The terminal adds the target coordinates of the key point in the target image to the coordinates of the upper left corner to obtain the coordinates of the key point in the first live image.

[0178] In an embodiment of the present application, by using affine transformation parameters to correct the target image, it is avoided that the extracted key point coordinates produce errors due to errors in the target position frame coordinates, thereby affecting the accuracy of the terminal's motion detection. The terminal corrects the target image according to the affine transformation parameters so that the portrait of the target anchor is in the middle position of the target image, thereby improving the accuracy of motion detection.

[0179] In one embodiment, based on the above Figure 7 The embodiment shown, see Fig.11 This embodiment relates to a process in which a terminal obtains the key point coordinates of each key point of a target anchor in a first live broadcast image based on the similarity between the target image and the previous frame captured image. Fig.11 As shown, the process includes step 801, step 802 and step 803:

[0180] Step 801, detecting the similarity between the target image and the previous frame captured image.

[0181] The previous frame captured image is an image corresponding to the target portrait captured from the third live image before the first live image, and the third live image is adjacent to the first target image in time sequence.

[0182] The terminal obtains the first live broadcast image of the target anchor, then obtains the target position frame coordinates corresponding to the target portrait and the first live broadcast image, and captures the target image from the first live broadcast image according to the target position frame coordinates. The terminal detects the similarity between the target image and the previous frame captured image, which is obtained by the terminal from the third live broadcast image adjacent to the first live broadcast image according to the portrait detection position frame coordinates corresponding to the target portrait.

[0183] In a possible implementation, step 804 may include the following steps A51, A52, A53, A54, and A55:

[0184] Step A51, performing hash processing on the target image to obtain a first hash value.

[0185] Step A52, performing hash processing on the previous frame captured image to obtain a second hash value.

[0186] Step A53: Calculate the absolute value of the difference between the first Hash value and the second Hash value.

[0187] Step A54: if the absolute value of the difference is greater than the difference threshold, then the similarity between the target image and the previous frame captured image is less than the similarity threshold.

[0188] Step A55: if the absolute value of the difference is less than or equal to the difference threshold, it is determined that the similarity between the target image and the previous frame captured image is greater than or equal to the similarity threshold.

[0189] In an embodiment of the present application, the terminal may use a differential hash algorithm to perform hash processing on the target image to obtain a first hash value, and perform hash processing on the previous frame captured image to obtain a second hash value. The terminal calculates the absolute value of the difference between the first hash value and the second hash value. If the absolute value of the difference is greater than a difference threshold, the terminal determines that the similarity between the target image and the previous frame captured image is less than a similarity threshold, that is, the similarity between the target image and the previous frame captured image is small; if the absolute value of the difference is less than or equal to the difference threshold, the terminal determines that the similarity between the target image and the previous frame captured image is greater than or equal to the similarity threshold, that is, the similarity between the target image and the previous frame captured image is large, and the target image and the previous frame captured image are highly similar.

[0190] Step 802: If the similarity between the target image and the image captured in the previous frame is less than the similarity threshold, the target image is input into the posture estimation model to obtain a heat map of each key point.

[0191] If the similarity between the target image and the previous frame captured image is less than the similarity threshold, it means that the similarity between the target image and the previous frame captured image is small. Therefore, the key point coordinates of the previous frame captured image have no reference to the target image. The terminal will Figure 7 In the illustrated embodiment, the terminal inputs the target image into the posture estimation model to obtain a heat map of each key point, and obtains the key point coordinates of each key point based on the heat map of each key point.

[0192] Step 803: If the similarity between the target image and the previous frame captured image is greater than or equal to the similarity threshold, the key point coordinates of the key points of the target anchor in the third live broadcast image are determined as the key point coordinates of the key points of the target anchor in the first live broadcast image.

[0193] If the similarity between the target image and the previous frame captured image is greater than or equal to the similarity threshold, it indicates that the target image and the previous frame captured image are highly similar, so that the terminal determines that the positions of the key points of the first live image remain unchanged from the previous frame, that is, the key points in the third live image remain unchanged. Therefore, the terminal determines the key point coordinates of the key points of the target anchor in the third live image as the key point coordinates of the key points of the target anchor in the first live image. In this way, the terminal does not need to recalculate the key point coordinates of the key points of the target anchor in the first live image, further reducing the amount of data calculation of the terminal, improving the speed of the terminal in obtaining the key point coordinates of each key point, and improving the speed of terminal action detection.

[0194] In one embodiment, based on the above Figure 7 The embodiment shown, see Fig.12 , this embodiment relates to the training process of the posture estimation model. Fig.12 As shown, the process includes step 901, step 902 and step 903:

[0195] Step 901, obtaining a sample target image.

[0196] Step 902: Perform affine transformation on the sample target image to obtain a dual sample image.

[0197] Step 903: train an initial pose estimation model based on the sample target image and the dual sample image to obtain a pose estimation model.

[0198] In the embodiment of the present application, in order to improve the richness of the training samples, the terminal performs an affine transformation on the acquired sample target image to obtain a dual sample image. The terminal trains an initial posture estimation model based on the sample target image and the dual sample image to obtain a posture estimation model, so that the posture estimation model can learn the features of the image after the affine transformation during the training process, so that in the application process of the posture estimation model, it can have good robustness to the live broadcast image with small-scale jitter such as translation, scaling or rotation.

[0199] During the training process of the posture estimation model, in an intermediate iteration process, the sample target image and the dual sample image are input into the intermediate posture estimation model to obtain the first predicted key point coordinates corresponding to the sample target image and the second predicted key point coordinates corresponding to the dual sample image; an affine transformation is performed on the first predicted key point coordinates to obtain the corrected predicted point coordinates; the loss value corresponding to the current iteration process is calculated based on the second predicted key point coordinates and the corrected predicted point coordinates; and the model parameters of the intermediate posture estimation model are adjusted based on the loss value.

[0200] In the embodiment of the present application, in order to perform learning constraints during the training process of the posture estimation model, so that the posture estimation model can learn more features of jitter within a small range such as translation, scaling or rotation of the training sample image, the loss function loss of the posture estimation model is total Add affine transformation consistency loss function loss aff , loss aff It is obtained by the terminal based on the second predicted key point coordinates and the corrected predicted point coordinates.

[0201] For example, the sample target image is I a , I a The key point coordinates of each key point of the corresponding portrait are (x a1 ,y a1 ),(x a2,y a2 ),...,(x aK ,y aK ), where K is the number of key points. The terminal transforms the sample target image into I according to the affine transformation matrix a After affine transformation, we get the dual sample image I b , I b The key point coordinates of each key point are (x b1 ,y b1 ),(x b2 ,y b2 ),...,(x bK ,y bK ), the affine transformation matrix is

[0202] The terminal sends the sample target image I a Input intermediate pose estimation model, I a The coordinates of the first predicted key point predicted by the intermediate posture estimation model are (x′ a1 ,y′ a1 ),(x′ a2 ,y′ a2 ),...,(x′ aK ,y′ aK ), the terminal uses an affine transformation matrix to transform the first predicted key point coordinates according to the following formula 2 to obtain the corrected predicted point coordinates (x″ b1 ,y″ b1 ),(x″ b2 ,y″ b2 ),...,(x″ bK ,y″ bK ):

[0203]

[0204] The terminal will pair the sample image I b Input intermediate pose estimation model, I b The coordinates of the second predicted key point predicted by the intermediate posture estimation model are (x′ b1 ,y′ b1 ),(x′ b2 ,y′ b2 ),...,(x′ bK ,y′ bK ). Affine transformation consistency loss function loss aff for:

[0205]

[0206] Loss function of the pose estimation model totalThe expression is:

[0207] loss total =λ a *loss aff +loss a +loss b +loss ma +loss mb

[0208] Among them, λ a is the weight of the affine transformation consistency loss function, ranging from 1 to 10; loss a 、loss bmb They are sample target images I a and dual sample image I b The actual key point coordinates and predicted key point coordinates are in Figure 8 The Euclidean distance loss function in Stage 1 shown, loss ma 、loss mb is the sample target image I a and dual sample image I b The actual key point coordinates and predicted key point coordinates are in Figure 8 The OHKM (Online Hard Keypoints Mining) distance loss function in Stage 2 is shown.

[0209] Take the sample target image I a OHKM distance loss function loss ma For example, the terminal first converts the sample target image I a The Euclidean distances between the actual key point coordinates and the predicted key point coordinates are sorted in descending order. The terminal selects the first σ Euclidean distances with the largest sorted Euclidean distances to participate in the OHKM distance calculation. The value range of σ is 0.25*M*K~M*K, where K is the number of key points and M is the size of the training sample set, which ranges from 32 to 256.

[0210] In this way, the terminal is based on the loss total The expression calculates the loss value corresponding to the current iteration process, and adjusts the model parameters of the intermediate posture estimation model according to the loss value. The posture estimation model is obtained by repeatedly iterating the training until the model converges.

[0211] Therefore, the terminal inputs the target image into the posture estimation model, and then obtains the heat map of each key point. The terminal obtains the key point coordinates of each key point based on the heat map of each key point. Since the posture estimation model uses affine transformation consistency loss function for learning constraints during the training process, the posture estimation model can fully learn the jitter within a small range such as translation, scaling or rotation of the sample image, which is beneficial to improve the prediction accuracy of the posture estimation model and the accuracy of the key point coordinates of each key point. According to the key point coordinates of each key point, the action amplitude of the target anchor for the preset action is detected, thereby realizing accurate detection of effective preset actions for the target anchor.

[0212] It should be understood that, although the various steps in the above-mentioned flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flowchart may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0213] In one embodiment, Fig.13 As shown, a motion detection device is provided, comprising:

[0214] The acquisition module 10 is used to acquire a first live broadcast image of a target anchor, and acquire key point coordinates of key points of the target anchor in the first live broadcast image according to the first live broadcast image;

[0215] A first detection module 20 is used to detect whether the action amplitude of the target anchor for the preset action is greater than a first amplitude threshold according to the coordinates of each key point;

[0216] A second detection module 30 is used to determine that the target anchor starts the preset action if the action amplitude is greater than the first amplitude threshold, and detect whether the target anchor ends the preset action according to at least one second live broadcast image of the target anchor, wherein the first live broadcast image and the at least one second live broadcast image are sequentially adjacent in time sequence;

[0217] The determination module 40 is used to determine that the target anchor has completed the preset action once if the target anchor finishes the preset action.

[0218] Optionally, the second detection module 30 includes:

[0219] A first detection unit is configured to detect, in order from front to back of each of the second live broadcast images, whether the action amplitude of the target anchor for the preset action in each of the second live broadcast images is less than a second amplitude threshold, and the second amplitude threshold is less than the first amplitude threshold;

[0220] The first determining unit determines that the target anchor ends the preset action if it is detected that the action amplitude of the target anchor for the preset action in the target second live broadcast image is less than the second amplitude threshold.

[0221] Optionally, the first detection module 20 includes:

[0222] A first acquisition unit is used to acquire an angle value between the body central axis of the target anchor and the vertical direction in the first live broadcast image according to the coordinates of each key point;

[0223] The second determining unit is configured to determine that the action amplitude of the target anchor for the preset action is greater than the first amplitude threshold if the absolute value of the angle value is greater than a preset angle threshold.

[0224] Optionally, each of the key point coordinates includes a left shoulder key point coordinate, a right shoulder key point coordinate, a left waist key point coordinate and a right waist key point coordinate, and the acquisition unit is specifically used to acquire a first midpoint coordinate according to the left shoulder key point coordinate and the right shoulder key point coordinate; acquire a second midpoint coordinate according to the left waist key point coordinate and the right waist key point coordinate; and calculate an angle value between the central axis of the body of the target anchor in the first live broadcast image and the vertical direction according to the first midpoint coordinate and the second midpoint coordinate.

[0225] Optionally, the acquisition module 10 includes:

[0226] A second acquisition unit is used to acquire the target position frame coordinates corresponding to the target portrait and the first live image;

[0227] A capture unit, configured to capture a target image from the first live image according to the target position frame coordinates;

[0228] A prediction unit, used for inputting the target image into a posture estimation model to obtain a heat map of each key point;

[0229] The third acquisition unit is used to acquire the key point coordinates of each key point according to the heat map of each key point.

[0230] Optionally, the second acquisition unit is specifically used to obtain portrait detection position frame coordinates, where the portrait detection position frame coordinates are position frame coordinates corresponding to the target portrait detected from the target live image before the first live image using a portrait detection model; the portrait detection position frame coordinates are enlarged according to a preset magnification ratio to obtain the target position frame coordinates.

[0231] Optionally, the third acquisition unit is specifically used to obtain the target coordinates of the key point in the target image according to the heat map for each key point; and obtain the key point coordinates of the key point in the first live image according to the position of the target image in the first live image and the target coordinates of the key point in the target image.

[0232] Optionally, the prediction unit is specifically used to input the target image into an affine transformation model to obtain affine transformation parameters corresponding to the target image; perform affine transformation on the target image using the affine transformation parameters to obtain a corrected image; input the corrected image into the posture estimation model to obtain a heat map of each of the key points.

[0233] Optionally, the third acquisition unit is specifically used to obtain the corrected coordinates of the key point in the corrected image according to the heat map for each key point; inversely transform the corrected coordinates according to the affine transformation parameters to obtain the target coordinates of the key point in the target image; and obtain the key point coordinates of the key point in the first live image according to the position of the target image in the first live image and the target coordinates of the key point in the target image.

[0234] Optionally, the device further comprises:

[0235] a third detection module, configured to detect a similarity between the target image and a previous frame captured image, wherein the previous frame captured image is an image corresponding to the target portrait captured from a third live image before the first live image, and the third live image is adjacent to the first target image in time sequence;

[0236] An execution module is used to execute the step of inputting the target image into the posture estimation model to obtain the heat map of each key point if the similarity between the target image and the captured image of the previous frame is less than a similarity threshold.

[0237] Optionally, the third detection module includes:

[0238] A first processing unit, configured to perform hash processing on the target image to obtain a first hash value;

[0239] A second processing unit, configured to perform hash processing on the previous frame captured image to obtain a second hash value;

[0240] a calculation unit, configured to calculate an absolute value of a difference between the first Hash value and the second Hash value;

[0241] A third determining unit, configured to determine that the similarity between the target image and the previous frame captured image is less than the similarity threshold if the absolute value of the difference is greater than the difference threshold;

[0242] The fourth determining unit is configured to determine that the similarity between the target image and the previous frame captured image is greater than or equal to the similarity threshold if the absolute value of the difference is less than or equal to the difference threshold.

[0243] Optionally, the device further comprises:

[0244] A determination module is used to determine the key point coordinates of each key point of the target anchor in the third live image as the key point coordinates of each key point of the target anchor in the first live image if the similarity between the target image and the previous frame captured image is greater than or equal to the similarity threshold.

[0245] Optionally, the device further comprises:

[0246] A sample acquisition module is used to acquire a sample target image;

[0247] An affine transformation module, used for performing an affine transformation on the sample target image to obtain a dual sample image;

[0248] A training module, used for training an initial posture estimation model according to the sample target image and the dual sample image to obtain the posture estimation model;

[0249] Among them, in an intermediate iteration process, the sample target image and the dual sample image are input into the intermediate posture estimation model to obtain the first predicted key point coordinates corresponding to the sample target image and the second predicted key point coordinates corresponding to the dual sample image; the first predicted key point coordinates are affine transformed to obtain the corrected predicted point coordinates; the loss value corresponding to the current iteration process is calculated based on the second predicted key point coordinates and the corrected predicted point coordinates; and the model parameters of the intermediate posture estimation model are adjusted based on the loss value.

[0250] For the specific definition of the action detection device, please refer to the definition of the action detection method above, which will not be repeated here. Each module in the above-mentioned action detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0251] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.14 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an action detection method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0252] Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0253] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0254] Acquire a first live broadcast image of a target anchor, and acquire key point coordinates of key points of the target anchor in the first live broadcast image according to the first live broadcast image;

[0255] According to the coordinates of each of the key points, detecting whether the target anchor's action amplitude for the preset action is greater than a first amplitude threshold;

[0256] If the action amplitude is greater than the first amplitude threshold, it is determined that the target anchor starts the preset action, and whether the target anchor ends the preset action is detected according to at least one second live broadcast image of the target anchor, and the first live broadcast image and the at least one second live broadcast image are sequentially adjacent in time sequence;

[0257] If the target anchor finishes the preset action, it is determined that the target anchor completes the preset action once.

[0258] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0259] According to the time sequence of each of the second live broadcast images from front to back, it is detected in turn whether the action amplitude of the target anchor for the preset action in each of the second live broadcast images is less than a second amplitude threshold, and the second amplitude threshold is less than the first amplitude threshold;

[0260] If it is detected that in the target second live broadcast image, the action amplitude of the target anchor for the preset action is less than the second amplitude threshold, it is determined that the target anchor ends the preset action.

[0261] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0262] According to the coordinates of each of the key points, an angle value between the central axis of the body of the target anchor in the first live broadcast image and the vertical direction is obtained;

[0263] If the absolute value of the angle value is greater than the preset angle threshold, it is determined that the action amplitude of the target anchor for the preset action is greater than the first amplitude threshold.

[0264] In one embodiment, the key point coordinates include left shoulder key point coordinates, right shoulder key point coordinates, left waist key point coordinates and right waist key point coordinates, and the processor further implements the following steps when executing the computer program:

[0265] Acquire the first midpoint coordinates according to the left shoulder key point coordinates and the right shoulder key point coordinates;

[0266] Acquire the second midpoint coordinates according to the left waist key point coordinates and the right waist key point coordinates;

[0267] According to the first midpoint coordinates and the second midpoint coordinates, the angle value between the central axis of the body of the target anchor in the first live broadcast image and the vertical direction is calculated.

[0268] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0269] Acquire target position frame coordinates corresponding to the target portrait and the first live image;

[0270] According to the target position frame coordinates, intercepting a target image from the first live image;

[0271] Inputting the target image into a posture estimation model to obtain a heat map of each key point;

[0272] According to the heat map of each of the key points, the key point coordinates of each of the key points are obtained.

[0273] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0274] Acquire portrait detection position frame coordinates, where the portrait detection position frame coordinates are position frame coordinates corresponding to the target portrait detected from a target live image before the first live image using a portrait detection model;

[0275] The portrait detection position frame coordinates are enlarged according to a preset enlargement ratio to obtain the target position frame coordinates.

[0276] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0277] For each of the key point's heat maps, obtaining the target coordinates of the key point in the target image according to the heat map;

[0278] According to the position of the target image in the first live image and the target coordinates of the key point in the target image, the key point coordinates of the key point in the first live image are acquired.

[0279] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0280] Inputting the target image into an affine transformation model to obtain affine transformation parameters corresponding to the target image;

[0281] Performing an affine transformation on the target image using the affine transformation parameters to obtain a corrected image;

[0282] The corrected image is input into the posture estimation model to obtain a heat map of each key point.

[0283] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0284] For each of the key point's heat maps, obtaining the corrected coordinates of the key point in the corrected image according to the heat map;

[0285] According to the affine transformation parameters, the corrected coordinates are inversely transformed to obtain the target coordinates of the key point in the target image;

[0286] According to the position of the target image in the first live image and the target coordinates of the key point in the target image, the key point coordinates of the key point in the first live image are acquired.

[0287] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0288] detecting a similarity between the target image and a previous frame captured image, wherein the previous frame captured image is an image corresponding to the target portrait captured from a third live image before the first live image, and the third live image is adjacent to the first target image in time sequence;

[0289] If the similarity between the target image and the previous frame captured image is less than the similarity threshold, the step of inputting the target image into the posture estimation model to obtain the heat map of each key point is performed.

[0290] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0291] Performing hash processing on the target image to obtain a first hash value;

[0292] Performing hash processing on the previous frame captured image to obtain a second hash value;

[0293] Calculate an absolute value of a difference between the first Hash value and the second Hash value;

[0294] If the absolute value of the difference is greater than the difference threshold, determining that the similarity between the target image and the previous frame captured image is less than the similarity threshold;

[0295] If the absolute value of the difference is less than or equal to the difference threshold, it is determined that the similarity between the target image and the previous frame captured image is greater than or equal to the similarity threshold.

[0296] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0297] If the similarity between the target image and the captured image of the previous frame is greater than or equal to the similarity threshold, the key point coordinates of each key point of the target anchor in the third live image are determined as the key point coordinates of each key point of the target anchor in the first live image.

[0298] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0299] Get a sample target image;

[0300] Performing an affine transformation on the sample target image to obtain a dual sample image;

[0301] Training an initial pose estimation model according to the sample target image and the dual sample image to obtain the pose estimation model;

[0302] Among them, in an intermediate iteration process, the sample target image and the dual sample image are input into the intermediate posture estimation model to obtain the first predicted key point coordinates corresponding to the sample target image and the second predicted key point coordinates corresponding to the dual sample image; the first predicted key point coordinates are affine transformed to obtain the corrected predicted point coordinates; the loss value corresponding to the current iteration process is calculated based on the second predicted key point coordinates and the corrected predicted point coordinates; and the model parameters of the intermediate posture estimation model are adjusted based on the loss value.

[0303] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0304] Acquire a first live broadcast image of a target anchor, and acquire key point coordinates of key points of the target anchor in the first live broadcast image according to the first live broadcast image;

[0305] According to the coordinates of each of the key points, detecting whether the target anchor's action amplitude for the preset action is greater than a first amplitude threshold;

[0306] If the action amplitude is greater than the first amplitude threshold, it is determined that the target anchor starts the preset action, and whether the target anchor ends the preset action is detected according to at least one second live broadcast image of the target anchor, and the first live broadcast image and the at least one second live broadcast image are sequentially adjacent in time sequence;

[0307] If the target anchor finishes the preset action, it is determined that the target anchor completes the preset action once.

[0308] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0309] According to the time sequence of each of the second live broadcast images from front to back, it is detected in turn whether the action amplitude of the target anchor for the preset action in each of the second live broadcast images is less than a second amplitude threshold, and the second amplitude threshold is less than the first amplitude threshold;

[0310] If it is detected that in the target second live broadcast image, the action amplitude of the target anchor for the preset action is less than the second amplitude threshold, it is determined that the target anchor ends the preset action.

[0311] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0312] According to the coordinates of each of the key points, an angle value between the central axis of the body of the target anchor in the first live broadcast image and the vertical direction is obtained;

[0313] If the absolute value of the angle value is greater than the preset angle threshold, it is determined that the action amplitude of the target anchor for the preset action is greater than the first amplitude threshold.

[0314] In one embodiment, the key point coordinates include left shoulder key point coordinates, right shoulder key point coordinates, left waist key point coordinates and right waist key point coordinates, and the computer program further implements the following steps when executed by the processor:

[0315] Acquire the first midpoint coordinates according to the left shoulder key point coordinates and the right shoulder key point coordinates;

[0316] Acquire the second midpoint coordinates according to the left waist key point coordinates and the right waist key point coordinates;

[0317] According to the first midpoint coordinates and the second midpoint coordinates, the angle value between the central axis of the body of the target anchor in the first live broadcast image and the vertical direction is calculated.

[0318] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0319] Acquire target position frame coordinates corresponding to the target portrait and the first live image;

[0320] According to the target position frame coordinates, intercepting a target image from the first live image;

[0321] Inputting the target image into a posture estimation model to obtain a heat map of each key point;

[0322] According to the heat map of each of the key points, the key point coordinates of each of the key points are obtained.

[0323] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0324] Acquire portrait detection position frame coordinates, where the portrait detection position frame coordinates are position frame coordinates corresponding to the target portrait detected from a target live image before the first live image using a portrait detection model;

[0325] The portrait detection position frame coordinates are enlarged according to a preset enlargement ratio to obtain the target position frame coordinates.

[0326] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0327] For each of the key point's heat maps, obtaining the target coordinates of the key point in the target image according to the heat map;

[0328] According to the position of the target image in the first live image and the target coordinates of the key point in the target image, the key point coordinates of the key point in the first live image are acquired.

[0329] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0330] Inputting the target image into an affine transformation model to obtain affine transformation parameters corresponding to the target image;

[0331] Performing an affine transformation on the target image using the affine transformation parameters to obtain a corrected image;

[0332] The corrected image is input into the posture estimation model to obtain a heat map of each key point.

[0333] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0334] For each of the key point's heat maps, obtaining the corrected coordinates of the key point in the corrected image according to the heat map;

[0335] According to the affine transformation parameters, the corrected coordinates are inversely transformed to obtain the target coordinates of the key point in the target image;

[0336] According to the position of the target image in the first live image and the target coordinates of the key point in the target image, the key point coordinates of the key point in the first live image are acquired.

[0337] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0338] detecting a similarity between the target image and a previous frame captured image, wherein the previous frame captured image is an image corresponding to the target portrait captured from a third live image before the first live image, and the third live image is adjacent to the first target image in time sequence;

[0339] If the similarity between the target image and the previous frame captured image is less than the similarity threshold, the step of inputting the target image into the posture estimation model to obtain the heat map of each key point is performed.

[0340] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0341] Performing hash processing on the target image to obtain a first hash value;

[0342] Performing hash processing on the previous frame captured image to obtain a second hash value;

[0343] Calculate an absolute value of a difference between the first Hash value and the second Hash value;

[0344] If the absolute value of the difference is greater than the difference threshold, determining that the similarity between the target image and the previous frame captured image is less than the similarity threshold;

[0345] If the absolute value of the difference is less than or equal to the difference threshold, it is determined that the similarity between the target image and the previous frame captured image is greater than or equal to the similarity threshold.

[0346] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0347] If the similarity between the target image and the captured image of the previous frame is greater than or equal to the similarity threshold, the key point coordinates of each key point of the target anchor in the third live image are determined as the key point coordinates of each key point of the target anchor in the first live image.

[0348] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0349] Get a sample target image;

[0350] Performing an affine transformation on the sample target image to obtain a dual sample image;

[0351] Training an initial pose estimation model according to the sample target image and the dual sample image to obtain the pose estimation model;

[0352] Among them, in an intermediate iteration process, the sample target image and the dual sample image are input into the intermediate posture estimation model to obtain the first predicted key point coordinates corresponding to the sample target image and the second predicted key point coordinates corresponding to the dual sample image; the first predicted key point coordinates are affine transformed to obtain the corrected predicted point coordinates; the loss value corresponding to the current iteration process is calculated based on the second predicted key point coordinates and the corrected predicted point coordinates; and the model parameters of the intermediate posture estimation model are adjusted based on the loss value.

[0353] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0354] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0355] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A motion detection method, characterized in that: The method comprises: Acquire a first live broadcast image of a target anchor, and acquire portrait detection position frame coordinates, where the portrait detection position frame coordinates are position frame coordinates corresponding to the target anchor detected from a target live broadcast image before the first live broadcast image using a portrait detection model; According to a preset magnification ratio, the portrait detection position frame coordinates are enlarged to obtain the target position frame coordinates; According to the target position frame coordinates, intercepting a target image from the first live image; detecting a similarity between the target image and a previous frame captured image, wherein the previous frame captured image is an image corresponding to the target person portrait captured from a third live image before the first live image, and the third live image is adjacent to the first live image in time sequence; If the similarity between the target image and the captured image of the previous frame is less than the similarity threshold, the target image is input into the posture estimation model to obtain a heat map of each key point; wherein the posture estimation model is obtained by training the initial posture estimation model based on the sample target image and the dual sample image, and the dual sample image is obtained by performing an affine transformation on the sample target image. The training process of the posture estimation model includes: in an intermediate iteration process, the sample target image and the dual sample image are input into the intermediate posture estimation model to obtain the first predicted key point coordinates corresponding to the sample target image and the second predicted key point coordinates corresponding to the dual sample image; the first predicted key point coordinates are affine transformed to obtain the corrected predicted point coordinates; the loss value corresponding to the current iteration process is calculated based on the second predicted key point coordinates and the corrected predicted point coordinates; the model parameters of the intermediate posture estimation model are adjusted according to the loss value; According to the heat map of each key point, the key point coordinates of each key point are obtained; According to the coordinates of each of the key points, detecting whether the target anchor's action amplitude for the preset action is greater than a first amplitude threshold; If the action amplitude is greater than the first amplitude threshold, it is determined that the target anchor starts the preset action, and whether the target anchor ends the preset action is detected according to at least one second live broadcast image of the target anchor, and the first live broadcast image and the at least one second live broadcast image are sequentially adjacent in time sequence; If the target anchor finishes the preset action, it is determined that the target anchor completes the preset action once.

2. The method according to claim 1, characterized in that: The detecting, according to at least one second live broadcast image of the target anchor, whether the target anchor has finished the preset action includes: According to the time sequence of each of the second live broadcast images from front to back, it is detected in turn whether the action amplitude of the target anchor for the preset action in each of the second live broadcast images is less than a second amplitude threshold, and the second amplitude threshold is less than the first amplitude threshold; If it is detected that in the target second live broadcast image, the action amplitude of the target anchor for the preset action is less than the second amplitude threshold, it is determined that the target anchor ends the preset action.

3. The method according to claim 1, characterized in that The detecting, according to the coordinates of each of the key points, whether the action amplitude of the target anchor for the preset action is greater than a first amplitude threshold includes: According to the coordinates of each of the key points, an angle value between the central axis of the body of the target anchor in the first live broadcast image and the vertical direction is obtained; If the absolute value of the angle value is greater than the preset angle threshold, it is determined that the action amplitude of the target anchor for the preset action is greater than the first amplitude threshold.

4. The method according to claim 3, characterized in that The key point coordinates include left shoulder key point coordinates, right shoulder key point coordinates, left waist key point coordinates, and right waist key point coordinates. The step of obtaining the angle value between the body central axis of the target anchor in the first live broadcast image and the vertical direction according to the key point coordinates includes: Acquire the first midpoint coordinates according to the left shoulder key point coordinates and the right shoulder key point coordinates; Acquire the second midpoint coordinates according to the left waist key point coordinates and the right waist key point coordinates; According to the first midpoint coordinates and the second midpoint coordinates, the angle value between the central axis of the body of the target anchor in the first live broadcast image and the vertical direction is calculated.

5. The method according to claim 1, characterized in that The step of obtaining the key point coordinates of each key point according to the heat map of each key point includes: For each of the key point's heat maps, obtaining the target coordinates of the key point in the target image according to the heat map; According to the position of the target image in the first live image and the target coordinates of the key point in the target image, the key point coordinates of the key point in the first live image are acquired.

6. The method according to claim 1, characterized in that The step of inputting the target image into a posture estimation model to obtain a heat map of each key point includes: Inputting the target image into an affine transformation model to obtain affine transformation parameters corresponding to the target image; Performing an affine transformation on the target image using the affine transformation parameters to obtain a corrected image; The corrected image is input into the posture estimation model to obtain a heat map of each key point.

7. The method according to claim 6, characterized in that The step of obtaining the key point coordinates of each key point according to the heat map of each key point includes: For each of the key point's heat maps, obtaining the corrected coordinates of the key point in the corrected image according to the heat map; According to the affine transformation parameters, the corrected coordinates are inversely transformed to obtain the target coordinates of the key point in the target image; According to the position of the target image in the first live image and the target coordinates of the key point in the target image, the key point coordinates of the key point in the first live image are acquired.

8. The method according to claim 1, characterized in that The detecting the similarity between the target image and the previous frame captured image comprises: Performing hash processing on the target image to obtain a first hash value; Performing hash processing on the previous frame captured image to obtain a second hash value; Calculate an absolute value of a difference between the first Hash value and the second Hash value; If the absolute value of the difference is greater than the difference threshold, determining that the similarity between the target image and the previous frame captured image is less than the similarity threshold; If the absolute value of the difference is less than or equal to the difference threshold, it is determined that the similarity between the target image and the previous frame captured image is greater than or equal to the similarity threshold.

9. The method according to claim 8, characterized in that After detecting the similarity between the target image and the previous frame captured image, the method further includes: If the similarity between the target image and the captured image of the previous frame is greater than or equal to the similarity threshold, the key point coordinates of each key point of the target anchor in the third live image are determined as the key point coordinates of each key point of the target anchor in the first live image.

10. The method according to claim 1, characterized in that The method further comprises: Get a sample target image; Performing an affine transformation on the sample target image to obtain a dual sample image; An initial pose estimation model is trained according to the sample target image and the dual sample image to obtain the pose estimation model.

11. A motion detection device, characterized in that: The device comprises: An acquisition module, used to acquire the first live broadcast image of the target anchor; The acquisition module also includes: a second acquisition unit, the second acquisition unit being specifically used to acquire portrait detection position frame coordinates, the portrait detection position frame coordinates being position frame coordinates corresponding to the target anchor detected from a target live image before the first live image by using a portrait detection model; the portrait detection position frame coordinates are amplified according to a preset amplification ratio to obtain target position frame coordinates; A capture unit, configured to capture a target image from the first live image according to the target position frame coordinates; a third detection module, configured to detect the similarity between the target image and a captured image of a previous frame, wherein the captured image of the previous frame is an image corresponding to the target portrait captured from a third live image before the first live image, and the third live image is similar to the first live image. The images are temporally adjacent; An execution module, configured to input the target image into a posture estimation model to obtain a heat map of each key point if the similarity between the target image and the previous frame captured image is less than a similarity threshold; Wherein, the posture estimation model is obtained by training the initial posture estimation model according to the sample target image and the dual sample image, and the dual sample image is obtained by performing an affine transformation on the sample target image. The training process of the posture estimation model includes: in an intermediate iteration process, the sample target image and the dual sample image are input into the intermediate posture estimation model to obtain the first predicted key point coordinates corresponding to the sample target image and the second predicted key point coordinates corresponding to the dual sample image; the first predicted key point coordinates are affine transformed to obtain the corrected predicted point coordinates; the loss value corresponding to the current iteration process is calculated according to the second predicted key point coordinates and the corrected predicted point coordinates; the model parameters of the intermediate posture estimation model are adjusted according to the loss value; A third acquisition unit, used for acquiring the key point coordinates of each of the key points according to the heat map of each of the key points; A first detection module is used to detect whether the action amplitude of the target anchor for the preset action is greater than a first amplitude threshold according to the coordinates of each of the key points; a second detection module, configured to determine that the target anchor has started the preset action if the action amplitude is greater than the first amplitude threshold, and detect whether the target anchor has ended the preset action according to at least one second live broadcast image of the target anchor, wherein the first live broadcast image and the at least one second live broadcast image are sequentially adjacent in time sequence; The determination module is used to determine that the target anchor has completed the preset action once if the target anchor finishes the preset action.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images