Abnormal key point detection method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to an abnormal key point detection method and device, electronic equipment and storage medium, and relates to the field of computer vision. The method comprises: obtaining a target video frame and a key point set of the target video frame; for each key point in the key point set, determining a target parameter value set corresponding to a preset target weight set associated with the key point according to the target video frame and at least one reference video frame belonging to the same video segment as the target video frame; determining a target value according to the target weight set and the target parameter value set; and determining an abnormal key point in the key point set according to the target value and a preset threshold. The present disclosure can detect abnormal key points in the video frame and improve the accuracy of key point detection.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to the field of computer vision, and particularly to an anomaly key point detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] In video-based human pose recognition and extraction tasks, neural networks cannot guarantee perpetual accuracy. In certain complex situations, a few frames may produce random erroneous results, manifesting as rapidly and erratically detected joint movements. Algorithms are needed to determine whether the data provided by the neural network is anomalous; otherwise, even after filtering, the resulting data will still be insufficiently smooth, compromising its authenticity and consistency. Therefore, detecting anomalous data in the neural network output is crucial. Summary of the Invention

[0003] Embodiments of this disclosure provide an abnormal key point detection method, apparatus, electronic device, and storage medium.

[0004] In a first aspect, embodiments of this disclosure provide an abnormal key point detection method, comprising: acquiring a target video frame and a set of key points of the target video frame; for each key point in the key point set, determining a set of target parameter values ​​associated with the key point and corresponding to a preset target weight set based on the target video frame and at least one reference video frame belonging to the same video segment as the target video frame; determining a target value based on the target weight set and the target parameter value set; and determining abnormal key points in the key point set based on the target value and a preset threshold.

[0005] Secondly, embodiments of this disclosure provide an abnormal keypoint detection device, comprising: a keypoint acquisition unit configured to acquire a target video frame and a set of keypoints of the target video frame; a parameter value determination unit configured to, for each keypoint in the keypoint set, determine a set of target parameter values ​​associated with the keypoint and corresponding to a preset target weight set, based on the target video frame and at least one reference video frame belonging to the same video segment as the target video frame; a target value determination unit configured to determine a target value based on the target weight set and the target parameter value set; and an abnormal point detection unit configured to determine abnormal keypoints in the keypoint set based on the target value and a preset threshold.

[0006] Thirdly, embodiments of this disclosure provide an electronic device including a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the abnormal key point detection method as described in the first aspect.

[0007] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the abnormal key point detection method as described in the first aspect.

[0008] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0010] Figure 1 An exemplary system architecture diagram in which the anomaly key point detection method of this disclosure can be applied is shown;

[0011] Figure 2 This is a flowchart illustrating an embodiment of the abnormal key point detection method disclosed herein;

[0012] Figure 3 This is a flowchart illustrating another embodiment of the abnormal key point detection method disclosed herein;

[0013] Figure 4 This is a schematic diagram of the structure of one embodiment of the abnormal key point detection device of this disclosure;

[0014] Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device disclosed herein. Detailed Implementation

[0015] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0016] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0017] Where there is no conflict, the embodiments and features described herein can be combined with each other.

[0018] To make the technical solutions and advantages of this disclosure clearer, the following description, in conjunction with the accompanying drawings and specific embodiments, will provide a more detailed account of this disclosure.

[0019] Figure 1 An exemplary system architecture 100 is shown, to which embodiments of the abnormal key point detection method or abnormal key point detection device of this disclosure can be applied.

[0020] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0021] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications, such as video detection applications, can be installed on terminal devices 101, 102, and 103.

[0022] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0023] Server 105 can be a server that provides various services, such as a backend server that supports the video sent by terminal devices 101, 102, and 103. The backend server can perform keypoint detection on objects in the video, identify abnormal keypoints, and then report the abnormal keypoints back to terminal devices 101, 102, and 103.

[0024] It should be noted that server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0025] It should be noted that the abnormal key point detection method provided in this embodiment is generally executed by server 105. Accordingly, the abnormal key point detection device is generally installed in server 105.

[0026] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0027] Figure 2 A flowchart 200 of an embodiment of the abnormal key point detection method of this disclosure is shown. For example... Figure 2 As shown, the abnormal key point detection method in this embodiment may include the following steps:

[0028] Step 201: Obtain the target video frame and the set of key points of the target video frame.

[0029] In this embodiment, the execution body of the abnormal key point detection method (e.g.) Figure 1 The server 105 shown can acquire the target video frame and its keypoint set through various methods. The target video frame can include various objects, such as the human body. The keypoint set of the target video frame can be obtained by a neural network performing keypoint detection on the target video frame. The keypoint set can include the position coordinates of multiple keypoints.

[0030] Step 202: For each key point in the key point set, based on the target video frame and at least one reference video frame belonging to the same video segment as the target video frame, determine the set of target parameter values ​​associated with the key point and corresponding to the preset target weight set.

[0031] In this embodiment, the executing entity may also acquire at least one reference video frame belonging to the same video segment as the target video frame. The video segment to which the target video frame belongs may include multiple video frames. The reference video frame may be a video frame adjacent to the target video frame, or it may be a video frame at a specified location. The reference video frame may include the same object as the target video frame.

[0032] The executing entity can analyze each keypoint in the keypoint set to determine whether it is an anomalous keypoint. Specifically, the executing entity can determine the set of target parameter values ​​associated with each keypoint based on the target video frame and each reference video frame. The target parameter value set can include multiple parameter values, and each parameter value can correspond to a weight in the weight set. That is, there is a correspondence between the target weight set and the target parameter value set. Here, each parameter value in the target parameter value set can be a feature value of each keypoint, or a value related to the position of each keypoint (e.g., the pixel value of the pixels occupied by the keypoint, etc.).

[0033] Step 203: Determine the target value based on the target weight set and the target parameter value set.

[0034] After obtaining the set of target parameter values, the executing entity can perform weighted calculations on each parameter value and each parameter in the target weight set to obtain the target value.

[0035] Step 204: Based on the target value and the preset threshold, determine the abnormal key points in the key point set.

[0036] After determining the target value, the executing entity can compare it with a preset threshold. If the target value is greater than the preset threshold, the key point corresponding to the target value is considered an abnormal key point. By calculating the associated target value for each key point in the key point set, the abnormal key points in the key point set can be identified.

[0037] The abnormal key point detection method provided in the above embodiments of this disclosure can analyze each key point in the key point set detected by the neural network, thereby determining the abnormal key points.

[0038] See also Figure 3 This illustrates flow 300 of another embodiment of the abnormal key point detection method according to the present disclosure. Figure 3 As shown, this embodiment may include the following steps:

[0039] In this embodiment, the executing entity can first determine the target weight set through steps 301 to 304.

[0040] Step 301: Obtain at least one labeled anomalous frame from the video segment.

[0041] Before determining the anomalous key points in the target video frame, the executing entity can first obtain at least one annotated anomalous frame from the video segment to which the target video frame belongs. This annotated anomalous frame includes the annotated anomalous key points. These anomalous key points in the annotated anomalous frames can be manually annotated based on experience.

[0042] Step 302: For each abnormal key point in the labeled abnormal frame, determine the mean, variance, velocity, and acceleration associated with the abnormal key point based on the labeled abnormal frame and the M, N, and P video frames at intervals.

[0043] The executing entity can analyze each anomalous key point in the anomaly frame. Specifically, the executor's body can determine three displacements, three means, and thus three variances based on the position of each anomalous key point in the anomaly frame and its position in the aforementioned M, N, and P video frames. Then, combining the time intervals between the anomaly frame and the aforementioned three video frames, three velocities and three accelerations are determined. Here, M, N, and P are preset natural numbers. In some specific practices, M, N, and P can represent the values ​​at the beginning, middle, and end of a video segment, respectively.

[0044] Step 303: Based on the determined mean, variance, speed, and acceleration, determine the first set of parameter values ​​and the first set of weights corresponding to the first set of parameter values.

[0045] The executing entity can perform various calculations on the above parameters to obtain other parameters that are different from the above parameters, ultimately obtaining a first set of parameter values. Then, the executing entity can generate a weight for each parameter value in the first set of parameter values, obtaining a first set of weights.

[0046] In some optional implementations of this embodiment, the executing entity can perform calculations on the above parameters through the following steps to obtain the first parameter value set: for each abnormal key point, determine the absolute difference between the position of the abnormal key point in the target video frame and the mean value corresponding to each reference video frame; determine the ratio by dividing the absolute difference by the corresponding variance; take the reciprocal of the ratio, variance, velocity, and acceleration respectively, and divide and multiply the obtained reciprocals pairwise to obtain the initial parameter value set.

[0047] The executing entity can obtain three ratios by taking the ratio of the absolute difference between the means of the anomaly frame and the three video frames to their corresponding variances. These six values ​​(three variances and three ratios) are then used as a set of displacement indices. By calculating the velocity and acceleration between any two video frames between the anomaly frame and the three video frames, six velocity indices and six acceleration indices are obtained, resulting in a total of 18 indices. Taking the reciprocal of each of these 18 indices yields 18 new indices. Taking the ratio between each pair of these new 18 indices (without dividing by itself) results in 18 × 17 new indices. Multiplying these pairs yields another 18 × 18 indices. This results in a total of 666 indices. These 666 indices can be called the first parameter value set.

[0048] The executing entity can multiply each of these indicators by a weight and sum them up, plus a constant bias. These 667 parameters can be called the first weight set.

[0049] Step 304: Optimize the first weight set to obtain the target weight set.

[0050] The executing entity can optimize the first weight set by removing parameters corresponding to unimportant indicators or considering only parameters with larger weight values. Specifically, the executing entity can perform iterative optimization regression on each weight value in the first weight set. Then, the obtained values ​​are sorted from largest to smallest, and the top N weights are taken as the target weight set.

[0051] In some optional implementations of this embodiment, the executing entity may perform at least one iterative optimization regression on the first weight set, select a portion of the weights from the first weight set, and obtain the target weight set.

[0052] In this implementation, the executing entity can use an existing iterative optimization algorithm to perform at least one iterative optimization regression on the first weight set. For example, the iterative optimizer Ceres can be used to regress these 667 parameters, and the loss function value of the Ceres iterative optimizer can be used to control whether to continue iterative optimization. Through the above at least one iterative optimization regression, the executing entity can select a portion of the weights from the first weight set to obtain the target weight set. Specifically, the executing entity can select a combination of weights with values ​​greater than a preset threshold as the target weight set. Alternatively, it can combine the top N relatively large weight values ​​as the target weight set.

[0053] In some specific practices, after each iteration of optimization regression, the implementing entity can sort the weights from largest to smallest and select the top K weight values; then perform the next iteration of optimization regression on the selected weight values ​​until the iteration optimization regression termination condition is met, thus obtaining the target weight set.

[0054] Here, the executing entity can select K weight values ​​through regression optimization in each iteration. Then, iterative optimization is performed on these K weight values. Before each iteration, the executing entity can check whether the regression termination condition is met. If it is met, the iteration optimization stops, and the resulting weight values ​​are combined into the target weight set. If the condition is not met, further optimization can proceed.

[0055] Step 305: Obtain the target video frame and the set of key points of the target video frame.

[0056] In this embodiment, the set of key points in the target video frame can be detected by a neural network.

[0057] Step 306: For each keypoint in the keypoint set, determine the target parameter value set based on the keypoint's position in the target video frame and its position in at least one reference video frame.

[0058] In this embodiment, the executing entity can first determine the position of each key point in the target video frame. The position can be a three-dimensional coordinate in the target video frame. The executing entity can also determine the position of each key point in each reference video frame. Specifically, the executing entity can determine the key points in each reference video frame that correspond to the key points in the target video frame, and use the position of the corresponding key points as the position of each key point in each reference video frame.

[0059] The executing entity can determine the set of target parameter values ​​based on the position of each keypoint in the target video frame and its position in each reference video frame. This set of target parameter values ​​can include displacement values, velocity values, acceleration values, and so on.

[0060] In some optional implementations of this embodiment, step 306 can be implemented by the following sub-steps:

[0061] Step 3061: Based on the position of the key point in the target video frame and its position in at least one reference video frame, determine at least one displacement, at least one mean, and at least one variance corresponding to the key point.

[0062] For each keypoint, the executing entity can subtract its position in the target video frame from its position in each of the reference video frames to obtain at least one displacement. It is understood that the number of displacements is the same as the number of reference video frames. Simultaneously, the executing entity can average the keypoint's position in the target video frame and its position in each of the reference video frames to obtain at least one mean. Then, based on the obtained means, at least one variance is calculated.

[0063] Step 3062: Determine at least one velocity and at least one acceleration corresponding to the key point based on at least one displacement and the time interval between the target video frame and each reference video frame.

[0064] After determining each displacement, the executing entity can determine at least one velocity and at least one acceleration by combining the time interval between the target video frame and each reference video frame.

[0065] Step 3063: Determine the set of initial parameter values ​​based on at least one mean, at least one variance, at least one velocity, and at least one acceleration.

[0066] The executing entity can perform various calculations on the obtained means, variances, velocities, and accelerations to obtain an initial set of parameter values. Specifically, the executing entity can perform operations such as adding and subtracting the means, variances, velocities, and accelerations. Alternatively, the executing entity can select a subset of values ​​for calculation each time.

[0067] In some specific practices, the calculations for mean, variance, velocity, and acceleration here can be the same as those in step 303.

[0068] Step 307: Determine the target value based on the target weight set and the target parameter value set.

[0069] Step 308: Based on the target value and the preset threshold, determine the abnormal key points in the key point set.

[0070] The abnormal key point detection method provided in the above embodiments of this disclosure can effectively detect abnormal key points in video frames, thereby supplementing the deficiencies of neural networks and improving the accuracy of key point detection.

[0071] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an anomaly key point detection device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0072] like Figure 4 As shown, the abnormal key point detection device 400 of this embodiment includes: a key point acquisition unit 401, a parameter value determination unit 402, a target value determination unit 403, and an abnormal point detection unit 404.

[0073] The key point acquisition unit 401 is configured to acquire the target video frame and the set of key points of the target video frame;

[0074] The parameter value determination unit 402 is configured to, for each key point in the key point set, determine the set of target parameter values ​​associated with the key point and corresponding to the preset target weight set, based on the target video frame and at least one reference video frame belonging to the same video segment as the target video frame.

[0075] The target value determination unit 403 is configured to determine the target value based on the target weight set and the target parameter value set;

[0076] Anomaly detection unit 404 is configured to determine abnormal key points in the key point set based on the target value and a preset threshold.

[0077] In addition, an electronic device is also proposed in the technical solution of this application.

[0078] Figure 5 A schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure is shown.

[0079] like Figure 5As shown, the electronic device may include a processor 501, a memory 502, a bus 503, and a computer program stored in the memory 502 and executable on the processor 501. The processor 501 and the memory 502 communicate with each other via the bus 503. When the processor 501 executes the computer program, it implements the steps of the above method, including, for example: acquiring a target video frame and a set of key points for the target video frame; for each key point in the key point set, determining a set of target parameter values ​​associated with the key point and corresponding to a preset target weight set, based on the target video frame and at least one reference video frame belonging to the same video segment as the target video frame; determining a target value based on the target weight set and the target parameter value set; and determining abnormal key points in the key point set based on the target value and a preset threshold.

[0080] In addition, one embodiment of this disclosure also provides a non-transitory computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the above-described method, including, for example,: acquiring a target video frame and a set of key points of the target video frame; for each key point in the set of key points, determining a set of target parameter values ​​associated with the key point and corresponding to a preset set of target weights, based on the target video frame and at least one reference video frame belonging to the same video segment as the target video frame; determining a target value based on the set of target weights and the set of target parameter values; and determining abnormal key points in the set of key points based on the target value and a preset threshold.

[0081] In summary, the technical solution disclosed herein can effectively detect abnormal key points in video frames, thereby supplementing the deficiencies of neural networks and improving the accuracy of key point detection.

[0082] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for detecting abnormal key points, comprising: Obtain the target video frame and the set of key points of the target video frame; For each key point in the key point set, based on the target video frame and at least one reference video frame belonging to the same video segment as the target video frame, determine the set of target parameter values ​​associated with the key point and corresponding to the preset target weight set. The target value is determined based on the target weight set and the target parameter value set; Based on the target value and the preset threshold, determine the abnormal key points in the key point set; For each keypoint in the keypoint set, based on the target video frame and at least one reference video frame belonging to the same video segment as the target video frame, the set of target parameter values ​​associated with the keypoint and corresponding to a preset target weight set is determined, including: For each key point in the key point set, the target parameter value set is determined based on the position of the key point in the target video frame and the position in the at least one reference video frame; Determining the target parameter value set based on the position of the key point in the target video frame and its position in at least one reference video frame includes: Based on the position of the key point in the target video frame and the position in the at least one reference video frame, the key point has at least one displacement, at least one mean, and at least one variance. Based on the at least one displacement and the time interval between the target video frame and each of the reference video frames, determine at least one velocity and at least one acceleration corresponding to the key point; Determine the initial parameter value set based on the at least one mean, the at least one variance, the at least one velocity, and the at least one acceleration; Select a set of target parameter values ​​from the initial set of parameter values ​​that corresponds to the target weight set.

2. The method according to claim 1, wherein, The step of determining abnormal key points in the key point set based on the target value set and a preset threshold includes: Key points corresponding to target values ​​in the target value set that are greater than the preset threshold are designated as abnormal key points.

3. The method according to claim 1, wherein, The method further includes the following steps for determining the target weight set: Obtain at least one labeled anomalous frame from the video segment, wherein the labeled anomalous frame includes labeled anomalous key points; For each anomalous key point in the labeled anomalous frame, the mean, variance, velocity, and acceleration associated with the anomalous key point are determined based on the labeled anomalous frame and the M, N, and P video frames at intervals, where M, N, and P are preset natural numbers. Based on the determined mean, variance, velocity, and acceleration, a first set of parameter values ​​and a first set of weights corresponding to the first set of parameter values ​​are determined. The first weight set is optimized to obtain the target weight set.

4. The method according to claim 3, wherein, The optimization of the first weight set to obtain the target weight set includes: Perform at least one iterative optimization regression on the first weight set, and select a portion of the weights from the first weight set to obtain the target weight set.

5. The method according to claim 4, wherein, The step of selecting a subset of weights from the first weight set includes: After each iteration of regression optimization, the weights are sorted from largest to smallest, and the top K weight values ​​are selected. The selected weight values ​​are subjected to the next iteration of optimization regression until the termination condition of the iteration optimization regression is met, thus obtaining the target weight set.

6. The method according to claim 3, wherein, The step of determining a first set of parameter values ​​and a first set of weights corresponding to the first set of parameter values ​​based on the determined mean, variance, speed, and acceleration includes: For each anomalous key point, determine the absolute difference between the position of the anomalous key point in the target video frame and the mean value corresponding to each reference video frame; The ratio is determined by dividing the absolute difference by the corresponding variance; Take the reciprocal of the ratio, the variance, the velocity, and the acceleration, and divide and multiply the resulting reciprocals pairwise to obtain the first set of parameter values.

7. The method according to claim 1, wherein, The at least one reference video frame includes video frames from the beginning, middle, and end of the video segment, respectively.

8. An anomaly key point detection device, comprising: The key point acquisition unit is configured to acquire a target video frame and a set of key points of the target video frame; The parameter value determination unit is configured to, for each key point in the key point set, determine the set of target parameter values ​​associated with the key point and corresponding to a preset target weight set, based on the target video frame and at least one reference video frame belonging to the same video segment as the target video frame. The target value determination unit is configured to determine a target value based on the target weight set and the target parameter value set; An anomaly detection unit is configured to determine abnormal key points in the key point set based on the target value and a preset threshold. The parameter value determination unit is further configured to: For each key point in the key point set, the target parameter value set is determined based on the position of the key point in the target video frame and the position in the at least one reference video frame; Determining the target parameter value set based on the position of the key point in the target video frame and its position in at least one reference video frame includes: Based on the position of the key point in the target video frame and the position in the at least one reference video frame, the key point has at least one displacement, at least one mean, and at least one variance. Based on the at least one displacement and the time interval between the target video frame and each of the reference video frames, determine at least one velocity and at least one acceleration corresponding to the key point; Determine the initial parameter value set based on the at least one mean, the at least one variance, the at least one velocity, and the at least one acceleration; Select a set of target parameter values ​​from the initial set of parameter values ​​that corresponds to the target weight set.

9. An electronic device comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the abnormal key point detection method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the abnormal key point detection method as described in any one of claims 1 to 7.

Images