A system and method for automatically obtaining ROI based on human posture clustering

Abstract

The application belongs to the technical field of image analysis, and particularly relates to a ROI automatic acquisition system and method based on human posture clustering, which comprises a conversion module, a detection module, a category judgment module and a selection module; the conversion module is used for converting an acquired video stream into a video frame sequence; the detection module is used for acquiring all human key points in a current video frame; the acquired human key points are taken as key point instances, and the acquired results are preliminarily judged to obtain the number of people; the category judgment module is used for, when the number of people is not zero, performing category judgment on each key point instance in combination with a pre-constructed mapping file to obtain positive and negative example attributes to which each key point instance belongs; and the selection module is used for traversing all key point instances determined as positive example attributes, and selecting a positive example instance corresponding to a maximum value of maximum similarity as a ROI to be analyzed; the application has better scene adaptability, can automatically screen a picture area concerned by a later task, and improves the speed of subsequent analysis.

Description

Technical Field

[0001] This invention relates to the field of image analysis technology, specifically to an automatic ROI acquisition system and method based on human pose clustering. Background Technology

[0002] ROI (Region of Interest) is an important concept in computer vision and image analysis, used to express the region in an image that actually participates in the analysis.

[0003] With the development of deep learning, human pose estimation has begun to be widely used in fields such as smart security, smart education, and AI-assisted sports training. In human pose estimation applications, how to obtain the region of interest (ROI) to be analyzed from the image is a common and universal problem. In this regard, there are currently two main approaches:

[0004] 1. Manually set the ROI region: This method involves manually selecting the region to be analyzed. The advantage of this method is high accuracy, but the disadvantage is that it requires manual intervention, which does not meet the expectation of program automation.

[0005] 2. Using object detection algorithms and certain rule logic to filter ROIs has the advantages of high automation and speed; however, it has the disadvantages of limited rule generalization, inability to handle specific situations, and a tendency to misidentify. In addition, this approach is pose-independent, which cannot handle scenarios where the area where a person in a specific pose is located needs to be taken as the ROI. Summary of the Invention

[0006] In view of the technical deficiencies mentioned in the background art, the purpose of this invention is to provide an automatic ROI acquisition system and method based on human posture clustering.

[0007] To achieve the above objectives, in a first aspect, embodiments of the present invention provide an automatic ROI acquisition system based on human pose clustering, the system comprising:

[0008] The conversion module is used to decode the acquired video stream and convert the video stream into a video frame sequence;

[0009] The detection module is used for:

[0010] Perform a keypoint detection task on each video frame to obtain all human keypoints in the current video frame;

[0011] The acquired human body key points are used as key point instances, and the acquired results are initially judged to determine the number of people.

[0012] The category determination module is used to determine the category of each key point instance in conjunction with a pre-built mapping file when the number of people is not zero, so as to obtain the positive and negative example attributes to which each key point instance belongs; wherein, the mapping file includes cluster result center instances and corresponding positive and negative example attributes;

[0013] The selection module is used to traverse all keypoint instances that are determined to be positive examples and select the positive example instance corresponding to the maximum value of the maximum similarity as the ROI to be analyzed; wherein, the maximum similarity is obtained by traversing all cluster result center instances in the mapping file for each keypoint instance to perform similarity calculation.

[0014] Preferably, the ROI automatic acquisition system based on human pose clustering further includes a processing module, which is used for:

[0015] Take bounding rectangles for all keypoint instances and calculate the standard deviations of the width and height of these bounding rectangles; traverse all ROIs to be analyzed, take their bounding rectangles, and if the standard deviation is greater than 0.1 times the width and height, keep the center coordinates of the bounding rectangle unchanged, increase the width and height by 2 corresponding standard deviations, and then use it as a pre-selected ROI;

[0016] If the standard deviation is less than 0.1 times the width and height, then the width and height are increased by 0.2 times respectively, and used as a pre-selected ROI.

[0017] Preferably, the ROI automatic acquisition system based on human pose clustering further includes a correction module, which is used for:

[0018] Image boundary correction is performed on the pre-selected ROI to ensure that the obtained ROI does not exceed the image boundary. The result after boundary correction is the final ROI of the corresponding frame.

[0019] Preferably, the category determination specifically involves:

[0020] Each key point instance is normalized in two dimensions and used as a query instance;

[0021] For the query instance, iterate through all cluster result center instances in the mapping file to calculate similarity, and take the positive and negative example attribute results corresponding to the cluster result center instance with the maximum value as the category of the query instance.

[0022] Preferably, the mapping file is obtained by clustering human poses, and the specific steps are as follows:

[0023] Collect unlabeled datasets that meet the requirements of the final task scenario;

[0024] The collected dataset is cleaned to obtain the original image dataset for further analysis;

[0025] After obtaining the original image dataset, the dataset is traversed, and the human keypoint detector is used to infer for each image to obtain all human keypoint instances, which are then stored separately.

[0026] By traversing the data, we obtain a dataset of individual human keypoint instances corresponding to the original image dataset, and then summarize them as a keypoint dataset.

[0027] Read the key point dataset, perform two-dimensional normalization on each key point instance, and then input it into the clustering algorithm for clustering iteration to obtain the clustering result. Extract the cluster center instance.

[0028] The cluster center instances are denormalized to become normal-sized keypoint instances;

[0029] Visualize and render the denormalized keypoint instances, and store the results for later viewing;

[0030] The obtained visualization rendering results are manually interpreted, and the set of instances that meet the requirements of the final task is selected and labeled as positive example attributes, while the remaining instances are labeled as negative example attributes.

[0031] The results obtained above are used to create the mapping file, which should include the data of the cluster center instances and the labeled positive and negative example attributes.

[0032] Then, a small number of samples are randomly selected from the key point dataset. The selected samples are traversed, and each sample is normalized and then the key point similarity is calculated with all cluster result center instances stored in the mapping file. The positive and negative example attributes corresponding to the result with the highest similarity are selected as the sample attribute and sample result.

[0033] The accuracy rate is calculated based on the sample results, and the results are judged. If the accuracy rate is too low, the above steps should be repeated until the accuracy rate is acceptable.

[0034] The mapping file that meets the accuracy requirements is used as the constructed mapping file, and the clustering process is now complete.

[0035] Secondly, embodiments of the present invention also provide a method for automatically obtaining ROI based on human pose clustering, the method comprising:

[0036] The acquired video stream is decoded and converted into a video frame sequence;

[0037] Perform a keypoint detection task on each video frame to obtain all human keypoints in the current video frame;

[0038] The acquired human body key points are used as key point instances, and the acquired results are initially judged to determine the number of people.

[0039] When the number of people is not zero, each key point instance is classified in conjunction with a pre-built mapping file to obtain the positive and negative example attributes of each key point instance; wherein, the mapping file includes cluster result center instances and corresponding positive and negative example attributes;

[0040] Iterate through all keypoint instances that are determined to be positive examples, and select the positive example instance corresponding to the maximum value of the maximum similarity as the ROI to be analyzed; wherein, the maximum similarity is obtained by traversing all cluster result center instances in the mapping file for each keypoint instance to perform similarity calculation.

[0041] By implementing the embodiments of the present invention, the key point data of the human body is acquired and the key point data is combined with a pre-constructed mapping file for interpretable classification. It has high adaptability in human posture analysis application scenarios, thereby automatically filtering out the image area of ​​interest for subsequent tasks, so that it only analyzes the area of ​​interest, and also automatically filters noise in the task results, while improving the accuracy and speed of subsequent analysis. Attached Figure Description

[0042] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.

[0043] Figure 1 A schematic diagram of the structure of an automatic ROI acquisition system based on human pose clustering provided in an embodiment of the present invention;

[0044] Figure 2 An interactive schematic diagram provided for an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of a mapping file acquisition process provided in an embodiment of the present invention;

[0046] Figure 4 This is a flowchart of an automatic ROI acquisition method based on human pose clustering provided in an embodiment of the present invention.

[0047] The realization of the objectives, functional features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0050] Example 1:

[0051] Please refer to Figures 1 to 3 This invention provides an automatic ROI acquisition system based on human pose clustering, the system comprising:

[0052] The conversion module is used to decode the acquired video stream and convert the video stream into a video frame sequence;

[0053] The detection module is used for:

[0054] Perform a keypoint detection task on each video frame to obtain all human keypoints in the current video frame;

[0055] The acquired human body key points are used as key point instances, and the acquired results are initially judged to determine the number of people.

[0056] The category determination module is used to determine the category of each key point instance in conjunction with a pre-built mapping file when the number of people is not zero, so as to obtain the positive and negative example attributes to which each key point instance belongs; wherein, the mapping file includes cluster result center instances and corresponding positive and negative example attributes;

[0057] The category determination is specifically as follows:

[0058] Each key point instance is normalized in two dimensions and used as a query instance;

[0059] For the query instance, iterate through all cluster result center instances in the mapping file to calculate the similarity, and take the positive and negative example attribute results corresponding to the cluster result center instance with the maximum value as the category result of the query instance; if the obtained result is a positive example, then store the query instance and its corresponding maximum similarity.

[0060] The selection module is used to traverse all keypoint instances that are determined to be positive examples and select the positive example instance corresponding to the maximum value of the maximum similarity as the ROI to be analyzed; wherein, the maximum similarity is obtained by traversing all cluster result center instances in the mapping file for each keypoint instance to perform similarity calculation.

[0061] In implementation, a human keypoint detector is used on the video frame to obtain all human keypoints in the current video frame; wherein, the human keypoint detector refers to a class of algorithms that can obtain predefined human keypoint coordinates from an image. In this embodiment, the open-source model YOLO-POSE can be selected. The above is just an example and is not intended to limit it.

[0062] If no human key points are detected, repeat the above process, read video frames, and perform human key point detection; when the number of people is confirmed to be non-zero, perform two-dimensional normalization on each key point instance, traverse the cluster result center instances in all mapping files, record the maximum similarity corresponding to the key point instance, and finally determine the positive and negative example attributes of the key point.

[0063] The mapping file is obtained by clustering human poses, and the specific steps are as follows:

[0064] Collect unlabeled datasets that meet the requirements of the final task scenario; where the final task scenario is explained as follows: First, this invention solves a sub-problem in the product, such as an application that counts sit-ups. The final task of the application is to complete the count of sit-ups, and the problem solved by this invention is to filter out the area of ​​the screen where the person doing sit-ups is located.

[0065] The collected dataset is cleaned to obtain the original image dataset for further analysis; wherein, the cleaning includes removing images that are unsuitable for further analysis, such as those with duplicate appearances, severe occlusion, abnormal exposure, or abnormal viewpoint.

[0066] After obtaining the original image dataset, the dataset is traversed, and a human keypoint detector is used to infer for each image to obtain all human keypoint instances, which are then stored separately. During storage, the original image path information and keypoint information should be retained for subsequent interpretable verification.

[0067] By traversing the data, we obtain a dataset of individual human keypoint instances corresponding to the original image dataset, and then summarize them as a keypoint dataset.

[0068] The keypoint dataset is read, and each keypoint instance is normalized in two dimensions. This normalization is then used as input to a clustering algorithm for clustering iteration. The clustering results are obtained, and the cluster center instances are extracted. The clustering algorithm is such as kmes++, and the two-dimensional normalization refers to normalizing the keypoint coordinates in both the x-axis and y-axis dimensions.

[0069] The cluster center instances are denormalized to become normal-sized keypoint instances;

[0070] Visualize and render the denormalized keypoint instances, and store the results for later viewing;

[0071] The obtained visualization rendering results are manually interpreted, and the set of instances that meet the requirements of the final task is selected and labeled as positive example attributes, while the remaining instances are labeled as negative example attributes.

[0072] The results obtained above are used to create the mapping file, which should include the data of the cluster center instances and the labeled positive and negative example attributes.

[0073] Then, a small number of samples are randomly selected from the key point dataset. The selected samples are traversed, and each sample is normalized and then the key point similarity is calculated with all cluster result center instances stored in the mapping file. The positive and negative example attributes corresponding to the result with the highest similarity are selected as the sample attribute and sample result.

[0074] Based on the sample results, the accuracy rate is statistically analyzed, and the statistical results are judged. If the accuracy rate is too low, the above steps should be repeated until the accuracy rate is acceptable; that is, the sample results are manually checked and the accuracy rate is statistically analyzed.

[0075] The mapping file that meets the accuracy requirements is used as the constructed mapping file, and the clustering process is now complete.

[0076] The key point similarity calculation mentioned in the text can also be understood as a similarity calculation, as detailed below:

[0077]

[0078] Where i refers to the key point number; n is the total number of key points; single point visibility refers to whether the point is visible, if visible the value is 1, otherwise the value is 0;

[0079] The method for calculating single-point similarity is as follows:

[0080]

[0081] Where d: refers to the Euclidean distance between corresponding points in the two sets of keypoints to be calculated; k: is a constant that varies depending on the point's location; the calculation method for k in single-point similarity is as follows:

[0082] Based on the clustering results mentioned in the specific implementation of interpretable classification, the standard deviation of the key points at the corresponding locations of all central instances is calculated, and twice the standard deviation is taken as the k value.

[0083] The range of keypoint similarity is (0,1]. The more similar the two sets of keypoints being measured, the closer the result is to 1. Two sets of keypoints that are completely identical will get 1. Conversely, the less similar the two sets of keypoints being measured, the closer the similarity result is to 0.

[0084] When applying this method, for applications that only need to obtain one ROI, iterate through all instances that were identified as positive examples in the previous step, compare their maximum similarity recorded in the previous step, and select the instance corresponding to the maximum value of the maximum similarity as the positive example to be analyzed.

[0085] For applications that need to obtain multiple ROIs, all key point instances that are determined to be positive examples are recorded and used as positive examples to be analyzed; the positive examples to be analyzed here are the ROIs to be analyzed, and the maximum similarity has the same meaning as the maximum similarity.

[0086] In another embodiment, based on the above technical solution, to obtain more information, the aforementioned ROI automatic acquisition system based on human pose clustering further includes a processing module, which is used to:

[0087] Take bounding rectangles for all keypoint instances and calculate the standard deviations of the width and height of these bounding rectangles;

[0088] Iterate through all ROIs to be analyzed, take their bounding rectangles, and if the standard deviation is greater than 0.1 times the width and height, keep the center coordinates of the bounding rectangle unchanged, and increase the width and height by 2 corresponding standard deviations (the standard deviations have been calculated in the previous step), and then use it as a pre-selected ROI;

[0089] If the standard deviation is less than 0.1 times the width and height, then the width and height are increased by 0.2 times respectively, and used as a pre-selected ROI.

[0090] Furthermore, the aforementioned ROI automatic acquisition system based on human pose clustering also includes a correction module, which is used to:

[0091] Image boundary correction is performed on the pre-selected ROI to ensure that the obtained ROI does not exceed the image boundary. The result after boundary correction is the final ROI of the corresponding frame.

[0092] Specifically, the ROI obtained in each video frame is compared with the ROI obtained in the previous frame to calculate IOU. When the IOU is consistently greater than 0.9, the ROI is considered stable. The specific method for calculating IOU is to divide the intersection area of ​​the two bounding boxes to be calculated by the area of ​​their union.

[0093] Once the ROI stabilizes, the ROI of the current frame is used as the final output ROI.

[0094] The subsequent task sets the analysis area of ​​the screen based on the final output ROI. Performing model inference within the ROI will reduce the amount of computation and improve the inference speed.

[0095] By acquiring human key point data and combining it with pre-built mapping files for interpretable classification, this method has high adaptability in human pose analysis applications. It can automatically filter out the image areas of interest for subsequent tasks, allowing them to analyze only the areas of interest. It also automatically filters noise from the task results, while improving the accuracy and speed of subsequent analysis.

[0096] Example 2:

[0097] Based on the same inventive concept, embodiments of the present invention also provide a method for automatically obtaining ROI based on human pose clustering, such as... Figure 4 As shown, the method includes:

[0098] S1, decode the acquired video stream and convert the video stream into a video frame sequence;

[0099] S2, perform keypoint detection task on each video frame to obtain all human keypoints in the current video frame;

[0100] S3, take the acquired human body key points as key point instances, and make a preliminary judgment on the acquired results to determine the number of people;

[0101] S4, when the number of people is not zero, classify each key point instance by combining it with a pre-built mapping file to obtain the positive and negative example attributes to which each key point instance belongs; wherein, the mapping file includes cluster result center instances and corresponding positive and negative example attributes;

[0102] S5, iterate through all keypoint instances that are determined to be positive examples, and select the positive example instance corresponding to the maximum value of the maximum similarity as the ROI to be analyzed; wherein, the maximum similarity is obtained by iterating through all cluster result center instances in the mapping file for each keypoint instance to perform similarity calculation.

[0103] The mapping file is obtained by clustering human poses, and the specific steps are as follows:

[0104] Collect unlabeled datasets that meet the requirements of the final task scenario;

[0105] The collected dataset is cleaned to obtain the original image dataset for further analysis;

[0106] After obtaining the original image dataset, the dataset is traversed, and the human keypoint detector is used to infer for each image to obtain all human keypoint instances, which are then stored separately.

[0107] By traversing the data, we obtain a dataset of individual human keypoint instances corresponding to the original image dataset, and then summarize them as a keypoint dataset.

[0108] Read the key point dataset, perform two-dimensional normalization on each key point instance, and then input it into the clustering algorithm for clustering iteration to obtain the clustering result. Extract the cluster center instance.

[0109] The cluster center instances are denormalized to become normal-sized keypoint instances;

[0110] Visualize and render the denormalized keypoint instances, and store the results for later viewing;

[0111] The obtained visualization rendering results are manually interpreted, and the set of instances that meet the requirements of the final task is selected and labeled as positive example attributes, while the remaining instances are labeled as negative example attributes.

[0112] The results obtained above are used to create the mapping file, which should include the data of the cluster center instances and the labeled positive and negative example attributes.

[0113] Then, a small number of samples are randomly selected from the key point dataset. The selected samples are traversed, and each sample is normalized and then the key point similarity is calculated with all cluster result center instances stored in the mapping file. The positive and negative example attributes corresponding to the result with the highest similarity are selected as the sample attribute and sample result.

[0114] The accuracy rate is calculated based on the sample results, and the results are judged. If the accuracy rate is too low, the above steps should be repeated until the accuracy rate is acceptable.

[0115] The mapping file that meets the accuracy requirements is used as the constructed mapping file, and the clustering process is now complete.

[0116] The category determination is specifically as follows:

[0117] Each key point instance is normalized in two dimensions and used as a query instance;

[0118] For the query instance, iterate through all cluster result center instances in the mapping file to calculate similarity, and take the positive and negative example attribute results corresponding to the cluster result center instance with the maximum value as the category of the query instance.

[0119] Furthermore, the method also includes:

[0120] Take bounding rectangles for all keypoint instances and calculate the standard deviations of the width and height of these bounding rectangles;

[0121] Iterate through all ROIs to be analyzed, take their bounding rectangles, and if the standard deviation is greater than 0.1 times the width and height, keep the center coordinates of the bounding rectangle unchanged, increase the width and height by 2 corresponding standard deviations, and then use it as a pre-selected ROI;

[0122] If the standard deviation is less than 0.1 times the width and height, then the width and height are increased by 0.2 times respectively, and used as a pre-selected ROI.

[0123] Furthermore, the method also includes:

[0124] Image boundary correction is performed on the pre-selected ROI to ensure that the obtained ROI does not exceed the image boundary. The result after boundary correction is the final ROI of the corresponding frame.

[0125] It should be noted that for a more detailed workflow of the method embodiments, please refer to the aforementioned system embodiments section, which will not be repeated here.

[0126] To better understand this invention, a specific application will be used as an example for illustration below.

[0127] In this embodiment, the ultimate goal is to count the number of sit-ups performed by a subject undergoing sit-up training in a video frame. One technical challenge faced by this embodiment is how to identify the subject from the video and filter out irrelevant individuals. This invention is precisely applicable to this scenario.

[0128] First, the mapping file is constructed, specifically:

[0129] 1. Collect a large number of videos taken by cameras during the sit-ups of test subjects;

[0130] 2. Preprocess the video to remove duplicate frames and obtain rough raw image data;

[0131] 3. Manually screen the rough raw image data obtained in the previous step. During the screening process, pay attention to selecting images of subjects doing sit-ups, and ensure that the images of subjects appearing in the images are not less than 50%.

[0132] 4. Perform a second manual screening on the images selected in the previous step to remove images with incomplete figures, obscured subjects, abnormal exposure, etc., to form the original image dataset;

[0133] 5. Traverse the original image dataset and perform human keypoint detection on the images. In this example, the open-source model YOLO-POSE is chosen as the human keypoint detector, but in fact any human keypoint detector with the required accuracy can do the job (due to the special nature of the sit-up posture, we retrained the keypoint detector). For the results obtained by the human keypoint detector, perform two-dimensional normalization. Specifically, divide the x-coordinate by the width of the corresponding human keypoint bounding box to get the x-coordinate in the result, and divide the y-coordinate by the height of the human keypoint bounding box to get the y-coordinate in the result.

[0134] 6. Record the image paths of the original image dataset and the keypoint instance data generated in the previous step after two-dimensional normalization, so that they are in the following format:

[0135] Each row contains the fields: [Sequence Number], [Image Path], and [Keypoint Instance Coordinates].

[0136] Wherein, the keypoint instance coordinates are x, y, s, where x and y are the coordinates of the keypoint, and s is the confidence level generated during model inference;

[0137] In this embodiment, we use 17-point keypoint coordinates, so there are 17 different sets of x, y, s coordinates for keypoint instances; this step will generate a keypoint dataset.

[0138] 7. First, a small number of samples are randomly selected from the keypoint dataset for subsequent validation;

[0139] 8. Input the key point dataset into the clustering model for clustering iteration. In this embodiment, kmeans++ is used as the clustering method, but theoretically any clustering method can be used.

[0140] 9. After determining the optimal number of clusters using the clustering method, obtain the clustering results and extract each central instance from the clustering results;

[0141] 10. Perform inverse normalization on the center instance obtained in the previous step. In this embodiment, 175 pixels are used as the side length data for the inverse normalization operation. The specific steps of the inverse normalization operation are as follows: multiply the x and y coordinates of the center instance by the side length respectively, and use the results as the x and y coordinates of the inverse normalization operation.

[0142] 11. After inverse normalization is completed, each cluster result center instance can be rendered to generate a binary image containing only the human body key point skeleton structure.

[0143] 12. For the binary image generated in the previous step, use human experience to determine which postures are of interest to the sit-up task and mark them as positive examples, while the remaining central instances are marked as negative examples;

[0144] 13. Generate a mapping file in the following format based on the marking results from the previous step:

[0145] [Key Point Instance Coordinates][Positive and Negative Examples]

[0146] The key point instance coordinates are explained in the same way as in step 6. Positive and negative examples are represented by numbers, with positive examples recorded as 1 and negative examples recorded as 0.

[0147] 14. Iterate through the samples extracted in step 7 for verification. The specific verification process is as follows: For each sample to be verified, perform keypoint similarity calculations one by one with all keypoint instances stored in the mapping file, and take the positive and negative examples corresponding to the instances that obtain the maximum keypoint similarity as the result of this sample; this step will generate a small set of sample results to be evaluated.

[0148] 15. Perform manual accuracy statistics on the sample set to be evaluated generated in the previous step, and judge the statistical results. If the accuracy is too low, the above steps should be repeated until the accuracy is acceptable.

[0149] 16. After the previous step, a mapping file with the required accuracy is generated. This mapping file is stored for later use, and the clustering work is now complete. It should be noted that in the description of the clustering process, all the key point instances stored in the mapping file are the clustering result center instances.

[0150] 17. After the clustering work is completed, the ROI can be automatically obtained. After reading the video stream, it is decoded into video frames, and operations are performed on each video frame.

[0151] 18. Perform keypoint detection on video frames, make a preliminary judgment on the results, and if the number of people contained is 0, then re-detect;

[0152] 19. If the number of people in the keypoint detection results obtained in the previous step is greater than or equal to 1, obtain the bounding box of each keypoint instance and record the width and height of the bounding box. Then calculate the standard deviation of the width and height respectively. If the number of people is only 1, then record the standard deviation of the width and height as 0.

[0153] 20. Perform category determination for each keypoint instance. The specific steps for category determination are as follows:

[0154] a. Perform two-dimensional normalization on key point instances, which will be referred to as query instances below;

[0155] b. For a query instance, iterate through all keypoint instances in the mapping file to calculate similarity, and take the positive and negative examples corresponding to the keypoint instance with the maximum value as the category result of the query instance;

[0156] c. If the result obtained in the previous step is a positive example, then store the query instance and its corresponding maximum similarity for later use;

[0157] 21. Check if a positive example was obtained in the previous step. If no positive example was obtained, repeat steps 18 to 20.

[0158] 22. If the number of positive examples obtained is greater than 1, then traverse all the positive example information stored in step 20 and select the positive example instance corresponding to the maximum similarity value as the pre-selected instance; if the number of positive examples is 1, then directly use the positive example as the pre-selected instance (this embodiment only requires one ROI, so the approach in this step is used. If multiple ROIs are required, you can also refer to the previous part of the description of this invention).

[0159] 23. Obtain the bounding box of the pre-selected instance and compare it with the standard deviation obtained in step 19. Specifically, determine whether 0.1 times the height of the bounding box is greater than the height standard deviation. If it is, the height increment is 0.2 times the height of the bounding box. Determine whether 0.1 times the width of the bounding box is greater than the width standard deviation. If it is, the width increment is 0.2 times the width of the bounding box.

[0160] 24. Keep the center coordinates of the bounding box of the preselected instance unchanged, and increase the width and height by the width increment and height increment obtained in the previous step to obtain the preselected ROI;

[0161] 25. Correct the out-of-bounds errors of the pre-selected ROIs and obtain the final result, which is the ROI of this frame;

[0162] 26. The ROI obtained for each video frame is compared with the ROI obtained for the previous frame to calculate IOU. When the IOU is consistently greater than 0.9, the ROI is considered stable. The specific method for calculating IOU is to divide the intersection area of ​​the two bounding boxes to be calculated by the area of ​​their union.

[0163] 27. Once the ROI stabilizes, use the ROI of the current frame as the final output;

[0164] 28. The subsequent task sets the analysis area of ​​the screen according to the ROI output in step 27. Performing model inference within the ROI will reduce the amount of computation and improve the inference speed.

[0165] The present invention has the following advantages in solving the ROI acquisition problem for human pose estimation related tasks:

[0166] Reduce reliance on manual annotation; no large amount of manually annotated datasets are required, only the necessary set of unannotated images needs to be collected and cleaned.

[0167] The results are more interpretable. Compared with the approach of directly feeding key points into the neural network for classification, this method has better interpretability. All classification results can be intuitively explained through visualization. In scenarios with high interpretability requirements, such as assisted training and education, this method has higher scenario adaptability.

[0168] Resource conservation is achieved because the use of human keypoint detectors is unavoidable for human pose estimation tasks. This method directly uses the results of human keypoint detectors as input, making reasonable reuse of already used resources without significantly increasing the demand for computational resources.

[0169] In this application, the descriptions of the various embodiments have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0170] The technical features of the present application can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application.

[0171] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application.

Claims

1. An automatic ROI acquisition system based on human pose clustering, characterized in that, The system includes: The conversion module is used to decode the acquired video stream and convert the video stream into a video frame sequence; The detection module is used for: Perform a keypoint detection task on each video frame to obtain all human keypoints in the current video frame; The acquired human body key points are used as key point instances, and the acquired results are initially judged to determine the number of people. The category determination module is used to determine the category of each key point instance in conjunction with a pre-built mapping file when the number of people is not zero, so as to obtain the positive and negative example attributes to which each key point instance belongs; wherein, the mapping file includes cluster result center instances and corresponding positive and negative example attributes; The selection module is used to traverse all keypoint instances that are determined to be positive examples and select the positive example instance corresponding to the maximum value of the maximum similarity as the ROI to be analyzed; wherein, the maximum similarity is calculated by traversing all cluster result center instances in the mapping file for each keypoint instance; The mapping file is obtained by clustering human poses, and the specific steps are as follows: Collect unlabeled datasets that meet the requirements of the final task scenario; The collected dataset is cleaned to obtain the original image dataset for further analysis; After obtaining the original image dataset, the dataset is traversed, and the human keypoint detector is used to infer for each image to obtain all human keypoint instances, which are then stored separately. By traversing the data, we obtain a dataset of individual human keypoint instances corresponding to the original image dataset, and then summarize them as a keypoint dataset. Read the key point dataset, perform two-dimensional normalization on each key point instance, and then input it into the clustering algorithm for clustering iteration to obtain the clustering result. Extract the cluster center instance. The cluster center instances are denormalized to become normal-sized keypoint instances; Visualize and render the denormalized keypoint instances, and store the results for later viewing; The obtained visualization rendering results are manually interpreted, and the set of instances that meet the requirements of the final task is selected and labeled as positive example attributes, while the remaining instances are labeled as negative example attributes. The results obtained above are used to create the mapping file, which should include the data of the cluster center instances and the labeled positive and negative example attributes. Then, a small number of samples are randomly selected from the key point dataset. The selected samples are traversed, and each sample is normalized and then the key point similarity is calculated with all cluster result center instances stored in the mapping file. The positive and negative example attributes corresponding to the result with the highest similarity are selected as the sample attribute and sample result. The accuracy rate is calculated based on the sample results, and the results are judged. If the accuracy rate is too low, the above steps should be repeated until the accuracy rate is acceptable. The mapping file that meets the accuracy requirements is used as the constructed mapping file, and the clustering process is now complete.

2. The ROI automatic acquisition system based on human pose clustering according to claim 1, characterized in that, It also includes a processing module, which is used for: Take bounding rectangles for all keypoint instances and calculate the standard deviations of the width and height of these bounding rectangles; Iterate through all ROIs to be analyzed, take their bounding rectangles, and if the standard deviation is greater than 0.1 times the width and height, keep the center coordinates of the bounding rectangle unchanged, increase the width and height by 2 corresponding standard deviations, and then use it as a pre-selected ROI; If the standard deviation is less than 0.1 times the width and height, then the width and height are increased by 0.2 times respectively, and used as a pre-selected ROI.

3. The ROI automatic acquisition system based on human pose clustering according to claim 2, characterized in that, It also includes a correction module, which is used for: Image boundary correction is performed on the pre-selected ROI to ensure that the obtained ROI does not exceed the image boundary. The result after boundary correction is the final ROI of the corresponding frame.

4. The ROI automatic acquisition system based on human pose clustering according to claim 1, characterized in that, The category determination is specifically as follows: Each key point instance is normalized in two dimensions and used as a query instance; For the query instance, iterate through all cluster result center instances in the mapping file to calculate similarity, and take the positive and negative example attribute results corresponding to the cluster result center instance with the maximum value as the category of the query instance.

5. A method for automatically acquiring ROI based on human pose clustering, characterized in that, The method includes: The acquired video stream is decoded and converted into a video frame sequence; Perform a keypoint detection task on each video frame to obtain all human keypoints in the current video frame; The acquired human body key points are used as key point instances, and the acquired results are initially judged to determine the number of people. When the number of people is not zero, each key point instance is classified in conjunction with a pre-built mapping file to obtain the positive and negative example attributes of each key point instance; wherein, the mapping file includes cluster result center instances and corresponding positive and negative example attributes; Iterate through all keypoint instances that are determined to be positive examples, and select the positive example instance corresponding to the maximum value of the maximum similarity as the ROI to be analyzed; wherein, the maximum similarity is calculated by iterating through all cluster result center instances in the mapping file for each keypoint instance; The mapping file is obtained by clustering human poses, and the specific steps are as follows: Collect unlabeled datasets that meet the requirements of the final task scenario; The collected dataset is cleaned to obtain the original image dataset for further analysis; After obtaining the original image dataset, the dataset is traversed, and the human keypoint detector is used to infer for each image to obtain all human keypoint instances, which are then stored separately. By traversing the data, we obtain a dataset of individual human keypoint instances corresponding to the original image dataset, and then summarize them as a keypoint dataset. Read the key point dataset, perform two-dimensional normalization on each key point instance, and then input it into the clustering algorithm for clustering iteration to obtain the clustering result. Extract the cluster center instance. The cluster center instances are denormalized to become normal-sized keypoint instances; Visualize and render the denormalized keypoint instances, and store the results for later viewing; The obtained visualization rendering results are manually interpreted, and the set of instances that meet the requirements of the final task is selected and labeled as positive example attributes, while the remaining instances are labeled as negative example attributes. The results obtained above are used to create the mapping file, which should include the data of the cluster center instances and the labeled positive and negative example attributes. Then, a small number of samples are randomly selected from the key point dataset. The selected samples are traversed, and each sample is normalized and then the key point similarity is calculated with all cluster result center instances stored in the mapping file. The positive and negative example attributes corresponding to the result with the highest similarity are selected as the sample attribute and sample result. The accuracy rate is calculated based on the sample results, and the results are judged. If the accuracy rate is too low, the above steps should be repeated until the accuracy rate is acceptable. The mapping file that meets the accuracy requirements is used as the constructed mapping file, and the clustering process is now complete.

6. The method for automatically acquiring ROI based on human pose clustering according to claim 5, characterized in that, The method further includes: Take bounding rectangles for all keypoint instances and calculate the standard deviations of the width and height of these bounding rectangles; Iterate through all ROIs to be analyzed, take their bounding rectangles, and if the standard deviation is greater than 0.1 times the width and height, keep the center coordinates of the bounding rectangle unchanged, increase the width and height by 2 corresponding standard deviations, and then use it as a pre-selected ROI; If the standard deviation is less than 0.1 times the width and height, then the width and height are increased by 0.2 times respectively, and used as a pre-selected ROI.

7. The method for automatically acquiring ROI based on human pose clustering according to claim 6, characterized in that, The method further includes: Image boundary correction is performed on the pre-selected ROI to ensure that the obtained ROI does not exceed the image boundary. The result after boundary correction is the final ROI of the corresponding frame.

8. The method for automatically acquiring ROI based on human pose clustering according to claim 5, characterized in that, The category determination is specifically as follows: Each key point instance is normalized in two dimensions and used as a query instance; For the query instance, iterate through all cluster result center instances in the mapping file to calculate similarity, and take the positive and negative example attribute results corresponding to the cluster result center instance with the maximum value as the category of the query instance.

Images