Image annotation method, action recognition method, device and electronic device
By extracting image features and identifying key points of human body parts and generating labeling information, the problem of low accuracy and efficiency of human body parts in the image is solved, and efficient detection and labeling of human body parts is achieved.
Patent Information
- Application Number
- CN202311766853.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-20
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-12-20
AI Technical Summary
The prior art has low accuracy and efficiency in human body parts labeling in images or videos, and manual labeling has problems of erroneous operation and high cost.
By acquiring preprocessed images or video frames, extracting image features, identifying human body parts and determining their key points, generating label information, including posture information and detection frames, thereby improving the accuracy and efficiency of human body parts detection.
It improves the accuracy of human body parts detection in the image and the accuracy of subsequent tasks, reduces the cost and misoperation of manual labeling, and improves the efficiency of image labeling.
Smart Images

Figure CN117746502B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to computer vision, augmented reality, virtual reality, deep learning and other technical fields in the field of artificial intelligence technology, and can be applied to scenes such as artificial intelligence content generation and digital humans, and in particular to an image annotation method, action recognition method, device and electronic equipment. Background Art
[0002] In many application scenarios, it is necessary to analyze and process the acquired images or videos, for example, to analyze the objects included in the images or the change information of the objects in the videos.
[0003] In order to analyze and process an image or a video, multiple image frames in the image or the video may be annotated.
[0004] For images or video image frames including human images, the accuracy and efficiency of marking human body parts in these images or video image frames need to be improved. Summary of the invention
[0005] The present disclosure provides an image annotation method, an action recognition method, an apparatus and an electronic device.
[0006] According to a first aspect of the present disclosure, there is provided an image annotation method, the method comprising: acquiring a preprocessed first image, the first image comprising a user image; extracting image features from the first image, and identifying a first region corresponding to a human body part in the user image based on the extracted image features; for each human body part, identifying a plurality of first key points of the human body part in the first region corresponding to the human body part, and determining position information of a second region surrounding the human body part and posture information of the human body part based on the plurality of first key points; generating annotation information of the human body part in the first image, the annotation information comprising posture information and a detection frame obtained according to the second region.
[0007] According to a second aspect of the present disclosure, a method for action recognition is provided, the method comprising: obtaining a video clip, wherein multiple frames of images in the video clip include a user image; annotating human body parts for each frame of the multiple frames of images to obtain annotation information of the human body parts in the frame of images, wherein the annotating human body parts for each frame of images comprises: preprocessing the frame of images and extracting image features, identifying a first region of human body parts according to the image features, for each human body part, identifying multiple first key points of the human body part from the first region of the human body part, determining a second region and posture information of the human body part according to the multiple first key points, determining a detection frame of the human body part according to the second region, the annotation information including the detection frame and the posture information of the human body part; filtering the human body part detection frames corresponding to the multiple frames of images respectively to obtain the human body part detection frames of the filtered multiple frames of images; and identifying the user's action based on the annotation information of the human body parts of the filtered multiple frames of images.
[0008] According to a third aspect of the present disclosure, an image annotation device is provided, the device comprising: a first acquisition unit, used to acquire a preprocessed first image, the first image comprising a user image; a first determination unit, used to extract image features from the first image, and identify a first region corresponding to a human body part in the user image based on the extracted image features; a second determination unit, used to identify, for each human body part, a plurality of first key points of the human body part in the first region corresponding to the human body part, and determine position information of a second region surrounding the human body part and posture information of the human body part based on the plurality of first key points; a first annotation unit, used to generate annotation information of the human body part in the first image, the annotation information comprising posture information and a detection box obtained according to the second region.
[0009] According to a fourth aspect of the present disclosure, a motion recognition device is provided, the device comprising: a second acquisition unit, for acquiring a video clip, wherein multiple frames of images in the video clip include a user image; a second annotation unit, for annotating human body parts for each frame of the multiple frames of images, and obtaining annotation information of the human body parts in the frame of the image, wherein annotating human body parts for each frame of the image comprises: preprocessing the frame of the image and extracting image features, identifying a first region of the human body part according to the image features, for each human body part, identifying multiple first key points of the human body part from the first region of the human body part, determining a second region and posture information of the human body part according to the multiple first key points, determining a detection frame of the human body part according to the second region, and the annotation information including the detection frame and the posture information of the human body part; a filtering unit, for filtering the human body part detection frames corresponding to the multiple frames of images respectively, and obtaining the human body part detection frames of the multiple frames of images after filtering; and a motion recognition unit, for recognizing the user's motion based on the annotation information of the human body part of the multiple frames of images after filtering.
[0010] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods provided in the first and second aspects.
[0011] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the methods provided in the first aspect and the second aspect.
[0012] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect and the second aspect.
[0013] According to the scheme of the present disclosure, a preprocessed first image is obtained, wherein the first image includes a user image; image features are extracted from the first image, and a first region corresponding to a human body part in the user image is identified based on the extracted image features; for each human body part, multiple first key points of the human body part are identified in the first region corresponding to the human body part, and position information of a second region surrounding the human body part and posture information of the human body part are determined based on the multiple first key points; annotation information of the human body part is generated in the first image, wherein the annotation information includes posture information and a detection frame obtained based on the second region. After the first region of the human body part is first identified through image features, the first key points of the human body part are finely identified in the first region of the human body part, and then the detection frame of the human body part is determined based on the multiple first key points of the human body part. Thus, the accuracy of the human body part detection result in the image is improved. Furthermore, when a corresponding task is subsequently performed based on the human body part detection result in the image, the accuracy of the subsequent task execution result can also be improved due to the high accuracy of the human body part detection result.
[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0016] Figure 1 is a scene graph that can implement the embodiments of the present disclosure;
[0017] Figure 2 is a schematic diagram according to a first embodiment of the present disclosure;
[0018] Figure 3 is a schematic diagram according to a second embodiment of the present disclosure;
[0019] Figure 4 is a schematic diagram according to a third embodiment of the present disclosure;
[0020] Figure 5 is a schematic diagram according to a fourth embodiment of the present disclosure;
[0021] Figure 6 is a schematic diagram according to a fifth embodiment of the present disclosure;
[0022] Figure 7 is a schematic diagram according to a sixth embodiment of the present disclosure;
[0023] Figure 8 is a schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION
[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0025] In many application scenarios, it is necessary to label the targets in the image, such as in machine learning model training scenarios and video action recognition scenarios.
[0026] For human body images, it is necessary to detect various parts of the human body (such as the human body, hands, face, legs, etc.) in the human body images. Since each part of the human body has a different appearance, shape and posture, and is interfered by factors such as illumination and occlusion during imaging, it is not easy to detect various parts of the human body from the image.
[0027] Usually, image annotation includes giving a detection box (bounding box) of the target in the image. At present, in many scenarios where image annotation is required, the targets in the image are annotated manually. Manual annotation of targets in the image (such as human body parts) can obtain better annotation results. However, the manpower cost required for manual annotation is high. In addition, when using manual annotation, it is sometimes difficult to avoid incorrect annotation due to misoperation. These incorrect annotations will introduce noise, reduce the annotation accuracy, and cause deviations in subsequent tasks based on the annotation.
[0028] The present disclosure provides an image annotation method, an action recognition device, and an electronic device, which are applied to computer vision, augmented reality, virtual reality, deep learning and other technical fields in the field of artificial intelligence technology, and can be applied to scenes such as artificial intelligence content generation and digital humans, so as to achieve the purpose of improving the low efficiency of annotating targets in images in related technologies and solving the problem of low annotation accuracy caused by noise introduced by manual annotation.
[0029] Please refer to Figure 1 , Figure 1 This is a schematic diagram of an application scenario provided by the present disclosure. Figure 1 As shown, the terminal device 101 can use the image acquisition device arranged thereon to acquire images, can mark targets on the images, and then send the marked images to the server 102 for subsequent task processing.
[0030] In one implementation, the image acquisition device can acquire video, and then the terminal device 101 can annotate each frame of the acquired video, and then perform action sequence recognition on the video with the annotated information, and send the action sequence recognition result to the server, which performs subsequent task processing.
[0031] In one implementation, the terminal device can send the captured video to the server 102, which annotates each frame of the video, identifies an action sequence based on the annotated video, and then performs subsequent task processing based on the identified action sequence.
[0032] Please refer to Figure 2 , Figure 2 is a schematic diagram of the first embodiment of the present disclosure, such as Figure 2 As shown, the image annotation method provided in this embodiment includes:
[0033] S201: Acquire a preprocessed first image, where the first image includes a user image.
[0034] In this embodiment, the execution subject of the image annotation method may be a terminal device or a server.
[0035] In the application scenario of monocular motion capture, the user's action sequence can be recognized based on the input video. In order to better recognize the user's action sequence, it is necessary to obtain more local details of human body parts such as face and hands, so it is necessary to detect the human body frame and human body parts such as face and left and right hands.
[0036] An image acquisition device (eg, a monocular camera) may be used to acquire an initial image, and the acquired initial image may include a user image (ie, a human body image).
[0037] The initial image can be a single image taken by the image acquisition device, or any frame of image extracted from the video acquired by the image acquisition device. In order to avoid the accuracy of human body part recognition affected by image size, resolution, uneven lighting, etc., the acquired image can be preprocessed.
[0038] In some application scenarios, the above-mentioned execution subject may receive an image captured by an image capture device from an image capture device, and then preprocess the image to obtain a preprocessed first image.
[0039] In some other application scenarios, the initial image captured by the image acquisition device may be sent to other electronic devices, which preprocess the image to obtain the first image. The execution subject may obtain the preprocessed first image from the other electronic device.
[0040] The above-mentioned preprocessing of the image includes one or more of the following preprocessing: size adjustment, color space conversion, histogram equalization, deblurring, etc.
[0041] Resizing, including but not limited to: image scaling, cropping, and compression, refers to enlarging or reducing the size of the target image so that the size of the target image meets the preset size requirements. After image normalization, the original image is converted into an image of a specified size.
[0042] Color space conversion refers to the process of converting the representation of an image from one color space to another, such as the conversion from RSB to HSV.
[0043] By adjusting the histogram of the target image to make it evenly distributed, the grayscale balance effect can be achieved, the overly dark, overly bright and unclear points can be improved, and the overall contrast of the image can be enhanced.
[0044] Deblurring refers to preprocessing a target image using a deblurring algorithm (such as blind deconvolution) to improve the clarity of the target image.
[0045] The first image after preprocessing removes the effects of non-standard size, resolution, lighting and the like that do not meet the requirements, and therefore is conducive to accurately marking the human body parts in the image.
[0046] S202: Extracting image features from the first image, and identifying a first region corresponding to a human body part in the user image based on the extracted image features.
[0047] As an implementation method, the image features may be matched with pre-extracted human body part features, and whether the target image includes the human body part may be determined based on the matching result.
[0048] The above-mentioned human body parts include at least one human body part, such as face, hands, torso and legs.
[0049] For each human body part, the image features corresponding to the human body part can be stored in advance. It is understandable that for each human body part, the image features of the human body part in different postures can be saved. Schematically, for the face, the image features of the face when smiling, the image features of the face when crying, and the image features of the face when normal can be saved. For the hand, the image features of the fingers when open, the image features of the fingers when closed, and the image features of the hand when making a fist, etc. can be saved.
[0050] The above image features can be matched with image features of at least one human body part in different postures, and whether the target image includes an image of the human body part can be determined based on the matching result. If it is determined that the target image includes an image of the human body part, it can be further determined which human body part is included, the area of each human body part included, and the posture information corresponding to the human body part.
[0051] For each human body part, the image region corresponding to the human body part may be regarded as the first region corresponding to the human body part.
[0052] In practice, the features of multiple parts of the human body can be combined to identify whether the target image includes a human body image. After determining that the target image includes a human body image, the first area where the human body part is located is identified.
[0053] S203: For each human body part, identify a plurality of first key points of the human body part in a first area corresponding to the human body part, and determine position information of a second area surrounding the human body part and posture information of the human body part based on the plurality of first key points.
[0054] The first area may be a rectangular area, and the first area includes a complete image of a human body part. The first area of a human body part surrounds the entire image of the human body part and a partial image around the human body part.
[0055] As an implementation method, for each human body part, a first region can be cut out from the human body image (that is, an image of the human body part is cut out), and then the first region is input into the recognition model corresponding to the human body part after image preprocessing (for example, adjusting the image size), and the recognition model of the human body part recognizes multiple first key points of the human body part and the posture corresponding to the human body part. The above recognition model can be a pre-trained model based on deep learning, which is used to recognize the fine first key points of the human body part in the image input to the human body part and recognize the posture corresponding to the human body part.
[0056] After determining the multiple first key points of the human body part, a minimum rectangular frame including the multiple first key points may be determined in the image of the human body part.
[0057] For example, for a face, the first area corresponding to the face can be input into a face recognition model. The face recognition model recognizes multiple first key points of the face, such as 68 preset first key points or 150 preset first key points. The face recognition model can provide the position information of the multiple first key points of the face image in the face image and the posture information of the face.
[0058] After determining the position information of the plurality of preset first key points of the face in the face image, the minimum rectangular frame surrounding the plurality of preset first key points can be determined in the face image, and the minimum rectangular frame is mapped to the first image according to the coordinate transformation to determine the second area of the face in the first image.
[0059] For example, for the hand, the first area corresponding to the hand is extracted from the user image as a hand image, and the hand image is preprocessed (for example, the image size is adjusted) and then input into the hand recognition model. The hand recognition model recognizes multiple preset first key points of the hand, for example, 21 preset first key points, and the posture of the hand.
[0060] After determining the position information of the plurality of preset first key points of the hand in the hand image, the minimum rectangle surrounding the plurality of preset first key points can be determined in the hand image, and the minimum rectangular area is mapped to the first image according to the coordinate transformation to determine the second area of the hand in the first image.
[0061] As another implementation, for each human body part, a regression-based algorithm or a heat map-based algorithm can be used to predict multiple preset first key points of the human body part and the posture of the human body part. The regression-based algorithm includes a direct regression algorithm, a cascade regression algorithm, and a deep learning regression algorithm.
[0062] S204: Generate annotation information of the human body part in the first image, where the annotation information includes posture information and a detection frame obtained according to the second area.
[0063] After obtaining the second region and posture information of the human body part, the execution state can detect the rectangular outline of the second region in the first image according to the second region, and write out the posture of the human body part, thereby obtaining the labeling information of multiple human body parts.
[0064] It is understandable that the first image may include images of multiple users. The above steps S201 to S204 may be used for each user image to determine the labeling information of the human body parts in each user image.
[0065] In this embodiment, a first image after preprocessing is obtained, and the first image includes a user image; image features are extracted from the first image, and a first region corresponding to a human body part in the user image is identified based on the extracted image features; for each human body part, multiple first key points of the human body part are identified in the first region corresponding to the human body part, and the position information of the second region of the human body part and the posture information of the human body part are determined based on the multiple first key points; and annotation information of the human body part is generated in the first image, and the annotation information includes posture information and a detection frame obtained according to the second region. Since the first region of the human body part is first identified by the image features, the first key points of the human body part are finely identified in the first region of the human body part, and then the detection frame of the human body part is determined according to the multiple first key points of the human body part. Thereby, the accuracy of the human body part detection result in the image is improved. Then, when the corresponding subsequent tasks are performed according to the human body part detection results in the image, the accuracy of the subsequent task execution results can also be improved due to the high accuracy of the human body part detection results. This solution does not require manual annotation, which improves the efficiency of image annotation.
[0066] In some embodiments, the above step S202 includes:
[0067] The above image features are input into a pre-trained classifier, and the classifier outputs a first region corresponding to a human body part in the user image. The classifier is used to identify the human body part and the corresponding region based on the image features.
[0068] In these embodiments, the image features of the first image may include Histogram of Oriented Gradient (HOG) features. The HOG features may have good edge structure features and good block representation, and better human body part recognition may be obtained through the HOG features and then through the classifier.
[0069] The above-mentioned classifier can be various types of classifiers, such as a multi-classification support vector machine (SVM) classifier. The above-mentioned classifier is used to identify different human body parts and the first regions corresponding to each human body part according to image features. It is understood that the position of the first region here can be represented by the coordinates of the four vertices or diagonal vertices of the rectangular outline of the first region.
[0070] In these embodiments, by inputting the above-mentioned image features into a pre-trained classifier, each human body part in the first image and the first area corresponding to each human body part can be quickly determined.
[0071] Please refer to Figure 3 , Figure 3 is a schematic diagram of the second embodiment of the present disclosure, such as Figure 3 As shown, the image annotation method provided in this embodiment includes:
[0072] S301: Acquire a preprocessed first image, where the first image includes a user image.
[0073] In this embodiment, the execution subject of the image annotation method may be a terminal device or a server.
[0074] The specific implementation of the above step S301 can refer to Figure 1 The description of step S101 of the illustrated embodiment is not repeated here.
[0075] S302: Extract image features from the first image.
[0076] S303: Determine a human body image region according to the image features, and extract a human body image from the first image based on the human body image region.
[0077] Specifically, the first image may be input into a convolutional neural network to extract image features of the first image. The convolutional neural network may be a convolutional neural network pre-trained to extract image features from an image.
[0078] The image features can be matched with the features of the human body image, and the human body image region can be determined from the first image according to the matching result. The human body image region can be a rectangular region.
[0079] Then, the human body image is extracted from the first image according to the human body image region. For example, the human body image region is cut out from the first image to obtain the human body image.
[0080] As an implementation method, the above image features can be input into a human body detection model, and the human body detection model outputs a human body image region, and then the human body image is extracted from the first image according to the human body image region.
[0081] The human body detection model can identify a human body image region surrounding the human body image from the first image according to the image features. The outline of the human body image region can be a rectangle.
[0082] S304: Input the human body image into a pre-trained human body key point recognition model to obtain a plurality of human body second key points.
[0083] S305: Determine first regions corresponding to different human body parts based on multiple second key points of the human body.
[0084] The human body image cut from the first image can be pre-processed (e.g., enlarged or reduced) and then input into a pre-trained human body key point recognition model, which outputs a plurality of human body second key points. The human body key point recognition model can be various human body key point recognition models for recognizing full-body key points in human body images.
[0085] The multiple second key points of the human body in this embodiment may include multiple preset human body key points, for example, 18 preset key points, including key points corresponding to major joints such as the top of the head, facial features, neck, and limbs.
[0086] The above-mentioned human key point recognition model includes but is not limited to: OpenPose model, MoveNet model, PoseNet model, etc. The above-mentioned human key point recognition model can recognize human key points and posture information.
[0087] The human body image can be input into the human body key recognition model, and the human body key point recognition model outputs the human body key point information. The human body key point information can include the location of each second key point of the human body, the posture of the corresponding part and the semantic information.
[0088] In some embodiments, for each human body part, a rectangular box surrounding at least one second key point corresponding to the human body part is determined based on the position of the at least one second key point corresponding to the human body part, and the first area is determined based on the rectangular box; or a rectangular box surrounding the human body part is determined based on at least one second key point of the human body part and at least one second key point of an adjacent human body part and human body part distribution information, and the first area is determined based on the rectangular box.
[0089] In some application scenarios, for each human body part, an initial rectangular frame surrounding the at least one second key point of the human body part can be determined, and then the initial rectangular area can be enlarged by a preset multiple to obtain the rectangular frame of the human body part.
[0090] For example, for a face, based on the key points of the two ears and the key point of the nose, the initial rectangular frame is taken as the center and the box including the two ears is enlarged 1.5 times to determine the rectangular frame including the face.
[0091] In some application scenarios, a rectangular frame surrounding the human body part can be determined based on at least one second key point of the human body part, at least one second key point of an adjacent human body part, and distribution characteristics of the human body parts, and the first area can be determined based on the rectangular frame.
[0092] For example, with respect to the hand, according to the distribution characteristics of human body parts, the length and width of the hand will not be greater than the distance from the wrist to the elbow.
[0093] The distance between the detected second key point of the elbow and the second key point of the wrist may be used as the side length to generate a square frame including the human hand, and the square frame may be used as a rectangular frame of the hand.
[0094] For each human body part, after determining the rectangular frame corresponding to the human body part in the human body image, the area enclosed by the rectangular frame can be mapped to the first image based on the coordinate conversion relationship to obtain the first area of the human body part in the first image.
[0095] In these embodiments, the first area of each human body part can be identified by determining the first area based on a rectangular frame surrounding at least one second key point corresponding to the human body part; or, determining the first area based on at least one second key point of the human body part and at least one second key point of an adjacent human body part and distribution characteristics of the human body parts.
[0096] Image features can be extracted from the preprocessed target image, for example, directional gradient histogram features of the target image can be extracted. The directional gradient histogram features can better outline the contour of the human object in the image. The human image region is identified based on the image features, for example, the directional gradient histogram features are used to use human contour and edge detection to determine the human image region. The human image region can be a rectangular region.
[0097] S306. For each human body part, identify multiple first key points of the human body part in a first area corresponding to the human body part, and determine position information of a second area surrounding the human body part and posture information of the human body part based on the multiple first key points.
[0098] After obtaining the first regions corresponding to multiple human body parts, for each first region of a human body part, the first region may include other image information other than the human body part, so the first region is a larger region. The first region carries other interfering image information other than the human body part, which is not conducive to the execution of subsequent tasks.
[0099] In order to more accurately identify the area surrounding the human body part and reduce interference image information, the first area needs to be further refined based on the first area corresponding to the human body part.
[0100] Taking the face as an example, after obtaining the first area corresponding to the face, the image in the first area can be processed again to determine multiple first key points corresponding to the face. The number of the first key points of the face is greater than the number of the second key points of the face identified in the human body key point recognition.
[0101] As an implementation method, the above step S305 includes the following steps:
[0102] Firstly, for a human body part, an image of the human body part is captured based on a first area of the human body part.
[0103] Secondly, the image of the human body part is input into a pre-trained human body part recognition model, and the human body part recognition model outputs a plurality of first key points of the human body part and posture information of the human body part.
[0104] Still taking the face as an example, the face image can be extracted from the first image according to the first area corresponding to the face. The face image is then preprocessed, such as scaling, so that the scaled face image meets the input requirements of the face recognition model.
[0105] The face recognition model can analyze and process the input facial image, identify multiple preset first key points included in the facial image, and recognize facial gestures according to the distribution of the multiple preset first key points.
[0106] The face recognition model identifies multiple preset first key points of the face, such as 68 preset first key points or 150 first key points. The face recognition model can provide the position information of the multiple first key points of the face image in the face image and the posture information of the face.
[0107] After determining the position information of the plurality of preset first key points of the face in the face image, the minimum rectangular frame surrounding the plurality of preset first key points can be determined in the face image, and the minimum rectangular frame is mapped to the first image according to the coordinate transformation to determine the second area of the face in the first image.
[0108] In these embodiments, by inputting an image of a human body part into a corresponding human body part recognition model, a plurality of first key points corresponding to the human body part and posture information corresponding to the human body part can be quickly recognized.
[0109] S307: Generate annotation information of the human body part in the first image, where the annotation information includes posture information and a detection frame obtained according to the second area.
[0110] After obtaining the second region and posture information of the human body part, the execution state can draw a rectangular outline of the second region in the first image according to the second region, and write the posture of the human body part, thereby obtaining labeling information of multiple human body parts.
[0111] In this embodiment, it is described that a human body image is extracted from a first image, and the human body image is input into a pre-trained human body key point recognition model to obtain multiple second key points of the human body, and then the contents of the first areas corresponding to each human body part are determined according to the multiple second key points, so that the first areas corresponding to each human body part can be identified more accurately, and the accuracy of the detection frame of the identified human body part can be improved.
[0112] Figure 4 is a schematic diagram according to the third embodiment of the present disclosure. Figure 4 As shown, the action recognition method provided in this embodiment includes the following steps:
[0113] S401: Acquire a video clip, where multiple frames of images in the video clip include a user image.
[0114] The execution subject of this embodiment may be a terminal device or a server.
[0115] In some application scenarios, the video clips may be video clips collected in real time or video clips collected in advance.
[0116] The video clip may include an image frame sequence composed of multiple image frames in time sequence. The multiple image frames of the video clip may include a user image.
[0117] Each frame of image may include user images of one or more users.
[0118] S402: annotating human body parts for each frame of multiple images to obtain annotation information of the human body parts in the frame of images, wherein annotating human body parts for each frame of images includes: preprocessing the frame of images and extracting image features, identifying a first region of the human body parts according to the image features, for each human body part, identifying a plurality of first key points of the human body part from the first region of the human body part, determining a second region and posture information of the human body part according to the plurality of first key points, determining a detection frame of the human body part according to the second region, and the annotation information includes the detection frame and the posture information of the human body part.
[0119] Image preprocessing can be performed on each frame of the image, and then human body parts detection can be performed on the preprocessed image.
[0120] When performing human body part detection on each frame image, the image features of the frame image can be extracted, and multiple second key points of the human body can be identified based on the image features. Different human body parts and the first areas corresponding to each human body part are identified based on the multiple second key points of the human body. For each human body part, multiple first key points corresponding to the human body part are identified based on the image in the first area corresponding to the human body part. The number of first key points of a human body part is greater than the number of second key points corresponding to the human body part. The minimum rectangular frame surrounding the multiple first key points is determined based on the positions of the multiple first key points, and the posture information of each part of the human body is identified based on the multiple first key points. The detection frame of the human body part in the frame image is determined based on the minimum rectangular frame.
[0121] Specifically, please refer to Figure 1 or Figure 2 The description of the illustrated embodiment is not repeated here.
[0122] S403: Filtering the human body part detection frames corresponding to the multiple frames of images to obtain the human body part detection frames of the multiple frames of images after filtering.
[0123] After obtaining the human body part detection frames corresponding to each of the multiple frame images, for each human body part, a preset filtering method can be used to filter the multiple detection frames of the human body part, such as using oneeuro filtering or using smoothnet based on deep learning to filter the multiple detection frames.
[0124] When detecting the human body parts in each frame image, it is inevitable that detection errors will occur. In order to reduce the deviation of the final action recognition result caused by the above errors, the human body part detection frames in each frame image in the image sequence can be filtered to eliminate abnormal detection frames.
[0125] After filtering the human body part detection frames corresponding to the multiple frames of images, the abnormal detection frames can be eliminated, so that the connection line of the center points of the detection frames corresponding to the retained multiple frames of images is smooth. Figure 1 or Figure 2 The illustrated embodiment introduces erroneous annotations when annotating each image.
[0126] S404: Recognize the user's actions based on the labeled information of the human body parts in the filtered multi-frame images.
[0127] For continuous image frames in a video, the user's actions can be determined by combining position change information of detection frames of different human body parts and posture change information.
[0128] For each human body part, a detection frame sequence carrying position information and posture information of the human body part is generated according to the time and detection frames corresponding to the multiple frames of images, the position change information of the detection frame is determined according to the position information of each detection frame in the detection frame sequence, and the posture change information of the human body part is determined according to the posture information of each detection frame.
[0129] The motion information of the human body part is determined based on the position change information of the detection frame and the posture change information of the human body part.
[0130] As an implementation method, the above step S404 includes the following steps:
[0131] Firstly, for each human body part, a labeling information sequence of the human body part is generated according to the labeling information of the human body part in the filtered multi-frame images.
[0132] Secondly, the labeling information sequences corresponding to the multiple human body parts are input into the action recognition model, and the action recognition model recognizes the user action information by combining the labeling information sequences corresponding to the multiple human body parts.
[0133] The above-mentioned action recognition model can be a machine learning model that is pre-trained using a supervised training method.
[0134] When training the action recognition model, the annotation information sequences and actions corresponding to multiple human body parts extracted from video clips corresponding to different actions can be used as training data pairs, and the action recognition model can be trained using multiple training data pairs to obtain a trained action recognition model.
[0135] The trained action recognition model can recognize the user's action information according to the input multiple human body parts corresponding to the label information sequence. The recognized user's action may include an action sequence formed by multiple actions. Since the human action is recognized by combining the label information sequence corresponding to the multiple human body parts, the accuracy of the recognized action can be improved.
[0136] In this embodiment, a video clip is obtained, and multiple frames of images in the video clip include user images; each frame of the multiple frames is annotated with human body parts to obtain the annotated information of the human body parts in the frame, wherein the annotating of human body parts for each frame includes: preprocessing the frame and extracting image features, identifying the first region of the human body parts according to the image features, identifying multiple first key points of the human body parts from the first region of the human body parts for each human body part, determining the second region and posture information of the human body parts according to the multiple first key points, determining the detection frame of the human body parts according to the second region, and the annotated information includes the detection frame and the posture information of the human body parts; filtering the human body part detection frames corresponding to the multiple frames of images respectively to obtain the human body part detection frames of the filtered multiple frames of images; and identifying the user's actions based on the annotated information of the human body parts of the filtered multiple frames of images. Since the obtained detection frame of the human body parts has a high precision, in addition, filtering the detection frame sequence before performing action recognition, and performing action recognition according to the filtered detection frame information can improve the accuracy of action recognition.
[0137] Figure 5 is a schematic diagram according to the fourth embodiment of the present disclosure. Figure 5 As shown, the image annotation device 500 provided in this embodiment includes: a first acquisition unit 501, a first determination unit 502, a second determination unit 503 and a first annotation unit 504.
[0138] A first acquisition unit 501 is used to acquire a preprocessed first image, where the first image includes a user image;
[0139] A first determining unit 502 is used to extract image features from the first image, and identify a first region corresponding to a human body part in the user image based on the extracted image features;
[0140] A second determining unit 503 is used to identify, for each human body part, a plurality of first key points of the human body part in a first area corresponding to the human body part, and determine position information of a second area surrounding the human body part and posture information of the human body part based on the plurality of first key points;
[0141] The first labeling unit 503 is used to generate labeling information of the human body part in the first image, where the labeling information includes posture information and a detection frame obtained according to the second area.
[0142] In some embodiments, the first determining unit 502 includes a first identifying module 5021, and the first identifying module 5021 is used to:
[0143] The image features are input into a pre-trained classifier, and the classifier outputs a first region corresponding to a human body part in the user image. The classifier is used to identify the human body part and the corresponding region based on the image features.
[0144] In some embodiments, the first determination unit 502 includes a human image extraction module 5022, a first input module 5023 and a first determination module 5024, wherein:
[0145] A human image extraction module 5022, configured to determine a human image region according to image features, and extract a human image from the first image based on the human image region;
[0146] The first input module 5023 is used to input the human body image into a pre-trained human body key point recognition model to obtain a plurality of human body second key points;
[0147] The first determination module 5024 is used to determine first regions corresponding to different human body parts based on multiple second key points of the human body.
[0148] In some embodiments, the first determining module 5024 is further configured to:
[0149] For each human body part, a rectangular frame surrounding at least one second key point corresponding to the human body part is determined according to the position of the at least one second key point, and the first area is determined according to the rectangular frame; or
[0150] A rectangular frame surrounding the human body part is determined based on at least one second key point of the human body part, at least one second key point of an adjacent human body part, and distribution characteristics of the human body parts, and a first area is determined based on the rectangular frame.
[0151] In some embodiments, the second determination unit 503 includes an interception module 5031 and a second input module 5032;
[0152] The capture module 5031 is used for capturing an image of a human body part based on a first region of the human body part;
[0153] The second input module 5032 is used to input the image of the human body part into the pre-trained human body part recognition model, and the human body part recognition model outputs multiple first key points of the human body part and the posture information of the human body part.
[0154] In some embodiments, preprocessing includes one or more of the following:
[0155] Resizing, color space conversion, histogram equalization, deblurring.
[0156] The device of this embodiment can execute Figure 2The specific implementation process and technical principles of the technical solution in the method shown are the same and will not be repeated here.
[0157] Figure 6 is a schematic diagram according to a fifth embodiment of the present disclosure, Figure 6 As shown, the action recognition device 600 provided in this embodiment includes a second acquisition unit 601, a second labeling unit 602, a filtering unit 603 and an action recognition unit 604.
[0158] A second acquisition unit 601 is used to acquire a video clip, where multiple frames of images in the video clip include a user image;
[0159] The second labeling unit 602 is used to label human body parts for each frame of the multiple frames of images to obtain labeling information of the human body parts in the frame of the image, wherein labeling human body parts for each frame of the image includes: preprocessing the frame of the image and extracting image features, identifying a first region of the human body parts according to the image features, for each human body part, identifying a plurality of first key points of the human body part from the first region of the human body part, determining a second region and posture information of the human body part according to the plurality of first key points, determining a detection frame of the human body part according to the second region, and the labeling information includes the detection frame and the posture information of the human body part;
[0160] A filtering unit 603 is used to filter the human body part detection frames corresponding to the multiple frames of images to obtain the human body part detection frames of the multiple frames of images after filtering;
[0161] The action recognition unit 604 is used to recognize the user's action based on the annotation information of the human body parts in the filtered multiple frames of images.
[0162] In some embodiments, the action recognition unit 604 includes a sequence generation module 6041 and an action recognition module 6042;
[0163] The sequence generation module 6041 is used to generate, for each human body part, a sequence of annotation information of the human body part according to the annotation information of the human body part in the filtered multiple frames of images.
[0164] The action recognition module 6042 is used to input the labeling information sequences corresponding to multiple human body parts into the action recognition model, and the action recognition model combines the labeling information sequences corresponding to multiple human body parts to recognize user action information.
[0165] The device of this embodiment can execute Figure 4 The specific implementation process and technical principles of the technical solution in the method shown are the same and will not be repeated here.
[0166] Figure 7 is a schematic diagram according to a sixth embodiment of the present disclosure, Figure 7 As shown, the electronic device 700 in this embodiment may include: a processor 701 and a memory 702 .
[0167] The memory 702 is used to store programs. The memory 702 may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc. The memory may also include non-volatile memory, such as flash memory. The memory 702 is used to store computer programs (such as applications, functional modules, etc. that implement the above methods), computer instructions, etc. The above computer programs, computer instructions, etc. may be partitioned and stored in one or more memories 702. And the above computer programs, computer instructions, data, etc. may be called by the processor 701.
[0168] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories 702 in partitions. And the above-mentioned computer programs, computer instructions, etc. may be called by the processor 701.
[0169] The processor 701 is used to execute the computer program stored in the memory 702 to implement each step of the method involved in the above embodiment.
[0170] For details, please refer to the relevant description in the previous method embodiment.
[0171] The processor 701 and the memory 702 may be independent structures or integrated structures. When the processor 701 and the memory 702 are independent structures, the memory 702 and the processor 701 may be coupled and connected via a bus 703 .
[0172] The electronic device of this embodiment can execute the technical solution in the above method, and its specific implementation process and technical principle are the same, which will not be repeated here.
[0173] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0174] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute a solution provided by any of the above embodiments.
[0175] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.
[0176] Figure 8 A schematic block diagram of an example electronic device that can be used to implement an embodiment of the present disclosure is shown. Among them, the electronic device 800 can be a terminal device or a server. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0177] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0178] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0179] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the generation operation process of the electronic case report form. For example, in some embodiments, the generation operation process of the electronic case report form may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the generation operation process of the electronic case report form described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the electronic case report form generation operation process in any other appropriate manner (eg, by means of firmware).
[0180] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0181] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0182] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0183] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0184] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0185] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.
[0186] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0187] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. An image annotation method, comprising: Acquire a preprocessed first image, where the first image includes a user image; Extracting image features from the first image, and identifying a first region corresponding to a human body part in the user image based on the extracted image features; For each human body part, a plurality of first key points of the human body part are identified in a first area corresponding to the human body part, and position information of a second area surrounding the human body part and posture information of the human body part are determined based on the plurality of first key points, wherein the accuracy of the position information of the second area is higher than that of the first area; Generate annotation information of the human body part in the first image, wherein the annotation information includes posture information and a detection frame obtained according to the second area.
2. The method according to claim 1, characterized in that The step of extracting image features from the first image and identifying a first region corresponding to a human body part in the user image based on the extracted image features includes: The image features are input into a pre-trained classifier, and the classifier outputs a first region corresponding to a human body part in the user image. The classifier is used to identify the human body part and the corresponding region based on the image features.
3. The method according to claim 1, characterized in that The step of extracting image features from the first image and identifying a first region corresponding to a human body part in the user image based on the extracted image features includes: Determine a human body image region according to the image features, and extract a human body image from the first image based on the human body image region; Inputting the human body image into a pre-trained human body key point recognition model to obtain a plurality of human body second key points; Based on the multiple second key points of the human body, first regions corresponding to different parts of the human body are determined.
4. The method according to claim 3, characterized in that: The determining of first regions corresponding to different human body parts respectively based on the plurality of second key points of the human body comprises: For each human body part, a rectangular frame surrounding at least one second key point is determined according to the position of at least one second key point corresponding to the human body part, and the first area is determined according to the rectangular frame; or A rectangular frame surrounding the human body part is determined based on at least one second key point of the human body part, at least one second key point of an adjacent human body part, and distribution characteristics of the human body parts, and a first area is determined based on the rectangular frame.
5. The method according to claim 1, characterized in that The method of identifying, for each human body part, a plurality of first key points of the human body part in a first area corresponding to the human body part, and determining position information of a second area surrounding the human body part and posture information of the human body part based on the plurality of first key points, comprises: For a human body part, based on a first area of the human body part, capturing an image of the human body part; The image of the human body part is input into the pre-trained human body part recognition model, and the human body part recognition model outputs a plurality of first key points of the human body part and posture information of the human body part.
6. The method according to any one of claims 1 to 5, characterized in that The pre-processing includes one or more of the following: Resizing, color space conversion, histogram equalization, deblurring.
7. A method for motion recognition, comprising: Acquire a video clip, wherein the multiple frames of images in the video clip include a user image; For each frame of multiple images, human body parts are annotated to obtain the annotated information of the human body parts in the frame of images, wherein the human body parts annotated for each frame of images comprises: preprocessing the frame of images and extracting image features, identifying a first region of human body parts according to the image features, for each human body part, identifying a plurality of first key points of the human body part from the first region of the human body part, determining a second region and posture information of the human body part according to the plurality of first key points, determining a detection frame of the human body part according to the second region, the annotated information comprising the detection frame and the posture information of the human body part, and the accuracy of the position information of the second region is higher than that of the first region; Filtering the human body part detection frames corresponding to the multiple frames of images respectively to obtain the human body part detection frames of the multiple frames of images after filtering; Based on the labeled information of human body parts in the filtered multi-frame images, the user's actions are recognized.
8. The method according to claim 7, characterized in that The identifying of the user's actions based on the labeling information of the human body parts of the filtered multiple-frame images includes: For each human body part, generating a labeling information sequence of the human body part according to the labeling information of the human body part in the filtered multiple frames of images; The annotation information sequences corresponding to the multiple human body parts are input into the action recognition model, and the action recognition model recognizes the user action information by combining the annotation information sequences corresponding to the multiple human body parts.
9. An image annotation device, comprising: A first acquisition unit, configured to acquire a preprocessed first image, wherein the first image includes a user image; A first determining unit, configured to extract image features from the first image, and identify a first region corresponding to a human body part in the user image based on the extracted image features; A second determining unit is used to identify, for each human body part, a plurality of first key points of the human body part in a first area corresponding to the human body part, and determine position information of a second area surrounding the human body part and posture information of the human body part based on the plurality of first key points, wherein the accuracy of the position information of the second area is higher than that of the first area; The first labeling unit is used to generate labeling information of the human body part in the first image, where the labeling information includes posture information and a detection frame obtained according to the second area.
10. The device according to claim 9, characterized in that The first determining unit includes a first identifying module, and the first identifying module is used to: The image features are input into a pre-trained classifier, and the classifier outputs a first region corresponding to a human body part in the user image. The classifier is used to identify the human body part and the corresponding region based on the image features.
11. The device according to claim 10, characterized in that The first determination unit includes a human image extraction module, a first input module and a first determination module, wherein: The human image extraction module is used to determine a human image region according to the image features, and extract a human image from the first image based on the human image region; The first input module is used to input the human body image into a pre-trained human body key point recognition model to obtain a plurality of human body second key points; The first determination module is used to determine first areas corresponding to different human body parts based on the multiple second key points of the human body.
12. The device according to claim 11, characterized in that The first determining module is further used for: For each human body part, determine a rectangular frame surrounding the at least one second key point according to the position of the at least one second key point corresponding to the human body part, and determine the first area according to the rectangular frame; or, A rectangular frame surrounding the human body part is determined based on at least one second key point of the human body part, at least one second key point of an adjacent human body part, and distribution characteristics of the human body parts, and a first area is determined based on the rectangular frame.
13. The device according to claim 9, characterized in that The second determination unit includes an interception module and a second input module; The capture module is used to capture an image of a human body part based on a first area of the human body part; The second input module is used to input the image of the human body part into the pre-trained human body part recognition model, and the human body part recognition model outputs multiple first key points of the human body part and posture information of the human body part.
14. The device according to any one of claims 9 to 13, characterized in that The pre-processing includes one or more of the following: Resizing, color space conversion, histogram equalization, deblurring.
15. A motion recognition device, comprising: A second acquisition unit, configured to acquire a video clip, wherein the multiple frames of images in the video clip include a user image; A second labeling unit is used to label human body parts for each frame of the multiple frames of images to obtain labeling information of the human body parts in the frame of the images, wherein labeling human body parts for each frame of the images includes: preprocessing the frame of the images and extracting image features, identifying a first region of human body parts according to the image features, for each human body part, identifying a plurality of first key points of the human body part from the first region of the human body part, determining a second region and posture information of the human body part according to the plurality of first key points, determining a detection frame of the human body part according to the second region, the labeling information includes the detection frame and the posture information of the human body part, and the accuracy of the position information of the second region is higher than that of the first region; A filtering unit, used for filtering the human body part detection frames corresponding to the multiple frames of images respectively, to obtain the human body part detection frames of the multiple frames of images after filtering; The action recognition unit is used to recognize the user's action based on the annotation information of the human body parts in the filtered multi-frame images.
16. The device according to claim 15, characterized in that The action recognition unit includes a sequence generation module and an action recognition module; The sequence generation module is used to generate a sequence of annotation information of each human body part according to the annotation information of the human body part in the filtered multiple frames of images; The action recognition module is used to input the labeling information sequences corresponding to multiple human body parts into the action recognition model, and the action recognition model combines the labeling information sequences corresponding to the multiple human body parts to recognize the user action information.
17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.
18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.
19. A computer program product, comprising a computer program, which implements the steps of the method according to any one of claims 1 to 8 when executed by a processor.
