Motion image labeling method and device based on edge detection, equipment and medium

By extracting the effective frame sequence of key action in motion video and using the Canny edge detection method, the problems of low motion image annotation efficiency and insufficient matching degree in the prior art are solved, and efficient motion image annotation and matching effect are achieved.

CN118918587APending Publication Date: 2024-11-08BEIJING BILI INFORMATION & TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411300333.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-18
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing moving image annotation method is inefficient and cannot keep up with the moving image frequency. The motion profile in the labeled image does not match the moving image.

Method used

By obtaining motion video, the effective frame sequence of key actions is extracted and the action outline edge is extracted using the Canny edge detection method to generate motion labeling prompts to improve the efficiency and matching of image labeling.

Benefits of technology

It improves the efficiency and matching degree of moving image annotation, reduces labor costs, and ensures a high degree of matching between the image and moving image after the annotation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918587A_ABST
    Figure CN118918587A_ABST
Patent Text Reader

Abstract

The invention provides a motion image labeling method and device based on edge detection, electronic equipment and a storage medium, and the method comprises the steps: firstly obtaining a motion video; then, extracting a key action effective frame sequence in the motion video; then, extracting an action contour edge of each key action effective frame in the key action effective frame sequence; and finally, based on the action contour edge of each key action effective frame, generating a motion labeling prompt for each key action effective frame, and adding the motion labeling prompt to the corresponding key action effective frame in the key action effective frame sequence to obtain a target motion image sequence with action labeling. According to the method, the key action effective frame sequence in the motion video is extracted, and the extracted key action effective frame is subjected to automatic motion contour extraction and labeling, so that the efficiency of labeling the motion image is improved, and the matching degree of the motion contour in the labeled image and the motion image is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to a motion image annotation method and device based on edge detection, an electronic device, and a storage medium. Background Art

[0002] In addition to being mainly powered by oxygen, aerobic exercise also requires the participation of major muscle groups throughout the body, and the exercise lasts for a long time and is rhythmic. Aerobic exercise can train the heart and lung functions, allowing the cardiovascular system to more effectively and quickly transfer oxygen to every part of the body. Through regular aerobic exercise, the human heart function is stronger, the pulse output is greater, the oxygen supply capacity is stronger, and the pulse rate will be appropriately reduced.

[0003] Since aerobic exercise often involves non-standard movements or forgotten movements, it is necessary to display movement prompts in the motion image. Existing motion prompts in motion images are mostly achieved by manually drawing and annotating, such as Figure 1A , 1B As shown, the existing method not only has the defects of low efficiency and inability to keep up with the frequency of motion images, but also has the problem that the motion contour in the annotated image cannot match the motion image. Summary of the invention

[0004] The present disclosure at least provides a method and device for annotating a moving image based on edge detection, an electronic device, and a storage medium to improve the efficiency of image annotation and the matching degree between the motion contour in the annotated image and the moving image. According to one aspect of the present disclosure, a method for annotating a moving image based on edge detection is provided, comprising: Get sports videos; Extracting a key action effective frame sequence in the sports video; Extracting the action contour edge of each key action valid frame in the key action valid frame sequence; Based on the action contour edge of each key action valid frame, motion annotation hints are generated for each key action valid frame, and the motion annotation hints are added to the corresponding key action valid frames in the key action valid frame sequence to obtain a target motion image sequence with action annotations.

[0005] In a possible implementation manner, the first key action valid frame in the key action valid frame sequence is determined according to the following steps: Starting from the first frame image of the motion video, determine whether the image is a key action valid frame according to the following steps until the first key action valid frame is determined: The frame image is divided into image blocks of a preset size, and the current image block is taken as the first image block in order from left to right and from top to bottom. If the first image block has a nearest left image block, the nearest left image block is taken as the second image block; if the nearest left image block does not exist, the nearest image block above the first image block is taken as the second image block; For the first image block and the second image block, calculate the difference of pixel values ​​pixel by pixel, and calculate the cumulative value of the absolute value of the difference; When the accumulated value is greater than or equal to the preset value, the frame image is used as the first key action valid frame.

[0006] In a possible implementation manner, at least some of the key action valid frames in the key action valid frame sequence except the first key action valid frame are determined according to the following steps: The first key action valid frame is taken as the standard frame; The following steps are executed repeatedly to determine the key action valid frame until the standard frame is the N-1th frame, where N is the number of frames of the motion video: Each frame image following the standard frame in the motion video is traversed until an image whose similarity to the standard frame is less than or equal to a preset similarity is determined, and the determined image is used as a key action valid frame and a new standard frame.

[0007] In a possible implementation, the method of the present disclosure further includes: If the similarity between the Nth frame image and the N-1th frame image in the motion video is less than the preset similarity, the Nth frame image is used as a key action valid frame.

[0008] In a possible implementation manner, extracting a key action valid frame sequence in the motion video further includes: According to the time sequence of the key action valid frames in the motion image, the key action valid frames are combined to obtain a key action valid frame sequence.

[0009] In a possible implementation manner, extracting the action contour edge of each key action valid frame in the key action valid frame sequence includes: The Canny edge detection method is used to extract the edge of the action contour of each key action valid frame in the key action valid frame sequence.

[0010] In a possible implementation, the method of the present disclosure further includes: Use MSE or SSIM to determine the similarity between images.

[0011] According to another aspect of the present disclosure, a motion image annotation device based on edge detection is provided, comprising: A video acquisition module, used to acquire motion videos; An effective frame determination module, used for extracting a key action effective frame sequence in the motion video; A contour extraction module, used to extract the action contour edge of each key action valid frame in the key action valid frame sequence; The annotation module is used to generate motion annotation prompts for each key action valid frame based on the motion contour edge of each key action valid frame, and add the motion annotation prompts to the corresponding key action valid frame in the key action valid frame sequence to obtain a target motion image sequence with action annotations.

[0012] According to another aspect of the present disclosure, an electronic device is provided, comprising a memory, a processor and a computer program stored on the memory, wherein the processor implements any of the above methods when executing the computer program. According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program implements any of the above methods when executed by the processor. The motion image annotation method based on edge detection disclosed in the present disclosure first acquires a motion video; then, extracts a key action valid frame sequence in the motion video; then, extracts the motion contour edge of each key action valid frame in the key action valid frame sequence; finally, based on the motion contour edge of each key action valid frame, generates motion annotation prompts for each key action valid frame, and adds the motion annotation prompts to the corresponding key action valid frames in the key action valid frame sequence, thereby obtaining a target motion image sequence with action annotations. The present disclosure improves the efficiency of motion image annotation by extracting a key action valid frame sequence in a motion video, and automatically extracting and annotating the extracted key action valid frames, thereby improving the matching degree between the motion contour in the annotated image and the motion image. It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description.

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Figure 1A It’s a moving image; Figure 1B It is a manually annotated image; Figure 2 is a flow chart of a motion image annotation method based on edge detection according to the present disclosure; Figure 3A is a schematic diagram of segmenting an image according to the solution of the present disclosure; Figure 3B is an image annotated using the solution disclosed herein; Figure 4 is a structural schematic diagram of a motion image annotation device based on edge detection according to the present disclosure; Figure 5 Detailed description of the invention The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, which includes various details of the embodiments of the present disclosure to facilitate understanding, and they should be considered as merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Likewise, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. In view of the problems of low efficiency, inability to keep up with the frequency of motion images, and inability to match the motion contours in the annotated images with the motion images in the existing methods of annotating motion images, the present disclosure provides a motion image annotation method and device based on edge detection, an electronic device, and a storage medium. The scheme of the present disclosure improves the efficiency of motion image annotation by extracting a sequence of key action valid frames in a motion video, and automatically extracting and annotating the contours of the extracted key action valid frames, thereby improving the matching degree of the motion contours in the annotated images with the motion images. The technical scheme of the present disclosure is described below through specific embodiments. As shown in FIG. Figure 2 FIG. 1 is a flowchart of a motion image annotation method based on edge detection in this embodiment. The execution subject of this embodiment is a computing device or component with data processing capability. Specifically, the method in this embodiment may include the following steps: S110: Obtain sports video.

[0014] Sports videos are videos shot while athletes are exercising.

[0015] S120: Extracting a key action valid frame sequence in the sports video.

[0016] Before executing this step, each frame of the image is completely extracted according to the original frame rate of the motion video, and then the key action valid frame sequence is extracted from the extracted image.

[0017] The key action valid frame sequence includes multiple key action valid frames, and these key action valid frames all include the key actions performed by the athletes. By extracting the key action valid frames, invalid images with high repetition can be removed, thereby improving the efficiency of action annotation.

[0018] S130 , extracting the action contour edge of each key action valid frame in the key action valid frame sequence.

[0019] Specifically, the Canny edge detection method may be used to extract the edge of the action contour of each key action valid frame in the key action valid frame sequence.

[0020] S140 , based on the action contour edge of each key action valid frame, generate motion annotation hints for each key action valid frame, and add the motion annotation hints to the corresponding key action valid frames in the key action valid frame sequence to obtain a target motion image sequence with action annotations.

[0021] In this step, compensation adjustment can also be performed based on the edge contour of each frame of the image, and then motion annotation and other processing can be performed to generate a continuous motion annotation prompt effect diagram, that is, the above-mentioned target motion image sequence.

[0022] The above embodiment can quickly extract a valid frame sequence including key actions, thereby improving the efficiency of motion contour annotation. The extracted motion contour and the motion image have a very high similarity, thereby effectively guiding users to perform healthy aerobic exercises.

[0023] In some embodiments, the first key action valid frame in the key action valid frame sequence may be determined according to the following steps: Starting from the first frame image of the motion video, determine whether the image is a key action valid frame according to the following steps until the first key action valid frame is determined: The frame image is divided into image blocks of preset sizes, and the current image block is taken as the first image block in order from left to right and from top to bottom. If the first image block has a nearest left image block, the nearest left image block is taken as the second image block; if the nearest left image block does not exist, the nearest image block above the first image block is taken as the second image block; for the first image block and the second image block, the difference of pixel values ​​is calculated pixel by pixel, and the cumulative value of the absolute value of the difference is calculated; when the cumulative value is greater than or equal to the preset value, the frame image is taken as the first frame key action valid frame.

[0024] like Figure 3A As shown, starting from the first frame of the motion video, each frame is divided into image blocks according to the n*m ​​pixel matrix. The size of the image block can be set dynamically. The data of the image blocks are obtained in sequence according to [i * n, j * m], i, j starting from 0, and the data that is not large enough for the image block size is discarded.

[0025] The image block at position [0, 0] is skipped directly, and from left to right and from top to bottom, the current image block B1, i.e. the first image block mentioned above, is taken. If there is a nearest image block on the left, it is recorded as image block B2, i.e. the second image block mentioned above; if there is no nearest image block on the left, the nearest image block above is taken, recorded as B2. The absolute value of the difference between the B1 and B2 image blocks is calculated pixel by pixel, and then all the absolute values ​​are accumulated, and the sum is recorded as S. If S is less than the preset value T, the current frame image is skipped and the next frame image is continued. The above steps are repeated until S>=T, and the frame is recorded as the first key action valid frame.

[0026] In some embodiments, at least some of the key action valid frames in the key action valid frame sequence except the first key action valid frame can be determined according to the following steps: The first key action valid frame is used as the standard frame; the following steps are performed cyclically to determine the key action valid frame until the standard frame is the N-1th frame, where N is the number of frames of the motion video: Each frame image following the standard frame in the motion video is traversed until an image whose similarity to the standard frame is less than or equal to a preset similarity is determined, and the determined image is used as a key action valid frame and a new standard frame.

[0027] Starting from the i-th frame image, the initial value of i is the first key action valid frame, traverse the subsequent frame images, and calculate the similarity with the i-th frame frame by frame until the j-th frame image whose similarity is less than or equal to the preset similarity stops. Then the j-th frame image is recorded as the key action valid frame, and the j-th frame is recorded as the new i-frame.

[0028] Specifically, MSE or SSIM may be used to determine the similarity between images.

[0029] In some embodiments, the following steps may be used to detect whether the last frame of a motion video is a key action valid frame: If the similarity between the Nth frame image and the N-1th frame image in the motion video is less than the preset similarity, the Nth frame image is used as a key action valid frame. The motion video has a total of N frames of images.

[0030] After obtaining all the key action valid frames, follow the steps below to compose the frame sequence: According to the time sequence of the key action valid frames in the motion image, the key action valid frames are combined to obtain a key action valid frame sequence.

[0031] contrast Figure 1A , Figure 1B and Figure 3BIt can be seen that the motion contour edges extracted by the method in the above embodiment of the present disclosure are highly matched with the images in the motion video, and due to the extraction of effective frame sequences of key actions, the efficiency of motion prompt annotation is effectively improved, while the labor cost is reduced.

[0032] Based on the same inventive concept, the present disclosure provides a motion image annotation device based on edge detection, and the steps performed by the components of the device are the same or similar to those of the above method, so similar parts are not repeated. Figure 4 As shown, the motion image annotation device based on edge detection of this embodiment includes: The video acquisition module 410 is used to acquire motion video.

[0033] The valid frame determination module 420 is used to extract a key action valid frame sequence in the motion video.

[0034] The contour extraction module 430 is used to extract the action contour edge of each key action valid frame in the key action valid frame sequence.

[0035] The annotation module 440 is used to generate motion annotation hints for each key action valid frame based on the motion contour edge of each key action valid frame, and add the motion annotation hints to the corresponding key action valid frame in the key action valid frame sequence to obtain a target motion image sequence with action annotations.

[0036] In some embodiments, when determining the first key action valid frame in the key action valid frame sequence, the valid frame determination module 420 is specifically configured to: Starting from the first frame image of the motion video, determine whether the image is a key action valid frame according to the following steps until the first key action valid frame is determined: The frame image is divided into image blocks of a preset size, and the current image block is taken as the first image block in order from left to right and from top to bottom. If the first image block has a nearest left image block, the nearest left image block is taken as the second image block; if the nearest left image block does not exist, the nearest image block above the first image block is taken as the second image block; For the first image block and the second image block, calculate the difference of pixel values ​​pixel by pixel, and calculate the cumulative value of the absolute value of the difference; When the accumulated value is greater than or equal to the preset value, the frame image is used as the first key action valid frame.

[0037] In some embodiments, when determining at least some of the key action valid frames in the key action valid frame sequence except the first key action valid frame, the valid frame determining module 420 is specifically configured to: The first key action valid frame is taken as the standard frame; The following steps are executed repeatedly to determine the key action valid frame until the standard frame is the N-1th frame, where N is the number of frames of the motion video: Each frame image following the standard frame in the motion video is traversed until an image whose similarity to the standard frame is less than or equal to a preset similarity is determined, and the determined image is used as a key action valid frame and a new standard frame.

[0038] In some embodiments, the valid frame determination module 420 is further configured to: If the similarity between the Nth frame image and the N-1th frame image in the motion video is less than the preset similarity, the Nth frame image is used as a key action valid frame.

[0039] In some embodiments, when extracting the key action valid frame sequence in the motion video, the valid frame determination module 420 is further used to: According to the time sequence of the key action valid frames in the motion image, the key action valid frames are combined to obtain a key action valid frame sequence.

[0040] In some embodiments, when extracting the action contour edge of each key action valid frame in the key action valid frame sequence, the contour extraction module 430 is specifically used to: The Canny edge detection method is used to extract the edge of the action contour of each key action valid frame in the key action valid frame sequence.

[0041] In some embodiments, the valid frame determination module 420 is further configured to: Use MSE or SSIM to determine the similarity between images.

[0042] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a computer-readable storage medium.

Claims

1. A motion image annotation method based on edge detection, characterized in that: include: Get sports videos; Extracting a key action effective frame sequence in the sports video; Extracting the action contour edge of each key action valid frame in the key action valid frame sequence; Based on the action contour edge of each key action valid frame, motion annotation hints are generated for each key action valid frame, and the motion annotation hints are added to the corresponding key action valid frames in the key action valid frame sequence to obtain a target motion image sequence with action annotations.

2. The method according to claim 1, characterized in that The first key action valid frame in the key action valid frame sequence is determined according to the following steps: Starting from the first frame image of the motion video, determine whether the image is a key action valid frame according to the following steps until the first key action valid frame is determined: The frame image is divided into image blocks of a preset size, and the current image block is taken as the first image block in order from left to right and from top to bottom. If the first image block has a nearest left image block, the nearest left image block is taken as the second image block; if the nearest left image block does not exist, the nearest image block above the first image block is taken as the second image block; For the first image block and the second image block, calculate the difference of pixel values ​​pixel by pixel, and calculate the cumulative value of the absolute value of the difference; When the accumulated value is greater than or equal to the preset value, the frame image is used as the first key action valid frame.

3. The method according to claim 2, characterized in that At least some of the key action valid frames in the key action valid frame sequence except the first key action valid frame are determined according to the following steps: The first key action valid frame is taken as the standard frame; The following steps are executed repeatedly to determine the key action valid frame until the standard frame is the N-1th frame, where N is the number of frames of the motion video: Each frame image following the standard frame in the motion video is traversed until an image whose similarity to the standard frame is less than or equal to a preset similarity is determined, and the determined image is used as a key action valid frame and a new standard frame.

4. The method according to claim 3, characterized in that Also includes: If the similarity between the Nth frame image and the N-1th frame image in the motion video is less than the preset similarity, the Nth frame image is used as a key action valid frame.

5. The method according to claim 4, characterized in that The step of extracting a key action valid frame sequence from the motion video further includes: According to the time sequence of the key action valid frames in the motion image, the key action valid frames are combined to obtain a key action valid frame sequence.

6. The method according to claim 1, characterized in that The step of extracting the action contour edge of each key action valid frame in the key action valid frame sequence comprises: The Canny edge detection method is used to extract the edge of the action contour of each key action valid frame in the key action valid frame sequence.

7. The method according to claim 3, characterized in that Also includes: Use MSE or SSIM to determine the similarity between images.

8. A motion image annotation device based on edge detection, characterized in that: include: A video acquisition module, used to acquire motion videos; An effective frame determination module, used for extracting a key action effective frame sequence in the motion video; A contour extraction module, used to extract the action contour edge of each key action valid frame in the key action valid frame sequence; The annotation module is used to generate motion annotation prompts for each key action valid frame based on the motion contour edge of each key action valid frame, and add the motion annotation prompts to the corresponding key action valid frame in the key action valid frame sequence to obtain a target motion image sequence with action annotations.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.