Action scoring method, terminal, and computer-readable storage medium
By using a detection segmentation network and a score regression network to extract features and score skiing actions, the problem of low accuracy in skiing action evaluation is solved, and more objective scoring results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG DAHUA TECH CO LTD
- Filing Date
- 2022-07-18
- Publication Date
- 2026-04-17
AI Technical Summary
Current technology has low accuracy in judging skiing maneuvers, and is greatly affected by the subjective factors of the judges.
A detection and segmentation network is used to extract features from the image to be detected, generating a mask image and local images. An action scoring network is then used to eliminate the influence of external factors.
This improves the accuracy of skiing action evaluation, reduces the influence of subjective factors, and ensures the fairness of the scoring.
Smart Images
Figure CN115439925B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image recognition technology, and in particular to an action scoring method, a terminal, and a computer-readable storage medium. Background Technology
[0002] Freestyle skiing, as a skill demonstration sport (such as slopestyle), has high entertainment value. Due to venue limitations, judges cannot closely observe the athletes' performance. Competition results are mainly determined by judges from different countries using high-speed cameras and their professional experience. This leads to a significant influence of subjective factors on the final results, resulting in inaccurate judgments. Summary of the Invention
[0003] The main technical problem solved by this invention is to provide a motion scoring method, terminal and computer-readable storage medium to solve the problem of low accuracy of skiing motion judgment results in the prior art.
[0004] To solve the above-mentioned technical problems, the first technical solution adopted by the present invention is: to provide an action scoring method for skiing, the action scoring method comprising: extracting features from an acquired image to be detected to obtain a feature map; the image to be detected contains skiing actions of a target object; processing the feature map of the image to be detected through a detection segmentation network to obtain a mask image of the image to be detected and a local image containing the target object; fusing the mask image and the local image of the target object to determine the action score of the skiing action.
[0005] The process of extracting features from the acquired image to be detected to obtain a feature map includes: extracting features from the image to be detected through a detection and segmentation network to obtain a feature map of the target object; wherein the detection and segmentation network includes a top-down module, a dilated convolution module, and a bottom-up module cascaded in sequence.
[0006] The detection and segmentation network further includes a binarization module and a target detection module, which are connected to the bottom-up module respectively. The network processes the feature map of the image to be detected to obtain a mask image and a local image containing the target object. This process includes: binarizing the feature map of the image to be detected using the binarization module to obtain the mask image; performing target detection on the feature map of the image to be detected using the target detection module to obtain target object detection information; and extracting the local image containing the target object from the image to be detected based on the target object detection information.
[0007] The detection information includes target detection boxes and their confidence scores. A detection and segmentation network processes the feature map of the image to be detected to obtain a mask image and a local image containing the target object. This includes: performing target detection on the feature map of the image to be detected to obtain the target detection box and its confidence score; extracting the target detection box from the image to be detected to obtain the local image containing the target object; and comparing the grayscale values of each pixel in the feature map of the image to be detected with a segmentation threshold to determine the mask image of the image to be detected. The segmentation threshold of the image to be detected is positively correlated with the confidence score.
[0008] The training method for the detection and segmentation network includes the following steps: obtaining a first training sample set, which includes multiple first sample images containing targets, and the first sample images are associated with labeled mask images and target ground truth bounding boxes containing the targets; processing the first sample images through the detection and segmentation network to obtain predicted mask images of the first sample images and target predicted bounding boxes of the targets; iteratively training the detection and segmentation network based on the sum of a first error value between the target ground truth bounding box and the target predicted bounding box corresponding to the same first sample image and a second error value between the labeled mask image and the predicted mask image of the same first sample image.
[0009] The process of fusing the mask image and local images of the target object to determine the action score of the skiing action includes: weighted fusing of the mask image and local images of the target object to obtain a mixed image of the target object, which consists only of the pixel regions of the target object; and recognizing the mixed image of the target object to determine the action score corresponding to the skiing action of the target object.
[0010] The process of fusing the mask image and local images of the target object to determine the action score for a skiing action includes: simultaneously inputting the mask image and local images of the target object into a score regression network to obtain the action score of the target object; the training method of the score regression network includes the following steps: obtaining a second training sample set, which includes multiple sub-image sets, each sub-image set including a mask sample image of the target and a local sample image containing the target; the sub-image sets are associated with the labeled scores of the target; simultaneously inputting the mask sample image and local sample image in the sub-image set into the score regression network to obtain the predicted score of the target corresponding to the sub-image set; and iteratively training the score regression network based on a third error value between the predicted score and the labeled score of the target corresponding to the same sub-image set.
[0011] The image to be detected is any video frame in the video stream. The action scoring method further includes: in response to the fact that all video frames in the video stream containing the skiing action of the target object correspond to action scores, the preset number of video frames corresponding to the highest action scores and / or the video frames corresponding to the lowest action scores are output.
[0012] To solve the above-mentioned technical problems, the second technical solution adopted by the present invention is to provide a terminal, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. The processor is used to execute program data to implement the steps in the above-mentioned action scoring method.
[0013] To solve the above-mentioned technical problems, the third technical solution adopted by the present invention is to provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps in the above-mentioned action scoring method.
[0014] The beneficial effects of this invention are as follows: Unlike existing technologies, it provides an action scoring method, terminal, and computer-readable storage medium. The action scoring method includes: extracting features from an acquired image to be detected to obtain a feature map; the image to be detected contains a skiing action of a target object; processing the feature map of the image to be detected using a detection and segmentation network to obtain a mask image of the image to be detected and a local image containing the target object; fusing the mask image and the local image of the target object to determine the action score of the skiing action. This application determines the action score of the target object contained in the image to be detected by using the mask image and the local image containing the target object detected by the detection and segmentation network, eliminating the influence of external factors on the action score of the target object, and only scoring the technical actions of the target object, thus improving the accuracy of the evaluation results. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a flowchart illustrating the action scoring method provided by the present invention;
[0017] Figure 2 This is a flowchart illustrating an embodiment of the action scoring method provided by the present invention;
[0018] Figure 3 yes Figure 2 A schematic diagram illustrating the training of the detection segmentation network and the score regression network in the provided action scoring method;
[0019] Figure 4 yes Figure 2 A flowchart illustrating a specific embodiment of step S21 in the provided action scoring method;
[0020] Figure 5 yes Figure 2 A flowchart illustrating a specific embodiment of step S22 in the provided action scoring method;
[0021] Figure 6 This is a flowchart illustrating a specific embodiment of the action scoring method provided by the present invention;
[0022] Figure 7 This is a schematic block diagram of one embodiment of the motion scoring device provided by the present invention;
[0023] Figure 8 This is a schematic block diagram of one embodiment of the terminal provided by the present invention;
[0024] Figure 9 This is a schematic block diagram of an embodiment of the computer-readable storage medium provided by the present invention. Detailed Implementation
[0025] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0026] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.
[0027] In this article, the term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "more" in this article means two or more objects.
[0028] To enable those skilled in the art to better understand the technical solution of the present invention, the action scoring method provided by the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0029] Please see Figure 1 , Figure 1 This is a flowchart illustrating the motion scoring method provided by the present invention. This embodiment provides a motion scoring method applicable to scoring skiing movements during skiing, and also applicable to motion scoring in other types of sports, such as diving and gymnastics. The motion scoring method includes the following steps.
[0030] S11: Extract features from the acquired image to be detected to obtain a feature map.
[0031] Specifically, the image to be detected contains the skiing motion of the target object. The image to be detected can be an image acquired in real time by an image acquisition device, or it can be any video frame from an offline video stream.
[0032] In one embodiment, a detection and segmentation network is used to extract features from the image to be detected, thereby obtaining a feature map of the target object. The detection and segmentation network includes a top-down module, a dilated convolutional module, and a bottom-up module, all cascaded together.
[0033] S12: The feature map of the image to be detected is processed by the detection segmentation network to obtain the mask image of the image to be detected and the local image containing the target object.
[0034] Specifically, the detection and segmentation network also includes a binarization module and a target detection module, which are connected to the bottom-up module respectively.
[0035] In one embodiment, a binarization module performs binarization processing on the feature map of the image to be detected to obtain a mask image of the image to be detected; a target detection module performs target detection on the feature map of the image to be detected to obtain the detection information of the target object; and based on the detection information of the target object, a local image containing the target object is extracted from the image to be detected.
[0036] In one specific embodiment, the detection information includes target detection boxes and their confidence scores. The grayscale values of each pixel in the feature map of the image to be detected are compared with a segmentation threshold to determine the mask image of the image to be detected. Target detection is performed on the feature map of the image to be detected to obtain the target detection boxes and confidence scores of the target objects. The confidence scores of the image to be detected are positively correlated with the segmentation threshold. The target detection boxes are extracted from the image to be detected to obtain a local image containing the target objects.
[0037] S13: Fuse the mask image and local image of the target object to determine the action score of the skiing action.
[0038] Specifically, the mask image and local image of the target object are weighted and fused to obtain a mixed image of the target object, which consists only of the pixel area of the target object; the mixed image of the target object is then used to identify the action score corresponding to the skiing action of the target object.
[0039] In one embodiment, the mask image and local image of the target object are simultaneously input into a score regression network to obtain the action score of the target object.
[0040] In one embodiment, in response to the fact that all video frames in the video stream containing the target object correspond to the action score of the target object, a preset number of video frames corresponding to the highest action score and / or the lowest action score are selected.
[0041] The action scoring method provided in this embodiment includes: extracting features from the acquired image to be detected to obtain a feature map; the image to be detected contains skiing actions of a target object; processing the feature map of the image to be detected through a detection and segmentation network to obtain a mask image of the image to be detected and a local image containing the target object; fusing the mask image and the local image of the target object to determine the action score of the skiing action. This application determines the action score of the target object contained in the image to be detected by using the mask image and the local image containing the target object detected by the detection and segmentation network, eliminating the influence of external factors on the action score of the target object, and only scoring the technical actions of the target object, thereby improving the accuracy of the evaluation results.
[0042] Please see Figures 2 to 6 , Figure 2 This is a flowchart illustrating an embodiment of the action scoring method provided by the present invention; Figure 3 yes Figure 2 A schematic diagram illustrating the training of the detection segmentation network and the score regression network in the provided action scoring method; Figure 4 yes Figure 2 A flowchart illustrating a specific embodiment of step S21 in the provided action scoring method; Figure 5 yes Figure 2 A flowchart illustrating a specific embodiment of step S22 in the provided action scoring method; Figure 6 This is a flowchart illustrating a specific embodiment of the action scoring method provided by the present invention.
[0043] This embodiment provides a motion scoring method, which is applicable to scoring skiing movements during skiing, and also applicable to motion scoring in other types of sports, such as diving and gymnastics. The motion scoring method includes the following steps.
[0044] S21: Training the detection and segmentation network.
[0045] Specifically, please refer to Figure 3 , Figure 4 and Figure 6 The training method for the detection segmentation network includes the following steps.
[0046] S211: Obtain the first training sample set.
[0047] Specifically, the first training sample set includes multiple first sample images containing the target, and the first sample images are associated with labeled mask images and target ground truth bounding boxes containing the target.
[0048] In one embodiment, the first sample image is an image containing skiers in a professional skiing competition scene. The first sample image contains a target performing a skiing action. For example, a skiing action could be a jump, a flight, or a landing. The first sample image is associated with a corresponding labeled mask image and a ground truth bounding box containing the target. The ground truth bounding box containing the target is the target's human body bounding box.
[0049] S212: The first sample image is processed by a detection segmentation network to obtain the predicted mask image of the first sample image and the target prediction box of the target.
[0050] Specifically, the detection and segmentation network includes a cascaded top-down module, a dilated convolution module, a bottom-up module, a binarization module, and an object detection module. The top-down module, dilated convolution module, and bottom-up module are cascaded sequentially, and the binarization module and object detection module are connected to the bottom-up module. The detection and segmentation network includes a feature parymid network (FPN) and an atrous spatial pyramid pooling (ASPP) network structure. Specifically, the top-down module and the bottom-up module form the FPN, and the dilated convolution module is an ASPP network structure. The object detection module can be an anchor-free detection algorithm. In this embodiment, by setting a dilated convolution module in the FPN, the receptive field of the network can be increased without changing the size of the input image, enhancing the network's ability to obtain multi-scale context.
[0051] In one specific embodiment, a first feature map of the first sample image is obtained by performing feature extraction and downsampling on the first sample image through a top-down module; a second feature map of the first sample image is obtained by convolving the first feature map through a dilated convolution module; and a third feature map of the first sample image is obtained by performing feature extraction and upsampling on the second feature map through a bottom-up module.
[0052] In one embodiment, the top-down module includes a first feature extraction unit, a second feature extraction unit, and a third feature extraction unit cascaded sequentially, and each of the first, second, and third feature extraction units includes a first feature extraction layer and a downsampling layer cascaded sequentially. The feature parameters of the first, second, and third feature extraction units can be the same or different, depending on the specific circumstances.
[0053] The first feature extraction unit performs feature extraction and downsampling on the first sample image to obtain a first sub-feature map of the first sample image. The second feature extraction unit performs feature extraction and downsampling on the first sub-feature map to obtain a second sub-feature map of the first sample image. The third feature extraction unit performs feature extraction and downsampling on the second sub-feature map to obtain a first feature map of the first sample image.
[0054] The dilated convolution module can be connected to at least one of the first feature extraction unit, the second feature extraction unit, and the third feature extraction unit. In one embodiment, the dilated convolution module performs dilated convolution on the first feature map to obtain a second feature map of the first sample image.
[0055] In one embodiment, the bottom-up module includes a fourth feature extraction unit, a fifth feature extraction unit, and a sixth feature extraction unit cascaded sequentially, and each of the fourth, fifth, and sixth feature extraction units includes a second feature extraction layer and an upsampling layer cascaded sequentially. The feature parameters of the fourth, fifth, and sixth feature extraction units can be the same or different, depending on the specific circumstances.
[0056] The fourth feature extraction unit performs feature extraction and upsampling on the second sample image to obtain the third sub-feature map of the first sample image. Based on the third sub-feature map and the first feature map, it performs lateral connection and feature fusion to obtain the fourth sub-feature map. The fifth feature extraction unit performs feature extraction and upsampling on the fourth sub-feature map to obtain the fifth sub-feature map of the first sample image. Based on the fifth sub-feature map and the second sub-feature map, it performs lateral connection and feature fusion to obtain the sixth sub-feature map. The sixth feature extraction unit performs feature extraction and upsampling on the sixth sub-feature map to obtain the seventh sub-feature map of the first sample image. Based on the seventh sub-feature map and the first sub-feature map, it performs lateral connection and feature fusion to obtain the third feature map.
[0057] In this embodiment, the third feature map has the same size as the first sub-feature map, the sixth sub-feature map has the same size as the second sub-feature map, and the fourth sub-feature map has the same size as the first feature map.
[0058] In one embodiment, the target detection module upsamples the third feature map before performing target detection to obtain the target prediction box of the target in the first sample image. Specifically, a keypoint-based detection algorithm detects the keypoints at the top left and top right corners of the target to determine the target prediction box. Alternatively, the target prediction box can be determined by detecting the center point of the target. Alternatively, the target prediction box containing the target in the first sample image can be obtained by detecting keypoints and performing width and height regression prediction.
[0059] In one embodiment, a binarization module upsamples and binarizes the third feature map based on the grayscale values of each pixel in the third feature map to obtain a predicted mask image corresponding to the first sample image. Specifically, the grayscale values of each pixel in the third feature map are compared with a segmentation threshold. If the grayscale value of a pixel exceeds the segmentation threshold, the pixel is assigned a value of 1; if the grayscale value of a pixel does not exceed the segmentation threshold, the pixel is assigned a value of 0, thereby forming the predicted mask image corresponding to the first sample image. The size of the predicted mask image is the same as the size of the corresponding first sample image.
[0060] S213: Iteratively train the detection and segmentation network by summing the first error value between the ground truth bounding box and the predicted bounding box of the same first sample image and the second error value between the labeled mask image and the predicted mask image of the same first sample image.
[0061] Specifically, a first error value for the first sample image is calculated using the ground truth bounding box and the predicted bounding box of the target corresponding to the same first sample image; a second error value is calculated using the labeled mask image and the predicted mask image of the same first sample image. An optimal error value is determined based on the weighted sum of the first and second error values for the same first sample image. The detection and segmentation network is then iteratively trained based on the optimal error value.
[0062] In one specific embodiment, the optimal loss value is obtained by summing a first error value between the ground truth bounding box and the predicted bounding box of the same first sample image and a second error value between the labeled mask image and the predicted mask image of the same first sample image, based on a loss function. The detection and segmentation network is then iteratively trained based on this optimal loss value. The loss function can be cross-entropy.
[0063] In one optional embodiment, the results of the detection and segmentation network are backpropagated. The optimal loss value is obtained by summing the first error value between the ground truth bounding box and the predicted bounding box of the same first sample image and the second error value between the labeled mask image and the predicted mask image of the same first sample image. The weights of the detection and segmentation network are then corrected using the optimal loss value to train the detection and segmentation network.
[0064] The first sample image is input into the detection and segmentation network, which processes it. Training of the detection and segmentation network stops when the optimal loss value, obtained by summing the first error value between the ground truth bounding box and the predicted bounding box of the same first sample image and the second error value between the labeled mask image and the predicted mask image of the same first sample image, no longer decreases with increasing training iterations.
[0065] S22: Training score regression network.
[0066] Specifically, please refer to Figure 3 and Figure 5 The training method for the score regression network includes the following steps.
[0067] S221: Obtain the second training sample set.
[0068] Specifically, the second training sample set includes multiple sub-image sets, each of which includes a mask sample image of the target and a local sample image containing the target; each sub-image set is associated with a target's labeled score. The labeled score is between 0 and 1.
[0069] In one embodiment, a first training sample set is obtained, whereby the first sample images are associated with labeled scores for actions involving the target. The first sample images are input into the detection and segmentation network trained in step S21. The detection and segmentation network processes the first sample images to obtain target detection boxes and a mask image corresponding to the first sample images. Images within the target detection boxes in the first sample images are extracted to obtain local sample images containing the target. The mask image corresponding to the first sample image is used as the mask sample image. The local sample image containing the target and the mask sample image corresponding to the first sample image form a subset of images in the second training sample set, and the labeled scores for actions involving the target in the first sample images are the labeled scores for this subset of images.
[0070] S222: Input the mask sample image and local sample image of the sub-image set into the score regression network at the same time to obtain the predicted score of the target corresponding to the sub-image set.
[0071] Specifically, the mask sample map and local sample map of the sub-image set are simultaneously input into the fractional regression network. The fractional regression network first fuses the information of the mask sample map and the local sample map to obtain a mixed image, which is a pixel region containing only the target.
[0072] In one embodiment, the mask sample image is an image that distinguishes the target region from the background region; the local sample image displays the feature information of the target; based on the target region in the mask sample image, pixel regions containing only the target are extracted in the local sample image. Then, based on the information from the pixel regions containing only the target, a predicted score for the target corresponding to the sub-image set is predicted.
[0073] S223: Iteratively train a score regression network based on the third error value between the predicted score and the labeled score of the target corresponding to the same sub-image set.
[0074] Specifically, the third error value of the sub-image set is calculated by comparing the predicted score and the labeled score of the target corresponding to the same sub-image set, and the score regression network is iteratively trained based on the third error value.
[0075] In one specific embodiment, a third error value is calculated based on the predicted score and the labeled score of the target corresponding to the same sub-image set, using a loss function. Then, a score regression network is trained based on this third error value. The loss function can be cross-entropy.
[0076] In one optional embodiment, the results of the fractional regression network are backpropagated, and a third error value is calculated based on the predicted score and the labeled score of the target corresponding to the same sub-image set. The weights of the fractional regression network are corrected by the third error value to achieve the training of the fractional regression network.
[0077] A subset of images is input into a fractional regression network, which processes local sample images and mask sample images within the subset. Training of the fractional regression network is stopped when the third error between the predicted score and the labeled score of the target corresponding to the same subset of images no longer decreases with increasing training iterations.
[0078] S23: Acquire the image to be detected.
[0079] Specifically, please refer to Figure 6 The system uses drones to follow a pre-set route along the race track, employing onboard cameras to continuously track and film skiers in real time. This means the target can be skiers or any person engaged in other sports. By acquiring stable, continuous, and real-time video or images via drones, the limitations of fixed cameras are avoided.
[0080] In another embodiment, an offline video stream is acquired, and video frames containing the target object in the offline video stream are used as images to be detected. These images are then sequentially input into a detection and segmentation network for feature extraction to obtain feature maps of the target object.
[0081] S24: The detection and segmentation network is used to extract features from the image to be detected, and the feature map of the target object is obtained.
[0082] Specifically, the current video frame is used as the image to be detected, and the detection and segmentation network trained in step S21 above is used to extract features from the image to be detected to obtain the feature map of the target object.
[0083] In this embodiment, the top-down module, dilated convolution module, and bottom-up module in the detection segmentation network sequentially extract features from the image to be detected, resulting in a feature map of the image to be detected. The size of the feature map is smaller than the size of the image to be detected.
[0084] In one embodiment, after acquiring the image to be detected, it can be preprocessed to obtain a preprocessed image. The preprocessed image is then input into a detection and segmentation network to extract features from it, resulting in a feature map of the image to be detected. The preprocessing includes resizing, color resizing, and normalization. For example, the acquired video frames can be scaled to adjust all frames to a preset size.
[0085] S25: Based on the feature map of the image to be detected, the detection segmentation network determines the mask image and the local image containing the target object of the image to be detected.
[0086] Specifically, the grayscale values of each pixel in the feature map of the image to be detected are compared with the segmentation threshold to determine the mask image of the image to be detected. Target detection is then performed on the feature map of the image to be detected to obtain the target detection box and confidence score; the confidence score of the image to be detected is positively correlated with the segmentation threshold.
[0087] In this embodiment, the target detection module performs target detection on the feature map of the image to be detected, obtaining the target detection box and the confidence score of the target object. The binarization module performs binarization processing on the feature map of the image to be detected, obtaining the mask image of the image to be detected.
[0088] In one embodiment, during skiing, the human body rotates in the air, resulting in situations where the body is facing away from or towards the camera. When the body is facing away from the camera, if a fixed segmentation threshold is used to determine the mask image, the grayscale values of some pixels corresponding to the body edges will be low, leading to false filtering. Furthermore, when the body is facing away from the camera, the confidence level of the target detection box is also low. Therefore, the segmentation threshold during the binarization process can be dynamically adjusted based on the confidence level of the detected target detection boxes.
[0089] In one specific embodiment, the segmentation threshold of the image to be detected during the binarization process can be determined based on the following formula 1.
[0090] Thr seg =sigmoid[α·(Conf det -β)] (Formula 1)
[0091] In Formula 1, Thr seg Indicates the segmentation threshold; Conf det This represents the confidence level of the target detection box; α and β are preset parameters, α = 5, β = 0.5.
[0092] In one embodiment, a local image containing the target object is extracted from the image to be detected based on the detection information of the target object. Specifically, a target detection bounding box is extracted from the image to be detected to obtain a local image containing the target object.
[0093] S26: Input the mask image and local image of the target object into the score regression network simultaneously to obtain the action score of the target object.
[0094] Specifically, the target object's mask image and local image are fused using the fractional regression network in step S22 above to obtain a mixed image of the target object, which consists only of the pixel regions of the target object. The mixed image of the target object is then used for recognition to determine the target object's action score.
[0095] S27: Determine whether all video frames in the video stream containing the target object correspond to the action score of the target object.
[0096] If all video frames in the video stream that contain the target object correspond to the target object's action score, then proceed directly to step S28; if only some video frames in the video stream that contain the target object correspond to the target object's action score, then proceed directly to step S23.
[0097] S28: Output a preset number of video frames corresponding to the highest and / or lowest motion scores.
[0098] Specifically, in response to the condition that all video frames containing the target object in the video stream correspond to the target object's action score, a preset number of video frames corresponding to the highest and / or lowest action scores are selected and output, along with their corresponding action scores. This allows judges to score the athlete's skiing actions, improving the reliability of the scoring and avoiding the influence of the judges' subjective factors on the scoring results.
[0099] The action scoring method provided in this embodiment includes: extracting features from the acquired image to be detected to obtain a feature map; the image to be detected contains skiing actions of a target object; processing the feature map of the image to be detected through a detection and segmentation network to obtain a mask image of the image to be detected and a local image containing the target object; fusing the mask image and the local image of the target object to determine the action score of the skiing action. This application determines the action score of the target object contained in the image to be detected by using the mask image and the local image containing the target object detected by the detection and segmentation network, eliminating the influence of external factors on the action score of the target object, and only scoring the technical actions of the target object, thus improving the accuracy of the evaluation results.
[0100] See Figure 7 , Figure 7This is a schematic block diagram of one embodiment of the motion scoring device provided by the present invention. This embodiment provides a motion scoring device 60, which is suitable for scoring motions during skiing, and is also suitable for scoring motions in other types of sports, such as diving and gymnastics. The motion scoring device 60 includes a feature extraction module 61, an image processing module 62, and a motion scoring module 63.
[0101] The feature extraction module 61 is used to extract features from the acquired image to be detected to obtain a feature map; the image to be detected contains the skiing action of the target object.
[0102] The image processing module 62 is used to process the feature map of the image to be detected through the detection segmentation network to obtain the mask image of the image to be detected and the local image containing the target object.
[0103] The motion scoring module 63 is used to fuse the mask image and local image of the target object to determine the motion score of the skiing motion.
[0104] The target object action scoring device provided in this embodiment determines the target object action score contained in the image to be detected by detecting the mask image output by the detection segmentation network and the local image containing the target object. This eliminates the influence of external factors on the target object action score and only scores the technical actions of the target object, thereby improving the accuracy of the evaluation results.
[0105] Please see Figure 8 , Figure 8 This is a schematic block diagram of one embodiment of the terminal provided by the present invention. The terminal 80 includes a memory 81 and a processor 82 coupled to each other. The processor 82 is used to execute program instructions stored in the memory 81 to implement the steps of any of the above-described action scoring method embodiments. In a specific implementation scenario, the terminal 80 may include, but is not limited to, a microcomputer, a server, etc. In addition, the terminal 80 may also include mobile devices such as laptops and tablets, which are not limited here.
[0106] Specifically, processor 82 controls itself and memory 81 to implement the steps of any of the above-described action scoring method embodiments. Processor 82 can also be referred to as a CPU (Central Processing Unit). Processor 82 may be an integrated circuit chip with signal processing capabilities. Processor 82 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 82 can be implemented using integrated circuit chips.
[0107] Please see Figure 9 , Figure 9 This is a schematic block diagram of one embodiment of the computer-readable storage medium provided by the present invention. The computer-readable storage medium 90 stores program instructions 901 that can be executed by a processor. The program instructions 901 are used to implement the steps of any of the above-described action scoring method embodiments.
[0108] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0109] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.
[0110] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.
[0111] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0112] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0113] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.
[0114] The above are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A method of action scoring for a ski sport, characterized in that, include: The detection and segmentation network extracts features from the acquired image to be detected to obtain a feature map of the target object; the image to be detected contains the skiing action of the target object; the detection and segmentation network includes a top-down module, a dilated convolution module, and a bottom-up module cascaded in sequence; the top-down module includes a first feature extraction unit, a second feature extraction unit, and a third feature extraction unit cascaded in sequence, and each of the first, second, and third feature extraction units includes a first feature extraction layer and a downsampling layer cascaded in sequence; the dilated convolution module is connected to at least one of the first, second, and third feature extraction units; the bottom-up module includes a fourth, fifth, and sixth feature extraction unit cascaded in sequence, and each of the fourth, fifth, and sixth feature extraction units includes a second feature extraction layer and an upsampling layer cascaded in sequence; The feature map of the image to be detected is processed by the detection and segmentation network to obtain a mask image of the image to be detected and a local image containing the target object. The mask image of the target object and the local image are fused to determine the action score of the skiing action; The step of fusing the mask image of the target object and the local image to determine the action score of the skiing action includes: Based on the target region of the mask image, a pixel region containing only the target object is extracted from the local image; The action score of the skiing action of the target object is determined based on information from the pixel region of the target object contained only in the local image.
2. The action scoring method according to claim 1, characterized in that, The detection and segmentation network further includes a binarization module and a target detection module, which are respectively connected to the bottom-up module. The step of processing the feature map of the image to be detected through a detection and segmentation network to obtain a mask image of the image to be detected and a local image containing the target object includes: The feature map of the image to be detected is binarized by the binarization module to obtain the mask image of the image to be detected. The target detection module performs target detection on the feature map of the image to be detected to obtain the detection information of the target object; Based on the detection information of the target object, a local image containing the target object is extracted from the image to be detected.
3. The action scoring method according to claim 2, characterized in that, The detection information includes the target detection box and the confidence level of the target detection box; The step of processing the feature map of the image to be detected through a detection and segmentation network to obtain a mask image of the image to be detected and a local image containing the target object includes: Target detection is performed on the feature map of the image to be detected to obtain the target detection box of the target object and the confidence score. Extract the target detection box from the image to be detected to obtain a local image containing the target object; The grayscale value of each pixel in the feature map of the image to be detected is compared with the segmentation threshold to determine the mask image of the image to be detected; wherein the segmentation threshold of the image to be detected is positively correlated with the confidence level.
4. The action scoring method according to claim 1, characterized in that, The training method for the detection and segmentation network includes the following steps: Obtain a first training sample set, which includes multiple first sample images containing the target. The first sample images are associated with an labeled mask image and a target ground truth bounding box containing the target. The first sample image is processed by the detection and segmentation network to obtain the predicted mask image of the first sample image and the target prediction box of the target. The detection and segmentation network is iteratively trained based on the sum of a first error value between the ground truth bounding box and the predicted bounding box of the same first sample image and a second error value between the labeled mask image and the predicted mask image of the same first sample image.
5. The action scoring method according to claim 1, characterized in that, The step of fusing the mask image of the target object and the local image to determine the action score of the skiing action includes: The mask image of the target object and the local image are weighted and fused to obtain a mixed image of the target object, wherein the mixed image of the target object consists only of the pixel region of the target object; The composite image of the target object is identified to determine the action score corresponding to the skiing action of the target object.
6. The action scoring method according to claim 1, characterized in that, The step of fusing the mask image of the target object and the local image to determine the action score of the skiing action includes: The mask image and the local image of the target object are simultaneously input into a score regression network to obtain the action score of the target object. The training method for the score regression network includes the following steps: A second training sample set is obtained, which includes multiple sub-image sets. Each sub-image set includes a mask sample image of the target and a local sample image containing the target. The sub-image set is associated with the target's annotation score. The mask sample image and the local sample image in the sub-image set are simultaneously input into the score regression network to obtain the predicted score of the target corresponding to the sub-image set; The score regression network is iteratively trained based on a third error value between the predicted score and the labeled score of the target corresponding to the same sub-image set.
7. The action scoring method according to claim 1, characterized in that, The image to be detected is any video frame in the video stream, and the action scoring method further includes: In response to the fact that all video frames in the video stream containing the skiing action of the target object correspond to the action score, a preset number of video frames corresponding to the highest action score and / or the lowest action score are output.
8. A terminal, characterized in that, The terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor, the processor being used to execute program data to implement the steps in the action scoring method as described in any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the action scoring method as described in any one of claims 1 to 7.