User motion video image enhancement method and system based on machine vision

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a user motion video image enhancement method and system based on machine vision, and the method comprises the steps: obtaining a stable coordinate of a motion video; dividing the motion video into a plurality of video segments according to the change rule of the definition of the video frame; obtaining node coordinates of each video frame and predicted coordinates of virtual nodes based on an attitude estimation algorithm and a standard skeleton model; establishing a loss function of the video segments based on the change of the predicted coordinates of the virtual nodes of the continuous video frames, the conformity of the nodes and the virtual nodes with motion inertia and the conformity of the nodes and the virtual nodes with a human body structure; acquiring actual coordinates of the virtual nodes; and carrying out image enhancement on each video segment by using a posture migration network. According to the invention, the target performance and accuracy of motion video image enhancement are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more particularly to a method and system for enhancing user motion video images based on machine vision. Background Art

[0002] With the development of artificial intelligence, more and more fields are beginning to use AI to expand its application scope and optimize user experience. For example, for users to receive professional training offline at the gym, it costs a lot of time and money. However, using AI to establish an AI sports coach can not only reduce user costs, but also customize different training plans for different users, allowing users to train in any venue, greatly improving the user's exercise effect and ensuring the user's exercise experience. The AI ​​sports coach generates a digital human and the user's exercise plan based on the user's physical data. During the user's exercise, the camera captures the user's movements and compares the user's movements with those in a standard movement library to provide movement guidance. During this process, the motion capture based on the user's movement video requires image enhancement to avoid poor motion capture quality due to problems such as blur, which can lead to incorrect movement guidance.

[0003] Currently, techniques such as optical flow and pose priors are commonly used to estimate motion trajectories, clarify video enhancement targets, and ensure effective video enhancement. For example, Chinese patent application CN119445043B discloses a method for fusing image-based dynamic video with a 3D model. This method introduces the NeRF 3D geometry-aware generation network to ensure that the shape and position of objects consistently match the actual geometry of the scene, enhancing geometric capture capabilities and improving the accuracy of video generation.

[0004] However, unlike ideal geometric structure matching, loose clothing worn by users during exercise may obscure their own geometric structure, resulting in poor matching and incorrect target recognition for video enhancement. In addition, since the user may move at a relatively fast speed, the video may be blurred, resulting in information loss, incorrect target recognition for video enhancement, and poor video enhancement effects. Summary of the Invention

[0005] In order to solve the above-mentioned technical problem of poor motion video image enhancement effect, the present invention provides solutions in the following aspects.

[0006] In a first aspect, the present invention provides a method for enhancing a user motion video image based on machine vision, comprising: Establish a standard skeleton model, which includes several nodes, several edges, and the distance between connected nodes and the angle range of adjacent edges; obtain a motion video; obtain the stable coordinates of the motion video based on the grayscale change degree of pixel points with the same coordinates in all video frames; obtain the clarity of the video frame according to the grayscale value of the stable coordinates of the video frame, the number of grayscale extreme points of the video frame and the grayscale change degree on the grayscale extreme point connection line; divide the motion video into several video segments according to the change law of the clarity of the video frame; perform semantic segmentation on each video frame to obtain several feature areas of each video frame; obtain the node coordinates of each video frame and the predicted coordinates of the virtual nodes based on the posture estimation algorithm and the standard skeleton model; establish a loss function for the video segmentation based on the change of the predicted coordinates of the virtual nodes in consecutive video frames, the degree to which the nodes and virtual nodes conform to the motion inertia and the human body structure; record the predicted coordinates of each virtual node corresponding to the minimum value of the loss function as the actual coordinates of the virtual node; use the posture transfer network to perform image enhancement on each video segment based on all the node coordinates of all video frames and the actual coordinates of the virtual nodes.

[0007] This invention establishes a personalized skeletal model, enabling personalized posture estimation and motion analysis, thereby improving the accuracy of analysis of user motion videos. By using virtual nodes to simulate the skeletal nodes beneath a user's clothing, this invention enables smart fitness applications to maintain high accuracy and stability even in the user's most realistic and casual home environment. This can improve the targeting and accuracy of motion video image enhancement.

[0008] Preferably, the step of obtaining stable coordinates of the motion video includes: Record the pixel sequence of any coordinates as the target pixel sequence; obtain the grayscale value type of the target pixel sequence and the number of pixels of each grayscale value; Calculate the pixel stability of any coordinates based on the grayscale value range and grayscale dispersion of the pixel sequence; The video frame is divided into several rectangular areas, and the coordinates with the highest pixel stability in each rectangular area are obtained and recorded as stable coordinates.

[0009] Preferably, the pixel stability of any coordinate satisfies the expression: ; Where, Indicates the pixel stability of the corresponding coordinates of the target pixel sequence; Represents the grayscale value type of the target pixel sequence; The set of pixel numbers representing all grayscale values ​​of the target pixel sequence; Represents the number of pixels with the i-th grayscale value in the target pixel sequence; represents the maximum value function; represents the natural exponential function.

[0010] Preferably, obtaining the clarity of the video frame includes: Any video frame is recorded as the target video frame, and the clarity of the target video frame satisfies the expression: ; Where, Indicates the clarity of the target video frame; represents the number of stable coordinates; Indicates the number of pixels in the corresponding pixel sequence of the grayscale value of the cth stable coordinate of the target video frame; Indicates the number of video frames of motion video; Indicates the number of grayscale extreme point connections of the target video frame; Represents the number of pixels connected by the h-th grayscale extreme point of the target video frame; 、 Represents the grayscale value of the a+1th and ath pixel points on the line connecting the hth grayscale extreme value point of the target video frame; represents the absolute value function; Represents the normalization function.

[0011] The present invention adopts a composite definition index, which can intelligently identify the intervals of user movement and use this as an anchor point to divide the video stream into multiple logically complete movement units.

[0012] Preferably, dividing the motion video into a plurality of video segments comprises: If the clarity of the k+1th video frame is greater than or equal to the clarity of the kth video frame, the k+1th video frame is encoded as 1. If the clarity of the k+1th video frame is less than the clarity of the kth video frame, the k+1th video frame is encoded as -1 to obtain a clarity difference sequence of the motion video; if the kth video frame is encoded as 1, the k-1th video frame is encoded as 1, and the k+1th video frame is encoded as -1, the kth video frame is recorded as a feature video frame; all video frames are judged in turn to obtain several feature video frames; Any two adjacent feature video frames and all video frames between the two adjacent feature video frames are recorded as a video segment, thereby obtaining a plurality of video segments of the motion video.

[0013] The present invention segments motion videos, providing action units with complete start and end points for subsequent analysis, rather than isolated video frames. This allows the most computationally intensive global optimization steps to be accurately and specifically applied to these key motion units, thereby improving overall processing efficiency.

[0014] Preferably, the obtaining of several characteristic regions of each video frame includes: Input the target video frame into a first-class semantic segmentation model to obtain a binary mask of the target video frame, in which the human body area is 1 and the background area is 0; multiply the target video frame by the binary mask to obtain the human body area of ​​the target video frame, and input the human body area into a second-class semantic segmentation model to obtain a multi-class segmentation map, which contains several parts represented by several numerical values; The pixels contained in each part of the target video frame are recorded as a feature region, the feature regions corresponding to the parts belonging to the human body are recorded as first-class feature regions, and the feature regions corresponding to the parts not belonging to the human body are recorded as second-class feature regions.

[0015] Preferably, obtaining the node coordinates of each video frame and the predicted coordinates of the virtual nodes includes: A posture estimation algorithm is used to obtain the node coordinates of the first-category feature area of ​​the target video frame and the initial predicted coordinates of the virtual nodes of the second-category feature area. The initial predicted coordinates of each virtual node are moved within the preset conditions. The movable range is the predicted coordinates corresponding to each virtual node; the preset conditions are that they are within the distance range of the connected nodes, within the angle range of the adjacent edges, and within the corresponding second-category feature area.

[0016] The present invention determines the predicted coordinates of virtual nodes based on a standard skeleton model, so that the positions of the virtual nodes conform to the human body structure and avoid the problem of large errors caused by image enhancement.

[0017] Preferably, the loss function for establishing video segmentation includes: Segment any video into The virtual nodes are recorded as ;right Extract a prediction coordinate from the prediction coordinate set of each video frame and construct it in chronological order A predicted coordinate sequence of ; calculate A partial loss function of any predicted coordinate sequence of the video segment; extracting a predicted coordinate sequence from each set of predicted coordinate sequences of all virtual nodes of the video segment, which is recorded as a predicted coordinate sequence combination of the video segment; multiplying the normalized value of the mean of the partial loss function corresponding to all virtual nodes of the y-th predicted coordinate sequence combination of the video segment by the normalized value of the mean of the normal index to obtain the loss function of the y-th predicted coordinate sequence combination of the video segment, where the normal index is determined based on the positional relationship between the virtual node and the neighboring node and the virtual node.

[0018] The present invention estimates the overall loss function by jointly optimizing all video frames of the entire motion unit. Its solution is the optimal solution on the entire spatiotemporal path, rather than a local greedy optimum. This mechanism completely avoids error accumulation and jitter, and can ensure enhanced image coherence.

[0019] Preferably, the calculation The partial loss function of any predicted coordinate sequence includes: obtaining the motion vectors of all pixel points of the time-preceding video frame in any two vector video frames of the target video segment based on the optical flow estimation algorithm; The partial loss function of any predicted coordinate sequence satisfies the expression: ; Where, express Partial loss function for the d-th predicted coordinate sequence; The number of video frames representing the target video segment; 、 express The d-th predicted coordinate sequence is in , The predicted coordinates of video frames; Indicated by and The vector formed; Indicates the preset number of neighborhood pixels; express The motion vector of the sth neighborhood pixel; represents the absolute value function; Represents the normalization function.

[0020] In a second aspect, the present invention provides a user motion video image enhancement system based on machine vision, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned user motion video image enhancement method based on machine vision is implemented.

[0021] By adopting the above technical solution, the above-mentioned user motion video image enhancement method based on machine vision is generated into a computer program and stored in the memory to be loaded and executed by the processor, so that a terminal device is made based on the memory and the processor for easy use.

[0022] The beneficial effects of the present invention are: (1) The present invention predicts the position of virtual nodes, so that even if the user wears loose clothes during exercise, the corresponding skeletal node position can be obtained according to the movement relationship and coordinate changes, thereby meeting the purpose of enhancing the user's movement video image and improving the accuracy of the user's movement posture assessment; (2) The present invention enhances the image based on the skeleton structure so that the video frame is enhanced in a direction that matches the user's motion state, thereby improving the effect and accuracy of the user motion video image enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart schematically illustrating a method for enhancing a user motion video image based on machine vision in the present invention; Figure 2 is a diagram schematically showing a standard skeleton model. DETAILED DESCRIPTION

[0024] The embodiment of the present invention discloses a method for enhancing user motion video images based on machine vision, referring to Figure 1 , including steps S1 to S4: S1: Establish a standard skeleton model, wherein the standard skeleton model includes a plurality of nodes, a plurality of edges, the distance between connected nodes, and the angle range between adjacent edges; obtain a motion video.

[0025] It should be noted that the AI ​​sports coach performs motion capture on the user's motion video during the user's exercise process, and compares the user's movements with the movements in the standard movement library to achieve movement guidance. The key to whether the movement is standard lies in the relative position and relationship of the bones and limbs, such as the angle of the thigh and calf when squatting. Taking each joint as a node and measuring the relative position and relationship between nodes can simplify movement judgment. However, due to factors such as loose clothing and blurred images during the user's exercise process, errors in node judgment may occur, resulting in errors in movement judgment, so it is necessary to perform image enhancement on the motion video. Since the focus of the motion video is on the characters and movements, it is considered to enhance only the character part. Therefore, the present invention first obtains the user's motion video and a standard skeletal model that provides a reference for the user's body structure.

[0026] Specifically, a standard skeleton model is established, which includes several nodes, several edges, and the distance range of connected nodes and the angle range of adjacent edges. It should be noted that the standard skeleton model can be obtained through platforms such as Sketchfab and medical schools. The distance of connected nodes and the angle range of adjacent edges are consistent with the bone length of the human body's medical structure and the angle range that bones can bend. Figure 2 A schematic diagram of a standard skeleton model.

[0027] The user's motion process video is obtained through a terminal used by the user to record the motion process, and grayscale is performed frame by frame to obtain a motion video.

[0028] At this point, the standard skeleton model and motion video have been obtained.

[0029] S2: Based on the grayscale change degree of pixel points with the same coordinates in all video frames, the stable coordinates of the motion video are obtained; according to the grayscale value of the stable coordinates of the video frame, the number of grayscale extreme points of the video frame and the grayscale change degree on the line connecting the grayscale extreme points, the clarity of the video frame is obtained; according to the change pattern of the clarity of the video frame, the motion video is divided into several video segments.

[0030] It should be noted that the user's motion process can be considered a combination of several motion units. Taking jumping jacks as an example, the user moves their hands from the sides of their legs to their head while simultaneously opening and closing their legs, then returns their hands to the sides of their legs and closes their legs, forming a "stationary-moving-stationary" motion unit. Within this motion unit, the clarity of the video frames of the motion video changes from "clear-blurred-clear". Video frames with higher clarity can better reflect the user's movements. Blurred video frames can be enhanced based on the high-definition video frames according to motion inertia, thereby obtaining an enhanced blurred image that accurately reflects the user's movements, thereby achieving the purpose of enhancing the user's motion video image. Therefore, the present invention decomposes the motion video to obtain multiple video segments, each of which represents a motion unit.

[0031] It should be further explained that the difference in the clarity of video frames lies in the presentation of image details. Image details are mainly expressed through the grayscale changes of pixels. Grayscale extremes are the most prominent features in video frames. Therefore, for any video frame of a motion video, the number of grayscale extremes and the degree of grayscale change on the line connecting the grayscale extreme points are analyzed to obtain the clarity of each video frame. At the same time, there are some stable background areas in the video frame. The changes in the clarity of these areas can reflect the relative clarity of the video frame. The grayscale of the pixels in the stable area always returns to a relatively stable grayscale. Therefore, the present invention combines the grayscale changes of the video frame and the grayscale changes of the neighborhood of the stable pixels to obtain the clarity of the video frame.

[0032] Specifically, based on the grayscale change of pixels with the same coordinates in all video frames, the stable coordinates of the motion video are obtained, including: A motion video is composed of several sequences of pixels with the same coordinates. These sequences are composed of the grayscale values ​​of pixels at the same location in different video frames at different times, arranged in chronological order. The number of pixel sequences depends on the video display format of the motion video. For example, in a 1080P video display format, where the video frame resolution is 1920×1080 pixels, the number of pixel sequences is 1920×1080.

[0033] Record the pixel sequence of any coordinates as the target pixel sequence. Obtain the grayscale value type of the target pixel sequence and the number of pixels of each grayscale value.

[0034] It should be noted that the more dispersed the grayscale distribution of the target pixel sequence and the more grayscale values ​​there are, the weaker the pixel stability of the corresponding coordinates of the target pixel sequence. Therefore, the pixel stability of any coordinate is calculated based on the grayscale value range and grayscale dispersion of the pixel sequence.

[0035] The pixel stability of the target pixel sequence corresponding to the coordinates satisfies the expression: ; Where, Indicates the pixel stability of the corresponding coordinates of the target pixel sequence; Represents the grayscale value type of the target pixel sequence; The set of pixel numbers representing all grayscale values ​​of the target pixel sequence; Represents the number of pixels with the i-th grayscale value in the target pixel sequence; represents the maximum value function; represents the natural exponential function.

[0036] Where, Represents the maximum number of pixels of all grayscale value types in the target pixel sequence; Represents the difference between the number of pixels of the i-th grayscale value of the target pixel sequence and the maximum number of pixels. A larger value indicates a smaller number of pixels of the i-th grayscale value of the target pixel sequence. Represents the average difference between the number of pixels of all grayscale values ​​of the target pixel sequence and the maximum number of pixels. The larger the value is, the smaller the average number of pixels of all grayscale values ​​of the target pixel sequence is, and the more concentrated the grayscale distribution of the pixels of the corresponding coordinates of the target pixel sequence is. Therefore, the greater the pixel stability of the corresponding coordinates of the target pixel sequence is. At the same time, the fewer the types of grayscale values ​​of the target pixel sequence are, the higher the pixel stability of the corresponding coordinates of the target pixel sequence is.

[0037] Divide the video frame into several equal rectangular regions, and obtain the coordinates with the highest pixel stability in each rectangular region, recording them as stable coordinates. It should be noted that the number of equal rectangular regions is set by the implementer based on the actual implementation situation. For example, in the case of a 1920×1080 resolution, the greatest common factor of 1920 and 1080 is 20, so the rectangular regions can be set to 20×20 to ensure that the video frame can be equally divided.

[0038] So far, several stable coordinates of the motion video have been obtained.

[0039] The key point is that for any video frame, if pixels at stable coordinates are unstable (i.e., have fewer grayscale values), the video frame will be less clear. Furthermore, the lower the number of grayscale extremes and the degree of grayscale variation along the line connecting the extreme grayscale values, the more blurry the video frame is and the lower its clarity.

[0040] Preferably, obtaining the clarity of the video frame according to the grayscale value of the stable coordinate of the video frame, the number of grayscale extreme value points of the video frame, and the grayscale change degree on the line connecting the grayscale extreme value points includes: Record any video frame as the target video frame, obtain the grayscale extreme value points of the target video frame, obtain the grayscale values ​​of the pixel points on the line connecting any two grayscale extreme value points of the target video frame, and construct a grayscale sequence between the two grayscale extreme value points in order of position.

[0041] The clarity of the target video frame satisfies the expression: ; Where, Indicates the clarity of the target video frame; represents the number of stable coordinates; Indicates the number of pixels in the corresponding pixel sequence of the grayscale value of the cth stable coordinate of the target video frame; Indicates the number of video frames of motion video; Indicates the number of grayscale extreme point connections of the target video frame; Represents the number of pixels connected by the h-th grayscale extreme point of the target video frame; 、 Represents the grayscale value of the a+1th and ath pixel points on the line connecting the hth grayscale extreme value point of the target video frame; represents the absolute value function; Represents the normalization function.

[0042] Where, Indicates the ratio of the grayscale value of the cth stable coordinate of the target video frame to the number of pixels in the corresponding pixel sequence. The larger the value, the closer the cth stable coordinate of the target video frame is to being stable. Indicates the average proportion of the grayscale values ​​of all stable coordinates of the target video frame to the number of pixels in the corresponding pixel sequence. The larger the value, the more stable all the stable coordinates of the target video frame are, and therefore the clarity of the target video frame is higher; Represents the grayscale difference between the a+1th and ath pixels on the line connecting the hth grayscale extreme point of the target video frame, Represents the average grayscale difference of the pixels along the line connecting the h-th grayscale extreme value point of the target video frame. The larger the value, the greater the grayscale variation and the richer the grayscale values, and the higher the definition of the area connected by the h-th grayscale extreme value point of the target video frame. It represents the sum of the clarity of the area connected by all grayscale extreme value points of the target video frame. The larger the value is, the clearer the area connected by all grayscale extreme value points of the target video frame is, and the higher the clarity of the target video frame is.

[0043] At this point, the clarity of each video frame is obtained.

[0044] It should be noted that the clarity sequence of the motion video is obtained in chronological order. The clarity change of this sequence is correlated with the user's motion state. When the user is still or in between movements, the video frames are often the clearest. When the user is exercising vigorously or completing an action, the video frames are often the blurriest. Therefore, the motion video can be divided based on the clarity change of the video frames.

[0045] Preferably, the motion video is divided into several video segments according to the change pattern of the clarity of the video frames, including: A definition sequence of a motion video is obtained in chronological order, and the definition sequence is differentially encoded: if the definition of the k+1th video frame is greater than or equal to the definition of the kth video frame, the k+1th video frame is encoded as 1; if the definition of the k+1th video frame is less than the definition of the kth video frame, the k+1th video frame is encoded as -1, thereby obtaining a definition differential sequence of the motion video. It should be noted that in the definition differential sequence of the motion video, a continuous +1 portion indicates that the video frame is gradually becoming clearer, and a continuous -1 portion indicates that the video frame is gradually becoming blurred.

[0046] Obtaining characteristic video frames for a motion video: If the kth video frame is coded as 1, the k-1th video frame is coded as 1, and the k+1th video frame is coded as -1, then the kth video frame is recorded as the characteristic video frame. This is then evaluated sequentially for all video frames to obtain several characteristic video frames. It should be noted that the characteristic video frames are images at the relatively clearest moments in the motion video.

[0047] Any two adjacent feature video frames and all video frames between the two adjacent feature video frames are recorded as a video segment, thereby obtaining a plurality of video segments of the motion video.

[0048] At this point, several video segments of the sports video have been obtained.

[0049] S3: Perform semantic segmentation on each video frame to obtain several feature areas of each video frame; obtain the node coordinates of each video frame and the predicted coordinates of the virtual nodes based on the posture estimation algorithm and the standard skeleton model; establish a loss function for video segmentation based on the changes in the predicted coordinates of the virtual nodes in consecutive video frames, the degree to which the nodes and virtual nodes conform to the motion inertia and the human body structure; obtain the actual coordinates of the virtual nodes based on the loss function of video segmentation.

[0050] It should be noted that step S2 segments the motion video so that the motion units can be divided even during the user's motion, and the user's movements can be accurately analyzed in real time. In order to reduce the scope of analysis, considering the diverse environments in which the user is located, such as parks, living rooms, gyms, etc., the background is complex and has nothing to do with motion, so each video frame can be segmented and only the character part is enhanced. Currently, semantic segmentation models based on artificial intelligence are gradually enriched, which can not only segment characters and backgrounds, but also segment each part of the human body, such as hats, hair, shirts, left arms, etc. Therefore, the present invention performs semantic segmentation on all video frames of each video segment.

[0051] It should be further explained that in order to complete the motion evaluation, the user's body structure needs to be matched with the standard skeleton model to obtain the user's skeleton model. Among these segmented parts, some are human body parts and some are objects such as clothing that affect posture estimation. The human body parts can be matched with the standard skeleton model more accurately, but the human body parts covered by clothing need to be matched by predicting virtual nodes. The matching effect may not be good in one video frame, but in continuous video frames, the movement of the human body has rules and inertia. By comparing the possible positions of the virtual nodes of continuous video frames, the positions of the virtual nodes can be corrected. The nodes and virtual nodes together constitute the skeleton of the character area of ​​the video frame. On this basis, the video frame is enhanced to ensure the accuracy of the user's motion video image enhancement. Therefore, the present invention constructs a loss function for evaluating the position accuracy of the virtual nodes of the video segmentation, and when the loss function is minimum, the optimal positions of all virtual nodes are obtained.

[0052] Specifically, semantic segmentation is performed on each video frame to obtain several feature areas of each video frame, including: The target video frame is input into a semantic segmentation model to obtain a binary mask of the target video frame, where the human body area is 1 and the background area is 0. The semantic segmentation model is, for example, a DeepLabV3+ model trained using the segmentation dataset COCO.

[0053] The target video frame is multiplied by the binary mask to obtain the person region in the target video frame. This person region is then input into a two-class semantic segmentation model to generate a multi-class segmentation map containing several types of parts represented by various numerical values. The two-class semantic segmentation model can be trained using the CE2P model trained with a human body parsing dataset. The numerical values ​​and parts can be, for example, 1 for the head, 2 for the top, and 3 for the left arm.

[0054] The pixels contained in each part of the target video frame are recorded as a feature region, the feature regions corresponding to the parts belonging to the human body are recorded as first-class feature regions, and the feature regions corresponding to the parts not belonging to the human body are recorded as second-class feature regions.

[0055] So far, several feature regions of each video frame have been obtained.

[0056] It should be noted that a class of feature areas can better match the nodes in the standard skeleton model, so the present invention obtains nodes from a class of feature areas, and then predicts virtual nodes along the class of feature areas to the neighborhood class two feature areas. In order to evaluate the accuracy of prediction, the present invention sets a loss function, which is used to evaluate the degree to which virtual nodes conform to human body structure. Considering that motion has continuity, the positions of the same virtual nodes of continuous video frames should also have continuity, and optical flow can well reflect the continuity of motion. Therefore, the present invention combines the situation that the nodes and virtual nodes of a single frame conform to the standard skeleton model, and the situation that the position changes of the nodes and virtual nodes of continuous frames conform to the standard skeleton model, establishes a loss function, and when the loss function is minimum, the positions of all virtual nodes are the most accurate.

[0057] Preferably, based on the posture estimation algorithm and the standard skeleton model, obtaining the node coordinates of each video frame and the predicted coordinates of the virtual nodes includes: Using a pose estimation algorithm, the node coordinates of the first-category feature region of the target video frame and the initial predicted coordinates of the virtual nodes in the second-category feature region are obtained. The initial predicted coordinates of each virtual node are then moved within a preset range. The movable range corresponds to the predicted coordinates of each virtual node. The preset conditions are that the range must be within the distance range of connected nodes, within the angle range of adjacent edges, and within the corresponding second-category feature region.

[0058] At this point, the node coordinates of each video frame and the predicted coordinates of the virtual nodes are obtained.

[0059] Preferably, based on the change of the predicted coordinates of the virtual nodes in consecutive video frames, the degree to which the nodes and the virtual nodes conform to the motion inertia and the human body structure, a loss function for video segmentation is established, including: An arbitrary video segment is designated as the target video segment. An optical flow field of any two adjacent video frames of the target video segment is obtained using an optical flow estimation algorithm. The optical flow field includes motion vectors of all pixels in the temporally preceding frame of the two vector frames. The optical flow estimation algorithm can be a conventional technique, such as the Lucas-Kanade algorithm.

[0060] The target video is segmented into The virtual nodes are recorded as ,Will In the The predicted coordinate set of video frames is recorded as ,Will In the The predicted coordinate set of video frames is recorded as It should be noted that In the The actual coordinates of the video frame belong to ,and In the The actual coordinates of the video frames and In the The motion vector of the actual coordinates of a video frame should be consistent with the motion vector of the neighborhood pixel point. Therefore, based on the consistency of the motion vector, the loss function of the target video segmentation is constructed.

[0061] right Extract a prediction coordinate from the prediction coordinate set of each video frame and construct it in chronological order A prediction coordinate sequence is obtained by permutation and combination method R predicted coordinate sequences.

[0062] The partial loss function of any predicted coordinate sequence satisfies the expression: ; Where, express Partial loss function for the d-th predicted coordinate sequence; The number of video frames representing the target video segment; 、 express The d-th predicted coordinate sequence is in , The predicted coordinates of video frames; Indicated by and The vector formed points to the The predicted coordinates of video frames; Indicates the preset number of neighborhood pixels; express The motion vector of the sth neighborhood pixel; Represents the normalization function.

[0063] Where, Indicated by and The vector formed by The modulus of the difference in motion vector of the sth neighborhood pixel is represented by and The combined difference between the vector formed and the direction and length of human movement, the larger the value, the greater the The d-th predicted coordinate sequence is in , The more the video frames do not conform to the human body motion situation; express The dth predicted coordinate sequence does not conform to the degree of human motion between all consecutive video frames. The larger the value, the The larger the partial loss function value of the d-th predicted coordinate sequence.

[0064] A prediction coordinate sequence is extracted from each set of prediction coordinate sequences of all virtual nodes of the target video segment, and Y prediction coordinate sequence combinations of the target video segment are obtained through a permutation and combination method.

[0065] It should be noted that each predicted coordinate sequence combination represents a coordinate movement process of all virtual nodes in the target video segment. It not only needs to achieve the minimum loss function value of all virtual nodes, but also should conform to the distance range and angle range between nodes and virtual nodes, and between virtual nodes. Therefore, the present invention combines the loss function of all virtual nodes and the human body situation between nodes to obtain the loss function of the target video segment.

[0066] The loss function of the target video segment satisfies the expression: ; Where, represents the loss function for the combination of the y-th predicted coordinate sequence of the target video segment; The number of virtual nodes representing the target video segment; Represents the partial loss function corresponding to the zth virtual node of the yth predicted coordinate sequence combination of the target video segment; The normal index of the zth virtual node of the yth predicted coordinate sequence combination of the target video segment is represented. When the zth virtual node and the adjacent nodes or virtual nodes all meet the distance range of the connected nodes and the angle range of the adjacent edges, then ,otherwise ; Represents the normalization function.

[0067] At this point, the loss function of video segmentation is obtained.

[0068] Preferably, based on the loss function of the video segmentation, the actual coordinates of the virtual node are obtained, including: obtaining The predicted coordinate sequence combination when is the minimum value includes all predicted coordinate sequences and predicted coordinates as the actual coordinates of the corresponding virtual node in the corresponding video frame.

[0069] At this point, the actual coordinates of all virtual nodes of each video frame of each video segment are obtained.

[0070] S4: Based on the coordinates of all nodes in all video frames and the actual coordinates of virtual nodes, the pose transfer network is used to perform image enhancement on each video segment.

[0071] Specifically, the actual coordinates of all virtual nodes in the target video frame and the node coordinates together constitute a skeleton model of the human body at the corresponding moment. For each video segment, each video frame and the corresponding skeleton model are used as input to generate an enhanced image using a posture transfer network. The posture transfer network can be a GFLA network.

[0072] At this point, the enhancement of the user's motion video image is completed.

[0073] An embodiment of the present invention also discloses a user motion video image enhancement system based on machine vision, including a processor and a memory, wherein the memory stores computer program instructions. When the computer program instructions are executed by the processor, a user motion video image enhancement method based on machine vision according to the present invention is implemented.

[0074] The above system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface. The configuration and functions of these components are known in the art and will not be described in detail here.

[0075] While several embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous modifications, variations, and alternatives will occur to those skilled in the art without departing from the concept and spirit of the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the present invention.

Claims

1. A method for enhancing user motion video images based on machine vision, characterized in that: include: Establishing a standard skeleton model, wherein the standard skeleton model includes a plurality of nodes, a plurality of edges, a distance between connected nodes, and an angle range between adjacent edges; acquiring a motion video; Based on the grayscale change degree of pixel points with the same coordinates in all video frames, the stable coordinates of the motion video are obtained; based on the grayscale value of the stable coordinates of the video frame, the number of grayscale extreme value points in the video frame, and the grayscale change degree on the grayscale extreme value point connection line, the definition of the video frame is obtained; based on the change pattern of the definition of the video frame, the motion video is divided into several video segments; Perform semantic segmentation on each video frame to obtain several feature regions of each video frame; obtain the node coordinates and predicted coordinates of virtual nodes in each video frame based on a posture estimation algorithm and a standard skeleton model; establish a loss function for video segmentation based on the changes in the predicted coordinates of virtual nodes in consecutive video frames, the degree to which the nodes and virtual nodes conform to motion inertia, and the human body structure; record the predicted coordinates of each virtual node corresponding to the minimum value of the loss function as the actual coordinates of the virtual node; Based on the coordinates of all nodes in all video frames and the actual coordinates of virtual nodes, a pose transfer network is used to perform image enhancement on each video segment.

2. The method for enhancing user motion video images based on machine vision according to claim 1, characterized in that: The step of obtaining stable coordinates of a motion video includes: Record the pixel sequence of any coordinates as the target pixel sequence; obtain the grayscale value type of the target pixel sequence and the number of pixels of each grayscale value; Calculate the pixel stability of any coordinates based on the grayscale value range and grayscale dispersion of the pixel sequence; The video frame is divided into several rectangular areas, and the coordinates with the highest pixel stability in each rectangular area are obtained and recorded as stable coordinates.

3. The method for enhancing user motion video images based on machine vision according to claim 2, characterized in that: The pixel stability of any coordinate satisfies the expression: ； Where, Indicates the pixel stability of the corresponding coordinates of the target pixel sequence; Represents the grayscale value type of the target pixel sequence; The set of pixel numbers representing all grayscale values ​​of the target pixel sequence; Represents the number of pixels with the i-th grayscale value in the target pixel sequence; represents the maximum value function; represents the natural exponential function.

4. The method for enhancing user motion video images based on machine vision according to claim 1, characterized in that: The obtaining of the clarity of the video frame includes: Any video frame is recorded as the target video frame, and the clarity of the target video frame satisfies the expression: ； Where, Indicates the clarity of the target video frame; represents the number of stable coordinates; Indicates the number of pixels in the corresponding pixel sequence of the grayscale value of the cth stable coordinate of the target video frame; Indicates the number of video frames of motion video; Indicates the number of grayscale extreme point connections of the target video frame; Represents the number of pixels connected by the h-th grayscale extreme point of the target video frame; 、 Represents the grayscale value of the a+1th and ath pixel points on the line connecting the hth grayscale extreme value point of the target video frame; represents the absolute value function; Represents the normalization function.

5. The method for enhancing user motion video images based on machine vision according to claim 1, characterized in that: The step of dividing the motion video into several video segments includes: If the clarity of the k+1th video frame is greater than or equal to the clarity of the kth video frame, the k+1th video frame is encoded as 1. If the clarity of the k+1th video frame is less than the clarity of the kth video frame, the k+1th video frame is encoded as -1 to obtain a clarity difference sequence of the motion video; if the kth video frame is encoded as 1, the k-1th video frame is encoded as 1, and the k+1th video frame is encoded as -1, the kth video frame is recorded as a feature video frame; all video frames are judged in turn to obtain several feature video frames; Any two adjacent feature video frames and all video frames between the two adjacent feature video frames are recorded as a video segment, thereby obtaining a plurality of video segments of the motion video.

6. The method for enhancing user motion video images based on machine vision according to claim 1, characterized in that: The step of obtaining several characteristic regions of each video frame includes: Input the target video frame into a first-class semantic segmentation model to obtain a binary mask of the target video frame, in which the human body area is 1 and the background area is 0; multiply the target video frame by the binary mask to obtain the human body area of ​​the target video frame, and input the human body area into a second-class semantic segmentation model to obtain a multi-class segmentation map, which contains several parts represented by several numerical values; The pixels contained in each part of the target video frame are recorded as a feature region, the feature regions corresponding to the parts belonging to the human body are recorded as first-class feature regions, and the feature regions corresponding to the parts not belonging to the human body are recorded as second-class feature regions.

7. The method for enhancing user motion video images based on machine vision according to claim 6, characterized in that: The obtaining of the node coordinates of each video frame and the predicted coordinates of the virtual node includes: A posture estimation algorithm is used to obtain the node coordinates of the first-category feature area of ​​the target video frame and the initial predicted coordinates of the virtual nodes of the second-category feature area. The initial predicted coordinates of each virtual node are moved within the preset conditions. The movable range is the predicted coordinates corresponding to each virtual node; the preset conditions are that they are within the distance range of the connected nodes, within the angle range of the adjacent edges, and within the corresponding second-category feature area.

8. The method for enhancing user motion video images based on machine vision according to claim 1, characterized in that: The loss function for establishing video segmentation includes: Segment any video into The virtual nodes are recorded as ;right Extract a prediction coordinate from the prediction coordinate set of each video frame and construct it in chronological order A predicted coordinate sequence of ; calculate A partial loss function of any predicted coordinate sequence of the video segment; extracting a predicted coordinate sequence from each set of predicted coordinate sequences of all virtual nodes of the video segment, which is recorded as a predicted coordinate sequence combination of the video segment; multiplying the normalized value of the mean of the partial loss function corresponding to all virtual nodes of the y-th predicted coordinate sequence combination of the video segment by the normalized value of the mean of the normal index to obtain the loss function of the y-th predicted coordinate sequence combination of the video segment, where the normal index is determined based on the positional relationship between the virtual node and the neighboring node and the virtual node.

9. The method for enhancing user motion video images based on machine vision according to claim 8, characterized in that: The calculation The partial loss function of any predicted coordinate sequence includes: obtaining the motion vectors of all pixel points of the time-preceding video frame in any two vector video frames of the target video segment based on the optical flow estimation algorithm; The partial loss function of any predicted coordinate sequence satisfies the expression: ； Where, express Partial loss function for the d-th predicted coordinate sequence; The number of video frames representing the target video segment; 、 express The d-th predicted coordinate sequence is in , The predicted coordinates of video frames; Indicated by and The vector formed; Indicates the preset number of neighborhood pixels; express The motion vector of the sth neighborhood pixel; represents the absolute value function; Represents the normalization function.

10. A user motion video image enhancement system based on machine vision, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a method for enhancing a user motion video image based on machine vision according to any one of claims 1 to 9 is implemented.

Images