A user motion video image enhancement method and system based on machine vision

Abstract

The application belongs to the technical field of image processing, and particularly relates to a user motion video image enhancement method and system based on machine vision, which comprises the following steps: acquiring a stable coordinate of a motion video; dividing the motion video into several video segments according to the variation law of the definition of video frames; acquiring node coordinates of each video frame and predicted coordinates of virtual nodes based on a pose estimation algorithm and a standard skeleton model; establishing a loss function of the video segments based on the variation of the predicted coordinates of the virtual nodes of the continuous video frames, the degree of compliance of the nodes and the virtual nodes with motion inertia, and the degree of compliance with the human body structure; acquiring actual coordinates of the virtual nodes; and using a pose migration network to perform image enhancement on each video segment. The application improves the target property and accuracy of motion video image enhancement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing. More particularly, the present application relates to a user motion video image enhancement method and system based on machine vision. BACKGROUND

[0002] With the development of artificial intelligence, more and more fields begin to expand the application range and optimize the user experience through artificial intelligence. For example, users need to spend a lot of time and cost to receive professional training offline. If an AI motion coach is established through artificial intelligence, it can not only reduce the cost of users, but also develop different training plans for different users, so that users can train in any place, greatly improve the exercise effect of users and ensure the exercise experience of users. The AI motion coach generates a digital person based on the physical data of the user and a motion scheme of the user. In the user motion process, the user's action is compared with the action in the standard action library through the camera motion capture, so as to realize the action guidance. In this process, based on the user motion video, the video needs to be image enhanced to avoid poor motion capture effect due to blurring and other problems, so as to avoid incorrect action guidance.

[0003] Currently, optical flow, pose prior and other technologies are commonly used to estimate motion trajectories, determine video enhancement targets and ensure video enhancement effects. In related technologies, for example, the Chinese patent document with the authorization announcement number CN119445043B discloses a fusion method of dynamic video and three-dimensional model based on image generation. It discloses the introduction of a NeRF three-dimensional geometric perception generation network, which makes the shape and position of the object always match the actual geometric structure in the scene, enhances the geometric capture ability, and improves the precision of video generation.

[0004] However, unlike the geometric structure matching in ideal conditions, the user may cover the geometric structure when wearing loose clothes during exercise, which may result in poor matching effect and incorrect video enhancement target recognition. In addition, since the user may move at a high speed during the exercise process, the video may be blurred, which may lose information, result in incorrect video enhancement target recognition, and poor video enhancement effect. SUMMARY

[0005] To solve the above technical problem of poor motion video image enhancement effect, the present application provides a solution in the following aspects.

[0006] In a first aspect, the present application provides a user motion video image enhancement method based on machine vision, comprising:

[0007] A standard skeleton model is established, the standard skeleton model comprising a plurality of nodes, a plurality of edges, and distances of connected nodes and angle ranges of adjacent edges; a motion video is acquired; stable coordinates of the motion video are acquired based on a degree of gray scale change of pixel points with the same coordinates in all video frames; a definition of the video frames is acquired according to a gray scale value of the stable coordinates of the video frames, a number of gray scale extreme points of the video frames and a degree of gray scale change on a connecting line of the gray scale extreme points; the motion video is divided into a plurality of video segments according to a change rule of the definition of the video frames; a plurality of feature regions of each video frame are acquired by performing semantic segmentation on the video frames; node coordinates and predicted coordinates of virtual nodes of each video frame are acquired based on a pose estimation algorithm and the standard skeleton model; a loss function of the video segments is established based on a change of the predicted coordinates of the virtual nodes of consecutive video frames, degrees of compliance of the nodes and the virtual nodes with motion inertia and degrees of compliance with a human body structure; the predicted coordinates of each virtual node corresponding to a minimum value of the loss function are recorded as actual coordinates of the virtual node; and image enhancement is performed on each video segment by using a pose transfer network based on all node coordinates of all video frames and the actual coordinates of the virtual nodes.

[0008] The application can make pose estimation and motion analysis be different for different people by establishing a personalized skeleton model, thereby improving analysis accuracy of a user's motion video. The application can make a smart fitness application still maintain high accuracy and stability in a most real and most casual home environment of a user by simulating skeleton nodes under clothes of the user by using virtual nodes. The application can improve target property and accuracy of image enhancement of a motion video.

[0009] Preferably, the stable coordinates of the motion video are acquired by:

[0010] A pixel point sequence with an arbitrary coordinate is recorded as a target pixel point sequence; a type of gray scale values of the target pixel point sequence and a number of pixel points of each gray scale value are acquired;

[0011] Pixel stability of the arbitrary coordinate is calculated according to a gray scale value range and a gray scale dispersion of the pixel point sequence;

[0012] The video frame is equally divided into a plurality of rectangular regions; coordinates with the highest pixel stability of each rectangular region are recorded as stable coordinates.

[0013] Preferably, the pixel stability of the arbitrary coordinate satisfies an expression:

[0014] ;

[0015] In the expression, represents the pixel stability of the coordinate corresponding to the target pixel point sequence; represents the type of gray scale values of the target pixel point sequence; The set of pixel numbers representing all grayscale values ​​of the target pixel sequence; Represents the number of pixels with the i-th grayscale value in the target pixel sequence; represents the maximum value function; represents the natural exponential function.

[0016] Preferably, obtaining the clarity of the video frame includes:

[0017] Any video frame is recorded as the target video frame, and the clarity of the target video frame satisfies the expression:

[0018] ;

[0019] Where, Indicates the clarity of the target video frame; represents the number of stable coordinates; Indicates the number of pixels in the corresponding pixel sequence of the grayscale value of the cth stable coordinate of the target video frame; Indicates the number of video frames of motion video; Indicates the number of grayscale extreme point connections of the target video frame; Represents the number of pixels connected by the h-th grayscale extreme point of the target video frame; 、 Represents the grayscale value of the a+1th and ath pixel points on the line connecting the hth grayscale extreme value point of the target video frame; represents the absolute value function; Represents the normalization function.

[0020] The present invention adopts a composite definition index, which can intelligently identify the intervals of user movement and use this as an anchor point to divide the video stream into multiple logically complete movement units.

[0021] Preferably, dividing the motion video into a plurality of video segments comprises:

[0022] If the clarity of the k+1th video frame is greater than or equal to the clarity of the kth video frame, the k+1th video frame is encoded as 1. If the clarity of the k+1th video frame is less than the clarity of the kth video frame, the k+1th video frame is encoded as -1 to obtain a clarity difference sequence of the motion video; if the kth video frame is encoded as 1, the k-1th video frame is encoded as 1, and the k+1th video frame is encoded as -1, the kth video frame is recorded as a feature video frame; all video frames are judged in turn to obtain several feature video frames;

[0023] Any two adjacent feature video frames and all video frames between the two adjacent feature video frames are recorded as a video segment, thereby obtaining a plurality of video segments of the motion video.

[0024] The motion video is segmented in the application, which provides motion units with complete start and end points for subsequent analysis instead of isolated video frames, so that the global optimization step with the largest amount of calculation can be accurately and specifically applied to the inside of the key motion units, improving the overall processing efficiency.

[0025] Preferably, the method further comprises:

[0026] inputting the target video frame into a semantic segmentation model to obtain a binary mask of the target video frame, wherein the human body region is 1 and the background region is 0; multiplying the target video frame and the binary mask to obtain a human region of the target video frame; inputting the human region into a multi-class semantic segmentation model to obtain a multi-class segmentation graph containing a plurality of types of parts represented by a plurality of types of values.

[0027] Each part contained in the target video frame is recorded as a feature region, the feature region corresponding to the part belonging to the human body is recorded as a first type of feature region, and the feature region corresponding to the part not belonging to the human body is recorded as a second type of feature region.

[0028] Preferably, the method further comprises:

[0029] obtaining the node coordinates of the first type of feature region and the initial predicted coordinates of the virtual nodes of the second type of feature region by using a pose estimation algorithm; moving the initial predicted coordinates of each virtual node within a preset condition range, wherein the movable range of each virtual node is the predicted coordinates corresponding to the virtual node; and the preset condition is to satisfy the distance range of the connected nodes, the angle range of the adjacent edges and the corresponding second type of feature region.

[0030] The application determines the predicted coordinates of the virtual nodes based on a standard skeleton model, so that the positions of the virtual nodes conform to the human body structure, and the problem of large errors caused by image enhancement is avoided.

[0031] Preferably, the method further comprises:

[0032] the first virtual node of any video segment is recorded as ; a predicted coordinate is extracted from the set of predicted coordinates of each video frame to form a predicted coordinate sequence of ; and the loss function of the video segment is calculated as ​​a partial loss function of any prediction coordinate sequence, extracting a prediction coordinate sequence from a set of prediction coordinate sequences of all virtual nodes of the video segment, and denoted as a prediction coordinate sequence combination of the video segment; multiplying a normalized value of a mean value of the partial loss function of all virtual nodes of the yth prediction coordinate sequence combination of the video segment by a normalized value of a mean value of the normal index to obtain a loss function of the yth prediction coordinate sequence combination of the video segment, wherein the normal index is determined based on a positional relationship between the virtual node and the neighboring node and the virtual node.

[0033] The application estimates an overall loss function by jointly optimizing all video frames of the entire motion unit, and the solution is an optimal solution on the entire space-time path rather than a local greedy optimal solution, thereby completely avoiding error accumulation and jitter phenomenon in mechanism and being capable of guaranteeing the continuity of the enhanced image.

[0034] Preferably, the calculating a partial loss function of any prediction coordinate sequence, comprises: obtaining motion vectors of all pixel points of a video frame in time in any two vector video frames of the target video segment based on an optical flow estimation algorithm; The partial loss function of any prediction coordinate sequence satisfies an expression:

[0035] In the expression, represents a partial loss function of the dth prediction coordinate sequence; represents a number of video frames of the target video segment; represents prediction coordinates of the dth prediction coordinate sequence of the target video segment in the th video frame and the th video frame; represents a vector composed of and represents a preset number of neighboring pixel points; represents a motion vector of the st neighboring pixel point of the target video segment; represents an absolute value function; represents a normalization function. In the second aspect, the application provides a user motion video image enhancement system based on machine vision, comprising a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are executed by the processor to realize the above-mentioned user motion video image enhancement method based on machine vision.

[0036] In the second aspect, the application provides a user motion video image enhancement system based on machine vision, comprising a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are executed by the processor to realize the above-mentioned user motion video image enhancement method based on machine vision.

[0037] ​​​​​​​​By adopting the technical scheme, the computer program of the user motion video image enhancement method based on machine vision is generated and stored in the memory, so as to be loaded and executed by the processor, thereby the terminal device is manufactured according to the memory and the processor, and the use is facilitated.

[0038] The present application has the advantages of:

[0039] (1) The present application can obtain the corresponding bone node position according to the motion relationship and the coordinate change by predicting the position of the virtual node even if the user wears loose, so as to meet the purpose of user motion video image enhancement and improve the evaluation accuracy of the user motion posture.

[0040] (2) The present application can enhance the video frame towards the user motion state, so as to improve the effect and accuracy of the user motion video image enhancement. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a flow chart illustrating the user motion video image enhancement method based on machine vision in the present application;

[0042] Figure 2 is a schematic view illustrating the standard bone model. DETAILED DESCRIPTION

[0043] The present application discloses a user motion video image enhancement method based on machine vision, referring to Figure 1 , including steps S1-S4:

[0044] S1: establishing a standard bone model, the standard bone model containing a plurality of nodes, a plurality of edges, and the distance of the connected nodes and the angle range of the adjacent edges; obtaining a motion video.

[0045] It should be noted that the AI motion coach captures the motion video of the user during the user's motion, compares the user's action with the action in the standard action library, so as to realize the action guidance. The key to whether the action is standard lies in the relative position and relationship of the skeleton and the limbs, for example, the angle of the thigh and the calf when squatting. Each joint is regarded as a node, and the relative position and relationship between the nodes can simplify the action judgment. However, the user may make a wrong judgment of the nodes and the action due to the loose clothing, blurred image and other factors during the motion, so it is necessary to enhance the image of the motion video. Since the motion video focuses on the person and the action, only the person part is enhanced. Therefore, the present application first obtains the motion video of the user and the standard bone model for providing reference for the body structure of the user.

[0046] Specifically, a standard skeleton model is established, which includes several nodes, several edges, and the distance range of connected nodes and the angle range of adjacent edges. It should be noted that the standard skeleton model can be obtained through platforms such as Sketchfab and medical schools. The distance of connected nodes and the angle range of adjacent edges are consistent with the bone length of the human body's medical structure and the angle range that bones can bend. Figure 2 A schematic diagram of a standard skeleton model.

[0047] The user's motion process video is obtained through a terminal used by the user to record the motion process, and grayscale is performed frame by frame to obtain a motion video.

[0048] At this point, the standard skeleton model and motion video have been obtained.

[0049] S2: Based on the grayscale change degree of pixel points with the same coordinates in all video frames, the stable coordinates of the motion video are obtained; according to the grayscale value of the stable coordinates of the video frame, the number of grayscale extreme points of the video frame and the grayscale change degree on the line connecting the grayscale extreme points, the clarity of the video frame is obtained; according to the change pattern of the clarity of the video frame, the motion video is divided into several video segments.

[0050] It should be noted that the user's motion process can be considered a combination of several motion units. Taking jumping jacks as an example, the user moves their hands from the sides of their legs to their head while simultaneously opening and closing their legs, then returns their hands to the sides of their legs and closes their legs, forming a "stationary-moving-stationary" motion unit. Within this motion unit, the clarity of the video frames of the motion video changes from "clear-blurred-clear". Video frames with higher clarity can better reflect the user's movements. Blurred video frames can be enhanced based on the high-definition video frames according to motion inertia, thereby obtaining an enhanced blurred image that accurately reflects the user's movements, thereby achieving the purpose of enhancing the user's motion video image. Therefore, the present invention decomposes the motion video to obtain multiple video segments, each of which represents a motion unit.

[0051] It should be further explained that the difference in the clarity of video frames lies in the presentation of image details. Image details are mainly expressed through the grayscale changes of pixels. Grayscale extremes are the most prominent features in video frames. Therefore, for any video frame of a motion video, the number of grayscale extremes and the degree of grayscale change on the line connecting the grayscale extreme points are analyzed to obtain the clarity of each video frame. At the same time, there are some stable background areas in the video frame. The changes in the clarity of these areas can reflect the relative clarity of the video frame. The grayscale of the pixels in the stable area always returns to a relatively stable grayscale. Therefore, the present invention combines the grayscale changes of the video frame and the grayscale changes of the neighborhood of the stable pixels to obtain the clarity of the video frame.

[0052] Specifically, based on the degree of change of the gray scale of the pixel points with the same coordinates in all video frames, the stable coordinates of the motion video are obtained, including:

[0053] A plurality of pixel point sequences with the same coordinates constitute the motion video, and the pixel point sequence is formed by the gray scale values of the pixel points at the same position of the video frames at different times in chronological order. The number of pixel point sequences is determined according to the video display format of the motion video. For example, under the video display format of 1080P, the resolution of the video frame is 1920×1080 pixels, and the number of pixel point sequences is 1920×1080.

[0054] The pixel point sequence of any coordinates is denoted as a target pixel point sequence. The gray scale value types of the target pixel point sequence and the number of pixel points of each gray scale value are obtained.

[0055] It should be noted that the more dispersed the gray scale distribution of the target pixel point sequence is and the more gray scale values the target pixel point sequence has, the weaker the pixel stability of the coordinates corresponding to the target pixel point sequence is. Therefore, according to the gray scale value range and the gray scale dispersion of the pixel point sequence, the pixel stability of any coordinates is calculated.

[0056] The pixel stability of the coordinates corresponding to the target pixel point sequence satisfies the expression:

[0057] ;

[0058] In the formula, represents the pixel stability of the coordinates corresponding to the target pixel point sequence; represents the gray scale value types of the target pixel point sequence; represents the set of the number of pixel points of all gray scale values of the target pixel point sequence; represents the number of pixel points of the i-th gray scale value of the target pixel point sequence; represents the maximum function; represents the natural exponential function.

[0059] In the formula, represents the maximum number of pixel points of all gray scale value types of the target pixel point sequence; represents the difference between the number of pixel points of the i-th gray scale value of the target pixel point sequence and the maximum number of pixel points. The larger the value is, the fewer the number of pixel points of the i-th gray scale value of the target pixel point sequence is; The average difference between the number of pixels representing all the gray values of the target pixel point sequence and the maximum number of pixels, the greater the value, the fewer the average number of pixels of all the gray values of the target pixel point sequence, and the more concentrated the pixel gray distribution of the coordinates corresponding to the target pixel point sequence, so the greater the pixel stability of the coordinates corresponding to the target pixel point sequence, and the fewer the types of gray values of the target pixel point sequence, the higher the pixel stability of the coordinates corresponding to the target pixel point sequence.

[0060] The video frame is equally divided into a plurality of rectangular regions, and the coordinates with the highest pixel stability of each rectangular region are obtained, which are denoted as stable coordinates. It should be noted that the number of equally divided rectangular regions is set by the implementer according to the actual implementation, for example, in the case of a 1920x1080 resolution, the greatest common divisor of 1920 and 1080 is 20, and the rectangular region can be set to 20x20, so that the video frame can be equally divided.

[0061] At this point, a plurality of stable coordinates of the motion video are obtained.

[0062] What is needed is that, for any video frame, if the pixel point at the stable coordinate position is in an unstable state, that is, in a gray value type with fewer pixel points, then the definition of the video frame is relatively weak. In addition, the lower the number of gray extreme values of the video frame and the degree of gray change on the gray extreme point connecting line, the higher the degree of blurring of the video frame, and the lower the definition.

[0063] Preferably, the definition of the video frame is obtained according to the gray value of the stable coordinate of the video frame, the number of gray extreme points of the video frame, and the degree of gray change on the gray extreme point connecting line, comprising:

[0064] Any video frame is denoted as a target video frame, the gray extreme points of the target video frame are obtained, the gray values of the pixel points on the connecting line between any two gray extreme points of the target video frame are obtained, and a gray sequence between the two gray extreme points is constructed in the order of position.

[0065] The definition of the target video frame satisfies the expression:

[0066] ;

[0067] In the formula, denotes the definition of the target video frame; denotes the number of stable coordinates; denotes the number of pixels of the gray value of the cth stable coordinate of the target video frame at the corresponding pixel point sequence; denotes the number of video frames of the motion video; denotes the number of gray extreme point connecting lines of the target video frame; denotes the number of pixels of the hth gray extreme point connecting line of the target video frame; , represents the gray scale value of the a+1th and ath pixel point of the hth gray scale extreme point connecting line of the target video frame; represents an absolute value function; represents a normalization function.

[0068] In the formula, represents the proportion of the gray scale value of the cth stable coordinate of the target video frame in the pixel point quantity of the corresponding pixel point sequence, and the greater the value, the more the cth stable coordinate of the target video frame tends to be stable; represents the average proportion of the gray scale value of all stable coordinates of the target video frame in the pixel point quantity of the corresponding pixel point sequence, and the greater the value, the more all stable coordinates of the target video frame tend to be stable, and thus the higher the definition of the target video frame is; represents the gray scale difference of the a+1th and ath pixel point of the hth gray scale extreme point connecting line of the target video frame, represents the average gray scale difference of the pixel point of the hth gray scale extreme point connecting line of the target video frame, and the greater the value, the greater the gray scale change of the hth gray scale extreme point connecting line of the target video frame, and the richer the gray scale value, and thus the higher the area definition of the hth gray scale extreme point connecting line of the target video frame is; represents the area definition sum of all gray scale extreme point connecting lines of the target video frame, and the greater the value, the more the area of all gray scale extreme point connecting lines of the target video frame is clear, and thus the higher the definition of the target video frame is.

[0069] Up to now, the definition of each video frame is obtained.

[0070] It should be noted that the definition sequence of the motion video is obtained in time sequence, and the change of the definition of the sequence has certain relevance with the motion state of the user. When the user is static or in motion interval, the video frame is often the clearest, and when the user moves violently or in the process of completing the action, the video frame is often the most blurred, and thus the motion video can be divided based on the change of the definition of the video frame.

[0071] Preferably, the motion video is divided into a plurality of video segments according to the change rule of the definition of the video frame, including:

[0072] In time sequence, a definition sequence of the motion video is obtained, and the definition sequence is differentially encoded: if the definition of the k+1th video frame is greater than or equal to the definition of the kth video frame, the k+1th video frame is encoded as 1, if the definition of the k+1th video frame is less than the definition of the kth video frame, the k+1th video frame is encoded as -1, and a definition differential sequence of the motion video is obtained. It should be noted that in the definition differential sequence of the motion video, the part of continuous +1 represents the process of the video frame gradually becoming clear, and the part of continuous -1 represents the process of the video frame gradually becoming blurred.

[0073] A feature video frame of the motion video is obtained: if the kth video frame is encoded as 1, the k-1th video frame is encoded as 1, and the k+1th video frame is encoded as -1, the kth video frame is recorded as a feature video frame; all video frames are sequentially judged to obtain a plurality of feature video frames. It should be noted that the feature video frame is an image of a relatively clearest moment in the motion video.

[0074] Any two adjacent feature video frames and all video frames between the two adjacent feature video frames are recorded as a video segment, and a plurality of video segments of the motion video are obtained.

[0075] At this point, a plurality of video segments of the motion video are obtained.

[0076] S3: performing semantic segmentation on each video frame to obtain a plurality of feature regions of each video frame; based on a pose estimation algorithm and a standard skeleton model, obtaining node coordinates of each video frame and predicted coordinates of virtual nodes; based on changes in the predicted coordinates of the virtual nodes of the continuous video frames, the degree of compliance of the nodes and the virtual nodes with the motion inertia, and the degree of compliance with the human body structure, establishing a loss function of the video segment; based on the loss function of the video segment, obtaining actual coordinates of the virtual nodes.

[0077] It should be noted that the step S2 segments the motion video so that the motion unit can be divided during the user's movement, and the user's action can be analyzed in real time and accurately. In order to reduce the analysis range, considering that the user is in various environments, such as parks, living rooms, gyms, etc., the background is complex and irrelevant to the movement, therefore each video frame can be segmented and only the human part can be enhanced. At present, the semantic segmentation model based on artificial intelligence is gradually enriched, which can not only segment the human, background, but also segment each part of the human body, such as hat, hair, shirt, left arm, etc. Therefore, the present application performs semantic segmentation on all video frames of each video segment.

[0078] It needs to be further explained that in order to complete the action evaluation, the user's body structure also needs to be matched with the standard bone model to obtain the user's bone model. In these segmented parts, part is the human body part, part is the clothing and other objects that affect the posture estimation. The human body part can be matched with the standard bone model more accurately, but the human body part under the clothing cover needs to be matched through the predicted virtual node. The matching effect may not be good in one video frame, but in continuous video frames, the movement of the human body has rules and inertia. By comparing the possible positions of the virtual nodes in the continuous video frames, the position of the virtual node can be corrected. The nodes and the virtual nodes together constitute the skeleton of the character region of the video frame. On this basis, the video frame is enhanced, which can ensure the accuracy of the user motion video image enhancement. Therefore, the loss function is constructed to evaluate the position accuracy of the virtual nodes of the video segments. When the loss function is minimum, the best position of all virtual nodes is obtained.

[0079] Specifically, the semantic segmentation is performed on each video frame to obtain a plurality of feature regions of each video frame, including:

[0080] The target video frame is input into a one-class semantic segmentation model to obtain a binary mask of the target video frame, wherein the human body region is 1 and the background region is 0. The one-class semantic segmentation model is, for example, a DeepLabV3+ model trained using a segmentation dataset COCO.

[0081] The target video frame is multiplied by the binary mask to obtain a character region of the target video frame, and the character region is input into a two-class semantic segmentation model to obtain a multi-class segmentation graph containing a plurality of types of parts represented by a plurality of types of values. The two-class semantic segmentation model is, for example, a CE2P model trained using a human parsing dataset. The values and parts are, for example, 1 representing the head, 2 representing the upper garment, 3 representing the left arm, and the like.

[0082] Each part of the target video frame contains a pixel point, which is recorded as a feature region. The feature region corresponding to the part belonging to the human body is recorded as a one-class feature region, and the feature region corresponding to the part not belonging to the human body is recorded as a two-class feature region.

[0083] At this point, a plurality of feature regions of each video frame are obtained.

[0084] It should be noted that a type of feature region can be better matched to the nodes in the standard skeleton model, so the application obtains nodes from a type of feature region, and then predicts virtual nodes from the type of feature region to the adjacent type of feature region. In order to evaluate the accuracy of the prediction, the application sets a loss function for evaluating the degree to which the virtual nodes conform to the human body structure. Considering the continuity of motion, the positions of the same virtual nodes in consecutive video frames should also have continuity, and the optical flow can well reflect the continuity of motion. Therefore, the application combines the cases in which the nodes and virtual nodes of a single frame conform to the standard skeleton model, and the cases in which the position changes of the nodes and virtual nodes of consecutive frames conform to the standard skeleton model, to establish a loss function, and when the loss function is minimized, the positions of all virtual nodes are the most accurate.

[0085] Preferably, based on the pose estimation algorithm and the standard skeleton model, the node coordinates of each video frame and the predicted coordinates of the virtual nodes are obtained, including:

[0086] The pose estimation algorithm is used to obtain the node coordinates of the type of feature region of the target video frame, and the initial predicted coordinates of the virtual nodes of the type of feature region. The initial predicted coordinates of each virtual node are moved within a predetermined condition range, and the movable range is the predicted coordinates corresponding to each virtual node. The predetermined condition is to meet the distance range of the connected nodes, the angle range of the adjacent edges, and the corresponding type of feature region.

[0087] At this point, the node coordinates of each video frame and the predicted coordinates of the virtual nodes are obtained.

[0088] Preferably, based on the changes in the predicted coordinates of the virtual nodes of consecutive video frames, the degree to which the nodes and virtual nodes conform to the motion inertia and conform to the human body structure, a loss function for video segmentation is established, including:

[0089] Any video segment is recorded as a target video segment, and an optical flow field of any two adjacent video frames of the target video segment is obtained based on an optical flow estimation algorithm. The optical flow field contains the motion vectors of all pixel points in the video frame that is earlier in time in the two vector video frames. The optical flow estimation algorithm is a prior art, such as the Lucas-Kanade algorithm.

[0090] The first virtual node of the target video segment is recorded as , and the predicted coordinate set of the first video frame is recorded as , and the predicted coordinate set of the first video frame is recorded as . It should be noted that the actual coordinates of the first video frame belong to ,and In the The actual coordinates of the video frames and In the The motion vector of the actual coordinates of a video frame should be consistent with the motion vector of the neighborhood pixel point. Therefore, based on the consistency of the motion vector, the loss function of the target video segmentation is constructed.

[0091] right Extract a prediction coordinate from the prediction coordinate set of each video frame and construct it in chronological order A prediction coordinate sequence is obtained by permutation and combination method R predicted coordinate sequences.

[0092] The partial loss function of any predicted coordinate sequence satisfies the expression:

[0093] ;

[0094] Where, express Partial loss function for the d-th predicted coordinate sequence; The number of video frames representing the target video segment; 、 express The d-th predicted coordinate sequence is in , The predicted coordinates of video frames; Indicated by and The vector formed points to the The predicted coordinates of video frames; Indicates the preset number of neighborhood pixels; express The motion vector of the sth neighborhood pixel; Represents the normalization function.

[0095] Where, Indicated by and The vector formed by The modulus of the difference in motion vector of the sth neighborhood pixel is represented by and The combined difference between the vector formed and the direction and length of human movement, the larger the value, the greater the The d-th predicted coordinate sequence is in , The more the video frames do not conform to the human body motion situation; express The greater the value of the dth predicted coordinate sequence of the target video segment is, the greater the degree of inconsistency of the dth predicted coordinate sequence of the target video segment with the human body motion between all consecutive video frames is The greater the value of the partial loss function of the dth predicted coordinate sequence of the target video segment is.

[0096] A predicted coordinate sequence is extracted from the set of predicted coordinate sequences of all virtual nodes of the target video segment, and Y predicted coordinate sequence combinations of the target video segment are obtained through permutation and combination.

[0097] It should be noted that each predicted coordinate sequence combination represents a coordinate movement process of all virtual nodes in the target video segment, which not only needs to achieve the minimum loss function value of all virtual nodes, but also needs to meet the distance range and angle range between the nodes and the virtual nodes and between the virtual nodes. Therefore, the loss function of the target video segment is obtained in combination with the loss function of all virtual nodes and the human body condition between the nodes.

[0098] The loss function of the target video segment satisfies the expression:

[0099] ;

[0100] In the formula, represents the loss function of the yth predicted coordinate sequence combination of the target video segment; represents the number of virtual nodes of the target video segment; represents the partial loss function corresponding to the zth virtual node of the yth predicted coordinate sequence combination of the target video segment; represents the normal index of the zth virtual node of the yth predicted coordinate sequence combination of the target video segment, when the zth virtual node and the adjacent node or virtual node both meet the distance range of the connected nodes and the angle range of the adjacent edges, then , otherwise ; represents a normalization function.

[0101] Thus, the loss function of the video segment is obtained.

[0102] Preferably, based on the loss function of the video segment, the actual coordinates of the virtual nodes are obtained, including: When the predicted coordinate sequence combination is the minimum value, all the predicted coordinate sequences contained therein and the predicted coordinate are marked as the actual coordinates of the corresponding virtual node in the corresponding video frame.

[0103] Thus, the actual coordinates of all virtual nodes of each video frame of each video segment are obtained.

[0104] S4: Based on all node coordinates and actual coordinates of virtual nodes of all video frames, image enhancement is performed on each video segment using a pose transfer network.

[0105] Specifically, the actual coordinates and the node coordinates of all virtual nodes of the target video frame jointly constitute a skeleton model of the human body at the corresponding moment, for each video segment, taking each video frame and the corresponding skeleton model as input, using a pose transfer network to generate an enhanced image. The pose transfer network can be a GFLA network.

[0106] At this point, the enhancement of the user motion video image is completed.

[0107] The embodiment of the application further discloses a user motion video image enhancement system based on machine vision, comprising a processor and a memory, the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a user motion video image enhancement method based on machine vision according to the application is realized.

[0108] The above system also includes a communication bus and a communication interface and other components familiar to those skilled in the art, the setting and function of which are known in the art, and therefore will not be described here.

[0109] Although the present specification has shown and described several embodiments of the present application, it will be apparent to those skilled in the art that many modifications, changes and substitutions can be made thereto without departing from the spirit and scope of the present application. It should be understood that various alternatives to the embodiments of the application described herein can be employed in practicing the application.

Claims

1. A method for enhancing user motion video images based on machine vision, characterized in that: include: Establishing a standard skeleton model, wherein the standard skeleton model includes a plurality of nodes, a plurality of edges, a distance between connected nodes, and an angle range between adjacent edges; acquiring a motion video; Based on the grayscale change degree of pixel points with the same coordinates in all video frames, the stable coordinates of the motion video are obtained; based on the grayscale value of the stable coordinates of the video frame, the number of grayscale extreme value points in the video frame, and the grayscale change degree on the grayscale extreme value point connection line, the definition of the video frame is obtained; based on the change pattern of the definition of the video frame, the motion video is divided into several video segments; Perform semantic segmentation on each video frame to obtain several feature regions of each video frame; obtain the node coordinates and predicted coordinates of virtual nodes in each video frame based on a posture estimation algorithm and a standard skeleton model; establish a loss function for video segmentation based on the changes in the predicted coordinates of virtual nodes in consecutive video frames, the degree to which the nodes and virtual nodes conform to motion inertia, and the human body structure; record the predicted coordinates of each virtual node corresponding to the minimum value of the loss function as the actual coordinates of the virtual node; Based on the coordinates of all nodes in all video frames and the actual coordinates of virtual nodes, a pose transfer network is used to perform image enhancement on each video segment.

2. The method for enhancing user motion video images based on machine vision according to claim 1, characterized in that: The step of obtaining stable coordinates of a motion video includes: Record the pixel sequence of any coordinates as the target pixel sequence; obtain the grayscale value type of the target pixel sequence and the number of pixels of each grayscale value; Calculate the pixel stability of any coordinates based on the grayscale value range and grayscale dispersion of the pixel sequence; The video frame is divided into several rectangular areas, and the coordinates with the highest pixel stability in each rectangular area are obtained and recorded as stable coordinates.

3. The method for enhancing user motion video images based on machine vision according to claim 2, characterized in that: The pixel stability of any coordinate satisfies the expression: ； Where, Indicates the pixel stability of the corresponding coordinates of the target pixel sequence; Represents the grayscale value type of the target pixel sequence; The set of pixel numbers representing all grayscale values ​​of the target pixel sequence; Represents the number of pixels with the i-th grayscale value in the target pixel sequence; represents the maximum value function; represents the natural exponential function.

4. The method for enhancing user motion video images based on machine vision according to claim 1, characterized in that: The obtaining of the clarity of the video frame includes: Any video frame is recorded as the target video frame, and the clarity of the target video frame satisfies the expression: ； Where, Indicates the clarity of the target video frame; represents the number of stable coordinates; Indicates the number of pixels in the corresponding pixel sequence of the grayscale value of the cth stable coordinate of the target video frame; Indicates the number of video frames of motion video; Indicates the number of grayscale extreme point connections of the target video frame; Represents the number of pixels connected by the h-th grayscale extreme point of the target video frame; 、 Represents the grayscale value of the a+1th and ath pixel points on the line connecting the hth grayscale extreme value point of the target video frame; represents the absolute value function; Represents the normalization function.

5. The method for enhancing user motion video images based on machine vision according to claim 1, characterized in that: The step of dividing the motion video into several video segments includes: If the clarity of the k+1th video frame is greater than or equal to the clarity of the kth video frame, the k+1th video frame is encoded as 1. If the clarity of the k+1th video frame is less than the clarity of the kth video frame, the k+1th video frame is encoded as -1 to obtain a clarity difference sequence of the motion video; if the kth video frame is encoded as 1, the k-1th video frame is encoded as 1, and the k+1th video frame is encoded as -1, the kth video frame is recorded as a feature video frame; all video frames are judged in turn to obtain several feature video frames; Any two adjacent feature video frames and all video frames between the two adjacent feature video frames are recorded as a video segment, thereby obtaining a plurality of video segments of the motion video.

6. The method for enhancing user motion video images based on machine vision according to claim 1, characterized in that: The step of obtaining several characteristic regions of each video frame includes: Input the target video frame into a first-class semantic segmentation model to obtain a binary mask of the target video frame, in which the human body area is 1 and the background area is 0; multiply the target video frame by the binary mask to obtain the human body area of ​​the target video frame, and input the human body area into a second-class semantic segmentation model to obtain a multi-class segmentation map, which contains several parts represented by several numerical values; The pixels contained in each part of the target video frame are recorded as a feature region, the feature regions corresponding to the parts belonging to the human body are recorded as first-class feature regions, and the feature regions corresponding to the parts not belonging to the human body are recorded as second-class feature regions.

7. The method for enhancing user motion video images based on machine vision according to claim 6, characterized in that: The obtaining of the node coordinates of each video frame and the predicted coordinates of the virtual node includes: A posture estimation algorithm is used to obtain the node coordinates of the first-category feature area of ​​the target video frame and the initial predicted coordinates of the virtual nodes of the second-category feature area. The initial predicted coordinates of each virtual node are moved within the preset conditions. The movable range is the predicted coordinates corresponding to each virtual node; the preset conditions are that they are within the distance range of the connected nodes, within the angle range of the adjacent edges, and within the corresponding second-category feature area.

8. The method for enhancing user motion video images based on machine vision according to claim 1, characterized in that: The loss function for establishing video segmentation includes: Segment any video into The virtual nodes are recorded as ;right Extract a prediction coordinate from the prediction coordinate set of each video frame and construct it in chronological order A predicted coordinate sequence of ; calculate A partial loss function of any predicted coordinate sequence of the video segment; extracting a predicted coordinate sequence from each set of predicted coordinate sequences of all virtual nodes of the video segment, which is recorded as a predicted coordinate sequence combination of the video segment; multiplying the normalized value of the mean of the partial loss function corresponding to all virtual nodes of the y-th predicted coordinate sequence combination of the video segment by the normalized value of the mean of the normal index to obtain the loss function of the y-th predicted coordinate sequence combination of the video segment, where the normal index is determined based on the positional relationship between the virtual node and the neighboring node and the virtual node.

9. The method for enhancing user motion video images based on machine vision according to claim 8, characterized in that: The calculation The partial loss function of any predicted coordinate sequence includes: obtaining the motion vectors of all pixel points of the time-preceding video frame in any two vector video frames of the target video segment based on the optical flow estimation algorithm; The partial loss function of any predicted coordinate sequence satisfies the expression: ; Where, express Partial loss function for the d-th predicted coordinate sequence; The number of video frames representing the target video segment; 、 express The d-th predicted coordinate sequence is in , The predicted coordinates of video frames; Indicated by and The vector formed; Indicates the preset number of neighborhood pixels; express The motion vector of the sth neighborhood pixel; represents the absolute value function; Represents the normalization function.

10. A user motion video image enhancement system based on machine vision, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a method for enhancing a user motion video image based on machine vision according to any one of claims 1 to 9 is implemented.

Images