Human body optical flow estimation method based on frequency domain prior

Abstract

A human body optical flow estimation method based on frequency domain prior comprises the steps that S1, a pre-training data set is constructed, and the corresponding relation between a three-dimensional shape and a two-dimensional image is obtained in a three-dimensional rendering mode; s2, calculating a frequency domain feature of a three-dimensional point cloud corresponding to each pixel in the two-dimensional image; s3, constructing a deep neural network comprising an image feature extractor and an optical flow estimator, and realizing three-dimensional frequency domain prior-based unsupervised pre-training on the network by using constraint conditions of shape registration based on the corresponding relationship and the frequency domain features; and S4, only loading the pre-training parameters of the image feature extractor, and training the image feature extractor and the optical flow estimator on the human body optical flow estimation data set in a fully supervised manner to obtain a deep neural network model for estimating human body motion. According to the method, the three-dimensional geometric features and the two-dimensional image features are combined, three-dimensional non-rigid shape registration is adopted as a pre-training task, the three-dimensional human body frequency domain features are effectively utilized, and the accuracy of human body optical flow estimation is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for estimating human optical flow based on frequency domain prior. Background Art

[0002] Optical flow estimation is one of the important research topics in the field of computer vision. It is used to estimate the motion pattern of pixels in an image sequence, including the direction and speed of motion. Traditional optical flow estimation algorithms assume that the brightness of objects in an image is constant within a short time interval, and that the motion of objects is small and smooth. Therefore, traditional optical flow estimation methods are highly sensitive to light and are prone to estimation bias. In recent years, optical flow estimation based on deep learning has made significant progress in the field of computer vision. Optical flow estimation can be used to solve many computer vision problems, such as motion detection, object tracking, video compression, 3D reconstruction, robot navigation, etc., so it has high research value.

[0003] However, the current optical flow estimation algorithms focus on rigid objects, such as chairs and vehicles, and rarely involve complex and changeable motion detection, such as pedestrians. Therefore, the present invention is committed to solving the optical flow estimation of non-rigid objects, especially the optical flow estimation of the human body. The human body is a complex structure with multiple joints. Optical flow estimation must not only consider the global motion, but also optimize and adjust the local motion of each joint and limb of the human body.

[0004] Optical flow estimation:

[0005] Optical flow estimation refers to the technology of estimating the movement speed and direction of pixels by analyzing the brightness changes of pixels between adjacent frames in an image sequence. Its purpose is to calculate the displacement of each pixel on the image plane, that is, the motion information of the object, by comparing two or more consecutive frames in the image. Optical flow is a vector with direction and length. Depending on whether sparse points in the image are selected for optical flow estimation, optical flow estimation can be divided into sparse optical flow and dense optical flow. In the present invention, dense optical flow estimation is adopted, focusing on the three-dimensional frequency domain characteristics of the human body structure, and focusing on the comparative study of the accuracy and generalization ability of different methods for human optical flow estimation.

[0006] Non-rigid shape registration:

[0007] Non-rigid shape registration refers to the task of aligning or matching different instances of non-rigid shapes (such as faces, hands, biological tissues, etc.) or shapes at different time points. In non-rigid shape registration, the shapes may be deformed by bending, twisting, stretching, etc., so it is necessary to establish a correspondence between the shapes for registration. Non-rigid shape registration has a wide range of applications in computer vision, medical image processing, computer-aided design and other fields.

[0008] Depth function mapping:

[0009] Deep Functional Maps (DFM) is a non-rigid shape matching technology based on deep learning, which is used to align points between two shapes. DFM uses function mapping to describe the relationship between shapes, which is a method of mapping each point of one shape to the corresponding point on another shape. DFM automatically learns function mapping through deep neural networks to achieve shape alignment. The advantage of DFM is that it can handle nonlinear deformations between different shapes and can handle missing data in shapes. In addition, DFM can also be applied to aspects such as shape generation and editing.

[0010] The basic idea of ​​DFM is to represent shapes as functions. The matching between functions can be transformed into a vector matching problem in the function space. In this process, DFM uses deep learning technology to learn a function mapping network to map the function of one shape to the function of another shape, thereby achieving alignment between shapes. The main challenge of DFM is how to design a robust and effective function mapping network to cope with complex deformations between different shapes.

[0011] Optical flow estimation algorithms based on deep learning can be divided into CNN and Transformer network frameworks. Existing CNN framework algorithms include RAFT and GMA, and representative Transformer framework-based algorithms include FlowFormer, FlowFormer++, and GMFlow. Among them, FlowFormer and FlowFormer++ also use a pre-trained framework and use image classification as a network pre-training task, which effectively improves the accuracy and generalization of optical flow estimation, but its network complexity also increases.

[0012] The existing algorithm can be mainly divided into two parts: image feature extractor and optical flow estimator. The input is a set of image pairs, usually the previous and next frames in the video. The feature extractor learns the key information of the input image, realizes the relationship between image pixels based on feature matching, estimates the direction and speed of movement, and then optimizes the optical flow through the optical flow estimator, and finally obtains a predicted value similar to the actual optical flow.

[0013] The prior art methods mainly include the following disadvantages:

[0014] 1) Existing optical flow estimation algorithms focus on rigid objects and ignore the optical flow estimation of complex non-rigid shapes such as crowds.

[0015] 2) In the existing pre-training algorithms for optical flow estimation, only two-dimensional information is used for network enhancement, and three-dimensional information is not effectively utilized.

[0016] 3) Less attention has been paid to human optical flow estimation, and the accuracy of existing methods is low.

[0017] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0018] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a method for estimating human optical flow based on frequency domain prior.

[0019] To achieve the above object, the present invention adopts the following technical solutions:

[0020] A method for estimating human optical flow based on frequency domain prior includes the following steps:

[0021] S1. Construct a pre-training data set and use 3D rendering to obtain the correspondence between 3D shapes and 2D images.

[0022] S2, calculating the frequency domain features of the three-dimensional point cloud corresponding to each pixel in the two-dimensional image;

[0023] S3, constructing a deep neural network including an image feature extractor and an optical flow estimator, and based on the corresponding relationship and the frequency domain features, using the constraint condition of shape registration, implementing unsupervised pre-training based on three-dimensional frequency domain prior for the deep neural network;

[0024] S4. Only the pre-trained parameters of the image feature extractor are loaded, and the image feature extractor and the optical flow estimator are trained in a fully supervised manner on a human optical flow estimation dataset to obtain a deep neural network model that can accurately estimate human motion based on a two-dimensional image sequence.

[0025] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for estimating human optical flow based on frequency domain prior is implemented.

[0026] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the method for estimating human optical flow based on frequency domain prior is implemented.

[0027] The present invention has the following beneficial effects:

[0028] The human optical flow estimation method based on frequency domain prior of the present invention innovatively integrates three-dimensional geometric features and two-dimensional image features, establishes a close connection between three-dimensional human shape and two-dimensional image through three-dimensional rendering, not only enhances the performance of optical flow estimation feature extractor, but also significantly improves the accuracy of optical flow estimation for complex non-rigid objects, especially human motion. Using three-dimensional non-rigid shape registration as a pre-training task, the present invention effectively utilizes the frequency domain features of three-dimensional human body, and enhances the network's ability to capture details and complex movements such as turning and twisting the head. In addition, the present invention adopts an unsupervised pre-training method, which can improve the performance of the existing optical flow estimation model without increasing network parameters, thereby achieving higher-precision human motion capture in various dynamic scenes. In addition, the technical solution of the present invention also improves the network's ability to estimate the optical flow of image pairs with a large number of frame intervals, and effectively distinguishes the movement direction of the human body and the background. Experimental verification shows its wide application potential and practical application value in the field of computer vision.

[0029] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Flow chart of human optical flow estimation based on frequency domain prior according to an embodiment of the present invention

[0031] Figure 2 A schematic diagram of a three-dimensional rendering process of constructing a pre-training data set based on D-FAUST according to an embodiment of the present invention.

[0032] Figure 3 This is a visualization comparison diagram of SHOF single-person optical flow estimation according to an embodiment of the present invention.

[0033] Figure 4 This is a visualization comparison diagram of the SHOF single-person optical flow estimation interval of 20 frames according to an embodiment of the present invention.

[0034] Figure 5 This is a visualization comparison diagram of MHOF multi-person optical flow estimation according to an embodiment of the present invention.

[0035] Figure 6This is a visualization comparison diagram of MHOF multi-person optical flow estimation at an interval of 10 frames according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0037] The present invention proposes a method for estimating human optical flow based on frequency domain priors, and uses a method for three-dimensional human registration. Through a pre-training method based on three-dimensional frequency domain features, the present invention effectively solves the problem that the existing methods are not accurate enough in estimating the optical flow of non-rigid shapes such as human bodies, and have weak generalization capabilities on other similar data sets. The present invention associates the three-dimensional human body patch structure with the two-dimensional image through three-dimensional rendering, converts the dense registration information between different three-dimensional human body shapes into the pairing relationship of corresponding pixels between different images, and then converts it into two-dimensional optical flow, effectively associating the three-dimensional geometric information with the two-dimensional image semantic information.

[0038] See also Figure 1 The embodiment of the present invention provides a method for estimating human optical flow based on frequency domain prior, comprising the following steps:

[0039] Step S1: construct a pre-training data set and use a three-dimensional rendering method to obtain the correspondence between the three-dimensional shape and the two-dimensional image.

[0040] In a preferred embodiment, step S1 specifically includes: preparing a basic data set, wherein the basic data set stores three-dimensional information of human bodies of different body shapes, genders and exercise types in the form of three-dimensional facets; randomly selecting multiple action sequences of men and women from the basic data set to ensure the diversity and representativeness of the data set; performing data enhancement processing on the selected sequences, and using UV atlas technology to perform texture mapping on the sequences to supplement the missing human texture information in the data set; performing normalization operations on the three-dimensional human body data to ensure consistency and comparability between different body shapes and action data; using three-dimensional rendering technology, such as Pyrender, to render the three-dimensional shape to generate a two-dimensional image with a predetermined resolution; during the rendering process, determining the camera ray corresponding to each pixel point according to the set camera posture, thereby tracing back to the precise position of the three-dimensional facet; through camera ray tracing, obtaining the vertex information of the three-dimensional facet and the coordinates of the corresponding points to establish the corresponding relationship between the pixel point and the three-dimensional facet; through the rendering process, obtaining the RGB rendered image, the human body area mask image and the partial point cloud data corresponding to the rendered image, providing data support for subsequent frequency domain feature calculation and deep neural network training.

[0041] Step S2: Calculate the frequency domain features of the three-dimensional point cloud corresponding to each pixel in the two-dimensional image.

[0042] In a preferred embodiment, step S2 specifically includes: using the Laplace-Beltrami algorithm to perform frequency domain feature analysis on the vertices of the three-dimensional human body patch, calculating the frequency domain eigenvalues ​​and eigenvectors of each vertex, so as to capture the motion pattern and deformation characteristics of the vertex; for the case where the camera ray is projected to a non-vertex position during the rendering process, the frequency domain eigenvalues ​​and eigenvectors of the point are calculated by the triangle barycentric coordinate method; using the known three-dimensional patch vertex information and the coordinates of point P, the position of point P in the triangular patch is determined by solving the weight, wherein the solution of the weight satisfies the constraint that point P is located in the plane formed by the vertices of the triangular patch, that is, the sum of the weights is equal to one, and the coordinates of P are the weighted average of the vertex coordinates; according to the vertex eigenvector and the weight of point P, the eigenvector of point P is calculated, and the eigenvector is a weighted combination of the vertex eigenvectors, reflecting the characteristic performance of point P in the frequency domain.

[0043] Step S3: construct a deep neural network including an image feature extractor and an optical flow estimator, and based on the correspondence relationship and the frequency domain features, utilize the constraints of shape registration to implement unsupervised pre-training of the deep neural network based on three-dimensional frequency domain prior.

[0044] In a preferred embodiment, step S3 specifically includes: using RAFT and GMA as the basic framework to construct a deep neural network, which includes an image feature extractor and an optical flow estimator; inputting the paired two-dimensional RGB images into the network, learning the image features through the image feature extractor, and performing optical flow estimation on each pixel to obtain a preliminary optical flow field; according to the coordinates of the pixel points and their estimated optical flow, the pixel coordinates of a two-dimensional RGB image are converted to the coordinate system of another associated two-dimensional RGB image to achieve coordinate alignment; using the human body area mask image to identify and select the effective human body pixels in the image, and converting the optical flow information into point-to-point registration information; according to the calculated frequency domain features, the registration information is further converted into frequency domain registration information to utilize the frequency domain features of the three-dimensional human body shape; using the characteristics of the deep function mapping DFM, including its symmetry and orthogonality, to construct a constraint function to ensure the accuracy and consistency of the frequency domain registration information. In addition, displacement perturbation is introduced during the pre-training process to enhance the generalization ability of the network so that it can better adapt to different human body movements and posture changes.

[0045] Step S4, only load the pre-trained parameters of the image feature extractor, and train the image feature extractor and the optical flow estimator in a fully supervised manner on a human optical flow estimation dataset to obtain a deep neural network model that can accurately estimate human motion based on a two-dimensional image sequence.

[0046] In a preferred embodiment, in step S4, based on different downstream tasks, including single-person optical flow estimation and multi-person optical flow estimation tasks, given the actual optical flow, the image feature extractor and the optical flow estimator are trained simultaneously in a fully supervised manner.

[0047] The present invention effectively improves the accuracy of human optical flow estimation, utilizes three-dimensional geometric information to improve the performance of two-dimensional optical flow estimation feature extractors, and solves the problem of optical flow estimation for non-rigid shapes such as human bodies. Compared with the prior art, the main advantages of the present invention are as follows: 1) Three-dimensional shape registration is used as a pre-training task for the optical flow estimation algorithm, and three-dimensional geometric information is combined with two-dimensional image information to effectively identify the optical flow estimation of complex human movements, such as turning around, twisting the head, and other movements. 2) By utilizing the inherent characteristics of DFM, an unsupervised approach is used to implement pre-training tasks without increasing network parameters, effectively improving the accuracy and generalization ability of human optical flow estimation. 3) Three-dimensional rendering is used to associate the three-dimensional human body shape with the two-dimensional rendered image, ensuring that the pixel points containing the human body on the two-dimensional image can be matched to the corresponding three-dimensional point cloud data on the three-dimensional human body shape.

[0048] The following further describes specific embodiments of the present invention, algorithm examples and experimental verification.

[0049] like Figure 1 As shown, a method for estimating human optical flow based on frequency domain prior in the present invention comprises the following steps:

[0050] 1) Construct a pre-training dataset and use 3D rendering to obtain the correspondence between 3D shapes and 2D images.

[0051] 2) Calculate the frequency domain features of the three-dimensional point cloud corresponding to each pixel in the image.

[0052] 3) The existing optical flow estimation algorithm is split into a feature extractor and an optical flow estimator. Based on the correspondence and frequency domain features, the constraints of shape registration are used to achieve unsupervised pre-training based on three-dimensional frequency domain priors.

[0053] 4) Only the pre-trained parameters of the feature extractor are loaded, and the feature extractor and optical flow estimator are trained in a fully supervised manner on the human optical flow estimation dataset.

[0054] In step 1), for the construction of the pre-training dataset, D-FAUST is used as the basic dataset. D-FAUST stores 3D human information in the form of 3D patches, including 3D data of motion sequences of different body shapes, different genders, and different types. In order to improve the network effect, 10 male sequences and 10 female sequences were randomly selected. Since the D-FAUST dataset lacks human texture data, in order to ensure the consistency of the sequence and the performance of the optical flow estimation image feature extractor, the UV atlas is used to color the sequence, and the 3D human data is uniformly normalized. After that, Pyrender is used for 3D shape rendering. The size of the rendered image is 384×384. According to the camera pose set during the rendering process, the camera ray corresponding to each pixel in the rendered image can be determined. The camera ray can be traced back to the position of the 3D patch, including the vertex information A, B, C of the 3D patch and the coordinates of the point P. During the rendering process, RGB rendered images, human area mask images, and partial point cloud data corresponding to the rendered image can be obtained.

[0055] In step 2), for the existing three-dimensional human body shape, the Laplace-Beltrami algorithm can be used to calculate the frequency domain eigenvalues ​​and eigenvectors of each vertex of the three-dimensional face. Since the camera ray will be projected to the non-vertex position of the face during the rendering process, the present invention uses the coordinates of the triangle center of gravity to further calculate the eigenvalue and eigenvector of the point P. According to step 1), the vertex information A, B, C of the three-dimensional face and the coordinates of the point P are known, and the weight (u, v, w) can be solved, and its constraints are:

[0056] P=uA+vB+wC,u+v+w=1,

[0057] In addition, the eigenvectors of vertices A, B, and C can be calculated as Φ A ,Φ B ,Φ C , combined with the weights (u,v,w), the feature vector Φ of P P It can be expressed as:

[0058] Φ P =uΦ A +vΦ B +wΦ C

[0059] In step 3), the present invention uses RAFT and GMA as the basic optical flow estimation network framework, and divides the network into an image feature extractor and an optical flow estimator. The input of the network is a set of two-dimensional RGB images I i ,I j , the image feature extractor learns the image features according to the input image and estimates the optical flow of each pixel to obtain F ijAccording to the coordinate information of each pixel and the estimated optical flow, I i The coordinates of the image are moved to I j Next, use the human body region in step 1) to cover the image M i ,M j Select effective pixels of the human body to convert the optical flow into point-by-point registration information Π ji According to the feature vector calculated in step 2), the point-by-point registration information can be converted into frequency domain registration information C ij According to the characteristics of DFM, its symmetry and orthogonality can be expressed as:

[0060] E bij =||C ij C ji -I|| 2 +||C ji C ij -I|| 2

[0061]

[0062] Then, its constraint function can be defined as:

[0063] L DFM =λ bij E bij +λ ortho

[0064] Among them, λ bij and ortho is the constraint weight of the two features. In order to improve the generalization ability of the network, displacement perturbation is added during the pre-training process.

[0065] In step 4), only the pre-trained parameters of the image feature extractor are loaded. Based on different downstream tasks, including single-person optical flow estimation and multi-person optical flow estimation tasks, given the actual optical flow, the network image feature extractor and optical flow estimator are trained simultaneously in a fully supervised manner.

[0066] Experimental Results

[0067] Figure 2 The three-dimensional rendering process of constructing a pre-training data set based on D-FAUST according to an embodiment of the present invention is shown. Figure 3 This is a visualization comparison diagram of the SHOF single-person optical flow estimation of an embodiment of the present invention (columns 2-4 are experiments using RAFT as the basic framework, and columns 5-7 are experiments using GMA as the basic framework. Odd-numbered rows are output optical flows of each method, and even-numbered rows are error diagrams). Figure 4 This is a visualization comparison diagram of the SHOF single-person optical flow estimation at an interval of 20 frames according to an embodiment of the present invention. Figure 5 This is a visualization comparison of MHOF multi-person optical flow estimation according to an embodiment of the present invention (columns 2-3 are experiments using RAFT as the basic framework, and columns 4-5 are experiments using GMA as the basic framework. Odd-numbered rows are output optical flows of each method, and even-numbered rows are error diagrams). Figure 6 This is a visualization comparison diagram of MHOF multi-person optical flow estimation at an interval of 10 frames according to an embodiment of the present invention.

[0068] In summary, the technical innovations and advantages of the present invention include:

[0069] 1) Using 3D human body registration as a pre-training task for optical flow estimation can effectively improve the accuracy and generalization ability of human optical flow estimation.

[0070] 2) Using 3D rendering, 3D geometric information is associated with 2D image information, and 3D frequency domain features are effectively used to enhance 2D feature extraction capabilities.

[0071] 3) Using unsupervised pre-training method, the network performance of the existing optical flow estimation model can be improved without increasing the network parameters.

[0072] 4) Effectively utilize 3D geometry to solve the optical flow estimation of complex human movements such as turning and bending.

[0073] 5) Compared with the existing optical flow estimation algorithms, the pre-training task of the present invention can effectively improve the performance of human optical flow estimation without increasing network parameters.

[0074] 6) Effectively estimate the optical flow between image pairs with a large frame interval, and effectively segment the movement direction of the human body and the background.

[0075] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0076] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0077] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0078] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0079] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0080] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0081] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0082] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0083] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0084] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0085] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0086] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0087] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for estimating human optical flow based on frequency domain prior, characterized in that: The following steps are involved: S1. Construct a pre-training data set and use 3D rendering to obtain the correspondence between 3D shapes and 2D images. S2, calculating the frequency domain features of the three-dimensional point cloud corresponding to each pixel in the two-dimensional image; S3, constructing a deep neural network including an image feature extractor and an optical flow estimator, and based on the corresponding relationship and the frequency domain features, using the constraint condition of shape registration, implementing unsupervised pre-training based on three-dimensional frequency domain prior for the deep neural network; S4. Only the pre-trained parameters of the image feature extractor are loaded, and the image feature extractor and the optical flow estimator are trained in a fully supervised manner on a human optical flow estimation dataset to obtain a deep neural network model for estimating human motion based on a two-dimensional image sequence.

2. The method for estimating human optical flow based on frequency domain prior according to claim 1, characterized in that: Step S1 specifically includes: Prepare a basic data set, wherein the basic data set stores three-dimensional information of human bodies of different body shapes, genders, and sports types in the form of three-dimensional patches; Randomly selecting a plurality of male and female action sequences from the base dataset; Performing data enhancement processing on the selected sequence, and performing texture mapping on the sequence using UV atlas technology to supplement the missing human body texture information in the data set; Performing normalization operation on three-dimensional human body data; Rendering the three-dimensional shape to generate a two-dimensional image with a predetermined resolution; during the rendering process, determining the camera ray corresponding to each pixel point according to the set camera posture; Through camera ray tracing, the vertex information of the three-dimensional surface and the coordinates of the corresponding points are obtained to establish the corresponding relationship between the pixel points and the three-dimensional surface; The RGB rendering image, the human body area mask image, and the partial point cloud data corresponding to the rendering image are obtained through the rendering process.

3. The method for estimating human optical flow based on frequency domain prior according to claim 1 or 2, characterized in that: Step S2 specifically includes: Calculate the frequency domain eigenvalue and eigenvector of each vertex of the three-dimensional human body patch; When the camera ray is projected to a non-vertex position during rendering, the frequency domain eigenvalue and eigenvector of the point are calculated using the triangle barycentric coordinate method. Using the known 3D patch vertex information and the coordinates of point P, the position of point P in the triangular patch is determined by solving the weights, where the weights are solved to satisfy the constraint that point P is located in the plane formed by the vertices of the triangular patch, that is, the sum of the weights is equal to one, and the coordinates of P are the weighted average of the vertex coordinates; According to the feature vector of the vertex and the weight of point P, the feature vector of point P is calculated. The feature vector is a weighted combination of the vertex feature vectors and reflects the characteristic performance of point P in the frequency domain.

4. The method for estimating human optical flow based on frequency domain prior according to claim 3, characterized in that: The Laplace-Beltrami algorithm is used to calculate the frequency domain eigenvalues ​​and eigenvectors of each vertex of the three-dimensional human body patch.

5. The method for estimating human optical flow based on frequency domain prior according to any one of claims 1 to 4, characterized in that: Step S3 specifically includes: Using the optical flow estimation network as the basic framework, a deep neural network including an image feature extractor and an optical flow estimator is constructed; Input the paired associated two-dimensional RGB images into the deep neural network, learn the image features through the image feature extractor, and estimate the optical flow of each pixel to obtain a preliminary optical flow field; According to the coordinates of the pixel points and their estimated optical flows, the pixel coordinates of a two-dimensional RGB image are converted into the coordinate system of another associated two-dimensional RGB image to achieve coordinate alignment; Use the human body area mask image to identify and select the effective human body pixels in the image, and convert the optical flow information into point-to-point registration information; According to the calculated frequency domain features, the registration information is further converted into frequency domain registration information; The constraint function is constructed by mapping the properties of DFM using the depth function, including its symmetry and orthogonality.

6. The method for estimating human optical flow based on frequency domain prior according to claim 5, characterized in that: The optical flow estimation network is a RAFT network or a GMA network.

7. The method for estimating human optical flow based on frequency domain prior according to claim 5 or 6, characterized in that: In step S3, displacement perturbations are introduced during the pre-training process to enhance the generalization ability of the network so that it can better adapt to different human body movements and posture changes.

8. The method for estimating human optical flow based on frequency domain prior according to any one of claims 1 to 7, characterized in that: In step S4, based on different downstream tasks, including single-person optical flow estimation and multi-person optical flow estimation tasks, given the actual optical flow, the image feature extractor and the optical flow estimator are trained simultaneously in a fully supervised manner.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for estimating human optical flow based on frequency domain prior as described in any one of claims 1 to 8 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for estimating human optical flow based on frequency domain prior as described in any one of claims 1 to 8 is implemented.

Images