A deep estimation network training method, device and equipment and storage medium
By combining stacked hourglass networks and cLSTM networks with a self-supervised training method, the accuracy problem of human depth estimation was solved, especially the poor performance of depth estimation in different poses and environments, achieving higher accuracy and distortion correction.
Patent Information
- Application Number
- CN202211596702.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-12
- Publication Date
- 2026-01-16
- Estimated Expiration
- 2042-12-12
AI Technical Summary
Existing technologies cannot accurately capture the geometric details of the human body, especially with different human postures and clothing details, making it difficult to accurately estimate depth. Furthermore, the acquisition of depth maps in the field is difficult, resulting in poor depth estimation performance.
By combining stacked hourglass networks and cLSTM networks, a self-supervised training method is used to train the network using human RGB images and joint coordinates, combined with normal maps, and the network parameters are updated until the preset conditions are met, thereby obtaining the spatiotemporal features of continuous video frames.
It improves the accuracy of depth estimation, corrects distortions between body parts, and enhances depth estimation capabilities in different postures and environments.
Smart Images
Figure CN115830093B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to a depth estimation network training method and device, equipment and a storage medium. BACKGROUND
[0002] For human body image depth estimation, the depth map obtained based on camera and other infrastructure cannot accurately capture the geometric details of the human body, and is greatly affected by the natural environment, which easily causes local holes in the depth map or even fails to capture the body parts. Most existing human body image-based depth estimation models are difficult to cope with different human body postures and clothing details, and the current models still have the problem of distortion (i.e. the relative order between body parts is incorrect) for the depth order between human body joints. At the same time, it is more difficult to obtain various real depth maps in the wild. SUMMARY
[0003] The present application aims to provide a depth estimation network training method, device, equipment and storage medium, which aims to solve the problem of poor estimation effect of human body image depth estimation network based on video due to the fact that the prior art cannot provide an effective depth estimation network training method.
[0004] In one aspect, the present application provides a depth estimation network training method, which comprises the following steps:
[0005] The human body RGB image set and the 17 joint coordinates of each RGB image are taken as the input of the first stacked hourglass network, the first stacked hourglass network is trained, and the trained first stacked hourglass network is obtained;
[0006] Obtain the continuous video frame image for the current training and the normal map of the continuous video frame image;
[0007] Input the continuous video frame image and the normal map into the trained first stacked hourglass network and the pre-constructed depth estimation network, obtain the output of the first stacked hourglass network and the depth estimation network, obtain the current estimation error of the depth estimation network according to the output of the first stacked hourglass network and the depth estimation network, and the depth estimation network comprises a second stacked hourglass network and a cLSTM network;
[0008] When the current estimation error does not meet the first preset condition, update the network parameters of the depth estimation network according to the current estimation error, and jump to the step of obtaining the continuous video frame image for the current training and the normal map of the continuous video frame image, to continue training the depth estimation network until the first preset condition is met, to obtain the trained depth estimation network.
[0009] In another aspect, the present application provides a human body image depth estimation method based on video, comprising the following steps:
[0010] sequentially obtaining a preset number of continuous video frame images from an input video file, and obtaining a normal map of the continuous video frame images;
[0011] inputting the continuous video frame images and the normal map of the continuous video frame images into the depth estimation network trained by the depth estimation network training method, to obtain a depth map of the continuous video frame images.
[0012] In another aspect, the present application further provides a depth estimation network training device, comprising:
[0013] a first training unit configured to take a human body RGB image set and 17 joint coordinates of each RGB image as inputs of a first stacked hourglass network, train the first stacked hourglass network, and obtain a trained first stacked hourglass network;
[0014] an image acquisition unit configured to acquire continuous video frame images and normal maps of the continuous video frame images for current training;
[0015] a first error acquisition unit configured to input the continuous video frame images and the normal maps into the trained first stacked hourglass network and a pre-constructed depth estimation network, acquire outputs of the first stacked hourglass network and the depth estimation network, and acquire a current estimation error of the depth estimation network according to the outputs of the first stacked hourglass network and the depth estimation network, wherein the depth estimation network comprises a second stacked hourglass network and a cLSTM network; and
[0016] a second training unit configured to, when the current estimation error does not meet a first preset condition, update network parameters of the depth estimation network according to the current estimation error, trigger the image acquisition unit to acquire continuous video frame images and normal maps of the continuous video frame images for current training, and continue training the depth estimation network until the first preset condition is met, to obtain a trained depth estimation network.
[0017] In another aspect, the present application further provides a computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method as described above when executing the computer program.
[0018] In another aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the steps of the method described above.
[0019] The present application trains the first stacked hourglass network by using a human RGB image set and 17 joint coordinates of each RGB image, inputs the trained first stacked hourglass network and the depth estimation network including the second stacked hourglass network and the cLSTM network with a continuous video frame image and a normal map, obtains the output of the first stacked hourglass network and the depth estimation network, obtains the current estimation error of the depth estimation network according to the output of the first stacked hourglass network and the depth estimation network, updates the network parameters of the depth estimation network according to the current estimation error when the current estimation error does not meet the first preset condition, continues to train the depth estimation network until a trained depth estimation network is obtained, thereby obtaining the spatiotemporal features of the continuous video frame image through the cLSTM network, obtaining more features of each frame image, improving the accuracy of depth estimation of the depth estimation network, and correcting the distortion between body parts in the depth estimation process through the cooperation of the first stacked hourglass network. BRIEF DESCRIPTION OF DRAWINGS
[0020] Figure 1 is an implementation flowchart of the depth estimation network training method provided by the first embodiment of the present application;
[0021] Figure 2 is an implementation flowchart of the depth estimation network training method provided by the second embodiment of the present application;
[0022] Figure 3 is an implementation flowchart of the human image depth estimation method based on video provided by the third embodiment of the present application;
[0023] Figure 4 is a structural schematic diagram of the depth estimation network training device provided by the fourth embodiment of the present application;
[0024] Figure 5 is a structural schematic diagram of the computing device provided by the fifth embodiment of the present application. DETAILED DESCRIPTION
[0025] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.
[0026] The specific implementation of the present application is described in detail below with reference to specific embodiments:
[0027] Example One:
[0028] Figure 1 The implementation process of the depth estimation network training method provided by the embodiment of the present application is shown, only the parts related to the embodiments of the present application are shown for the convenience of description, and the details are as follows:
[0029] In step S101, the human body RGB image set and the 17 joint coordinates of each RGB image are taken as the input of the first stacked hourglass network, the first stacked hourglass network is trained, and the trained first stacked hourglass network is obtained;
[0030] The embodiment of the present application is applicable to a computing device for training a depth estimation network. In the embodiment of the present application, a pre-constructed stacked hourglass network (Stacked Hourglass Networks) is pre-trained, and for the convenience of description, the stacked hourglass network is referred to as the first stacked hourglass network. Specifically, since the human body has a total of 17 joints, here, the first stacked hourglass network is supervised trained using the RGB image set with human body images and the 17 joint coordinates of each labeled RGB image, and the trained first stacked hourglass network is obtained for subsequent training of the depth estimation network.
[0031] In step S102, the continuous video frame images for the current training and the normal map of the continuous video frame images are obtained;
[0032] In the embodiment of the present application, the continuous video frame images in the video are used to train the depth estimation network at each time or each round of training, and the number of continuous video frame images can be set as needed, preferably, the number of continuous video frame images is 5 frames, so as to improve the accuracy of the trained depth estimation network and improve the training efficiency. After obtaining the continuous video frame images for the current training, the normal map corresponding to the continuous video frame images is obtained.
[0033] In step S103, the continuous video frame images and the normal map are input into the trained first stacked hourglass network and the pre-constructed depth estimation network, the outputs of the first stacked hourglass network and the depth estimation network are obtained, and the current estimation error of the depth estimation network is obtained according to the outputs of the first stacked hourglass network and the depth estimation network;
[0034] In the embodiment of the present application, the depth estimation network comprises a stacked hourglass network and a cLSTM network (convolutional LSTM Network), for the convenience of description, the stacked hourglass network is referred to as a second stacked hourglass network, the second stacked hourglass network is used to extract spatial features of the continuous video frame images, the cLSTM network is used to extract temporal features of the continuous video frame images, and the spatial features extracted by the second stacked hourglass network are combined to obtain the spatiotemporal features of the continuous video frame images. Specifically, when the current estimation error of the depth estimation network is obtained, the obtained continuous video frame images and the normal map are input into the depth estimation network, the output of the depth estimation network is obtained, the obtained continuous video frame images are input into the trained first stacked hourglass network, the output of the first stacked hourglass network is obtained, and then the current estimation error of the depth estimation network is obtained according to the outputs of the first stacked hourglass network and the depth estimation network. In this way, when the current estimation error is calculated, the target value (true value) is not required, and only the outputs of the first stacked hourglass network and the depth estimation network are required to obtain the current estimation error of the depth estimation network, thereby realizing the self-supervised training of the depth estimation network.
[0035] In step S104, it is judged whether the current estimation error meets the first preset condition, if yes, step S105 is executed, otherwise, step S106 is executed.
[0036] In the embodiment of the present application, when it is judged whether the current estimation error meets the first preset condition, it can be judged whether the current estimation error and the error change of the previous preset time meet the preset range, if yes, it can be considered that the current estimation error meets the first preset condition, otherwise, it can be considered that the first preset condition is not met.
[0037] In step S105, the trained depth estimation network is obtained.
[0038] In the embodiment of the present application, when the current estimation error meets the first preset condition, it can be considered that the depth estimation network has been trained, at this time, the trained depth estimation network can be obtained or output.
[0039] In step S106, the network parameters of the depth estimation network are updated according to the current estimation error, and the process jumps to step S102.
[0040] In the embodiment of the present application, when the current estimation error does not meet the first preset condition, it can be considered that the depth estimation network has not been trained, at this time, the network parameters of the depth estimation network are updated according to the current estimation error, and the process jumps to step S102 to obtain the continuous video frame images and the normal map of the continuous video frame images for the next (round) training, and the training of the depth estimation network is continued until the first preset condition is met, so as to obtain the trained depth estimation network.
[0041] The embodiment of the application obtains the space-time features of the continuous video frame images through the cLSTM network, can obtain more features of each frame image, improves the accuracy of the depth estimation of the depth estimation network, and corrects the distortion between the body parts in the depth estimation process through the cooperation of the first stacked hourglass network.
[0042] Example Two:
[0043] Figure 2 The implementation process of the depth estimation network training method provided by the second embodiment of the application is shown, only the parts related to the embodiments of the application are shown for the convenience of description, and the details are as follows:
[0044] In step S201, the human body RGB image set and the 17 joint coordinates of each RGB image are taken as the input of the first stacked hourglass network, the first stacked hourglass network is trained, and the trained first stacked hourglass network is obtained;
[0045] In the embodiment of the application, since the human body has a total of 17 joints, here, the first stacked hourglass network is supervised trained using the human body image RGB image set and the 17 joint coordinates of each labeled RGB image, and the trained first stacked hourglass network is obtained for the subsequent training of the depth estimation network.
[0046] In the specific implementation, the first stacked hourglass network can be trained through the following steps:
[0047] (1) obtaining the human body RGB image for the current training and the 17 joint coordinates from the human body RGB image set;
[0048] (2) taking the obtained human body RGB image and the 17 joint coordinates as the input of the first stacked hourglass network, and obtaining the depth values of the 17 joint coordinates corresponding to the joint nodes through the stacked hourglass network;
[0049] (3) using the first loss function to obtain the current estimation error of the first stacked hourglass network according to the obtained depth values and the preset target values;
[0050] (4) when the current estimation error of the first stacked hourglass network does not satisfy the second preset condition, updating the network parameters of the depth estimation network according to the current estimation error of the first stacked hourglass network, and jumping to step (1) to continue training the first stacked hourglass network until the second preset condition is met, so as to obtain the trained first stacked hourglass network.
[0051] In the embodiment of the application, the first loss function is Wherein, Z represents accumulation, K is the number of joint group combination, K = 153, (i k ,j k ) represents the k pair of joints, is the predicted depth of joint i, z jk is the predicted depth of joint j, r k is determined by the target depth value of joint i and joint j, wherein, when joint i is closer than joint j, r k = +1, when joint i is farther than joint j, r k = -1, when joint i is equal to joint j, r k = 0. Through the formula, the accuracy of depth order prediction is increased, thereby improving the accuracy of this error calculation.
[0052] When judging whether the current estimation error of the first stacked hourglass network meets the second preset condition, whether the current estimation error and the error of the previous preset time change is located in the preset range can be judged, yes, it can be considered that the current estimation error meets the second preset condition, otherwise, it can be considered that it does not meet the second preset condition, of course, other ways can also be used for judgment.
[0053] In step S202, continuous video frame images used for the current training and normal maps of the continuous video frame images are obtained;
[0054] In the embodiment of the application, during each or each round of training of the depth estimation network, the continuous video frame images in the video are used for training, and the number of continuous video frame images can be set as required, preferably, the number of continuous video frame images is 5 frames, so as to improve the accuracy of the depth estimation network obtained by training and improve the training efficiency. After obtaining the continuous video frame images used for the current training, the normal maps corresponding to the continuous video frame images are obtained.
[0055] In step S203, the continuous video frame images and the normal maps are input into the second stacked hourglass network to obtain the spatial feature maps of the continuous video frame images;
[0056] In the embodiment of the application, the depth estimation network comprises a second stacked hourglass network and a cLSTM network, the second stacked hourglass network receives the input continuous video frame images and normal maps to obtain the spatial feature maps corresponding to each frame of image in the continuous video frame images.
[0057] In step S204, the spatial feature maps of the continuous video frame images are input into the cLSTM network to obtain the first depth map of the continuous video frame images;
[0058] In the embodiment of the present application, the cLSTM network is used to extract the time features of the continuous video frame images, and the spatial features extracted by the second stacked hourglass network are combined to obtain the spatiotemporal features of the continuous video frame images, and finally the depth map corresponding to the spatiotemporal features is output, that is, the depth map of the continuous video frame images. For ease of description, the depth map is referred to as the first depth map here.
[0059] In an embodiment, when the spatial feature maps of the continuous video frame images are input into the cLSTM network to obtain the first depth map of the continuous video frame images, the first depth map of each frame image in the continuous video frame images is sequentially obtained. Specifically, when the first depth map of each frame image is obtained, the spatial feature maps of the current video frame image and the previous video frame image are input into the cLSTM network to obtain the first depth map of the current video frame image, thereby retaining the temporal information of the previous frame. If the current frame is the first frame in the continuous video frame images, there is no previous frame image, at this time, the spatial feature map of the current video frame image is directly input into the cLSTM network to obtain the depth map (first depth map) of the first frame image.
[0060] In step S205, the normal map of the first depth map is obtained, and each first depth map is warped to obtain the second depth map of the next frame image of each video frame image in the continuous video frame images.
[0061] In the embodiment of the present application, after the cLSTM network obtains all the first depth maps corresponding to the continuous video frame images, the normal map of each first depth map is obtained, and each first depth map is warped to obtain the depth map of the next frame image of each video frame image in the continuous video frame images. For ease of description, the depth map is referred to as the second depth map. As an example, if the first depth maps of the continuous video frame images F1, F2, F3, F4, F5 are F'1, F'2, F'3, F'4, F'5, the second depth maps of F2, F3, F4, F5 can be obtained after the first depth maps F'1, F'2, F'3, F'4 are warped, and the second depth map of the video frame image F1 can be set to be the same as the first depth map. At this time, the second depth maps of the continuous video frame images F1-F5 can be obtained.
[0062] In step S206, the continuous video frame images are input into the first stacked hourglass network to obtain the third depth map of the continuous 5 video frame images.
[0063] In the embodiment of the present application, the continuous video frame images are input into the first stacked hourglass network trained in step S201 to obtain the depth map of the continuous video frame images. For ease of description, the depth map is referred to as the third depth map.
[0064] In step S207, the current estimation error of the depth estimation network is obtained according to the first depth map and the second depth map, the first depth map and the third depth map, and the normal map of the first depth map and the normal map of the continuous video frame image.
[0065] In the embodiment of the present application, the current estimation error of the depth estimation network is obtained according to the first depth map and the second depth map, the first depth map and the third depth map, and the normal map of the first depth map and the normal map of the continuous video frame image. In this way, when calculating the current estimation error of the depth estimation network, the target value (true value) is not needed, and only the output of the first stacked hourglass network and the depth estimation network is needed to obtain the current estimation error of the depth estimation network, thereby realizing the self-supervised training of the depth estimation network.
[0066] In an embodiment, the current estimation error of the depth estimation network is obtained by the following steps:
[0067] The first estimation error of the depth estimation network is obtained using the second loss function according to the first depth map and the second depth map;
[0068] The second estimation error of the depth estimation network is obtained using the third loss function according to the first depth map and the third depth map;
[0069] The third estimation error of the depth estimation network is obtained using the fourth loss function according to the normal map of the first depth map and the normal map of the continuous video frame image;
[0070] The first estimation error, the second estimation error and the third estimation error are determined as the current estimation error of the depth estimation network.
[0071] In the embodiment, the estimation error determined by the first depth map and the second depth map, the first depth map and the third depth map, and the normal map of the first depth map and the normal map of the continuous video frame image is determined as the current estimation error of the depth estimation network, which improves the calculation accuracy of the current estimation error and further improves the training efficiency of the depth estimation network. Preferably, the third loss function is the same as the first loss function, and the fourth loss function is a mean square error (MSE) loss function, and the current estimation error of the depth estimation network can be accurately determined by using these loss functions.
[0072] In step S208, it is judged whether the current estimation error meets the first preset condition. If yes, step S209 is performed, otherwise step S210 is performed.
[0073] In the embodiment of the present application, when judging whether the current estimation error meets the first preset condition, it can be judged whether the current estimation error and the error variation of the previous preset time meet the preset range. If yes, it can be considered that the current estimation error meets the first preset condition, otherwise, it can be considered that the current estimation error does not meet the first preset condition. If the first estimation error, the second estimation error and the third estimation error are determined as the current estimation error of the depth estimation network as described in step 207, it can be respectively judged whether the first estimation error, the second estimation error and the third estimation error meet the preset condition.
[0074] In step S209, the trained depth estimation network is obtained;
[0075] In the embodiment of the present application, when the current estimation error meets the first preset condition, it can be considered that the depth estimation network has been trained and completed, at this time, the trained depth estimation network can be obtained or output.
[0076] In step S210, the network parameters of the depth estimation network are updated according to the current estimation error, and the step S202 is jumped to.
[0077] In the embodiment of the present application, when the current estimation error does not meet the first preset condition, it can be considered that the depth estimation network has not been trained and completed, at this time, the network parameters of the depth estimation network are updated according to the current estimation error, and the step S202 is jumped to, so as to obtain the continuous video frame images and the normal map of the continuous video frame images for the next (round) training, continue to train the depth estimation network until the first preset condition is met, so as to obtain the trained depth estimation network.
[0078] The embodiment of the present application can obtain more features of each frame of image by obtaining the space-time features of the continuous video frame images through the cLSTM network, improve the accuracy of the depth estimation of the depth estimation network, and correct the distortion between the body parts in the depth estimation process through the cooperation of the first stacked hourglass network.
[0079] Example Three:
[0080] Figure 3 The implementation process of the human image depth estimation method based on video provided by the embodiment three of the present application is shown, only the part related to the embodiment of the present application is shown for the convenience of description, and the details are as follows:
[0081] In step S301, a preset number of continuous video frame images are sequentially obtained from the input video file, and the normal map of the continuous video frame images is obtained;
[0082] In the embodiment of the present application, when a depth map of a video file needs to be obtained, a preset number of continuous video frame images are sequentially obtained from the video file, wherein the video file can be a video segment.
[0083] In step S302, the continuous video frame images and the normal maps of the continuous video frame images are input into a preset depth estimation network to obtain the depth map of the continuous video frame images.
[0084] In the embodiment of the present application, the preset depth estimation network is trained by the depth estimation network training method of the aforementioned embodiments one and two, the depth map of the continuous video frame images is obtained through the depth estimation network, and then the next preset number of continuous video frame images and the normal maps of the next continuous video frame images can be obtained from the input video file, the depth map of the next continuous video frame images is obtained through the depth estimation network, and so on until the depth map corresponding to the video file is obtained.
[0085] Example Four:
[0086] Figure 4 The structure of the depth estimation network training device provided in the fourth embodiment of the present application is shown, only the parts related to the embodiments of the present application are shown for the convenience of description, and the structure includes:
[0087] The first training unit 41 is configured to input the human body RGB image set and the 17 joint coordinates of each RGB image into the first stacked hourglass network as inputs of the first stacked hourglass network, train the first stacked hourglass network, and obtain the trained first stacked hourglass network.
[0088] The image acquisition unit 42 is configured to acquire the continuous video frame images and the normal maps of the continuous video frame images used for the current training.
[0089] The first error acquisition unit 43 is configured to input the continuous video frame images and the normal maps into the trained first stacked hourglass network and the pre-constructed depth estimation network, acquire the outputs of the first stacked hourglass network and the depth estimation network, acquire the current estimation error of the depth estimation network according to the outputs of the first stacked hourglass network and the depth estimation network, and the depth estimation network includes the second stacked hourglass network and the cLSTM network; and
[0090] The second training unit 44 is configured to, when the current estimation error does not meet the first preset condition, update the network parameters of the depth estimation network according to the current estimation error, trigger the image acquisition unit 42 to acquire the continuous video frame images and the normal maps of the continuous video frame images used for the current training, and continue to train the depth estimation network until the first preset condition is met to obtain the trained depth estimation network.
[0091] Specifically, the first training unit 41 can include:
[0092] a training data acquisition unit configured to acquire, from the human RGB image set, a human RGB image for current training and 17 joint node coordinates;
[0093] a depth value acquisition unit configured to acquire, by using the human RGB image and the 17 joint node coordinates as inputs of the first stacked hourglass network, depth values of joint nodes corresponding to the 17 joint node coordinates by using the stacked hourglass network;
[0094] a second error acquisition unit configured to acquire, by using the first loss function, a current estimation error of the first stacked hourglass network according to the acquired depth values and a preset target value;
[0095] a third training unit configured to, when the current estimation error of the first stacked hourglass network does not satisfy the second preset condition, update network parameters of the depth estimation network according to the current estimation error of the first stacked hourglass network, and trigger the training data acquisition unit to acquire, from the human RGB image set, the human RGB image for current training and the 17 joint node coordinates, so as to continue training the first stacked hourglass network until the second preset condition is satisfied, to obtain the trained first stacked hourglass network.
[0096] In the embodiment of the present application, each unit of the depth estimation network training apparatus can be realized by a corresponding hardware or software unit, and each unit can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which does not limit the present application. The specific embodiment of each unit can refer to the description of the first and second embodiments, which will not be repeated here.
[0097] Example Five:
[0098] Figure 5 The structure of the computing device provided in the fourth embodiment of the present application is shown, and only the parts related to the embodiments of the present application are shown for the convenience of description.
[0099] The computing device 5 of the embodiment of the present application includes a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. When the processor 50 executes the computer program 52, the steps in the above-mentioned depth estimation network training method and the human body image depth estimation method based on video embodiment are implemented, for example Figure 1 the steps S101 to S106 shown. Alternatively, when the processor 50 executes the computer program 52, the functions of each unit in the above-mentioned depth estimation network training apparatus embodiment are implemented, for example Figure 5 the functions of the units 41 to 44 shown.
[0100] The embodiment of the present application trains the first stacked hourglass network by using a human RGB image set and 17 joint coordinates of each RGB image, inputs the trained first stacked hourglass network and the depth estimation network including the second stacked hourglass network and the cLSTM network with a continuous video frame image and a normal map, obtains the output of the first stacked hourglass network and the depth estimation network, obtains the current estimation error of the depth estimation network according to the output of the first stacked hourglass network and the depth estimation network, updates the network parameters of the depth estimation network according to the current estimation error when the current estimation error does not meet the first preset condition, continues to train the depth estimation network until the trained depth estimation network is obtained, thereby obtaining the space-time features of the continuous video frame image through the cLSTM network, obtaining more features of each frame image, improving the accuracy of depth estimation of the depth estimation network, and correcting the distortion between body parts in the depth estimation process through the cooperation of the first stacked hourglass network.
[0101] The computing device of the embodiment of the present application can be a personal computer or a server. The steps implemented when the processor 50 in the computing device 5 executes the computer program 52 to implement the depth estimation network training method and the human body image depth estimation method based on video can refer to the description of the foregoing method embodiments, and will not be described here.
[0102] Example Five:
[0103] In the embodiment of the present application, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps in the above depth estimation network training method and the human body image depth estimation method based on video are implemented, for example, Figure 1 the steps S101 to S106 shown. Alternatively, when the computer program is executed by the processor, the functions of the units in the above apparatus embodiment are implemented, for example Figure 4 the functions of the units 41 to 44 shown.
[0104] The computer readable storage medium of the embodiment of the present application can include any entity or device capable of carrying computer program code, recording medium, such as ROM / RAM, magnetic disk, optical disk, flash memory, etc.
[0105] The above only describes the preferred embodiments of the present application and should not be used to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims
1. A method for training a depth estimation network, the method comprising: The method comprises the following steps: The human body RGB image set and the 17 joint node coordinates of each RGB image are input into a first stacked hourglass network, the first stacked hourglass network is trained, and a trained first stacked hourglass network is obtained; Continuous video frame images used for current training and normal maps of the continuous video frame images are obtained; The continuous video frame images and the normal maps are input into the trained first stacked hourglass network and a pre-constructed depth estimation network, the outputs of the first stacked hourglass network and the depth estimation network are obtained, the current estimation error of the depth estimation network is obtained according to the outputs of the first stacked hourglass network and the depth estimation network, and the depth estimation network comprises a second stacked hourglass network and a cLSTM network; When the current estimation error does not meet a first preset condition, the network parameters of the depth estimation network are updated according to the current estimation error, and the step of obtaining the continuous video frame images used for current training and the normal maps of the continuous video frame images is jumped to, so that the training of the depth estimation network is continued until the first preset condition is met, so that a trained depth estimation network is obtained; The step of inputting the continuous video frame images and the normal maps into the trained first stacked hourglass network and the pre-constructed depth estimation network, obtaining the outputs of the first stacked hourglass network and the depth estimation network, and obtaining the current estimation error of the depth estimation network according to the outputs of the first stacked hourglass network and the depth estimation network comprises: The continuous video frame images and the normal maps are input into the second stacked hourglass network, and spatial feature maps of the continuous video frame images are obtained; The spatial feature maps of the continuous video frame images are input into the cLSTM network, first depth maps of the continuous video frame images are obtained, the normal maps of the first depth maps are obtained, each first depth map is warped, second depth maps of next frame images of each video frame image in the continuous video frame images are obtained, the continuous video frame images are input into the first stacked hourglass network, third depth maps of the continuous video frame images are obtained, and the current estimation error of the depth estimation network is obtained according to the first depth maps and the second depth maps, the first depth maps and the third depth maps, and the normal maps of the first depth maps and the normal maps of the continuous video frame images.
2. The method of claim 1, wherein, The step of inputting the human body RGB image set and the 17 joint node coordinates of each RGB image into the first stacked hourglass network and training the first stacked hourglass network comprises: A human body RGB image used for current training and 17 joint node coordinates are obtained from the human body RGB image set; The obtained human body RGB image and 17 joint node coordinates are input into the first stacked hourglass network, and depth values of joint nodes corresponding to the 17 joint node coordinates are obtained through the stacked hourglass network. According to the acquired depth value and the preset target value, a first loss function is used to acquire a current estimation error of the first stacked hourglass network; When the current estimation error of the first stacked hourglass network does not satisfy a second preset condition, the network parameters of the depth estimation network are updated according to the current estimation error of the first stacked hourglass network, and the step of acquiring the current training human body RGB image and the 17 joint coordinates from the human body RGB image set is jumped to, so as to continue training the first stacked hourglass network until the second preset condition is satisfied, so as to obtain the trained first stacked hourglass network.
3. The method of claim 1, wherein, The step of inputting the spatial feature map of the continuous video frame image into the cLSTM network to obtain the first depth map of the continuous video frame image comprises: The spatial feature maps of the current video frame image and the previous video frame image are input into the cLSTM network to obtain the first depth map of the current video frame image.
4. The method of claim 1, wherein, The step of acquiring the current estimation error of the depth estimation network according to the first depth map, the second depth map, the first depth map and the third depth map, and the normal map of the first depth map and the normal map of the continuous video frame image comprises: According to the first depth map and the second depth map, a second loss function is used to acquire a first estimation error of the depth estimation network; According to the first depth map and the third depth map, a third loss function is used to acquire a second estimation error of the depth estimation network; According to the normal map of the first depth map and the normal map of the continuous video frame image, a fourth loss function is used to acquire a third estimation error of the depth estimation network; The first estimation error, the second estimation error and the third estimation error are determined as the current estimation error of the depth estimation network.
5. A method for depth estimation of human body image based on video, characterized in that, The method comprises the following steps: A preset number of continuous video frame images are sequentially acquired from an input video file, and a normal map of the continuous video frame image is acquired; The continuous video frame image and the normal map of the continuous video frame image are input into the depth estimation network trained by the depth estimation network training method of any one of claims 1-4 to obtain a depth map of the continuous video frame image.
6. A depth estimation network training apparatus, comprising: It comprises: A first training unit is configured to train the first stacked hourglass network by taking the human body RGB image set and the 17 joint coordinates of each RGB image as inputs of the first stacked hourglass network, and obtain a trained first stacked hourglass network; An image acquisition unit is configured to acquire a continuous video frame image for current training and a normal map of the continuous video frame image; An image acquisition unit is configured to acquire a continuous video frame image for current training and a normal map of the continuous video frame image; a first error obtaining unit, configured to input the continuous video frame image and the normal map into the trained first stacked Hourglass Network and a pre-constructed depth estimation network, obtain outputs of the first stacked Hourglass Network and the depth estimation network, and obtain a current estimation error of the depth estimation network according to the outputs of the first stacked Hourglass Network and the depth estimation network, the depth estimation network comprising a second stacked Hourglass Network and a cLSTM network; and a second training unit, configured to, when the current estimation error does not meet a first preset condition, update network parameters of the depth estimation network according to the current estimation error, trigger the image obtaining unit to obtain a continuous video frame image and a normal map of the continuous video frame image for current training, and continue training the depth estimation network until the first preset condition is met, so as to obtain the trained depth estimation network; the first error obtaining unit, when inputting the continuous video frame image and the normal map into the trained first stacked Hourglass Network and the pre-constructed depth estimation network, obtaining the outputs of the first stacked Hourglass Network and the depth estimation network, and obtaining the current estimation error of the depth estimation network according to the outputs of the first stacked Hourglass Network and the depth estimation network, comprises: inputting the continuous video frame image and the normal map into the second stacked Hourglass Network to obtain a spatial feature map of the continuous video frame image; inputting the spatial feature map of the continuous video frame image into the cLSTM network to obtain a first depth map of the continuous video frame image, obtaining a normal map of the first depth map, and warping each first depth map to obtain a second depth map of a next frame image of each video frame image in the continuous video frame image; inputting the continuous video frame image into the first stacked Hourglass Network to obtain a third depth map of the continuous video frame image; and obtaining the current estimation error of the depth estimation network according to the first depth map and the second depth map, the first depth map and the third depth map, and the normal map of the first depth map and the normal map of the continuous video frame image.
7. The apparatus of claim 6, wherein, the first training unit comprises: a training data obtaining unit, configured to obtain a human body RGB image and 17 joint coordinates for current training from the human body RGB image set; a depth value obtaining unit, configured to take the obtained human body RGB image and 17 joint coordinates as inputs of the first stacked Hourglass Network, and obtain depth values of joints corresponding to the 17 joint coordinates by the stacked Hourglass Network; a second error obtaining unit, configured to obtain a current estimation error of the first stacked Hourglass Network by using a first loss function according to the obtained depth values and a preset target value; and a second training unit, configured to, when the current estimation error does not meet a first preset condition, update network parameters of the depth estimation network according to the current estimation error, trigger the image obtaining unit to obtain a continuous video frame image and a normal map of the continuous video frame image for current training, and continue training the depth estimation network until the first preset condition is met, so as to obtain the trained depth estimation network; the first error obtaining unit, when inputting the continuous video frame image and the normal map into the trained first stacked Hourglass Network and the pre-constructed depth estimation network, obtaining the outputs of the first stacked Hourglass Network and the depth estimation network, and obtaining the current estimation error of the depth estimation network according to the outputs of the first stacked Hourglass Network and the depth estimation network, comprises: inputting the continuous video frame image and the normal map into the second stacked Hourglass Network to obtain a spatial feature map of the continuous video frame image; inputting the spatial feature map of the continuous video frame image into the cLSTM network to obtain a first depth map of the continuous video frame image, obtaining a normal map of the first depth map, and warping each first depth map to obtain a second depth map of a next frame image of each video frame image in the continuous video frame image; inputting the continuous video frame image into the first stacked Hourglass Network to obtain a third depth map of the continuous video frame image; and obtaining the current estimation error of the depth estimation network according to the first depth map and the second depth map, the first depth map and the third depth map, and the normal map of the first depth map and the normal map of the continuous video frame image. the first training unit comprises: a training data obtaining unit, configured to obtain a human body RGB image and 17 joint coordinates for current training from the human body RGB image set; a depth value obtaining unit, configured to take the obtained human body RGB image and 17 joint coordinates as inputs of the first stacked Hourglass Network, and obtain depth values of joints corresponding to the 17 joint coordinates by the stacked Hourglass Network; a second error obtaining unit, configured to obtain a current estimation error of the first stacked Hourglass Network by using a first loss function according to the obtained depth values and a preset target value; and A third training unit is configured to update the network parameters of the deep estimation network according to the current estimation error of the first stacked hourglass network when the current estimation error of the first stacked hourglass network does not satisfy the second preset condition, and trigger the training data acquisition unit to acquire the current training human RGB image and the 17 joint coordinates from the human RGB image set to continue training the first stacked hourglass network until the second preset condition is satisfied, so as to obtain the trained first stacked hourglass network.
8. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 5.
9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 5.