Method, apparatus and device for estimating depth based on object motion, and medium

Abstract

The application discloses a depth estimation method and device based on object motion, equipment and medium, wherein the method comprises the following steps: determining an i-1th initial depth map of an i-1th initial image according to a single-frame depth estimation model; determining object motion data of each object mask corresponding to the i-1th initial image and the i-1th initial image according to an object motion estimation model; adjusting the position of the object in the i-1th initial image according to the object motion data, the i-1th initial depth map and the object mask corresponding to the i-1th initial image, and obtaining an i-1th adjusted image; inputting the i-1th initial image and the i-1th adjusted image into a time-series multi-view depth estimation model to perform depth estimation, and obtaining an i-1th target depth map. Thus, the scene in the two images input into the time-series multi-view depth estimation model is consistent, and the accuracy of the i-1th target depth map is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and medium for depth estimation based on object motion. Background Technology

[0002] In the field of mobile robot visual perception, accurate estimation of scene depth is crucial for enabling mobile robots to perceive moving objects in a scene in real time, facilitating path planning and control. Existing temporal multi-frame depth estimation methods are affected by moving objects in the scene; that is, moving objects can cause two frames captured by a moving camera to depict different scenes. When objects in an image are moving, the scene changes between the two frames, leading to errors in object depth estimation, and consequently, scene depth estimation. For example, when an object moving in the same direction and at a similar speed to the camera appears in front of it, the model tends to predict the object's depth as infinity. This is because the process of recovering depth from a moving image assumes the scene is static, while moving objects do not change much in the image; only objects at infinity remain relatively unchanged in moving images. Summary of the Invention

[0003] The main objective of this application is to provide a depth estimation method, apparatus, device, and medium based on object motion, aiming to solve the technical problem that when existing time-series multi-frame depth estimation methods estimate scene depth, the scene changes between two frames due to the motion of objects in the image, leading to errors in object depth estimation and consequently, errors in scene depth estimation.

[0004] To achieve the aforementioned objectives, this application proposes a depth estimation method based on object motion, the method comprising:

[0005] Obtain the initial image of the i-th frame and the object masks corresponding to the initial image of the i-th frame;

[0006] Based on the preset single-frame depth estimation model, depth estimation is performed on each pixel of the acquired initial image of the (i-1)th frame to obtain the initial depth map of the (i-1)th frame.

[0007] According to the preset object motion estimation model, object motion estimation is performed on each of the object masks corresponding to the initial image of the i-th frame and each of the object masks corresponding to the initial image of the (i-1)-th frame to obtain object motion data;

[0008] Based on the motion data of each object, the initial depth map of the (i-1)th frame and the object masks corresponding to the initial image of the (i-1)th frame, the positions of the objects in the initial image of the (i-1)th frame are adjusted to obtain the adjusted image of the (i-1)th frame.

[0009] The initial image of the i-th frame and the adjusted image of the (i-1)-th frame are input into a preset temporal multi-view depth estimation model to perform depth estimation, thereby obtaining the target depth map of the i-th frame.

[0010] Further, before the step of performing depth estimation for each pixel of the acquired initial image of the (i-1)th frame according to a preset single-frame depth estimation model to obtain the initial depth map of the (i-1)th frame, the following steps are included:

[0011] Acquire multiple first training samples, a first counter, and a second counter, wherein each first training sample is a temporally adjacent image, the first counter is initialized to 2, and the second counter is initialized to 1;

[0012] The first training sample corresponding to the first counter is obtained from each of the first training samples as the first target sample, and the first training sample corresponding to the second counter is obtained from each of the first training samples as the reference sample;

[0013] The first target sample is input into the depth estimation module of the preset first model to estimate the depth of each pixel to obtain a first depth map. According to the camera pose transformation estimation module of the first model, the camera pose transformation data of the reference sample and the first target sample is predicted. According to the camera pose transformation data and the first depth map, the reference sample is projected onto the viewpoint corresponding to the first target sample to obtain a projected image.

[0014] Based on the structural similarity loss function, the loss function is calculated according to the projected image and the first target sample to obtain the first loss value. The network parameters of the first model are updated according to the first loss value. The first counter and the second counter are both incremented by 1. The step of obtaining the first training sample corresponding to the first counter from each of the first training samples as the first target sample is repeated until the preset first training termination condition is reached.

[0015] The depth estimation module of the first model is used as the single-frame depth estimation model.

[0016] Further, the step of performing object motion estimation on each object mask corresponding to the initial image of the i-th frame and each object mask corresponding to the initial image of the (i-1)-th frame according to a preset object motion estimation model to obtain object motion data includes:

[0017] Based on the IOU calculation method, each object mask corresponding to the initial image of the i-th frame and each object mask corresponding to the initial image of the (i-1)-th frame are associated to obtain the common object mask set of the (i-1)-th frame and the common object mask set of the i-th frame.

[0018] Based on the common object mask set of the i-th frame, non-object regions are filtered out from the initial image of the i-th frame to obtain a first retained image. Based on the common object mask set of the (i-1)-th frame, non-object regions are filtered out from the initial image of the (i-1)-th frame to obtain a second retained image.

[0019] The first preserved image and the second preserved image are stitched together along the channel dimension to obtain a stitched image;

[0020] The stitched image is input into the object motion estimation model to estimate the object motion, thereby obtaining the motion data of each object.

[0021] Furthermore, before the step of inputting the stitched image into the object motion estimation model to perform object motion estimation and obtain the motion data of each object, the method further includes:

[0022] Obtain multiple second training samples, wherein the second training samples include: two temporally adjacent image frames;

[0023] Obtain any one of the second training samples as the second target sample from each of the second training samples;

[0024] Two image samples from the second target sample are input into a preset second model to perform head frame depth map estimation and object pose transformation data estimation, and then synthesize a composite image of the object region from the tail frame perspective based on the head frame depth map and the object pose transformation data.

[0025] Based on the structural similarity loss function, the loss value is calculated according to the object region images corresponding to the two frames of image samples in the second target sample, and the synthesis loss value is obtained.

[0026] Two image samples from the second target sample are input into the second model to estimate the object motion and obtain the object motion estimate.

[0027] Based on the L1 loss function, the loss value is calculated according to the estimated object motion value and the pseudo-label of the object motion corresponding to the second target sample to obtain the motion loss value.

[0028] Based on the synthetic loss value and the motion loss value, update the network parameters of the second model, and repeat the step of obtaining any one of the second training samples from each of the second training samples as the second target sample until the preset second training termination condition is met, and use the second model as the object motion estimation model.

[0029] The object motion pseudo-label corresponding to the second target sample is obtained from the object pose change data based on two image samples in the second target sample, the camera intrinsic parameters corresponding to the second target sample, and the ICP point cloud registration algorithm.

[0030] Further, the step of adjusting the position of the objects in the initial image of the (i-1)th frame based on the motion data of each object, the initial depth map of the (i-1)th frame, and the object masks corresponding to the initial image of the (i-1)th frame, to obtain the adjusted image of the (i-1)th frame, includes:

[0031] Based on the initial depth map of the (i-1)th frame and the camera intrinsic parameters corresponding to the initial image of the i-th frame, point cloud mapping is performed on the initial image of the (i-1)th frame to obtain the initial point cloud;

[0032] Based on the object masks corresponding to the initial image of the (i-1)th frame, the object point cloud is located in the initial point cloud to obtain the first object point cloud;

[0033] Based on the motion data of each object, pose transformation is performed on each of the first object point clouds to obtain the second object point cloud.

[0034] The initial point cloud is updated based on the point clouds of each of the second objects to obtain the target point cloud;

[0035] Based on the camera intrinsic parameters corresponding to the initial image of the i-th frame, image projection processing is performed on the target point cloud to obtain the adjusted image of the (i-1)-th frame.

[0036] Further, before the step of inputting the initial image of the i-th frame and the adjusted image of the (i-1)-th frame into a preset temporal multi-view depth estimation model for depth estimation to obtain the target depth map of the i-th frame, the method further includes:

[0037] Obtain multiple third-party training samples;

[0038] Obtain any one of the third training samples as the third target sample from each of the third training samples;

[0039] The two image samples from the third target sample are respectively input into the backbone network of the preset third model for feature extraction to obtain the first frame feature and the last frame feature.

[0040] The first frame features and the last frame features are input into the pose transformation module of the third model for feature concatenation and pose transformation prediction in sequence to obtain the pose transformation prediction value.

[0041] Using the depth estimation module of the third model, a cost body, encoding, and tail frame depth decoding are sequentially constructed from the head frame features and the tail frame features to obtain the tail frame depth map;

[0042] The image synthesis module of the third model is used to synthesize the pose transformation prediction value and the tail frame depth map from the tail frame perspective to obtain the tail frame synthesized image.

[0043] Based on the photometric consistency loss, the loss value is calculated according to the last frame image samples of the synthetic last frame image and the last frame image samples of the two frames of image samples in the third target sample, and the second loss value is obtained.

[0044] The network parameters of the third model are updated according to the second loss value. The step of obtaining any one of the third training samples from each of the third training samples as the third target sample is repeated until the preset third training termination condition is reached, and the third model is used as a temporal multi-view depth estimation model.

[0045] Further, before the step of obtaining the initial image of the i-th frame and the object masks corresponding to the initial image of the i-th frame, the following steps are included:

[0046] A preset instance segmentation method is used to perform instance segmentation on the i-th initial image to obtain the object mask corresponding to each object in the i-th initial image.

[0047] This application also proposes a depth estimation device based on object motion, the device comprising:

[0048] The data acquisition module is used to acquire the initial image of the i-th frame and the object masks corresponding to the initial image of the i-th frame;

[0049] The (i-1)th frame initial depth map determination module is used to perform depth estimation on each pixel of the acquired (i-1)th frame initial image according to a preset single-frame depth estimation model, so as to obtain the (i-1)th frame initial depth map.

[0050] The object motion data determination module is used to perform object motion estimation on each of the object masks corresponding to the initial image of the i-th frame and each of the object masks corresponding to the initial image of the (i-1)-th frame according to a preset object motion estimation model, so as to obtain the motion data of each object.

[0051] The i-1th frame adjusted image determination module is used to adjust the position of the objects in the i-1th frame initial image according to the motion data of each object, the initial depth map of the i-1th frame and the object mask corresponding to the initial image of the i-1th frame, so as to obtain the i-1th frame adjusted image.

[0052] The i-th frame target depth map determination module is used to input the initial image of the i-th frame and the adjusted image of the (i-1)-th frame into a preset temporal multi-view depth estimation model to perform depth estimation and obtain the i-th frame target depth map.

[0053] This application also proposes a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0054] This application also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0055] This application discloses a depth estimation method, apparatus, device, and medium based on object motion. The method involves: estimating the depth of each pixel in an acquired initial image of frame (i-1) according to a preset single-frame depth estimation model, resulting in an initial depth map of frame (i-1); estimating the motion of each object mask corresponding to the initial image of frame (i) and the initial image of frame (i-1) according to a preset object motion estimation model, resulting in object motion data; adjusting the positions of objects in the initial image of frame (i-1) based on the object motion data, the initial depth map of frame (i-1), and the object masks corresponding to the initial image of frame (i-1), resulting in an adjusted image of frame (i-1); and inputting the initial image of frame (i) and the adjusted image of frame (i-1) into a preset temporal multi-view depth estimation model for depth estimation, resulting in a target depth map of frame (i). By estimating object motion using object masks based on two initial images, instance-level 3D motion estimation for each object is introduced. By adjusting the object's position based on the motion data, the positional changes of the object in the two initial images in the world coordinate system are eliminated based on the motion estimation results. This ensures that the scene in the two images input to the temporal multi-view depth estimation model remains consistent, avoiding the technical problem that changes in the scene between the two images due to the motion of the object itself, which leads to incorrect object depth estimation and consequently incorrect scene depth estimation. This improves the accuracy of the target depth map in the i-th frame. Attached Figure Description

[0056] Figure 1This is a flowchart illustrating a depth estimation method based on object motion according to an embodiment of this application;

[0057] Figure 2 This is a schematic block diagram of a depth estimation device based on object motion according to an embodiment of this application;

[0058] Figure 3 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application.

[0059] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] In the field of mobile robot visual perception, to enable robots to perceive moving objects in a scene in real time, facilitating path planning and control, this application applies temporal multi-frame motion object estimation to multi-frame depth estimation, obtaining more accurate scene depth while acquiring scene object motion information.

[0062] This scheme mainly consists of two parts. The first part is temporal object motion estimation, which requires two temporally consecutive images and the corresponding object localization within those images as input. This localization can be a two-dimensional instance mask of the object (i.e., an object mask). The second part is multi-frame depth estimation. In monocular depth estimation, temporal images contain some geometric information. Utilizing two temporally adjacent images can yield more accurate depth estimation results. The input is the same two temporally consecutive images as the object motion estimation part. Interactive modeling of the two images can be performed by constructing a cost volume. This method of estimating depth from moving images is affected by moving objects in the scene; that is, moving objects in the scene can cause two images captured during motion to not represent the same scene. However, the result of estimating depth from temporally consecutive moving objects can eliminate object motion in the image. Therefore, when interacting with the adjusted two temporally consecutive images, more accurate geometric information can be extracted, leading to a more accurate depth prediction.

[0063] This scheme fully utilizes information from two time-series images to estimate the depth of the scene and the motion of objects in the scene. It also allows the object motion estimation task to complement the depth estimation task, alleviating the problem of scene inconsistency caused by object motion in the two frames. Furthermore, it can simultaneously output the depth of the image and the 3D motion state of each object in the image end-to-end.

[0064] Reference Figure 1 This application provides a depth estimation method based on object motion, the method comprising:

[0065] S1: Obtain the initial image of the i-th frame and the object masks corresponding to the initial image of the i-th frame;

[0066] Specifically, the initial image of the i-th frame input by the user and the object masks corresponding to the initial image of the i-th frame can be obtained, or the initial image of the i-th frame and the object masks corresponding to the initial image of the i-th frame can be obtained from the storage space, or the initial image of the i-th frame and the object masks corresponding to the initial image of the i-th frame can be obtained from a third-party application.

[0067] The initial image of the i-th frame is the image captured by the camera in the i-th frame. i is an integer greater than 1.

[0068] The object mask corresponding to the initial image of the i-th frame is the mask of each object in the initial image of the i-th frame.

[0069] S2: Based on the preset single-frame depth estimation model, perform depth estimation on each pixel of the acquired initial image of the (i-1)th frame to obtain the initial depth map of the (i-1)th frame.

[0070] The initial image of frame i-1 is the image of frame i-1 captured by the camera.

[0071] Specifically, the initial image of frame i-1 is input into the single-frame depth estimation model to estimate the depth of each pixel, thus obtaining the initial depth map of frame i-1.

[0072] In other words, the initial depth map of frame i-1 is the depth map corresponding to the initial image of frame i-1, and the size of the initial depth map of frame i-1 is the same as that of the initial image of frame i-1.

[0073] The single-frame depth estimation model is a model trained based on a monocular self-supervised depth estimation method.

[0074] The monocular self-supervised depth estimation method predicts the depth of a given image and the pose transformation between two frames, projects it into the viewpoint of another frame, and then trains the model by minimizing the image reconstruction error. The image reconstruction error can be calculated using the structural similarity loss function (SSIM). Here, the two frames refer to two temporally adjacent frames.

[0075] S3: Based on the preset object motion estimation model, perform object motion estimation on each of the object masks corresponding to the initial image of the i-th frame and each of the object masks corresponding to the initial image of the (i-1)-th frame to obtain object motion data;

[0076] Existing object motion estimation methods estimate object motion using two temporally adjacent frames. There are two approaches: one approach uses two temporally adjacent frames as input and a neural network to estimate the scene flow of the entire scene, which is the movement of each pixel in the image in three directions at its corresponding 3D spatial position, resulting in an h*w*3 image; the other approach uses two mask regions corresponding to the same object in two temporally adjacent frames as input and estimates the motion of each object separately. The motion estimation is to handle the problem of moving objects during the synthesis of new perspectives. Even if the depth estimation of the first frame (head frame) and the pose estimation between the two temporally adjacent frames are completely accurate, the movement of objects in the two temporally adjacent frames will cause inconsistencies between the synthesized tail frame from the head frame and the actual tail frame in the region of the moving object.

[0077] Specifically, image association is performed between each object mask corresponding to the initial image of the i-th frame and each object mask corresponding to the initial image of the (i-1)-th frame. The associated images are then input into the object motion estimation model to estimate the object motion, obtaining the 6-DOF motion data of each object in three-dimensional space. This data is used as the object motion data. The object motion data is the 6-DOF motion data of an object in three-dimensional space.

[0078] The motion data of the object is represented as (tx,ty,tz,rx,ry,rz), ​​where tx, ty, and tz represent translations along the x, y, and z directions, respectively, and rx, ry, and rz represent Euler angles about the x, y, and z axes, respectively.

[0079] An object motion estimation model is used to estimate the 6-DOF motion of each object in a correlated image in 3D space.

[0080] An object motion estimation model can be a convolutional neural network, which includes a feature encoding structure and a decoding structure.

[0081] S4: Based on the motion data of each object, the initial depth map of the (i-1)th frame and the object masks corresponding to the initial image of the (i-1)th frame, adjust the position of the objects in the initial image of the (i-1)th frame to obtain the adjusted image of the (i-1)th frame.

[0082] Specifically, based on the object motion data output by the object motion estimation network, the initial depth map of the (i-1)th frame estimated by the single-frame depth estimation model for the initial image of the (i-1)th frame, and the object masks corresponding to the initial image of the (i-1)th frame, the object positions in the initial image of the (i-1)th frame are adjusted so that the positions of the objects in the adjusted initial image of the (i-1)th frame and the initial image of the i-th frame are consistent in the world coordinate system, thereby eliminating the influence of object motion on the scene, and the image with the object motion eliminated is used as the adjusted image of the (i-1)th frame.

[0083] Understandably, existing methods generally filter out moving objects. This application achieves the same goal of eliminating the adverse effects of moving objects on depth estimation by adjusting the position of the objects.

[0084] S5: Input the initial image of the i-th frame and the adjusted image of the (i-1)-th frame into a preset temporal multi-view depth estimation model to perform depth estimation and obtain the target depth map of the i-th frame.

[0085] Specifically, by inputting the initial image of the i-th frame and the adjusted image of the (i-1)-th frame into a preset temporal multi-view depth estimation model for depth estimation, the ability to enhance the expressive power of features by utilizing temporal information is realized, thereby obtaining a more accurate depth estimation result compared to the single-frame depth estimation model.

[0086] The temporal multi-view depth estimation model is a model trained based on the temporal multi-frame depth estimation method.

[0087] The main difference between temporal multi-frame depth estimation methods and monocular self-supervised depth estimation methods is that the depth is predicted by using two temporally adjacent frames of input images. These two frames are taken at a very small time interval during the vehicle's motion, hence the term "depth recovery from motion". In order to utilize the information of the two input frames, a cost volume is generally used to interact the information of the two temporally adjacent frames. Then, the cost volume is encoded and decoded to obtain the estimated depth.

[0088] Since the model calculates the loss using two moving frames, when the object in the image is moving, the scene changes between the two frames, leading to incorrect depth prediction. For example, when an object appears in front of the camera that moves in the same direction and at a similar speed to the camera, the model tends to predict the object's depth as infinity. This is because the process of recovering depth from a moving image assumes that the scene is static, and such moving objects do not change much in the image. Only objects at infinity remain relatively unchanged in moving images.

[0089] Compared to existing technologies, this application fully utilizes the two-frame information from multiple temporal perspectives, while decoupling the moving object from the background region. For the background region (non-object region), depth can be better estimated using two temporally adjacent frames; for the moving object, its motion is directly estimated using two temporally adjacent frames and their 2D localization. The estimated object motion cancels out the differences caused by the object's motion in the two temporally adjacent frames, and then the depth of the moving object region is estimated using the two temporally adjacent frames. Each model in this application is a multi-task learning process, simultaneously completing depth estimation and object motion estimation tasks. Furthermore, the motion estimation task assists the temporal depth estimation task, unifying the two tasks within a single framework. Both tasks are crucial components of mobile robot perception.

[0090] This embodiment achieves object motion estimation by using an object mask based on two initial images, thus introducing instance-level 3D motion estimation for each object. By adjusting the object's position based on the motion data, the positional changes of the object in the two initial images in the world coordinate system are eliminated based on the object motion estimation results. This ensures that the scene in the two frames of the input temporal multi-view depth estimation model remains consistent, avoiding the technical problem that changes in the scene between the two frames when the object itself moves, leading to incorrect object depth estimation and consequently incorrect scene depth estimation. This improves the accuracy of the target depth map in the i-th frame.

[0091] In one embodiment, before the step of performing depth estimation for each pixel of the acquired initial image of the (i-1)th frame according to a preset single-frame depth estimation model to obtain the initial depth map of the (i-1)th frame, the following steps are included:

[0092] S21: Obtain multiple first training samples, a first counter, and a second counter, wherein each of the first training samples is a temporally adjacent image, initialize the first counter to 2, and initialize the second counter to 1;

[0093] Specifically, it can obtain multiple first training samples, first counters, and second counters input by the user, or obtain multiple first training samples, first counters, and second counters from storage space, or obtain multiple first training samples, first counters, and second counters from third-party applications.

[0094] Each of the first training samples is a temporally adjacent image; that is, two adjacent first training samples are two temporally adjacent frames.

[0095] S22: Obtain the first training sample whose order corresponds to the first counter from each of the first training samples as the first target sample, and obtain the first training sample whose order corresponds to the second counter from each of the first training samples as the reference sample;

[0096] Specifically, the first training sample corresponding to the first counter is obtained from each of the first training samples as the first target sample, and the first training sample corresponding to the second counter is obtained from each of the first training samples as the reference sample, so that the reference sample and the first target sample are two temporally adjacent images, the reference sample is the first frame of the two temporally adjacent images, and the first target sample is the second frame of the two temporally adjacent images.

[0097] S23: Input the first target sample into the depth estimation module of the preset first model to perform depth estimation for each pixel to obtain a first depth map. According to the camera pose transformation estimation module of the first model, perform camera pose transformation data prediction on the reference sample and the first target sample. According to the camera pose transformation data and the first depth map, project the reference sample onto the viewpoint corresponding to the first target sample to obtain a projected image.

[0098] Specifically, the first target sample is input into the depth estimation module of the preset first model to perform depth estimation for each pixel to obtain a first depth map. That is, the pixel value of each pixel in the first depth map is a depth value, and the pixels in the first depth map correspond one-to-one with the pixels in the first target sample.

[0099] The camera pose transformation estimation module is used to predict the pose transformation between two temporally adjacent images. This module is based on a model trained using a neural network.

[0100] Specifically, based on the camera pose transformation data and the first depth map, the reference sample is projected onto a viewpoint corresponding to the first target sample, and the resulting image is used as the projected image. In other words, the projected image is the image from the viewpoint corresponding to the first target sample.

[0101] S24: Based on the structural similarity loss function, calculate the loss function according to the projected image and the first target sample to obtain the first loss value, update the network parameters of the first model according to the first loss value, increment the first counter and the second counter by 1, and repeat the step of obtaining the first training sample with the sorting corresponding to the first counter from each of the first training samples as the first target sample until the preset first training end condition is reached.

[0102] Specifically, the method of calculating a first loss value based on the structural similarity loss function, using the projected image and the first target sample, and updating the network parameters of the first model based on the first loss value, can be selected from existing technologies and will not be elaborated here.

[0103] Increment both the first counter and the second counter by 1; that is, increment the first counter by 1 and increment the second counter by 1.

[0104] Specifically, the step of repeatedly executing the step of obtaining the first training sample from each of the first training samples and sorting it according to the first counter as the first target sample is to repeatedly execute steps S22 to S24 until the preset first training end condition is reached.

[0105] Optionally, the first training termination condition is that the first loss value converges to a preset first value.

[0106] S25: Use the depth estimation module of the first model as the single-frame depth estimation model.

[0107] Specifically, the depth estimation module of the first model that has reached the first training termination condition is used as the single-frame depth estimation model.

[0108] In this embodiment, a limited sequence of images is used as input to train the first model. The image projected from the predicted first depth map and the pose transformation between two temporally adjacent images is used as the label for self-supervised training, which reduces the training cost.

[0109] In one embodiment, the step of performing object motion estimation on each object mask corresponding to the initial image of the i-th frame and each object mask corresponding to the initial image of the (i-1)-th frame, based on a preset object motion estimation model, to obtain object motion data, includes:

[0110] S31: Based on the IOU calculation method, associate each object mask corresponding to the initial image of the i-th frame with each object mask corresponding to the initial image of the (i-1)-th frame to obtain the common object mask set of the (i-1)-th frame and the common object mask set of the i-th frame;

[0111] The IOU (Intersection over Union) calculation method is a method for calculating the overlap rate.

[0112] Specifically, based on the IOU calculation method, the object masks corresponding to the initial image of the i-th frame and the object masks corresponding to the initial image of the (i-1)-th frame are associated to realize the association of objects. After the association is completed, the objects that appear in both frames (also called common objects) and the object masks of the common objects in the two frames are obtained. The object masks of the common objects in the two frames are the common object mask set of the (i-1)-th frame and the common object mask set of the i-th frame.

[0113] S32: Based on the common object mask set of the i-th frame, filter out non-object regions in the initial image of the i-th frame to obtain a first retained image; based on the common object mask set of the (i-1)-th frame, filter out non-object regions in the initial image of the (i-1)-th frame to obtain a second retained image.

[0114] Specifically, based on the common object mask set of the i-th frame, non-object region filtering is performed on the initial image of the i-th frame, that is, filtering out the regions in the initial image of the i-th frame other than common objects, and the retained image region is used as the first retained image; based on the common object mask set of the (i-1)-th frame, non-object region filtering is performed on the initial image of the (i-1)-th frame, that is, filtering out the regions in the initial image of the (i-1)-th frame other than common objects, and the retained image region is used as the second retained image.

[0115] S33: Perform channel-level stitching on the first retained image and the second retained image to obtain a stitched image;

[0116] Specifically, the first preserved image and the second preserved image are stitched together along the channel dimension, and the stitched data is used as the stitched image.

[0117] S34: Input the stitched image into the object motion estimation model to estimate the object motion and obtain the motion data of each object.

[0118] Specifically, the stitched image is input into the object motion estimation model to estimate the object motion. The object motion estimation model will output the motion data of each object, thereby obtaining the six degrees of freedom motion of the object.

[0119] This embodiment extracts images containing only the object from the initial image of the i-th frame and images containing only the object from the initial image of the (i-1)-th frame. Then, the extracted images are stitched together and input into the object motion estimation model to estimate the object motion, thereby accurately estimating the six degrees of freedom motion of the object. This provides a foundation for eliminating the influence of moving objects based on the six degrees of freedom motion of the object.

[0120] In one embodiment, before the step of inputting the stitched image into the object motion estimation model to perform object motion estimation and obtain the motion data of each object, the method further includes:

[0121] S341: Obtain multiple second training samples, wherein the second training samples include: two temporally adjacent image samples;

[0122] Specifically, multiple second training samples can be obtained from user input, multiple second training samples can be obtained from storage space, or multiple second training samples can be obtained from third-party applications.

[0123] An image sample is a single image.

[0124] S342: Obtain any one of the second training samples as the second target sample from each of the second training samples;

[0125] S343: Input the two image samples from the second target sample into the preset second model to perform head frame depth map estimation and object pose transformation data estimation respectively, and synthesize the object region composite image under the tail frame view based on the head frame depth map and the object pose transformation data.

[0126] Specifically, the two image samples from the second target sample are input into a preset second model to estimate the depth map of the first frame (i.e., the head frame) of the two image samples, and the pose transformation of the object in the two image samples is estimated. The estimated data is used as the object pose transformation data. Based on the head frame depth map and the object pose transformation data, an image of the object region in the second frame (i.e., the tail frame) of the two image samples is synthesized, and the synthesized image is used as the object region composite image.

[0127] S344: Based on the structural similarity loss function, the loss value is calculated according to the object region images corresponding to the two frames of image samples in the second target sample, and the synthesis loss value is obtained.

[0128] Specifically, the method of calculating the loss value based on the structural similarity loss function (i.e., the SSIM loss function) and the object region image corresponding to the tail frame of the two image samples in the second target sample can be selected from the existing technology and will not be elaborated here.

[0129] S345: Input two image samples from the second target sample into the second model to estimate the object motion and obtain the object motion estimate value;

[0130] Specifically, two frames of image samples from the second target sample are input into the second model to estimate the object motion between the two frames, and the estimated data is used as the object motion estimate.

[0131] S346: Based on the L1 loss function, the loss value is calculated according to the estimated object motion value and the pseudo-label of the object motion corresponding to the second target sample to obtain the motion loss value;

[0132] L1 loss, also known as Mean Absolute Error (MAE), is the average of the sum of the absolute differences between the actual and predicted values.

[0133] Specifically, the specific methods and steps for calculating the loss value based on the L1 loss function, according to the estimated value of the object motion and the pseudo-label of the object motion corresponding to the second target sample, can be selected from existing technologies and will not be elaborated here.

[0134] S347: Update the network parameters of the second model according to the synthetic loss value and the motion loss value, and repeat the step of obtaining any one of the second training samples from each of the second training samples as the second target sample until the preset second training termination condition is reached, and use the second model as the object motion estimation model.

[0135] The object motion pseudo-label corresponding to the second target sample is obtained from the object pose change data based on two image samples in the second target sample, the camera intrinsic parameters corresponding to the second target sample, and the ICP point cloud registration algorithm.

[0136] Specifically, the synthetic loss value and the motion loss value are weighted and summed to obtain a comprehensive loss value; the network parameters of the second model are updated based on the comprehensive loss value.

[0137] The specific methods and steps for updating the network parameters of the second model based on the comprehensive loss value can be selected from existing technologies and will not be elaborated here.

[0138] Repeat the step of obtaining any one of the second training samples from each of the second training samples as the second target sample, that is, repeat steps S342 to S347.

[0139] Optionally, the second training termination condition is that the combined loss value obtained based on the synthetic loss value and the motion loss value converges to a preset second value.

[0140] Scale supervision is achieved through the object motion pseudo-labels. Since there is a single-frame depth estimation model trained based on a monocular self-supervised depth estimation method, depth can be estimated for two adjacent frames to obtain depth maps. The object region of each depth map is mapped to a point cloud (i.e., an object point cloud) using camera intrinsic parameters. The object point clouds of two adjacent frames are transformed into the same coordinate system through pose transformation between the two frames. Then, the object motion predicted by depth estimation is obtained from the changes in the object point clouds of the two frames. The method for calculating the changes in the object point clouds of the two frames can be the ICP (Iterative Closest Point) registration method to calculate the pose change between the object point clouds of the two frames. This calculated pose change is used as the object motion pseudo-label.

[0141] This embodiment improves the accuracy of the trained object motion estimation model based on self-supervision and scale supervision of depth estimation.

[0142] In one embodiment, the step of adjusting the position of the objects in the initial image of the (i-1)th frame based on the motion data of each object, the initial depth map of the (i-1)th frame, and the object masks corresponding to the initial image of the (i-1)th frame, to obtain the adjusted image of the (i-1)th frame, includes:

[0143] S41: Based on the initial depth map of the (i-1)th frame and the camera intrinsic parameters corresponding to the initial image of the i-th frame, perform point cloud mapping on the initial image of the (i-1)th frame to obtain the initial point cloud;

[0144] Specifically, since the initial image of the (i-1)th frame and the initial image of the i-th frame were taken by the same camera, the camera intrinsic parameters corresponding to the initial image of the i-th frame are the same as those corresponding to the initial image of the (i-1)th frame. Based on the initial depth map of the (i-1)th frame and the camera intrinsic parameters corresponding to the initial image of the i-th frame, point cloud mapping is performed on the initial image of the (i-1)th frame to obtain the point cloud under the view of the initial image of the (i-1)th frame, and this point cloud is used as the initial point cloud.

[0145] It is understandable that each point in the initial point cloud retains the RGB colors (red, green, and blue channels) of the initial image in the (i-1)th frame.

[0146] S42: Based on the object masks corresponding to the initial image of the (i-1)th frame, locate the object point cloud in the initial point cloud to obtain the first object point cloud;

[0147] Specifically, based on the positional relationship of each object mask corresponding to the (i-1)th frame of the initial image, the object point cloud corresponding to each object is located in the initial point cloud, and each located object point cloud is used as the first object point cloud. Since the camera imaging process is conical, the first object point cloud is a conical point cloud.

[0148] S43: Based on the motion data of each of the objects, perform pose transformation on each of the first object point clouds to obtain the second object point cloud;

[0149] Specifically, based on the motion data of each object, a pose transformation is performed on each of the first object point clouds, thereby converting the first object point cloud to the viewpoint corresponding to the initial image of the i-th frame. In other words, the second object point cloud is the object point cloud under the viewpoint corresponding to the initial image of the i-th frame.

[0150] S44: Update the initial point cloud based on the point clouds of each of the second objects to obtain the target point cloud;

[0151] Specifically, the initial point cloud is updated based on each of the second object point clouds, and the updated initial point cloud is used as the target point cloud. This ensures that the viewpoint of the object point cloud in the target point cloud is the viewpoint corresponding to the initial image of the i-th frame.

[0152] S45: Based on the camera intrinsic parameters corresponding to the initial image of the i-th frame, perform image projection processing on the target point cloud to obtain the adjusted image of the (i-1)-th frame.

[0153] Specifically, since the camera intrinsic parameters corresponding to the initial image of the i-th frame are the same as those corresponding to the initial image of the (i-1)-th frame, the target point cloud is subjected to image projection processing according to the camera intrinsic parameters corresponding to the initial image of the i-th frame, and the projected image is used as the adjusted image of the (i-1)-th frame.

[0154] Note that since the target point cloud has changed compared to the initial point cloud obtained by mapping in step S41, and there is an occlusion relationship, when projecting the target point cloud onto the image, only the RGB colors of the point cloud closer to the camera coordinate system are retained.

[0155] This embodiment achieves the elimination of the positional changes of objects in the world coordinate system in the two initial images based on the object motion estimation results by keeping the position of objects in the world coordinate system consistent in the i-frame and i-1-frame, thereby eliminating the influence of object motion on the scene.

[0156] In one embodiment, before the step of inputting the initial image of the i-th frame and the adjusted image of the (i-1)-th frame into a preset temporal multi-view depth estimation model for depth estimation to obtain the target depth map of the i-th frame, the method further includes:

[0157] S51: Obtain multiple third training samples;

[0158] Specifically, it can obtain multiple third training samples from user input, multiple third training samples from storage space, or multiple third training samples from third-party applications.

[0159] S52: Obtain any one of the third training samples as the third target sample from each of the third training samples;

[0160] S53: Input the two image samples from the third target sample into the backbone network of the preset third model for feature extraction to obtain the first frame feature and the last frame feature.

[0161] Specifically, two image samples from the third target sample are input into the backbone network of a preset third model for feature extraction. Head frame features are extracted from the first frame of the two image samples, and tail frame features are extracted from the second frame. In computer vision tasks, the network that extracts features from images is called the backbone network, which can be considered the core of downstream tasks.

[0162] S54: Input the first frame features and the last frame features into the pose transformation module of the third model to perform feature concatenation and pose transformation prediction in sequence to obtain the pose transformation prediction value;

[0163] Specifically, the features of the first frame and the features of the last frame are concatenated, and the concatenated features are input into the pose transformation module of the third model for pose transformation prediction to predict the pose transformation between the two frames. The pose transformation is represented by translation in three directions and Euler angles of rotation in three directions, and the predicted data is used as the pose transformation prediction value.

[0164] S55: Using the depth estimation module of the third model, the cost body, encoding, and tail frame depth decoding are constructed sequentially for the head frame features and the tail frame features to obtain the tail frame depth map;

[0165] Specifically, the depth estimation module of the third model is used to construct a cost body between the features of the first frame and the features of the last frame. Then the cost body is encoded, and finally the depth map of the second frame (last frame) of the two adjacent frames is decoded from the encoded data. This depth map is used as the depth map of the last frame.

[0166] S56: Using the image synthesis module of the third model, the pose transformation prediction value and the tail frame depth map are used to synthesize the image from the tail frame perspective to obtain the tail frame synthesized image.

[0167] Specifically, the image synthesis module of the third model is used to synthesize the pose transformation prediction value and the tail frame depth map from the perspective of the tail frame, so as to project the first frame of the two image samples in the third target sample onto the perspective of the second frame of the two image samples in the third target sample, and use the synthesized image as the tail frame synthesized image.

[0168] S57: Based on photometric consistency loss, the loss value is calculated according to the last frame image samples of the last frame synthesized image and the last frame image samples of the two image samples in the third target sample to obtain the second loss value;

[0169] Specifically, the method and steps for calculating the loss value based on the photometric consistency loss, according to the tail frame image sample of the synthetic image of the tail frame and the tail frame image sample of the two image samples in the third target sample, can be selected from the prior art and will not be elaborated here.

[0170] S58: Update the network parameters of the third model according to the second loss value, and repeat the step of obtaining any one of the third training samples from each of the third training samples as the third target sample until the preset third training termination condition is reached, and use the third model as a temporal multi-view depth estimation model.

[0171] Specifically, the method steps for updating the network parameters of the third model based on the second loss value can be selected from existing technologies and will not be elaborated here.

[0172] Repeat the step of obtaining any one of the third training samples from each of the third training samples as the third target sample, that is, repeat steps S52 to S58.

[0173] Optionally, the third training termination condition is that the second loss value converges to a preset third value.

[0174] This embodiment uses the depth estimation module of the third model to construct a cost body, encode, and decode the tail frame depth of the first frame features and the tail frame features in sequence to obtain the tail frame depth map. The image synthesis module of the third model is used to synthesize the pose transformation prediction value and the tail frame depth map from the tail frame perspective to obtain the tail frame synthesized image, thereby realizing self-supervised model training and reducing the cost of model training.

[0175] In one embodiment, before the steps of obtaining the initial image of the i-th frame and the object masks corresponding to the initial image of the i-th frame, the following steps are included:

[0176] S11: Using a preset instance segmentation method, perform instance segmentation on the i-th frame initial image to obtain the object mask corresponding to each object in the i-th frame initial image.

[0177] First, scene videos need to be acquired. These video images are captured by a robot carrying a camera moving in the environment. Then, the image sequence in the video is cropped into multiple frames (i.e., initial images), and the order of each frame in the original video is recorded to represent the temporal relationship. In addition, the camera needs to be calibrated to obtain the camera's intrinsic parameters.

[0178] The initial image of frame i-1, which is the initial image of frame i-1. The initial image of frame i, which is the initial image of frame i.

[0179] Specifically, a preset instance segmentation method is used to segment the initial image of the i-th frame to obtain a mask for each object, and the mask of each object is used as an object mask.

[0180] This embodiment uses a preset instance segmentation method to perform instance segmentation on the initial image of the i-th frame, providing a basis for object motion estimation based on the object mask obtained from instance segmentation.

[0181] Reference Figure 2 This application provides a depth estimation device based on object motion, the device comprising:

[0182] The data acquisition module 100 is used to acquire the initial image of the i-th frame and the object masks corresponding to the initial image of the i-th frame;

[0183] The (i-1)th frame initial depth map determination module 200 is used to perform depth estimation on each pixel of the acquired (i-1)th frame initial image according to a preset single-frame depth estimation model, so as to obtain the (i-1)th frame initial depth map.

[0184] The object motion data determination module 300 is used to perform object motion estimation on each of the object masks corresponding to the initial image of the i-th frame and each of the object masks corresponding to the initial image of the (i-1)-th frame according to a preset object motion estimation model, so as to obtain the motion data of each object.

[0185] The i-1th frame adjusted image determination module 400 is used to adjust the position of the objects in the i-1th frame initial image according to the motion data of each object, the initial depth map of the i-1th frame and the object mask corresponding to the initial image of the i-1th frame, so as to obtain the i-1th frame adjusted image.

[0186] The i-th frame target depth map determination module 500 is used to input the initial image of the i-th frame and the adjusted image of the (i-1)-th frame into a preset temporal multi-view depth estimation model to perform depth estimation and obtain the i-th frame target depth map.

[0187] This embodiment achieves object motion estimation by using an object mask based on two initial images, thus introducing instance-level 3D motion estimation for each object. By adjusting the object's position based on the motion data, the positional changes of the object in the two initial images in the world coordinate system are eliminated based on the object motion estimation results. This ensures that the scene in the two frames of the input temporal multi-view depth estimation model remains consistent, avoiding the technical problem that changes in the scene between the two frames when the object itself moves, leading to incorrect object depth estimation and consequently incorrect scene depth estimation. This improves the accuracy of the target depth map in the i-th frame.

[0188] Reference Figure 3 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as depth estimation methods based on object motion. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a depth estimation method based on object motion. The object motion-based depth estimation method includes: acquiring an initial image of frame i and object masks corresponding to the initial image of frame i; performing depth estimation on each pixel of the acquired initial image of frame (i-1) according to a preset single-frame depth estimation model to obtain an initial depth map of frame (i-1); performing object motion estimation on the object masks corresponding to the initial image of frame i and the object masks corresponding to the initial image of frame (i-1) according to a preset object motion estimation model to obtain object motion data; adjusting the positions of objects in the initial image of frame (i-1) according to the object motion data, the initial depth map of frame (i-1), and the object masks corresponding to the initial image of frame (i-1) to obtain an adjusted image of frame (i-1); and inputting the initial image of frame i and the adjusted image of frame (i-1) into a preset temporal multi-view depth estimation model for depth estimation to obtain a target depth map of frame i.

[0189] This embodiment achieves object motion estimation by using an object mask based on two initial images, thus introducing instance-level 3D motion estimation for each object. By adjusting the object's position based on the motion data, the positional changes of the object in the two initial images in the world coordinate system are eliminated based on the object motion estimation results. This ensures that the scene in the two frames of the input temporal multi-view depth estimation model remains consistent, avoiding the technical problem that changes in the scene between the two frames when the object itself moves, leading to incorrect object depth estimation and consequently incorrect scene depth estimation. This improves the accuracy of the target depth map in the i-th frame.

[0190] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a depth estimation method based on object motion, including the following steps: acquiring an initial image of frame i and object masks corresponding to the initial image of frame i; performing depth estimation on each pixel of the acquired initial image of frame (i-1) according to a preset single-frame depth estimation model to obtain an initial depth map of frame (i-1); performing object motion estimation on the object masks corresponding to the initial image of frame i and the object masks corresponding to the initial image of frame (i-1) according to a preset object motion estimation model to obtain object motion data; adjusting the positions of objects in the initial image of frame (i-1) according to the object motion data, the initial depth map of frame (i-1), and the object masks corresponding to the initial image of frame (i-1) to obtain an adjusted image of frame (i-1); and inputting the initial image of frame i and the adjusted image of frame (i-1) into a preset temporal multi-view depth estimation model for depth estimation to obtain a target depth map of frame i.

[0191] The aforementioned object motion-based depth estimation method estimates object motion by using an object mask based on two initial images. This introduces instance-level 3D motion estimation for each object. By adjusting the object's position based on the motion data, the method eliminates positional changes of the object in the world coordinate system in the two initial images based on the motion estimation results. This ensures that the scene in the two images input to the temporal multi-view depth estimation model remains consistent, avoiding the technical problem of scene changes in the two images caused by the motion of the object itself, which leads to object depth estimation errors and subsequently scene depth estimation errors. This improves the accuracy of the target depth map in the i-th frame.

[0192] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0193] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0194] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A depth estimation method based on object motion, characterized in that, The method includes: Obtain the initial image of the i-th frame and the object masks corresponding to the initial image of the i-th frame; Based on the preset single-frame depth estimation model, depth estimation is performed on each pixel of the acquired initial image of the (i-1)th frame to obtain the initial depth map of the (i-1)th frame. According to the preset object motion estimation model, object motion estimation is performed on each of the object masks corresponding to the initial image of the i-th frame and each of the object masks corresponding to the initial image of the (i-1)-th frame to obtain object motion data; Based on the motion data of each object, the initial depth map of the (i-1)th frame and the object masks corresponding to the initial image of the (i-1)th frame, the positions of the objects in the initial image of the (i-1)th frame are adjusted to obtain the adjusted image of the (i-1)th frame. The initial image of the i-th frame and the adjusted image of the (i-1)-th frame are input into a preset temporal multi-view depth estimation model to perform depth estimation, thereby obtaining the target depth map of the i-th frame.

2. The depth estimation method based on object motion according to claim 1, characterized in that, Before the step of performing depth estimation on each pixel of the acquired initial image of the (i-1)th frame according to a preset single-frame depth estimation model to obtain the initial depth map of the (i-1)th frame, the following steps are included: Acquire multiple first training samples, a first counter, and a second counter, wherein each first training sample is a temporally adjacent image, the first counter is initialized to 2, and the second counter is initialized to 1; The first training sample corresponding to the first counter is obtained from each of the first training samples as the first target sample, and the first training sample corresponding to the second counter is obtained from each of the first training samples as the reference sample; The first target sample is input into the depth estimation module of the preset first model to estimate the depth of each pixel to obtain a first depth map. According to the camera pose transformation estimation module of the first model, the camera pose transformation data of the reference sample and the first target sample is predicted. According to the camera pose transformation data and the first depth map, the reference sample is projected onto the viewpoint corresponding to the first target sample to obtain a projected image. Based on the structural similarity loss function, the loss function is calculated according to the projected image and the first target sample to obtain the first loss value. The network parameters of the first model are updated according to the first loss value. The first counter and the second counter are both incremented by 1. The step of obtaining the first training sample corresponding to the first counter from each of the first training samples as the first target sample is repeated until the preset first training termination condition is reached. The depth estimation module of the first model is used as the single-frame depth estimation model.

3. The depth estimation method based on object motion according to claim 1, characterized in that, The step of estimating the motion of each object according to a preset object motion estimation model for each object mask corresponding to the initial image of the i-th frame and each object mask corresponding to the initial image of the (i-1)-th frame to obtain motion data for each object includes: Based on the IOU calculation method, each object mask corresponding to the initial image of the i-th frame and each object mask corresponding to the initial image of the (i-1)-th frame are associated to obtain the common object mask set of the (i-1)-th frame and the common object mask set of the i-th frame. Based on the common object mask set of the i-th frame, non-object regions are filtered out from the initial image of the i-th frame to obtain a first retained image. Based on the common object mask set of the (i-1)-th frame, non-object regions are filtered out from the initial image of the (i-1)-th frame to obtain a second retained image. The first preserved image and the second preserved image are stitched together along the channel dimension to obtain a stitched image; The stitched image is input into the object motion estimation model to estimate the object motion, thereby obtaining the motion data of each object.

4. The depth estimation method based on object motion according to claim 3, characterized in that, Before the step of inputting the stitched image into the object motion estimation model to perform object motion estimation and obtain the motion data of each object, the method further includes: Obtain multiple second training samples, wherein the second training samples include: two temporally adjacent image frames; Obtain any one of the second training samples as the second target sample from each of the second training samples; Two image samples from the second target sample are input into a preset second model to perform head frame depth map estimation and object pose transformation data estimation, and then synthesize a composite image of the object region from the tail frame perspective based on the head frame depth map and the object pose transformation data. Based on the structural similarity loss function, the loss value is calculated according to the object region images corresponding to the two frames of image samples in the second target sample, and the synthesis loss value is obtained. Two image samples from the second target sample are input into the second model to estimate the object motion and obtain the object motion estimate. Based on the L1 loss function, the loss value is calculated according to the estimated object motion value and the pseudo-label of the object motion corresponding to the second target sample to obtain the motion loss value. Based on the synthetic loss value and the motion loss value, update the network parameters of the second model, and repeat the step of obtaining any one of the second training samples from each of the second training samples as the second target sample until the preset second training termination condition is met, and use the second model as the object motion estimation model. The object motion pseudo-label corresponding to the second target sample is obtained from the object pose change data based on two image samples in the second target sample, the camera intrinsic parameters corresponding to the second target sample, and the ICP point cloud registration algorithm.

5. The depth estimation method based on object motion according to claim 1, characterized in that, The step of adjusting the positions of objects in the initial image of the (i-1)th frame based on the motion data of each object, the initial depth map of the (i-1)th frame, and the object masks corresponding to the initial image of the (i-1)th frame, to obtain the adjusted image of the (i-1)th frame, includes: Based on the initial depth map of the (i-1)th frame and the camera intrinsic parameters corresponding to the initial image of the i-th frame, point cloud mapping is performed on the initial image of the (i-1)th frame to obtain the initial point cloud; Based on the object masks corresponding to the initial image of the (i-1)th frame, the object point cloud is located in the initial point cloud to obtain the first object point cloud; Based on the motion data of each object, pose transformation is performed on each of the first object point clouds to obtain the second object point cloud. The initial point cloud is updated based on the point clouds of each of the second objects to obtain the target point cloud; Based on the camera intrinsic parameters corresponding to the initial image of the i-th frame, image projection processing is performed on the target point cloud to obtain the adjusted image of the (i-1)-th frame.

6. The depth estimation method based on object motion according to claim 1, characterized in that, Before the step of inputting the initial image of the i-th frame and the adjusted image of the (i-1)-th frame into a preset temporal multi-view depth estimation model for depth estimation to obtain the target depth map of the i-th frame, the method further includes: Obtain multiple third-party training samples; Obtain any one of the third training samples as the third target sample from each of the third training samples; The two image samples from the third target sample are respectively input into the backbone network of the preset third model for feature extraction to obtain the first frame feature and the last frame feature. The first frame features and the last frame features are input into the pose transformation module of the third model for feature concatenation and pose transformation prediction in sequence to obtain the pose transformation prediction value. Using the depth estimation module of the third model, a cost body, encoding, and tail frame depth decoding are sequentially constructed from the head frame features and the tail frame features to obtain the tail frame depth map; The image synthesis module of the third model is used to synthesize the pose transformation prediction value and the tail frame depth map from the tail frame perspective to obtain the tail frame synthesized image. Based on the photometric consistency loss, the loss value is calculated according to the last frame image samples of the synthetic last frame image and the last frame image samples of the two frames of image samples in the third target sample, and the second loss value is obtained. The network parameters of the third model are updated according to the second loss value. The step of obtaining any one of the third training samples from each of the third training samples as the third target sample is repeated until the preset third training termination condition is reached, and the third model is used as a temporal multi-view depth estimation model.

7. The depth estimation method based on object motion according to claim 1, characterized in that, Before the step of obtaining the initial image of the i-th frame and the object masks corresponding to the initial image of the i-th frame, the following steps are included: A preset instance segmentation method is used to perform instance segmentation on the i-th initial image to obtain the object mask corresponding to each object in the i-th initial image.

8. A depth estimation device based on object motion, characterized in that, The device includes: The data acquisition module is used to acquire the initial image of the i-th frame and the object masks corresponding to the initial image of the i-th frame; The (i-1)th frame initial depth map determination module is used to perform depth estimation on each pixel of the acquired (i-1)th frame initial image according to a preset single-frame depth estimation model, so as to obtain the (i-1)th frame initial depth map. The object motion data determination module is used to perform object motion estimation on each of the object masks corresponding to the initial image of the i-th frame and each of the object masks corresponding to the initial image of the (i-1)-th frame according to a preset object motion estimation model, so as to obtain the motion data of each object. The i-1th frame adjusted image determination module is used to adjust the position of the objects in the i-1th frame initial image according to the motion data of each object, the initial depth map of the i-1th frame and the object mask corresponding to the initial image of the i-1th frame, so as to obtain the i-1th frame adjusted image. The i-th frame target depth map determination module is used to input the initial image of the i-th frame and the adjusted image of the (i-1)-th frame into a preset temporal multi-view depth estimation model to perform depth estimation and obtain the i-th frame target depth map.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images