Scene reconstruction method, device, storage medium and electronic device

Abstract

Disclosed are a scene reconstruction method, device, storage medium and electronic device, wherein the method comprises: performing background segmentation on a first image sequence collected by a camera set on a movable device to obtain a second image sequence; performing sparse reconstruction of a point cloud using the second image sequence to obtain a first reconstructed point cloud and a first camera pose sequence of the camera; performing dynamic and static segmentation on the first image sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene; performing dense reconstruction of the point cloud using the first reconstructed point cloud, the first camera pose sequence and the third image sequence to obtain a second reconstructed point cloud; performing sparse reconstruction and dense reconstruction of the point cloud in sequence using the first camera pose sequence and the fourth image sequence to obtain a third reconstructed point cloud; and determining a scene reconstruction result corresponding to the first image sequence based on the second reconstructed point cloud and the third reconstructed point cloud. The present disclosure can reduce the implementation cost of scene reconstruction.

Description

Technical Field

[0001] The present disclosure relates to driving technology, and in particular to a scene reconstruction method, device, storage medium and electronic device. Background Art

[0002] In some cases, scene reconstruction is required, such as outdoor scenes for unmanned driving. The scene reconstruction results are of great value for scene reproduction and data annotation. At present, in order to achieve scene reconstruction, it is often necessary to rely on LiDAR. Summary of the invention

[0003] Current scene reconstruction relies on lidar, which is costly to implement.

[0004] In order to solve the above technical problems, the present disclosure provides a scene reconstruction method, device, medium and electronic device to reduce the implementation cost of scene reconstruction.

[0005] According to one aspect of an embodiment of the present disclosure, a scene reconstruction method is provided, including:

[0006] Performing background segmentation on a first image sequence captured by a camera disposed on a movable device to obtain a second image sequence;

[0007] Using the second image sequence, performing sparse reconstruction of the point cloud to obtain a first reconstructed point cloud and a first camera pose sequence of the camera;

[0008] Performing static and dynamic segmentation on the first image sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene;

[0009] Using the first reconstructed point cloud, the first camera pose sequence and the third image sequence, densely reconstruct the point cloud to obtain a second reconstructed point cloud;

[0010] Using the first camera pose sequence and the fourth image sequence, sequentially performing sparse reconstruction and dense reconstruction of the point cloud to obtain a third reconstructed point cloud;

[0011] A scene reconstruction result corresponding to the first image sequence is determined based on the second reconstructed point cloud and the third reconstructed point cloud.

[0012] According to another aspect of an embodiment of the present disclosure, there is provided a scene reconstruction device, comprising:

[0013] A first segmentation module is used to perform background segmentation on a first image sequence captured by a camera disposed on a movable device to obtain a second image sequence;

[0014] A first reconstruction module, configured to perform sparse reconstruction of a point cloud using the second image sequence obtained by the first segmentation module to obtain a first reconstructed point cloud and a first camera pose sequence;

[0015] A second segmentation module is used to perform dynamic and static segmentation on the first image sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene;

[0016] A second reconstruction module is used to perform dense reconstruction of the point cloud using the first reconstructed point cloud obtained by the first reconstruction module and the first camera pose sequence, and the third image sequence obtained by the second segmentation module to obtain a second reconstructed point cloud;

[0017] A third reconstruction module is used to use the first camera pose sequence obtained by the first reconstruction module and the fourth image sequence obtained by the second segmentation module to perform sparse reconstruction and dense reconstruction of the point cloud in sequence to obtain a third reconstructed point cloud;

[0018] A determination module is used to determine a scene reconstruction result corresponding to the first image sequence based on the second reconstructed point cloud obtained by the second reconstruction module and the third reconstructed point cloud obtained by the third reconstruction module.

[0019] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned scene reconstruction method.

[0020] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0021] processor;

[0022] a memory for storing instructions executable by the processor;

[0023] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above-mentioned scene reconstruction method.

[0024] According to another aspect of the embodiments of the present disclosure, a computer program product is provided. When instructions in the computer program product are executed by a processor, the above-mentioned scene reconstruction method is performed.

[0025] Based on the scene reconstruction method, device, medium and electronic device provided by the above embodiments of the present disclosure, in the embodiments of the present disclosure, the background segmentation can be performed on the multi-view images (i.e., the first image sequence) collected by the camera, and the second image sequence obtained by the background segmentation can be used to perform sparse reconstruction of the point cloud to obtain the first reconstructed point cloud and the first camera pose sequence. In addition, the multi-view images can also be segmented dynamically and statically to obtain a third image sequence corresponding to the static scene and a fourth image sequence corresponding to the dynamic scene. The third image sequence can be used together with the first reconstructed point cloud and the first camera pose sequence for dense reconstruction of the point cloud to obtain a second reconstructed point cloud that can effectively represent the static scene. The fourth image sequence can be used together with the first camera pose sequence for sparse reconstruction and dense reconstruction of the point cloud to obtain a third reconstructed point cloud that can effectively represent the dynamic scene. The scene reconstruction result corresponding to the first image sequence can be determined by combining the second reconstructed point cloud and the third reconstructed point cloud. It can be seen that in the embodiments of the present disclosure, scene reconstruction is achieved by a series of processing of the multi-view images collected by the camera. Compared with the laser radar, the price of the camera is very low, so the implementation cost of scene reconstruction can be reduced. In addition, in the embodiments of the present disclosure, a divide-and-conquer strategy is adopted for static scenes and dynamic scenes, that is, the static scenes and dynamic scenes are reconstructed separately, which is conducive to reducing the difficulty of reconstruction and improving the reconstruction speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1-1 It is a schematic diagram of the motion model of a static scene.

[0027] Figure 1-2 It is a schematic diagram of the motion model of a dynamic scene.

[0028] Figure 2 It is a flowchart of a scene reconstruction method provided by an exemplary embodiment of the present disclosure.

[0029] Figure 3 It is one of the flowcharts of the dynamic and static segmentation method provided by an exemplary embodiment of the present disclosure.

[0030] Figure 4 This is the second flowchart of the dynamic and static segmentation method provided by an exemplary embodiment of the present disclosure.

[0031] Figure 5-1 This is the third flowchart of the dynamic and static segmentation method provided by an exemplary embodiment of the present disclosure.

[0032] Figure 5-2 This is the fourth flowchart of the dynamic and static segmentation method provided by an exemplary embodiment of the present disclosure.

[0033] Figure 6 It is a flowchart of a method for reconstructing a dynamic scene provided by an exemplary embodiment of the present disclosure.

[0034] Figure 7-1 It is a flowchart of a method for correcting an object pose sequence provided by an exemplary embodiment of the present disclosure.

[0035] Figure 7-2 It is a flowchart of a method for correcting an object pose sequence provided by another exemplary embodiment of the present disclosure.

[0036] Figure 8 It is a flowchart of a dense reconstruction method of a dynamic scene provided by an exemplary embodiment of the present disclosure.

[0037] Fig. 9 It is a flowchart of a dense reconstruction method for a static scene provided by an exemplary embodiment of the present disclosure.

[0038] Fig.10 It is a flowchart of a dense reconstruction method for a static scene provided by another exemplary embodiment of the present disclosure.

[0039] Fig.11 It is a flowchart of a dense reconstruction method of a static scene provided by yet another exemplary embodiment of the present disclosure.

[0040] Fig.12 It is a flowchart of a scene reconstruction method provided by another exemplary embodiment of the present disclosure.

[0041] Fig.13 It is a structural schematic diagram of a scene reconstruction device provided by an exemplary embodiment of the present disclosure.

[0042] Fig.14 It is a structural schematic diagram of a scene reconstruction device provided by another exemplary embodiment of the present disclosure.

[0043] Figure 15-1 It is a structural schematic diagram of a scene reconstruction device provided by yet another exemplary embodiment of the present disclosure.

[0044] Figure 15-2 It is a structural schematic diagram of a scene reconstruction device provided by yet another exemplary embodiment of the present disclosure.

[0045] Fig.16 It is a structural schematic diagram of a scene reconstruction device provided by yet another exemplary embodiment of the present disclosure.

[0046] Fig.17 It is a structural schematic diagram of a scene reconstruction device provided by yet another exemplary embodiment of the present disclosure.

[0047] Fig.18 It is a structural schematic diagram of a scene reconstruction device provided by yet another exemplary embodiment of the present disclosure.

[0048] Fig.19is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0050] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0051] Application Overview

[0052] Laser radar (LiDAR) is a radar system that detects the target's position, speed, orientation, posture, shape and other characteristic quantities by emitting laser beams.

[0053] Lidar can be used for scene reconstruction, for example, for unmanned driving outdoor scene reconstruction, but Lidar is expensive, which leads to high cost of scene reconstruction. How to reduce the cost of scene reconstruction is a problem worthy of attention for those skilled in the art.

[0054] Exemplary Systems

[0055] The following text involves multiple models, such as optical flow prediction model, depth prediction model and instance segmentation model, as well as motion model of static scene and motion model of dynamic scene. In order to facilitate the understanding of the following text, these models are briefly introduced here.

[0056] The optical flow prediction model, the depth prediction model, and the instance segmentation model can all be convolutional neural network (CNN) models. A frame of image is provided as input to the optical flow prediction model, and the optical flow prediction model can predict the optical flow value of each pixel in the frame of image; wherein the optical flow value of any pixel can be used to indicate the pixel position corresponding to the pixel position of the pixel in the next frame of image. A frame of image is provided as input to the depth prediction model, and the depth prediction model can predict the depth value of each pixel in the frame of image; wherein the depth value of any pixel can be used to determine the spatial position of the pixel in the camera coordinate system. A frame of image is provided as input to the instance segmentation model, and the instance segmentation model can predict the category of each pixel in the frame of image and locate different instances in the frame of image.

[0057] The motion model of a static scene can be a mathematical model used to describe the motion law of objects in a static scene. The motion model of a static scene can be found in Figure 1-1 The motion model of a dynamic scene can be a mathematical model used to describe the motion law of objects in a dynamic scene. The motion law of a dynamic scene can be found in Figure 1-2 .like Figure 1-1 As shown, the camera is moving, and the optical center of the camera at time t-1, time t, and time t+1 is O t-1 , O t , O t+1 , the object (such as lane lines, buildings, etc.) is stationary, and the kth point on the object is always point P k It means that in the images captured by the camera at time t-1, time t, and time t+1, the projection points of the kth point on the object are respectively Figure 1-2 and Figure 1-1 The main difference is that the camera and the object (such as a vehicle, pedestrian, etc.) are both moving, and the kth point on the object at time t-1, time t, and time t+1 can be expressed as The posture of the object at time t-1, time t, and time t+1 can be expressed as

[0058] Exemplary Methods

[0059] Figure 2 It is a flowchart of a scene reconstruction method provided by an exemplary embodiment of the present disclosure. Figure 2 The method shown may include step 210, step 220, step 230, step 240, step 250 and step 260, and each step is described below.

[0060] Step 210 , performing background segmentation on a first image sequence captured by a camera disposed on a movable device to obtain a second image sequence.

[0061] Optionally, the movable device includes but is not limited to vehicles, trains, ships, etc.; the cameras provided on the movable device include but are not limited to front-view cameras, rear-view cameras, side-view cameras, etc.

[0062] The camera provided on the mobile device can capture images at a certain frame rate to obtain a first image sequence including multiple captured images. For ease of understanding, the embodiments of the present disclosure are described by taking the case where the multiple captured images in the first image sequence are N captured images as an example.

[0063] For each of the N frames of acquired images included in the first image sequence, instance segmentation can be performed on the acquired image through an instance segmentation model to determine the respective predicted categories of each pixel in the acquired image, thereby determining which areas in the acquired image are areas where objects that may move (such as vehicles, pedestrians, etc.) are located, so as to generate masks in these areas, thereby removing objects that may move in the acquired image and retaining only completely stationary objects (such as the ground, walls, buildings, trees, etc.), thereby segmenting the background image from the acquired image.

[0064] According to the method described in the previous paragraph, N frames of background images corresponding to the N frames of captured images can be obtained, and the N frames of background images can be arranged in time sequence to form a second image sequence.

[0065] The above is to achieve background segmentation of the first image sequence by using the instance segmentation model, thereby obtaining the second image sequence. In some cases, the foreground and background separation of each frame of the first image sequence can also be achieved by using the semantic segmentation model, thereby obtaining the second image sequence.

[0066] Step 220: sparsely reconstruct the point cloud using the second image sequence to obtain a first reconstructed point cloud and a first camera pose sequence of the camera.

[0067] For each background image in the N background images included in the second image sequence, a key point detection algorithm may be used to perform key point detection on the background image; wherein the key point detection algorithm includes but is not limited to a scale-invariant feature transform (SIFT) algorithm, an algorithm based on deep learning, and the like.

[0068] When the key points of each of the N frames of background images are known, the key points can be matched and screened. Combined with the use of the incremental motion recovery structure (Structure From Motion, SFM) algorithm, the camera pose at the acquisition time corresponding to each of the N frames of acquired images can be determined to obtain N camera poses. The N camera poses are arranged in time sequence to form a first camera pose sequence. In addition, several three-dimensional points in the world coordinate system can be determined to form a first reconstructed point cloud.

[0069] Of course, the algorithm used to generate the first reconstructed point cloud and the first camera pose sequence is not limited to the incremental SFM algorithm, but can also be a full SFM algorithm, a hybrid SFM algorithm, etc., which are not listed here one by one.

[0070] Step 230 , performing static-dynamic segmentation on the first image sequence to obtain a third image sequence corresponding to static scenes and a fourth image sequence corresponding to dynamic scenes.

[0071] For each of the N acquired images included in the first image sequence, it is possible to determine which objects in the acquired image are objects that have actually moved and which objects are objects that have not moved. The acquired image can thereby be segmented into a segmented image that only includes objects that have moved (which can be considered as a segmented image corresponding to a dynamic scene) and a segmented image that only includes objects that have not moved (which can be considered as a segmented image corresponding to a static scene), thereby achieving dynamic and static segmentation of the acquired image.

[0072] According to the method introduced in the previous paragraph, the N frames of acquired images can be subjected to static and dynamic segmentation to obtain N frames of segmented images corresponding to static scenes and N frames of segmented images corresponding to dynamic scenes. The N frames of segmented images corresponding to the static scenes are arranged in time sequence to form a third image sequence, and the N frames of segmented images corresponding to the dynamic scenes are arranged in time sequence to form a fourth image sequence.

[0073] Step 240 , densely reconstructing the point cloud using the first reconstructed point cloud, the first camera pose sequence, and the third image sequence to obtain a second reconstructed point cloud.

[0074] Given the first reconstructed point cloud, the first camera pose sequence and the third image sequence, combined Figure 1-1 The motion model shown can use the Multi-View Stereo (MVS) algorithm to find pixels with photometric consistency in different images in the third image sequence, and perform stereo matching on this basis to determine a number of points in the world coordinate system to form a second reconstructed point cloud. The second reconstructed point cloud is denser than the first reconstructed point cloud, and the second reconstructed point cloud can effectively represent static scenes.

[0075] Step 250 , using the first camera pose sequence and the fourth image sequence, sequentially perform sparse reconstruction and dense reconstruction of the point cloud to obtain a third reconstructed point cloud.

[0076] When the first camera pose sequence and the fourth image sequence are known, sparse reconstruction can be performed using the SFM algorithm first, and then dense reconstruction can be performed using the MVS algorithm to obtain a third reconstructed point cloud, which can effectively characterize dynamic scenes.

[0077] Step 260: Determine a scene reconstruction result corresponding to the first image sequence based on the second reconstructed point cloud and the third reconstructed point cloud.

[0078] Optionally, the second reconstructed point cloud and the third reconstructed point cloud may be modeled respectively to obtain a first three-dimensional model corresponding to the second reconstructed point cloud and a second three-dimensional model corresponding to the third reconstructed point cloud, and then texture mapping is performed on the first three-dimensional model and the second three-dimensional model respectively, and the scene reconstruction result corresponding to the first image sequence may include the first three-dimensional model after texture mapping and the second three-dimensional model after texture mapping. In some embodiments, the scene reconstruction result corresponding to the first image sequence may also include a motion trajectory of a movable device, a motion trajectory of objects that move around the movable device, etc.

[0079] In the embodiment of the present disclosure, the background segmentation can be performed on the multi-view images (i.e., the first image sequence) captured by the camera, and the second image sequence obtained by the background segmentation can be used to perform sparse reconstruction of the point cloud to obtain the first reconstructed point cloud and the first camera pose sequence. In addition, the multi-view images can also be segmented statically and dynamically to obtain a third image sequence corresponding to the static scene and a fourth image sequence corresponding to the dynamic scene. The third image sequence can be used together with the first reconstructed point cloud and the first camera pose sequence for dense reconstruction of the point cloud to obtain a second reconstructed point cloud that can effectively represent the static scene. The fourth image sequence can be used together with the first camera pose sequence for sparse reconstruction and dense reconstruction of the point cloud to obtain a third reconstructed point cloud that can effectively represent the dynamic scene. The second reconstructed point cloud and the third reconstructed point cloud can be combined to determine the scene reconstruction result corresponding to the first image sequence. It can be seen that in the embodiment of the present disclosure, the scene reconstruction is achieved by a series of processing of the multi-view images captured by the camera. Compared with the laser radar, the price of the camera is very low, so the implementation cost of the scene reconstruction can be reduced. In addition, in the embodiments of the present disclosure, a divide-and-conquer strategy is adopted for static scenes and dynamic scenes, that is, the static scenes and dynamic scenes are reconstructed separately, which is conducive to reducing the difficulty of reconstruction and improving the reconstruction speed.

[0080] Figure 3 It is a flowchart of a dynamic and static segmentation method provided by an exemplary embodiment of the present disclosure. Figure 3 The method shown may include step 310, step 320 and step 330, each of which is described below.

[0081] Step 310: Perform optical flow prediction on the first image sequence using an optical flow prediction model to obtain an optical flow prediction result.

[0082] For each of the N frames of acquired images included in the first image sequence, the acquired image can be provided as input to the optical flow prediction model, and the optical flow prediction model can perform calculations based on it to obtain the optical flow value of each pixel point in the acquired image. The optical flow value of each pixel point in the acquired image can constitute the optical flow prediction data corresponding to the acquired image.

[0083] According to the method introduced in the previous paragraph, the optical flow prediction data corresponding to each of the N frames of acquired images can be obtained, and the optical flow prediction data corresponding to each of the N frames of acquired images can constitute the optical flow prediction result corresponding to the first image sequence.

[0084] Step 320: Perform depth prediction on the first image sequence using a depth prediction model to obtain a depth prediction result.

[0085] For each of the N acquired images included in the first image sequence, the acquired image can be provided as an input to the depth prediction model, and the depth prediction model can perform operations based on it to obtain the depth value of each pixel in the acquired image, and the depth value corresponding to each pixel in the acquired image can constitute the depth prediction data corresponding to the acquired image. Optionally, the depth prediction data corresponding to the acquired image can be in the form of a depth map.

[0086] According to the method introduced in the previous paragraph, the depth prediction data corresponding to each of the N frames of acquired images can be obtained, and the depth prediction data corresponding to each of the N frames of acquired images can constitute the depth prediction result corresponding to the first image sequence.

[0087] Step 330 , based on the optical flow prediction result, the depth prediction result and the first camera pose sequence, the first image sequence is segmented into static and dynamic images to obtain a third image sequence corresponding to the static scene and a fourth image sequence corresponding to the dynamic scene.

[0088] Optionally, step 330 may be performed as Figure 2 An optional implementation of step 230 in the illustrated embodiment.

[0089] In some optional embodiments of the present disclosure, Figure 4 As shown, step 330 may include step 3301 , step 3303 and step 3305 .

[0090] Step 3301: Determine the first pixel position correspondence between the previous and next frame images in the first image sequence based on the optical flow prediction result.

[0091] Assuming that the N captured images in the first image sequence are sequentially represented as image 1, image 2, image 3, ..., image N, and the optical flow value corresponding to pixel point 1 in image 1 is used to indicate that the pixel position of pixel point 2 in image 2 corresponds to the pixel position of pixel point 1 in image 1, then the first pixel position correspondence relationship may include: the correspondence relationship between the pixel position of pixel point 1 and the pixel position of pixel point 2. In this way, the first pixel position correspondence relationship may include the correspondence relationship between the pixel positions of multiple pairs of matching points; wherein pixel point 1 and pixel point 2 may form a pair of matching points.

[0092] Step 3303: Determine a second pixel position correspondence relationship between the previous and next frame images in the first image sequence based on the depth prediction result and the first camera pose sequence.

[0093] The first camera pose sequence may include: a camera pose corresponding to image 1 and a camera pose corresponding to image 2. Using the camera pose corresponding to image 1 and the camera pose corresponding to image 2, the camera's translation matrix and rotation matrix from the capture time corresponding to image 1 to the capture time corresponding to image 2 may be calculated.

[0094] refer to Figure 1-1 For the motion model shown, when the depth value corresponding to pixel 1 is known, and the translation matrix and rotation matrix of the camera from the acquisition time corresponding to image 1 to the acquisition time corresponding to image 2 are known, combined with the pre-calibrated camera intrinsic parameters, the pixel 2' corresponding to pixel 1 on image 2 can be determined using the following formula:

[0095]

[0096] in, represents the pixel position of pixel 2', K represents the pre-calibrated camera intrinsic parameter in matrix form, represents the rotation matrix of the camera from the acquisition time corresponding to image 1 to the acquisition time corresponding to image 2, K -1 represents the inverse matrix of K, represents the pixel position of pixel 1, Indicates the depth value of pixel 1 in the depth prediction result. Represents the translation matrix of the camera from the acquisition time corresponding to image 1 to the acquisition time corresponding to image 2.

[0097] The above formula can be understood as first converting pixel point 1 in image 2 from the pixel coordinate system to the camera coordinate system, and then converting from the camera coordinate system to the world coordinate system to obtain the space point 1 corresponding to pixel point 1 in the world coordinate system, and then converting the space point 1 from the world coordinate system to the camera coordinate system, and then converting from the camera coordinate system to the pixel coordinate system, so as to determine the pixel point 2' corresponding to pixel point 1 on image 2. The second pixel position correspondence relationship can include the correspondence between the pixel position of pixel point 1 and the pixel position of pixel point 2'. In this way, the second pixel position correspondence relationship can include the correspondence between the pixel positions of multiple pairs of matching points; wherein, pixel point 1 and pixel point 2' can form a pair of matching points.

[0098] Step 3305 , based on the first pixel position correspondence relationship and the second pixel position correspondence relationship, perform static and dynamic segmentation on the first image sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene.

[0099] In some optional embodiments of the present disclosure, Figure 5-1 As shown, step 3305 includes step 33051a, step 33052a, step 33053a, step 33054a, step 33055a and step 33056a.

[0100] Step 33051a, for the first target pixel point in the first target image in the first image sequence, determine the first target pixel position corresponding to the pixel position of the first target pixel point from the second target image according to the first pixel position correspondence relationship; wherein the second target image is the next frame image of the first target image in the first image sequence.

[0101] Step 33052a, according to the second pixel position correspondence, determine the second target pixel position corresponding to the pixel position of the first target pixel point from the second target image.

[0102] It should be noted that the first target image may be any frame image except the last frame image in the first image sequence, and the first target pixel point may be any pixel point in the first target image.

[0103] Assuming that the first target image is image 1, the second target image is image 2, and the first target pixel is pixel 1, then according to the first pixel position correspondence, the first target pixel position determined from the second target image can be the pixel position of pixel 2 in image 2, and the second target pixel position determined from the second target image can be the pixel position of pixel 2' in image 2.

[0104] Step 33053a, based on the position error between the first target pixel position and the second target pixel position, determine the pixel attribute of the first target pixel; wherein the pixel attribute of any pixel is used to characterize whether the pixel belongs to a dynamic object.

[0105] Assuming that the first target pixel position is represented by f(p) and the second target pixel position is represented by u(p), the position error between the first target pixel position and the second target pixel position can be represented by |f(p)-u(p)|. For ease of explanation, the position error between the first target pixel position and the second target pixel position is referred to as pixel position error. Based on the pixel position error and the preset position error, the pixel point attribute of the first target pixel point can be determined.

[0106] Optionally, if the pixel position error is greater than a preset position error, the pixel point attribute of the first target pixel point can be determined as a first-category attribute to characterize that the first target pixel point belongs to a dynamic object; if the pixel position error is less than or equal to the preset position error, the pixel point attribute of the first target pixel point can be determined as a second-category attribute to characterize that the first target pixel point does not belong to a dynamic object, that is, the first target pixel point belongs to a static object.

[0107] Of course, the method of determining pixel attributes is not limited to this. For example, the difference between the pixel position error and the preset position error can be calculated, and the ratio of the absolute value of the difference to the preset position error can be calculated. If the calculated ratio is greater than the preset ratio, the pixel attribute of the first target pixel can be determined as a first-class attribute to characterize that the first target pixel belongs to a dynamic object; if the calculated ratio is less than or equal to the preset ratio, the pixel attribute of the first target pixel can be determined as a second-class attribute to characterize that the first target pixel does not belong to a dynamic object, that is, the first target pixel belongs to a static object.

[0108] Step 33054a, based on the pixel attributes of the first target pixel points, the first target image is segmented into static and dynamic images to obtain a first segmented image corresponding to the static scene and a second segmented image corresponding to the dynamic scene.

[0109] According to the method of determining the pixel attributes described above, the pixel attributes of each pixel in the first target image can be obtained. By generating a mask in the area where the pixel attributes of the first type of attributes are distributed in the first target image, the dynamic objects in the first target image can be removed, thereby segmenting the first segmented image corresponding to the static scene. Similarly, by generating a mask in the area where the pixel attributes of the second type of attributes are distributed in the first target image, the static objects in the first target image can be removed, thereby segmenting the second segmented image corresponding to the dynamic scene.

[0110] Step 33055a, determining a third image sequence based on the first segmented image.

[0111] Step 33056a, determining a fourth image sequence based on the second segmented image.

[0112] The above describes the dynamic and static segmentation method of the first target image. Each frame of the acquired image in the first image sequence can be segmented in a similar way, thereby obtaining N frame segmented images corresponding to static scenes and N frame segmented images corresponding to dynamic scenes. The N frame segmented images corresponding to static scenes are arranged in time sequence to form a third image sequence; the N frame segmented images corresponding to dynamic scenes are arranged in time sequence to form a fourth image sequence.

[0113] In some embodiments, after obtaining N frame segmented images corresponding to static scenes and N frame segmented images corresponding to dynamic scenes, these segmented images can be corrected (for example, filtering) using a certain algorithm, and then the third image sequence and the fourth image sequence can be determined based on the corrected segmented images.

[0114] In this embodiment, by comparing the position error between the two target pixel positions obtained based on the first pixel position correspondence relationship and the second pixel position correspondence relationship, the position error can be used as reference information to determine which pixels in the first target image belong to dynamic objects and which pixels belong to static objects. On this basis, the first target image can be efficiently and reliably segmented into dynamic and static objects, so as to obtain the third image sequence and the fourth image sequence for subsequent steps.

[0115] In some optional embodiments of the present disclosure, Figure 5-2 As shown, the method provided by the embodiment of the present disclosure also includes step 3304.

[0116] Step 3304: perform instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result.

[0117] For each of the N frames of acquired images included in the first image sequence, the acquired image can be provided as input to the instance segmentation model, and the instance segmentation model can perform operations based on it to obtain the predicted category of each pixel point in the acquired image, and the instance area of ​​each instance in the acquired image (which can be either a static object or a dynamic object). This information can constitute the instance segmentation data corresponding to the acquired image.

[0118] According to the method described in the previous paragraph, instance segmentation data corresponding to each of the N frames of acquired images can be obtained, and the instance segmentation data corresponding to each of the N frames of acquired images can constitute the instance segmentation result corresponding to the first image sequence.

[0119] Step 3305 includes step 33051b, step 33052b, step 33053b, step 33054b, step 33055b and step 33056b.

[0120] Step 33051b, for the first target image in the first image sequence, determine the instance region of each instance in the first target image based on the instance segmentation result.

[0121] Optionally, instance segmentation data corresponding to the first target image may be directly extracted from the instance segmentation result, and the instance region of each instance in the first target image may be located based on the instance segmentation data corresponding to the first target image.

[0122] Step 33052b, based on the first pixel position correspondence relationship and the second pixel position correspondence relationship, determine the pixel attributes of each pixel in the first target image; wherein the pixel attribute of any pixel is used to characterize whether the pixel belongs to a dynamic object.

[0123] It should be noted that, based on the first pixel position correspondence relationship and the second pixel position correspondence relationship, the method for determining the pixel point attribute of each pixel point in the first target image is as follows: Figure 5-1 The relevant introduction in the illustrated implementation manner will suffice and will not be repeated here.

[0124] Step 33053b, for each instance area in the first target image, statistics are performed on the pixel attributes of each pixel in the instance area to obtain statistical results.

[0125] Optionally, for each instance area in the first target image, a first number and a first proportion of pixels in the instance area whose pixel attributes are the first type of attributes, as well as a second number and a second proportion of pixels in the instance area whose pixel attributes are the second type of attributes, can be counted, and the first number, first proportion, second number, and second proportion corresponding to each instance area in the first target image can constitute the statistical result.

[0126] In some cases, the statistical result may also include only the first quantity and the second quantity corresponding to each instance region in the first target image, or only the first proportion and the second proportion corresponding to each instance region in the first target image.

[0127] Step 33054b, based on the statistical results corresponding to each instance area in the first target image, the first target image is segmented into static and dynamic scenes to obtain a third segmented image corresponding to the static scene and a fourth segmented image corresponding to the dynamic scene.

[0128] In some optional implementations of the present disclosure, step 33054b includes:

[0129] For each instance region in the first target image, based on the statistical result corresponding to the instance region, determining the proportion of pixels in the instance region whose attributes are used to characterize that the current pixel does not belong to a dynamic object;

[0130] Based on the proportions corresponding to each instance area in the first target image, the first target image is segmented into static and dynamic scenes to obtain a third segmented image corresponding to the static scene and a fourth segmented image corresponding to the dynamic scene.

[0131] Assuming that the statistical results include the second proportion corresponding to each instance area in the first target image, the second proportion corresponding to any instance area can be considered as the proportion of pixel points in the instance area used by the pixel point attributes to characterize that the current pixel point does not belong to a dynamic object. Then, the target image can be directly segmented into dynamic and static parts based on the second proportion corresponding to each instance area in the first target image and the preset proportion.

[0132] Assuming that the statistical result does not include the second proportion corresponding to each instance region in the first target image, but includes the first proportion corresponding to each instance region in the first target image, then for each instance region, the difference between 1 as a preset value and the first proportion corresponding to the instance region can be calculated to obtain the second proportion corresponding to the instance region. Then, based on the second proportion corresponding to each instance region in the first target image and the preset proportion, the target image is segmented dynamically and statically.

[0133] Optionally, the preset proportion may be 70%, 75%, 80%, etc., which are not listed here one by one.

[0134] For any instance region in the first target image, if the second proportion corresponding to the instance region is greater than the preset proportion, it can be determined that the instance region belongs to a static object; if the second proportion corresponding to the instance region is less than or equal to the preset proportion, it can be determined that the instance region belongs to a dynamic object. By generating a mask for each instance region belonging to a dynamic object in the first target image, the dynamic object in the first target image can be removed, thereby segmenting a third segmented image corresponding to a static scene. Similarly, by generating a mask for each instance region belonging to a static object in the first target image, the static object in the first target image can be removed, thereby segmenting a fourth segmented image corresponding to a dynamic scene.

[0135] The above paragraph introduces the case of performing dynamic and static segmentation with reference to the second proportion corresponding to each instance area. In some cases, dynamic and static segmentation can also be performed with reference to the first proportion corresponding to each instance area. For example, for any instance area in the first target image, if the first proportion corresponding to the instance area is less than or equal to the preset proportion, it can be determined that the instance area belongs to a static object. If the first proportion corresponding to the instance area is greater than the preset proportion, it can be determined that the instance area belongs to a dynamic object. On this basis, the third segmented image corresponding to the static scene and the fourth segmented image corresponding to the dynamic scene can also be segmented by generating a mask.

[0136] Step 33055b, determining a third image sequence based on the third segmented image.

[0137] Step 33056b, determining a fourth image sequence based on the fourth segmented image.

[0138] It should be noted that the specific implementation of steps 33055b and 33056b can refer to the relevant introduction to steps 33055a and 33056a above, which will not be repeated here.

[0139] It should be noted that both the depth prediction model and the optical flow prediction model have certain prediction errors, and the process of obtaining the first camera pose sequence also has certain estimation errors. In order to reduce the impact of these errors on the scene reconstruction results, the instance segmentation model can be used to determine the instance area of ​​each instance in the first target image, and for each instance area, the pixel attributes are statistically analyzed so that the statistical results can be used for dynamic and static segmentation of the first target image. For example, the proportion of pixels with the second type of attributes in each instance area can be used for dynamic and static segmentation. This is equivalent to introducing certain constraints for the dynamic and static segmentation process (the motion states of each point on the same object are often consistent, that is, these points are either fixed or move together), which is conducive to making the dynamic and static segmentation results more robust. The dynamic and static segmentation results are used in subsequent steps to ensure the final scene reconstruction effect.

[0140] In the embodiments of the present disclosure, by using the optical flow prediction model, the optical flow prediction result corresponding to the first image can be determined efficiently and reliably, and the optical flow prediction result can be used to determine the first pixel position correspondence between the front and rear frame images. By using the depth prediction model, the depth prediction result corresponding to the first image sequence can be determined efficiently and reliably, and the depth prediction result can be used together with the first camera pose sequence to determine the second pixel position correspondence between the front and rear frame images. The first pixel position correspondence can be regarded as an object tracking result obtained in one way, and the second pixel position correspondence can be regarded as an object tracking result obtained in another way. Combining the first pixel position correspondence and the second pixel position correspondence, the position error can be determined, and the determined position error can be regarded as a residual flow. With the residual flow as reference information, the dynamic objects and static objects in the image can be distinguished more accurately to ensure the dynamic and static segmentation effect of the first image sequence. In specific implementation, some more complex tracking algorithms can also be introduced, and the tracking results obtained by the tracking algorithm are used for dynamic and static segmentation.

[0141] Figure 6 It is a flowchart of a method for reconstructing a dynamic scene provided by an exemplary embodiment of the present disclosure. Figure 6 The method shown includes step 610, step 620, step 630, step 640 and step 650. Optionally, a combination of step 620 to step 650 can be used as an optional implementation of step 250.

[0142] Step 610: Perform optical flow prediction on the first image sequence using an optical flow prediction model to obtain an optical flow prediction result.

[0143] It should be noted that the specific implementation of step 610 can refer to the relevant introduction to step 310 above, which will not be repeated here.

[0144] Step 620: Determine a third pixel position correspondence relationship between the previous and next frame images in the fourth image sequence based on the optical flow prediction result.

[0145] It should be noted that the specific implementation of step 620 can refer to the above introduction to step 3301 and will not be repeated here.

[0146] Step 630 , performing sparse reconstruction of the point cloud using the first camera pose sequence, the fourth image sequence and the third pixel position correspondence to obtain a fourth reconstructed point cloud and an object pose sequence of the dynamic object in the fourth image sequence.

[0147] When the correspondence between the first camera pose sequence, the fourth image sequence and the third pixel position is known, the SFM algorithm can be used in combination with Figure 1-2 The motion model shown in the figure obtains the fourth reconstructed point cloud and the object pose sequence of the dynamic object in the fourth image sequence; wherein the object pose sequence may include the object pose of the dynamic object in the fourth image sequence at the respective acquisition moments of the N frames of acquired images.

[0148] It should be noted that the references given above Figure 1-1 The motion model shown determines the formula for pixel 2' corresponding to pixel 1 in image 2, and explains the basic principle of the formula. Here, when using the first camera pose sequence, the fourth image sequence and the third pixel position correspondence to perform sparse reconstruction of the point cloud, it is also necessary to find multiple pairs of matching points. The main difference from the basic principle explained above is that the process of finding multiple pairs of matching points here needs to consider the movement of the object.

[0149] Assume that pixel 1 in image 1 belongs to moving object c. Pixel 1 may be a point with obvious features of moving object c, such as a point with a large gradient. Here, pixel 1 in image 1 may be first converted from the pixel coordinate system to the camera coordinate system, and then from the camera coordinate system to the world coordinate system, to obtain the acquisition time corresponding to image 1, and the spatial point 1 corresponding to pixel 1 in the world coordinate system. Next, referring to the translation matrix of moving object c from the acquisition time corresponding to image 1 to the acquisition time corresponding to image 2, spatial point 1 may be converted from the acquisition time corresponding to image 1 to the acquisition time corresponding to image 2, to obtain spatial point 1'. After that, spatial point 1' may be first converted from the world coordinate system to the camera coordinate system, and then from the camera coordinate system to the pixel coordinate system, so as to determine pixel 2'' corresponding to pixel 1 in image 2, and pixel 1 and pixel 2'' may form a pair of matching points.

[0150] Step 640: Correct the object posture sequence according to a preset correction method.

[0151] In some optional embodiments of the present disclosure, Figure 7-1 As shown, the method provided by the embodiment of the present disclosure also includes step 632.

[0152] Step 632: Perform depth prediction on the first image sequence using a depth prediction model to obtain a depth prediction result.

[0153] It should be noted that the specific implementation of step 632 can refer to the relevant introduction to step 320 above, which will not be repeated here.

[0154] Step 640 includes step 6401, step 6402 and step 6403.

[0155] Step 6401: Determine a fourth pixel position correspondence relationship between previous and next frame images in a fourth image sequence based on the depth prediction result, the first camera pose sequence, and the object pose sequence.

[0156] It should be noted that the method for determining the fourth pixel position correspondence relationship is similar to the method for determining the second pixel position correspondence relationship in the above text, and the main difference is that the fourth pixel position correspondence relationship is determined with reference to Figure 1-2 The motion model shown in FIG. 1 needs to consider the motion of the object during the determination process (for details, please refer to the above description of the method for determining the pixel point 2'' corresponding to the pixel point 1 in the image 2). Therefore, the object posture sequence needs to be applied, and the determination of the corresponding relationship of the second pixel position is based on Figure 1-1 The motion model shown does not require the motion of the object to be considered during the determination process.

[0157] Step 6402: Determine a reprojection error based on the third pixel position correspondence relationship and the fourth pixel position correspondence relationship.

[0158] It should be noted that the above describes a method for determining the position error for the first target pixel based on the first pixel position correspondence and the second pixel position correspondence. Here, based on the third pixel position correspondence and the fourth pixel position correspondence, a similar method can be used to determine the position errors corresponding to the multiple target pixels, and then the reprojection error can be determined based on the position errors corresponding to the multiple target pixels. For example, the average value of the position errors corresponding to the multiple target pixels can be directly calculated, and the calculated average value is used as the reprojection error. For another example, the position errors with obviously abnormal values ​​(for example, the position errors are too large or too small) can be removed from the position errors corresponding to the multiple target pixels, and then the average value of the remaining position errors is calculated, and the calculated average value is used as the reprojection error.

[0159] Step 6403, correct the object pose sequence by minimizing the reprojection error.

[0160] Optionally, the reprojection error may be minimized by specifying an optimization algorithm, where the specified optimization algorithm may be either a linear optimization algorithm or a nonlinear optimization algorithm, for example, either a linear least squares method or a nonlinear least squares method.

[0161] The calculation of the reprojection error needs to be applied to the fourth pixel position correspondence, and the determination of the fourth pixel position correspondence needs to be applied to the object pose sequence, that is, the object pose sequence and the reprojection error are interrelated. In this way, by minimizing the reprojection error, the object pose sequence can be optimized, and the corrected object pose sequence can be used for subsequent point cloud reconstruction, which is beneficial to improving the point cloud reconstruction effect.

[0162] In some other optional embodiments of the present disclosure, Figure 7-2 As shown, the method provided by the embodiment of the present disclosure also includes step 634.

[0163] Step 634: perform instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result.

[0164] It should be noted that the specific implementation of step 634 can refer to the relevant introduction to step 320 above, which will not be repeated here.

[0165] Step 640 includes step 6404, step 6405, step 6406 and step 6407.

[0166] Step 6404: determine the motion trajectory of the dynamic object in the fourth image sequence based on the object posture sequence.

[0167] For any dynamic object in the fourth image sequence, an object position sequence of the dynamic object can be extracted from the object posture sequence, and each position in the extracted object position sequence can constitute the motion trajectory of the dynamic object. Alternatively, after the object position sequence is extracted, the positions in the extracted object position sequence that are obviously abnormal can be corrected, and each position in the corrected object position sequence can constitute the motion trajectory of the dynamic object.

[0168] Step 6405: Determine a first predicted category of the dynamic object in the fourth image sequence based on the instance segmentation result.

[0169] Optionally, a predicted category of any dynamic object in the fourth image sequence may be extracted from the instance segmentation result, and the predicted category may be used as the first predicted category.

[0170] Step 6406, determining a preset trajectory corresponding to the first prediction category.

[0171] Optionally, the correspondence between the category and the trajectory may be preset, so that the trajectory corresponding to the first prediction category may be efficiently and reliably determined based on the preset correspondence, and the determined trajectory may be used as the preset trajectory in step 6406 .

[0172] Step 6407, correcting the object posture sequence by minimizing the trajectory error between the motion trajectory and the preset trajectory.

[0173] Optionally, a trajectory error between the motion trajectory and a preset trajectory may be calculated, and the trajectory error may be minimized by using the above specified optimization algorithm.

[0174] The calculation of trajectory error needs to be applied to the motion trajectory, and the determination of the motion trajectory needs to be applied to the object pose sequence, that is, the object pose sequence and the trajectory error are interrelated. In this way, by minimizing the trajectory error, the object pose sequence can be optimized, and the corrected object pose sequence can be used for subsequent point cloud reconstruction, which is beneficial to improving the point cloud reconstruction effect.

[0175] Step 650 , densely reconstructing the point cloud using the first camera pose sequence, the fourth image sequence, the third pixel position correspondence, the fourth reconstructed point cloud and the corrected object pose sequence to obtain a third reconstructed point cloud.

[0176] In the case where the first camera pose sequence, the fourth image sequence, the third pixel position correspondence, the fourth reconstructed point cloud and the corrected object pose sequence are known, combined Figure 1-2The motion model shown can achieve dense reconstruction of the point cloud through the application of the MVS algorithm to obtain the third reconstructed point cloud.

[0177] In the embodiments of the present disclosure, by using the optical flow prediction model, the optical flow prediction result corresponding to the first image can be obtained efficiently and reliably, and the optical flow prediction result can be used to determine the third pixel position correspondence between the previous and next frame images. The third pixel position correspondence can be used together with the first camera pose sequence and the fourth image sequence for sparse reconstruction of the dynamic scene to obtain a fourth reconstructed point cloud and an object pose sequence. After that, the object pose sequence can be corrected in a certain manner, and the corrected object pose sequence can be used together with the first camera pose sequence, the fourth image sequence, the third pixel position correspondence and the fourth reconstructed point cloud for dense reconstruction of the dynamic scene. In this way, the accuracy and reliability of the data on which the dense reconstruction of the dynamic scene depends can be better guaranteed, which is conducive to ensuring the final scene reconstruction effect.

[0178] Figure 8 It is a flowchart of a dense reconstruction method of a dynamic scene provided by an exemplary embodiment of the present disclosure. Figure 8 The method shown includes step 810, step 820, step 830, step 840 and step 850. Optionally, the combination of step 820 to step 840 can be used as an optional implementation of step 630 above, and step 850 can be used as an optional implementation of step 650.

[0179] Step 810: Perform instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result.

[0180] It should be noted that the specific implementation of step 810 can refer to the relevant introduction to step 3304 above, which will not be repeated here.

[0181] Step 820, based on the instance segmentation result and the third pixel position correspondence, determine whether the predicted categories of the corresponding pixel positions in the previous and next frame images in the fourth image sequence are consistent, and obtain a determination result.

[0182] For the pixel position x1 in any image except the last frame in the fourth image sequence, the pixel position x2 corresponding to the pixel position x1 in the next frame of the image can be determined based on the third pixel position correspondence relationship. In addition, based on the instance segmentation result, the predicted category y1 of the pixel position x1 and the predicted category y2 of the pixel position x2 can be determined, and then the predicted category y1 and the predicted category y2 can be compared to see if they are consistent to obtain a confirmed result.

[0183] Step 830: based on the determination result, correct the third pixel position correspondence.

[0184] If the pixel position x1 and the pixel position x2 belong to the same moving object, theoretically, the prediction categories of the pixel position x1 and the pixel position x2 should be consistent. In view of this, if the determination result indicates that the prediction category y1 and the prediction category y2 are inconsistent, it can be determined that the pixel position x1 and the pixel position x2 do not actually belong to the same moving object, and the pixel position x1 and the pixel position x2 do not actually have a corresponding relationship. Therefore, the corresponding relationship between the pixel position x1 and the pixel position x2 can be deleted from the third pixel position corresponding relationship, or an invalid mark can be added to the corresponding relationship between the pixel position x1 and the pixel position x2 in the third pixel position corresponding relationship to achieve the correction of the third pixel position corresponding relationship.

[0185] Step 840 , performing sparse reconstruction of the point cloud using the first camera pose sequence, the fourth image sequence and the corrected third pixel position correspondence to obtain a fourth reconstructed point cloud and an object pose sequence of the dynamic object in the fourth image sequence.

[0186] It should be noted that the specific implementation of step 840 may be similar to the specific implementation of step 630 introduced above, the main difference being that the corrected third pixel position correspondence is used in step 840, which is conducive to improving the sparse reconstruction effect of dynamic scenes.

[0187] Step 850 , densely reconstructing the point cloud using the first camera pose sequence, the fourth image sequence, the corrected third pixel position correspondence, the fourth reconstructed point cloud and the corrected object pose sequence to obtain a third reconstructed point cloud.

[0188] It should be noted that the specific implementation of step 850 may be similar to the specific implementation of step 650 introduced above. The main difference is that the corrected third pixel position correspondence is used in step 850, which is conducive to improving the dense reconstruction effect of dynamic scenes.

[0189] In the embodiments of the present disclosure, by using the instance segmentation model, the instance segmentation results of the first image sequence can be obtained efficiently and reliably. By organically using the instance segmentation results and the third pixel position correspondence, the third pixel position correspondence can be corrected, and the corrected third pixel position correspondence is used for sparse reconstruction and dense reconstruction of the point cloud, which is beneficial to improving the reconstruction effect of dynamic scenes.

[0190] Fig. 9 It is a flowchart of a dense reconstruction method for a static scene provided by an exemplary embodiment of the present disclosure. Fig. 9The method shown includes step 910, step 920, step 930, step 940, step 950 and step 960. Optionally, a combination of step 920 to step 960 can be used as an optional implementation of step 240.

[0191] Step 910: Perform instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result.

[0192] It should be noted that the specific implementation of step 910 can refer to the relevant introduction to step 3304 above, which will not be repeated here.

[0193] Step 920 : for a second target pixel in a third target image in a third image sequence, determine a second predicted category of the second target pixel based on the instance segmentation result.

[0194] It should be noted that the third target image may be any image included in the third image sequence, and the second target pixel point may be any pixel point in the third target image.

[0195] Optionally, the predicted category of the second target pixel point may be directly extracted from the instance segmentation result, and the extracted predicted category may be used as the second predicted category.

[0196] Step 930 : Determine the size of the local region matching the second prediction category.

[0197] Optionally, the correspondence between the category and the window size may be preset, so that the window size corresponding to the second prediction category may be determined according to the preset correspondence, and the window size may be used as the local area size matching the second prediction category.

[0198] Step 940: determine the target local area from the third target image based on the pixel position of the second target pixel and the size of the local area.

[0199] Optionally, the local area size may include a width size and a height size, and then a region with the pixel position of the second target pixel point as the center and having the width size and the height size may be intercepted from the third target image as the target local area. In some cases, a region with the pixel position of the second target pixel point as the upper left corner and having the width size and the height size may also be intercepted from the third target image as the target local area.

[0200] Step 950 , for the local target area, densely reconstruct the point cloud using the first reconstructed point cloud, the first camera pose sequence, and other images in the third image sequence except the third target image, to obtain a local point cloud corresponding to the local target area.

[0201] After determining the local area of ​​the target, the first reconstructed point cloud, the first camera pose sequence and the pre-calibrated camera intrinsic parameters are used, and then the Figure 1-1 The motion model shown can find a spatial region corresponding to the local area of ​​the target in the world coordinate system, and the spatial region can be a rectangular region. Then, spatial points can be sampled in the spatial region to obtain multiple spatial points, for example, M spatial points.

[0202] For each of the M spatial points, use the first camera pose sequence and the pre-calibrated camera intrinsic parameters, and then refer to Figure 1-1 The motion model shown can determine the pixel point corresponding to the spatial point in other images except the third target image in the third image sequence, and the pixel position of the pixel point can be expressed as pixel position x3. Since the spatial region is associated with the local target region, the pixel point corresponding to the spatial point in the local target region can also be calculated, and the pixel position of the pixel point can be expressed as pixel position x4. By matching between images, the pixel position corresponding to pixel position x4 in other images can also be determined, and the pixel position can be expressed as pixel position x5. After that, by subtracting pixel position x5 from pixel position x4, the corresponding position error can be obtained. According to the position errors corresponding to each of the M spatial points, the reprojection error can be obtained by averaging. According to the obtained reprojection error, the normal vector of the spatial region and the depth values ​​of the M spatial points can be optimized. The M spatial points after the depth value is optimized can form a local point cloud corresponding to the local target region.

[0203] Step 960: Determine a second reconstructed point cloud based on the local point cloud.

[0204] The above describes a method for reconstructing a local point cloud for a local target area corresponding to a second target pixel point. According to this method, local point clouds can be reconstructed for multiple pixel points in the third image sequence, and these local point clouds can form a second reconstructed point cloud. Alternatively, after obtaining these local point clouds, these reconstructed point clouds can be optimized in a certain manner, and these optimized reconstructed point clouds can form a second reconstructed point cloud.

[0205] In the embodiments of the present disclosure, by using the instance segmentation model, the instance segmentation results of the first image sequence can be obtained efficiently and reliably. The instance segmentation results can be used to determine the second prediction category of the second target pixel in the third target image in the third image sequence. With reference to the second prediction category, the appropriate local area size can be determined, and the corresponding local area can be determined accordingly for reconstruction of the local point cloud corresponding to the second target pixel. For example, if the second target pixel is in a weak texture area such as the ground, a large window can be used to reconstruct the local point cloud to reconstruct more scene details. If the second target pixel is in a strong texture area, a small window can be used to reconstruct the local point cloud to improve the reconstruction accuracy. Therefore, the embodiments of the present disclosure can better ensure the dense reconstruction effect of static scenes.

[0206] Fig.10 It is a flowchart of a dense reconstruction method for a static scene provided by another exemplary embodiment of the present disclosure. Fig.10 The method shown includes step 1010, step 1020 and step 1030. Optionally, the combination of step 1020 and step 1030 can be used as an optional implementation of step 240.

[0207] Step 1010: Perform optical flow prediction on the first image sequence using an optical flow prediction model to obtain an optical flow prediction result.

[0208] It should be noted that the specific implementation of step 1010 can be referred to the relevant introduction to step 310 above, which will not be repeated here.

[0209] Step 1020: Determine the fifth pixel position correspondence between the previous and next frame images in the third image sequence based on the optical flow prediction result.

[0210] It should be noted that the specific implementation of step 1020 can refer to the relevant introduction to step 3301 above, which will not be repeated here.

[0211] Step 1030 , densely reconstructing the point cloud using the first reconstructed point cloud, the first camera pose sequence, the third image sequence, and the fifth pixel position correspondence to obtain a second reconstructed point cloud.

[0212] It should be noted that the specific implementation of step 1030 may be similar to the specific implementation of step 240 introduced above. The main difference is that the dense reconstruction in step 1031 also utilizes the fifth pixel position correspondence relationship. The fifth pixel position correspondence relationship can be regarded as an object tracking result obtained in a certain way. Under the guidance of the object tracking result, it is helpful to clarify which pixel points in the previous and next frames correspond to the same spatial point, so as to provide an effective reference for the reconstruction process, thereby improving the dense reconstruction effect of static scenes.

[0213] Fig.11 It is a flowchart of a dense reconstruction method of a static scene provided by yet another exemplary embodiment of the present disclosure. Fig.11 The method shown includes step 1110, step 1120 and step 1130. Optionally, the combination of step 1110 to step 1130 can be used as another optional implementation of step 240.

[0214] Step 1110 , determining a second camera pose sequence of the camera based on data collected by a non-visual sensor disposed on the movable device.

[0215] Optionally, non-visual sensors include but are not limited to positioning sensors, inertial measurement units, wheel speed meters, etc.; among which, positioning sensors include but are not limited to positioning sensors based on the Global Positioning System (GPS), positioning sensors based on the Global Navigation Satellite System (GNSS), etc.

[0216] It should be noted that, for the sensors included in the non-visual sensors and located in the chassis system of the movable device, the data collected by these sensors can be obtained from a controller area network (CAN) bus.

[0217] Assuming that the non-visual sensor includes a positioning sensor and inertial measurement data, the camera position at each time the N frames are captured can be determined based on the positioning data collected by the positioning sensor, and the camera attitude at each time the N frames are captured can be determined based on the inertial measurement data collected by the inertial measurement unit. By integrating these camera positions and camera attitudes, a second camera pose sequence can be obtained.

[0218] Step 1120 , using the second camera pose sequence as an alignment reference, performing similarity transformation alignment on the first camera pose sequence.

[0219] It should be noted that any feasible similarity transformation alignment algorithm may be used to perform similarity transformation alignment on the first camera pose sequence.

[0220] Step 1130 , densely reconstructing the point cloud using the first reconstructed point cloud, the first camera pose sequence aligned by similarity transformation, and the third image sequence to obtain a second reconstructed point cloud.

[0221] It should be noted that when the SFM algorithm is used for sparse reconstruction to obtain the first camera pose sequence, each pose in the first camera pose sequence is unscaled, that is, it does not have a real scale. Therefore, the data collected by the non-visual sensor can be used to determine the second camera pose sequence with a real scale, and the first camera pose sequence and the second camera pose sequence can be aligned using the similarity transformation alignment algorithm, so that the first camera pose sequence after similarity transformation alignment has a real scale. The first camera pose sequence after similarity transformation alignment is used for dense reconstruction of static scenes, which is conducive to ensuring the dense reconstruction effect of static scenes.

[0222] In some optional embodiments of the present disclosure, Fig.12 As shown, a forward-looking monocular video can be obtained by a forward-looking camera arranged on a movable device, and based on the forward-looking monocular video, the first image sequence mentioned above can be obtained. The first image sequence is processed by a large visual perception model including an instance segmentation model, a depth prediction model and an optical flow prediction model, and an instance segmentation result, a depth prediction result and an optical flow prediction result can be obtained. By performing background segmentation on the first image sequence, a second image sequence can be obtained. Using the second image sequence, sparse reconstruction can be performed to obtain a sparse reconstruction result of a static scene. The sparse reconstruction result of the static scene can include a first reconstruction point cloud and a first camera pose sequence. Using the data collected by a non-visual sensor arranged on the movable device, the first camera pose sequence can be aligned by similarity transformation. By performing dynamic and static segmentation on the first image sequence, a third image sequence corresponding to the static scene and a fourth image sequence corresponding to the dynamic scene can be obtained. Based on the sparse reconstruction results of the third image sequence and the static scene, the static scene can be densely reconstructed. In addition, based on the fourth image sequence, the dynamic scene can be sparsely reconstructed first, and then the dynamic scene can be densely reconstructed, and finally the scene reconstruction result corresponding to the first image sequence can be obtained.

[0223] In summary, in the embodiments of the present invention, a visual method is used to achieve scene reconstruction. Specifically, a method of dividing and conquering the easy parts first and then the difficult parts can be used to perform scene reconstruction. For example, the static scene is reconstructed first, and then the dynamic scene is reconstructed after the static scene reconstruction is completed. This is beneficial to reduce the reconstruction cost and reconstruction difficulty, and can obtain a high-precision dense point cloud.

[0224] Exemplary Devices

[0225] Fig.13 It is a structural schematic diagram of a scene reconstruction device provided by an exemplary embodiment of the present disclosure. Fig.12The device shown includes a first segmentation module 1310 , a first reconstruction module 1320 , a second segmentation module 1330 , a second reconstruction module 1340 , a third reconstruction module 1350 and a determination module 1360 .

[0226] The first segmentation module 1310 is used to perform background segmentation on a first image sequence captured by a camera disposed on a movable device to obtain a second image sequence;

[0227] A first reconstruction module 1320 is used to perform sparse reconstruction of a point cloud using the second image sequence obtained by the first segmentation module 1310 to obtain a first reconstructed point cloud and a first camera pose sequence;

[0228] The second segmentation module 1330 is used to perform static and dynamic segmentation on the first image sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene;

[0229] A second reconstruction module 1340 is used to perform dense reconstruction of a point cloud using the first reconstructed point cloud obtained by the first reconstruction module 1320 and the first camera pose sequence, and the third image sequence obtained by the second segmentation module 1330, to obtain a second reconstructed point cloud;

[0230] A third reconstruction module 1350 is used to use the first camera pose sequence obtained by the first reconstruction module 1320 and the fourth image sequence obtained by the second segmentation module 1330 to perform sparse reconstruction and dense reconstruction of the point cloud in sequence to obtain a third reconstructed point cloud;

[0231] The determination module 1360 is used to determine the scene reconstruction result corresponding to the first image sequence based on the second reconstructed point cloud obtained by the second reconstruction module 1340 and the third reconstructed point cloud obtained by the third reconstruction module 1350.

[0232] In some optional examples, such as Fig.14 As shown, the device provided by the embodiment of the present disclosure also includes:

[0233] A first prediction module 1370 is used to perform optical flow prediction on the first image sequence through an optical flow prediction model to obtain an optical flow prediction result;

[0234] A second prediction module 1380 is used to perform depth prediction on the first image sequence through a depth prediction model to obtain a depth prediction result;

[0235] The second segmentation module 1330 is specifically used to perform dynamic and static segmentation on the first image sequence based on the optical flow prediction result obtained by the first prediction module 1370, the depth prediction result obtained by the second prediction module 1380 and the first camera pose sequence obtained by the first reconstruction module 1320 to obtain a third image sequence corresponding to the static scene and a fourth image sequence corresponding to the dynamic scene.

[0236] In some optional examples, such as Fig.14 As shown, the second segmentation module 1330 includes:

[0237] A first determination submodule 13301 is used to determine a first pixel position correspondence relationship between previous and next frame images in a first image sequence based on the optical flow prediction result obtained by the first prediction module 1370;

[0238] A second determination submodule 13303 is used to determine a second pixel position correspondence relationship between the front and rear frame images in the first image sequence based on the depth prediction result obtained by the second prediction module 1380 and the first camera pose sequence obtained by the first reconstruction module 1320;

[0239] The segmentation submodule 13305 is used to perform dynamic and static segmentation on the first image sequence based on the first pixel position correspondence determined by the first determination submodule 13301 and the second pixel position correspondence determined by the second determination submodule 13303 to obtain a third image sequence corresponding to the static scene and a fourth image sequence corresponding to the dynamic scene.

[0240] In some optional examples, the segmentation submodule 13305 includes:

[0241] A first determining unit is used to determine, for a first target pixel point in a first target image in the first image sequence, a first target pixel position corresponding to the pixel position of the first target pixel point from a second target image according to the first pixel position correspondence determined by the first determining submodule 13301; wherein the second target image is a subsequent frame image of the first target image in the first image sequence;

[0242] A second determining unit, configured to determine, from the second target image, a second target pixel position corresponding to the pixel position of the first target pixel point according to the second pixel position correspondence relationship determined by the second determining submodule 13303;

[0243] A third determining unit is used to determine a pixel point attribute of a first target pixel point based on a position error between a first target pixel point determined by the first determining unit and a second target pixel point determined by the second determining unit; wherein the pixel point attribute of any pixel point is used to characterize whether the pixel point belongs to a dynamic object;

[0244] A first segmentation unit is used to perform dynamic and static segmentation on the first target image based on the pixel attributes of the first target pixel determined by the third determination unit to obtain a first segmented image corresponding to the static scene and a second segmented image corresponding to the dynamic scene;

[0245] A fourth determining unit, configured to determine a third image sequence based on the first segmented image obtained by the first segmenting unit;

[0246] The fifth determining unit is configured to determine a fourth image sequence based on the second segmented image obtained by the first segmenting unit.

[0247] In an alternative example, Fig.14 As shown, the device provided by the embodiment of the present disclosure also includes:

[0248] A third segmentation module 1390 is used to perform instance segmentation on the first image sequence by using an instance segmentation model to obtain an instance segmentation result;

[0249] The segmentation submodule 13305 includes:

[0250] a fifth determining unit, configured to determine, for a first target image in the first image sequence, an instance region of each instance in the first target image based on the instance segmentation result obtained by the third segmentation module 1390;

[0251] A sixth determining unit, configured to determine the pixel attributes of each pixel in the first target image based on the first pixel position correspondence determined by the first determining submodule 13301 and the second pixel position correspondence determined by the second determining submodule 13303; wherein the pixel attribute of any pixel is used to characterize whether the pixel belongs to a dynamic object;

[0252] a statistical unit, configured to perform statistics on the pixel attributes of each pixel in each instance region determined by the sixth determining unit for each instance region in the first target image, to obtain a statistical result;

[0253] A second segmentation unit is used to perform static and dynamic segmentation on the first target image based on the statistical results corresponding to each instance region in the first target image obtained by the statistical unit, so as to obtain a third segmented image corresponding to the static scene and a fourth segmented image corresponding to the dynamic scene;

[0254] a seventh determining unit, configured to determine a third image sequence based on the third segmented image obtained by the second segmenting unit;

[0255] The eighth determining unit is configured to determine a fourth image sequence based on the fourth segmented image obtained by the second segmenting unit.

[0256] In some optional examples, the second segmentation unit includes:

[0257] a determination subunit, configured to determine, for each instance region in the first target image, based on the statistical result corresponding to the instance region obtained by the statistical unit, a proportion of pixels in the instance region in which the pixel attribute is used to characterize that the current pixel does not belong to a dynamic object;

[0258] The segmentation subunit is used to perform dynamic and static segmentation on the first target image based on the corresponding proportion of each instance area in the first target image determined by the determination subunit, so as to obtain a third segmented image corresponding to the static scene and a fourth segmented image corresponding to the dynamic scene.

[0259] In some optional examples, such as Figure 15-1 , Figure 15-2 As shown, the device provided by the embodiment of the present disclosure also includes:

[0260] A first prediction module 1370 is used to perform optical flow prediction on the first image sequence through an optical flow prediction model to obtain an optical flow prediction result;

[0261] The third reconstruction module 1350 includes:

[0262] A third determination submodule 13501 is used to determine a third pixel position correspondence relationship between the front and rear frame images in the fourth image sequence obtained by the segmentation submodule 13305 based on the optical flow prediction result obtained by the first prediction module 1370;

[0263] The first reconstruction submodule 13503 is used to perform sparse reconstruction of the point cloud by using the first camera pose sequence obtained by the first reconstruction module 1320, the fourth image sequence obtained by the second segmentation module 1330 and the third pixel position correspondence determined by the third determination submodule 13501, so as to obtain a fourth reconstructed point cloud and an object pose sequence of the dynamic object in the fourth image sequence;

[0264] A correction submodule 13507, used to correct the object posture sequence obtained by the first reconstruction submodule 13503 according to a preset correction method;

[0265] The second reconstruction submodule 13509 is used to perform dense reconstruction of the point cloud using the first camera pose sequence obtained by the first reconstruction module 1320, the fourth image sequence obtained by the second segmentation module 1330, the third pixel position correspondence determined by the third determination submodule 13501, the fourth reconstructed point cloud obtained by the first reconstruction submodule 13503 and the corrected object pose sequence obtained by the correction submodule 13507 to obtain a third reconstructed point cloud.

[0266] In some optional examples, such as Figure 15-1 As shown, the device provided by the embodiment of the present disclosure also includes:

[0267] A second prediction module 1380 is used to perform depth prediction on the first image sequence through a depth prediction model to obtain a depth prediction result;

[0268] Modify submodule 13507, including:

[0269] a ninth determining unit, configured to determine a fourth pixel position correspondence relationship between the front and rear frame images in the fourth image sequence obtained by the second segmentation module 1330 based on the depth prediction result obtained by the second prediction module 1380, the first camera pose sequence obtained by the first reconstruction module 1320, and the object pose sequence obtained by the first reconstruction submodule 13503;

[0270] a tenth determining unit, configured to determine a reprojection error based on the third pixel position correspondence relationship determined by the third determining submodule 13501 and the fourth pixel position correspondence relationship determined by the ninth determining unit;

[0271] The first correction sub-unit is used to correct the object posture sequence obtained by the first reconstruction sub-module 13503 by minimizing the reprojection error determined by the tenth determination unit.

[0272] In some optional examples, such as Figure 15-2 As shown, the device provided by the embodiment of the present disclosure also includes:

[0273] A third segmentation module 1390 is used to perform instance segmentation on the first image sequence by using an instance segmentation model to obtain an instance segmentation result;

[0274] Modify submodule 13507, including:

[0275] an eleventh determination unit, configured to determine a motion trajectory of a dynamic object in the fourth image sequence obtained by the second segmentation module 1330 based on the object pose sequence obtained by the first reconstruction submodule 13503;

[0276] a twelfth determining unit, configured to determine, based on the instance segmentation result obtained by the third segmentation module 1390, a first predicted category of the dynamic object in the fourth image sequence obtained by the second segmentation module 1330;

[0277] a thirteenth determining unit, configured to determine a preset trajectory corresponding to the first prediction category determined by the twelfth determining unit;

[0278] The second correction sub-unit is used to correct the object posture sequence obtained by the first reconstruction sub-module 13503 by minimizing the trajectory error between the motion trajectory determined by the eleventh determination unit and the preset trajectory determined by the thirteenth determination unit.

[0279] In some optional examples, such as Fig.16As shown, the device provided by the embodiment of the present disclosure also includes:

[0280] A third segmentation module 1390 is used to perform instance segmentation on the first image sequence by using an instance segmentation model to obtain an instance segmentation result;

[0281] The first reconstruction submodule 13503 includes:

[0282] A fourteenth determining unit is used to determine whether the predicted categories of corresponding pixel positions in the previous and next frame images in the fourth image sequence obtained by the second segmentation module 1330 are consistent based on the instance segmentation result obtained by the third segmentation module 1390 and the third pixel position correspondence determined by the third determination submodule 13501, and obtain a determination result;

[0283] a correction unit, configured to correct the third pixel position correspondence relationship determined by the third determination submodule 13501 based on the determination result obtained by the fourteenth determination unit;

[0284] A reconstruction unit, configured to perform sparse reconstruction of the point cloud by using the first camera pose sequence obtained by the first reconstruction module 1320, the fourth image sequence obtained by the second segmentation module 1330, and the corrected third pixel position correspondence obtained by the correction unit, to obtain a fourth reconstructed point cloud and an object pose sequence of the dynamic object in the fourth image sequence;

[0285] The second reconstruction submodule 13509 is specifically used to perform dense reconstruction of the point cloud using the first camera pose sequence, the fourth image sequence, the corrected third pixel position correspondence, the fourth reconstructed point cloud and the corrected object pose sequence to obtain a third reconstructed point cloud.

[0286] In some optional examples, such as Fig.16 As shown, the device provided by the embodiment of the present disclosure also includes:

[0287] A third segmentation module 1390 is used to perform instance segmentation on the first image sequence by using an instance segmentation model to obtain an instance segmentation result;

[0288] The second reconstruction module 1340 includes:

[0289] A fifth determination submodule 13401 is used to determine, for a second target pixel in the third target image in the third image sequence obtained by the second segmentation module 1330, a second prediction category of the second target pixel based on the instance segmentation result obtained by the third segmentation module 1390;

[0290] A sixth determining submodule 13403, configured to determine a size of a local region matching the second prediction category determined by the fifth determining submodule 13401;

[0291] A seventh determination submodule 13405 is used to determine a target local area from the third target image based on the pixel position of the second target pixel and the local area size determined by the sixth determination submodule 13403;

[0292] The third reconstruction submodule 13407 is used to perform dense reconstruction of the point cloud for the target local area determined by the seventh determination submodule 13405, using the first reconstructed point cloud obtained by the first reconstruction module 1320, the first camera pose sequence obtained by the first reconstruction module 1320, and other images in the third image sequence obtained by the second segmentation module 1330 except the third target image, to obtain a local point cloud corresponding to the target local area;

[0293] The eighth determination submodule 13409 is used to determine the second reconstructed point cloud based on the local point cloud obtained by the third reconstruction submodule 13407.

[0294] In some optional examples, such as Fig.17 As shown, the device provided by the embodiment of the present disclosure also includes:

[0295] A first prediction module 1370 is used to perform optical flow prediction on the first image sequence through an optical flow prediction model to obtain an optical flow prediction result;

[0296] The second reconstruction module 1340 includes:

[0297] A ninth determination submodule 13411, configured to determine a fifth pixel position correspondence relationship between the front and rear frame images in the third image sequence obtained by the second segmentation module 1330 based on the optical flow prediction result obtained by the first prediction module 1370;

[0298] The fourth reconstruction submodule 13413 is used to perform dense reconstruction of the point cloud using the first reconstructed point cloud obtained by the first reconstruction module 1320, the first camera pose sequence obtained by the first reconstruction module 1320, the third image sequence obtained by the second segmentation module 1330 and the fifth pixel position correspondence determined by the ninth determination submodule 13411 to obtain a second reconstructed point cloud.

[0299] In some optional examples, such as Fig.18 As shown, the second reconstruction module 1340 includes:

[0300] A tenth determination submodule 13415 is used to determine a second camera pose sequence of the camera based on data collected by a non-visual sensor disposed on the movable device;

[0301] A similarity transformation alignment submodule 13417 is used to perform similarity transformation alignment on the first camera pose sequence obtained by the first reconstruction module 1320 by using the second camera pose sequence determined by the tenth determination submodule 13415 as an alignment reference;

[0302] The fifth reconstruction submodule 13419 is used to perform dense reconstruction of the point cloud using the first reconstructed point cloud obtained by the first reconstruction module 1320, the first camera pose sequence obtained by the similarity transformation alignment submodule 13417 after similarity transformation alignment, and the third image sequence obtained by the second segmentation module 1330 to obtain a second reconstructed point cloud.

[0303] In the device of the present disclosure, the various optional embodiments, optional implementation methods and optional examples disclosed above can be flexibly selected and combined as needed to achieve corresponding functions and effects, and the present disclosure does not list them one by one.

[0304] The beneficial technical effects corresponding to the exemplary embodiment of the present device can be found in the corresponding beneficial technical effects of the above exemplary method section, which will not be repeated here.

[0305] Exemplary Electronic Devices

[0306] Fig.19 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated. The electronic device 1900 includes one or more processors 1910 and a memory 1920 .

[0307] The processor 1910 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1900 to perform desired functions.

[0308] The memory 1920 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1910 may execute one or more computer program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0309] In one example, the electronic device 1900 may further include: an input device 1930 and an output device 1940 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0310] The input device 1930 may also include, for example, a keyboard, a mouse, etc.

[0311] The output device 1940 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0312] Of course, to simplify, Fig.19 Only some of the components related to the present disclosure in the electronic device 1900 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application scenarios, the electronic device 1900 may also include any other appropriate components.

[0313] Exemplary computer program products and computer-readable storage media

[0314] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the scene reconstruction method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0315] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0316] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the scene reconstruction method according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0317] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0318] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations. It cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present disclosure. The specific details disclosed above are only for the purpose of illustration and ease of understanding, not limitation. The above details do not limit the present disclosure to being implemented by adopting the above specific details.

[0319] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A scene reconstruction method, comprising: Performing background segmentation on a first image sequence captured by a camera disposed on a movable device to obtain a second image sequence; Using the second image sequence, performing sparse reconstruction of the point cloud to obtain a first reconstructed point cloud and a first camera pose sequence of the camera; Performing static and dynamic segmentation on the first image sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene; Using the first reconstructed point cloud, the first camera pose sequence and the third image sequence, densely reconstruct the point cloud to obtain a second reconstructed point cloud; Using the first camera pose sequence and the fourth image sequence, sequentially performing sparse reconstruction and dense reconstruction of the point cloud to obtain a third reconstructed point cloud; A scene reconstruction result corresponding to the first image sequence is determined based on the second reconstructed point cloud and the third reconstructed point cloud.

2. The method according to claim 1, wherein: The method further comprises: Performing optical flow prediction on the first image sequence using an optical flow prediction model to obtain an optical flow prediction result; Performing depth prediction on the first image sequence using a depth prediction model to obtain a depth prediction result; The step of performing static and dynamic segmentation on the first image sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene includes: Based on the optical flow prediction result, the depth prediction result and the first camera pose sequence, the first image sequence is segmented into static and dynamic images to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene.

3. The method according to claim 2, wherein: The step of performing static and dynamic segmentation on the first image sequence based on the optical flow prediction result, the depth prediction result and the first camera pose sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene includes: Based on the optical flow prediction result, determining a first pixel position correspondence relationship between previous and next frame images in the first image sequence; Determine a second pixel position correspondence relationship between front and rear frame images in the first image sequence based on the depth prediction result and the first camera pose sequence; Based on the first pixel position correspondence relationship and the second pixel position correspondence relationship, the first image sequence is segmented into static and dynamic scenes to obtain a third image sequence corresponding to static scenes and a fourth image sequence corresponding to dynamic scenes.

4. The method according to claim 3, wherein: The step of performing static-dynamic segmentation on the first image sequence based on the first pixel position correspondence relationship and the second pixel position correspondence relationship to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene includes: For a first target pixel point in a first target image in the first image sequence, determining a first target pixel position corresponding to the pixel position of the first target pixel point from a second target image according to the first pixel position correspondence relationship; wherein the second target image is an image frame subsequent to the first target image in the first image sequence; According to the second pixel position correspondence relationship, determining a second target pixel position corresponding to the pixel position of the first target pixel point from the second target image; Determining a pixel attribute of the first target pixel based on a position error between the first target pixel position and the second target pixel position; wherein the pixel attribute of any pixel is used to characterize whether the pixel belongs to a dynamic object; Based on the pixel attributes of the first target pixel points, the first target image is segmented into static and dynamic images to obtain a first segmented image corresponding to a static scene and a second segmented image corresponding to a dynamic scene; Based on the first segmented image, determining the third image sequence; Based on the second segmented image, the fourth image sequence is determined.

5. The method according to claim 3, wherein: The method further comprises: Performing instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result; The step of performing static-dynamic segmentation on the first image sequence based on the first pixel position correspondence relationship and the second pixel position correspondence relationship to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene includes: For a first target image in the first image sequence, determining, based on the instance segmentation result, an instance region of each instance in the first target image; Based on the first pixel position correspondence relationship and the second pixel position correspondence relationship, determining the pixel point attribute of each pixel point in the first target image; wherein the pixel point attribute of any pixel point is used to indicate whether the pixel point belongs to a dynamic object; For each instance region in the first target image, statistics are performed on the pixel attributes of each pixel in the instance region to obtain a statistical result; Based on the statistical results corresponding to each instance region in the first target image, the first target image is segmented into static and dynamic images to obtain a third segmented image corresponding to the static scene and a fourth segmented image corresponding to the dynamic scene; Based on the third segmented image, determining the third image sequence; The fourth image sequence is determined based on the fourth segmented image.

6. The method according to claim 5, wherein: The step of performing static and dynamic segmentation on the first target image based on the statistical results corresponding to each instance region in the first target image to obtain a third segmented image corresponding to a static scene and a fourth segmented image corresponding to a dynamic scene includes: For each instance region in the first target image, based on the statistical result corresponding to the instance region, determining the proportion of pixels in the instance region whose attributes are used to characterize that the current pixel does not belong to a dynamic object; Based on the proportion corresponding to each instance area in the first target image, the first target image is segmented into static and dynamic scenes to obtain a third segmented image corresponding to the static scene and a fourth segmented image corresponding to the dynamic scene.

7. The method according to claim 1, wherein: The method further comprises: Performing optical flow prediction on the first image sequence using an optical flow prediction model to obtain an optical flow prediction result; The step of using the first camera pose sequence and the fourth image sequence to sequentially perform sparse reconstruction and dense reconstruction of the point cloud to obtain a third reconstructed point cloud includes: Based on the optical flow prediction result, determining a third pixel position correspondence relationship between the front and rear frame images in the fourth image sequence; Performing sparse reconstruction of a point cloud by using the first camera pose sequence, the fourth image sequence and the third pixel position correspondence to obtain a fourth reconstructed point cloud and an object pose sequence of the dynamic object in the fourth image sequence; Correcting the position and posture sequence of the object according to a preset correction method; Dense reconstruction of the point cloud is performed using the first camera pose sequence, the fourth image sequence, the third pixel position correspondence, the fourth reconstructed point cloud and the corrected object pose sequence to obtain the third reconstructed point cloud.

8. The method according to claim 7, wherein: The method further comprises: Performing depth prediction on the first image sequence using a depth prediction model to obtain a depth prediction result; The step of correcting the object posture sequence according to a preset correction method includes: Determine a fourth pixel position correspondence relationship between previous and next frame images in the fourth image sequence based on the depth prediction result, the first camera pose sequence, and the object pose sequence; Determining a reprojection error based on the third pixel position correspondence relationship and the fourth pixel position correspondence relationship; The object pose sequence is corrected by minimizing the reprojection error.

9. The method according to claim 7, wherein: The method further comprises: Performing instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result; The step of correcting the object posture sequence according to a preset correction method includes: Determining a motion trajectory of a dynamic object in the fourth image sequence based on the object posture sequence; Determining a first predicted category of a dynamic object in the fourth image sequence based on the instance segmentation result; Determining a preset trajectory corresponding to the first prediction category; The object posture sequence is corrected by minimizing the trajectory error between the motion trajectory and the preset trajectory.

10. The method according to claim 7, wherein: The method further comprises: Performing instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result; The method of performing sparse reconstruction of a point cloud by using the first camera pose sequence, the fourth image sequence and the third pixel position correspondence to obtain a fourth reconstructed point cloud and an object pose sequence of the dynamic object in the fourth image sequence comprises: Based on the instance segmentation result and the third pixel position correspondence relationship, determining whether the predicted categories of corresponding pixel positions in the previous and next frame images in the fourth image sequence are consistent, and obtaining a determination result; Based on the determination result, correcting the third pixel position correspondence relationship; Performing sparse reconstruction of a point cloud by using the first camera pose sequence, the fourth image sequence and the corrected third pixel position correspondence to obtain a fourth reconstructed point cloud and an object pose sequence of the dynamic object in the fourth image sequence; The method of performing dense reconstruction of a point cloud by using the first camera pose sequence, the fourth image sequence, the third pixel position correspondence, the fourth reconstructed point cloud and the corrected object pose sequence to obtain the third reconstructed point cloud comprises: Dense reconstruction of the point cloud is performed using the first camera pose sequence, the fourth image sequence, the corrected third pixel position correspondence, the fourth reconstructed point cloud and the corrected object pose sequence to obtain the third reconstructed point cloud.

11. The method according to claim 1, wherein: The method further comprises: Performing instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result; The method of performing dense reconstruction of a point cloud using the first reconstructed point cloud, the first camera pose sequence, and the third image sequence to obtain a second reconstructed point cloud includes: For a second target pixel in a third target image in the third image sequence, determining a second predicted category of the second target pixel based on the instance segmentation result; determining a size of a local region matching the second prediction category; Determining a target local area from the third target image based on a pixel position of the second target pixel and a size of the local area; For the local target area, densely reconstruct the point cloud by using the first reconstructed point cloud, the first camera pose sequence, and other images in the third image sequence except the third target image, to obtain a local point cloud corresponding to the local target area; Based on the local point cloud, the second reconstructed point cloud is determined.

12. A scene reconstruction device, comprising: A first segmentation module is used to perform background segmentation on a first image sequence captured by a camera disposed on a movable device to obtain a second image sequence; A first reconstruction module, configured to perform sparse reconstruction of a point cloud using the second image sequence obtained by the first segmentation module to obtain a first reconstructed point cloud and a first camera pose sequence; A second segmentation module is used to perform dynamic and static segmentation on the first image sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene; A second reconstruction module is used to perform dense reconstruction of the point cloud using the first reconstructed point cloud obtained by the first reconstruction module and the first camera pose sequence, and the third image sequence obtained by the second segmentation module to obtain a second reconstructed point cloud; A third reconstruction module is used to use the first camera pose sequence obtained by the first reconstruction module and the fourth image sequence obtained by the second segmentation module to perform sparse reconstruction and dense reconstruction of the point cloud in sequence to obtain a third reconstructed point cloud; A determination module is used to determine a scene reconstruction result corresponding to the first image sequence based on the second reconstructed point cloud obtained by the second reconstruction module and the third reconstructed point cloud obtained by the third reconstruction module.

13. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the scene reconstruction method described in any one of claims 1 to 11.

14. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the scene reconstruction method described in any one of claims 1-11 above.

Images