Scene reconstruction method and apparatus, and storage medium and electronic device

Abstract

Disclosed in the embodiments of the present disclosure are a scene reconstruction method and apparatus, and a storage medium and an electronic device. The method comprises: performing background segmentation on a first image sequence which is collected by a camera provided on a mobile device, so as to obtain a second image sequence; using the second image sequence to perform sparse reconstruction of a point cloud, so as to obtain a first reconstructed point cloud and a first camera pose sequence of the camera; performing dynamic-static segmentation on the first image sequence, so as to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene; using the first reconstructed point cloud, the first camera pose sequence and the third image sequence to perform dense reconstruction of the point cloud, so as to obtain a second reconstructed point cloud; using the first camera pose sequence and the fourth image sequence to sequentially perform sparse reconstruction and dense reconstruction of the point cloud, so as to obtain a third reconstructed point cloud; and on the basis of the second reconstructed point cloud and the third reconstructed point cloud, determining a scene reconstruction result corresponding to the first image sequence. The present disclosure can reduce the implementation cost of scene reconstruction.

Description

Scene reconstruction method, device, storage medium and electronic device

[0001] This disclosure claims priority to Chinese patent application number CN202310806421.2 filed with the State Intellectual Property Office on June 30, 2023, entitled “Scene Reconstruction Method, Device, Storage Medium and Electronic Device,” the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The present disclosure relates to driving technology, and in particular to a scene reconstruction method, device, storage medium, and electronic device. Background Art

[0003] In some cases, scene reconstruction is necessary, such as for autonomous driving in outdoor environments. The results of scene reconstruction are invaluable for scene reproduction and data annotation. Currently, scene reconstruction often relies on lidar.

[0004] Summary of the Invention

[0005] Current scene reconstruction relies on lidar, which is very expensive to implement.

[0006] In order to solve the above technical problems, the present disclosure provides a scene reconstruction method, device, medium and electronic device, which can reduce the implementation cost of scene reconstruction.

[0007] According to one aspect of an embodiment of the present disclosure, a scene reconstruction method is provided, comprising:

[0008] Perform background segmentation on a first image sequence captured by a camera set on a movable device to obtain a second image sequence; use the second image sequence to perform sparse reconstruction of a point cloud to obtain a first reconstructed point cloud and a first camera pose sequence of the camera; perform dynamic and static segmentation on the first image sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene; use the first reconstructed point cloud, the first camera pose sequence and the third image sequence to perform dense reconstruction of the point cloud to obtain a second reconstructed point cloud; use the first camera pose sequence and the fourth image sequence to perform sparse reconstruction and dense reconstruction of the point cloud in sequence to obtain a third reconstructed point cloud; based on the second reconstructed point cloud and the third reconstructed point cloud, determine the scene reconstruction result corresponding to the first image sequence.

[0009] According to another aspect of an embodiment of the present disclosure, a scene reconstruction device is provided, comprising:

[0010] A first segmentation module is used to perform background segmentation on a first image sequence captured by a camera provided on a movable device to obtain a second image sequence; a first reconstruction module is used to perform sparse reconstruction of a point cloud using the second image sequence obtained by the first segmentation module to obtain a first reconstructed point cloud and a first camera pose sequence; a second segmentation module is used to perform dynamic and static segmentation on the first image sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene; a second reconstruction module is used to perform dense reconstruction of a point cloud using the first reconstructed point cloud and the first camera pose sequence obtained by the first reconstruction module, and the third image sequence obtained by the second segmentation module to obtain a second reconstructed point cloud; a third reconstruction module is used to perform sparse reconstruction and dense reconstruction of a point cloud in sequence using the first camera pose sequence obtained by the first reconstruction module and the fourth image sequence obtained by the second segmentation module to obtain a third reconstructed point cloud; a determination module is used to determine a scene reconstruction result corresponding to the first image sequence based on the second reconstructed point cloud obtained by the second reconstruction module and the third reconstructed point cloud obtained by the third reconstruction module.

[0011] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned scene reconstruction method.

[0012] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device including:

[0013] processor;

[0014] a memory for storing instructions executable by the processor;

[0015] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above-mentioned scene reconstruction method.

[0016] According to another aspect of the embodiments of the present disclosure, a computer program product is provided. When instructions in the computer program product are executed by a processor, the above-mentioned scene reconstruction method is performed.

[0017] Based on the scene reconstruction method, device, medium, and electronic device provided by the above-mentioned embodiments of the present disclosure, the multi-view images (i.e., the first image sequence) captured by the camera can be segmented for background, and the second image sequence obtained by the background segmentation can be used to perform sparse reconstruction of the point cloud to obtain a first reconstructed point cloud and a first camera pose sequence. In addition, the multi-view images can be segmented for static and dynamic scenes to obtain a third image sequence corresponding to the static scene and a fourth image sequence corresponding to the dynamic scene. The third image sequence can be used together with the first reconstructed point cloud and the first camera pose sequence for dense reconstruction of the point cloud to obtain a second reconstructed point cloud that can effectively represent the static scene. The fourth image sequence can be used together with the first camera pose sequence for sparse and dense reconstruction of the point cloud to obtain a third reconstructed point cloud that can effectively represent the dynamic scene. The scene reconstruction result corresponding to the first image sequence can be determined by combining the second and third reconstructed point clouds. It can be seen that in the embodiments of the present disclosure, scene reconstruction is achieved by a series of processing of the multi-view images captured by the camera. Compared with lidar, the price of the camera is relatively low, so the implementation cost of scene reconstruction can be reduced. In addition, in the embodiments of the present disclosure, a divide-and-conquer strategy is adopted for static scenes and dynamic scenes, that is, static scenes and dynamic scenes are reconstructed separately, which is conducive to reducing the reconstruction difficulty and improving the reconstruction speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1-1 is a schematic diagram of the motion model of a static scene.

[0019] Figure 1-2 is a schematic diagram of the motion model of a dynamic scene.

[0020] FIG2 is a flowchart of a scene reconstruction method provided by an exemplary embodiment of the present disclosure.

[0021] FIG3 is a flowchart of a dynamic and static segmentation method provided by an exemplary embodiment of the present disclosure.

[0022] FIG4 is a second flowchart of a dynamic and static segmentation method provided by an exemplary embodiment of the present disclosure.

[0023] FIG5-1 is a third flowchart of a dynamic and static segmentation method provided by an exemplary embodiment of the present disclosure.

[0024] FIG5-2 is a fourth flowchart of a dynamic and static segmentation method provided by an exemplary embodiment of the present disclosure.

[0025] FIG6 is a flowchart of a method for reconstructing a dynamic scene provided by an exemplary embodiment of the present disclosure.

[0026] FIG7-1 is a flowchart of a method for correcting an object pose sequence provided by an exemplary embodiment of the present disclosure.

[0027] FIG7-2 is a flowchart of a method for correcting an object pose sequence provided by another exemplary embodiment of the present disclosure.

[0028] FIG8 is a flowchart of a dense reconstruction method for a dynamic scene provided by an exemplary embodiment of the present disclosure.

[0029] FIG9 is a flowchart of a dense reconstruction method for a static scene provided by an exemplary embodiment of the present disclosure.

[0030] FIG10 is a flowchart of a dense reconstruction method for a static scene provided by another exemplary embodiment of the present disclosure.

[0031] FIG11 is a flowchart of a dense reconstruction method for a static scene provided by yet another exemplary embodiment of the present disclosure.

[0032] FIG12 is a flowchart of a scene reconstruction method provided by another exemplary embodiment of the present disclosure.

[0033] FIG13 is a schematic structural diagram of a scene reconstruction device provided by an exemplary embodiment of the present disclosure.

[0034] FIG14 is a schematic structural diagram of a scene reconstruction device provided by another exemplary embodiment of the present disclosure.

[0035] FIG15-1 is a schematic structural diagram of a scene reconstruction device provided by yet another exemplary embodiment of the present disclosure.

[0036] FIG15-2 is a schematic structural diagram of a scene reconstruction device provided by yet another exemplary embodiment of the present disclosure.

[0037] FIG16 is a schematic structural diagram of a scene reconstruction device provided by yet another exemplary embodiment of the present disclosure.

[0038] FIG17 is a schematic structural diagram of a scene reconstruction device provided by yet another exemplary embodiment of the present disclosure.

[0039] FIG18 is a schematic structural diagram of a scene reconstruction device provided by yet another exemplary embodiment of the present disclosure.

[0040] FIG19 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0042] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0043] Application Overview

[0044] Laser radar (LiDAR) is a radar system that detects the target's position, speed, direction, posture, shape and other characteristic quantities by emitting laser beams.

[0045] LiDAR can be used for scene reconstruction, such as in autonomous driving outdoor scenes. However, LiDAR is expensive, which leads to high implementation costs. How to reduce the implementation cost of scene reconstruction is a question worth considering for those skilled in the art.

[0046] Exemplary Systems

[0047] The following text involves multiple models, such as the optical flow prediction model, the depth prediction model, and the instance segmentation model, as well as the motion model for static scenes and the motion model for dynamic scenes. In order to facilitate the understanding of the following text, these models are briefly introduced here.

[0048] The optical flow prediction model, the depth prediction model, and the instance segmentation model can all be convolutional neural network (CNN) models. A frame of image is provided as input to the optical flow prediction model, and the optical flow prediction model can predict the optical flow value of each pixel in the frame of image; wherein, the optical flow value of any pixel can be used to indicate the pixel position corresponding to the pixel position of the pixel in the next frame of image. A frame of image is provided as input to the depth prediction model, and the depth prediction model can predict the depth value of each pixel in the frame of image; wherein, the depth value of any pixel can be used to determine the spatial position of the pixel in the camera coordinate system. A frame of image is provided as input to the instance segmentation model, and the instance segmentation model can predict the category of each pixel in the frame of image and locate the different instances in the frame of image.

[0049] The motion model of a static scene can be a mathematical model used to describe the motion law of objects in a static scene. The motion model of a static scene can be shown in Figure 1-1. The motion model of a dynamic scene can be a mathematical model used to describe the motion law of objects in a dynamic scene. The motion law of a dynamic scene can be shown in Figure 1-2. As shown in Figure 1-1, the camera is moving, and the optical center of the camera at time t-1, time t, and time t+1 are respectively O t-1 , O t , O t+1 , the object (such as lane lines, buildings, etc.) is stationary, and the kth point on the object is always represented by point P k Indicates that in the images captured by the camera at time t-1, time t, and time t+1, the projection points of the kth point on the object are The main difference between Figure 1-2 and Figure 1-1 is that both the camera and the object (such as a vehicle, pedestrian, etc.) are moving. The kth point on the object at time t-1, time t, and time t+1 can be expressed as The posture of the object at time t-1, time t, and time t+1 can be expressed as

[0050] Exemplary Methods

[0051] Figure 2 is a flow chart of a scene reconstruction method provided by an exemplary embodiment of the present disclosure. The method shown in Figure 2 may include steps 210, 220, 230, 240, 250, and 260, each of which is described below.

[0052] Step 210 : performing background segmentation on a first image sequence captured by a camera disposed on a mobile device to obtain a second image sequence.

[0053] Optionally, the movable device includes but is not limited to vehicles, trains, ships, etc.; the cameras provided on the movable device include but are not limited to front-view cameras, rear-view cameras, side-view cameras, etc.

[0054] The camera provided on the mobile device can capture images at a certain frame rate to obtain a first image sequence including multiple captured frames. For ease of understanding, the embodiments of the present disclosure are described using the case where the multiple captured frames in the first image sequence are N captured frames as an example.

[0055] For each of the N frames of acquired images included in the first image sequence, the acquired image can be instance segmented using an instance segmentation model to determine the predicted category of each pixel in the acquired image, thereby determining which areas in the acquired image are areas where objects that may move (such as vehicles, pedestrians, etc.) are located, so as to generate masks in these areas, thereby removing objects that may move in the acquired image and retaining only completely stationary objects (such as the ground, walls, buildings, trees, etc.), thereby segmenting the background image from the acquired image.

[0056] According to the method described in the previous paragraph, N frames of background images corresponding to the N frames of captured images can be obtained, and the N frames of background images are arranged in time sequence to form a second image sequence.

[0057] The above is how the instance segmentation model is used to segment the background of the first image sequence, thereby obtaining the second image sequence. In some cases, the semantic segmentation model can also be used to separate the foreground and background of each frame captured in the first image sequence, thereby obtaining the second image sequence.

[0058] In an optional example, step 210 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the first segmentation module executed by the processor.

[0059] Step 220 : Utilize the second image sequence to perform sparse reconstruction of the point cloud to obtain a first reconstructed point cloud and a first camera pose sequence of the camera.

[0060] For each background image in the N background images included in the second image sequence, a key point detection algorithm may be used to perform key point detection on the background image; wherein the key point detection algorithm includes but is not limited to a scale-invariant feature transform (SIFT) algorithm, a deep learning-based algorithm, and the like.

[0061] When the key points of each of N frames of background images are known, key point matching and screening can be performed. Combined with the use of the incremental Structure From Motion (SFM) algorithm, the camera pose at the acquisition time corresponding to each of the N frames of acquired images can be determined, obtaining N camera poses. Arranging the N camera poses in time sequence can form a first camera pose sequence. In addition, several three-dimensional points in the world coordinate system can be determined to form the first reconstructed point cloud.

[0062] Of course, the algorithm used to generate the first reconstructed point cloud and the first camera pose sequence is not limited to the incremental SFM algorithm, but can also be a full SFM algorithm, a hybrid SFM algorithm, etc., which are not listed here one by one.

[0063] In an optional example, step 220 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the first reconstruction module executed by the processor.

[0064] Step 230 : performing static and dynamic segmentation on the first image sequence to obtain a third image sequence corresponding to static scenes and a fourth image sequence corresponding to dynamic scenes.

[0065] For each of the N acquired images included in the first image sequence, it is possible to determine which objects in the acquired image are actually moving and which are not moving. This allows the acquired image to be segmented into a segmented image that includes only objects that have moved and a segmented image that includes only objects that have not moved, thereby achieving dynamic and static segmentation of the acquired image. The segmented image that includes only objects that have moved can be considered a segmented image corresponding to a dynamic scene. The segmented image that includes only objects that have not moved can be considered a segmented image corresponding to a static scene.

[0066] According to the method introduced in the previous paragraph, the N frames of captured images can be segmented into static and dynamic images to obtain N frames of segmented images corresponding to static scenes and N frames of segmented images corresponding to dynamic scenes. The N frames of segmented images corresponding to static scenes are arranged in time sequence to form a third image sequence. The N frames of segmented images corresponding to dynamic scenes are arranged in time sequence to form a fourth image sequence.

[0067] In an optional example, step 230 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the second segmentation module executed by the processor.

[0068] Step 240 : Perform dense reconstruction of the point cloud using the first reconstructed point cloud, the first camera pose sequence, and the third image sequence to obtain a second reconstructed point cloud.

[0069] When the first reconstructed point cloud, the first camera pose sequence and the third image sequence are known, combined with the motion model shown in Figure 1-1, the multi-view stereo vision (MVS) algorithm can be used to find pixels with photometric consistency in different images in the third image sequence, and stereo matching can be performed on this basis to determine several points in the world coordinate system to form the second reconstructed point cloud. The second reconstructed point cloud is denser than the first reconstructed point cloud, and the second reconstructed point cloud can effectively represent static scenes.

[0070] In an optional example, step 240 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by a second reconstruction module executed by the processor.

[0071] Step 250 : Using the first camera pose sequence and the fourth image sequence, perform sparse reconstruction and dense reconstruction of the point cloud in sequence to obtain a third reconstructed point cloud.

[0072] When the first camera pose sequence and the fourth image sequence are known, sparse reconstruction can be performed using the SFM algorithm first, and then dense reconstruction can be performed using the MVS algorithm to obtain a third reconstructed point cloud, which can effectively represent dynamic scenes.

[0073] In an optional example, step 250 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by a third reconstruction module executed by the processor.

[0074] Step 260 : Determine a scene reconstruction result corresponding to the first image sequence based on the second reconstructed point cloud and the third reconstructed point cloud.

[0075] Optionally, the second reconstructed point cloud and the third reconstructed point cloud can be modeled separately to obtain a first three-dimensional model corresponding to the second reconstructed point cloud and a second three-dimensional model corresponding to the third reconstructed point cloud. Texture mapping can then be performed on the first three-dimensional model and the second three-dimensional model, respectively. The scene reconstruction result corresponding to the first image sequence can include the textured first three-dimensional model and the textured second three-dimensional model. In some embodiments, the scene reconstruction result corresponding to the first image sequence can also include the motion trajectory of the movable device, the motion trajectory of objects moving around the movable device, etc.

[0076] In an optional example, step 260 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by a determination module executed by the processor.

[0077] In an embodiment of the present disclosure, the multi-view images (i.e., the first image sequence) captured by the camera can be segmented for background, and the second image sequence obtained by the background segmentation can be used to perform sparse reconstruction of the point cloud to obtain a first reconstructed point cloud and a first camera pose sequence. In addition, the multi-view images can be segmented for static and dynamic scenes to obtain a third image sequence corresponding to the static scene and a fourth image sequence corresponding to the dynamic scene. The third image sequence can be used together with the first reconstructed point cloud and the first camera pose sequence for dense reconstruction of the point cloud to obtain a second reconstructed point cloud that can effectively represent the static scene. The fourth image sequence can be used together with the first camera pose sequence for sparse reconstruction and dense reconstruction of the point cloud to obtain a third reconstructed point cloud that can effectively represent the dynamic scene. The scene reconstruction result corresponding to the first image sequence can be determined by combining the second reconstructed point cloud and the third reconstructed point cloud. It can be seen that in the embodiment of the present disclosure, scene reconstruction is achieved by a series of processing of the multi-view images captured by the camera. Compared with lidar, the price of the camera is very low, so the implementation cost of scene reconstruction can be reduced. In addition, in the embodiments of the present disclosure, a divide-and-conquer strategy is adopted for static scenes and dynamic scenes, that is, static scenes and dynamic scenes are reconstructed separately, which is conducive to reducing the reconstruction difficulty and improving the reconstruction speed.

[0078] Figure 3 is a flow chart of a dynamic and static segmentation method provided by an exemplary embodiment of the present disclosure. The method shown in Figure 3 may include steps 310, 320, and 330, each of which is described below.

[0079] Step 310 : Perform optical flow prediction on the first image sequence using an optical flow prediction model to obtain an optical flow prediction result.

[0080] For each of the N frames of acquired images included in the first image sequence, the acquired image can be provided as input to the optical flow prediction model, and the optical flow prediction model can perform calculations based on this to obtain the optical flow value of each pixel point in the acquired image. The optical flow value of each pixel point in the acquired image can constitute the optical flow prediction data corresponding to the acquired image.

[0081] According to the method described in the previous paragraph, the optical flow prediction data corresponding to each of the N frames of acquired images can be obtained, and the optical flow prediction data corresponding to each of the N frames of acquired images can constitute the optical flow prediction result corresponding to the first image sequence.

[0082] In an optional example, step 310 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the first prediction module executed by the processor.

[0083] Step 320 : Perform depth prediction on the first image sequence using a depth prediction model to obtain a depth prediction result.

[0084] For each of the N acquired images included in the first image sequence, the acquired image can be provided as input to a depth prediction model, which can perform calculations based on the acquired image to obtain a depth value for each pixel in the acquired image. The depth values ​​corresponding to each pixel in the acquired image can constitute depth prediction data corresponding to the acquired image. Optionally, the depth prediction data corresponding to the acquired image can be in the form of a depth map.

[0085] According to the method introduced in the previous paragraph, the depth prediction data corresponding to each of the N frames of acquired images can be obtained, and the depth prediction data corresponding to each of the N frames of acquired images can constitute the depth prediction result corresponding to the first image sequence.

[0086] In an optional example, step 320 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the second prediction module executed by the processor.

[0087] Step 330 : Based on the optical flow prediction result, the depth prediction result, and the first camera pose sequence, the first image sequence is segmented into static and dynamic images to obtain a third image sequence corresponding to the static scene and a fourth image sequence corresponding to the dynamic scene.

[0088] In an optional example, step 330 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the second segmentation module executed by the processor.

[0089] Optionally, step 330 may be used as an optional implementation of step 230 in the embodiment shown in FIG. 2 .

[0090] In some optional implementations of the present disclosure, as shown in FIG. 4 , step 330 may include step 3301 , step 3303 , and step 3305 .

[0091] Step 3301: Determine a first pixel position correspondence between previous and next frame images in a first image sequence based on the optical flow prediction result.

[0092] Assuming that the N captured images in a first image sequence are sequentially represented as image 1, image 2, image 3, ..., image N, and the optical flow value corresponding to pixel 1 in image 1 is used to indicate that the pixel position of pixel 2 in image 2 corresponds to the pixel position of pixel 1 in image 1, the first pixel position correspondence relationship can include: the correspondence between the pixel position of pixel 1 and the pixel position of pixel 2. In this way, the first pixel position correspondence relationship can include the correspondence between the pixel positions of multiple pairs of matching points; where pixel 1 and pixel 2 can form a pair of matching points.

[0093] In an optional example, step 3301 may be executed by the processor calling a corresponding instruction stored in a memory, or may be executed by the first determination submodule executed by the processor.

[0094] Step 3303: Determine a second pixel position correspondence between the previous and next frame images in the first image sequence based on the depth prediction result and the first camera pose sequence.

[0095] The first camera pose sequence may include: a camera pose corresponding to image 1 and a camera pose corresponding to image 2. The camera pose corresponding to image 1 and the camera pose corresponding to image 2 can be used to calculate the translation matrix and rotation matrix of the camera from the acquisition time corresponding to image 1 to the acquisition time corresponding to image 2.

[0096] Referring to the motion model shown in Figure 1-1, given the depth value corresponding to pixel 1, the camera's translation matrix and rotation matrix from the acquisition time corresponding to image 1 to the acquisition time corresponding to image 2, and the pre-calibrated camera intrinsic parameters, the following formula can be used to determine the pixel 2' corresponding to pixel 1 on image 2:

[0097] in, represents the pixel position of pixel 2', K represents the pre-calibrated camera intrinsic parameter in matrix form, represents the rotation matrix of the camera from the acquisition time corresponding to image 1 to the acquisition time corresponding to image 2, K -1 represents the inverse matrix of K, represents the pixel position of pixel 1, Indicates the depth value of pixel 1 in the depth prediction result. Represents the translation matrix of the camera from the acquisition moment corresponding to image 1 to the acquisition moment corresponding to image 2.

[0098] The above formula can be understood as first converting pixel point 1 in image 2 from the pixel coordinate system to the camera coordinate system, and then from the camera coordinate system to the world coordinate system, obtaining spatial point 1 corresponding to pixel point 1 in the world coordinate system, and then converting spatial point 1 from the world coordinate system to the camera coordinate system, and then from the camera coordinate system to the pixel coordinate system, thereby determining pixel point 2' corresponding to pixel point 1 on image 2. The second pixel position correspondence can include the correspondence between the pixel position of pixel point 1 and the pixel position of pixel point 2'. In this way, the second pixel position correspondence can include the correspondence between the pixel positions of multiple pairs of matching points; among which pixel point 1 and pixel point 2' can form a pair of matching points.

[0099] In an optional example, step 3303 may be executed by the processor calling a corresponding instruction stored in a memory, or may be executed by a second determining submodule executed by the processor.

[0100] Step 3305 : Based on the first pixel position correspondence relationship and the second pixel position correspondence relationship, the first image sequence is segmented into static and dynamic scenes to obtain a third image sequence corresponding to static scenes and a fourth image sequence corresponding to dynamic scenes.

[0101] In an optional example, step 3305 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by a segmentation submodule executed by the processor.

[0102] In some optional embodiments of the present disclosure, as shown in FIG5-1, step 3305 includes step 33051a, step 33052a, step 33053a, step 33054a, step 33055a and step 33056a.

[0103] Step 33051a, for the first target pixel point in the first target image in the first image sequence, determine the first target pixel position corresponding to the pixel position of the first target pixel point from the second target image according to the first pixel position correspondence; wherein the second target image is the next frame image of the first target image in the first image sequence.

[0104] Step 33052a: Determine, from the second target image, a second target pixel position corresponding to the pixel position of the first target pixel point according to the second pixel point correspondence relationship.

[0105] It should be noted that the first target image may be any frame image except the last frame image in the first image sequence, and the first target pixel point may be any pixel point in the first target image.

[0106] Assuming that the first target image is image 1, the second target image is image 2, and the first target pixel point is pixel point 1, then according to the first pixel position correspondence, the first target pixel position determined from the second target image can be the pixel position of pixel point 2 in image 2, and the second target pixel position determined from the second target image can be the pixel position of pixel point 2' in image 2.

[0107] Step 33053a, based on the position error between the first target pixel position and the second target pixel position, determine the pixel attribute of the first target pixel; wherein the pixel attribute of any pixel is used to indicate whether the pixel belongs to a dynamic object.

[0108] Assuming the first target pixel position is denoted as f(p) and the second target pixel position is denoted as u(p), the position error between the first and second target pixel positions can be expressed as |f(p) - u(p)|. For ease of explanation, the position error between the first and second target pixel positions will be referred to as the pixel position error. Based on the pixel position error and the preset position error, the pixel attribute of the first target pixel can be determined.

[0109] Optionally, if the pixel position error is greater than the preset position error, the pixel point attribute of the first target pixel point can be determined as a first-category attribute to characterize that the first target pixel point belongs to a dynamic object; if the pixel position error is less than or equal to the preset position error, the pixel point attribute of the first target pixel point can be determined as a second-category attribute to characterize that the first target pixel point does not belong to a dynamic object, that is, the first target pixel point belongs to a static object.

[0110] Of course, the method for determining pixel attributes is not limited to this. For example, the difference between the pixel position error and the preset position error can be calculated, and the ratio of the absolute value of the difference to the preset position error can be calculated. If the calculated ratio is greater than the preset ratio, the pixel attribute of the first target pixel can be determined as a first-category attribute to indicate that the first target pixel belongs to a dynamic object; if the calculated ratio is less than or equal to the preset ratio, the pixel attribute of the first target pixel can be determined as a second-category attribute to indicate that the first target pixel does not belong to a dynamic object, that is, the first target pixel belongs to a static object.

[0111] Step 33054a: Based on the pixel attributes of the first target pixel, the first target image is segmented into static and dynamic images to obtain a first segmented image corresponding to the static scene and a second segmented image corresponding to the dynamic scene.

[0112] According to the pixel attribute determination method described above, the pixel attributes of each pixel in the first target image can be obtained. By generating a mask for the area in the first target image where pixels with the first type of attributes are distributed, dynamic objects in the first target image can be removed, thereby segmenting a first segmented image corresponding to the static scene. Similarly, by generating a mask for the area in the first target image where pixels with the second type of attributes are distributed, static objects in the first target image can be removed, thereby segmenting a second segmented image corresponding to the dynamic scene.

[0113] Step 33055a: Determine a third image sequence based on the first segmented image.

[0114] Step 33056a: Determine a fourth image sequence based on the second segmented image.

[0115] The above describes the static and dynamic segmentation method for the first target image. Each captured image frame in the first image sequence can be similarly segmented, resulting in N segments corresponding to static scenes and N segments corresponding to dynamic scenes. Arranging the N segments corresponding to static scenes in time sequence forms the third image sequence; arranging the N segments corresponding to dynamic scenes in time sequence forms the fourth image sequence.

[0116] In some embodiments, after obtaining N frames of segmented images corresponding to static scenes and N frames of segmented images corresponding to dynamic scenes, a certain algorithm can be used to correct these segmented images (for example, filtering processing), and then the third image sequence and the fourth image sequence can be determined based on these corrected segmented images.

[0117] In an optional example, the above steps 33051a to 33056a can be executed by the processor calling the corresponding instructions stored in the memory, or can be executed by the corresponding units in the segmentation submodule executed by the processor.

[0118] In this embodiment, by comparing the position error between the two target pixel positions obtained based on the first pixel point correspondence and the second pixel point correspondence, the position error can be used as reference information to determine which pixels in the first target image belong to dynamic objects and which pixels belong to static objects. On this basis, the first target image can be efficiently and reliably segmented into dynamic and static images, so as to obtain the third image sequence and the fourth image sequence for subsequent steps.

[0119] In some optional implementations of the present disclosure, as shown in FIG. 5-2 , the method provided by the embodiment of the present disclosure further includes step 3304 .

[0120] Step 3304: Perform instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result.

[0121] For each of the N frames of acquired images included in the first image sequence, the acquired image can be provided as input to the instance segmentation model. The instance segmentation model can perform calculations based on this to obtain the predicted category of each pixel point in the acquired image, as well as the instance area of ​​each instance in the acquired image (which can be either a static object or a dynamic object). This information can constitute the instance segmentation data corresponding to the acquired image.

[0122] In an optional example, step 3304 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by a third segmentation module executed by the processor.

[0123] According to the method described in the previous paragraph, instance segmentation data corresponding to each of the N frames of acquired images can be obtained, and the instance segmentation data corresponding to each of the N frames of acquired images can constitute the instance segmentation result corresponding to the first image sequence.

[0124] Step 3305 includes step 33051b, step 33052b, step 33053b, step 33054b, step 33055b and step 33056b.

[0125] In an optional example, step 3305 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by a segmentation submodule executed by the processor.

[0126] Step 33051b: For the first target image in the first image sequence, determine the instance region of each instance in the first target image based on the instance segmentation result.

[0127] Optionally, instance segmentation data corresponding to the first target image may be directly extracted from the instance segmentation result, and the instance region of each instance in the first target image may be located based on the instance segmentation data corresponding to the first target image.

[0128] Step 33052b, based on the first pixel position correspondence and the second pixel position correspondence, determine the pixel attributes of each pixel in the first target image; wherein the pixel attribute of any pixel is used to indicate whether the pixel belongs to a dynamic object.

[0129] It should be noted that, for the method of determining the pixel attributes of each pixel in the first target image based on the first pixel position correspondence and the second pixel position correspondence, please refer to the relevant introduction in the implementation shown in Figure 5-1, and will not be repeated here.

[0130] Step 33053b: For each instance region in the first target image, statistics are performed on the pixel attributes of each pixel in the instance region to obtain statistical results.

[0131] Optionally, for each instance area in the first target image, the first number and first proportion of pixels in the instance area whose pixel attributes are the first type of attributes, as well as the second number and second proportion of pixels in the instance area whose pixel attributes are the second type of attributes, can be counted. The first number, first proportion, second number, and second proportion corresponding to each instance area in the first target image can constitute the statistical results.

[0132] In some cases, the statistical results may also include only the first quantity and the second quantity corresponding to each instance area in the first target image, or only the first proportion and the second proportion corresponding to each instance area in the first target image.

[0133] Step 33054b: Based on the statistical results corresponding to each instance area in the first target image, the first target image is segmented into static and dynamic images to obtain a third segmented image corresponding to the static scene and a fourth segmented image corresponding to the dynamic scene.

[0134] In some optional embodiments of the present disclosure, step 33054b includes:

[0135] For each instance area in the first target image, based on the statistical results corresponding to the instance area, determine the proportion of pixel points in the instance area whose attributes are used to characterize that the current pixel point does not belong to a dynamic object; based on the proportion corresponding to each instance area in the first target image, perform dynamic and static segmentation on the first target image to obtain a third segmented image corresponding to the static scene and a fourth segmented image corresponding to the dynamic scene.

[0136] Assuming that the statistical results include the second proportion corresponding to each instance area in the first target image, the second proportion corresponding to any instance area can be considered as the proportion of pixel points in the instance area used by the pixel point attributes to characterize that the current pixel point does not belong to a dynamic object. Then, the target image can be directly segmented into dynamic and static parts based on the second proportion corresponding to each instance area in the first target image and the preset proportion.

[0137] Assuming that the statistical results do not include the second proportion corresponding to each instance region in the first target image, but instead include the first proportion corresponding to each instance region in the first target image, the second proportion corresponding to each instance region can be obtained by calculating the difference between 1 (a preset value) and the first proportion corresponding to the instance region. Then, based on the second proportion corresponding to each instance region in the first target image and the preset proportion, the target image is segmented into static and dynamic regions.

[0138] Optionally, the preset proportion may be 70%, 75%, 80%, etc., which are not listed here one by one.

[0139] For any instance region in the first target image, if the second proportion corresponding to the instance region is greater than the preset proportion, the instance region can be determined to be a static object; if the second proportion corresponding to the instance region is less than or equal to the preset proportion, the instance region can be determined to be a dynamic object. By generating a mask for each instance region in the first target image that is a dynamic object, the dynamic object in the first target image can be removed, thereby segmenting a third segmented image corresponding to the static scene. Similarly, by generating a mask for each instance region in the first target image that is a static object, the static object in the first target image can be removed, thereby segmenting a fourth segmented image corresponding to the dynamic scene.

[0140] The above paragraph introduces the case of performing dynamic and static segmentation with reference to the second proportion corresponding to each instance area. In some cases, dynamic and static segmentation can also be performed with reference to the first proportion corresponding to each instance area. For example, for any instance area in the first target image, if the first proportion corresponding to the instance area is less than or equal to the preset proportion, it can be determined that the instance area belongs to a static object. If the first proportion corresponding to the instance area is greater than the preset proportion, it can be determined that the instance area belongs to a dynamic object. On this basis, a third segmented image corresponding to the static scene and a fourth segmented image corresponding to the dynamic scene can also be segmented by generating a mask.

[0141] Step 33055b: Determine a third image sequence based on the third segmented image.

[0142] Step 33056b: Determine a fourth image sequence based on the fourth segmented image.

[0143] It should be noted that the specific implementation of steps 33055b and 33056b can refer to the relevant introduction to steps 33055a and 33056a above, and will not be repeated here.

[0144] It should be noted that both the depth prediction model and the optical flow prediction model may have certain prediction errors, and the process of obtaining the first camera pose sequence also has certain estimation errors. In order to reduce the impact of these errors on the scene reconstruction results, the instance segmentation model can be used to determine the instance area of ​​each instance in the first target image, and for each instance area, the pixel attributes are statistically analyzed so that the statistical results can be used for dynamic and static segmentation of the first target image. For example, the proportion of pixels in each instance area whose pixel attributes are the second type of attributes can be used for dynamic and static segmentation. This is equivalent to introducing certain constraints for the dynamic and static segmentation process (the motion states of each point on the same object are often consistent, that is, these points are either fixed or move together), which is conducive to making the dynamic and static segmentation results more robust. Using the dynamic and static segmentation results in subsequent steps is conducive to improving the final scene reconstruction effect.

[0145] In an optional example, the above steps 33051b to 33056b can be executed by the processor calling the corresponding instructions stored in the memory, or can be executed by the corresponding units in the segmentation submodule executed by the processor.

[0146] In the embodiments of the present disclosure, by using an optical flow prediction model, an optical flow prediction result corresponding to a first image can be determined efficiently and reliably. The optical flow prediction result can be used to determine a first pixel position correspondence between the preceding and following frame images. By using a depth prediction model, a depth prediction result corresponding to a first image sequence can be determined efficiently and reliably. The depth prediction result can be used together with the first camera pose sequence to determine a second pixel position correspondence between the preceding and following frame images. The first pixel position correspondence can be regarded as an object tracking result obtained by one method, and the second pixel position correspondence can be regarded as an object tracking result obtained by another method. Combining the first pixel position correspondence and the second pixel position correspondence, a position error can be determined. The determined position error can be regarded as a residual flow. Using the residual flow as reference information, dynamic objects and static objects in the image can be distinguished more accurately to improve the dynamic and static segmentation effect of the first image sequence. In specific implementation, some more complex tracking algorithms can also be introduced, and the tracking results obtained by the tracking algorithm can be used for dynamic and static segmentation.

[0147] FIG6 is a flow chart illustrating a method for reconstructing a dynamic scene according to an exemplary embodiment of the present disclosure. The method shown in FIG6 includes steps 610, 620, 630, 640, and 650. Optionally, a combination of steps 620 to 650 may be used as an alternative implementation of step 250.

[0148] Step 610: Perform optical flow prediction on the first image sequence using an optical flow prediction model to obtain an optical flow prediction result.

[0149] It should be noted that the specific implementation of step 610 can be referred to the above introduction to step 310 and will not be repeated here.

[0150] In an optional example, step 610 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the first prediction module executed by the processor.

[0151] Step 620 : Determine a third pixel position correspondence between the previous and next frame images in the fourth image sequence based on the optical flow prediction result.

[0152] It should be noted that the specific implementation of step 620 can be referred to the above introduction to step 3301 and will not be repeated here.

[0153] In an optional example, step 620 may be executed by the processor calling a corresponding instruction stored in the memory, or may be executed by a third determining submodule executed by the processor.

[0154] Step 630 : Perform sparse reconstruction of the point cloud using the first camera pose sequence, the fourth image sequence, and the third pixel position correspondence to obtain a fourth reconstructed point cloud and an object pose sequence of the dynamic object in the fourth image sequence.

[0155] When the correspondence between the first camera pose sequence, the fourth image sequence and the third pixel position is known, the SFM algorithm can be used in combination with the motion model shown in Figures 1-2 to obtain the fourth reconstructed point cloud and the object pose sequence of the dynamic objects in the fourth image sequence; wherein the object pose sequence may include the object pose of the dynamic objects in the fourth image sequence at the respective acquisition moments of the N frames of acquired images.

[0156] It should be noted that the above text gives the motion model shown in reference Figure 1-1, determines the formula for pixel 2' in image 2 corresponding to pixel 1, and explains the basic principle of the formula. Here, when using the first camera pose sequence, the fourth image sequence and the third pixel position correspondence to perform sparse reconstruction of the point cloud, it is also necessary to find multiple pairs of matching points. The main difference from the basic principle explained above is that the process of finding multiple pairs of matching points here needs to take the motion of the object into consideration.

[0157] Assume that pixel 1 in image 1 belongs to moving object c. Pixel 1 can be a point with relatively obvious features of moving object c, such as a point with a large gradient. Here, pixel 1 in image 1 can be first converted from the pixel coordinate system to the camera coordinate system, and then from the camera coordinate system to the world coordinate system, to obtain the acquisition time corresponding to image 1 and the spatial point 1 corresponding to pixel 1 in the world coordinate system. Next, referring to the translation matrix of moving object c from the acquisition time corresponding to image 1 to the acquisition time corresponding to image 2, spatial point 1 can be converted from the acquisition time corresponding to image 1 to the acquisition time corresponding to image 2, to obtain spatial point 1'. Spatial point 1' can then be converted from the world coordinate system to the camera coordinate system, and then from the camera coordinate system to the pixel coordinate system, thereby determining pixel 2'' in image 2 corresponding to pixel 1. Pixel 1 and pixel 2'' can form a pair of matching points.

[0158] In an optional example, step 630 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the first reconstruction submodule executed by the processor.

[0159] Step 640: Correct the object pose sequence according to a preset correction method.

[0160] In an optional example, step 640 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by a correction submodule executed by the processor.

[0161] In some optional implementations of the present disclosure, as shown in FIG. 7-1 , the method provided by the embodiment of the present disclosure further includes step 632 .

[0162] Step 632: Perform depth prediction on the first image sequence using a depth prediction model to obtain a depth prediction result.

[0163] It should be noted that the specific implementation of step 632 can be referred to the above description of step 320 and will not be repeated here.

[0164] In an optional example, step 632 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the second prediction module executed by the processor.

[0165] Step 640 includes step 6401, step 6402 and step 6403.

[0166] Step 6401: Determine a fourth pixel position correspondence between previous and next frame images in a fourth image sequence based on the depth prediction result, the first camera pose sequence, and the object pose sequence.

[0167] It should be noted that the method for determining the fourth pixel position correspondence is similar to the method for determining the second pixel position correspondence above. The main difference is that the determination of the fourth pixel position correspondence refers to the motion model shown in Figure 1-2, and the movement of the object needs to be considered during the determination process. For details, please refer to the above introduction to the method for determining the pixel point 2'' corresponding to the pixel point 1 in image 2. Therefore, the object posture sequence needs to be applied, while the determination of the second pixel position correspondence refers to the motion model shown in Figure 1-1, and the movement of the object does not need to be considered during the determination process.

[0168] Step 6402: Determine a reprojection error based on the third pixel position correspondence and the fourth pixel position correspondence.

[0169] It should be noted that the above describes a method for determining the position error for the first target pixel based on the first pixel position correspondence and the second pixel position correspondence. Here, based on the third pixel position correspondence and the fourth pixel position correspondence, a similar method can be used to determine the position errors corresponding to multiple target pixel points. Then, based on the position errors corresponding to multiple target pixel points, the reprojection error can be determined. For example, the average value of the position errors corresponding to multiple target pixel points can be directly calculated, and the calculated average value can be used as the reprojection error. For another example, the position errors with obviously abnormal values ​​(for example, the position errors are too large or too small) can be removed from the position errors corresponding to multiple target pixel points, and then the average value of the remaining position errors can be calculated, and the calculated average value can be used as the reprojection error.

[0170] Step 6403, correct the object pose sequence by minimizing the reprojection error.

[0171] Optionally, the reprojection error can be minimized by specifying an optimization algorithm, where the specified optimization algorithm can be either a linear optimization algorithm or a nonlinear optimization algorithm, for example, either a linear least squares method or a nonlinear least squares method.

[0172] The calculation of the reprojection error requires the application of the fourth pixel position correspondence, and the determination of the fourth pixel position correspondence requires the application of the object pose sequence. That is, the object pose sequence and the reprojection error are interrelated. In this way, by minimizing the reprojection error, the object pose sequence can be optimized, and the corrected object pose sequence can be used for subsequent point cloud reconstruction, which is beneficial to improving the point cloud reconstruction effect.

[0173] In an optional example, steps 6401 to 6403 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by corresponding units in the correction submodule executed by the processor.

[0174] In some other optional implementations of the present disclosure, as shown in FIG. 7-2 , the method provided by the embodiment of the present disclosure further includes step 634 .

[0175] Step 634 : Perform instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result.

[0176] It should be noted that the specific implementation of step 634 can be referred to the relevant introduction to step 320 above, and will not be repeated here.

[0177] In an optional example, step 634 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by a third segmentation module executed by the processor.

[0178] Step 640 includes step 6404, step 6405, step 6406 and step 6407.

[0179] Step 6404: Determine the motion trajectory of the dynamic object in the fourth image sequence based on the object posture sequence.

[0180] For any dynamic object in the fourth image sequence, an object position sequence for the dynamic object can be extracted from the object pose sequence, and each position in the extracted object position sequence can constitute the motion trajectory of the dynamic object. Alternatively, after the object position sequence is extracted, positions in the extracted object position sequence that are obviously abnormal can be corrected, and each position in the corrected object position sequence can constitute the motion trajectory of the dynamic object.

[0181] Step 6405 : Determine a first predicted category of the dynamic object in the fourth image sequence based on the instance segmentation result.

[0182] Optionally, a predicted category of any dynamic object in the fourth image sequence may be extracted from the instance segmentation result, and the predicted category may be used as the first predicted category.

[0183] Step 6406: Determine the preset trajectory corresponding to the first prediction category.

[0184] Optionally, the correspondence between the categories and the trajectories may be preset. Thus, based on the preset correspondence, the trajectory corresponding to the first predicted category may be determined efficiently and reliably, and the determined trajectory may be used as the preset trajectory in step 6406 .

[0185] Step 6407 , correcting the object pose sequence by minimizing the trajectory error between the motion trajectory and the preset trajectory.

[0186] Optionally, a trajectory error between the motion trajectory and a preset trajectory may be calculated, and the trajectory error may be minimized using the above-specified optimization algorithm.

[0187] In an optional example, the above steps 6404 to 6407 can be executed by the processor calling the corresponding instructions stored in the memory, or can be executed by the corresponding units in the correction submodule executed by the processor.

[0188] The calculation of trajectory error needs to be applied to the motion trajectory, and the determination of the motion trajectory needs to be applied to the object pose sequence. That is, the object pose sequence and the trajectory error are interrelated. In this way, by minimizing the trajectory error, the object pose sequence can be optimized. The corrected object pose sequence can be used for subsequent point cloud reconstruction, which is beneficial to improving the point cloud reconstruction effect.

[0189] Step 650 : densely reconstruct the point cloud using the first camera pose sequence, the fourth image sequence, the third pixel position correspondence, the fourth reconstructed point cloud, and the corrected object pose sequence to obtain a third reconstructed point cloud.

[0190] When the first camera pose sequence, the fourth image sequence, the third pixel position correspondence, the fourth reconstructed point cloud and the corrected object pose sequence are known, combined with the motion model shown in Figure 1-2, the MVS algorithm can be used to achieve dense reconstruction of the point cloud to obtain the third reconstructed point cloud.

[0191] In an optional example, step 650 may be executed by the processor calling corresponding instructions stored in the memory, or may be executed by the second reconstruction submodule executed by the processor.

[0192] In the embodiments of the present disclosure, through the application of the optical flow prediction model, the optical flow prediction result corresponding to the first image can be obtained efficiently and reliably, and the optical flow prediction result can be used to determine the third pixel position correspondence between the previous and next frame images. The third pixel position correspondence can be used together with the first camera pose sequence and the fourth image sequence for sparse reconstruction of the dynamic scene to obtain a fourth reconstructed point cloud and an object pose sequence. After that, the object pose sequence can be corrected in a certain way, and the corrected object pose sequence can be used together with the first camera pose sequence, the fourth image sequence, the third pixel position correspondence and the fourth reconstructed point cloud for dense reconstruction of the dynamic scene. In this way, the accuracy and reliability of the data on which the dense reconstruction of the dynamic scene depends can be improved, which is conducive to improving the final scene reconstruction effect.

[0193] FIG8 is a flowchart illustrating a dense reconstruction method for a dynamic scene according to an exemplary embodiment of the present disclosure. The method shown in FIG8 includes steps 810, 820, 830, 840, and 850. Alternatively, the combination of steps 820 to 840 may serve as an alternative implementation of step 630 described above, and step 850 may serve as an alternative implementation of step 650.

[0194] Step 810: Perform instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result.

[0195] It should be noted that the specific implementation of step 810 can be referred to the relevant introduction to step 3304 above, and will not be repeated here.

[0196] Step 820 : Based on the instance segmentation result and the third pixel position correspondence, determine whether the predicted categories of corresponding pixel positions in the preceding and following frame images in the fourth image sequence are consistent, and obtain a determination result.

[0197] For the pixel position x1 in any image except the last frame in the fourth image sequence, based on the third pixel position correspondence relationship, the pixel position x2 corresponding to the pixel position x1 in the next frame of the image can be determined. In addition, based on the instance segmentation result, the predicted category y1 of the pixel position x1 and the predicted category y2 of the pixel position x2 can be determined. Then, the predicted category y1 and the predicted category y2 can be compared to see if they are consistent to obtain a confirmed result.

[0198] Step 830: Based on the determination result, correct the third pixel position correspondence.

[0199] If pixel position x1 and pixel position x2 belong to the same moving object, theoretically, the predicted categories of pixel position x1 and pixel position x2 should be consistent. In view of this, if the determination result indicates that the predicted category y1 and the predicted category y2 are inconsistent, it can be determined that pixel position x1 and pixel position x2 do not actually belong to the same moving object and pixel position x1 and pixel position x2 do not actually have a corresponding relationship. Therefore, the correspondence between pixel position x1 and pixel position x2 can be deleted from the third pixel position correspondence relationship, or an invalid flag can be added to the correspondence between pixel position x1 and pixel position x2 in the third pixel position correspondence relationship to achieve correction of the third pixel position correspondence relationship.

[0200] Step 840 , performing sparse reconstruction of the point cloud using the first camera pose sequence, the fourth image sequence, and the corrected third pixel position correspondence to obtain a fourth reconstructed point cloud and an object pose sequence of the dynamic object in the fourth image sequence.

[0201] It should be noted that the specific implementation of step 840 can be similar to the specific implementation of step 630 introduced above. The main difference is that the corrected third pixel position correspondence is used in step 840, which is conducive to improving the sparse reconstruction effect of dynamic scenes.

[0202] Step 850 , performing dense reconstruction of the point cloud using the first camera pose sequence, the fourth image sequence, the corrected third pixel position correspondence, the fourth reconstructed point cloud, and the corrected object pose sequence to obtain a third reconstructed point cloud.

[0203] It should be noted that the specific implementation of step 850 can be similar to the specific implementation of step 650 introduced above. The main difference is that the corrected third pixel position correspondence is used in step 850, which is conducive to improving the dense reconstruction effect of dynamic scenes.

[0204] In an optional example, the above steps 810 to 850 can be executed by the processor calling the corresponding instructions stored in the memory, or can be executed by the corresponding module run by the processor.

[0205] In the embodiments of the present disclosure, by using the instance segmentation model, the instance segmentation results of the first image sequence can be obtained efficiently and reliably. By organically using the instance segmentation results and the third pixel position correspondence, the third pixel position correspondence can be corrected, and the corrected third pixel position correspondence is used for sparse reconstruction and dense reconstruction of the point cloud, which is beneficial to improving the reconstruction effect of dynamic scenes.

[0206] Figure 9 is a flowchart illustrating a dense reconstruction method for a static scene according to an exemplary embodiment of the present disclosure. The method shown in Figure 9 includes steps 910, 920, 930, 940, 950, and 960. Alternatively, a combination of steps 920 through 960 may be used as an alternative implementation of step 240.

[0207] Step 910: Perform instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result.

[0208] It should be noted that the specific implementation of step 910 can be referred to the relevant introduction to step 3304 above, and will not be repeated here.

[0209] Step 920 : For a second target pixel in a third target image in a third image sequence, determine a second predicted category of the second target pixel based on the instance segmentation result.

[0210] It should be noted that the third target image may be any image included in the third image sequence, and the second target pixel may be any pixel in the third target image.

[0211] Optionally, the predicted category of the second target pixel point may be directly extracted from the instance segmentation result, and the extracted predicted category may be used as the second predicted category.

[0212] Step 930 : Determine the size of the local region matching the second predicted category.

[0213] Optionally, a correspondence between categories and window sizes may be preset, so that the window size corresponding to the second prediction category may be determined based on the preset correspondence, and the window size may be used as the size of the local area matching the second prediction category.

[0214] Step 940 : Determine a target local area from the third target image based on the pixel position of the second target pixel and the size of the local area.

[0215] Optionally, the local area size may include a width size and a height size. In this case, a region centered at the pixel position of the second target pixel point and having the width size and the height size may be captured from the third target image as the target local area. In some cases, a region centered at the pixel position of the second target pixel point and having the width size and the height size may also be captured from the third target image as the target local area.

[0216] Step 950 , for the target local area, densely reconstruct the point cloud using the first reconstructed point cloud, the first camera pose sequence, and the other images in the third image sequence except the third target image to obtain a local point cloud corresponding to the target local area.

[0217] After determining the target local region, the first reconstructed point cloud, the first camera pose sequence, and the pre-calibrated camera intrinsic parameters are used, and the motion model shown in Figure 1-1 is referred to. The spatial region corresponding to the target local region in the world coordinate system can be found. The spatial region can be a rectangular area. Spatial points can then be sampled within the spatial region to obtain multiple spatial points, for example, M spatial points.

[0218] For each of the M spatial points, using the first camera pose sequence and pre-calibrated camera intrinsic parameters, and referring to the motion model shown in Figure 1-1, the pixel corresponding to the spatial point in the third image sequence, excluding the third target image, can be determined. The pixel position of this pixel can be represented as pixel position x3. Because the spatial region and the local target region are associated, the pixel corresponding to the spatial point in the local target region can also be calculated. The pixel position of this pixel can be represented as pixel position x4. Through image matching, the pixel position corresponding to pixel position x4 in the other image can also be determined. This pixel position can be represented as pixel position x5. The corresponding position error can then be obtained by subtracting pixel position x5 from pixel position x4. Based on the position errors corresponding to the M spatial points, the reprojection error can be obtained by averaging them. Based on the obtained reprojection error, the normal vector of the spatial region and the depth values ​​of the M spatial points can be optimized. After the depth values ​​are optimized, the M spatial points can form a local point cloud corresponding to the local target region.

[0219] Step 960: Determine a second reconstructed point cloud based on the local point cloud.

[0220] The above describes a method for reconstructing a local point cloud for the target local area corresponding to the second target pixel. Using this method, local point clouds can be reconstructed for multiple pixels in the third image sequence, and these local point clouds can form the second reconstructed point cloud. Alternatively, after obtaining these local point clouds, they can be optimized using a certain method, and these optimized reconstructed point clouds can form the second reconstructed point cloud.

[0221] In an optional example, the above steps 910 to 960 can be executed by the processor calling the corresponding instructions stored in the memory, or can be executed by the corresponding module run by the processor.

[0222] In the embodiments of the present disclosure, through the application of the instance segmentation model, the instance segmentation results of the first image sequence can be obtained efficiently and reliably. The instance segmentation results can be used to determine the second prediction category of the second target pixel point in the third target image in the third image sequence. With reference to the second prediction category, the appropriate local area size can be determined, and the corresponding local area can be determined accordingly for the reconstruction of the local point cloud corresponding to the second target pixel point. For example, if the second target pixel point is in a weak texture area such as the ground, a large window can be used to reconstruct the local point cloud, so that more scene details can be reconstructed. If the second target pixel point is in a strong texture area, a small window can be used to reconstruct the local point cloud, so that the reconstruction accuracy can be improved. Therefore, the embodiments of the present disclosure can improve the dense reconstruction effect of static scenes.

[0223] FIG10 is a flowchart illustrating a dense reconstruction method for a static scene according to another exemplary embodiment of the present disclosure. The method shown in FIG10 includes steps 1010, 1020, and 1030. Optionally, a combination of steps 1020 and 1030 can be used as an optional implementation of step 240.

[0224] Step 1010: Perform optical flow prediction on the first image sequence using an optical flow prediction model to obtain an optical flow prediction result.

[0225] It should be noted that the specific implementation of step 1010 can be referred to the above introduction to step 310 and will not be repeated here.

[0226] Step 1020: Determine a fifth pixel position correspondence between the previous and next frame images in the third image sequence based on the optical flow prediction result.

[0227] It should be noted that the specific implementation of step 1020 can be referred to the relevant introduction to step 3301 above, and will not be repeated here.

[0228] Step 1030 : Perform dense reconstruction of the point cloud using the first reconstructed point cloud, the first camera pose sequence, the third image sequence, and the fifth pixel position correspondence to obtain a second reconstructed point cloud.

[0229] In an optional example, the above steps 1010 to 1030 can be executed by the processor calling the corresponding instructions stored in the memory, or can be executed by the corresponding module run by the processor.

[0230] It should be noted that the specific implementation of step 1030 can be similar to the specific implementation of step 240 introduced above. The main difference is that the dense reconstruction in step 1031 also utilizes the fifth pixel position correspondence relationship. The fifth pixel position correspondence relationship can be regarded as an object tracking result obtained in a certain way. Under the guidance of the object tracking result, it is helpful to clarify which pixel points in the previous and next frames correspond to the same spatial point, so as to provide an effective reference for the reconstruction process, thereby improving the dense reconstruction effect of static scenes.

[0231] FIG11 is a flowchart illustrating a dense reconstruction method for a static scene provided by another exemplary embodiment of the present disclosure. The method shown in FIG11 includes steps 1110, 1120, and 1130. Optionally, a combination of steps 1110 to 1130 can be used as another optional implementation of step 240.

[0232] Step 1110 : determining a second camera pose sequence of the camera based on data collected by a non-visual sensor disposed on the movable device.

[0233] Optionally, non-visual sensors include but are not limited to positioning sensors, inertial measurement units, wheel speed meters, etc.; among them, positioning sensors include but are not limited to positioning sensors based on the Global Positioning System (GPS), positioning sensors based on the Global Navigation Satellite System (GNSS), etc.

[0234] It should be noted that, for the sensors located in the chassis system of the mobile device included in the non-visual sensors, the data collected by these sensors can be obtained from a Controller Area Network (CAN) bus.

[0235] Assuming that the non-visual sensor includes a positioning sensor and inertial measurement data, the camera position at the time of each acquisition of N frames of image acquisition can be determined based on the positioning data collected by the positioning sensor, and the camera posture at the time of each acquisition of N frames of image acquisition can be determined based on the inertial measurement data collected by the inertial measurement unit. By integrating these camera positions and camera postures, a second camera pose sequence can be obtained.

[0236] Step 1120 : Using the second camera pose sequence as an alignment reference, perform similarity transformation alignment on the first camera pose sequence.

[0237] It should be noted that any feasible similarity transformation alignment algorithm can be used to perform similarity transformation alignment on the first camera pose sequence.

[0238] Step 1130 , densely reconstructing the point cloud using the first reconstructed point cloud, the first camera pose sequence aligned by similarity transformation, and the third image sequence to obtain a second reconstructed point cloud.

[0239] It should be noted that when the SFM algorithm is used for sparse reconstruction to obtain the first camera pose sequence, the individual poses in the first camera pose sequence are unscaled, meaning they do not have a true scale. Therefore, data collected by non-visual sensors can be used to determine a second camera pose sequence with a true scale. A similarity transform alignment algorithm can then be used to align the first and second camera pose sequences. This ensures that the first camera pose sequence, after similarity transform alignment, has a true scale. Using this first camera pose sequence for dense reconstruction of static scenes can improve the quality of dense reconstruction of static scenes.

[0240] In an optional example, the above steps 1110 to 1130 can be executed by the processor calling the corresponding instructions stored in the memory, or can be executed by the corresponding module run by the processor.

[0241] In some optional embodiments of the present disclosure, FIG12 is a flow chart of a scene reconstruction method provided by another exemplary embodiment of the present disclosure. As shown in FIG12 , a forward-looking monocular video can be obtained by a forward-looking camera provided on a mobile device. On this basis, the flow of the scene reconstruction method of the embodiment of the present disclosure may include the following steps:

[0242] Step 1210: Determine a first image sequence. That is, based on the front-view monocular video, the first image sequence mentioned above can be obtained.

[0243] In step 1220 , the first image sequence is processed by a large visual perception model including an instance segmentation model, a depth prediction model, and an optical flow prediction model to obtain an instance segmentation result, a depth prediction result, and an optical flow prediction result.

[0244] Step 1230 : performing background segmentation on the first image sequence to obtain a second image sequence.

[0245] In step 1240, sparse reconstruction is performed using the second image sequence to obtain a sparse reconstruction result of the static scene. The sparse reconstruction result of the static scene may include a first reconstructed point cloud and a first camera pose sequence.

[0246] Step 1250 : Using data collected by a non-visual sensor provided on the movable device, similarity transformation alignment is performed on the first camera pose sequence to obtain the first camera pose sequence after similarity transformation alignment.

[0247] In step 1260, the first image sequence is subjected to static and dynamic segmentation to obtain a third image sequence corresponding to the static scene and a fourth image sequence corresponding to the dynamic scene. For example, based on the instance segmentation results, optical flow prediction results, and depth prediction results obtained in step 1220, the first image sequence can be subjected to static and dynamic segmentation to obtain a third image sequence corresponding to the static scene and a fourth image sequence corresponding to the dynamic scene. For details, please refer to the aforementioned related embodiments, such as steps 3301, 3303, 3304, and 3305, and the specific expanded embodiments of each step, such as steps 33051a to 33056a and steps 33051b to 33056b.

[0248] Step 1270 , densely reconstructing the static scene. That is, based on the first camera pose sequence, the first reconstructed point cloud, and the third image sequence aligned by similarity transformation, the static scene can be densely reconstructed to obtain a second reconstructed point cloud.

[0249] Step 1280: sparsely reconstruct the dynamic scene. That is, based on the fourth image sequence, the dynamic scene may be sparsely reconstructed first.

[0250] In step 1290, dense reconstruction is performed on the dynamic scene, ultimately obtaining a scene reconstruction result corresponding to the first image sequence. Specifically, the scene reconstruction result corresponding to the first image sequence is obtained by sequentially performing sparse reconstruction and dense reconstruction on the dynamic scene to obtain the third reconstructed point cloud and the second reconstructed point cloud obtained in step 1270.

[0251] In summary, the embodiments of the present invention adopt a visual method to achieve scene reconstruction. Specifically, a divide-and-conquer approach can be adopted to reconstruct the scene from the easy to the difficult. For example, the static scene is reconstructed first, and then the dynamic scene is reconstructed after the static scene reconstruction is completed. This is conducive to reducing the reconstruction cost and reconstruction difficulty, and can obtain a high-precision dense point cloud.

[0252] Any of the scene reconstruction methods provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server. Alternatively, any of the scene reconstruction methods provided in the embodiments of the present disclosure can be executed by a processor, such as a processor that executes any of the scene reconstruction methods mentioned in the embodiments of the present disclosure by invoking corresponding instructions stored in a memory. This will not be further described below.

[0253] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0254] Exemplary devices

[0255] FIG13 is a schematic diagram of a scene reconstruction device according to an exemplary embodiment of the present disclosure. The device shown in FIG12 includes a first segmentation module 1310 , a first reconstruction module 1320 , a second segmentation module 1330 , a second reconstruction module 1340 , a third reconstruction module 1350 , and a determination module 1360 .

[0256] A first segmentation module 1310 is configured to perform background segmentation on a first image sequence captured by a camera disposed on a mobile device to obtain a second image sequence;

[0257] A first reconstruction module 1320 is configured to perform sparse reconstruction of a point cloud using the second image sequence obtained by the first segmentation module 1310 to obtain a first reconstructed point cloud and a first camera pose sequence;

[0258] A second segmentation module 1330 is configured to perform static and dynamic segmentation on the first image sequence to obtain a third image sequence corresponding to static scenes and a fourth image sequence corresponding to dynamic scenes;

[0259] A second reconstruction module 1340 is configured to perform dense reconstruction of a point cloud using the first reconstructed point cloud obtained by the first reconstruction module 1320 and the first camera pose sequence, and the third image sequence obtained by the second segmentation module 1330, to obtain a second reconstructed point cloud;

[0260] A third reconstruction module 1350 is configured to sequentially perform sparse reconstruction and dense reconstruction of a point cloud using the first camera pose sequence obtained by the first reconstruction module 1320 and the fourth image sequence obtained by the second segmentation module 1330 to obtain a third reconstructed point cloud;

[0261] The determination module 1360 is configured to determine a scene reconstruction result corresponding to the first image sequence based on the second reconstructed point cloud obtained by the second reconstruction module 1340 and the third reconstructed point cloud obtained by the third reconstruction module 1350 .

[0262] In some optional examples, as shown in FIG14 , the apparatus provided by the embodiments of the present disclosure further includes:

[0263] A first prediction module 1370 is configured to perform optical flow prediction on the first image sequence using an optical flow prediction model to obtain an optical flow prediction result;

[0264] A second prediction module 1380 is configured to perform depth prediction on the first image sequence using a depth prediction model to obtain a depth prediction result;

[0265] The second segmentation module 1330 is specifically used to perform dynamic and static segmentation on the first image sequence based on the optical flow prediction result obtained by the first prediction module 1370, the depth prediction result obtained by the second prediction module 1380, and the first camera pose sequence obtained by the first reconstruction module 1320, to obtain a third image sequence corresponding to the static scene and a fourth image sequence corresponding to the dynamic scene.

[0266] In some optional examples, as shown in FIG14 , the second segmentation module 1330 includes:

[0267] A first determining submodule 13301 is configured to determine a first pixel position correspondence between previous and next frame images in the first image sequence based on the optical flow prediction result obtained by the first prediction module 1370;

[0268] A second determination submodule 13303 is configured to determine a second pixel position correspondence between previous and next frame images in the first image sequence based on the depth prediction result obtained by the second prediction module 1380 and the first camera pose sequence obtained by the first reconstruction module 1320;

[0269] The segmentation submodule 13305 is used to perform dynamic and static segmentation on the first image sequence based on the first pixel position correspondence determined by the first determination submodule 13301 and the second pixel position correspondence determined by the second determination submodule 13303 to obtain a third image sequence corresponding to the static scene and a fourth image sequence corresponding to the dynamic scene.

[0270] In some optional examples, the segmentation submodule 13305 includes:

[0271] a first determining unit configured to determine, for a first target pixel point in a first target image in the first image sequence, a first target pixel position corresponding to the pixel position of the first target pixel point in a second target image according to the first pixel position correspondence determined by the first determining submodule 13301; wherein the second target image is an image frame subsequent to the first target image in the first image sequence;

[0272] A second determining unit is configured to determine, from the second target image, a second target pixel position corresponding to the pixel position of the first target pixel point according to the second pixel point correspondence relationship determined by the second determining submodule 13303;

[0273] a third determining unit, configured to determine a pixel attribute of the first target pixel based on a position error between the first target pixel position determined by the first determining unit and the second target pixel position determined by the second determining unit; wherein the pixel attribute of any pixel is used to indicate whether the pixel belongs to a dynamic object;

[0274] a first segmentation unit, configured to perform static and dynamic segmentation on the first target image based on the pixel attributes of the first target pixel determined by the third determination unit, to obtain a first segmented image corresponding to the static scene and a second segmented image corresponding to the dynamic scene;

[0275] a fourth determining unit, configured to determine a third image sequence based on the first segmented image obtained by the first segmenting unit;

[0276] The fifth determining unit is configured to determine a fourth image sequence based on the second segmented image obtained by the first segmenting unit.

[0277] In an optional example, as shown in FIG14 , the apparatus provided by an embodiment of the present disclosure further includes:

[0278] A third segmentation module 1390 is configured to perform instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result;

[0279] The segmentation submodule 13305 includes:

[0280] a fifth determining unit, configured to determine, for a first target image in the first image sequence, an instance region of each instance in the first target image based on the instance segmentation result obtained by the third segmentation module 1390;

[0281] A sixth determining unit is configured to determine a pixel attribute of each pixel in the first target image based on the first pixel position correspondence determined by the first determining submodule 13301 and the second pixel position correspondence determined by the second determining submodule 13303; wherein the pixel attribute of any pixel is used to indicate whether the pixel belongs to a dynamic object;

[0282] a statistical unit, configured to perform statistics on the pixel attributes of each pixel in each instance region determined by the sixth determining unit in the instance region in the first target image, to obtain a statistical result;

[0283] a second segmentation unit, configured to perform static and dynamic segmentation on the first target image based on the statistical results corresponding to each instance region in the first target image obtained by the statistical unit, to obtain a third segmented image corresponding to the static scene and a fourth segmented image corresponding to the dynamic scene;

[0284] a seventh determining unit, configured to determine a third image sequence based on the third segmented image obtained by the second segmenting unit;

[0285] The eighth determining unit is configured to determine a fourth image sequence based on the fourth segmented image obtained by the second segmenting unit.

[0286] In some optional examples, the second segmentation unit includes:

[0287] a determining subunit configured to determine, for each instance region in the first target image, based on the statistical result corresponding to the instance region obtained by the statistical unit, a proportion of pixels in the instance region whose attributes are used to represent that the current pixel does not belong to a dynamic object;

[0288] The segmentation subunit is used to perform dynamic and static segmentation on the first target image based on the corresponding proportion of each instance area in the first target image determined by the determination subunit, and obtain a third segmented image corresponding to the static scene and a fourth segmented image corresponding to the dynamic scene.

[0289] In some optional examples, as shown in FIG15-1 and FIG15-2 , the apparatus provided by the embodiments of the present disclosure further includes:

[0290] A first prediction module 1370 is configured to perform optical flow prediction on the first image sequence using an optical flow prediction model to obtain an optical flow prediction result;

[0291] The third reconstruction module 1350 includes:

[0292] A third determining submodule 13501 is configured to determine, based on the optical flow prediction result obtained by the first prediction module 1370, a third pixel position correspondence between the previous and next frame images in the fourth image sequence obtained by the segmentation submodule 13305;

[0293] A first reconstruction submodule 13503 is configured to perform sparse reconstruction of the point cloud using the first camera pose sequence obtained by the first reconstruction module 1320, the fourth image sequence obtained by the second segmentation module 1330, and the third pixel position correspondence determined by the third determination submodule 13501, to obtain a fourth reconstructed point cloud and an object pose sequence of the dynamic object in the fourth image sequence;

[0294] A correction submodule 13507 is used to correct the object pose sequence obtained by the first reconstruction submodule 13503 according to a preset correction method;

[0295] The second reconstruction submodule 13509 is used to perform dense reconstruction of the point cloud using the first camera pose sequence obtained by the first reconstruction module 1320, the fourth image sequence obtained by the second segmentation module 1330, the third pixel position correspondence determined by the third determination submodule 13501, the fourth reconstructed point cloud obtained by the first reconstruction submodule 13503 and the corrected object pose sequence obtained by the correction submodule 13507 to obtain a third reconstructed point cloud.

[0296] In some optional examples, as shown in FIG15-1 , the apparatus provided by the embodiment of the present disclosure further includes:

[0297] A second prediction module 1380 is configured to perform depth prediction on the first image sequence using a depth prediction model to obtain a depth prediction result;

[0298] Modify submodule 13507, including:

[0299] a ninth determining unit, configured to determine a fourth pixel position correspondence between the preceding and following frame images in the fourth image sequence obtained by the second segmentation module 1330 based on the depth prediction result obtained by the second prediction module 1380, the first camera pose sequence obtained by the first reconstruction module 1320, and the object pose sequence obtained by the first reconstruction submodule 13503;

[0300] a tenth determining unit, configured to determine a reprojection error based on the third pixel position correspondence determined by the third determining submodule 13501 and the fourth pixel position correspondence determined by the ninth determining unit;

[0301] The first correction sub-unit is used to correct the object pose sequence obtained by the first reconstruction sub-module 13503 by minimizing the re-projection error determined by the tenth determination unit.

[0302] In some optional examples, as shown in FIG15-2 , the apparatus provided by the embodiment of the present disclosure further includes:

[0303] A third segmentation module 1390 is configured to perform instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result;

[0304] Modify submodule 13507, including:

[0305] an eleventh determining unit, configured to determine a motion trajectory of the dynamic object in the fourth image sequence obtained by the second segmentation module 1330 based on the object pose sequence obtained by the first reconstruction submodule 13503;

[0306] a twelfth determining unit, configured to determine a first predicted category of the dynamic object in the fourth image sequence obtained by the second segmentation module 1330 based on the instance segmentation result obtained by the third segmentation module 1390;

[0307] a thirteenth determining unit, configured to determine a preset trajectory corresponding to the first prediction category determined by the twelfth determining unit;

[0308] The second correction subunit is used to correct the object posture sequence obtained by the first reconstruction submodule 13503 by minimizing the trajectory error between the motion trajectory determined by the eleventh determination unit and the preset trajectory determined by the thirteenth determination unit.

[0309] In some optional examples, as shown in FIG16 , the apparatus provided by the embodiments of the present disclosure further includes:

[0310] A third segmentation module 1390 is configured to perform instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result;

[0311] The first reconstruction submodule 13503 includes:

[0312] a fourteenth determining unit, configured to determine, based on the instance segmentation result obtained by the third segmentation module 1390 and the third pixel position correspondence determined by the third determining submodule 13501, whether the predicted categories of corresponding pixel positions in the preceding and following frame images in the fourth image sequence obtained by the second segmentation module 1330 are consistent, and obtain a determination result;

[0313] a correction unit, configured to correct the third pixel position correspondence determined by the third determination submodule 13501 based on the determination result obtained by the fourteenth determination unit;

[0314] a reconstruction unit configured to perform sparse reconstruction of the point cloud using the first camera pose sequence obtained by the first reconstruction module 1320, the fourth image sequence obtained by the second segmentation module 1330, and the corrected third pixel position correspondence obtained by the correction unit, to obtain a fourth reconstructed point cloud and an object pose sequence of the dynamic object in the fourth image sequence;

[0315] The second reconstruction submodule 13509 is specifically used to use the first camera pose sequence, the fourth image sequence, the corrected third pixel position correspondence, the fourth reconstructed point cloud and the corrected object pose sequence to perform dense reconstruction of the point cloud to obtain a third reconstructed point cloud.

[0316] In some optional examples, as shown in FIG16 , the apparatus provided by the embodiments of the present disclosure further includes:

[0317] A third segmentation module 1390 is configured to perform instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result;

[0318] The second reconstruction module 1340 includes:

[0319] a fifth determination submodule 13401, configured to determine, for a second target pixel in the third target image in the third image sequence obtained by the second segmentation module 1330, a second predicted category of the second target pixel based on the instance segmentation result obtained by the third segmentation module 1390;

[0320] a sixth determining submodule 13403, configured to determine a size of a local region matching the second prediction category determined by the fifth determining submodule 13401;

[0321] a seventh determining submodule 13405 for determining a target local area from the third target image based on the pixel position of the second target pixel and the size of the local area determined by the sixth determining submodule 13403;

[0322] A third reconstruction submodule 13407 is configured to perform dense point cloud reconstruction on the target local area determined by the seventh determination submodule 13405 using the first reconstructed point cloud obtained by the first reconstruction module 1320, the first camera pose sequence obtained by the first reconstruction module 1320, and the other images in the third image sequence obtained by the second segmentation module 1330, excluding the third target image, to obtain a local point cloud corresponding to the target local area.

[0323] The eighth determination submodule 13409 is configured to determine a second reconstructed point cloud based on the local point cloud obtained by the third reconstruction submodule 13407 .

[0324] In some optional examples, as shown in FIG17 , the apparatus provided by the embodiments of the present disclosure further includes:

[0325] A first prediction module 1370 is configured to perform optical flow prediction on the first image sequence using an optical flow prediction model to obtain an optical flow prediction result;

[0326] The second reconstruction module 1340 includes:

[0327] a ninth determining submodule 13411 configured to determine, based on the optical flow prediction result obtained by the first prediction module 1370 , a fifth pixel position correspondence between the preceding and following frame images in the third image sequence obtained by the second segmentation module 1330 ;

[0328] The fourth reconstruction submodule 13413 is used to use the first reconstructed point cloud obtained by the first reconstruction module 1320, the first camera pose sequence obtained by the first reconstruction module 1320, the third image sequence obtained by the second segmentation module 1330 and the fifth pixel position correspondence determined by the ninth determination submodule 13411 to perform dense reconstruction of the point cloud to obtain a second reconstructed point cloud.

[0329] In some optional examples, as shown in FIG18 , the second reconstruction module 1340 includes:

[0330] a tenth determining submodule 13415 , configured to determine a second camera pose sequence of the camera based on data collected by a non-visual sensor disposed on the movable device;

[0331] a similarity transformation alignment submodule 13417 for performing similarity transformation alignment on the first camera pose sequence obtained by the first reconstruction module 1320 using the second camera pose sequence determined by the tenth determination submodule 13415 as an alignment reference;

[0332] The fifth reconstruction submodule 13419 is used to perform dense reconstruction of the point cloud using the first reconstructed point cloud obtained by the first reconstruction module 1320, the first camera pose sequence obtained by the similarity transformation alignment submodule 13417 after similarity transformation alignment, and the third image sequence obtained by the second segmentation module 1330 to obtain a second reconstructed point cloud.

[0333] In the device of the embodiment of the present disclosure, the various optional embodiments, optional implementation methods and optional examples disclosed above can be flexibly selected and combined as needed to achieve corresponding functions and effects, and the present disclosure does not list them one by one.

[0334] The beneficial technical effects corresponding to the exemplary embodiment of this device can be found in the corresponding beneficial technical effects of the above exemplary method part, which will not be repeated here.

[0335] Exemplary electronic devices

[0336] FIG19 illustrates a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 1900 includes one or more processors 1910 and a memory 1920 .

[0337] The processor 1910 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1900 to perform desired functions.

[0338] The memory 1920 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1910 may execute the one or more computer program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0339] In one example, the electronic device 1900 may further include an input device 1930 and an output device 1940 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0340] The input device 1930 may also include, for example, a keyboard, a mouse, and the like.

[0341] The output device 1940 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0342] Of course, for simplicity, FIG19 only shows some of the components related to the present disclosure in the electronic device 1900, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1900 may further include any other appropriate components according to specific application scenarios.

[0343] Exemplary computer program products and computer-readable storage media

[0344] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions that, when executed by a processor, enable the processor to execute the steps of the scene reconstruction method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0345] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0346] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the scene reconstruction method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0347] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0348] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and non-limiting, and should not be construed as necessarily possessed by each embodiment of the present disclosure. The specific details disclosed above are provided for illustrative purposes and to facilitate understanding, and are not intended to be limiting. The above details do not limit the present disclosure to necessarily being implemented using the above specific details.

[0349] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A scene reconstruction method, comprising: Performing background segmentation on a first image sequence captured by a camera disposed on a movable device to obtain a second image sequence; Using the second image sequence, performing sparse reconstruction of the point cloud to obtain a first reconstructed point cloud and a first camera pose sequence of the camera; Performing static and dynamic segmentation on the first image sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene; Using the first reconstructed point cloud, the first camera pose sequence and the third image sequence, densely reconstruct the point cloud to obtain a second reconstructed point cloud; Using the first camera pose sequence and the fourth image sequence, sequentially performing sparse reconstruction and dense reconstruction of the point cloud to obtain a third reconstructed point cloud; A scene reconstruction result corresponding to the first image sequence is determined based on the second reconstructed point cloud and the third reconstructed point cloud.

2. The method according to claim 1, wherein: The method further comprises: Performing optical flow prediction on the first image sequence using an optical flow prediction model to obtain an optical flow prediction result; Performing depth prediction on the first image sequence using a depth prediction model to obtain a depth prediction result; The step of performing static and dynamic segmentation on the first image sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene includes: Based on the optical flow prediction result, the depth prediction result and the first camera pose sequence, the first image sequence is segmented into static and dynamic images to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene.

3. The method according to claim 2, wherein: The step of performing static and dynamic segmentation on the first image sequence based on the optical flow prediction result, the depth prediction result and the first camera pose sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene includes: Based on the optical flow prediction result, determining a first pixel position correspondence relationship between previous and next frame images in the first image sequence; Determine a second pixel position correspondence relationship between front and rear frame images in the first image sequence based on the depth prediction result and the first camera pose sequence; Based on the first pixel position correspondence relationship and the second pixel position correspondence relationship, the first image sequence is segmented into static and dynamic scenes to obtain a third image sequence corresponding to static scenes and a fourth image sequence corresponding to dynamic scenes.

4. The method according to claim 3, wherein: The step of performing static-dynamic segmentation on the first image sequence based on the first pixel position correspondence relationship and the second pixel position correspondence relationship to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene includes: For a first target pixel point in a first target image in the first image sequence, determining a first target pixel position corresponding to the pixel position of the first target pixel point from a second target image according to the first pixel position correspondence relationship; wherein the second target image is an image frame subsequent to the first target image in the first image sequence; According to the second pixel point correspondence relationship, determining a second target pixel position corresponding to the pixel position of the first target pixel point from the second target image; Determining a pixel attribute of the first target pixel based on a position error between the first target pixel position and the second target pixel position; wherein the pixel attribute of any pixel is used to characterize whether the pixel belongs to a dynamic object; Based on the pixel attributes of the first target pixel points, the first target image is segmented into static and dynamic images to obtain a first segmented image corresponding to a static scene and a second segmented image corresponding to a dynamic scene; Based on the first segmented image, determining the third image sequence; Based on the second segmented image, the fourth image sequence is determined.

5. The method according to claim 3, wherein: The method further comprises: Performing instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result; The step of performing static-dynamic segmentation on the first image sequence based on the first pixel position correspondence relationship and the second pixel position correspondence relationship to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene includes: For a first target image in the first image sequence, determining, based on the instance segmentation result, an instance region of each instance in the first target image; Based on the first pixel position correspondence relationship and the second pixel position correspondence relationship, determining the pixel point attribute of each pixel point in the first target image; wherein the pixel point attribute of any pixel point is used to indicate whether the pixel point belongs to a dynamic object; For each instance region in the first target image, statistics are performed on the pixel attributes of each pixel in the instance region to obtain a statistical result; Based on the statistical results corresponding to each instance region in the first target image, the first target image is segmented into static and dynamic images to obtain a third segmented image corresponding to the static scene and a fourth segmented image corresponding to the dynamic scene; Based on the third segmented image, determining the third image sequence; The fourth image sequence is determined based on the fourth segmented image.

6. The method according to claim 5, wherein: The step of performing static and dynamic segmentation on the first target image based on the statistical results corresponding to each instance region in the first target image to obtain a third segmented image corresponding to a static scene and a fourth segmented image corresponding to a dynamic scene includes: For each instance region in the first target image, based on the statistical result corresponding to the instance region, determining the proportion of pixels in the instance region whose attributes are used to characterize that the current pixel does not belong to a dynamic object; Based on the proportion corresponding to each instance area in the first target image, the first target image is segmented into static and dynamic scenes to obtain a third segmented image corresponding to the static scene and a fourth segmented image corresponding to the dynamic scene.

7. The method according to claim 1, wherein: The method further comprises: Performing optical flow prediction on the first image sequence using an optical flow prediction model to obtain an optical flow prediction result; The step of using the first camera pose sequence and the fourth image sequence to sequentially perform sparse reconstruction and dense reconstruction of the point cloud to obtain a third reconstructed point cloud includes: Based on the optical flow prediction result, determining a third pixel position correspondence relationship between the front and rear frame images in the fourth image sequence; Performing sparse reconstruction of a point cloud by using the first camera pose sequence, the fourth image sequence and the third pixel position correspondence to obtain a fourth reconstructed point cloud and an object pose sequence of the dynamic object in the fourth image sequence; Correcting the position and posture sequence of the object according to a preset correction method; Dense reconstruction of the point cloud is performed using the first camera pose sequence, the fourth image sequence, the third pixel position correspondence, the fourth reconstructed point cloud and the corrected object pose sequence to obtain the third reconstructed point cloud.

8. The method according to claim 7, wherein: The method further comprises: Performing depth prediction on the first image sequence using a depth prediction model to obtain a depth prediction result; The step of correcting the object posture sequence according to a preset correction method includes: Determine a fourth pixel position correspondence relationship between front and rear frame images in the fourth image sequence based on the depth prediction result, the first camera pose sequence, and the object pose sequence; Determining a reprojection error based on the third pixel position correspondence relationship and the fourth pixel position correspondence relationship; The object pose sequence is corrected by minimizing the reprojection error.

9. The method according to claim 7, wherein: The method further comprises: Performing instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result; The step of correcting the object posture sequence according to a preset correction method includes: Determining a motion trajectory of a dynamic object in the fourth image sequence based on the object posture sequence; Determining a first predicted category of a dynamic object in the fourth image sequence based on the instance segmentation result; Determining a preset trajectory corresponding to the first prediction category; The object posture sequence is corrected by minimizing the trajectory error between the motion trajectory and the preset trajectory.

10. The method according to claim 7, wherein: The method further comprises: Performing instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result; The method of performing sparse reconstruction of a point cloud by using the first camera pose sequence, the fourth image sequence and the third pixel position correspondence to obtain a fourth reconstructed point cloud and an object pose sequence of the dynamic object in the fourth image sequence comprises: Based on the instance segmentation result and the third pixel position correspondence relationship, determining whether the predicted categories of corresponding pixel positions in the previous and next frame images in the fourth image sequence are consistent, and obtaining a determination result; Based on the determination result, correcting the third pixel position correspondence relationship; Performing sparse reconstruction of a point cloud by using the first camera pose sequence, the fourth image sequence and the corrected third pixel position correspondence to obtain a fourth reconstructed point cloud and an object pose sequence of the dynamic object in the fourth image sequence; The method of performing dense reconstruction of a point cloud by using the first camera pose sequence, the fourth image sequence, the third pixel position correspondence, the fourth reconstructed point cloud and the corrected object pose sequence to obtain the third reconstructed point cloud comprises: Dense reconstruction of the point cloud is performed using the first camera pose sequence, the fourth image sequence, the corrected third pixel position correspondence, the fourth reconstructed point cloud and the corrected object pose sequence to obtain the third reconstructed point cloud.

11. The method according to claim 1, wherein: The method further comprises: Performing instance segmentation on the first image sequence using an instance segmentation model to obtain an instance segmentation result; The method of performing dense reconstruction of a point cloud using the first reconstructed point cloud, the first camera pose sequence, and the third image sequence to obtain a second reconstructed point cloud includes: For a second target pixel in a third target image in the third image sequence, determining a second predicted category of the second target pixel based on the instance segmentation result; determining a size of a local region matching the second prediction category; Determining a target local area from the third target image based on a pixel position of the second target pixel and a size of the local area; For the local target area, densely reconstruct the point cloud by using the first reconstructed point cloud, the first camera pose sequence, and other images in the third image sequence except the third target image, to obtain a local point cloud corresponding to the local target area; Based on the local point cloud, the second reconstructed point cloud is determined.

12. A scene reconstruction device, comprising: A first segmentation module is used to perform background segmentation on a first image sequence captured by a camera disposed on a movable device to obtain a second image sequence; A first reconstruction module, configured to perform sparse reconstruction of a point cloud using the second image sequence obtained by the first segmentation module to obtain a first reconstructed point cloud and a first camera pose sequence; A second segmentation module is used to perform dynamic and static segmentation on the first image sequence to obtain a third image sequence corresponding to a static scene and a fourth image sequence corresponding to a dynamic scene; A second reconstruction module is used to perform dense reconstruction of the point cloud using the first reconstructed point cloud obtained by the first reconstruction module and the first camera pose sequence, and the third image sequence obtained by the second segmentation module to obtain a second reconstructed point cloud; A third reconstruction module is used to use the first camera pose sequence obtained by the first reconstruction module and the fourth image sequence obtained by the second segmentation module to perform sparse reconstruction and dense reconstruction of the point cloud in sequence to obtain a third reconstructed point cloud; a determination module, configured to determine, based on the second reconstruction point cloud obtained by the second reconstruction module and the third reconstruction point cloud obtained by the third reconstruction module, Determine a scene reconstruction result corresponding to the first image sequence.

13. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the scene reconstruction method described in any one of claims 1 to 11.

14. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the scene reconstruction method described in any one of claims 1-11 above.

Images