Method and device for generating training data, computer-readable medium and electronic device

Abstract

The present disclosure provides a method for generating training data, a device for generating training data, a computer-readable medium and an electronic device, and relates to the field of image processing technology. The method comprises: obtaining a video set; the video set comprises at least one 2D video; performing frame screening on each 2D video based on a preset tracking condition to obtain a tracking frame sequence corresponding to each 2D video; performing three-dimensional reconstruction based on each tracking frame sequence to obtain a reconstructed image and depth data corresponding to the reconstructed image, and using the reconstructed image and the depth data corresponding to the reconstructed image as the first training data. The present disclosure can convert existing videos on the Internet, shot by different devices, and with rich scenes into depth true values, and generate a huge amount of training data with rich scenes, so as to train machine learning or deep learning networks related to depth data based on these data.

Description

Background Art

[0002] Depth maps are widely used in computer vision tasks, such as visual odometer, visual mapping, visual SLAM, 3D reconstruction, scene perception, spot rendering, free viewpoint rendering, etc. In related technologies, there are generally three methods to obtain the true depth value in the depth map: the first is to capture the true value data through a depth sensor, including Time-of-flight (ToF), structured light and binocular stereo matching; the second is to manually mark the distance relationship between two points on the color image; the third is based on machine learning or deep learning methods such as self-supervised learning, and obtain a depth map with consistent posture with the assistance of an inter-frame posture prediction network.

[0003] Among the above methods, the first method requires professional acquisition equipment, and each acquisition can only obtain a single type of scene. At the same time, the traditional depth sensors used for acquisition (such as Tof, structured light, and binocular stereo matching) have high equipment costs and high power consumption; the second method relies on a lot of manual operations, and can only obtain the distance relationship of sparse point pairs; the third method requires a lot of training data, and there are certain requirements for training data. For example, in the self-supervised learning method based on posenet+depthnet, the video data used for training is required to be shot in a one-shot manner. However, deep learning algorithms are highly dependent on training data, but it is impractical to collect data for an unlimited number of scenes through depth sensors. Summary of the invention

[0004] The purpose of the present disclosure is to provide a method for generating training data, a device for generating training data, a computer-readable medium and an electronic device, so as to achieve the goal of providing training data with abundant quantity and high quality at least to a certain extent.

[0005] According to a first aspect of the present disclosure, a method for generating training data is provided, comprising: acquiring a video set; the video set comprising at least one 2D video; performing frame screening on each 2D video based on a preset tracking condition to obtain a tracking frame sequence corresponding to each 2D video; performing three-dimensional reconstruction based on each tracking frame sequence to obtain a reconstructed image and depth data corresponding to the reconstructed image, and using the reconstructed image and the depth data corresponding to the reconstructed image as first training data.

[0006] According to a second aspect of the present disclosure, a training data generation device is provided, comprising: a video acquisition module, used to acquire a video set; the video set includes at least one 2D video; a data screening module, used to perform frame screening on each 2D video based on a preset tracking condition to obtain a tracking frame sequence corresponding to each 2D video; a data generation module, used to perform three-dimensional reconstruction based on each tracking frame sequence to obtain a reconstructed image and depth data corresponding to the reconstructed image, and use the reconstructed image and the depth data corresponding to the reconstructed image as first training data.

[0007] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0008] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; and a memory for storing one or more programs, which enables the one or more processors to implement the above method when the one or more programs are executed by the one or more processors.

[0009] A method for generating training data provided by an embodiment of the present disclosure can obtain tracking sequence frames corresponding to 2D videos that meet the preset tracking conditions by performing frame screening on each 2D video in the acquired video set based on preset tracking conditions, and then perform three-dimensional reconstruction based on the tracking frame sequence to obtain a reconstructed image and depth data corresponding to the reconstructed image, and then use the reconstructed image and depth data as the first training data. Through the above method, videos with rich scenes shot by different devices and available on the Internet can be converted into true depth values ​​to generate a huge amount of training data with rich scenes, so as to train machine learning or deep learning networks related to depth data based on these data. For example, the above huge amount of training data with rich scenes is very suitable for training a monocular depth prediction network with strong generalization.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0012] Figure 1 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0013] Figure 2 A schematic diagram showing an electronic device to which the embodiments of the present disclosure can be applied;

[0014] Figure 3 A flowchart schematically illustrates a method for generating training data in an exemplary embodiment of the present disclosure;

[0015] Figure 4 A flowchart schematically illustrates a frame screening processing method in an exemplary embodiment of the present disclosure;

[0016] Figure 5 A schematic diagram showing a comparison of dense three-dimensional results of 2D video based on different initialization methods in an exemplary embodiment of the present disclosure;

[0017] Figure 6 A schematic diagram showing a comparison of dense three-dimensional results of another 2D video based on different initialization methods in an exemplary embodiment of the present disclosure;

[0018] Figure 7 A schematic diagram schematically shows four common sparse reconstruction results that are wrong or do not conform to the actual motion situation;

[0019] Figure 8 Schematically illustrating a reconstructed image and a semantic segmentation image corresponding to the reconstructed image in an exemplary embodiment of the present disclosure;

[0020] Fig. 9 Schematically showing a comparison diagram before and after deep doping elimination in an exemplary embodiment of the present disclosure;

[0021] Fig.10 Schematically showing a reconstructed image, a geometrically consistent depth image, and a photometrically consistent depth image corresponding to a dense reconstruction result in an exemplary embodiment of the present disclosure;

[0022] Fig.11 A flow chart schematically illustrates a cleaning method for cleaning invalid debris in an exemplary embodiment of the present disclosure;

[0023] Fig.12 A flow chart schematically illustrates a cleaning method for cleaning an edge portion in an exemplary embodiment of the present disclosure;

[0024] Fig.13 Schematically illustrating a first confidence region and a second confidence region in an exemplary embodiment of the present disclosure;

[0025] Fig.14 Schematically illustrating a third confidence region and a fourth confidence region in an exemplary embodiment of the present disclosure;

[0026] Fig.15Schematically showing a comparison diagram of the edge portion before and after cleaning in an exemplary embodiment of the present disclosure;

[0027] Fig.16 A schematic diagram schematically illustrates a process of generating relative depth data in an exemplary embodiment of the present disclosure;

[0028] Fig.17 A schematic diagram schematically illustrates a process of eliminating deep doping of a reconstructed image corresponding to a dummy challenge video in an exemplary embodiment of the present disclosure;

[0029] Fig.18 A flow chart schematically illustrates a method for cleaning dense reconstruction results in an exemplary embodiment of the present disclosure;

[0030] Fig.19 A schematic diagram schematically illustrates a principle of generating second training data based on a 3D video in an exemplary embodiment of the present disclosure;

[0031] Fig. 20 Schematically illustrating reconstructed image data sets corresponding to two different scene categories in an exemplary embodiment of the present disclosure;

[0032] Fig.21 Schematically illustrates an exemplary embodiment of the present disclosure. Fig. 20 The reconstructed image dataset obtained by reducing the dimensionality of the two reconstructed image datasets in ;

[0033] Fig. 22 A flowchart schematically illustrates another method for generating training data in an exemplary embodiment of the present disclosure;

[0034] Fig.23 The following is a schematic diagram schematically showing the composition of a device for generating training data in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0036] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0037] Figure 1 A schematic diagram of a system architecture of an exemplary application environment in which a method and apparatus for generating training data according to an embodiment of the present disclosure can be applied is shown.

[0038] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with image processing functions, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0039] The method for generating training data provided in the embodiment of the present disclosure can be executed by the terminal devices 101, 102, and 103, and accordingly, the device for generating training data can be arranged in the terminal devices 101, 102, and 103. However, it is easy for a person skilled in the art to understand that the method for generating training data provided in the embodiment of the present disclosure can also be executed by the server 105, and accordingly, the device for generating training data can also be arranged in the server 105, which is not particularly limited in the present exemplary embodiment. For example, in an exemplary embodiment, a user may shoot a video through the terminal devices 101 and 102, and upload the video to various video servers on the Internet through the network 104, and then the terminal device 103 may download the video from various video websites through the network 104 to obtain a video set, and generate the first training data based on the video set.

[0040] An exemplary embodiment of the present disclosure provides an electronic device for implementing a method for generating training data, which may be Figure 1The terminal device 101, 102, 103 or the server 105 in the embodiment. The electronic device at least includes a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the method for generating training data by executing the executable instructions.

[0041] Below Figure 2 Taking the mobile terminal 200 in FIG. 1 as an example, the structure of the electronic device is exemplarily described. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2 The structure in the figure can also be applied to fixed type devices. In other embodiments, the mobile terminal 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the components is only shown schematically and does not constitute a structural limitation of the mobile terminal 200. In other embodiments, the mobile terminal 200 may also adopt the same Figure 2 Different interface connection methods, or a combination of multiple interface connection methods.

[0042] like Figure 2 As shown, the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyroscope sensor 2803, etc.

[0043] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0044] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. NPU can realize applications such as intelligent cognition of the mobile terminal 200, such as image recognition, face recognition, voice recognition, text understanding, etc. In some embodiments, NPU can be used for semantic segmentation, adaptive adjustment of preset parameters, and three-dimensional reconstruction.

[0045] The processor 210 is provided with a memory. The memory can store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and the execution is controlled by the processor 210.

[0046] The mobile terminal 200 implements the display function through a GPU, a display screen 290, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0047] The mobile terminal 200 can realize the shooting function through the ISP, the camera module 291, the video codec, the GPU, the display screen 290 and the application processor, etc. Among them, the ISP is used to process the data fed back by the camera module 291; the camera module 291 is used to capture static images or videos; the digital signal processor is used to process digital signals, and can process other digital signals in addition to digital image signals; the video codec is used to compress or decompress digital videos, and the mobile terminal 200 can also support one or more video codecs.

[0048] In some embodiments, the video may be rendered, segmented, and processed based on the GPU, display screen 290 , application processor, ISP, camera module 291 , video codec, and the like.

[0049] Depth maps are widely used in computer vision tasks, such as visual odometer, visual mapping, visual SLAM, 3D reconstruction, scene perception, spot rendering, free viewpoint rendering, etc. Traditional depth sensors (such as Tof, structured light, and binocular stereo matching) are expensive and power-hungry. Only one camera is needed to restore the depth map from a single color image, which can minimize the production cost and use cost. Depending on whether the signal is actively emitted during operation, depth sensors are divided into active sensors and passive sensors. Active sensors need to use lasers or pattern projectors to project special signals to the target scene, and restore the depth by analyzing the sensor's response to the signal. Passive sensors only require the target to be under natural light or light conditions visible to the naked eye. Traditional passive monocular depth estimation algorithms have special requirements for scenes and shooting methods, such as specific lighting, special focusing methods, local texture of target objects, etc., and are not suitable for general use scenarios.

[0050] Deep learning-based algorithms learn the nonlinear mapping relationship between the target scene and the depth map by updating (training) the model parameters of large-scale data, and then predict more accurate depth. However, deep learning algorithms are highly dependent on training data. For example, a model trained only with vehicle data has difficulty processing indoor images. In order to train a model with strong generalization, we need a huge amount of training data with rich scenes. Using depth sensors to collect data for an unlimited number of scenes is impractical.

[0051] Based on one or more of the above problems, this exemplary embodiment provides a method for generating training data. The method for generating training data can generate a large amount of training data with rich scenes based on videos with rich scenes shot by different devices uploaded by netizens. The method for generating training data can be applied to one or more of the above terminal devices 101, 102, 103, and can also be applied to the above server 105, which is not particularly limited in this exemplary embodiment. Figure 3 As shown, the method for generating training data may include the following steps S310 to S330:

[0052] In step S310, a video set is obtained.

[0053] The video set includes at least one 2D video.

[0054] In an exemplary embodiment, due to the rise of Internet streaming media, tens of thousands of netizens upload videos with rich scenes shot with different devices to video websites every day. The scenes shot by these videos have the advantages of rich content, wide time and place span, diverse shooting styles, and diverse shooting devices. Therefore, if Internet streaming media videos can be used to generate training data, the training data obtained will also have the characteristics of rich scene types, and thus obtain machine learning and deep learning models with wider trial scenarios. Based on the above reasons, when obtaining a video collection, video data can be crawled on the Internet based on a web crawler.

[0055] Specifically, we can use web packet capture analysis, dynamic loading of JSON data parsing, youtube-dl and other related technologies to achieve automatic video retrieval and batch downloading of multiple video websites. It should be noted that in order to obtain a video collection with rich scenes, we can first identify some video websites that can provide rich scenes and types before crawling videos, and then search and batch download based on keywords, video sources, etc.

[0056] For example, in an embodiment of batch downloading based on keyword search results, the automated testing framework selenium can first simulate browser operations to automatically search for open video website data with specified keywords, and then, through packet capture analysis, parse the URL of each video sub-page from the JSON data returned after the search result page receives the request, and parse the real address of the video from the JSON data returned after the video sub-page receives the request, and then, based on the written script, simulate the web page request through the requests library, automatically parse the JSON data to obtain the real address of the video file, download the video, format the name, and output the video information.

[0057] For another example, in an embodiment of batch downloading based on designated video authors, the URL rules of the open designated author's homepage, the video sub-page URL rules, etc. can also be found through packet capture analysis, and then the real address of the video can be parsed from the JSON data returned after the video sub-page receives the request, and then the homepage URL generation, homepage access request, video sub-page URL parsing, video file real address parsing, video downloading, formatting and naming, and video information output of the designated video author can be completed based on the written script.

[0058] It should be noted that the open video websites or open designated author homepages refer to websites or homepages that are free of copyright and other restrictions, or are available after consultation with the copyright owner or other parties.

[0059] In step S320, frames of each 2D video are screened based on a preset tracking condition to obtain a tracking frame sequence corresponding to each 2D video.

[0060] In an exemplary embodiment, since 3D reconstruction is a time-consuming process and the output file is very large, before 3D reconstruction, the video data can be screened to filter out video frames with a high probability of reconstruction failure or high similarity in the video, so as to shorten the time consumption of 3D reconstruction and make the output file smaller. Specifically, frame screening processing can be performed for each 2D video. The frame screening process may include: first extracting the first video frame from the 2D video, determining whether the first video frame satisfies the preset tracking condition, and when the first video frame satisfies the preset tracking condition, adding the first video frame to the video subsequence; continuing to extract the second video frame based on the preset step length, and continuing to determine whether the second video frame satisfies the preset tracking condition, and when the second video frame satisfies the preset tracking condition, adding the second video frame to the video subsequence; looping the above extraction and judgment process, when the i-th video frame extracted satisfies the preset tracking condition, adding the i-th video frame to the video subsequence, and continuing to extract the i+1-th video frame; when the i-th video frame extracted does not satisfy the preset tracking condition, determining the video subsequence generated by all video frames satisfying the preset tracking condition before the i-th video frame as the tracking frame sequence corresponding to the 2D video. Wherein, i is a positive integer.

[0061] It should be noted that when shooting a video, if the shooter moves slowly or is stationary, or the subject moves slowly or is prohibited, there may be high-similarity continuous frames, large-angle pure rotations, and other video frames in the video that are prone to tracking failures. Therefore, the video frames can be screened based on their characteristics and their connection with the previously extracted video frames. Specifically, the preset tracking conditions may include: the number of feature points in the video frame is greater than the number threshold, the average modulus of the optical flow matching vector corresponding to the video frame is less than the maximum value of the modulus range, the number of inliers corresponding to the video frame is greater than the inlier threshold, and the inter-frame rotation angle corresponding to the video frame is less than the angle threshold.

[0062] In particular, when the average modulus of the optical flow matching vector corresponding to the video frame is less than the minimum value of the modulus range, it can be considered that the two adjacent frames are almost stationary. To avoid extracting video frames with too high similarity, the i-th video frame can be directly extracted without adding it to the video frame subsequence.

[0063] In addition, since the first video frame cannot be determined based on the preset step length, a fixed number of frames can be set in advance as the first video frame, and then the first video frame is used as a reference to extract subsequent video frames using the preset step length. For example, the nth frame of the extracted video can be set as the first video frame; the first video frame can also be determined by other methods, which are not specifically limited in the present disclosure.

[0064] In an exemplary embodiment, since the three-dimensional reconstruction process requires a sufficient number of video frames, it is necessary to control the number of frames of the video subsequence screened out from the 2D video. Specifically, when the number of frames of the video subsequence is less than the minimum threshold, in order to avoid being unable to output a reconstruction result that meets the requirements during the three-dimensional reconstruction process, the sequence can be discarded, that is, the 2D video has no corresponding tracking sequence and cannot construct training data; and when the number of frames of the video subsequence is greater than or equal to the minimum threshold, the video subsequence can be directly determined as the tracking frame sequence corresponding to the 2D video; in addition, in order to prevent the three-dimensional reconstruction from taking too long due to too many frames in the tracking frame sequence, a maximum threshold can also be set. When the number of frames of the video subsequence is greater than the maximum threshold, the video subsequence can be cut, and the video subsequence with a frame number equal to the maximum threshold is used as the tracking frame sequence corresponding to the 2D video for subsequent processing.

[0065] It should be noted that when cutting a video subsequence, a continuous sequence with a frame number equal to the highest threshold needs to be cut out from the video subsequence as a tracking frame sequence, and the present disclosure does not specifically limit the specific cutting method. For example, when the highest threshold is 10 frames, if the video subsequence includes the 1st to 15th video frames extracted from the 2D video, the 1st to 10th video frames can be extracted as the tracking frame sequence, or the 2nd to 11th, the 3rd to 12th, etc. can be extracted, and the present disclosure does not specifically limit this.

[0066] The following example takes the preset tracking conditions including all the above conditions as an example. Figure 4 As shown, the frame filtering process is described in detail:

[0067] Step S401, extracting the i-th video frame according to a preset step size; wherein i is a positive integer of 1, 2, ...

[0068] The preset step size may be an empirical value selected based on normal video shooting experience, or may be determined by adding a model or algorithm that adaptively generates the corresponding preset step size based on the video. For example, the corresponding preset step size may be determined based on the average modulus of the vectors of the matching point pairs between the video frames in the video. The smaller the average modulus of the vectors of the matching point pairs, the greater the similarity between the video frames, so a larger preset step size may be selected to avoid screening out too many adjacent video frames with high similarity.

[0069] Step S403: extract feature points in the i-th video frame.

[0070] Step S405 , determining whether the number of feature points in the i-th video frame is greater than a number threshold.

[0071] The number threshold can be set based on experience. Since feature points in a video frame are usually used for tracking, if the number of feature points is too small, there may be a lack of sufficient feature points in the video frame for tracking, and tracking loss may occur at the position of the video frame during 3D reconstruction, which may cause 3D reconstruction failure. Therefore, video frames can be screened based on the number of feature points.

[0072] Step S407 , when the number of feature points in the i-th video frame is greater than the number threshold, optical flow calculation is performed on the i-th video frame to obtain an average module length of the optical flow matching vector.

[0073] When a sufficient number of feature points can be extracted, a point correspondence relationship can be established between the i-th video frame and the i-1-th video frame by optical flow calculation, and the average modulus length of the matching vectors between the matching point pairs can be calculated.

[0074] Step S409, determining whether the average modulus length of the optical flow matching vector is within the modulus length range.

[0075] If the average modulus is less than the minimum value of the modulus range, it is considered that the two adjacent frames are almost still. To avoid too high similarity, the current frame is skipped and the i+1th video frame is extracted in the video according to the preset step size. If the average modulus is greater than the maximum value of the preset range, it is considered that the parallax between the two adjacent frames is too large, which may easily cause tracking loss. Therefore, video frames can be screened based on the average modulus.

[0076] Step S411, when the average modulus of the optical flow matching vector is within the modulus range, a basic matrix is ​​solved based on the matching point pairs, and the number of inliers in the matching point pairs included in the video frame is calculated.

[0077] If the average modulus of the matching vectors between the matching point pairs meets the preset modulus range, the basic matrix can be solved based on the matching point pairs to determine the solved inliers. The solution method can include the RANSAC method, and the inliers refer to the matching point pairs that make the estimated basic matrix error smaller.

[0078] Step S413, determining whether the number of inliers in the matching point pairs in the video frame exceeds an inlier threshold.

[0079] If the number of inliers is too small, it is considered that there are not enough feature matching relationships between two adjacent frames to solve the inter-frame pose, which may easily cause tracking loss. Therefore, video frames can be screened based on the number of inliers.

[0080] Step S415, when the number of inliers exceeds the inlier threshold, decompose the basic matrix obtained in step S411 to solve the inter-frame pose and calculate the inter-frame rotation angle.

[0081] The inter-frame rotation angle is used to characterize the rotation change of the pose between adjacent frames. When a large angle rotation occurs between adjacent frames, it is easy to cause errors or failures in the inter-frame pose solution during tracking, which may lead to tracking loss. Therefore, video frames can be filtered based on the inter-frame rotation angle.

[0082] Step S417, determining whether the inter-frame rotation angle is less than an angle threshold.

[0083] Step S419: when the inter-frame rotation angle is less than the angle threshold, the i-th video frame may be added to the video subsequence, and the process returns to step S401 to continue extracting the i+1-th video frame.

[0084] If the i-th video frame meets the preset tracking conditions of the above series, it is considered that the current video frame is likely to be stably tracked during the 3D reconstruction process, so the frame is added to the current subsequence and the extraction and processing of the next video frame begins.

[0085] Step S421 , when the i-th video frame does not satisfy any of the above conditions, or the current 2D video cannot extract the i+1-th video frame, it is determined whether the number of frames of the existing video subsequence is less than a minimum threshold.

[0086] If the i-th video frame does not meet any of the above preset tracking conditions, it can be determined that if the i-th video frame is added to the video subsequence, it is likely that the tracking loss problem will occur at the i-th video frame during the subsequent three-dimensional reconstruction, so the i-th video frame cannot be added to the video subsequence. It should be noted that in the video subsequence, the order of the video frames is consistent with the order when they are extracted, and the order of the video frames in the 2D video. Among them, the current 2D video cannot extract the i+1th video frame means that when the i-th video frame is extracted, the last few video frames of the 2D video have been extracted, and the i+1th video frame cannot be obtained by continuing to extract according to the preset step size; when the i-th video frame does not meet any of the above conditions, the existing video subsequence refers to the video subsequence formed by the 1st video frame to the i-1th video frame; when the current 2D video cannot extract the i+1th video frame, the existing video subsequence refers to the video subsequence formed by the 1st video frame to the i-th video frame.

[0087] In addition, since the 3D reconstruction process requires a sufficient number of video frames, it is necessary to control the frame number of the video subsequences screened out from the 2D video.

[0088] Step S423: when the number of frames of the video subsequence is less than the minimum threshold, the video subsequence is discarded, and the tracking frame sequence of the 2D video frame is determined to be empty.

[0089] If the number of frames of a video subsequence is less than a minimum threshold, it is considered that the number of frames of the video subsequence is too small and cannot output a reconstruction result that meets the requirements in the subsequent 3D reconstruction. Therefore, the subsequence is discarded and the tracking frame sequence of the 2D video frame is determined to be empty, that is, it is impossible to continue to generate training data based on the 2D video.

[0090] Step S425: when the number of frames of the video subsequence is greater than or equal to the minimum threshold, the video subsequence is used as a tracking frame sequence.

[0091] It should be noted that a maximum threshold can also be set. If the number of frames in a video subsequence is greater than the maximum threshold, in order to prevent the 3D reconstruction from taking too long due to too many frames in the video subsequence, the video subsequence can be cut and the cut video subsequence can be used as the tracking frame sequence corresponding to the 2D video for subsequent processing.

[0092] In step S330, three-dimensional reconstruction is performed based on each tracking frame sequence to obtain a reconstructed image and depth data corresponding to the reconstructed image, and the reconstructed image and the depth data corresponding to the reconstructed image are used as first training data.

[0093] The reconstructed image refers to an image in RGB or other formats corresponding to the depth image obtained through 3D reconstruction. The image does not include depth data and is a complete scene image restored based on multiple video frames, such as Figure 5 and 6 In Figure 5 a and Figure 6 a; The first training data is used as the input of the model trained using the pre-training data (second training data), and further training is performed to obtain a more refined model.

[0094] In an exemplary embodiment, after obtaining a tracking frame sequence, for each tracking frame sequence, sparse 3D reconstruction may be performed first to obtain a sparse 3D reconstruction result; then dense 3D reconstruction may be performed based on the sparse 3D reconstruction result to obtain a dense reconstruction result; then a reconstructed image and depth data corresponding to the reconstructed image may be generated based on the dense reconstruction result. The depth data corresponding to the reconstructed image generated by the dense reconstruction result may include a depth image.

[0095] In an exemplary embodiment, when performing sparse 3D reconstruction, a method of Structure From Motion (SFM) can be used to perform multi-view sparse reconstruction based on a tracking frame sequence. In addition, sparse 3D reconstruction can also be performed by other methods, which are not specifically limited in the present disclosure.

[0096] It should be noted that in actual use, due to the multi-view sparse reconstruction, although some sequences can output sparse reconstruction results normally, there are still some sparse reconstruction results that are wrong or do not conform to the actual motion situation. Figure 7 As shown in the figure, the sparse reconstruction results that are wrong or do not conform to the actual motion situation include the following possibilities: the number of map points is small (such as Figure 7 a) Small number of cameras (e.g. Figure 7 b), abnormal inter-frame rotation (such as Figure 7 c), frame translation anomaly (such as Figure 7 d), etc. The darkest color represents the camera trajectory, and the rest represent the recovered sparse point clouds. Therefore, the sparse reconstruction results can be screened based on the continuity of the motion trajectory, and the wrong sparse reconstruction results or those that do not conform to the actual motion situation can be deleted to obtain the screened sparse reconstruction results, so as to perform dense 3D reconstruction based on the screened sparse reconstruction results.

[0097] Specifically, since the input tracking frame sequence in the sparse reconstruction process is an ordered continuous image sequence screened from the video, the real motion trajectory of this type of sequence is continuous and consistent. Therefore, the sparse reconstruction results can be jointly judged based on parameters such as the number of point clouds, the number of cameras, inter-frame rotation, and inter-frame displacement to filter out sparse reconstruction results with poor motion trajectory continuity, so as to improve the success rate of subsequent dense reconstruction. For example, the goal of joint judgment can be achieved by setting thresholds for parameters such as the number of point clouds, the number of cameras, inter-frame rotation, and inter-frame displacement. When the number of point clouds and the number of cameras are lower than the threshold, and the inter-frame rotation and inter-frame displacement are greater than the threshold, it is determined that the continuity of the motion trajectory is poor, and the sparse reconstruction result is deleted.

[0098] In an exemplary embodiment, after obtaining the sparse reconstruction result, dense 3D reconstruction can be performed based on the sparse reconstruction result to obtain a dense reconstruction result. When performing dense 3D reconstruction, various multi-view dense reconstruction methods can be used for reconstruction, and the present disclosure does not impose any special restrictions on this.

[0099] In an exemplary embodiment, a depth propagation method can be used for depth recovery. This method uses a random value to initialize the unknown depth, and then iteratively updates the depth based on the expectation maximization method, thereby obtaining a depth estimation result with higher accuracy. However, the use of random initial values ​​can easily lead to a large difference between the estimated depth and the true value or even a recovery failure. In order to solve this problem, the coordinates and sparse depth values ​​corresponding to each sparse point in the sparse reconstruction result can be obtained; then the sparse depth value is used as the corresponding initial depth value for three-dimensional dense reconstruction to obtain a dense reconstruction result. Through the sparse depth value in the sparse reconstruction result, a relatively accurate initial value can be used for initialization, so that the result of dense reconstruction is closer to the true value, and it also has a good reconstruction effect for the area where dense reconstruction fails.

[0100] For example, refer to Figure 5 and Figure 6 As shown, Figure 5 a and Figure 6 a is an RGB image, Figure 5 b and Figure 6 b is the effect of densely reconstructing the RGB image using random values ​​for initialization. Figure 5 c and 6c are the effects of dense reconstruction of RGB images using sparse depth values ​​for initialization. By comparison, it can be seen that compared with initialization using random values, the dense 3D reconstruction results initialized with sparse depth values ​​can restore the depth values ​​of more areas, as shown in the boxed part in the figure.

[0101] In an exemplary embodiment, the depth data generated by dense reconstruction may have some areas that are prone to large errors, such as fragmented areas, reflective areas, and weak texture areas. These problems will pollute the training data finally generated, which will lead to problems such as slow model training and suboptimal training results. Therefore, before determining the reconstructed image and the depth data corresponding to the reconstructed image based on the dense reconstruction result, the dense reconstruction result can be cleaned based on the semantic segmentation algorithm to obtain the cleaned dense reconstruction result, so as to determine the reconstructed image and the depth data corresponding to the reconstructed image based on the cleaned dense reconstruction result.

[0102] Among them, the semantic segmentation algorithm is one of the basic topics of computer vision. Its goal is to assign a semantic label to each pixel value of the image. Semantic labels mainly include semantic classifications such as tasks, animals, plants, buildings, sky, and water. For semantic segmentation tasks, the input is an RGB image, and the output is a mask image that assigns a category label to each pixel value. For example, Figure 8 b shows the Figure 8 a The semantic segmentation map obtained by semantic segmentation.

[0103] In an exemplary embodiment, under the premise of multi-frame three-dimensional reconstruction, if there is a situation in which a background object is blocked by a foreground object in the reconstructed image, for example, when a distant building is blocked by a nearby person, since the number of frames in which the background object is observed in the multi-frame data is often greater than that of the foreground object, it may cause background pixels to be mixed in the area corresponding to the foreground object in the depth image. To address this problem, when cleaning the dense reconstruction result based on the semantic segmentation algorithm, a depth image can be first determined by the depth data in the dense reconstruction result, and then a first area can be determined in the depth image by a semantic style algorithm, and then the first area is processed based on a preset algorithm to eliminate the depth mixing of the second area with the first area, and obtain a dense reconstruction result after cleaning. Among them, the depth of the second area is greater than that of the first area.

[0104] In an exemplary embodiment, when the semantic label of the first area corresponds to a non-living thing, the first area can be processed based on photometric consistency and geometric consistency. Specifically, dense three-dimensional reconstruction can obtain two results, namely a photometric consistency result and a geometric consistency result. It is possible to directly determine whether the geometric consistency depth value exceeds a preset multiple of the photometric consistency depth value. When the geometric consistency depth value exceeds the preset multiple of the photometric consistency depth value, the photometric consistency depth value is used, otherwise the geometric consistency depth value is used. Since the problem of background depth value doping with foreground depth value only occurs when the background depth value is sufficiently larger than the foreground depth value, the depth doping of the second area (background area) to the first area (foreground area) can be eliminated by a preset multiple. For example, Fig. 9 It shows that the above method eliminates the deep doping before the preset ratio is 1.1 ( Fig. 9 a) and after elimination ( Fig. 9 In the comparison image b), we can see that the background interference in the tree area in the foreground has been eliminated to a certain extent.

[0105] Among them, the photometric consistency result is only constrained by the photometric consistency of the local area of ​​the image to constrain the matching degree, while the geometric consistency result uses the multi-frame matching relationship to constrain the current match based on the photometric consistency. Therefore, the geometric consistency result can filter out most of the weak texture areas, such as the sky and the pure color ground, such as Fig.10 The original image in ( Fig.10 a) and geometrically consistent depth image ( Fig.10 It should be noted that, since it is difficult to find matching relationships in weak texture areas, only some areas of the depth image obtained by 3D reconstruction have reliable depth values. Correspondingly, only these reliable depth values ​​can be used as effective training data for sequencing. For example, Fig.10 The reconstructed image in ( Fig.10 a), geometrically consistent depth image ( Fig.10b) and photometrically consistent depth image ( Fig.10 The depth values ​​corresponding to the sky and road parts in c) are invalid.

[0106] In an exemplary embodiment, when the semantic label of the first region corresponds to a living thing, for different biological objects, since the background depth is often greater than the foreground depth, and the depth change in the same biological object is generally small, a median filter can be used to eliminate the doping of the second region (background region) with the biological region (first region). Specifically, different organisms, such as different people, can be determined based on the semantic segmentation results; then, for each biological region, median filtering is performed on the biological region to eliminate the depth doping of the background region with the biological region.

[0107] In an exemplary embodiment, in addition to the above-mentioned depth doping problem, there may also be a problem of invalid depth value fragmentation appearing in the depth image, so it is necessary to clean these fragmented invalid depth values ​​to determine the reconstructed image and the depth data corresponding to the reconstructed image based on the dense reconstruction result after cleaning. To address this problem, when cleaning the invalid fragments, specifically, refer to Fig.11 As shown, the following steps S1110 to S1140 may be included:

[0108] In step S1110 , invalid depth values ​​and valid depth values ​​are determined in the depth data in the dense reconstruction result based on a median filtering algorithm.

[0109] In an exemplary embodiment, a window of a preset pixel size can be used to perform median filtering on the depth image, and a depth image A before filtering and a depth image B after filtering can be obtained. Then, based on the set error ratio n, the area where A / B is greater than nB or the area where B / A is greater than nB is classified as a pixel with a large error, that is, a pixel with an invalid depth value. It should be noted that in order to distinguish between invalid depth values ​​and valid depth values ​​in subsequent steps, after determining the pixel with an invalid depth value, the depth value corresponding to the pixel can be set to a non-number. Among them, the above error ratio n can be taken as 1.1, or it can be set differently according to requirements. Through the analysis of the local window area of ​​the depth map, it can be seen that under normal circumstances, the depth value that differs too much from the median value often has a large error, so the invalid depth value with a large error can be distinguished based on the above median filtering method.

[0110] In step S1120 , a connected region is determined in the depth image determined by the dense reconstruction result by using a semantic segmentation algorithm.

[0111] In an exemplary embodiment, after the semantic labels are obtained by semantic segmentation, if there are many invalid depth values ​​in the connected area corresponding to the same semantic label, it can be determined that the depth value error of the connected area is large, so it is necessary to treat all the depth values ​​corresponding to the connected area as invalid depth values ​​for subsequent processing. At this time, the connected area corresponding to each segmentation category can be calculated based on the semantic segmentation map obtained by the semantic segmentation algorithm, and then further processed for each. Specifically, a static foreground mask can be extracted from the semantic segmentation map first, for example, it can include trees, organisms, fountains, sculptures, etc.; then for each category, such as the tree category, several connected areas are calculated; wherein the connected area means that each pixel in the area is of the same category, and these pixels can be connected to each other up, down, left, and right.

[0112] In step S1130, the invalid proportion corresponding to the connected area is calculated based on the invalid depth value and the valid depth value;

[0113] In step S1140 , when the invalid proportion is greater than the first proportion, the depth value corresponding to the connected region is determined as an invalid depth value, and a dense reconstruction result after cleaning is obtained.

[0114] In an exemplary embodiment, after the connected regions, invalid depth values, and valid depth values ​​are determined, the proportion of invalid depth values ​​in each connected region is calculated to determine whether the invalid proportion in each connected region is greater than the first proportion. If it is greater than the first proportion, it means that there are too many invalid depth values ​​in the connected region, and the depth value in the connected region can be determined as an invalid depth value to obtain a dense reconstruction result after cleaning. Similarly, in order to accurately distinguish invalid depth values ​​in subsequent processing, all depth values ​​in the connected region can be set to non-numbers.

[0115] In an exemplary embodiment, in addition to the above problems, unreliable depth values ​​may exist at the edges of areas with low confidence and high confidence, so the edge parts also need to be cleaned. Fig.12 As shown, the following steps S1210 to S1240 may be included:

[0116] In step S1210, semantic segmentation is performed on the reconstructed image determined by the dense reconstruction result by using a semantic segmentation algorithm to obtain a semantic segmentation image.

[0117] In an exemplary embodiment, it is often difficult to obtain correct depth values ​​in infinitely distant areas (such as the sky), narrow and long structure areas (such as street lamps), man-made pattern areas (such as posters), dynamic object areas (such as flags), reflective areas (such as glass), and dynamic reflective areas (such as water surfaces), and there are also other areas that are easy to reconstruct. Based on this, the reconstructed image determined by the dense reconstruction result can be semantically segmented by a semantic segmentation algorithm to obtain a semantic segmentation image, and then the confidence of each area in the reconstructed image can be determined based on the semantic segmentation image.

[0118] In step S1220, the semantic labels contained in the semantic segmentation image are classified based on the confidence classification method, so as to divide the reconstructed image determined by the dense reconstruction result into a first confidence area and a second confidence area through the semantic segmentation image.

[0119] The confidence level of the first confidence area is lower than the confidence level of the second confidence area.

[0120] In an exemplary embodiment, a semantic segmentation image is obtained, and the semantic labels contained in the semantic segmentation image can be classified based on a preset confidence classification method to obtain a class of semantic labels with low confidence and a class of semantic labels with high confidence, and then the corresponding area of ​​the class of semantic labels with low confidence in the reconstructed image is determined as the first confidence area, and the corresponding area of ​​the class of semantic labels with high confidence in the reconstructed image is determined as the second confidence area. It should be noted that in order for subsequent steps to directly determine which areas are the first confidence areas and which areas are the second confidence areas, different labels can be marked for the pixels of the two areas respectively. For example, the pixels contained in the first confidence area can be marked as 1, and the pixels contained in the second confidence area can be marked as 2.

[0121] For example, for Figure 8 The reconstructed image shown in a can be semantically segmented to obtain Figure 8 The semantic segmentation image shown in b is then classified into semantic segmentation labels, with the sky and street lights classified into one category and the rest classified into another category. Correspondingly, we can get Fig.13 The first confidence region shown in a and Fig.13 The second confidence region shown in b.

[0122] In step S1230, a morphological dilation process is performed on the first confidence region to obtain a third confidence region after dilation, and the depth value corresponding to the third confidence region is determined as an invalid depth value.

[0123] In an exemplary embodiment, after obtaining a first confidence region with a lower confidence, a morphological dilation process may be performed on the first confidence region to include the edge of the original first confidence region in the dilated third confidence region, and then the depth value corresponding to the third confidence region is determined as an invalid depth value. It should be noted that, as above, in order to facilitate the subsequent steps to distinguish between invalid depth values ​​and valid depth values, after determining a pixel with an invalid depth value, the depth value corresponding to the pixel may be set to a non-number.

[0124] For example, for Fig.13 The first confidence area (light-colored area) shown in a can be morphologically expanded with a radius of 4 pixels to obtain the following Fig.14 a shows the third confidence area (light-colored area), and sets all depth values ​​of the third confidence area to non-numbers.

[0125] In step S1240, morphological shrinkage processing is performed on the second confidence region to obtain a shrunk fourth confidence region, and the depth value corresponding to the region outside the fourth confidence region is determined as an invalid depth value to obtain a cleaned dense reconstruction result.

[0126] In an exemplary embodiment, after obtaining a second confidence region with a higher confidence, a morphological closing operation and a morphological erosion process may be performed on the second confidence region to exclude the edge contained in the original second confidence region from the fourth confidence region after contraction, and then the depth value corresponding to the region outside the third and fourth confidence regions is determined as an invalid depth value to obtain a dense reconstruction result after cleaning. It should be noted that, as above, in order to facilitate the subsequent steps to distinguish between invalid depth values ​​and valid depth values, after determining the pixel with an invalid depth value, the depth value corresponding to the pixel may be set to a non-number.

[0127] For example, for Fig.13 For the second confidence region (light-colored region) shown in b, a morphological closing operation with a radius of 2 pixels can be first performed on the second confidence region, and then a morphological erosion with a radius of 4 pixels can be performed.

[0128] It should be noted that in order to remove areas smaller than a certain size and improve the trainability of the effective depth, an area opening operation can be performed after the morphological closing operation. For example, after the above morphological closing operation, morphological erosion and area opening operation, the following Fig.13 The second confidence region shown in b is transformed into Fig.14 b The fourth confidence region (light-colored area) shown.

[0129] After confidence cleaning, the purpose of cleaning the edges of areas with low and high confidence depth values ​​can be achieved. The effect is shown in the table below. Fig.15 a (depth image before edge cleaning) and Fig.15 b (Depth image after edge cleaning).

[0130] In an exemplary embodiment, after the above-mentioned cleaning, a dense reconstruction result after cleaning can be obtained. However, the area ratio of the effective area in the depth image (the number of pixels in the effective area divided by the number of pixels in the depth map) is too small, and the data that can be used for training in the depth image is also relatively limited. At this time, relative depth data can be generated as training data based on the depth data. Specifically, when the area ratio corresponding to the effective depth value in the reconstructed image is less than the second ratio, relative depth data can be generated based on the depth data corresponding to the reconstructed image, and then the reconstructed image and the relative depth data corresponding to the reconstructed image are used as the first training data.

[0131] In an exemplary embodiment, when generating relative depth data, if there are large areas in the reconstructed image that are not reconstructed successfully (the depth values ​​corresponding to the pixels are invalid depth values, that is, the depth values ​​corresponding to the pixels are non-numeric), and the semantic labels in the semantic segmentation map are areas with specific labels (which may include labels such as people or animals that are usually foreground), it can be assumed that these areas are foreground areas, and relative depth data can be generated based on the foreground areas and the original depth data.

[0132] For example, it is assumed that the depth of the foreground area is smaller than that of static objects whose size exceeds 75% of the effective depth. That is, when the semantic segmentation map shows that there is a large area of ​​people in the picture, it can be assumed that the depth value of this person area is smaller than the depth value of the relatively distant building. Based on this assumption, the depth data corresponding to the foreground area determined based on the specific label and the reconstructed image can be used to generate relative depth data containing three values, namely far, near and invalid values. Then, during training, training is performed based on the relative depth data through sequential loss. Since dynamic objects rarely remain stationary, SfM and MVS have difficulty reconstructing the depth of these objects. The use of relative depth data allows the network to use this distance constraint (i.e., sequential loss) to learn approximate distance information (i.e., large people are closer than distant buildings) in the absence of static person data.

[0133] For example, refer to Fig.16 As shown, semantic segmentation can be used to Fig.16 The reconstructed image shown in a is semantically segmented to obtain three labels: person, building and sky (such as Fig.16 b), the depth image corresponding to the reconstructed image is as follows Fig.16As shown in c, the task label corresponding area is then determined as the foreground area, that is, the area with a value of near, and then based on the assumption that the depth of the foreground area is smaller than the static object whose size exceeds 75% of the effective depth and Fig.16 The depth of the person and the building in c determine that the building is a far area. At the same time, based on the above cleaning process, the depth value of the sky area can be determined as an invalid depth value, thereby obtaining relative depth data with values ​​of far, near and invalid values. The corresponding relative depth image is as follows Fig.16 As shown in d.

[0134] In particular, there is a special type of video on the Internet, in which the characters in the video will pose in various interesting poses and remain motionless. The photographer needs to hold the camera and shoot the scene in one shot. This type of video is called a dummy challenge video and is a popular online video. For this type of video, since any character remains still during the video shooting process, the accurate camera pose and dense depth can be restored through the SfM and MVS algorithms. The depth interference in this video can be eliminated in the following way.

[0135] Specifically, the reconstructed image can be first semantically segmented based on the semantic segmentation algorithm to segment different character areas and mark them; then, the area in the depth image corresponding to the character area corresponding to each character is traversed, and the valid depth value in the area is obtained to obtain a histogram of the depth value distribution in the character area; then, the depth threshold corresponding to the character area is calculated according to a preset percentage, and all depth values ​​greater than the depth threshold are filtered out, thereby eliminating depth contamination.

[0136] For example, refer to Fig.17 As shown, Fig.17 The reconstructed image shown in a is semantically segmented to obtain two character regions (such as Fig.17 b), and then refer to the character area for the depth image (as shown in Fig.17 c) traverses each character area, records the depth values ​​of valid points in the area, and obtains a depth value distribution histogram in each character area. At this time, assuming that the preset percentage is 75%, and assuming that there are 100 valid depth values ​​in the character area, the depth value of the 75th valid point in the 100 valid points when the depth values ​​are arranged from small to large can be determined based on the depth value distribution histogram, and then the depth value is used as the depth threshold, and the depth values ​​greater than the depth threshold (such as Fig.17 d) is filtered out to obtain a depth image without deep doping (as shown in the gray part in d). Fig.17 e). For example, among 100 points, 50 have a depth value of 1, 30 have a depth value of 2, and 20 have a depth value of 10. It can be determined that the depth value of the 75th point is 2, and the depth value greater than 2, that is, the depth value of 10, is filtered out.

[0137] The following is an example of a 2D video that is not a dummy challenge video and needs to be cleaned in the above three ways. Fig.18 As shown, the technical solution of the embodiment of the present disclosure is described in detail:

[0138] Step S1801, performing ratio judgment on the photometric consistency depth image and the geometric consistency depth image determined based on the dense reconstruction result, and obtaining a depth image that eliminates depth doping to a certain extent;

[0139] Step S1803, determining invalid depth values ​​and valid depth values ​​in the depth data in the dense reconstruction result based on a median filtering algorithm, and configuring the invalid depth values ​​to be non-numbers;

[0140] Step S1805, performing semantic segmentation on the depth image of the reconstructed image to obtain regions corresponding to different semantic labels;

[0141] Step S1807, calculating at least one connected area for the area corresponding to each tag;

[0142] Step S1809, for each connected area, the proportion of invalid depth values ​​in the connected area is calculated based on the invalid depth values ​​and the valid depth values. When the proportion of invalid depth values ​​is greater than the first proportion, the depth value corresponding to the connected area in the depth image after removing the depth doping is configured as a non-number (that is, it is determined to be an invalid depth value), and the depth image after debris cleaning is obtained; wherein, for the part of the connected area that has been configured as a non-number, it is not necessary to repeatedly set it;

[0143] Step S1811, based on the semantic labels obtained by the semantic segmentation algorithm, the reconstructed image determined by the dense reconstruction result is divided into a first confidence region (low confidence region) and a second confidence region (high confidence region);

[0144] Step S1813, performing morphological image processing on the first confidence region and the second confidence region respectively based on morphological image processing to obtain a third confidence region and a fourth confidence region;

[0145] Step S1815, in the depth image after debris cleaning, the depth value in the third confidence area is configured as a non-number (that is, it is determined to be an invalid depth value), so as to obtain a depth image without a low confidence area; wherein, for the part in the third confidence area that has been configured as a non-number, it is not necessary to repeatedly set it;

[0146] Step S1817: In the depth image without the low confidence region, the depth values ​​that are not in the fourth confidence region are configured as non-numbers (ie, they are determined to be invalid depth values), and a cleaned dense reconstruction result is obtained.

[0147] In an exemplary embodiment, in addition to 2D videos, 3D videos, i.e., three-dimensional videos shot with a stereo camera, may also be acquired when acquiring a video set. In this case, second training data may be generated based on the 3D videos, and the second training data may be used as pre-training data to train a rough initial model, and then the initial model may be further trained based on the first training data to obtain a more refined model.

[0148] Specifically, for each 3D video in the video set, binocular camera calibration and alignment are performed on each video frame in the 3D video to obtain a binocular image; then a disparity map is calculated based on the binocular image, and the binocular image and depth data calculated based on the disparity map are used as second training data.

[0149] For example, a 3D video shot with a stereo camera can usually be decomposed into two side-by-side left and right view images shot at the same time. For each frame of video, the homography matrix of the binocular image can be solved first, and then the homography matrix can be used to deform the two view images to align the binocular images, and then the disparity map is solved based on the aligned binocular images, and the binocular images and the depth data calculated based on the disparity map are used as the second training data.

[0150] Among them, when solving the homography matrix, you can use algorithms such as uncalibrated stereo image rectification. When solving the disparity map, you can use an optical flow network (such as Raft, FlowNet2) to solve the disparity map, and then calculate the depth data corresponding to the disparity map based on the following formula: depth = (focal length * baseline) / disparity. Fig.19 As shown, Fig.19 a and Fig.19 b are the images after the left and right views are aligned, Fig.19 c is the disparity map calculated using the optical flow network, where the contour structure is clearly visible and the distance is distinct.

[0151] In addition, due to the difference in viewing angles, there are some areas in the scene that can only be captured by one camera, so the depth values ​​of these blocked areas cannot be calculated through stereo matching. For these depth values, they can be eliminated through left-right consistency analysis. Specifically, Fig.19 d is the left-right consistency view, in which the darker the color, the greater the brightness difference. Fig.19 e and Fig.19 f represents Fig.19When the area with large brightness difference in d covers the original image and disparity map, it can be seen that most of the occluded areas are successfully identified. Based on this, the second training data can use the normalized multiscale gradient loss and disparity map to supervise the inverse depth, and then train the initial model with invariant scale.

[0152] In an exemplary embodiment, after obtaining the first training data or the second training data, the first training data and the second training data may be downsampled to balance data of various scene types contained in the training data.

[0153] After obtaining the first training data or the second training data, in order to improve the generalization ability of the model trained based on the first training data or the first training data and the second training data, it is necessary to increase the scene types contained in the training data as much as possible. Therefore, when obtaining the video set, it is necessary to obtain as many videos as possible. However, after obtaining the first training data or the second training data, if the data contained in each scene type in the first training data or the second training data is relatively different, that is, the amount of training data for a certain scene type is too much, it may cause the trained model to shift to the scene type, that is, the model will perform well for scene types with more data, but perform generally for scene types with less data. In view of this situation, if differentiated data sampling is performed for different scene types when training the model (for example, a low sampling probability is used for a single scene, and a high sampling probability is used for a complex scene), this adverse effect can be dealt with to a certain extent. However, this operation requires the training algorithm to use different sampling mechanisms according to different data, and the debugging cost is high and it is difficult to apply to newly added data.

[0154] Based on the above problems, in an exemplary embodiment, in order to minimize the impact of uneven distribution of training data, the training data can be downsampled according to the distribution of features of different data points in the embedding space (data space after dimensionality reduction). Specifically, when the first training data or the second training data is obtained, the first training data or the second training data can be first classified into scene categories; then, for each scene category, the reconstructed image or binocular image is subjected to global feature extraction through the trained feature extraction model, that is, a feature vector is used to describe the entire reconstructed image or binocular image; all feature vectors of each scene category are reduced in dimension to obtain feature vectors of a specific dimension; and then the images contained in the data set of each scene category are downsampled using a grid.

[0155] The following takes the first training data and the specific dimension as 2 as an example, referring to Fig. 20 and Fig.21 As shown, the above downsampling process is described in detail:

[0156] First, for each scene category, a classification network is used to extract global features of the reconstructed image contained in the scene category, that is, a feature vector is used to describe the entire reconstructed image.

[0157] Among them, the classification network refers to a classification network generated by long-term training using a large-scale data set (for example, ImageNet-21K contains 14 million images, and JFT-300M contains 300 million images), which has more powerful feature extraction capabilities than manual features. Common models include LeNet, AlexNet, VGG, Inception, and ResNet. In this embodiment, ResNet-50 trained using ImageNet is used as the feature extraction network, an image is input, and a 1024-dimensional feature vector is output.

[0158] Secondly, the output 1024-dimensional feature vector is reduced in dimensionality using t-distributed Stochastic Neighbor Embedding (TSNE).

[0159] TSNE is a data dimensionality reduction algorithm that converts the similarity between data points into joint probabilities and attempts to minimize the KL divergence (Kullback-Leibler divergence) between the joint probabilities in low-dimensional and high-dimensional spaces. In this embodiment, the 1024-dimensional feature vector is reduced to a 2-dimensional feature vector. Fig. 20 a and 20b respectively show the visualization diagrams of the datasets of two scene categories after dimensionality reduction. Fig. 20 a is a simple scene, containing 19287 reconstructed images; Fig. 20 b is a complex scene, containing 2690 reconstructed images;

[0160] Afterwards, the images of each scene dataset are downsampled using a grid.

[0161] Specifically, a suitable grid size is selected according to the standard deviation of the scene, and the view corresponding to the feature vector after dimensionality reduction is divided into regions according to the grid size; after obtaining the divided regions, random sampling is performed in each area divided by the grid until the image quality meets the requirements.

[0162] The scene standard deviation can represent the richness of the scene type, and based on the richness, different downsampling strengths are selected for scene types of different richness. Therefore, for each scene category, the 1024-dimensional feature vector obtained previously can be used to calculate the standard deviation for each dimension, and then the average of the corresponding standard deviations of all dimensions is calculated, and the calculated average is used as the standard deviation of the scene.

[0163] When selecting a suitable grid size according to the scene standard deviation, the corresponding relationship between the scene standard deviation and the grid size can be set in advance, and the selection can be made according to the corresponding relationship. For example, the corresponding relationship may include: if the standard deviation is less than 0.4, use a 3x3 grid; if it is greater than 0.4 and less than 0.45, use a 4x4 grid; if the standard deviation is greater than 0.45 and less than 0.55, use a 16*16 grid; if the standard deviation is greater than 0.55 and less than 0.65, use a 20*20 grid; if the standard deviation is greater than 0.65, use a 24*24 grid.

[0164] Based on the above method Fig. 20 a and Fig. 20 The data set of b is downsampled and the following is obtained: Fig.21 a and Fig.21 b shows a visualization diagram of the downsampling of the datasets of the two scene categories. The number of images of the simple scene is Fig. 20 The 19287 images in a become Fig.21 15 images in a; the number of images of complex scenes is Fig. 20 The 2690 images in b become Fig.21 b; the ratio of the number of images corresponding to simple scenes and complex scenes is reduced from 7.16:1 to 0.55:1, which effectively balances the data volume of each scene type.

[0165] It should be noted that after downsampling the reconstructed image, the reconstructed image remaining after the downsampling and the depth data corresponding to the part of the reconstructed image may be used as the first training data.

[0166] The following reference Fig. 22 As shown, taking the video set including both 2D video and 3D video as an example, the technical solution of the embodiment of the present disclosure is described in detail:

[0167] Step S2201, crawling a video set in the streaming media based on a web crawler; the video set includes 2D videos and 3D videos;

[0168] Step S2203, performing frame screening on each 2D video to obtain a tracking frame sequence corresponding to each 2D video;

[0169] Step S2205, performing sparse 3D reconstruction on the tracking frame sequence to obtain a sparse reconstruction result;

[0170] Step S2207, screening the sparse reconstruction results based on the continuity of the motion trajectory to obtain screened sparse reconstruction results;

[0171] Step S2209, performing dense 3D reconstruction based on the screened sparse reconstruction results to obtain dense reconstruction results;

[0172] Step S2211, cleaning the dense reconstruction result based on a semantic segmentation algorithm to obtain a cleaned dense reconstruction result;

[0173] Step S2213, determining a reconstructed image and depth data corresponding to the reconstructed image based on the cleaned dense reconstruction result to obtain first training data;

[0174] Step S2215, downsampling the reconstructed image in the first training data to obtain a downsampled reconstructed image, and determining the downsampled reconstructed image and the depth data corresponding to the downsampled reconstructed image as the downsampled first training data;

[0175] Step S2217, performing binocular camera calibration and alignment on each video frame in each 3D video to obtain a binocular image;

[0176] Step S2219, calculating a disparity map based on the binocular image, and calculating depth data corresponding to the binocular image based on the disparity map;

[0177] Step S2221, determining the second training data based on the binocular image and the depth data corresponding to the binocular image;

[0178] Step S2223, downsampling the binocular image in the second training data to obtain a downsampled binocular image, and determining the downsampled binocular image and the depth data corresponding to the downsampled binocular image as the downsampled second training data.

[0179] In summary, in this exemplary embodiment, an unlimited number of videos of different scene types can be obtained through a web crawler; based on the preset tracking conditions, the video frames that are not suitable for three-dimensional reconstruction can be efficiently filtered out by filtering the frames of the video that are not suitable for three-dimensional reconstruction, and the tracking sequence frames suitable for sparse three-dimensional reconstruction can be obtained; then, in order to reduce the time consumption of dense reconstruction, after sparse reconstruction, the sparse reconstruction results can be filtered by the continuity of the motion trajectory, and the tracking frame sequence that is not suitable for dense reconstruction can be eliminated, thereby generating the first training data; in addition, using 3D videos on the Internet for binocular stereo matching can obtain the second training data with rich scenes for pre-training at a low cost. Afterwards, in order to eliminate the model offset effect caused by the large difference in the amount of data of different scene types, the first training data and the second training data are adaptively downsampled respectively by downsampling, so that the amount of data of various scene types is more balanced. In particular, when the 3D dummy challenge data is collected, training data with rich scene types containing characters can be generated by segmenting the characters.

[0180] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0181] For further reference, Fig.23 As shown, in the embodiment of this example, a training data generating device 2300 is also provided, including a video acquisition module 2310, a data screening module 2320 and a data generating module 2330. Among them:

[0182] The video acquisition module 2310 may be used to acquire a video set; the video set includes at least one 2D video.

[0183] The data screening module 2320 may be used to screen frames of each 2D video based on a preset tracking condition to obtain a tracking frame sequence corresponding to each 2D video.

[0184] The data generation module 2330 can be used to perform three-dimensional reconstruction based on each tracking frame sequence, obtain a reconstructed image and depth data corresponding to the reconstructed image, and use the reconstructed image and the depth data corresponding to the reconstructed image as the first training data.

[0185] In an exemplary embodiment, the data filtering module 2320 can be used to perform frame filtering processing on each 2D video separately, and the frame filtering processing includes: extracting the i-th video frame in the 2D video based on a preset step size; where i is a positive integer; when the i-th video frame meets the preset tracking condition, adding the i-th video frame to the video subsequence, and continuing to extract the i+1-th video frame based on the preset step size; when the i-th video frame does not meet the preset tracking condition, determining the video subsequence generated by the video frame before the i-th video frame as the tracking frame sequence corresponding to the 2D video.

[0186] In an exemplary embodiment, the preset tracking conditions include a combination of one or more of the following conditions: the number of feature points in the video frame is greater than a number threshold, the average module length of the optical flow matching vector corresponding to the video frame is less than the maximum value of the module length range, the number of inliers corresponding to the video frame is greater than an inlier threshold, and the inter-frame rotation angle corresponding to the video frame is less than an angle threshold.

[0187] In an exemplary embodiment, the data screening module 2320 may be used to determine that the tracking frame sequence corresponding to the 2D video frame is empty when the number of frames of the video subsequence is less than a minimum threshold; and to determine that the video subsequence is a tracking frame sequence corresponding to the 2D video frame when the number of frames of the video subsequence is greater than or equal to the minimum threshold.

[0188] In an exemplary embodiment, the data generation module 2330 can be used to perform sparse three-dimensional reconstruction based on each tracking frame sequence to obtain a sparse reconstruction result; perform dense three-dimensional reconstruction based on the sparse reconstruction result to obtain a dense reconstruction result; and determine the reconstructed image and the depth data corresponding to the reconstructed image based on the dense reconstruction result.

[0189] In an exemplary embodiment, the data generation module 2330 can be used to obtain the coordinates and sparse depth values ​​corresponding to each sparse point in the sparse reconstruction result; perform dense three-dimensional reconstruction with the sparse depth value as the initial depth value corresponding to the coordinate to obtain a dense reconstruction result.

[0190] In an exemplary embodiment, the data screening module 2320 may be used to screen the sparse reconstruction results based on the continuity of the motion trajectory to obtain the screened sparse reconstruction results, so as to perform dense three-dimensional reconstruction based on the screened sparse reconstruction results.

[0191] In an exemplary embodiment, the data screening module 2320 can be used to clean the dense reconstruction result based on a semantic segmentation algorithm to obtain the cleaned dense reconstruction result, so as to determine the reconstructed image and the depth data corresponding to the reconstructed image based on the cleaned dense reconstruction result.

[0192] In an exemplary embodiment, the data screening module 2320 can be used to determine a first region in a depth image determined by a dense reconstruction result through a semantic segmentation algorithm; process the first region based on a preset algorithm to eliminate the depth doping of the second region to the first region, and obtain a cleaned dense reconstruction result; wherein the depth of the second region is greater than that of the first region.

[0193] In an exemplary embodiment, the data screening module 2320 can be used to determine invalid depth values ​​and valid depth values ​​in the depth data in the dense reconstruction result based on the median filtering algorithm; determine the connected area in the depth image determined by the dense reconstruction result through the semantic segmentation algorithm; calculate the invalid proportion corresponding to the connected area based on the invalid depth value and the valid depth value; when the invalid proportion is greater than the first proportion, determine the depth value corresponding to the connected area as an invalid depth value to obtain a cleaned dense reconstruction result.

[0194] In an exemplary embodiment, the data screening module 2320 can be used to perform semantic segmentation on the reconstructed image determined by the dense reconstruction result through a semantic segmentation algorithm to obtain a semantic segmentation image; classify the semantic labels contained in the semantic segmentation image based on a confidence classification method, so as to divide the reconstructed image determined by the dense reconstruction result into a first confidence area and a second confidence area through the semantic segmentation image; wherein the confidence of the first confidence area is lower than the confidence of the second confidence area; perform morphological dilation processing on the first confidence area to obtain a third confidence area after dilation, and determine the depth value corresponding to the third confidence area as an invalid depth value; perform morphological contraction processing on the second confidence area to obtain a fourth confidence area after contraction, and determine the depth value corresponding to the area outside the fourth confidence area as an invalid depth value, so as to obtain a cleaned dense reconstruction result.

[0195] In an exemplary embodiment, the data generation module 2330 can be used to generate relative depth data based on the depth data corresponding to the reconstructed image when the area ratio corresponding to the effective depth value in the reconstructed image is less than the second ratio; and use the reconstructed image and the relative depth data corresponding to the reconstructed image as the first training data.

[0196] In an exemplary embodiment, the data screening module 2320 may be used to perform data downsampling on the first training data to obtain downsampled first training data.

[0197] In an exemplary embodiment, when the video set also includes at least one 3D video, the data generation module 2330 can be used to perform binocular camera calibration and alignment for each video frame in each 3D video to obtain a binocular image; calculate a disparity map based on the binocular image, and use the binocular image and depth data calculated based on the disparity map as second training data.

[0198] In an exemplary embodiment, the data screening module 2320 may be used to perform data downsampling on the second training data to obtain downsampled second training data.

[0199] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0200] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0201] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification, for example, it can execute Figure 3 to Figure 4 , Figure 11 to Figure 12 , Fig.18 , Fig. 22 Any one or more steps in .

[0202] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0203] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0204] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0205] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0206] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for generating training data, characterized in that: include: Get video collection; The video set includes at least one 2D video; Perform frame screening on each of the 2D videos based on a preset tracking condition to obtain a tracking frame sequence corresponding to each of the 2D videos; Performing three-dimensional reconstruction based on each of the tracking frame sequences to obtain a reconstructed image and depth data corresponding to the reconstructed image, and using the reconstructed image and the depth data corresponding to the reconstructed image as first training data; The performing three-dimensional reconstruction based on each of the tracking frame sequences to obtain a reconstructed image and depth data corresponding to the reconstructed image includes: performing sparse three-dimensional reconstruction based on each of the tracking frame sequences to obtain a sparse reconstruction result; performing dense three-dimensional reconstruction based on the sparse reconstruction result to obtain a dense reconstruction result; cleaning the dense reconstruction result based on a semantic segmentation algorithm to obtain a cleaned dense reconstruction result; and determining a reconstructed image and depth data corresponding to the reconstructed image based on the cleaned dense reconstruction result; The method cleans the dense reconstruction result based on the semantic segmentation algorithm to obtain the cleaned dense reconstruction result, including: determining invalid depth values ​​and valid depth values ​​in the depth data in the dense reconstruction result based on the median filtering algorithm; determining the connected area in the depth image determined by the dense reconstruction result through the semantic segmentation algorithm; calculating the invalid proportion corresponding to the connected area based on the invalid depth value and the valid depth value; when the invalid proportion is greater than the first proportion, determining the depth value corresponding to the connected area as the invalid depth value, to obtain the cleaned dense reconstruction result.

2. The method according to claim 1, characterized in that The performing frame screening on each of the 2D videos based on the preset tracking condition to obtain a tracking frame sequence corresponding to each of the 2D videos includes: Performing frame filtering processing on each of the 2D videos, the frame filtering processing comprising: Extracting the i-th video frame from the 2D video based on a preset step size; wherein i is a positive integer; When the i-th video frame meets the preset tracking condition, adding the i-th video frame to the video subsequence, and continuing to extract the i+1-th video frame based on the preset step size; When the i-th video frame does not satisfy the preset tracking condition, a video subsequence generated by the video frame before the i-th video frame is determined as a tracking frame sequence corresponding to the 2D video.

3. The method according to claim 2, characterized in that The preset tracking conditions include one or more combinations of the following conditions: The number of feature points in the video frame is greater than the number threshold, the average modulus of the optical flow matching vector corresponding to the video frame is less than the maximum value of the modulus range, the number of inliers corresponding to the video frame is greater than the inlier threshold, and the inter-frame rotation angle corresponding to the video frame is less than the angle threshold.

4. The method according to claim 2, characterized in that: Before determining the video subsequence as a tracking frame sequence corresponding to the 2D video, the method further includes: When the number of frames of the video subsequence is less than a minimum threshold, determining that the tracking frame sequence corresponding to the 2D video frame is empty; When the number of frames of the video subsequence is greater than or equal to the minimum threshold, the video subsequence is determined to be a tracking frame sequence corresponding to the 2D video frame.

5. The method according to claim 1, characterized in that The performing dense three-dimensional reconstruction based on the sparse reconstruction result to obtain a dense reconstruction result includes: Obtain the coordinates and sparse depth values ​​corresponding to each sparse point in the sparse reconstruction result; A dense three-dimensional reconstruction is performed using the sparse depth value as an initial depth value corresponding to the coordinate to obtain a dense reconstruction result.

6. The method according to claim 1, characterized in that Before performing dense three-dimensional reconstruction based on the sparse reconstruction result to obtain a dense reconstruction result, the method further includes: The sparse reconstruction results are screened based on the continuity of the motion trajectory to obtain screened sparse reconstruction results, so as to perform dense three-dimensional reconstruction based on the screened sparse reconstruction results.

7. The method according to claim 1, characterized in that The step of cleaning the dense reconstruction result based on a semantic segmentation algorithm to obtain the cleaned dense reconstruction result includes: Determine a first region in the depth image determined by the dense reconstruction result by a semantic segmentation algorithm; The first region is processed based on a preset algorithm to eliminate the deep doping of the second region on the first region, thereby obtaining a dense reconstruction result after cleaning; wherein the depth of the second region is greater than that of the first region.

8. The method according to claim 1, characterized in that The step of cleaning the dense reconstruction result based on a semantic segmentation algorithm to obtain the cleaned dense reconstruction result includes: Performing semantic segmentation on the reconstructed image determined by the dense reconstruction result by using a semantic segmentation algorithm to obtain a semantic segmentation image; Classifying the semantic labels contained in the semantic segmentation image based on a confidence classification method, so as to divide the reconstructed image determined by the dense reconstruction result into a first confidence region and a second confidence region through the semantic segmentation image; wherein the confidence of the first confidence region is lower than the confidence of the second confidence region; Performing morphological dilation processing on the first confidence region to obtain a third confidence region after dilation, and determining a depth value corresponding to the third confidence region as an invalid depth value; The second confidence region is subjected to morphological shrinkage processing to obtain a shrunken fourth confidence region, and depth values ​​corresponding to regions outside the fourth confidence region are determined as invalid depth values ​​to obtain a cleaned dense reconstruction result.

9. The method according to claim 8, characterized in that After cleaning the dense reconstruction result based on the semantic segmentation algorithm to obtain the cleaned dense reconstruction result, the method further includes: When the area ratio corresponding to the effective depth value in the reconstructed image is less than the second ratio, generating relative depth data based on the depth data corresponding to the reconstructed image; The reconstructed image and relative depth data corresponding to the reconstructed image are used as the first training data.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: Downsampling is performed on the first training data to obtain downsampled first training data.

11. The method according to claim 1, characterized in that: The video set further includes at least one 3D video, and the method further includes: Performing binocular camera calibration and alignment for each video frame in each of the 3D videos to obtain a binocular image; A disparity map is calculated based on the binocular image, and the binocular image and depth data calculated based on the disparity map are used as second training data.

12. The method according to claim 11, characterized in that The method further comprises: Downsampling is performed on the second training data to obtain downsampled second training data.

13. A training data generating device, characterized in that: include: A video acquisition module, used to acquire a video collection; The video set includes at least one 2D video; A data screening module, used to screen frames of each of the 2D videos based on a preset tracking condition to obtain a tracking frame sequence corresponding to each of the 2D videos; a data generation module, configured to perform three-dimensional reconstruction based on each of the tracking frame sequences, obtain a reconstructed image and depth data corresponding to the reconstructed image, and use the reconstructed image and the depth data corresponding to the reconstructed image as first training data; The performing three-dimensional reconstruction based on each of the tracking frame sequences to obtain a reconstructed image and depth data corresponding to the reconstructed image includes: performing sparse three-dimensional reconstruction based on each of the tracking frame sequences to obtain a sparse reconstruction result; performing dense three-dimensional reconstruction based on the sparse reconstruction result to obtain a dense reconstruction result; cleaning the dense reconstruction result based on a semantic segmentation algorithm to obtain a cleaned dense reconstruction result; and determining a reconstructed image and depth data corresponding to the reconstructed image based on the cleaned dense reconstruction result; The method cleans the dense reconstruction result based on the semantic segmentation algorithm to obtain the cleaned dense reconstruction result, including: determining invalid depth values ​​and valid depth values ​​in the depth data in the dense reconstruction result based on the median filtering algorithm; determining the connected area in the depth image determined by the dense reconstruction result through the semantic segmentation algorithm; calculating the invalid proportion corresponding to the connected area based on the invalid depth value and the valid depth value; when the invalid proportion is greater than the first proportion, determining the depth value corresponding to the connected area as the invalid depth value, to obtain the cleaned dense reconstruction result.

14. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

15. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 12 by executing the executable instructions.

Images