A three-dimensional reconstruction method, device, equipment and storage medium for sound barrier engineering
By extracting multi-level features and supervised learning methods, the accuracy of the three-dimensional reconstruction of sound barriers is improved, and the problem of poor reconstruction of sound barriers in complex environments and multi-components in the existing technology is solved, achieving higher construction management accuracy and quality.
Patent Information
- Application Number
- CN202311511098.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-13
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-11-13
AI Technical Summary
The prior art has poor results in the three-dimensional reconstruction of complex environments and multi-component acoustic barriers, and cannot effectively improve the operational accuracy and construction quality of construction managers.
By acquiring the image of the sound barrier, extracting two-dimensional feature maps of multiple levels and generating depth feature maps, reverse projecting into three-dimensional point clouds, calculating three-dimensional voxels, and using a supervised three-dimensional convolutional neural network model for loss calculation and Optimize, fuse semantic information to improve reconstruction accuracy.
It significantly improves the accuracy of the three-dimensional voxel reconstruction of acoustic barriers, provides more directional construction guidance, reduces errors and rework during construction, and improves construction efficiency and accuracy.
Smart Images

Figure CN117576311B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer vision processing technology, and in particular to a three-dimensional reconstruction method, device, equipment and storage medium for sound barrier engineering. Background Art
[0002] In order to suppress the environmental pollution caused by the noise generated by rail trains and passing cars, sound barriers are usually set up on both sides of the railway tracks and expressways, which can effectively suppress the noise.
[0003] With the rapid development of cities, it is often necessary to rebuild some sound barrier structures. In the reconstruction project of the sound barrier, it is necessary to use the three-dimensional model of the sound barrier to enable construction management personnel to understand the detailed structure and connection methods of each part, so that construction management personnel can operate more directional, thereby further ensuring construction quality and safety.
[0004] Three-dimensional voxels are a form of expression of three-dimensional models. Currently, multi-view images are usually used to reconstruct the three-dimensional voxel model of the sound barrier using computer vision. However, this method mainly uses the single-scale features of the image for reconstruction. When the environment is more complex and the sound barrier has more components, the reconstruction effect is poor. Summary of the invention
[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.
[0006] The main purpose of the embodiments of the present invention is to propose a three-dimensional reconstruction method, device, equipment and storage medium for sound barrier projects, which can improve the accuracy of three-dimensional voxel reconstruction and make the operations of construction management personnel more directional, thereby further ensuring construction quality and safety.
[0007] A first aspect of an embodiment of the present invention provides a three-dimensional reconstruction method for a sound barrier project, the three-dimensional reconstruction method for a sound barrier project comprising:
[0008] Acquire an image of the sound barrier;
[0009] Extracting two-dimensional feature maps of multiple levels from the image respectively, generating a depth feature map corresponding to each level according to the two-dimensional feature maps of the multiple levels, reversely projecting the depth feature map corresponding to each level into a three-dimensional point cloud corresponding to each level, computing a first three-dimensional voxel corresponding to each level according to the three-dimensional point cloud corresponding to each level, and fusing the first three-dimensional voxels corresponding to each level to obtain a second three-dimensional voxel;
[0010] Extracting a three-dimensional feature corresponding to the second three-dimensional voxel, inputting the three-dimensional feature into a preset three-dimensional voxel reconstruction model, and obtaining a final three-dimensional voxel reconstructed by the three-dimensional voxel reconstruction model; the three-dimensional voxel reconstruction model is obtained by performing loss calculation on the three-dimensional voxel prediction value extracted from the sound barrier training image and the three-dimensional voxel true value of the sound barrier corresponding to the sound barrier training image, and performing back propagation optimization using the loss calculation;
[0011] A three-dimensional model of the sound barrier is constructed based on the final three-dimensional voxels.
[0012] In some embodiments of the present application, extracting two-dimensional feature maps of multiple levels from the image, and generating a depth feature map corresponding to each level according to the two-dimensional feature maps of the multiple levels, includes:
[0013] Performing convolution and downsampling on the image layer by layer, and selecting two-dimensional feature maps corresponding to multiple downsampling outputs;
[0014] Performing semantic segmentation on each of the two-dimensional feature maps to obtain a segmentation feature map corresponding to each of the two-dimensional feature maps;
[0015] An initial depth feature map is generated according to the image, and the initial depth feature map is optimized respectively through the segmentation feature map corresponding to each of the two-dimensional feature maps to obtain a depth feature map corresponding to each level.
[0016] In some embodiments of the present application, K-means is used to semantically segment the two-dimensional feature map.
[0017] In some embodiments of the present application, extracting the three-dimensional feature corresponding to the second three-dimensional voxel;
[0018] Build the first 3D convolutional neural network model;
[0019] Inputting the second three-dimensional voxel into the first three-dimensional convolutional neural network model to obtain a three-dimensional feature corresponding to the second three-dimensional voxel output by the first three-dimensional convolutional neural network model; wherein the training process of the first three-dimensional convolutional neural network model includes:
[0020] Construct an initial 3D convolutional neural network model;
[0021] Inputting a third three-dimensional voxel extracted from the sound barrier training image into the initial three-dimensional convolutional neural network model to obtain a first three-dimensional feature corresponding to the third three-dimensional voxel;
[0022] Inputting the three-dimensional voxel true value of the sound barrier corresponding to the sound barrier training image into the initial three-dimensional convolutional neural network model to obtain a second three-dimensional feature corresponding to the three-dimensional voxel true value;
[0023] The loss is calculated according to the first three-dimensional feature and the second three-dimensional feature, and the initial three-dimensional convolutional neural network model is optimized using the loss to obtain the optimized initial three-dimensional convolutional neural network model as the first three-dimensional convolutional neural network model.
[0024] In some embodiments of the present application, the three-dimensional voxel reconstruction model is obtained by training a three-dimensional convolutional neural network model.
[0025] In some embodiments of the present application, the image is collected by a drone.
[0026] In some embodiments of the present application, before computing the first three-dimensional voxel corresponding to each level according to the three-dimensional point cloud corresponding to each level, the three-dimensional reconstruction method of the sound barrier engineering further includes:
[0027] The three-dimensional point cloud is subjected to filtering processing.
[0028] A second aspect of an embodiment of the present invention provides a three-dimensional reconstruction device for a sound barrier project, the three-dimensional reconstruction device for a sound barrier project comprising:
[0029] An image acquisition unit, used to acquire an image of the sound barrier;
[0030] A first computing unit is used to extract two-dimensional feature maps of multiple levels from the image, generate a depth feature map corresponding to each level according to the two-dimensional feature maps of the multiple levels, reversely project the depth feature map corresponding to each level into a three-dimensional point cloud corresponding to each level, calculate a first three-dimensional voxel corresponding to each level according to the three-dimensional point cloud corresponding to each level, and fuse the first three-dimensional voxels corresponding to each level to obtain a second three-dimensional voxel;
[0031] A second calculation unit is used to extract the three-dimensional features corresponding to the second three-dimensional voxel, input the three-dimensional features into a preset three-dimensional voxel reconstruction model, and obtain the final three-dimensional voxel reconstructed by the three-dimensional voxel reconstruction model; the three-dimensional voxel reconstruction model is obtained by performing loss calculation on the three-dimensional voxel prediction values extracted from the sound barrier training image and the three-dimensional voxel true values of the sound barrier corresponding to the sound barrier training image, and performing back propagation optimization using the loss calculation;
[0032] A three-dimensional model construction unit is used to construct a three-dimensional model of the sound barrier based on the final three-dimensional voxels.
[0033] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present invention proposes an electronic device, comprising: at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the above-mentioned three-dimensional reconstruction method of a sound barrier project.
[0034] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present invention proposes a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned three-dimensional reconstruction method of a sound barrier project.
[0035] An embodiment of the present application provides a three-dimensional reconstruction method for a sound barrier project. The method of this embodiment reconstructs a second three-dimensional voxel of a multi-view image. In the first reconstruction, unlike the prior art, the two-dimensional features of different levels of the multi-view image are first extracted to enrich the multi-level information contained in the features, and then semantic segmentation is performed on each two-dimensional feature to obtain the segmentation feature. The initial depth feature map is then optimized by each of the two segmentation features to obtain a depth feature map corresponding to each level. The depth feature map corresponding to each level is then reversely projected into a three-dimensional point cloud corresponding to each level. The first three-dimensional voxel corresponding to each level is calculated based on the three-dimensional point cloud corresponding to each level, and the first three-dimensional voxel corresponding to each level is fused to obtain a second three-dimensional voxel. The fused second three-dimensional voxel can fully fuse the semantic information of different levels and improve the accuracy of reconstructing the three-dimensional voxel. Since the three-dimensional voxels constructed for the first time adopted an unsupervised reconstruction method, the accuracy of the three-dimensional voxel construction was improved. Therefore, in the second reconstruction, a supervised reconstruction method is used to perform a secondary reconstruction of the three-dimensional voxels. Therefore, the three-dimensional features corresponding to the second three-dimensional voxels are first extracted, and the three-dimensional features are input into a preset three-dimensional voxel reconstruction model to reconstruct the final three-dimensional voxels. Since the three-dimensional voxel reconstruction model is trained by a supervised reconstruction method, the final reconstructed three-dimensional voxels are more accurate than the second three-dimensional voxels constructed for the first time.
[0036] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0038] Figure 1 It is a flow chart of a three-dimensional reconstruction method of a sound barrier project provided by an embodiment of the present application;
[0039] Figure 2 yes Figure 1 Schematic diagram of the process of generating a depth feature map in step S120;
[0040] Figure 3 yes Figure 1 A schematic diagram of a process of extracting a three-dimensional feature corresponding to a second three-dimensional voxel in step S130;
[0041] Figure 4 is a schematic diagram of extracting multi-level features provided by an embodiment of the present application;
[0042] Figure 5 is a schematic diagram of generating a second three-dimensional voxel provided by an embodiment of the present application;
[0043] Figure 6 It is a schematic structural diagram of a three-dimensional reconstruction device for a sound barrier project provided by one embodiment of the present application;
[0044] Figure 7 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0046] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0048] In order to suppress the environmental pollution caused by the noise generated by rail trains and passing cars, sound barriers are usually set up on both sides of the railway tracks and expressways, which can effectively suppress the noise.
[0049] With the rapid development of cities, it is often necessary to rebuild some sound barrier structures. In the reconstruction project of the sound barrier, it is necessary to use the three-dimensional model of the sound barrier to enable construction management personnel to understand the detailed structure and connection methods of each part, so that construction management personnel can operate more directional, thereby further ensuring construction quality and safety.
[0050] Three-dimensional voxels are a form of expression of three-dimensional models. Currently, multi-view images are usually used to reconstruct the three-dimensional voxel model of the sound barrier using computer vision. However, this method mainly uses the single-scale features of the image for reconstruction. When the environment is more complex and the sound barrier has more components, the reconstruction effect is poor.
[0051] Reference Figures 1 to 5 In order to solve the above technical problems, a more accurate three-dimensional voxel model is reconstructed to provide important assistance for practical operators to obtain detailed structures and connection methods. An embodiment of the present application provides a three-dimensional reconstruction method for a sound barrier project, and the method includes:
[0052] Step S110: Acquire an image of the sound barrier.
[0053] Step S120, extracting multiple levels of two-dimensional feature maps from the image respectively, and generating a depth feature map corresponding to each level based on the multiple levels of two-dimensional feature maps, reversely projecting the depth feature map corresponding to each level into a three-dimensional point cloud corresponding to each level, computing the first three-dimensional voxel corresponding to each level based on the three-dimensional point cloud corresponding to each level, and fusing the first three-dimensional voxel corresponding to each level to obtain a second three-dimensional voxel.
[0054] Step S130, extract the three-dimensional features corresponding to the second three-dimensional voxel, input the three-dimensional features into a preset three-dimensional voxel reconstruction model, and obtain the final three-dimensional voxel reconstructed by the three-dimensional voxel reconstruction model; the three-dimensional voxel reconstruction model is obtained by performing loss calculation on the three-dimensional voxel prediction values extracted from the sound barrier training image and the three-dimensional voxel true values of the sound barrier corresponding to the sound barrier training image, and performing back propagation optimization using the loss calculation.
[0055] Step S140: construct a three-dimensional model of the sound barrier based on the final three-dimensional voxels.
[0056] In step S110, multi-view images of the sound barrier may be obtained by using drone aerial photography, or by using a robot to take pictures, or by capturing images from surveillance videos.
[0057] In step S120, the reconstruction of the first three-dimensional voxel of the sound barrier mainly adopts the extraction of multi-scale features to fully extract the features belonging to different levels, so that the reconstructed first three-dimensional voxel is more accurate than the three-dimensional voxel constructed by single-scale features. The specific process includes:
[0058] First, multi-scale feature extraction is performed on the image to extract two-dimensional feature maps of multiple levels with scales from low to high. In some embodiments, multiple levels of two-dimensional feature maps are extracted from the image respectively, and a depth feature map corresponding to each level is generated according to the multiple levels of two-dimensional feature maps, including the following steps S210, S220 and S230:
[0059] Step S210: perform convolution and down-sampling on the image layer by layer, and select two-dimensional feature maps corresponding to multiple down-sampling outputs.
[0060] Step S220: perform semantic segmentation on each two-dimensional feature map to obtain a segmentation feature map corresponding to each two-dimensional feature map.
[0061] Step S230: Generate an initial depth feature map according to the image, and optimize the initial depth feature map through the segmentation feature map corresponding to each two-dimensional feature map to obtain a depth feature map corresponding to each level.
[0062] In step S210, the image is convolved and downsampled multiple times. The features extracted previously are relatively low-level feature maps. As the number of convolution and downsampling increases, higher-level feature maps will gradually be obtained. Then, multiple two-dimensional feature maps are extracted from lower levels to higher levels. It should be noted that the order of extraction is not limited here. Each downsampled two-dimensional feature map can be selected from the low level to the high level. For example, after five convolutions and downsamplings, the two-dimensional feature maps of the subsequent four downsamplings can be selected. It can also be selected randomly. For example, after five convolutions and downsamplings, the feature maps of the last two and the second downsampling can be selected. Figure 4 As shown, from the bottom layer to the top layer, the higher the level.
[0063] In step S220, semantic segmentation is performed on each two-dimensional feature map to obtain a segmentation feature map corresponding to each two-dimensional feature map. For example, four two-dimensional feature maps are selected in the previous S210, and four segmentation feature maps are obtained for each two-dimensional feature map. In some embodiments, K-means (k-means clustering algorithm) is used to semantically segment the two-dimensional feature map.
[0064] Reference Figure 5 In step S230, an initial depth feature map is generated according to the image, and then the segmentation feature map corresponding to each two-dimensional feature map is used to guide the initial depth feature map to generate a corresponding depth feature map.
[0065] In step S120, the depth feature map corresponding to each level is back-projected into a three-dimensional point cloud corresponding to each level.
[0066] In this embodiment, the 3D point cloud can be converted into 3D voxels according to the 3D convolutional neural network model. Since the 3D point clouds corresponding to multiple levels are extracted above, the 3D voxels corresponding to each point cloud data can be obtained here, and finally the 3D voxels of multiple point cloud data are fused to obtain a total 3D voxel (here referred to as the second 3D voxel). Here, the 3D voxel is a form of expression of the 3D model, and the voxel is a cube in the 3D space, which is equivalent to a pixel in the 3D space.
[0067] Since the above steps S110 and S120 use multi-perspective images and extract multi-level features, multiple levels corresponding to multiple three-dimensional point clouds are obtained, and then the three-dimensional voxels corresponding to each three-dimensional point cloud are obtained. Combined with multiple three-dimensional voxels, a total second three-dimensional voxel is obtained, and the preliminary construction of the three-dimensional voxel is completed. At this time, the constructed three-dimensional voxels are three-dimensional voxels obtained using unsupervised technology, and there is still room for improvement in accuracy. Therefore, in step S130, supervised technology is used to perform secondary reconstruction on the preliminary constructed three-dimensional voxels, that is, the three-dimensional features corresponding to the second three-dimensional voxels are first extracted, and the three-dimensional features are input into a preset three-dimensional voxel reconstruction model to obtain the final three-dimensional voxels reconstructed by the three-dimensional voxel reconstruction model, wherein the three-dimensional voxel reconstruction model is obtained by using the three-dimensional voxel prediction value extracted from the sound barrier training image and the three-dimensional voxel true value of the sound barrier corresponding to the sound barrier training image to perform loss calculation, and the loss calculation is used for back propagation optimization.
[0068] In some embodiments of the present application, S130 extracting the three-dimensional feature corresponding to the second three-dimensional voxel includes the following:
[0069] Step S310: construct a first three-dimensional convolutional neural network model.
[0070] Step S320: input the second three-dimensional voxel into the first three-dimensional convolutional neural network model to obtain a three-dimensional feature corresponding to the second three-dimensional voxel output by the first three-dimensional convolutional neural network model; wherein the training process of the first three-dimensional convolutional neural network model includes:
[0071] Construct an initial three-dimensional convolutional neural network model. Input the third three-dimensional voxel extracted from the sound barrier training image into the initial three-dimensional convolutional neural network model to obtain the first three-dimensional feature corresponding to the third three-dimensional voxel. Input the three-dimensional voxel true value of the sound barrier corresponding to the sound barrier training image into the initial three-dimensional convolutional neural network model to obtain the second three-dimensional feature corresponding to the three-dimensional voxel true value. Calculate the loss based on the first three-dimensional feature and the second three-dimensional feature, and use the loss to optimize the initial three-dimensional convolutional neural network model to obtain the optimized initial three-dimensional convolutional neural network model.
[0072] By training the first three-dimensional convolutional neural network model in step S320, the accuracy of the first three-dimensional convolutional neural network model in extracting three-dimensional features in the second three-dimensional voxels can be improved, thereby increasing the accuracy of subsequent three-dimensional voxel model reconstruction.
[0073] In step S140, a three-dimensional model of the sound barrier is constructed based on the final three-dimensional voxels, and the three-dimensional model provides important assistance to practical operators in obtaining detailed structures and connection methods.
[0074] The method of this embodiment performs a secondary three-dimensional voxel reconstruction on the multi-view image. In the first reconstruction, unlike the prior art, the two-dimensional features of different levels of the multi-view image are first extracted to enrich the multi-level information contained in the features, and then semantic segmentation is performed on each two-dimensional feature to obtain the segmentation feature. The initial depth feature map is then optimized by each of the two segmentation features to obtain the depth feature map corresponding to each level. The depth feature map corresponding to each level is then reversely projected into a three-dimensional point cloud corresponding to each level. The first three-dimensional voxel corresponding to each level is calculated based on the three-dimensional point cloud corresponding to each level, and the first three-dimensional voxel corresponding to each level is fused to obtain the second three-dimensional voxel. The fused second three-dimensional voxel can fully fuse the semantic information of different levels and improve the accuracy of reconstructing the three-dimensional voxel. Since the three-dimensional voxels constructed for the first time adopted an unsupervised reconstruction method, the accuracy of the three-dimensional voxel construction was improved. Therefore, in the second reconstruction, a supervised reconstruction method is used to perform a secondary reconstruction of the three-dimensional voxels. Therefore, the three-dimensional features corresponding to the second three-dimensional voxels are first extracted, and the three-dimensional features are input into a preset three-dimensional voxel reconstruction model to reconstruct the final three-dimensional voxels. Since the three-dimensional voxel reconstruction model is trained by a supervised reconstruction method, the final reconstructed three-dimensional voxels are more accurate than the second three-dimensional voxels constructed for the first time.
[0075] In some embodiments, during the construction of a special project of sound barrier for the rapid transformation of a certain road in a certain city, the above method is used to construct a three-dimensional voxel model of the sound barrier, which can significantly reduce the problems of errors, omissions, collisions, etc. in the reconstruction according to the drawings, and avoid the current situation of frequent rework during the construction stage. At the same time, through the expression of the three-dimensional information model, the intention of the project reconstruction is intuitively reflected, thereby guiding the construction, and the efficiency and accuracy are greatly improved.
[0076] Reference Figure 6 In one embodiment of the present application, a three-dimensional reconstruction of a sound barrier project is provided. The apparatus includes: an image acquisition unit 1100, a first calculation unit 1200, a second calculation unit 1300 and a three-dimensional model construction unit 1400, as follows
[0077] The image acquisition unit 1100 is used to acquire an image of the sound barrier.
[0078] The first computing unit 1200 is used to extract two-dimensional feature maps of multiple levels from the image, generate a depth feature map corresponding to each level based on the two-dimensional feature maps of the multiple levels, reversely project the depth feature map corresponding to each level into a three-dimensional point cloud corresponding to each level, calculate the first three-dimensional voxel corresponding to each level based on the three-dimensional point cloud corresponding to each level, and fuse the first three-dimensional voxel corresponding to each level to obtain a second three-dimensional voxel.
[0079] The second calculation unit 1300 is used to extract the three-dimensional features corresponding to the second three-dimensional voxel, and input the three-dimensional features into a preset three-dimensional voxel reconstruction model to obtain the final three-dimensional voxel reconstructed by the three-dimensional voxel reconstruction model; the three-dimensional voxel reconstruction model is obtained by performing loss calculation on the three-dimensional voxel prediction values extracted from the sound barrier training image and the three-dimensional voxel true values of the sound barrier corresponding to the sound barrier training image, and performing back propagation optimization using the loss calculation.
[0080] The three-dimensional model construction unit 1400 is used to construct a three-dimensional model of the sound barrier according to the final three-dimensional voxels.
[0081] It should be noted that the three-dimensional reconstruction device for the sound barrier project provided in this embodiment and the three-dimensional reconstruction method embodiment for the sound barrier project described above are based on the same inventive concept. Therefore, the relevant contents of the three-dimensional reconstruction method embodiment for the sound barrier project described above are also applicable to the three-dimensional reconstruction device embodiment for the sound barrier project and will not be described in detail here.
[0082] The device of this embodiment performs a secondary three-dimensional voxel reconstruction on the multi-view image. In the first reconstruction, unlike the prior art, the two-dimensional features of different levels of the multi-view image are first extracted to enrich the multi-level information contained in the features, and then semantic segmentation is performed on each two-dimensional feature to obtain the segmentation feature. The initial depth feature map is then optimized by each of the two segmentation features to obtain the depth feature map corresponding to each level. The depth feature map corresponding to each level is then reversely projected into a three-dimensional point cloud corresponding to each level. The first three-dimensional voxel corresponding to each level is calculated based on the three-dimensional point cloud corresponding to each level, and the first three-dimensional voxel corresponding to each level is fused to obtain the second three-dimensional voxel. The fused second three-dimensional voxel can fully fuse the semantic information of different levels and improve the accuracy of reconstructing the three-dimensional voxel. Since the three-dimensional voxels constructed for the first time adopted an unsupervised reconstruction method, the accuracy of the three-dimensional voxel construction was improved. Therefore, in the second reconstruction, a supervised reconstruction method is used to perform a secondary reconstruction of the three-dimensional voxels. Therefore, the three-dimensional features corresponding to the second three-dimensional voxels are first extracted, and the three-dimensional features are input into a preset three-dimensional voxel reconstruction model to reconstruct the final three-dimensional voxels. Since the three-dimensional voxel reconstruction model is trained by a supervised reconstruction method, the final reconstructed three-dimensional voxels are more accurate than the second three-dimensional voxels constructed for the first time.
[0083] like Figure 7 The embodiment of the present application further provides an electronic device, the electronic device comprising:
[0084] at least one memory;
[0085] at least one processor;
[0086] at least one program;
[0087] The program is stored in the memory, and the processor executes at least one program to implement the three-dimensional reconstruction method of the sound barrier project implemented in the present disclosure.
[0088] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.
[0089] The electronic device according to the embodiment of the present application is described in detail below.
[0090] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;
[0091] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1700, and the processor 1600 calls and executes the three-dimensional reconstruction method of the sound barrier engineering of the embodiment of the present invention.
[0092] Input / output interface 1800, used to implement information input and output;
[0093] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);
[0094] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );
[0095] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .
[0096] An embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned three-dimensional reconstruction method of the sound barrier project.
[0097] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0098] The embodiments described in the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.
[0099] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0100] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0101] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0102] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0103] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0104] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0105] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0106] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0107] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.
[0108] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the embodiments of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the embodiments of the present application.
Claims
1. A three-dimensional reconstruction method for a sound barrier project, characterized in that: The three-dimensional reconstruction method of the sound barrier project includes: Acquire an image of the sound barrier; Extracting two-dimensional feature maps of multiple levels from the image respectively, and generating a depth feature map corresponding to each level according to the two-dimensional feature maps of the multiple levels, reversely projecting the depth feature map corresponding to each level into a three-dimensional point cloud corresponding to each level, calculating the first three-dimensional voxel corresponding to each level according to the three-dimensional point cloud corresponding to each level, and fusing the first three-dimensional voxel corresponding to each level to obtain a second three-dimensional voxel; extracting two-dimensional feature maps of multiple levels from the image respectively, and generating a depth feature map corresponding to each level according to the two-dimensional feature maps of the multiple levels, includes: using K-means to semantically segment the two-dimensional feature map; Performing convolution and downsampling on the image layer by layer, and selecting two-dimensional feature maps corresponding to multiple downsampling outputs; Performing semantic segmentation on each of the two-dimensional feature maps to obtain a segmentation feature map corresponding to each of the two-dimensional feature maps; Generate an initial depth feature map according to the image, and optimize the initial depth feature map respectively through the segmentation feature map corresponding to each of the two-dimensional feature maps to obtain a depth feature map corresponding to each layer; Extracting the three-dimensional features corresponding to the second three-dimensional voxels, inputting the three-dimensional features into a preset three-dimensional voxel reconstruction model, and obtaining the final three-dimensional voxels reconstructed by the three-dimensional voxel reconstruction model; the three-dimensional voxel reconstruction model is obtained by performing loss calculation on the three-dimensional voxel prediction values extracted from the sound barrier training image and the three-dimensional voxel real values of the sound barrier corresponding to the sound barrier training image, and performing back propagation optimization using the loss calculation; extracting the three-dimensional features corresponding to the second three-dimensional voxels; Build the first 3D convolutional neural network model; Inputting the second three-dimensional voxel into the first three-dimensional convolutional neural network model to obtain a three-dimensional feature corresponding to the second three-dimensional voxel output by the first three-dimensional convolutional neural network model; wherein the training process of the first three-dimensional convolutional neural network model includes: Construct an initial 3D convolutional neural network model; Inputting a third three-dimensional voxel extracted from the sound barrier training image into the initial three-dimensional convolutional neural network model to obtain a first three-dimensional feature corresponding to the third three-dimensional voxel; Inputting the three-dimensional voxel true value of the sound barrier corresponding to the sound barrier training image into the initial three-dimensional convolutional neural network model to obtain a second three-dimensional feature corresponding to the three-dimensional voxel true value; Calculating a loss according to the first three-dimensional feature and the second three-dimensional feature, and optimizing the initial three-dimensional convolutional neural network model using the loss to obtain an optimized initial three-dimensional convolutional neural network model as the first three-dimensional convolutional neural network model; A three-dimensional model of the sound barrier is constructed based on the final three-dimensional voxels.
2. The three-dimensional reconstruction method of the sound barrier engineering according to claim 1, characterized in that: The three-dimensional voxel reconstruction model is obtained by training a three-dimensional convolutional neural network model.
3. The three-dimensional reconstruction method of the sound barrier engineering according to claim 1, characterized in that: The images are collected by drone.
4. The three-dimensional reconstruction method of the sound barrier engineering according to claim 1, characterized in that: Before computing the first three-dimensional voxel corresponding to each level according to the three-dimensional point cloud corresponding to each level, the three-dimensional reconstruction method of the sound barrier engineering further includes: The three-dimensional point cloud is subjected to filtering processing.
5. A three-dimensional reconstruction device for a sound barrier project, characterized in that: The three-dimensional reconstruction device of the sound barrier project includes: An image acquisition unit, used for acquiring an image of the sound barrier; A first computing unit is used to extract two-dimensional feature maps of multiple levels from the image, generate a depth feature map corresponding to each level according to the two-dimensional feature maps of the multiple levels, reversely project the depth feature map corresponding to each level into a three-dimensional point cloud corresponding to each level, calculate the first three-dimensional voxel corresponding to each level according to the three-dimensional point cloud corresponding to each level, and fuse the first three-dimensional voxel corresponding to each level to obtain a second three-dimensional voxel; extracting two-dimensional feature maps of multiple levels from the image, and generating a depth feature map corresponding to each level according to the two-dimensional feature maps of the multiple levels, includes: semantically segmenting the two-dimensional feature map using K-means; Performing convolution and downsampling on the image layer by layer, and selecting two-dimensional feature maps corresponding to multiple downsampling outputs; Performing semantic segmentation on each of the two-dimensional feature maps to obtain a segmentation feature map corresponding to each of the two-dimensional feature maps; Generate an initial depth feature map according to the image, and optimize the initial depth feature map respectively through the segmentation feature map corresponding to each of the two-dimensional feature maps to obtain a depth feature map corresponding to each layer; A second calculation unit is used to extract the three-dimensional features corresponding to the second three-dimensional voxel, input the three-dimensional features into a preset three-dimensional voxel reconstruction model, and obtain the final three-dimensional voxel reconstructed by the three-dimensional voxel reconstruction model; the three-dimensional voxel reconstruction model is obtained by performing loss calculation on the three-dimensional voxel prediction values extracted from the sound barrier training image and the three-dimensional voxel real values of the sound barrier corresponding to the sound barrier training image, and performing back propagation optimization using the loss calculation; the three-dimensional features corresponding to the second three-dimensional voxel are extracted; Build the first 3D convolutional neural network model; Inputting the second three-dimensional voxel into the first three-dimensional convolutional neural network model to obtain a three-dimensional feature corresponding to the second three-dimensional voxel output by the first three-dimensional convolutional neural network model; wherein the training process of the first three-dimensional convolutional neural network model includes: Construct an initial 3D convolutional neural network model; Inputting a third three-dimensional voxel extracted from the sound barrier training image into the initial three-dimensional convolutional neural network model to obtain a first three-dimensional feature corresponding to the third three-dimensional voxel; Inputting the three-dimensional voxel true value of the sound barrier corresponding to the sound barrier training image into the initial three-dimensional convolutional neural network model to obtain a second three-dimensional feature corresponding to the three-dimensional voxel true value; Calculating a loss according to the first three-dimensional feature and the second three-dimensional feature, and optimizing the initial three-dimensional convolutional neural network model using the loss to obtain an optimized initial three-dimensional convolutional neural network model as the first three-dimensional convolutional neural network model; A three-dimensional model construction unit is used to construct a three-dimensional model of the sound barrier based on the final three-dimensional voxels.
6. An electronic device, characterized in that: include: at least one control processor and a memory for communicatively coupling with the at least one control processor; The memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the three-dimensional reconstruction method of the sound barrier project described in any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the three-dimensional reconstruction method of the sound barrier project described in any one of claims 1 to 4.
Citation Information
Patent Citations
Three-dimensional model reconstruction method based on grid deformation
CN110021069A
Three-dimensional reconstruction method and device based on multiple visual angles of unmanned aerial vehicle
CN111652966A
Multi-view saliency estimation method based on unsupervised learning
CN112329662A
Sensor fusion depth reconstruction data driving method based on attention mechanism
CN113989343A
Three-dimensional reconstruction method and system, machine equipment and computer readable storage medium
CN114119869A
