A method and device for obtaining key information features for three-dimensional reconstruction

By performing multiple iterative processing and feature fusion processing on multiple two-dimensional images, the problem of insufficient ability to capture complex textures and detailed information in deep learning models is solved, and the accuracy of three-dimensional reconstruction is improved.

CN117830774BActive Publication Date: 2024-11-22ZHEJIANG GEELY HLDG GRP CO LTD +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202311720722.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-13
Publication Date
2024-11-22
Estimated Expiration
2043-12-13

AI Technical Summary

Technical Problem

Existing deep learning models have weak information capture capabilities in handling complex textures, details, and edges, resulting in reduced accuracy of 3D reconstruction.

Method used

By performing multiple iterative processing on multiple two-dimensional images of the target scene in different orientations, combining feature extraction and fusion processing, richer multi-scale feature information is obtained, including n rounds of iterative processing for each image, each round of processing includes features 提取和融合处理,并将最终轮迭代处理输出的过渡图进行第二融合处理,以获得融合后特征图。

Benefits of technology

The ability to capture fine-grained image information is improved, and the extracted key information features more accurately reflect the feature information such as texture, details and edges, thereby improving the accuracy of three-dimensional reconstruction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117830774B_ABST
    Figure CN117830774B_ABST
Patent Text Reader

Abstract

The present specification provides a method and device for obtaining key information features for three-dimensional reconstruction, the method comprising: obtaining multiple two-dimensional images of a target scene at different orientations; performing n rounds of iterative processing on each two-dimensional image respectively; wherein each round of iterative processing performed on any two-dimensional image comprises: performing a first fusion processing on m feature maps of different scales extracted from an input image, and outputting a transition map generated by the fusion; obtaining a transition map output by the final round of iterative processing corresponding to each two-dimensional image, and performing a second fusion processing on the m feature maps of different scales extracted from each of the acquired transition maps respectively, to obtain a fused feature map; obtaining key information features for three-dimensional reconstruction of the target scene according to the fused feature maps respectively corresponding to the multiple two-dimensional images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for obtaining key information features for three-dimensional reconstruction. Background Art

[0002] 3D reconstruction technology has received extensive attention in application fields such as autonomous driving, robotics, digital cities, and virtual / mixed reality. In recent years, with the continuous development of deep learning technology, deep learning models have emerged that use neural networks to obtain key information features of point cloud data from a single image or image sequence for 3D reconstruction.

[0003] However, current deep learning models are relatively weak in capturing fine-grained information in images and may have limitations in processing complex textures, details, and edges, resulting in reduced modeling results and accuracy. Summary of the invention

[0004] In order to overcome the problems existing in the related art, this specification provides a method and device for obtaining key information features for three-dimensional reconstruction.

[0005] According to a first aspect of an embodiment of this specification, a method for acquiring key information features for three-dimensional reconstruction is provided, the method comprising:

[0006] Acquire multiple two-dimensional images of the target scene at different orientations;

[0007] n rounds of iterative processing are respectively performed on each two-dimensional image; wherein each round of iterative processing performed on any two-dimensional image includes: performing a first fusion processing on m feature maps of different scales extracted from the input image, and outputting a transition map generated by the fusion; and, the input image in the first round of iterative processing is any one of the two-dimensional images, and the input image in the non-first round of iterative processing is the transition map output by the previous round of iterative processing; n and m are both integers not less than 3;

[0008] Obtain a transition map output by the final round of iterative processing corresponding to each two-dimensional image, and perform a second fusion process on the m feature maps of different scales extracted from each of the obtained transition maps to obtain a fused feature map;

[0009] According to the fused feature maps respectively corresponding to the multiple two-dimensional images, key information features for three-dimensional reconstruction of the target scene are obtained.

[0010] According to a second aspect of an embodiment of this specification, there is provided a device for acquiring key information features for three-dimensional reconstruction, including:

[0011] An image acquisition unit is used to acquire multiple two-dimensional images of the target scene at different orientations;

[0012] an aggregation unit, configured to perform n rounds of iterative processing on each two-dimensional image respectively; wherein each round of iterative processing performed on any two-dimensional image comprises: performing a first fusion processing on m feature maps of different scales extracted from the input image, and outputting a transition map generated by the fusion; and, the input image in the first round of iterative processing is any one of the two-dimensional images, and the input image in the non-first round of iterative processing is the transition map output by the previous round of iterative processing; n and m are both integers not less than 3;

[0013] A fusion processing unit, used to obtain a transition map output by the final round of iterative processing corresponding to each two-dimensional image, and perform a second fusion processing according to m feature maps of different scales extracted from each of the obtained transition maps, so as to obtain a fused feature map;

[0014] The key information feature acquisition unit is used to acquire key information features for three-dimensional reconstruction of the target scene according to the fused feature maps corresponding to the multiple two-dimensional images.

[0015] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method described in the first aspect are implemented.

[0016] According to a fourth aspect of the embodiments of this specification, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method described in the first aspect.

[0017] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0018] In the embodiment of the present specification, for multiple two-dimensional images of the acquired target scene at different orientations, multiple rounds of iterative processing are performed for each two-dimensional image, each round of iterative processing includes two steps of feature extraction and feature fusion, and the transition map output by the final round of iterative processing is subjected to a second fusion processing according to the m different scale feature maps extracted from each acquired transition map, so as to obtain a fused feature map. The above method can be used to cyclically extract and fuse feature information of different scales of the image, so as to achieve the purpose of more in-depth and effective fusion of image features and extraction of richer multi-scale features, so as to facilitate the key information features extracted from the fused feature map to more accurately reflect the feature information such as texture, details and edges of the two-dimensional image.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0021] Figure 1 It is a flow chart of a method for obtaining key information features for three-dimensional reconstruction according to an exemplary embodiment of the present specification.

[0022] Figure 2 It is a schematic diagram of a method for acquiring a two-dimensional image according to an exemplary embodiment of the present specification.

[0023] Figure 3 It is a schematic diagram of a method for performing n rounds of iterative processing on a two-dimensional image according to an exemplary embodiment of the present specification.

[0024] Figure 4 It is a schematic diagram of a method for extracting features from an input image according to an exemplary embodiment of the present specification.

[0025] Figure 5 It is a schematic diagram of a method for performing a first fusion process on an input image according to an exemplary embodiment of the present specification.

[0026] Figure 6 It is a schematic diagram of a method for obtaining a fused feature map according to an exemplary embodiment of the present specification.

[0027] Figure 7 It is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present specification.

[0028] Figure 8 It is a block diagram of a device for obtaining key information features for three-dimensional reconstruction according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0030] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the" and "the" used in this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0031] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0032] 3D reconstruction technology has received extensive attention in application fields such as autonomous driving, robotics, digital cities, and virtual / mixed reality. In recent years, with the continuous development of deep learning technology, deep learning models have emerged that use neural networks to obtain key information features of point cloud data from a single image or image sequence for 3D reconstruction.

[0033] However, current deep learning models are relatively weak in capturing fine-grained information in images and may have limitations in processing complex textures, details, and edges, resulting in reduced modeling results and accuracy.

[0034] In response to the above problems, this application proposes a method for obtaining key information features for three-dimensional reconstruction. By more deeply and effectively fusing image features, richer multi-scale features are extracted to improve the ability to capture fine-grained information of the image, which is beneficial to subsequent three-dimensional reconstruction tasks.

[0035] Next, the embodiments of this specification are described in detail.

[0036] like Figure 1 As shown, Figure 1 This is a flowchart of a method for obtaining key information features for three-dimensional reconstruction according to an exemplary embodiment of the present specification, comprising the following steps:

[0037] Step 101: Acquire multiple two-dimensional images of a target scene at different orientations.

[0038] For different target scenes, such as indoor environments, buildings, or in-car environments, spatial image information that can reflect different angles of the target scenes can be obtained by placing cameras at different positions of the target scenes. Taking the target scene as the in-car environment as an example, the spatial image information of the in-car environment can be collected by cameras at the center front, left rear, and right rear of the car. The above is only an example, and this specification does not limit the position and number of cameras.

[0039] In one embodiment, if Figure 2 As shown, the encoder is used to map the feature information of multiple initial images I1, I2, ..., Ir taken at different orientations in the target scene to the latent vector space 20 to obtain the two-dimensional images P1, P2, ..., Pr corresponding to each initial image I1, I2, ..., Ir, where r is the total number of initial images taken by the camera. Among them, any two-dimensional image Pi is a low-dimensional representation of the corresponding initial image Ii. Optionally, the encoder can be an encoder of a VAE model.

[0040] Through the above embodiments, redundant information in the initial image can be significantly reduced; and compressing the image into a vector representation of the latent space can also reduce the data dimension, which is conducive to accelerating subsequent image reasoning tasks.

[0041] Step 102, respectively perform n rounds of iterative processing on each two-dimensional image; wherein each round of iterative processing performed on any two-dimensional image includes: performing a first fusion processing on m feature maps of different scales extracted from the input image, and outputting a transition map generated by the fusion; and, the input image in the first round of iterative processing is any one of the two-dimensional images, and the input image in the non-first round of iterative processing is the transition map output by the previous round of iterative processing; n and m are both integers not less than 3.

[0042] Since the logic of n rounds of iterative processing implemented on two-dimensional images P1, P2, ..., Pr in different orientations is the same, the following only introduces the processing logic of any two-dimensional image Pi:

[0043] In one embodiment, if Figure 3 As shown, n rounds of iterative processing are performed on the two-dimensional image Pi, wherein the processing logic of each round of iterative processing includes two adjacent steps of feature extraction 3011 and first fusion processing 3012. Feature extraction 3011 includes extracting m feature maps of different scales from the input image, and the first fusion processing 3012 includes fusing the extracted m feature maps of different scales, and finally obtaining a fused transition map.

[0044] First, in the first round of iterative processing 301, the input image is a two-dimensional image Pi, and the first round of iterative processing 301 is to extract m feature maps of different scales Ti-1, Ti-2, ..., Ti-m of the two-dimensional image Pi, and output the transition map D1 generated by the first fusion processing 3012 to the second round of iterative processing 302. Secondly, in non-first round of iterative processing, the input image is the transition map output by the previous round of iterative processing, and the processing steps are to extract m feature maps of different scales of the transition map, and output the transition map generated by the first fusion processing 3012 as the input image of the next round of iterative processing. Exemplarily, in the second round of iterative processing, the input image is the transition map D1 output by the first round of iterative processing 301, and the processing steps are to extract m feature maps of different scales of the transition map D1, and output the transition map D2 generated by the first fusion processing 3012. By analogy, the input image in the nth round of iterative processing 303 is the transition image Dn-1, and after the two steps of feature extraction 3011 and the first fusion processing 3012, the output image is the transition image Dn.

[0045] Next, we will introduce Figure 3 Each round of iterative processing includes two steps: feature extraction 3011 and first fusion processing 3012:

[0046] In one embodiment, for Figure 3 For each round of iterative processing of the input image, including the two-dimensional image Pi (the first round of iterative processing of the input image) or the transition image Dj (the non-first round of iterative processing of the input image), it can be obtained by Figure 4 The method shown extracts m feature maps of different scales Ti-1, Ti-2, ..., Ti-m. Figure 4 As shown, a convolutional neural network can be selected to extract m original feature maps Fi-1, Fi-2, ..., Fi-m of different scales from the input image. Optionally, the size of the convolution kernel and the number of channels of the image can be increased in an incremental manner to extract m original feature maps of different scales of the image. Optionally, the extracted original feature maps Fi-1, Fi-2, ..., Fi-m of m scales can be directly used in the subsequent first fusion processing 3012 step. Optionally, the original feature maps Fi-1, Fi-2, ..., Fi-m of each scale extracted can be processed using a self-attention mechanism model to obtain feature maps Ti-1, Ti-2, ..., Ti-m with contextual connections, and the obtained feature maps Ti-1, Ti-2, ..., Ti-m of m different scales with contextual connections can be used in the subsequent first fusion processing 3012 step.

[0047] Based on the above embodiments, Figure 4As shown, the embedding vector Ei of the orientation information corresponding to the image Ii in the target scene can also be obtained, and the embedding vector Ei and the original feature maps Fi-1, Fi-2, ..., Fi-m are jointly input into the self-attention mechanism model to be processed using the self-attention mechanism model. Among them, the orientation information of the image Ii can be the position of the camera that obtains the image. For example, if the target scene is an in-car scene, the position of the camera can be at the left rear, right rear or middle front of the car. Then, the embedding model is used to convert the orientation information of the image Ii into an embedding vector Ei of the same dimension as the two-dimensional image Pi, and they are jointly input into the self-attention mechanism model for processing.

[0048] against Figure 3 The step of the first fusion processing 3012 in is illustrated by taking m=3 as an example.

[0049] In one embodiment, if Figure 5 As shown, based on Figure 4 The method extracts m feature maps Ti-1, Ti-2, ..., Ti-3 of different scales of the input image, transforms the feature map Ti-1 of the first scale into a feature map Bi-1 of the same dimension as the feature map Ti-2 of the second scale through dimension transformation, and adds the feature map Bi-1 to the feature map Ti-2 of the second scale to obtain a first intermediate fusion feature map Ui-2; converts the first intermediate fusion feature map Ui-2 into the dimension of the feature map Ti-3 of the third scale, and adds the transformed first intermediate fusion feature map Bi-2 to the feature map Ti-3 of the third scale to obtain a transition map D.

[0050] It should be noted that, when m is greater than 3, the above processing logic is still used and will not be elaborated in this specification.

[0051] In this embodiment, since the image information captured by feature maps of different scales has different semantic levels and perceptual ranges, by performing dimensional transformation and accumulation processing on feature maps of different scales, the contributions of feature maps of different scales can be balanced to provide a richer and more comprehensive visual expression.

[0052] Step 103: Obtain a transition map outputted by the final round of iterative processing corresponding to each two-dimensional image, and perform a second fusion process on the m feature maps of different scales extracted from each of the obtained transition maps to obtain a fused feature map.

[0053] In one embodiment, obtaining Figure 3 The transition graph Dn output by the final round of iterative processing in the n-round iterative processing step is Figure 4 The feature extraction method shown extracts m feature maps Ti-1, Ti-2, ..., Ti-m of different scales from the transition map Dn.

[0054] In another embodiment, if Figure 6 As shown, m-1 rounds of feature fusion iterative processing are performed on feature maps of m different scales. First, the feature map Ti-1 of the first scale is converted to the dimension size of the feature map Ti-2 of the second scale, and the converted feature map is fused with the feature map Ti-2 of the second scale to obtain the second intermediate fusion feature map Ui-2; then, the second intermediate fusion feature map Ui-2 is converted to the dimension size of the feature map Ti-3 of the third scale, and the converted feature map is fused with the feature map Ti-3 of the third scale to obtain the third intermediate fusion feature map Ui-3; and so on, m-1 different intermediate fusion feature maps Ui-2, Ui-3, ..., Ui-m can be obtained. Finally, a second fusion process 50 can be performed based on the obtained m-1 intermediate fusion feature maps and the feature map Ti-1 of the first scale to obtain the fused feature map Ki corresponding to the two-dimensional image Pi. Optionally, the second fusion processing 50 may be to convert the acquired m-1 intermediate fusion feature maps Ui-2, Ui-3, ..., Ui-m and the first-scale feature map Ti-1 into the same dimension and then perform element-wise addition.

[0055] In one embodiment, before performing the second fusion process 50, each intermediate fusion feature map Ui-2, Ui-3, ..., Ui-m and the feature map Ti-1 of the first scale may be processed using a depthwise separable convolutional network to increase the receptive field of the network and extract rich semantic information. Then, the processed feature maps are subjected to the second fusion process 50 to obtain the fused feature map Ki.

[0056] Step 104: Acquire key information features for three-dimensional reconstruction of the target scene according to the fused feature maps corresponding to the multiple two-dimensional images.

[0057] In one embodiment, the fused feature maps K1, K2, ..., Kr corresponding to each of the obtained two-dimensional images P1, P2, ..., Pr can be respectively input into a multi-layer perceptron to obtain key information features for three-dimensional reconstruction of the target scene. Among them, the key information features can be the light intensity, transparency, and RGB value of the original image.

[0058] In another embodiment, the acquired key information features can be processed by a decoder and restored to a high-resolution image for subsequent 3D reconstruction tasks. Optionally, the decoder can be a decoder of a VAE model.

[0059] So far, a method for obtaining key information features for three-dimensional reconstruction has been introduced.

[0060] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of a device and a terminal to which it is applied.

[0061] like Figure 7 As shown, Figure 7 It is a structural diagram of an electronic device shown in this specification according to an exemplary embodiment. At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, a memory 708 and a non-volatile memory 710, and of course may also include hardware required for other services. One or more embodiments of this specification can be implemented based on software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into the memory 708 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or logic devices.

[0062] like Figure 8 As shown, Figure 8 is a block diagram of a device for obtaining key information features for three-dimensional reconstruction according to an exemplary embodiment of the present specification. The device can be applied to Figure 7 In the electronic device shown in the figure, the technical solution of this specification is implemented. The device includes:

[0063] An image acquisition unit 802 is used to acquire multiple two-dimensional images of a target scene at different orientations;

[0064] Aggregation unit 804, used to perform n rounds of iterative processing on each two-dimensional image respectively; wherein each round of iterative processing performed on any two-dimensional image includes: performing a first fusion processing on m feature maps of different scales extracted from the input image, and outputting a transition map generated by the fusion; and, the input image in the first round of iterative processing is any two-dimensional image, and the input image in the non-first round of iterative processing is the transition map output by the previous round of iterative processing; n and m are both integers not less than 3;

[0065] A fusion processing unit 806 is used to obtain a transition map output by the final round of iterative processing corresponding to each two-dimensional image, and perform a second fusion process according to the m feature maps of different scales extracted from each of the obtained transition maps to obtain a fused feature map;

[0066] The key information feature acquisition unit 808 is used to acquire key information features for three-dimensional reconstruction of the target scene according to the fused feature maps respectively corresponding to the multiple two-dimensional images.

[0067] The image acquisition unit 802 is specifically used to use an encoder to map the feature information of multiple initial images taken at different orientations in the target scene to a latent space to obtain two-dimensional images corresponding to each initial image; wherein the two-dimensional image is a low-dimensional representation of the corresponding initial image.

[0068] Aggregation unit 804 is specifically used to extract m original feature maps of different scales from any one of the images using a convolutional neural network; and process the original feature map of each scale separately using a self-attention mechanism model to obtain a feature map with contextual connection.

[0069] The aggregation unit 804 is specifically used to obtain an embedding vector of the orientation information corresponding to any image in the target scene; input the embedding vector and the original feature map into the self-attention mechanism model together to be processed using the self-attention mechanism model.

[0070] The aggregation unit 804 is specifically used to perform a first fusion process on the feature maps of m different scales, including: performing m-1 rounds of iterative fusion process on the feature maps of the m different scales; wherein the i-th round of iterative fusion process includes: converting the feature map of the i-th scale into the dimension size of the feature map of the i+1-th scale, and adding the converted feature map to the feature map of the i+1-th scale element by element until i=m-1; wherein the initial value of i is 1, and the value of i is incremented by 1 after each round of iterative fusion process.

[0071] The fusion processing unit 806 is specifically used to perform m-1 rounds of feature fusion iterative processing on the m feature maps of different scales extracted from each acquired transition map; wherein the j-th round of feature fusion iterative processing includes: converting the feature map of the j-th scale into the dimension size of the feature map of the j+1-th scale, and fusing the converted feature map with the feature map of the j+1-th scale to obtain the j+1-th intermediate fused feature map; wherein the initial value of j is 1, and the value of j is increased by 1 after each round of feature fusion iterative processing until j=m-1; and performing the second fusion processing according to all the acquired intermediate fused feature maps and the feature map of the first scale.

[0072] The fusion processing unit 806 is specifically used to process each intermediate fusion feature map and the feature map of the first scale using a deep separable convolutional network to obtain corresponding processed feature maps; and perform a second fusion process on all the processed feature maps.

[0073] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0074] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0075] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0076] Those skilled in the art will readily appreciate other embodiments of the specification after considering the specification and practicing the invention claimed herein. The specification is intended to cover any variations, uses or adaptations of the specification that follow the general principles of the specification and include common knowledge or customary techniques in the art that are not claimed in the specification. The specification and examples are to be considered exemplary only, and the true scope and spirit of the specification are indicated by the following claims.

[0077] It should be understood that the present description is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

[0078] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A method for obtaining key information features for three-dimensional reconstruction, characterized in that: The method comprises: Acquire multiple two-dimensional images of the target scene at different orientations; n rounds of iterative processing are respectively performed on each two-dimensional image; wherein each round of iterative processing performed on any two-dimensional image includes: performing a first fusion processing on m feature maps of different scales extracted from the input image, and outputting a transition map generated by the fusion; and, the input image in the first round of iterative processing is any one of the two-dimensional images, and the input image in the non-first round of iterative processing is the transition map output by the previous round of iterative processing; n and m are both integers not less than 3; Obtain a transition map outputted by the final round of iterative processing corresponding to each two-dimensional image, and perform a second fusion processing on the m feature maps of different scales extracted from each of the acquired transition maps, respectively, to obtain a fused feature map; the second fusion processing on the m feature maps of different scales extracted from each of the acquired transition maps, respectively, includes: performing m-1 rounds of feature fusion iterative processing on the m feature maps of different scales extracted from each of the acquired transition maps; wherein the j-th round of feature fusion iterative processing includes: converting the feature map of the j-th scale into the dimension size of the feature map of the j+1-th scale, and fusing the converted feature map with the feature map of the j+1-th scale to obtain the j+1-th intermediate fused feature map; wherein the initial value of j is 1, and the value of j is incremented by 1 after each round of feature fusion iterative processing until j=m-1; the second fusion processing is performed on all the acquired intermediate fused feature maps and the feature map of the first scale; According to the fused feature maps respectively corresponding to the multiple two-dimensional images, key information features for three-dimensional reconstruction of the target scene are obtained.

2. The method according to claim 1, characterized in that The step of acquiring a plurality of two-dimensional images of the target scene at different orientations includes: Mapping feature information of a plurality of initial images taken at different orientations under the target scene to a latent space using an encoder to obtain two-dimensional images corresponding to the initial images; The two-dimensional image is a low-dimensional representation of the corresponding initial image.

3. The method according to claim 1, characterized in that Extract m feature maps of different scales from any image, including: Extracting m original feature maps of different scales from any one of the images using a convolutional neural network; The self-attention mechanism model is used to process the original feature maps of each scale separately to obtain feature maps with contextual connections.

4. The method according to claim 3, characterized in that The self-attention mechanism model is used to process the original feature map of each scale separately, including: Obtaining an embedded vector of the position information corresponding to any one of the images in the target scene; The embedding vector and the original feature map are input into the self-attention mechanism model together for processing by the self-attention mechanism model.

5. The method according to claim 1, characterized in that The first fusion process of the m feature maps of different scales includes: Performing m-1 rounds of iterative fusion processing on the m feature maps of different scales; wherein the i-th round of iterative fusion processing includes: converting the feature map of the i-th scale into the dimension size of the feature map of the i+1-th scale, and adding the converted feature map to the feature map of the i+1-th scale element by element until i=m-1; The initial value of i is 1, and the value of i increases by 1 after each round of iterative fusion processing.

6. The method according to claim 1, characterized in that The second fusion processing is performed according to all the acquired intermediate fusion feature maps and the feature map of the first scale, including: For each intermediate fusion feature map and the feature map of the first scale, a deep separable convolutional network is used to process them respectively to obtain the corresponding processed feature maps; The second fusion process is performed on all processed feature maps.

7. A device for obtaining key information features for three-dimensional reconstruction, characterized in that: The device comprises: An image acquisition unit is used to acquire multiple two-dimensional images of the target scene at different orientations; an aggregation unit, configured to perform n rounds of iterative processing on each two-dimensional image respectively; wherein each round of iterative processing performed on any two-dimensional image comprises: performing a first fusion processing on m feature maps of different scales extracted from the input image, and outputting a transition map generated by the fusion; and, the input image in the first round of iterative processing is any one of the two-dimensional images, and the input image in the non-first round of iterative processing is the transition map output by the previous round of iterative processing; n and m are both integers not less than 3; A fusion processing unit is used to obtain a transition map output by the final round of iterative processing corresponding to each two-dimensional image, and perform a second fusion processing according to the m different scale feature maps extracted from each acquired transition map, so as to obtain a fused feature map; the second fusion processing according to the m different scale feature maps extracted from each acquired transition map comprises: performing m-1 rounds of feature fusion iterative processing on the m different scale feature maps extracted from each acquired transition map, respectively; wherein the j-th round of feature fusion iterative processing comprises: converting the feature map of the j-th scale into the dimension size of the feature map of the j+1-th scale, and fusing the converted feature map with the feature map of the j+1-th scale to obtain the j+1-th intermediate fusion feature map; wherein the initial value of j is 1, and the value of j is incremented by 1 after each round of feature fusion iterative processing until j=m-1; the second fusion processing is performed according to all the acquired intermediate fusion feature maps and the feature map of the first scale; The key information feature acquisition unit is used to acquire key information features for three-dimensional reconstruction of the target scene according to the fused feature maps corresponding to the multiple two-dimensional images.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Multi-view three-dimensional reconstruction method and device based on variable convolutional depth network

    CN113963117A

  • Image three-dimensional reconstruction method and device, electronic equipment and storage medium

    CN114548238A

  • Three-dimensional CT image reconstruction method and device based on single projection data

    CN115496659A

  • Multi-view three-dimensional reconstruction method based on deep learning

    CN116310095A

  • Multi-view three-dimensional reconstruction method based on attention mechanism and variable convolutional depth network

    CN116310098A