Three-dimensional feature optimization method, device and electronic device for visual scene understanding

Abstract

The present invention provides a three-dimensional feature optimization method, device and electronic device for visual scene understanding. In the method, a feature densification module is used to aggregate sparse three-dimensional features layer by layer to obtain dense three-dimensional features with global geometric and semantic information, which can significantly improve the model's ability to capture global and local information of three-dimensional scenes. In addition, the combination of random feature query and instance feature query, as well as the aggregation of dense three-dimensional features and image features, obtains optimized three-dimensional features, which solves the problem of uncertain areas in the scene and improves the robustness of the model and the prediction accuracy of key objects.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a three-dimensional feature optimization method, device and electronic equipment for visual scene understanding. Background Art

[0002] In the autonomous driving scenario, the 3D semantic occupancy prediction task needs to restore the 3D structure of the scene from the sparse 2D image. The specific process is: extract image features from the image of the current scene, then convert the image features into sparse 3D voxel features, and finally, predict whether each voxel in the 3D space is occupied by an obstacle based on the sparse 3D voxel features. If occupied, determine the category of the obstacle in the occupied voxel.

[0003] In the above process, sparse three-dimensional voxel features will lead to missing information in the scene, especially in highly dynamic scenes or complex structures. The above method is difficult to accurately capture global and local semantic information; in addition, the optimization of uncertain areas in the scene (such as occluded areas or edges) is often insufficient, which limits the accuracy of the extracted three-dimensional voxel features in representing the three-dimensional scene.

[0004] In summary, how to optimize three-dimensional features to avoid information loss in the scene and improve the accuracy of three-dimensional scene representation has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a three-dimensional feature optimization method, device and electronic device for visual scene understanding, so as to alleviate the technical problems that the three-dimensional features obtained by the prior art will lead to missing information in the scene and poor accuracy in representing the three-dimensional scene.

[0006] In a first aspect, an embodiment of the present invention provides a three-dimensional feature optimization method for visual scene understanding, comprising:

[0007] Get the sparse three-dimensional features of the current scene;

[0008] A feature densification module is used to aggregate the sparse three-dimensional features layer by layer to obtain dense three-dimensional features with global geometric and semantic information;

[0009] Generating uniformly distributed random query points on the dense three-dimensional features to obtain a random feature query;

[0010] Obtaining an instance feature query according to the initialized three-dimensional features and the image features corresponding to the current scene;

[0011] Merging the random feature query and the instance feature query to obtain a new feature query;

[0012] An optimized three-dimensional feature is obtained by aggregating the dense three-dimensional feature, the new feature query and the image feature.

[0013] Furthermore, the sparse three-dimensional features of the current scene are obtained, including:

[0014] Acquire multi-view images of the current scene;

[0015] Using a pre-trained network to extract features from the multi-view images to obtain multi-scale image features corresponding to the current scene;

[0016] The image features are converted into the sparse three-dimensional features by using a perspective transformation method.

[0017] Furthermore, the feature densification module is a feature densification module similar to the Hourglass network.

[0018] Further, according to the initialized three-dimensional features and the image features corresponding to the current scene, an instance feature query is obtained, including:

[0019] Obtaining the customized initialized three-dimensional features;

[0020] The instance feature query is obtained by using a deformable cross attention mechanism and taking the initialized three-dimensional features as queries and the image features as keys and values ​​for feature aggregation.

[0021] Further, obtaining optimized three-dimensional features according to the dense three-dimensional features, the new feature query and the image feature aggregation includes:

[0022] Based on the dense three-dimensional features, the new feature query is integrated to obtain an optimized feature query;

[0023] The optimized feature query and the image feature are aggregated through a deformable cross-attention mechanism to obtain the optimized three-dimensional feature.

[0024] Furthermore, the method further comprises:

[0025] Perform semantic occupancy prediction on the optimized three-dimensional features to obtain a semantic occupancy prediction result.

[0026] In a second aspect, an embodiment of the present invention provides a three-dimensional feature optimization device for visual scene understanding, comprising:

[0027] An acquisition unit, used for acquiring sparse three-dimensional features of the current scene;

[0028] A layer-by-layer aggregation unit, used to aggregate the sparse three-dimensional features layer by layer using a feature densification module to obtain dense three-dimensional features with global geometric and semantic information;

[0029] A random query unit, used for generating uniformly distributed random query points on the dense three-dimensional features to obtain a random feature query;

[0030] An instance query unit, used for obtaining an instance feature query according to the initialized three-dimensional features and the image features corresponding to the current scene;

[0031] A fusion unit, used for fusing the random feature query and the instance feature query to obtain a new feature query;

[0032] An aggregation unit is used to aggregate the dense three-dimensional features, the new feature query and the image features to obtain optimized three-dimensional features.

[0033] Furthermore, the acquisition unit is also used for:

[0034] Acquire multi-view images of the current scene;

[0035] Using a pre-trained network to extract features from the multi-view images to obtain multi-scale image features corresponding to the current scene;

[0036] The image features are converted into the sparse three-dimensional features by using a perspective transformation method.

[0037] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0038] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to execute any method described in the first aspect above.

[0039] In an embodiment of the present invention, a three-dimensional feature optimization method for visual scene understanding is provided, comprising: obtaining sparse three-dimensional features of a current scene; aggregating the sparse three-dimensional features layer by layer using a feature densification module to obtain dense three-dimensional features with global geometric and semantic information; generating uniformly distributed random query points on the dense three-dimensional features to obtain random feature queries; obtaining instance feature queries based on the initialized three-dimensional features and image features corresponding to the current scene; fusing the random feature queries with the instance feature queries to obtain new feature queries; and obtaining optimized three-dimensional features based on the aggregation of dense three-dimensional features, new feature queries and image features. From the above description, it can be seen that in the three-dimensional feature optimization method for visual scene understanding of the present invention, a feature densification module is used to aggregate sparse three-dimensional features layer by layer, and then dense three-dimensional features with global geometric and semantic information are obtained, which can significantly improve the model's ability to capture global and local information of three-dimensional scenes. In addition, the combination of random feature query and instance feature query, as well as the aggregation of dense three-dimensional features and image features, obtains optimized three-dimensional features, which solves the problem of uncertain areas in the scene, improves the robustness of the model and the prediction accuracy of key objects, and alleviates the technical problem that the three-dimensional features obtained by traditional technology will lead to missing information in the scene and poor accuracy in the representation of the three-dimensional scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A flowchart of a three-dimensional feature optimization method for visual scene understanding provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of three-dimensional feature optimization provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of feature densification provided by an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of random query and example query provided by an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of a three-dimensional feature optimization device for visual scene understanding provided by an embodiment of the present invention;

[0046] Figure 6A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The three-dimensional features obtained by traditional technology will lead to missing information in the scene and poor accuracy in representing the three-dimensional scene.

[0049] Based on this, in the three-dimensional feature optimization method for visual scene understanding of the present invention, a feature densification module is used to aggregate sparse three-dimensional features layer by layer to obtain dense three-dimensional features with global geometric and semantic information, which can significantly improve the model's ability to capture global and local information of three-dimensional scenes. In addition, the combination of random feature query and instance feature query, as well as the aggregation of dense three-dimensional features and image features, obtains optimized three-dimensional features, which solves the problem of uncertain areas in the scene and improves the robustness of the model and the prediction accuracy of key objects.

[0050] To facilitate understanding of this embodiment, a three-dimensional feature optimization method for visual scene understanding disclosed in an embodiment of the present invention is first introduced in detail.

[0051] Embodiment 1:

[0052] According to an embodiment of the present invention, an embodiment of a three-dimensional feature optimization method for visual scene understanding is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0053] Figure 1 is a flow chart of a three-dimensional feature optimization method for visual scene understanding according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0054] Step S102, obtaining sparse three-dimensional features of the current scene;

[0055] Step S104, using a feature densification module to aggregate the sparse three-dimensional features layer by layer to obtain dense three-dimensional features with global geometric and semantic information;

[0056] Specifically, Figure 2 and Figure 3As shown in the figure, the feature densification module captures global contextual semantic information and retains local semantic details by aggregating sparse 3D features (i.e., sparse 3D voxel features) layer by layer to generate dense 3D features (i.e., dense 3D voxel features) containing complete (i.e., global) geometric and semantic information.

[0057] Step S106, generating uniformly distributed random query points on the dense three-dimensional features to obtain a random feature query;

[0058] Specifically, Figure 2 and Figure 4 As shown, uniformly distributed random query points are generated on dense 3D features to capture the 3D voxel features of uncertain areas, which are used to model uncertainty areas in complex scenes.

[0059] Step S108, obtaining an instance feature query based on the initialized three-dimensional features and the image features corresponding to the current scene;

[0060] Specifically, Figure 2 and Figure 4 As shown, the above-mentioned initialized three-dimensional features are customized initialized three-dimensional features of each voxel (the number of voxels is also preset), wherein the initialized three-dimensional features of each voxel can be 128-dimensional, and instance feature queries of key objects (such as vehicles and pedestrians) can be obtained based on the initialized three-dimensional features and the image features corresponding to the current scene, so as to perform accurate modeling of the target area, thereby improving the analysis capability of key objects.

[0061] Step S110, fusing the random feature query and the instance feature query to obtain a new feature query;

[0062] Step S112, obtaining optimized three-dimensional features according to dense three-dimensional features, new feature queries and image feature aggregation.

[0063] Specifically, Figure 2 As shown in the figure, the optimized 3D features obtained through dense 3D features, new feature queries and image feature aggregation contain global geometric and semantic information, and the local detail information is also richer, which can achieve a more detailed and comprehensive scene representation.

[0064] In an embodiment of the present invention, a three-dimensional feature optimization method for visual scene understanding is provided, comprising: obtaining sparse three-dimensional features of a current scene; aggregating the sparse three-dimensional features layer by layer using a feature densification module to obtain dense three-dimensional features with global geometric and semantic information; generating uniformly distributed random query points on the dense three-dimensional features to obtain random feature queries; obtaining instance feature queries based on the initialized three-dimensional features and image features corresponding to the current scene; fusing the random feature queries with the instance feature queries to obtain new feature queries; and obtaining optimized three-dimensional features based on the aggregation of dense three-dimensional features, new feature queries and image features. From the above description, it can be seen that in the three-dimensional feature optimization method for visual scene understanding of the present invention, a feature densification module is used to aggregate sparse three-dimensional features layer by layer, and then dense three-dimensional features with global geometric and semantic information are obtained, which can significantly improve the model's ability to capture global and local information of three-dimensional scenes. In addition, the combination of random feature query and instance feature query, as well as the aggregation of dense three-dimensional features and image features, obtains optimized three-dimensional features, which solves the problem of uncertain areas in the scene, improves the robustness of the model and the prediction accuracy of key objects, and alleviates the technical problem that the three-dimensional features obtained by traditional technology will lead to missing information in the scene and poor accuracy in the representation of the three-dimensional scene.

[0065] The above content briefly introduces the three-dimensional feature optimization method for visual scene understanding of the present invention. The specific contents involved are described in detail below.

[0066] In an optional embodiment of the present invention, obtaining the sparse three-dimensional features of the current scene specifically includes the following steps:

[0067] (1) Obtain multi-view images of the current scene;

[0068] like Figure 2 As shown, the multi-view images may specifically be images of different viewpoints of the current scene captured by multiple cameras at the same time.

[0069] (2) Use a pre-trained network to extract features from multi-view images and obtain multi-scale image features corresponding to the current scene;

[0070] Specifically, the pre-trained network may be ResNet-50.

[0071] (3) The viewpoint transformation method is used to transform image features into sparse three-dimensional features.

[0072] Specifically, the sparse three-dimensional features may be sparse three-dimensional voxel features, that is, a cuboid composed of gridded voxels obtained by gridding the three-dimensional space, and the three-dimensional features of each voxel in the cuboid.

[0073] In an optional embodiment of the present invention, the feature densification module is a feature densification module of an hourglass network type.

[0074] Specifically, the feature densification module of the above-mentioned hourglass network-like network converts the sparse three-dimensional features generated from the image into dense three-dimensional features through downsampling and upsampling.

[0075] The Hourglass network is a convolutional neural network structure specially designed to capture multi-scale information in images. It consists of a series of downsampling (contraction) and upsampling (expansion) layers to form an architecture similar to the shape of an hourglass.

[0076] Symmetry: A notable feature of the Hourglass network is its symmetrical structure, that is, the contraction path and the expansion path have similar numbers of layers and filters. This symmetry helps maintain the spatial correspondence between input and output, allowing the network to locate feature points more accurately.

[0077] Skip Connections: To prevent the loss of important details during multiple downsampling, the Hourglass network introduces skip connections. These connections pass shallow features directly to deep layers, thus ensuring that fine-grained information is not over-smoothed. Skip connections also help alleviate the problem of gradient vanishing and facilitate gradient backpropagation during training.

[0078] Repeating modules: The Hourglass network can be nested, that is, a larger Hourglass structure contains several small Hourglass units. This design allows the network to learn abstract features at different levels and perform feature fusion at multiple scales.

[0079] Downsampling without pooling layer: Different from the traditional practice of downsampling by max pooling or average pooling, the Hourglass network tends to use convolution operations with a step size greater than 1 to reduce the spatial dimension of the feature map. This method can retain more feature information while reducing the dimension.

[0080] In an optional embodiment of the present invention, obtaining an instance feature query according to the initialized three-dimensional features and the image features corresponding to the current scene specifically includes the following steps:

[0081] (1) Obtaining customized initialization 3D features;

[0082] (2) Through the deformable cross-attention mechanism, the initialized three-dimensional features are used as queries and the image features are used as keys and values ​​for feature aggregation to obtain instance feature queries.

[0083] Specifically, refer to Figure 4 , the initialized 3D features and image features are aggregated through the deformable cross-attention mechanism to obtain instance feature query.

[0084] The calculation process of deformable cross attention usually involves three matrices: Query, Key, and Value.

[0085] In an optional embodiment of the present invention, obtaining optimized three-dimensional features according to dense three-dimensional features, new feature queries and image feature aggregation specifically includes the following steps:

[0086] (1) Based on dense 3D features, new feature queries are integrated to obtain optimized feature queries;

[0087] (2) The optimized feature query and image features are aggregated through the deformable cross-attention mechanism to obtain the optimized three-dimensional features.

[0088] Specifically, refer to Figure 2 , the dense three-dimensional features are fused with the new feature query to obtain the optimized feature query, and then the deformable cross-attention mechanism is used to perform feature aggregation with the optimized feature query as the query and the image features as the key and value to obtain the optimized three-dimensional features.

[0089] In an optional embodiment of the present invention, the method further comprises the following steps:

[0090] Semantic occupancy prediction is performed on the optimized three-dimensional features to obtain a semantic occupancy prediction result.

[0091] The three-dimensional feature optimization method for visual scene understanding of the present invention has the following advantages:

[0092] Sparse feature densification: Hourglass-like structures are used to convert sparse 3D features into dense 3D features, significantly improving the model's ability to capture global and local information of 3D scenes.

[0093] Multi-query optimization: The combination of random query and instance query solves the problem of uncertainty areas in the scene, improving the robustness of the model and the prediction accuracy of key objects.

[0094] Embodiment 2:

[0095] An embodiment of the present invention also provides a three-dimensional feature optimization device for visual scene understanding. The three-dimensional feature optimization device for visual scene understanding is mainly used to execute the three-dimensional feature optimization method for visual scene understanding provided in the first embodiment of the present invention. The following is a specific introduction to the three-dimensional feature optimization device for visual scene understanding provided in the embodiment of the present invention.

[0096] Figure 5 is a schematic diagram of a three-dimensional feature optimization device for visual scene understanding according to an embodiment of the present invention, such as Figure 5 As shown, the device mainly includes: an acquisition unit 10, a layer-by-layer aggregation unit 20, a random query unit 30, an example query unit 40, a fusion unit 50 and an aggregation unit 60, wherein:

[0097] An acquisition unit, used for acquiring sparse three-dimensional features of the current scene;

[0098] A layer-by-layer aggregation unit is used to aggregate sparse 3D features layer by layer using a feature densification module to obtain dense 3D features with global geometric and semantic information;

[0099] A random query unit, used to generate uniformly distributed random query points on dense three-dimensional features to obtain random feature queries;

[0100] An instance query unit, used for obtaining an instance feature query according to the initialized three-dimensional features and the image features corresponding to the current scene;

[0101] A fusion unit, used to fuse the random feature query and the instance feature query to obtain a new feature query;

[0102] The aggregation unit is used to obtain optimized 3D features based on dense 3D features, new feature queries and image features.

[0103] In an embodiment of the present invention, a three-dimensional feature optimization device for visual scene understanding is provided, comprising: obtaining sparse three-dimensional features of a current scene; aggregating the sparse three-dimensional features layer by layer using a feature densification module to obtain dense three-dimensional features with global geometric and semantic information; generating uniformly distributed random query points on the dense three-dimensional features to obtain random feature queries; obtaining instance feature queries based on the initialized three-dimensional features and the image features corresponding to the current scene; fusing the random feature queries with the instance feature queries to obtain new feature queries; and obtaining optimized three-dimensional features based on the aggregation of the dense three-dimensional features, the new feature queries and the image features. From the above description, it can be seen that in the three-dimensional feature optimization device for visual scene understanding of the present invention, a feature densification module is used to aggregate sparse three-dimensional features layer by layer, and then dense three-dimensional features with global geometric and semantic information are obtained, which can significantly improve the model's ability to capture global and local information of three-dimensional scenes. In addition, the combination of random feature query and instance feature query, as well as the aggregation of dense three-dimensional features and image features, obtains optimized three-dimensional features, which solves the problem of uncertain areas in the scene, improves the robustness of the model and the prediction accuracy of key objects, and alleviates the technical problem that the three-dimensional features obtained by traditional technology will lead to missing information in the scene and poor accuracy in the representation of the three-dimensional scene.

[0104] Optionally, the acquisition unit is also used to: acquire multi-view images of the current scene; use a pre-trained network to extract features from the multi-view images to obtain multi-scale image features corresponding to the current scene; and use a perspective transformation method to convert the image features into sparse three-dimensional features.

[0105] Optionally, the feature densification module is a feature densification module of an hourglass network-like network.

[0106] Optionally, the instance query unit is also used to: obtain a customized initialized three-dimensional feature; perform feature aggregation using the initialized three-dimensional feature as a query and the image feature as a key and a value through a deformable cross-attention mechanism to obtain an instance feature query.

[0107] Optionally, the aggregation unit is also used to: fuse new feature queries based on dense three-dimensional features to obtain optimized feature queries; aggregate the optimized feature queries and image features through a deformable cross-attention mechanism to obtain optimized three-dimensional features.

[0108] Optionally, the device is also used to: perform semantic occupancy prediction on the optimized three-dimensional features to obtain a semantic occupancy prediction result.

[0109] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0110] like Figure 6 As shown, an electronic device 600 provided in an embodiment of the present application includes: a processor 601, a memory 602 and a bus, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 through the bus, and the processor 601 executes the machine-readable instructions to perform the steps of the three-dimensional feature optimization method for visual scene understanding as described above.

[0111] Specifically, the above-mentioned memory 602 and processor 601 can be general-purpose memory and processor, which are not specifically limited here. When the processor 601 runs the computer program stored in the memory 602, it can execute the above-mentioned three-dimensional feature optimization method for visual scene understanding.

[0112] The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 601. The above processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 602, and the processor 601 reads the information in the memory 602 and completes the steps of the above method in combination with its hardware.

[0113] Corresponding to the above-mentioned three-dimensional feature optimization method for visual scene understanding, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the above-mentioned three-dimensional feature optimization method for visual scene understanding.

[0114] The three-dimensional feature optimization device for visual scene understanding provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiment of the present application, its implementation principle and the technical effect produced are the same as those in the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. It can be clearly understood by technicians in the relevant field that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0115] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0116] For another example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0117] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0118] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0119] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or partly be embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the three-dimensional feature optimization method for visual scene understanding described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0120] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0121] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solution of the present application, rather than to limit it. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solution recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiment of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A three-dimensional feature optimization method for visual scene understanding, characterized in that: include: Get the sparse three-dimensional features of the current scene; A feature densification module is used to aggregate the sparse three-dimensional features layer by layer to obtain dense three-dimensional features with global geometric and semantic information; Generating uniformly distributed random query points on the dense three-dimensional features to obtain a random feature query; Obtaining an instance feature query according to the initialized three-dimensional features and the image features corresponding to the current scene; Merging the random feature query and the instance feature query to obtain a new feature query; An optimized three-dimensional feature is obtained by aggregating the dense three-dimensional feature, the new feature query and the image feature.

2. The method according to claim 1, characterized in that Get the sparse 3D features of the current scene, including: Acquire multi-view images of the current scene; Using a pre-trained network to extract features from the multi-view images to obtain multi-scale image features corresponding to the current scene; The image features are converted into the sparse three-dimensional features by using a perspective transformation method.

3. The method according to claim 1, characterized in that The feature densification module is a feature densification module similar to the Hourglass network.

4. The method according to claim 1, characterized in that: According to the initialized three-dimensional features and the image features corresponding to the current scene, an instance feature query is obtained, including: Obtaining the customized initialized three-dimensional features; The instance feature query is obtained by using a deformable cross attention mechanism and taking the initialized three-dimensional features as queries and the image features as keys and values ​​for feature aggregation.

5. The method according to claim 1, characterized in that: The optimized three-dimensional features are obtained according to the dense three-dimensional features, the new feature query and the image feature aggregation, including: Based on the dense three-dimensional features, the new feature query is integrated to obtain an optimized feature query; The optimized feature query and the image feature are aggregated through a deformable cross-attention mechanism to obtain the optimized three-dimensional feature.

6. The method according to claim 1, characterized in that The method further comprises: Perform semantic occupancy prediction on the optimized three-dimensional features to obtain a semantic occupancy prediction result.

7. A three-dimensional feature optimization device for visual scene understanding, characterized in that: include: An acquisition unit, used for acquiring sparse three-dimensional features of the current scene; A layer-by-layer aggregation unit, used to aggregate the sparse three-dimensional features layer by layer using a feature densification module to obtain dense three-dimensional features with global geometric and semantic information; A random query unit, used for generating uniformly distributed random query points on the dense three-dimensional features to obtain a random feature query; An instance query unit, used for obtaining an instance feature query according to the initialized three-dimensional features and the image features corresponding to the current scene; A fusion unit, used for fusing the random feature query and the instance feature query to obtain a new feature query; An aggregation unit is used to aggregate the dense three-dimensional features, the new feature query and the image features to obtain optimized three-dimensional features.

8. The device according to claim 7, characterized in that The acquisition unit is also used for: Acquire multi-view images of the current scene; Using a pre-trained network to extract features from the multi-view images to obtain multi-scale image features corresponding to the current scene; The image features are converted into the sparse three-dimensional features by using a perspective transformation method.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to execute the method according to any one of claims 1 to 6.

Images