Three-dimensional feature optimization method, device and electronic device for visual scene understanding based on geometry guidance

Abstract

The present invention provides a three-dimensional feature optimization method, device and electronic device for visual scene understanding based on geometric guidance. In this method, the optimized three-dimensional features not only have three-dimensional voxel features, but also integrate geometric prior spatial position information and geometric guidance information, thereby enhancing the model's processing ability for spatially effective position features, improving computing efficiency, significantly improving the model's processing ability for occluded areas, improving the modeling ability for instance objects, and improving the reconstruction accuracy of three-dimensional scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and deep learning, and in particular to a three-dimensional feature optimization method, device and electronic device for visual scene understanding based on geometric guidance. Background Art

[0002] In the task of visual scene understanding, traditional models generally extract image features from the image of the current scene, then convert the image features into sparse three-dimensional voxel features, and finally realize visual scene understanding based on the sparse three-dimensional voxel features. For example, based on the sparse three-dimensional voxel features, it is determined whether each voxel in the 3D space is occupied by an obstacle. If occupied, the category of the obstacle in the occupied voxel is determined.

[0003] In the above process, there is no geometric structure information in the sparse three-dimensional voxel features, so it is difficult to process occluded areas or detail features in complex scenes; in addition, the existing feature optimization method (i.e., the process of determining whether each voxel in the 3D space is occupied by an obstacle based on the sparse three-dimensional voxel features) does not utilize geometric structure information, so its ability to capture local geometric priors is limited. Ultimately, the modeling accuracy of the relationship between scene geometry (detection box, position, speed, and angle information in each voxel) and semantics (whether each voxel is occupied, and the category of obstacles in the occupied voxels) is low, that is, the accuracy of the 3D space finally modeled is poor.

[0004] In summary, how to optimize three-dimensional features so that the optimized three-dimensional features cover geometric structure information and thus improve the accuracy of visual scene understanding has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a three-dimensional feature optimization method, device and electronic device for visual scene understanding based on geometric guidance, so as to alleviate the technical problems that the three-dimensional features obtained in the prior art cover a limited amount of information and have poor accuracy in representing the visual scene.

[0006] In a first aspect, an embodiment of the present invention provides a three-dimensional feature optimization method for visual scene understanding based on geometric guidance, comprising:

[0007] Perform depth estimation and feature extraction on the image to be processed to obtain depth information and image features;

[0008] Determine a pseudo point cloud according to the depth information and the image features, and process the pseudo point cloud using a point cloud processing network to obtain geometric a priori spatial position information, wherein the geometric a priori spatial position information is whether each voxel in the space is occupied by an obstacle and the category of the obstacle in the occupied voxel;

[0009] Performing a perspective transformation on the image features to obtain three-dimensional voxel features, and extracting geometric guidance information from the three-dimensional voxel features, wherein the geometric guidance information includes: a detection frame, a speed, and an angle in each voxel;

[0010] Fusing the geometric prior spatial position information, the geometric guidance information and the three-dimensional voxel feature to obtain a fused feature;

[0011] The fused features are processed using a deformable self-attention mechanism to obtain optimized three-dimensional features, wherein the deformable self-attention mechanism can capture information of occluded areas in the fused features through dynamic offset.

[0012] Furthermore, depth estimation and feature extraction are performed on the image to be processed, including:

[0013] Using a pre-trained depth estimation network to perform depth estimation on the image to be processed to obtain the depth information;

[0014] A feature extraction network is used to extract features of the image to be processed to obtain the image features.

[0015] Further, determining a pseudo point cloud according to the depth information and the image features includes:

[0016] Performing interpolation mapping on the image features to obtain features of corresponding positions in three-dimensional space;

[0017] The pseudo point cloud is determined according to the features of the corresponding position in the three-dimensional space and the depth information.

[0018] Furthermore, extracting geometric guidance information from the three-dimensional voxel features includes:

[0019] The geometric guidance information is extracted from the three-dimensional voxel features by an auxiliary head.

[0020] Furthermore, the geometric prior spatial position information, the geometric guidance information and the three-dimensional voxel feature are fused, including:

[0021] The geometric guidance information and the three-dimensional voxel features are stacked and convolved to obtain the initial fusion feature:

[0022] The initial fusion feature is fused twice with the spatial position information of the geometric prior to obtain the fusion feature.

[0023] Furthermore, a deformable self-attention mechanism is used to process the fused features, including:

[0024] Adopting deformable self-attention calculation formula The fused features are processed to obtain optimized three-dimensional features, wherein Attention(q, X) represents the optimized three-dimensional features, represents the attention weight of the sampling point, X represents the fusion feature, p q represents the location of the query point, represents the learnable offset, represents the bias weight, M represents the number of attention heads, and K represents the number of sampling points for each head.

[0025] Furthermore, the method further comprises:

[0026] Perform semantic occupancy prediction on the optimized three-dimensional features to obtain a semantic occupancy prediction result.

[0027] In a second aspect, an embodiment of the present invention provides a three-dimensional feature optimization device for visual scene understanding based on geometric guidance, comprising:

[0028] A depth estimation and feature extraction unit, used to perform depth estimation and feature extraction on the image to be processed to obtain depth information and image features;

[0029] a determination and processing unit, configured to determine a pseudo point cloud according to the depth information and the image features, and process the pseudo point cloud using a point cloud processing network to obtain geometric a priori spatial position information, wherein the geometric a priori spatial position information is whether each voxel in the space is occupied by an obstacle and the category of the obstacle in the occupied voxel;

[0030] A perspective transformation and extraction unit, configured to transform the image features to obtain three-dimensional voxel features, and extract geometric guidance information from the three-dimensional voxel features, wherein the geometric guidance information includes: a detection frame, a speed, and an angle in each voxel;

[0031] A fusion unit, used for fusing the geometric prior spatial position information, the geometric guidance information and the three-dimensional voxel feature to obtain a fusion feature;

[0032] A processing unit is used to process the fused features using a deformable self-attention mechanism to obtain optimized three-dimensional features, wherein the deformable self-attention mechanism can capture information of occluded areas in the fused features through dynamic offset.

[0033] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0034] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to execute any method described in the first aspect above.

[0035] In an embodiment of the present invention, a three-dimensional feature optimization method for visual scene understanding based on geometric guidance is provided, including: performing depth estimation and feature extraction on a to-be-processed image to obtain depth information and image features; determining a pseudo point cloud according to the depth information and image features, and processing the pseudo point cloud using a point cloud processing network to obtain geometric prior spatial position information, wherein the geometric prior spatial position information is whether each voxel in the space is occupied by an obstacle and the category of the obstacle in the occupied voxel; performing perspective transformation on the image features to obtain three-dimensional voxel features, and extracting geometric guidance information from the three-dimensional voxel features, wherein the geometric guidance information includes: a detection frame, a speed and an angle in each voxel; fusing the geometric prior spatial position information, the geometric guidance information and the three-dimensional voxel features to obtain a fused feature; processing the fused feature using a deformable self-attention mechanism to obtain an optimized three-dimensional feature, wherein the deformable self-attention mechanism can capture information of occluded areas in the fused feature through dynamic offset. From the above description, it can be seen that in the three-dimensional feature optimization method based on geometric guidance for visual scene understanding of the present invention, the optimized three-dimensional features not only have three-dimensional voxel features, but also integrate geometric prior spatial position information and geometric guidance information, thereby enhancing the model's processing ability for spatially effective position features, improving calculation efficiency, significantly improving the model's processing ability for occluded areas, and improving the modeling ability for instance objects, which can improve the reconstruction accuracy of three-dimensional scenes and alleviate the technical problems of the limited amount of information covered by the three-dimensional features obtained in the prior art and the poor accuracy of the representation of visual scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A flowchart of a three-dimensional feature optimization method for visual scene understanding based on geometric guidance provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of three-dimensional feature optimization provided by an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of a process for extracting geometric priori spatial position information provided by an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of a process for extracting geometric guidance information provided by an embodiment of the present invention;

[0041] Figure 5 A schematic diagram of a three-dimensional feature optimization device for visual scene understanding based on geometric guidance provided by an embodiment of the present invention;

[0042] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The three-dimensional features obtained by traditional technology cover a limited amount of information and have poor accuracy in representing visual scenes.

[0045] Based on this, in the three-dimensional feature optimization method based on geometric guidance for visual scene understanding of the present invention, the optimized three-dimensional features not only have three-dimensional voxel features, but also integrate geometric prior spatial position information and geometric guidance information, which enhances the model's processing ability for spatially effective position features, improves computational efficiency, significantly improves the model's processing ability for occluded areas, improves the ability to model instance objects, and can improve the reconstruction accuracy of three-dimensional scenes.

[0046] To facilitate understanding of this embodiment, a three-dimensional feature optimization method based on geometric guidance for visual scene understanding disclosed in an embodiment of the present invention is first introduced in detail.

[0047] Embodiment 1:

[0048] According to an embodiment of the present invention, an embodiment of a three-dimensional feature optimization method for visual scene understanding based on geometry guidance is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0049] Figure 1 is a flow chart of a three-dimensional feature optimization method for visual scene understanding based on geometric guidance according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0050] Step S102, performing depth estimation and feature extraction on the image to be processed to obtain depth information and image features;

[0051] Specifically, the flowchart of the above optimization method is as follows: Figure 2 shown.

[0052] Step S104, determining a pseudo point cloud according to the depth information and the image features, and processing the pseudo point cloud using a point cloud processing network to obtain geometric prior space position information, wherein the geometric prior space position information is whether each voxel in the space is occupied by an obstacle and the category of the obstacle in the occupied voxel;

[0053] Specifically, the geometric prior spatial position information is then combined with the 3D voxel features through a deformable self-attention mechanism to guide the feature optimization of the occluded area. This is because after obtaining the information about which voxels in the space are occupied, even if they are occluded, we know whether these voxels are occupied or occluded, which covers richer information. The extraction process of the geometric prior spatial position information is as follows: Figure 3 shown.

[0054] Step S106, performing perspective transformation on the image features to obtain three-dimensional voxel features, and extracting geometric guidance information from the three-dimensional voxel features, wherein the geometric guidance information includes: a detection frame, a speed, and an angle in each voxel;

[0055] Specifically, the geometric guidance information is used to further optimize the semantic and geometric accuracy of 3D occupancy prediction in downstream tasks. The extraction process of geometric guidance information is as follows: Figure 4 shown.

[0056] Step S108, fusing the geometric prior spatial position information, geometric guidance information and three-dimensional voxel features to obtain fused features;

[0057] The above fusion features can enhance the understanding of instances in space (specifically including: understanding of instance category information, instance detection box (ie, size, shape, position), speed and angle).

[0058] In step S110, the fused features are processed using a deformable self-attention mechanism to obtain optimized three-dimensional features, wherein the deformable self-attention mechanism can capture information of occluded areas in the fused features through dynamic offset.

[0059] Specifically, the information of the occluded area in the 3D voxel feature is captured through dynamic offset.

[0060] In an embodiment of the present invention, a three-dimensional feature optimization method for visual scene understanding based on geometric guidance is provided, including: performing depth estimation and feature extraction on a to-be-processed image to obtain depth information and image features; determining a pseudo point cloud according to the depth information and image features, and processing the pseudo point cloud using a point cloud processing network to obtain geometric prior spatial position information, wherein the geometric prior spatial position information is whether each voxel in the space is occupied by an obstacle and the category of the obstacle in the occupied voxel; performing perspective transformation on the image features to obtain three-dimensional voxel features, and extracting geometric guidance information from the three-dimensional voxel features, wherein the geometric guidance information includes: a detection frame, a speed and an angle in each voxel; fusing the geometric prior spatial position information, the geometric guidance information and the three-dimensional voxel features to obtain a fused feature; processing the fused feature using a deformable self-attention mechanism to obtain an optimized three-dimensional feature, wherein the deformable self-attention mechanism can capture information of occluded areas in the fused feature through dynamic offset. From the above description, it can be seen that in the three-dimensional feature optimization method based on geometric guidance for visual scene understanding of the present invention, the optimized three-dimensional features not only have three-dimensional voxel features, but also integrate geometric prior spatial position information and geometric guidance information, thereby enhancing the model's processing ability for spatially effective position features, improving calculation efficiency, significantly improving the model's processing ability for occluded areas, and improving the modeling ability for instance objects, which can improve the reconstruction accuracy of three-dimensional scenes and alleviate the technical problems of the limited amount of information covered by the three-dimensional features obtained in the prior art and the poor accuracy of the representation of visual scenes.

[0061] The above content briefly introduces the three-dimensional feature optimization method based on geometric guidance for visual scene understanding of the present invention. The specific contents involved are described in detail below.

[0062] In an optional embodiment of the present invention, depth estimation and feature extraction are performed on the image to be processed, specifically including the following steps:

[0063] (1) Use a pre-trained depth estimation network to perform depth estimation on the image to be processed to obtain depth information;

[0064] Specifically, depth information refers to the distance from the camera to an object or point in the scene, which is usually represented by a depth map.

[0065] (2) Use the feature extraction network to extract features from the image to be processed and obtain image features.

[0066] Specifically, the feature extraction network can be a 2D backbone. Figure 2 and Figure 3 shown.

[0067] In an optional embodiment of the present invention, determining a pseudo point cloud according to depth information and image features specifically includes the following steps:

[0068] (1) Interpolate and map the image features to obtain the features of the corresponding positions in the three-dimensional space;

[0069] refer to Figure 2 and Figure 3 Specifically, after interpolation mapping, the image features are converted to three-dimensional space. For example, the position (1, 1) of the two-dimensional image may correspond to the position (2, 2, 1) of the three-dimensional space. Then the feature of the position (2, 2, 1) in the three-dimensional space is the image feature of the above-mentioned position (1, 1).

[0070] (2) Determine the pseudo point cloud based on the features and depth information of the corresponding positions in the three-dimensional space.

[0071] In an optional embodiment of the present invention, extracting geometric guidance information from three-dimensional voxel features specifically includes the following steps:

[0072] Geometric guidance information is extracted from 3D voxel features through an auxiliary head.

[0073] Specifically, Figure 2 and Figure 4 As shown, the auxiliary head can specifically be a fully connected layer.

[0074] In an optional embodiment of the present invention, the geometric prior spatial position information, the geometric guidance information and the three-dimensional voxel features are fused, specifically comprising the following steps:

[0075] (1) Stacking and convolving the geometric guidance information and 3D voxel features to obtain the initial fusion features;

[0076] Specifically, Figure 2 As shown in the figure, assuming that the 3D voxel feature is 128-dimensional and the geometric guidance information is 10-dimensional, the two are stacked to form 138-dimensional, and then convolution processing is performed to obtain the initial fusion feature.

[0077] (2) The initial fusion features are fused with the spatial position information of the geometric prior for a second time to obtain the fusion features.

[0078] refer to Figure 2 Specifically, the secondary fusion is a superposition on the initial fusion feature. According to the geometric prior spatial position information (i.e., position occupancy information), a learnable parameter is added to the unoccupied area. This parameter value is superimposed on the feature value of the initial fusion feature obtained by the above geometric guided fusion to obtain the fusion feature.

[0079] In an optional embodiment of the present invention, a deformable self-attention mechanism is used to process the fusion features, which specifically includes the following steps:

[0080] Adopting deformable self-attention calculation formula

[0081] The fused features are processed to obtain optimized three-dimensional features, where Attention(q, X) represents the optimized three-dimensional features. represents the attention weight of the sampling point, X represents the fusion feature, and p q represents the location of the query point, Represents a learnable offset, which will be added to the original sampling point position. It is generated according to the current input context and can adaptively adjust the sampling position. represents the bias weight, M represents the number of attention heads, and K represents the number of sampling points for each head.

[0082] In an optional embodiment of the present invention, the method further comprises the following steps:

[0083] The semantic occupancy prediction is performed on the optimized three-dimensional features to obtain the semantic occupancy prediction result.

[0084] The method of the present invention has the following advantages:

[0085] Geometric position (geometric prior spatial position information) fusion: enhances the model's ability to process spatially effective position features and improves computational efficiency;

[0086] Geometry-guided fusion: significantly improves the model's ability to handle occluded areas, improves the modeling capabilities of instance objects, and improves the reconstruction accuracy of 3D scenes;

[0087] Semantic-geometric consistency enhancement: The optimized network generates 3D semantic occupancy predictions with higher geometric and semantic consistency.

[0088] Embodiment 2:

[0089] An embodiment of the present invention also provides a three-dimensional feature optimization device based on geometric guidance for visual scene understanding. The three-dimensional feature optimization device based on geometric guidance for visual scene understanding is mainly used to execute the three-dimensional feature optimization method based on geometric guidance for visual scene understanding provided in Example 1 of the present invention. The following is a specific introduction to the three-dimensional feature optimization device based on geometric guidance for visual scene understanding provided in the embodiment of the present invention.

[0090] Figure 5 is a schematic diagram of a three-dimensional feature optimization device for visual scene understanding based on geometric guidance according to an embodiment of the present invention, such as Figure 5As shown, the device mainly includes: a depth estimation and feature extraction unit 10, a determination and processing unit 20, a perspective transformation and extraction unit 30, a fusion unit 40, and a processing unit 50, wherein:

[0091] A depth estimation and feature extraction unit, used to perform depth estimation and feature extraction on the image to be processed to obtain depth information and image features;

[0092] A determination and processing unit, used to determine a pseudo point cloud according to the depth information and the image features, and process the pseudo point cloud using a point cloud processing network to obtain geometric prior space position information, wherein the geometric prior space position information is whether each voxel in the space is occupied by an obstacle and the category of the obstacle in the occupied voxel;

[0093] A perspective transformation and extraction unit, used to transform the perspective of the image features to obtain three-dimensional voxel features, and extract geometric guidance information from the three-dimensional voxel features, wherein the geometric guidance information includes: a detection frame, a speed and an angle in each voxel;

[0094] A fusion unit, used for fusing geometric prior spatial position information, geometric guidance information and three-dimensional voxel features to obtain fused features;

[0095] The processing unit is used to process the fused features by adopting a deformable self-attention mechanism to obtain optimized three-dimensional features, wherein the deformable self-attention mechanism can capture the information of the occluded area in the fused features through dynamic offset.

[0096] In an embodiment of the present invention, a three-dimensional feature optimization device for visual scene understanding based on geometric guidance is provided, including: performing depth estimation and feature extraction on a to-be-processed image to obtain depth information and image features; determining a pseudo point cloud according to the depth information and the image features, and processing the pseudo point cloud using a point cloud processing network to obtain geometric prior spatial position information, wherein the geometric prior spatial position information is whether each voxel in the space is occupied by an obstacle and the category of the obstacle in the occupied voxel; performing perspective transformation on the image features to obtain three-dimensional voxel features, and extracting geometric guidance information from the three-dimensional voxel features, wherein the geometric guidance information includes: a detection frame, a speed and an angle in each voxel; fusing the geometric prior spatial position information, the geometric guidance information and the three-dimensional voxel features to obtain a fused feature; processing the fused feature using a deformable self-attention mechanism to obtain an optimized three-dimensional feature, wherein the deformable self-attention mechanism can capture information of occluded areas in the fused feature through dynamic offset. From the above description, it can be seen that in the three-dimensional feature optimization device based on geometric guidance for visual scene understanding of the present invention, the optimized three-dimensional features not only have three-dimensional voxel features, but also integrate geometric prior spatial position information and geometric guidance information, thereby enhancing the model's processing ability for spatially effective position features, improving calculation efficiency, significantly improving the model's processing ability for occluded areas, and improving the modeling ability for instance objects, which can improve the reconstruction accuracy of three-dimensional scenes and alleviate the technical problem that the three-dimensional features obtained by the prior art cover a limited amount of information and have poor representation accuracy for visual scenes.

[0097] Optionally, the depth estimation and feature extraction unit is further used to: use a pre-trained depth estimation network to perform depth estimation on the image to be processed to obtain depth information; use a feature extraction network to perform feature extraction on the image to be processed to obtain image features.

[0098] Optionally, the determination and processing unit is further used to: perform interpolation mapping on the image features to obtain features of corresponding positions in the three-dimensional space; and determine the pseudo point cloud according to the features of the corresponding positions in the three-dimensional space and the depth information.

[0099] Optionally, the view transform and extraction unit is further used to: extract geometric guidance information from the three-dimensional voxel features through an auxiliary head.

[0100] Optionally, the fusion unit is further used to: stack and convolve the geometric guidance information and the three-dimensional voxel features to obtain an initial fused feature; and perform secondary fusion of the initial fused feature with the geometric prior spatial position information to obtain a fused feature.

[0101] Optionally, the processing unit is further configured to: use a deformable self-attention calculation formula

[0102] The fused features are processed to obtain optimized three-dimensional features, where Attention(q, X) represents the optimized three-dimensional features. represents the attention weight of the sampling point, X represents the fusion feature, and p q represents the location of the query point, represents the learnable offset, represents the bias weight, M represents the number of attention heads, and K represents the number of sampling points for each head.

[0103] Optionally, the device is also used to: perform semantic occupancy prediction on the optimized three-dimensional features to obtain a semantic occupancy prediction result.

[0104] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0105] like Figure 6 As shown, an electronic device 600 provided in an embodiment of the present application includes: a processor 601, a memory 602 and a bus, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 through the bus, and the processor 601 executes the machine-readable instructions to perform the steps of the three-dimensional feature optimization method for visual scene understanding based on geometric guidance as described above.

[0106] Specifically, the above-mentioned memory 602 and processor 601 can be general-purpose memory and processor, which are not specifically limited here. When the processor 601 runs the computer program stored in the memory 602, it can execute the above-mentioned three-dimensional feature optimization method based on geometric guidance for visual scene understanding.

[0107] The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 601. The above processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 602, and the processor 601 reads the information in the memory 602 and completes the steps of the above method in combination with its hardware.

[0108] Corresponding to the above-mentioned three-dimensional feature optimization method based on geometric guidance for visual scene understanding, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the above-mentioned three-dimensional feature optimization method based on geometric guidance for visual scene understanding.

[0109] The three-dimensional feature optimization device for visual scene understanding based on geometric guidance provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiment of the present application, its implementation principle and the technical effect produced are the same as those in the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. It can be clearly understood by technicians in the relevant field that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0110] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0111] For another example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0114] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the three-dimensional feature optimization method based on geometric guidance for visual scene understanding described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0115] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0116] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solution of the present application, rather than to limit it. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solution recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiment of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A three-dimensional feature optimization method for visual scene understanding based on geometry guidance, characterized in that: include: Perform depth estimation and feature extraction on the image to be processed to obtain depth information and image features; Determine a pseudo point cloud according to the depth information and the image features, and process the pseudo point cloud using a point cloud processing network to obtain geometric a priori spatial position information, wherein the geometric a priori spatial position information is whether each voxel in the space is occupied by an obstacle and the category of the obstacle in the occupied voxel; Performing a perspective transformation on the image features to obtain three-dimensional voxel features, and extracting geometric guidance information from the three-dimensional voxel features, wherein the geometric guidance information includes: a detection frame, a speed, and an angle in each voxel; Fusing the geometric prior spatial position information, the geometric guidance information and the three-dimensional voxel feature to obtain a fused feature; The fused features are processed using a deformable self-attention mechanism to obtain optimized three-dimensional features, wherein the deformable self-attention mechanism can capture information of occluded areas in the fused features through dynamic offset.

2. The method according to claim 1, characterized in that Depth estimation and feature extraction of the image to be processed, including: Using a pre-trained depth estimation network to perform depth estimation on the image to be processed to obtain the depth information; A feature extraction network is used to extract features of the image to be processed to obtain the image features.

3. The method according to claim 1, characterized in that Determining a pseudo point cloud according to the depth information and the image features includes: Performing interpolation mapping on the image features to obtain features of corresponding positions in three-dimensional space; The pseudo point cloud is determined according to the features of the corresponding position in the three-dimensional space and the depth information.

4. The method according to claim 1, characterized in that Extracting geometric guidance information from the three-dimensional voxel features includes: The geometric guidance information is extracted from the three-dimensional voxel features by an auxiliary head.

5. The method according to claim 1, characterized in that The geometric prior spatial position information, the geometric guidance information and the three-dimensional voxel feature are integrated, including: The geometric guidance information and the three-dimensional voxel features are stacked and convolved to obtain an initial fusion feature; The initial fusion feature is fused twice with the spatial position information of the geometric prior to obtain the fusion feature.

6. The method according to claim 1, characterized in that The fusion features are processed using a deformable self-attention mechanism, including: Adopting deformable self-attention calculation formula The fused features are processed to obtain optimized three-dimensional features, wherein Attention(q, X) represents the optimized three-dimensional features, represents the attention weight of the sampling point, X represents the fusion feature, p q represents the location of the query point, represents the learnable offset, represents the bias weight, X represents the number of attention heads, and K represents the number of sampling points for each head.

7. The method according to claim 1, characterized in that The method further comprises: Perform semantic occupancy prediction on the optimized three-dimensional features to obtain a semantic occupancy prediction result.

8. A three-dimensional feature optimization device for visual scene understanding based on geometric guidance, characterized in that: include: A depth estimation and feature extraction unit, used to perform depth estimation and feature extraction on the image to be processed to obtain depth information and image features; a determination and processing unit, configured to determine a pseudo point cloud according to the depth information and the image features, and process the pseudo point cloud using a point cloud processing network to obtain geometric a priori spatial position information, wherein the geometric a priori spatial position information is whether each voxel in the space is occupied by an obstacle and the category of the obstacle in the occupied voxel; A perspective transformation and extraction unit, configured to transform the image features to obtain three-dimensional voxel features, and extract geometric guidance information from the three-dimensional voxel features, wherein the geometric guidance information includes: a detection frame, a speed, and an angle in each voxel; A fusion unit, used for fusing the geometric prior spatial position information, the geometric guidance information and the three-dimensional voxel feature to obtain a fusion feature; A processing unit is used to process the fused features using a deformable self-attention mechanism to obtain optimized three-dimensional features, wherein the deformable self-attention mechanism can capture information of occluded areas in the fused features through dynamic offset.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to execute the method according to any one of claims 1 to 7.

Images