Method for enabling intelligent robot with body to look around aerial view and storage medium
By employing a dual-parameter convolutional residual module and a CNN-structured visual projection transformation in BEV technology, the problems of high computational cost and inability to deploy transformers on edge devices are solved, enabling fast and generalized BEV information generation.
Patent Information
- Application Number
- CN202512009045.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-29
- Publication Date
- 2026-04-07
AI Technical Summary
Existing BEV technology relies on transformer technology, which results in high computational costs, makes it difficult to deploy effectively on edge devices, and transformers do not run well on edge devices.
Five levels of features are constructed by cascading dual-parameter convolutional residual modules. BEV information is generated through visual projection transformation. A 3D position encoder for visual projection transformation is designed using a CNN-structured BEV network, which avoids the encoding computation of transformer.
It enables rapid and generalized BEV information generation on edge devices, reducing computational requirements and improving model deployment efficiency.
Smart Images
Figure CN121810480A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of bird's-eye view technology, specifically to a method and storage medium for creating a bird's-eye view image of an embodied intelligent robot based on a position encoder constructed from visual projection transformation. Background Technology
[0002] BEV is an abbreviation for bird's-eye view, representing a surround-view perception capability. This technology plays an important role in embodied intelligence because it can effectively eliminate occlusion problems and scale inconsistencies in single-camera imaging.
[0003] Currently, most BEV implementations utilize transformer technology to simultaneously encode multiple cameras and project them onto a bird's-eye view coordinate system, such as the well-known BEVFusion. Without the introduction of radar, a large number of transformers are often needed to learn the corresponding features of the images.
[0004] However, this scheme has the following drawbacks:
[0005] Disadvantage 1: Transformers often require a lot of computation, which means that traditional BEVs often need to be deployed on edge devices with large computing power.
[0006] Disadvantage 2: As of now, due to its multi-head attention computational characteristics, the transformer still cannot run well on edge devices. Summary of the Invention
[0007] In view of the above problems, this application provides a method and storage medium for embodied intelligent robot surround view bird's-eye view, which solves the problems that existing BEVs rely on transformer technology and that traditional BEVs often need to be deployed on edge devices with large computing power and still cannot run well on edge devices.
[0008] To achieve the above objectives, the inventors provide a method for creating a surround-view bird's-eye view for an embodied intelligent robot, comprising:
[0009] Five levels of features, F1 to F5, are constructed by concatenating dual-parameter convolutional residual modules.
[0010] Features F2, F3, and F4 are each concatenated into a dual-parameter convolutional residual module, and the number of channels is controlled to be the same as the number of channels in Fa4, resulting in new features Fa2, Fa3, and Fa4;
[0011] Upsample feature F5 to the size of Fa2, Fa3, and Fa4, and convert its accessibility to 1. Multiply it with Fa2, Fa3, and Fa4 as the high-dimensional attention to obtain features A2, A3, and A4.
[0012] The position encoder reads features A4, A3, and A2 respectively, reads the intrinsic parameter matrices of n cameras and the extrinsic parameter matrices relative to the subject, and obtains the corresponding BEV information through visual projection transformation;
[0013] Two dual-parameter convolutional residual modules are used to extract features from the results generated by the position encoder, and corresponding object information and 3D position information are generated based on the extracted feature results.
[0014] The corresponding object information and 3D position information are compared with the true value to calculate the loss, and backpropagation is performed through the reparameterized convolution before the reparameterization.
[0015] By reparameterizing all the reparameterized convolutions, we obtain the final model that generates BEV features.
[0016] In some embodiments, the deployment process of the dual-parameter convolutional residual module is as follows:
[0017] Read the input F from the previous module, and use convolution to transform the dimensions of F to obtain F0;
[0018] F0 is fed in parallel to n 3x3 convolutional blocks and a normalization module to obtain F1~Fn. F0 is fed in parallel to n 3x3 convolutional blocks and a normalization module to obtain F1~Fn. F1~Fn are then added together to obtain Ftrain_out.
[0019] During training, Ftrain_out is directly backpropagated;
[0020] During deployment, n 3x3 convolutional blocks and the normalization module are reparameterized to obtain reparameterized convolutions;
[0021] Two heavily parameterized convolutions are concatenated, and the input is added to the output of the two-parameter convolution through a residual connection to obtain a two-parameter convolution residual module.
[0022] In some embodiments, reparameterizing n 3x3 convolutional blocks and the normalization module to obtain a reparameterized convolution specifically includes the following steps:
[0023] The n 3x3 convolutional blocks and normalization modules are reparameterized and simplified into one 3x3 convolutional block and normalization module according to the mathematical formula, thus obtaining the reparameterized convolution.
[0024] In some embodiments, obtaining the corresponding BEV information through visual projection transformation specifically includes the following steps:
[0025] The orientation of each pixel is calculated using an intrinsic parameter matrix, and the distance of the pixel relative to the camera is represented by a distance feature.
[0026] By using the extrinsic parameter matrix, the corresponding distance and semantic information are projected onto a plane centered on the subject to obtain the corresponding BEV information.
[0027] In some embodiments, generating corresponding object information and 3D position information based on the extracted feature results specifically includes the following steps:
[0028] The BEVFusion 3DHeader generates corresponding object information and 3D position information from the feature results.
[0029] Another technical solution is also provided: a storage medium storing a computer program, which, when executed by a processor, performs the following steps:
[0030] Five levels of features, F1 to F5, are constructed by concatenating dual-parameter convolutional residual modules.
[0031] Features F2, F3, and F4 are each concatenated into a dual-parameter convolutional residual module, and the number of channels is controlled to be the same as the number of channels in Fa4, resulting in new features Fa2, Fa3, and Fa4;
[0032] Upsample feature F5 to the size of Fa2, Fa3, and Fa4, and convert its accessibility to 1. Multiply it with Fa2, Fa3, and Fa4 as the high-dimensional attention to obtain features A2, A3, and A4.
[0033] The position encoder reads features A4, A3, and A2 respectively, reads the intrinsic parameter matrices of n cameras and the extrinsic parameter matrices relative to the subject, and obtains the corresponding BEV information through visual projection transformation;
[0034] Two dual-parameter convolutional residual modules are used to extract features from the results generated by the position encoder, and corresponding object information and 3D position information are generated based on the extracted feature results.
[0035] The corresponding object information and 3D position information are compared with the true value to calculate the loss, and backpropagation is performed through the reparameterized convolution before the reparameterization.
[0036] By reparameterizing all the reparameterized convolutions, we obtain the final model that generates BEV features.
[0037] In some embodiments, the deployment process of the dual-parameter convolutional residual module is as follows:
[0038] Read the input F from the previous module, and use convolution to transform the dimensions of F to obtain F0;
[0039] F0 is fed in parallel to n 3x3 convolutional blocks and a normalization module to obtain F1~Fn. F0 is fed in parallel to n 3x3 convolutional blocks and a normalization module to obtain F1~Fn. F1~Fn are then added together to obtain Ftrain_out.
[0040] During training, Ftrain_out is directly backpropagated;
[0041] During deployment, n 3x3 convolutional blocks and the normalization module are reparameterized to obtain reparameterized convolutions;
[0042] Two heavily parameterized convolutions are concatenated, and the input is added to the output of the two-parameter convolution through a residual connection to obtain a two-parameter convolution residual module.
[0043] In some embodiments, reparameterizing n 3x3 convolutional blocks and the normalization module to obtain a reparameterized convolution specifically includes the following steps:
[0044] The n 3x3 convolutional blocks and normalization modules are reparameterized and simplified into one 3x3 convolutional block and normalization module according to the mathematical formula, thus obtaining the reparameterized convolution.
[0045] In some embodiments, obtaining the corresponding BEV information through visual projection transformation specifically includes the following steps:
[0046] The orientation of each pixel is calculated using an intrinsic parameter matrix, and the distance of the pixel relative to the camera is represented by a distance feature.
[0047] By using the extrinsic parameter matrix, the corresponding distance and semantic information are projected onto a plane centered on the subject to obtain the corresponding BEV information.
[0048] In some embodiments, generating corresponding object information and 3D position information based on the extracted feature results specifically includes the following steps:
[0049] The BEVFusion 3DHeader generates corresponding object information and 3D position information from the feature results.
[0050] Unlike existing technologies, the above technical solution uses a series of dual-parameter convolutional residual modules as the backbone of the network. The five-level features F1 to F5 are constructed by using the series of dual-parameter convolutional residual modules, and features F2, F3, and F4 are then seriesed with another dual-parameter convolutional residual module. The number of channels is uniformly controlled to the number of channels of feature Fa4, resulting in new features Fa2, Fa3, and Fa4. Feature F5 is upsampled to the size of features Fa2, Fa3, and Fa4 respectively, and its accessibility is converted to 1. It is then multiplied with features Fa2, Fa3, and Fa4 as high-dimensional attention to obtain features A2, A3, and A4. The position encoder sequentially reads features A4, A3, and A2, and reads the intrinsic parameter matrices of n cameras and the extrinsic parameter matrices relative to the subject through features A4, A3, and A2 respectively. The corresponding BEV information is obtained by visually projecting and transforming features A4, A3, and A2. Then, two dual-parameter convolutional residual modules are used to extract features from the results generated by the position encoder, and corresponding object information and 3D position information are generated based on the extracted feature results. Next, the corresponding object information and 3D position information are compared with the true values for loss calculation. Backpropagation is performed through the reparameterized convolutions before reparameterization, and all reparameterized convolutions are reparameterized to obtain the final model that generates BEV features. Simultaneously, a BEV network with a CNN structure is used, and a 3D position encoder with visual projection transformation is designed, providing a new design idea for BEV networks. It does not require encoding calculation through transformer technology, and since there is no transformer, it can be deployed relatively quickly and in a generalized manner on various edge devices.
[0051] The above description of the invention is merely an overview of the technical solution of this application. In order to enable those skilled in the art to better understand the technical solution of this application and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of this application easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of this application. Attached Figure Description
[0052] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of this application and other related content, and should not be considered as limitations on this application.
[0053] In the accompanying drawings of the instruction manual:
[0054] Figure 1 This is a schematic flowchart illustrating the surround-view bird's-eye view method for embodied intelligent robots described in a specific implementation.
[0055] Figure 2 A flowchart illustrating the deployment process of the dual-parameter convolutional residual module described in a specific implementation;
[0056] Figure 3 This is a schematic diagram of a process for obtaining corresponding BEV information through visual projection transformation as described in a specific implementation;
[0057] Figure 4 This is a schematic diagram of the structure of the storage medium described in a specific embodiment.
[0058] The reference numerals used in the above figures are explained as follows:
[0059] 410. Storage medium,
[0060] 420. Processor. Detailed Implementation
[0061] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.
[0062] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.
[0063] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.
[0064] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.
[0065] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.
[0066] Unless otherwise specified, the use of terms such as “comprising,” “including,” “having,” or other similar expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.
[0067] As understood in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments in this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.
[0068] In the description of the embodiments of this application, the space-related expressions used, such as "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "vertical," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," indicate the orientation or positional relationship based on the orientation or positional relationship shown in the specific embodiments or drawings. They are only for the purpose of describing the specific embodiments of this application or for the reader's understanding, and do not indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.
[0069] Unless otherwise expressly specified or limited, the terms "installation," "connection," "linking," "fixing," and "setting," as used in the description of the embodiments of this application, should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two components or the interaction between two components. For those skilled in the art to which this application pertains, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.
[0070] Please see Figure 1 This embodiment provides a method for creating a surround-view bird's-eye view of an embodied intelligent robot, including:
[0071] Step S110: Construct 5-level features F1~F5 by cascading dual-parameter convolutional residual modules;
[0072] Step S120: Concatenate features F2, F3, and F4 into a dual-parameter convolutional residual module, and control the number of channels to be the same as the number of channels in Fa4, to obtain new features Fa2, Fa3, and Fa4;
[0073] Step S130: Upsample feature F5 to the size of Fa2, Fa3, and Fa4, and convert its accessibility to 1. Multiply it with Fa2, Fa3, and Fa4 as high-dimensional attention to obtain features A2, A3, and A4.
[0074] Step S140: The position encoder reads features A4, A3 and A2 respectively, reads the intrinsic parameter matrices of n cameras and the extrinsic parameter matrices relative to the subject, and obtains the corresponding BEV information through visual projection transformation;
[0075] Step S150: Use two dual-parameter convolutional residual modules to extract features from the results generated by the position encoder, and generate corresponding object information and 3D position information based on the extracted feature results;
[0076] Step S160: Calculate the loss by comparing the corresponding object information and 3D position information with the true value, and perform backpropagation operation by reparameter convolution before reparameter ...
[0077] Step S170: Reparameterize all the reparameterized convolutions to obtain the final model that generates BEV features.
[0078] By using a cascaded dual-parameter convolutional residual module as the backbone of the network, the 5-level features F1~F5 constructed by the cascaded dual-parameter convolutional residual module are used to cascade features F2, F3, and F4, which are then cascaded again by a dual-parameter convolutional residual module. The number of channels is uniformly controlled to the number of channels of feature Fa4, resulting in new features Fa2, Fa3, and Fa4. Feature F5 is upsampled to the size of features Fa2, Fa3, and Fa4 respectively, and its accessibility is converted to 1. It is then multiplied with features Fa2, Fa3, and Fa4 as high-dimensional attention to obtain features A2, A3, and A4. The position encoder sequentially reads features A4, A3, and A2, and reads the intrinsic parameter matrices of n cameras and the extrinsic parameter matrices relative to the subject through features A4, A3, and A2 respectively. The corresponding BEV information is obtained by visually projecting and transforming features A4, A3, and A2. Then, two dual-parameter convolutional residual modules are used to extract features from the results generated by the position encoder, and corresponding object information and 3D position information are generated based on the extracted feature results. Next, the corresponding object information and 3D position information are compared with the true values for loss calculation. Backpropagation is performed through the reparameterized convolutions before reparameterization, and all reparameterized convolutions are reparameterized to obtain the final model that generates BEV features. Simultaneously, a BEV network with a CNN structure is used, and a 3D position encoder with visual projection transformation is designed, providing a new design idea for BEV networks. It does not require encoding calculation through transformer technology, and since there is no transformer, it can be deployed relatively quickly and in a generalized manner on various edge devices.
[0079] Please see Figure 2 In some embodiments, the deployment process of the dual-parameter convolutional residual module is as follows:
[0080] Step S210: Read the input F from the previous module, and use convolution to transform the dimensions of F to obtain F0; prepare for subsequent multi-branch convolution.
[0081] Step S220: Connect F0 in parallel to n 3x3 convolutional blocks and normalization modules to obtain F1~Fn. Connect F0 in parallel to n 3x3 convolutional blocks and normalization modules to obtain F1~Fn. Then add F1~Fn together to obtain Ftrain_out.
[0082] Step S230: During training, directly backpropagate Ftrain_out;
[0083] Step S240: During deployment, the n 3x3 convolutional blocks and the normalization module are reparameterized to obtain reparameterized convolutions;
[0084] Step S250: Concatenate two heavily parameterized convolutions, and add the input to the output of the dual parameterized convolution through a residual connection to obtain a dual parameterized convolution residual module.
[0085] A cascaded dual-parameter convolutional residual module is used as the backbone of the network, as detailed below:
[0086] Read the input F of the previous module. Perform a dimensionality transformation on F using convolution to obtain F0. The previous module refers to the direct preceding network component of the current dual-parameter convolutional residual module. If the current module is the first backbone module of the network, the previous module is usually the network's input layer / feature preprocessing module. If the current module is a backbone module in the middle of the network, the previous module is another dual-parameter convolutional residual module or a backbone residual module of the same type that precedes it in the network.
[0087] F0 is fed into n 3x3 convolutional blocks and a normalization module in parallel to obtain F1~Fn, and F1~Fn are added together to obtain Ftrain_out.
[0088] During training, Ftrain_out is directly backpropagated;
[0089] During deployment, n 3x3 convolutional blocks and normalization modules are reparameterized. Specifically, based on a mathematical formula, each block is reduced to a single 3x3 convolutional block and normalization module, simplifying the structure, improving inference speed and memory utilization, and thus enhancing model deployment capabilities. This part is known as reparameterized convolution.
[0090] Two concatenated double-parameter convolutions are connected in series, and a residual implementation is used to add the input to the output module of the double-parameter convolution. This yields the double-parameter convolution residual module.
[0091] Please see Figure 3 In some embodiments, obtaining the corresponding BEV information through visual projection transformation specifically includes the following steps:
[0092] Step S310: Calculate the orientation of each pixel using the intrinsic parameter matrix, and represent the distance of the pixel relative to the camera using distance features;
[0093] Step S320: Project the corresponding distance information and semantic information onto a plane centered on the subject using the extrinsic parameter matrix to obtain the corresponding BEV information.
[0094] Design a new visual projection transformation module (C2V) as a feature encoder for bird's-eye view projection. Details are as follows:
[0095] A. Read A4 and convert its channel count to twice the number of cameras n, representing semantic features and distance features respectively.
[0096] B. Read the intrinsic parameter matrix K and the relative extrinsic parameter matrix R|T of each of the n cameras.
[0097] C. Calculate the orientation of each pixel using the intrinsic parameter K, and use the distance feature to represent the distance of that pixel relative to the camera.
[0098] D. Project the corresponding distance and semantic information onto a plane centered on the subject using the extrinsic parameter matrix, and use this plane as the BEV information A4-BEV of the A4 feature.
[0099] E. Repeat AD to project the features from A4 to A2 to obtain BEV information A4-BEV, BEV information A3-BEV, and BEV information A2-BEV. Then perform channel connection operations.
[0100] In some embodiments, generating corresponding object information and 3D position information based on the extracted feature results specifically includes the following steps:
[0101] The BEVFusion 3DHeader generates corresponding object information and 3D position information from the feature results.
[0102] BEVFusion is a classic multimodal (camera + LiDAR) 3D detection framework. Its core is to fuse features from both the camera and LiDAR into a bird's-eye view (BEV), and then use the 3DHeader module to complete the final 3D detection and segmentation tasks. The 3DHeader is a key component in BEVFusion that receives the BEV fused features and outputs the 3D perception task results. Essentially, it's a combination of feature extraction for BEV features and a task head (detection / segmentation head), responsible for mapping the high-dimensional BEV fused features into 3D bounding boxes, categories, semantic segmentation masks, and other final outputs.
[0103] Please see Figure 4 In another embodiment, a storage medium 410 stores a computer program, which is executed by a processor 420 to perform the following steps:
[0104] Five levels of features, F1 to F5, are constructed by concatenating dual-parameter convolutional residual modules.
[0105] Features F2, F3, and F4 are each concatenated into a dual-parameter convolutional residual module, and the number of channels is controlled to be the same as the number of channels in Fa4, resulting in new features Fa2, Fa3, and Fa4;
[0106] Upsample feature F5 to the size of Fa2, Fa3, and Fa4, and convert its accessibility to 1. Multiply it with Fa2, Fa3, and Fa4 as the high-dimensional attention to obtain features A2, A3, and A4.
[0107] The position encoder reads features A4, A3, and A2 respectively, reads the intrinsic parameter matrices of n cameras and the extrinsic parameter matrices relative to the subject, and obtains the corresponding BEV information through visual projection transformation;
[0108] Two dual-parameter convolutional residual modules are used to extract features from the results generated by the position encoder, and corresponding object information and 3D position information are generated based on the extracted feature results.
[0109] The corresponding object information and 3D position information are compared with the true value to calculate the loss, and backpropagation is performed through the reparameterized convolution before the reparameterization.
[0110] By reparameterizing all the reparameterized convolutions, we obtain the final model that generates BEV features.
[0111] By using a cascaded dual-parameter convolutional residual module as the backbone of the network, the 5-level features F1~F5 constructed by the cascaded dual-parameter convolutional residual module are used to cascade features F2, F3, and F4, which are then cascaded again by a dual-parameter convolutional residual module. The number of channels is uniformly controlled to the number of channels of feature Fa4, resulting in new features Fa2, Fa3, and Fa4. Feature F5 is upsampled to the size of features Fa2, Fa3, and Fa4 respectively, and its accessibility is converted to 1. It is then multiplied with features Fa2, Fa3, and Fa4 as high-dimensional attention to obtain features A2, A3, and A4. The position encoder sequentially reads features A4, A3, and A2, and reads the intrinsic parameter matrices of n cameras and the extrinsic parameter matrices relative to the subject through features A4, A3, and A2 respectively. The corresponding BEV information is obtained by visually projecting and transforming features A4, A3, and A2. Then, two dual-parameter convolutional residual modules are used to extract features from the results generated by the position encoder, and corresponding object information and 3D position information are generated based on the extracted feature results. Next, the corresponding object information and 3D position information are compared with the true values for loss calculation. Backpropagation is performed through the reparameterized convolutions before reparameterization, and all reparameterized convolutions are reparameterized to obtain the final model that generates BEV features. Simultaneously, a BEV network with a CNN structure is used, and a 3D position encoder with visual projection transformation is designed, providing a new design idea for BEV networks. It does not require encoding calculation through transformer technology, and since there is no transformer, it can be deployed relatively quickly and in a generalized manner on various edge devices.
[0112] In some embodiments, the deployment process of the dual-parameter convolutional residual module is as follows:
[0113] Read the input F from the previous module, and use convolution to transform the dimensions of F to obtain F0; this prepares for subsequent multi-branch convolution.
[0114] F0 is fed in parallel to n 3x3 convolutional blocks and a normalization module to obtain F1~Fn. F0 is fed in parallel to n 3x3 convolutional blocks and a normalization module to obtain F1~Fn. F1~Fn are then added together to obtain Ftrain_out.
[0115] During training, Ftrain_out is directly backpropagated;
[0116] During deployment, n 3x3 convolutional blocks and the normalization module are reparameterized to obtain reparameterized convolutions;
[0117] Two heavily parameterized convolutions are concatenated, and the input is added to the output of the two-parameter convolution through a residual connection to obtain a two-parameter convolution residual module.
[0118] A cascaded dual-parameter convolutional residual module is used as the backbone of the network, as detailed below:
[0119] Read the input F of the previous module. Perform a dimensionality transformation on F using convolution to obtain F0. The previous module refers to the direct preceding network component of the current dual-parameter convolutional residual module. If the current module is the first backbone module of the network, the previous module is usually the network's input layer / feature preprocessing module. If the current module is a backbone module in the middle of the network, the previous module is another dual-parameter convolutional residual module or a backbone residual module of the same type that precedes it in the network.
[0120] F0 is fed into n 3x3 convolutional blocks and a normalization module in parallel to obtain F1~Fn, and F1~Fn are added together to obtain Ftrain_out.
[0121] During training, Ftrain_out is directly backpropagated;
[0122] During deployment, n 3x3 convolutional blocks and normalization modules are reparameterized. Specifically, based on a mathematical formula, each block is reduced to a single 3x3 convolutional block and normalization module, simplifying the structure, improving inference speed and memory utilization, and thus enhancing model deployment capabilities. This part is known as reparameterized convolution.
[0123] Two concatenated double-parameter convolutions are connected in series, and a residual implementation is used to add the input to the output module of the double-parameter convolution. This yields the double-parameter convolution residual module.
[0124] In some embodiments, obtaining the corresponding BEV information through visual projection transformation specifically includes the following steps:
[0125] The orientation of each pixel is calculated using an intrinsic parameter matrix, and the distance of the pixel relative to the camera is represented by a distance feature.
[0126] By using the extrinsic parameter matrix, the corresponding distance and semantic information are projected onto a plane centered on the subject to obtain the corresponding BEV information.
[0127] Design a new visual projection transformation module (C2V) as a feature encoder for bird's-eye view projection. Details are as follows:
[0128] A. Read A4 and convert its channel count to twice the number of cameras n, representing semantic features and distance features respectively.
[0129] B. Read the intrinsic parameter matrix K and the relative extrinsic parameter matrix R|T of each of the n cameras.
[0130] C. Calculate the orientation of each pixel using the intrinsic parameter K, and use the distance feature to represent the distance of that pixel relative to the camera.
[0131] D. Project the corresponding distance and semantic information onto a plane centered on the subject using the extrinsic parameter matrix, and use this plane as the BEV information A4-BEV of the A4 feature.
[0132] E. Repeat AD to project the features from A4 to A2 to obtain BEV information A4-BEV, BEV information A3-BEV, and BEV information A2-BEV. Then perform channel connection operations.
[0133] In some embodiments, generating corresponding object information and 3D position information based on the extracted feature results specifically includes the following steps:
[0134] The BEVFusion 3DHeader generates corresponding object information and 3D position information from the feature results.
[0135] BEVFusion is a classic multimodal (camera + LiDAR) 3D detection framework. Its core is to fuse features from both the camera and LiDAR into a bird's-eye view (BEV), and then use the 3DHeader module to complete the final 3D detection and segmentation tasks. The 3DHeader is a key component in BEVFusion that receives the BEV fused features and outputs the 3D perception task results. Essentially, it's a combination of feature extraction for BEV features and a task head (detection / segmentation head), responsible for mapping the high-dimensional BEV fused features into 3D bounding boxes, categories, semantic segmentation masks, and other final outputs.
[0136] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.
Claims
1. A method for creating a panoramic bird's-eye view image for an embodied intelligent robot, characterized in that, include: Five levels of features, F1 to F5, are constructed by concatenating dual-parameter convolutional residual modules. Features F2, F3, and F4 are each concatenated into a dual-parameter convolutional residual module, and the number of channels is controlled to be the same as the number of channels in Fa4, resulting in new features Fa2, Fa3, and Fa4; Upsample feature F5 to the size of Fa2, Fa3, and Fa4, and convert its accessibility to 1. Multiply it with Fa2, Fa3, and Fa4 as the high-dimensional attention to obtain features A2, A3, and A4. The position encoder reads features A4, A3, and A2 respectively, reads the intrinsic parameter matrices of n cameras and the extrinsic parameter matrices relative to the subject, and obtains the corresponding BEV information through visual projection transformation; Two dual-parameter convolutional residual modules are used to extract features from the results generated by the position encoder, and corresponding object information and 3D position information are generated based on the extracted feature results. The corresponding object information and 3D position information are compared with the true value to calculate the loss, and backpropagation is performed through the reparameterized convolution before the reparameterization. By reparameterizing all the reparameterized convolutions, we obtain the final model that generates BEV features.
2. The method for obtaining a panoramic bird's-eye view of an embodied intelligent robot according to claim 1, characterized in that, The deployment process of the dual-parameter convolutional residual module is as follows: Read the input F from the previous module, and use convolution to transform the dimensions of F to obtain F0; F0 is fed in parallel to n 3x3 convolutional blocks and a normalization module to obtain F1~Fn. F0 is fed in parallel to n 3x3 convolutional blocks and a normalization module to obtain F1~Fn. F1~Fn are then added together to obtain Ftrain_out. During training, Ftrain_out is directly backpropagated; During deployment, n 3x3 convolutional blocks and normalization modules are reparameterized and simplified into one 3x3 convolutional block and normalization module according to mathematical formulas, resulting in reparameterized convolution. Two heavily parameterized convolutions are concatenated, and the input is added to the output of the two-parameter convolution through a residual connection to obtain a two-parameter convolution residual module.
3. The method for obtaining a bird's-eye view of an embodied intelligent robot according to claim 2, characterized in that, The process of reparameterizing n 3x3 convolutional blocks and the normalization module to obtain reparameterized convolutions specifically includes the following steps: The n 3x3 convolutional blocks and normalization modules are reparameterized and simplified into one 3x3 convolutional block and normalization module according to the mathematical formula, thus obtaining the reparameterized convolution.
4. The method for obtaining a panoramic bird's-eye view of an embodied intelligent robot according to claim 1, characterized in that, The process of obtaining the corresponding BEV information through visual projection transformation specifically includes the following steps: The orientation of each pixel is calculated using an intrinsic parameter matrix, and the distance of the pixel relative to the camera is represented by a distance feature. By using the extrinsic parameter matrix, the corresponding distance and semantic information are projected onto a plane centered on the subject to obtain the corresponding BEV information.
5. The method for obtaining a panoramic bird's-eye view of an embodied intelligent robot according to claim 1, characterized in that, The process of generating corresponding object information and 3D position information based on the extracted feature results specifically includes the following steps: The BEVFusion 3DHeader generates corresponding object information and 3D position information from the feature results.
6. A storage medium storing a computer program, characterized in that, The computer program, when executed by the processor, performs the following steps: Five levels of features, F1 to F5, are constructed by concatenating dual-parameter convolutional residual modules. Features F2, F3, and F4 are each concatenated into a dual-parameter convolutional residual module, and the number of channels is controlled to be the same as the number of channels in Fa4, resulting in new features Fa2, Fa3, and Fa4; Upsample feature F5 to the size of Fa2, Fa3, and Fa4, and convert its accessibility to 1. Multiply it with Fa2, Fa3, and Fa4 as the high-dimensional attention to obtain features A2, A3, and A4. The position encoder reads features A4, A3, and A2 respectively, reads the intrinsic parameter matrices of n cameras and the extrinsic parameter matrices relative to the subject, and obtains the corresponding BEV information through visual projection transformation; Two dual-parameter convolutional residual modules are used to extract features from the results generated by the position encoder, and corresponding object information and 3D position information are generated based on the extracted feature results. The corresponding object information and 3D position information are compared with the true value to calculate the loss, and backpropagation is performed through the reparameterized convolution before the reparameterization. By reparameterizing all the reparameterized convolutions, we obtain the final model that generates BEV features.
7. The storage medium according to claim 6, characterized in that, The deployment process of the dual-parameter convolutional residual module is as follows: Read the input F from the previous module, and use convolution to transform the dimensions of F to obtain F0; F0 is fed in parallel to n 3x3 convolutional blocks and a normalization module to obtain F1~Fn. F0 is fed in parallel to n 3x3 convolutional blocks and a normalization module to obtain F1~Fn. F1~Fn are then added together to obtain Ftrain_out. During training, Ftrain_out is directly backpropagated; During deployment, n 3x3 convolutional blocks and normalization modules are reparameterized and simplified into one 3x3 convolutional block and normalization module according to mathematical formulas, resulting in reparameterized convolution. Two heavily parameterized convolutions are concatenated, and the input is added to the output of the two-parameter convolution through a residual connection to obtain a two-parameter convolution residual module.
8. The storage medium according to claim 7, characterized in that, The process of reparameterizing n 3x3 convolutional blocks and the normalization module to obtain reparameterized convolutions specifically includes the following steps: The n 3x3 convolutional blocks and normalization modules are reparameterized and simplified into one 3x3 convolutional block and normalization module according to the mathematical formula, thus obtaining the reparameterized convolution.
9. The storage medium according to claim 6, characterized in that, The process of obtaining the corresponding BEV information through visual projection transformation specifically includes the following steps: The orientation of each pixel is calculated using an intrinsic parameter matrix, and the distance of the pixel relative to the camera is represented by a distance feature. By using the extrinsic parameter matrix, the corresponding distance and semantic information are projected onto a plane centered on the subject to obtain the corresponding BEV information.
10. The storage medium according to claim 6, characterized in that, The process of generating corresponding object information and 3D position information based on the extracted feature results specifically includes the following steps: The BEVFusion 3DHeader generates corresponding object information and 3D position information from the feature results.