Target detection method combining vision and lidar applied to embodied intelligent agents

Abstract

This application provides a target detection method that fuses vision and LiDAR for embodied intelligent agents. The method includes: acquiring multi-view image data and LiDAR point cloud data of a target scene; extracting visual features from the multi-view image data to obtain multi-scale visual features; extracting LiDAR features from the LiDAR point cloud data to obtain LiDAR bird's-eye view features; updating all initial instances according to the multi-scale visual features to obtain multiple candidate instances, where each candidate instance includes corresponding candidate anchor points and candidate instance features; fusing the LiDAR bird's-eye view features into the candidate instance features according to the candidate anchor points to obtain fused instance features; and performing inference based on the fused instance features to determine the 3D target detection result in the target scene. This method can solve the problem of unstable 3D geometric information supplementation in related technologies.

Description

Technical Field

[0001] This application relates to the field of embodied intelligent perception technology, and in particular to a target detection method that integrates vision and lidar for use in embodied intelligent agents. Background Technology

[0002] Sparse-driven environment perception methods based on multi-view images have advantages such as unified structure, compact instance representation, and convenient end-to-end deployment, and are widely used in embodied intelligent systems such as autonomous driving systems, mobile robots, construction machinery, and delivery robots. Sparse-driven frameworks, represented by SparseDrive, typically maintain a fixed number of instance queries and anchor points in the detection head, continuously update the candidate target representation through multi-layer decoding, and can continue to provide the detection results for use by downstream modules.

[0003] However, purely visual sparse query methods still primarily rely on image reconstruction of 3D geometry, which is prone to problems such as unstable spatial boundaries, scale recovery deviations, and increased localization errors in long-distance, occluded, weakly textured, and nighttime scenes. LiDAR can directly provide high-quality 3D spatial structure information, making it suitable as an important geometric supplement to the visual branch. For embodied agents, they also need to perform actions such as movement, obstacle avoidance, target approach, grasping, or interactive tasks based on perception results in open physical environments, thus requiring higher stability in target spatial position, scale, and motion state.

[0004] Most existing LiDAR and vision fusion solutions revolve around input-level stitching, dense bird's-eye view fusion, or post-detection result fusion. The first two types of methods (i.e., input-level stitching and dense bird's-eye view fusion) usually have high computational costs and are difficult to couple naturally with the sparse instance query decoding link; the latter type of method (i.e., post-detection result fusion), although simple to implement, does not have LiDAR features directly involved in the layer-by-layer update process of candidate instances, thus providing limited enhancement to instance-level representations and making it difficult to stably supplement sparse queries with three-dimensional geometric constraints.

[0005] Furthermore, in embodied intelligence applications, agents need to make action decisions such as navigation and obstacle avoidance, grasping operations, collaborative transportation, and inspection and cleaning based on environmental perception results. If the estimated position, size, or motion state of the target is unstable, it will affect the safety and continuity of the interaction between the embodied agent and the real physical environment. Therefore, a multimodal target detection scheme that can provide stable three-dimensional spatial understanding for embodied agents is needed.

[0006] Therefore, the related technologies suffer from technical problems such as the inability to achieve stable three-dimensional geometric information supplementation and the difficulty in providing reliable spatial perception input for the action planning and environmental interaction of embodied intelligent agents. Summary of the Invention

[0007] This application provides a target detection method that integrates vision and lidar for embodied intelligent agents, in order to at least solve the technical problems in related technologies where stable three-dimensional geometric information cannot be supplemented and it is difficult to provide reliable spatial perception input for the action planning and environmental interaction of embodied intelligent agents.

[0008] According to one aspect of the embodiments of this application, a target detection method fused with vision and lidar applied to an embodied intelligent agent is provided, comprising: Acquire multi-view image data and LiDAR point cloud data of the target scene; Visual features are extracted from the multi-view image data to obtain multi-scale visual features; lidar features are extracted from the lidar point cloud data to obtain lidar bird's-eye view features. Based on the multi-scale visual features, all initial instances are initially updated to obtain multiple candidate instances, wherein each candidate instance includes: corresponding candidate anchor points and candidate instance features; According to the candidate anchor points, the features of the LiDAR bird's-eye view are fused into the features of the candidate instances to obtain the fused instance features; Based on the fused instance features, inference is performed to determine the 3D target detection results in the target scene.

[0009] Optionally, as described above, the step of initially updating all initial instances according to the multi-scale visual features to obtain multiple candidate instances includes: Using each initial anchor point as a condition, camera deformable feature aggregation is performed on the multi-scale visual features to obtain the visual features corresponding to each initial anchor point in the multi-scale visual features, wherein the initial anchor point is the anchor point contained in the initial instance; The visual features corresponding to each initial anchor point are fused with the initial instance features corresponding to each initial anchor point to obtain preliminary fused instance features; Each initial instance is adjusted based on the preliminary fusion instance features corresponding to each initial anchor point to obtain the multiple candidate instances.

[0010] Optionally, as described above, the step of fusing the lidar bird's-eye view features to the candidate instance features according to the candidate anchor points to obtain the fused instance features includes: For each candidate instance, the feature points of the candidate anchor points in the candidate instance are used as reference positions, and conditional sampling and weighted aggregation are performed on the LiDAR bird's-eye view features based on the candidate instance features in the candidate instance to obtain the fused instance features in each candidate instance.

[0011] Optionally, as described above, for each candidate instance, using the feature points of the candidate anchor points in the candidate instance as reference positions, and performing conditional sampling and weighted aggregation on the LiDAR bird's-eye view features based on the candidate instance features in the candidate instance to obtain the fused instance features corresponding to each candidate instance, includes: performing the fusion operation N times in a loop, and using the instance features in the fusion result obtained from the last fusion operation as the fused instance features, where N is an integer greater than or equal to 2; The fusion operation includes: Temporal graph attention processing is performed on each target instance to obtain each temporally enhanced instance in which historical motion information is incorporated into the features of the target instance. In the case of the first execution of the fusion operation, the target instance is the candidate instance. In the case of the non-first execution of the fusion operation, the target instance is the fusion result obtained from the previous fusion operation. Each of the time-series enhancement instances is subjected to self-attention processing to obtain a global logic instance corresponding to each of the time-series enhancement instances; By using the multi-scale visual features and adjusting the instance features in each global logical instance according to the candidate anchor points in each global logical instance, an enhanced instance corresponding to each global logical instance is obtained after enhancing the semantic description of the appearance. Using the feature points of the candidate anchor points in each enhancement instance as reference positions, and based on the enhancement instance features in each enhancement instance, conditional sampling and weighted aggregation are performed on the LiDAR bird's-eye view features to obtain point cloud aggregation features. Based on the candidate anchor points, the enhancement instance features in each enhancement instance, and the point cloud aggregation features, a multimodal instance corresponding to each enhancement instance is obtained. Each multimodal instance is processed sequentially through an FFN network and a refinement layer to obtain the fusion result corresponding to the features of each multimodal instance.

[0012] Optionally, as described above, the step of sequentially processing each multimodal instance through an FFN network and a refinement layer to obtain the fusion result corresponding to the features of each multimodal instance includes: By using the FFN network, the enhanced instance features and the point cloud aggregation features in each multimodal instance are fused to obtain the multimodal fused instance features corresponding to each multimodal instance. The fusion result is obtained by adjusting the multimodal fusion instance features corresponding to each multimodal instance through the refinement layer.

[0013] Optionally, as described above, the step of extracting lidar features from the lidar point cloud data to obtain lidar bird's-eye view features includes: The lidar point cloud data is processed sequentially through a voxelization module, a columnar feature encoding module, a bird's-eye view scattering module, a point cloud backbone network, and a point cloud neck network to obtain the processed features. Based on the dimensions of multi-scale visual features, channel compression is performed on the processed features to obtain the LiDAR bird's-eye view features that are compatible with the multi-scale visual features.

[0014] Optionally, as described above, the method further includes: The three-dimensional target detection results are transmitted to the embodied intelligent decision-making module; The embodied intelligent decision-making module plans the planned path in the target scene and the motion control commands for the embodied intelligent agent based on the three-dimensional target detection results, wherein the embodied intelligent agent is the entity in which the embodied intelligent decision-making module is located.

[0015] According to another aspect of the embodiments of this application, a target detection device integrating vision and lidar for use in embodied intelligent agents is also provided, comprising: The acquisition module is used to acquire multi-view image data and LiDAR point cloud data of the target scene; The extraction module is used to extract visual features from the multi-view image data to obtain multi-scale visual features; and to extract lidar features from the lidar point cloud data to obtain lidar bird's-eye view features. The initial update module is used to perform an initial update on all initial instances according to the multi-scale visual features to obtain multiple candidate instances, wherein each candidate instance includes: corresponding candidate anchor points and candidate instance features; The fusion module is used to fuse the features of the LiDAR bird's-eye view into the features of the candidate instance according to the candidate anchor points, so as to obtain the fused instance features; The inference module is used to perform inference based on the fused instance features to determine the 3D target detection results in the target scene.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein the memory is used to store a computer program; and the processor is used to execute the method steps of any of the above embodiments by running the computer program stored in the memory.

[0017] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the method steps of any of the above embodiments when running.

[0018] According to another aspect of the embodiments of this application, an embodied intelligent agent is also provided, including the electronic device as described in the foregoing embodiments.

[0019] In this embodiment, multi-view image data and LiDAR point cloud data of the target scene are acquired; visual features are extracted from the multi-view image data to obtain multi-scale visual features; LiDAR features are extracted from the LiDAR point cloud data to obtain LiDAR bird's-eye view features; according to the multi-scale visual features, all initial instances are initially updated to obtain multiple candidate instances, wherein each candidate instance includes corresponding candidate anchor points and candidate instance features; in the subsequent decoding stage, the LiDAR bird's-eye view features are fused to the candidate instance features according to the candidate anchor points to obtain fused instance features; inference is performed based on the fused instance features. The method for determining the 3D target detection results in the target scene involves adding a LiDAR feature extraction branch, enabling the LiDAR point cloud to be accessed through the sparse query decoding link in the form of LiDAR bird's-eye view features. This provides stable 3D geometric information supplementation for visual perception. Furthermore, by fusing the LiDAR bird's-eye view features with the candidate instance features according to the candidate anchor points, the fused instance features are obtained. This allows the LiDAR features to participate in the fusion around the candidate anchor point positions, thereby improving the accuracy of instance-level geometric alignment while supplementing 3D geometric information. This solves the problem of unstable 3D geometric information supplementation in related technologies. In embodied intelligence applications, the above 3D target detection results can also serve as spatial perception input for intelligent agent motion planning, grasping operations, inspection and cleaning, logistics delivery, or human-computer interaction, enabling the embodied intelligent agent to generate corresponding action decisions based on the fused instance features. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the hardware environment of an optional target detection method that integrates vision and lidar applied to an embodied intelligent agent, according to an embodiment of this application. Figure 2This is a flowchart illustrating an optional target detection method based on embodied intelligent agents, which combines vision and lidar according to an embodiment of this application. Figure 3 This is a flowchart illustrating an optional target detection method that integrates vision and lidar for use in embodied intelligent agents, according to an application example of this application. Figure 4 This is a structural block diagram of an optional target detection device that integrates vision and lidar for use in an embodied intelligent agent, according to an embodiment of this application. Figure 5 This is a structural block diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] According to one aspect of the embodiments of this application, a target detection method fusion vision and lidar applied to embodied intelligent agents is provided. Optionally, in this embodiment, the above-described target detection method fusion vision and lidar applied to embodied intelligent agents can be applied to, for example... Figure 1 The hardware environment shown consists of terminal 1402 and server 1404. For example... Figure 1 As shown, server 1404 is connected to terminal 1402 via a network and can be used to provide services (such as data analysis services, application services, etc.) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 1404.

[0026] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal may not be limited to PC, mobile phone, tablet computer, etc.

[0027] The 3D target detection method of this application embodiment can be executed by a server, a terminal, or both. Alternatively, the 3D target detection method of this application embodiment can be executed by a client installed on the terminal.

[0028] Taking the target detection method of fusion vision and lidar applied to embodied intelligent agents in this embodiment, which is executed by a terminal (such as a vehicle terminal, robot terminal, or other embodied intelligent agent terminal), as an example, Figure 2 A target detection method based on fused vision and lidar applied to an embodied intelligent agent, provided in this application embodiment, includes the following steps: Step S202: Acquire multi-view image data and lidar point cloud data of the target scene.

[0029] The 3D target detection method in this embodiment can be applied to embodied intelligent agents such as vehicles, mobile robots, construction machinery, and delivery robots to obtain target detection scenarios in their environment in real time.

[0030] Specifically, during the multi-view image data acquisition process, image acquisition devices mounted on embody intelligent devices can capture images of the entire target scene from different angles and directions, including horizontal, vertical, and near / far perspectives. This comprehensively captures the appearance, texture, color features, spatial layout, and details of objects within the scene, avoiding blind spots and information gaps inherent in single-view photography. Simultaneously, LiDAR devices can perform omnidirectional scanning of the target scene, accurately acquiring the 3D coordinates, distances, and morphological structures of all objects within the scene, generating high-density, high-precision 3D point cloud data. The simultaneous acquisition and complementary use of these two types of data preserves the visual details of the scene while accurately reconstructing the 3D spatial structure, comprehensively covering all core information of the scene. Furthermore, multi-view image data provides semantic and textural information, while LiDAR point cloud data provides stable 3D spatial structural information.

[0031] Step S204: Visual features are extracted from the multi-view image data to obtain multi-scale visual features.

[0032] Specifically, after acquiring multi-view image data from the target scene, refined visual feature extraction processing is required to mine deep, effective feature information and remove invalid and redundant information, providing core feature support for subsequent instance analysis. Optionally, the multi-view image data can be preprocessed, including but not limited to: unifying image resolution and size, correcting distortion and angular deviations caused by multi-view shooting, removing noise and light and shadow interference from the shooting environment, cropping invalid background redundant areas, and balancing image color and brightness. Then, the preprocessed multi-view image data is input into a dedicated image backbone network. Relying on the feature extraction capabilities of the backbone network, the image content is analyzed layer by layer, efficiently extracting multi-scale visual coding features of different details and levels in the image, covering basic feature information such as the contours, textures, colors, and edges of scene objects. On this basis, the multi-scale coding features are integrated, mapped, and optimized through a feature pyramid network, uniformly mapping the original features of different scales and dimensions into multi-scale visual features with standard embedding dimensions.

[0033] Step S206: Extract lidar features from lidar point cloud data to obtain lidar bird's-eye view features.

[0034] Specifically, after obtaining the lidar point cloud data, lidar feature extraction can be performed to obtain the corresponding lidar bird's-eye view features. As an optional embodiment, lidar feature extraction can be performed on the lidar point cloud data to obtain the lidar bird's-eye view features using the following method: The lidar point cloud data is processed sequentially through a voxelization module, a columnar feature encoding module, a bird's-eye view scattering module, a point cloud backbone network, and a point cloud neck network to obtain the processed features. Based on the dimensions of multi-scale visual features, channel compression is performed on the processed features to obtain LiDAR bird's-eye view features that are compatible with multi-scale visual features.

[0035] When this method is deployed on an embodied intelligent agent, the lidar point cloud data can also be used to characterize the passable space around the intelligent agent, obstacle boundaries, the outline of the manipulated object, and the distance to dynamic targets, providing geometric constraints for subsequent motion control or operation interaction.

[0036] In other words, after acquiring the raw LiDAR point cloud data of the scene, it is necessary to complete refined feature processing through multiple professional modules to fully mine the three-dimensional spatial feature information of the point cloud. First, the raw LiDAR point cloud data is input into the voxelization module to complete the point cloud regularization and sparse data structuring processing, standardizing the data distribution form. Then, the columnar feature encoding module extracts the local geometric features and spatial structure information of the point cloud data from the LiDAR point cloud data. Next, the bird's-eye view scattering module converts the three-dimensional point cloud data into a two-dimensional bird's-eye view representation to meet the requirements of subsequent network processing. Afterwards, deep semantic features are mined based on the point cloud backbone network, and feature fusion, optimization, and dimensional calibration are completed in combination with the point cloud neck network to output high-precision and highly robust deep point cloud features, resulting in the processed features. To achieve cross-modal fusion of image and point cloud data, it is also necessary to use the dimensions of multi-scale visual features as a unified standard, perform channel compression and dimension adaptation on the processed features, remove redundant channel information, and unify feature dimensions. Finally, we can obtain LiDAR bird's-eye view features that are fully compatible with multi-scale visual features, providing reliable data support for subsequent dual-modal feature fusion and accurate scene analysis.

[0037] Step S208: Based on multi-scale visual features, perform preliminary updates on all initial instances to obtain multiple candidate instances. Each candidate instance includes: corresponding candidate anchor points and candidate instance features.

[0038] Specifically, after completing the LiDAR feature processing and dimensional adaptation, a sparse instance query mechanism is further constructed to provide accurate retrieval basis for cross-modal feature fusion and instance-level perception. In this embodiment, based on a pre-built instance library, a fixed number (e.g., 900) of representative initial instances (i.e., including initial instance features and corresponding initial anchor points) are adaptively output to effectively filter out core target candidate instances in the scene and discard redundant and invalid information. On this basis, the obtained initial anchor points are finely encoded to deeply mine the spatial location, boundary range, and structural association information corresponding to the anchor points, generating accurate and reliable query location prior constraints. This location prior can accurately lock the distribution range of instances in the target scene, abandoning the inefficient mode of global indiscriminate retrieval, so that subsequent visual feature aggregation operations and LiDAR feature sampling are no longer carried out in a disordered manner across the entire domain, but are strictly focused and targeted around the selected initial instances. This sparse instance query method can effectively reduce computational redundancy, enhance the correlation and relevance of instance features, and significantly improve the accuracy and efficiency of multimodal feature matching, providing stable and efficient pre-support for subsequent scene target recognition, feature fusion and accurate analysis.

[0039] Furthermore, based on the approximate position of the initial anchor point in the initial instance, the detailed information of the corresponding position of the initial anchor point is accurately extracted from the multi-scale visual features, and the initial instance is updated for the first time to obtain the multiple candidate instances.

[0040] As an optional implementation, the initial update of all initial instances based on multi-scale visual features can be achieved through the following steps to obtain multiple candidate instances: Using each initial anchor point as a condition, camera deformable feature aggregation is performed on multi-scale visual features to obtain the visual features corresponding to each initial anchor point in the multi-scale visual features. Here, the initial anchor points are the anchor points contained in the initial instance. In other words, using all initial anchor points that correspond one-to-one with the initial instance as constraints, camera deformable feature aggregation is performed on the multi-scale visual feature map to generate visual features corresponding to each initial anchor point one by one. Camera deformable aggregation can also use the initial anchor points as a reference, extracting target-related texture and color information only from the visual features near the initial anchor points to update the instance features.

[0041] The visual features corresponding to each initial anchor point are fused with the initial instance features corresponding to each initial anchor point to obtain preliminary fused instance features. In other words, the aggregated visual features corresponding to each initial anchor point are deeply fused with the inherent features of the initial instance bound to that initial anchor point through the FFN network to generate preliminary fused instance features.

[0042] Each initial instance is adjusted based on the preliminary fusion instance features corresponding to each initial anchor point, resulting in multiple candidate instances. In other words, for each initial instance, a refinement layer is used to comprehensively correct and optimize the original initial instance from dimensions such as position, shape, scale, and confidence, based on the preliminary fusion instance features corresponding to the initial anchor points, ultimately outputting optimized candidate instances. This process yields all candidate instances.

[0043] Optionally, all candidate instances can be selected from all initial instances based on instance feature confidence. That is, after visual feature aggregation and semantic update, the instance features bound to each initial anchor point will output an initial classification confidence. Anchor points below the confidence threshold can be judged as background or invalid anchor points by pre-setting a confidence threshold, while high-confidence anchor points and their corresponding instance features are retained as candidate instances.

[0044] Step S210: According to the candidate anchor points, the features of the LiDAR bird's-eye view are fused into the candidate instance features to obtain the fused instance features.

[0045] Specifically, after completing the multi-scale visual feature extraction, cross-modal feature fusion operation needs to be carried out. By combining the precisely located candidate anchor points, the features of the LiDAR bird's-eye view and the features of the candidate images are deeply fused to generate fused instance features that have both visual semantics and spatial accuracy.

[0046] As an optional embodiment, the method described above can be implemented by fusing the LiDAR bird's-eye view features into candidate instance features according to the candidate anchor points to obtain fused instance features, including: for each candidate instance, using the feature points of the candidate anchor points in the candidate instance as reference positions, and performing conditional sampling and weighted aggregation on the LiDAR bird's-eye view features based on the candidate instance features in the candidate instance to obtain fused instance features in each candidate instance.

[0047] Specifically, for each candidate instance, the coordinates and position information of the feature point (e.g., the center or other selected point) of the candidate anchor point are first used as a reference position to accurately match the spatial region of the candidate instance in the image with the corresponding spatial region of the LiDAR bird's-eye view, completing the spatial alignment of dual-modal features, i.e., conditional sampling. Then, the 3D spatial, distance scale, terrain structure, and object contour geometric features contained in the LiDAR bird's-eye view are extracted and sequentially aligned and superimposed onto the corresponding image candidate instance features, i.e., weighted aggregation. By using candidate anchor point constraints, problems such as feature misalignment and region mismatch are avoided, effectively compensating for the shortcomings of insufficient spatial localization accuracy and lack of depth information in pure visual features. The final fused instance features retain both the rich semantic details of the image and the precise spatial geometric information of the LiDAR, significantly improving feature representation capabilities and providing high-quality fused feature support for subsequent instance decoding, target detection, and accurate scene perception.

[0048] Step S212: Based on the fused instance features, inference is performed to determine the 3D target detection results in the target scene.

[0049] In other words, after completing the fusion of dual-modal features into instance features, 3D target inference and detection can be performed based on these fused instance features, accurately outputting the 3D target detection results for the target scene. The fused instance features integrate image semantic information and LiDAR 3D spatial geometric information, possessing strong multi-dimensional representation capabilities and serving as the core input to the inference network. The model performs global inference analysis based on the optimized fused instance features, fully mining key information such as target category, contour shape, spatial location, size scale, and depth distance contained within the features, accurately identifying, filtering, and classifying various candidate targets within the scene. Simultaneously, relying on the 3D spatial attributes of the features, regression correction of the target's 3D bounding box is completed, eliminating invalid interference samples and erroneous candidate targets. Multi-dimensional feature constraints improve inference accuracy, effectively solving problems such as target occlusion, scale differences, and positioning deviations that easily occur in complex scenes. Ultimately, it accurately determines the category attributes and 3D spatial parameters of all targets within the scene, outputting complete and accurate 3D target detection results, thus meeting the needs of scene perception and subsequent applications such as intelligent driving analysis.

[0050] In embodied intelligence scenarios, 3D target detection results can include the location, size, category, orientation, and motion state of obstacles around the agent, operable targets, work objects, and fellow agents or humans. Embodied agents can then use these 3D target detection results for local path planning, dynamic obstacle avoidance, target approach, grasping posture selection, or task execution status updates.

[0051] In this embodiment, multi-view image data and LiDAR point cloud data of the target scene are acquired; visual features are extracted from the multi-view image data to obtain multi-scale visual features; LiDAR features are extracted from the LiDAR point cloud data to obtain LiDAR bird's-eye view features; based on the multi-scale visual features, all initial instances are initially updated to obtain multiple candidate instances, where each candidate instance includes: corresponding candidate anchor points and candidate instance features; based on the candidate anchor points, the LiDAR bird's-eye view features are fused into the candidate instance features to obtain fused instance features; based on the fused instance features, inference is performed to determine the target scene. The method for obtaining 3D target detection results in this paper, by adding a branch for LiDAR feature extraction, allows LiDAR point clouds to be accessed in the form of LiDAR bird's-eye view features in the sparse query decoding link. This provides stable 3D geometric information supplementation for visual perception. Furthermore, by fusing LiDAR bird's-eye view features with candidate instance features according to candidate anchor points, the resulting fused instance features allow LiDAR features to participate in the fusion around the candidate anchor point positions. This not only supplements 3D geometric information but also improves the accuracy of instance-level geometric alignment, thus solving the problem of unstable 3D geometric information supplementation in related technologies. In embodied intelligence applications, the above 3D target detection results can also serve as spatial perception input for intelligent agent motion planning, grasping operations, inspection and cleaning, logistics delivery, or human-computer interaction. This enables the embodied intelligent agent to generate corresponding action decisions based on the fused target category, position, size, orientation, and motion state.

[0052] As an optional embodiment, as described above, for each candidate instance, the feature points of the candidate anchor points in the candidate instance are used as reference positions, and based on the candidate instance features in the candidate instance, conditional sampling and weighted aggregation are performed on the LiDAR bird's-eye view features to obtain the fused instance features corresponding to each candidate instance, including: performing the fusion operation N times in a loop, and using the instance features of the fusion result obtained from the last fusion operation as the fused instance features, where N is an integer greater than or equal to 2; The fusion operation includes: For each target instance, temporal graph attention processing is performed to obtain each temporally enhanced instance whose features incorporate historical motion information. Specifically, in the case of the first fusion operation, the target instance is a candidate instance; in cases where the fusion operation is not performed for the first time, the target instance is the fusion result obtained from the previous fusion operation. Specifically, candidate instances in the current frame and instance information from historical frames are processed. The processing method is to associate and match instances in the current frame with instances in the past few frames. For example, connecting "the car seen this second" with "the car seen last second" confirms that they are the same object. This gives instances "temporal continuity," allowing us not only to know where the object is now, but also to predict its motion trend based on its past trajectory, thus solving the problem of objects being temporarily occluded or the image being blurred. This results in temporally enhanced instances incorporating historical motion information.

[0053] Each temporal augmentation instance undergoes self-attention processing to obtain a corresponding global logical instance. Specifically, this is achieved by capturing the global contextual relationships between different temporal augmentation instances. Through comparison and correlation, unreasonable isolated guesses are eliminated (e.g., a wheel cannot be suspended in the air), making the reasoning of the overall scene more logical. This results in a more globally logical global logical instance that considers the influence of other surrounding objects and corresponds to each temporal augmentation instance.

[0054] By utilizing multi-scale visual features and adjusting the instance features of each global logical instance according to candidate anchor points, an enhanced instance corresponding to each global logical instance is obtained, which enhances the semantic description of the appearance. In other words, based on the candidate anchor points in the global logical instance, image details (such as color, texture, and text labels) around the candidate anchor point location are flexibly extracted from the visual feature map. This allows for secondary refinement of the visual information. With temporal and global information as a foundation, the appearance details of the object are re-confirmed, supplementing richer semantic information (e.g., clarifying whether the surrounding vehicles are ambulances or ordinary vans). This results in the output of instance features with enhanced semantic description of the appearance, thus obtaining the enhanced instance.

[0055] Using the feature points of candidate anchor points in each augmented instance as reference positions, and based on the augmented instance features in each augmented instance, conditional sampling and weighted aggregation are performed on the LiDAR bird's-eye view features to obtain point cloud aggregation features. Based on the candidate anchor points, the augmented instance features in each augmented instance, and the point cloud aggregation features, a multimodal instance corresponding to each augmented instance is obtained. Specifically, conditional sampling can be performed on the LiDAR bird's-eye view features, centered on the feature points of candidate anchor points in each augmented instance, to obtain 3D structure, contour boundary, and distance scale information from the candidate instance's neighborhood. This can alleviate the problems of inaccurate pure visual ranging, scale ambiguity, and spatial positioning offset, resulting in multimodal instance features that integrate visual semantics and LiDAR geometric information, and thus, a multimodal instance.

[0056] Each multimodal instance is processed sequentially through an FFN network and a refinement layer to obtain the fusion result corresponding to the features of each multimodal instance. In other words, after obtaining the multimodal instances, the FFN network and refinement layer are used to perform deep integration of instance features and ultimately accurate correction and judgment of the entity, resulting in the fusion result corresponding to the features of each multimodal instance.

[0057] In other words, the fusion operation will be executed N times, and in the last fusion operation, the instance features of the fusion result obtained by the fusion operation will be used as the instance features after fusion.

[0058] By adopting a serial update method that first aggregates visual features and then enhances cross-attention with LiDAR, the visual semantic features and the geometric features corresponding to the LiDAR bird's-eye view features complement each other, thereby improving the stability of 3D detection in complex scenes.

[0059] As an optional implementation, each multimodal instance can be processed sequentially through an FFN network and a refinement layer to obtain the fusion result corresponding to the features of each multimodal instance through the following steps: The FFN network fuses the enhanced instance features and point cloud aggregation features of each multimodal instance to obtain the multimodal fused instance features corresponding to each multimodal instance. In other words, for each multimodal instance, the FFN network can perform deep fusion (e.g., nonlinear transformation) of the enhanced instance features and point cloud aggregation features contained in the multimodal instance features that integrate visual and LiDAR information, thereby achieving information integration and refinement to obtain the multimodal fused instance features corresponding to each multimodal instance.

[0060] The refinement layer adjusts the multimodal fusion instance features in each multimodal instance to obtain the fusion result. Specifically, for multimodal instances output by the FFN network, the refinement layer can perform local neighborhood feature completion, target contour boundary correction, and spatial position precision fine-tuning on the multimodal fusion instance features based on the scene's 3D spatial topology. This corrects the local feature loss and spatial offset issues caused by attention sampling, optimizes the 3D geometric details of entities such as size, depth, and orientation, and finally outputs a fusion result adapted for cross-modal fusion.

[0061] As an optional embodiment, the method described above further includes: The 3D object detection results are transmitted to the embodied intelligence decision-making module. Specifically, within the embodied intelligence framework, the 3D object detection results can be transmitted to the embodied intelligence decision-making module (e.g., a task planner, motion controller, or operation actuator). These 3D object detection results serve as sparse instances of the surrounding environment, i.e., obstacles (such as other vehicles and pedestrians) and map elements (such as lane lines) with precise 3D bounding box parameters and features. Furthermore, the initialization state of the vehicle instance can also be included: based on the features from the forward-looking camera and the vehicle's historical state, a vehicle instance containing semantic and geometric information is generated and submitted to the planner along with the 3D object detection results.

[0062] The embodied intelligent decision-making module plans the planned path and motion control commands for the embodied intelligent agent in the target scene based on the 3D target detection results. The embodied intelligent agent is the entity where the embodied intelligent decision-making module resides. Specifically, when the embodied intelligent decision-making module is located on a vehicle (i.e., the embodied intelligent agent is the vehicle), it can deeply interact with surrounding obstacle instances in the 3D target detection results (e.g., analyzing whether a nearby car will suddenly change lanes), while simultaneously predicting multiple possible trajectories for all surrounding traffic participants. During prediction, the embodied intelligent decision-making module generates multiple candidate planned trajectories for the vehicle in parallel. Then, it selects the most reasonable and safest trajectory from these candidate trajectories as the optimal planned path. Finally, this optimal planned path can be translated into specific motion control commands for the embodied intelligent agent (such as steering angle, acceleration, etc.) to direct the vehicle to drive safely, or to direct robots, construction machinery, and other embodied intelligent agents to complete obstacle avoidance, target approach, or operational actions.

[0063] like Figure 3 As shown, an application example of applying the method of any of the foregoing embodiments is provided, including: Step 1: Acquire multi-view image data and LiDAR point cloud data of the target scene. Multi-view image data provides semantic and texture information, while LiDAR point cloud data provides stable 3D spatial structure information.

[0064] Step 2: Perform visual feature extraction on multi-view image data. After the image is input into the visual backbone network + FPN (Feature Pyramid Network), visual coding features at multiple scales are extracted. These features are then mapped by the feature pyramid network to multi-scale visual features with a unified embedding dimension for subsequent instance query decoding.

[0065] Step 3: Perform LiDAR feature extraction on the LiDAR point cloud data. The LiDAR point cloud sequentially passes through a voxelization module, a columnar feature encoding module, a bird's-eye view scattering module, a point cloud backbone network, and a point cloud neck network to obtain the fused LiDAR bird's-eye view features (i.e., LiDAR BEV features); then, channel compression is used to map them to a dimension consistent with the dimension of multi-scale visual features.

[0066] Step 4: Construct a sparse instance query. The instance library outputs a fixed number (e.g., 900) of initial instance features and initial anchors, and generates a position code corresponding to each initial anchor through the anchor encoding module to construct a sparse instance query, so that subsequent visual aggregation and LiDAR feature sampling are both centered around candidate instances.

[0067] Step 5: Perform the first single-frame decoding stage, i.e., the first-layer decoding. Using the initial anchor point as a condition, perform camera deformable feature aggregation on multi-scale visual features to complete the initial semantic update and spatial localization of the instance query. Then, perform feedforward transformation on the initial instance features through FFN, and the refinement layer is used to organize the information output by FFN to obtain candidate instances, which are then input into the "temporal layer + self-attention" process.

[0068] Step Six: Perform LiDAR and visual cross-attention fusion in the subsequent decoding stage. The subsequent decoding stage may include: temporal graph attention (i.e., temporal layer), self-attention, camera deformable feature aggregation, LiDAR deformable attention, FFN, and refinement layer; and repeat N times to obtain the fusion result. Among them, the LiDAR deformable attention uses the center of the candidate anchor point or the candidate key point as the reference position, performs conditional sampling and weighted aggregation on the LiDAR bird's-eye view features, so that the point cloud geometric information is injected into the visual query representation (i.e., candidate instance features of candidate instances) at the instance granularity, and outputs the category, 3D bounding box parameters, and quality estimation results through the subsequent FFN and refinement layer.

[0069] Step 7: Perform target refinement and output. The refinement layer outputs the target category, 3D bounding box parameters, and quality estimation results based on the instance features in the fusion result. During the training phase, classification loss and regression loss can be combined for optimization. During the inference phase, the final 3D target detection result is output.

[0070] Step 8: Within the SparseDrive framework, the 3D target detection results output by the detection branch can be used as input for subsequent modules. In other words, the above method can be used solely for 3D target detection tasks, or it can be deployed within sparse-driven autonomous driving end-to-end frameworks like SparseDrive as an enhancement module for its detection branch. This maintains compatibility with the original sparse query decoding paradigm and the overall SparseDrive interface, thus enabling LiDAR feature enhancement without significantly increasing computational overhead. Similarly, the above method can also be deployed in embodied intelligence end-to-end systems, transmitting instance-level 3D perception results from the detection branch to task planners, motion controllers, or actuators, enabling agents to achieve environmental understanding and action decision-making loops in scenarios such as warehousing and logistics, park inspections, cleaning services, and engineering operations.

[0071] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0072] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0073] According to another aspect of the embodiments of this application, a target detection device for fusion vision and lidar applied to an embodied intelligent agent is also provided for implementing the above-described target detection method for fusion vision and lidar applied to an embodied intelligent agent. Figure 4 This is a structural block diagram of an optional target detection device based on an embodiment of this application, which combines vision and lidar for use in embodied intelligent agents. Figure 4 As shown, the device may include: The acquisition module 41 is used to acquire multi-view image data and lidar point cloud data of the target scene; Extraction module 42 is used to extract visual features from the multi-view image data to obtain multi-scale visual features; and to extract lidar features from the lidar point cloud data to obtain lidar bird's-eye view features. The initial update module 43 is used to perform an initial update on all initial instances according to the multi-scale visual features to obtain multiple candidate instances, wherein each candidate instance includes: corresponding candidate anchor points and candidate instance features. The fusion module 44 is used to fuse the features of the lidar bird's-eye view into the features of the candidate instance according to the candidate anchor points, so as to obtain the fused instance features; The reasoning module 45 is used to perform reasoning based on the fused instance features to determine the 3D target detection result in the target scene.

[0074] It should be noted that the acquisition module 41 in this embodiment can be used to perform the above step S202, the extraction module 42 in this embodiment can be used to perform the above steps S204 and S206, the preliminary update module 43 in this embodiment can be used to perform the above step S208, the fusion module 44 in this embodiment can be used to perform the above step S210, and the inference module 45 in this embodiment can be used to perform the above step S212.

[0075] In addition to the modules described above, the apparatus in this embodiment may also include modules that perform any of the methods described in any of the aforementioned embodiments of the target detection method for fusion vision and lidar applied to an embodied intelligent agent.

[0076] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in environments such as... Figure 1 The hardware environment shown can be implemented through software or hardware, and the hardware environment includes the network environment.

[0077] According to another aspect of the embodiments of this application, an electronic device is also provided for implementing the above-described target detection method for fused vision and lidar applied to an embodied intelligent agent. The electronic device may be a server, a terminal, or a combination thereof.

[0078] According to another embodiment of this application, an electronic device is also provided, comprising: Figure 5As shown, the electronic device may include: a processor 1501, a communication interface 1502, a memory 1503, and a communication bus 1504, wherein the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504.

[0079] Memory 1503 is used to store computer programs; When processor 1501 executes the program stored in memory 1503, it performs the following steps: Step S202: Acquire multi-view image data and lidar point cloud data of the target scene.

[0080] Step S204: Visual features are extracted from the multi-view image data to obtain multi-scale visual features.

[0081] Step S206: Extract lidar features from lidar point cloud data to obtain lidar bird's-eye view features.

[0082] Step S208: Based on multi-scale visual features, perform preliminary updates on all initial instances to obtain multiple candidate instances. Each candidate instance includes: corresponding candidate anchor points and candidate instance features.

[0083] Step S210: According to the candidate anchor points, the features of the LiDAR bird's-eye view are fused into the candidate instance features to obtain the fused instance features.

[0084] Step S212: Based on the fused instance features, inference is performed to determine the 3D target detection results in the target scene.

[0085] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic device and other devices.

[0086] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0087] As an example, the memory 1503 described above may include, but is not limited to, the acquisition module 41, extraction module 42, preliminary update module 43, fusion module 44, and inference module 45 of the target detection device using fused vision and lidar applied to an embodied intelligent agent. Furthermore, it may include, but is not limited to, other module units in the aforementioned three-dimensional target detection device, which will not be elaborated upon in this example.

[0088] The processor mentioned above can be a general-purpose processor, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0089] This application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the method steps of the above method embodiments when it runs.

[0090] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0091] This application also provides an embodied intelligent agent, including the electronic device described in the foregoing embodiments.

[0092] An embodied intelligent agent can be any device possessing mobility, perception, decision-making, or execution capabilities, including vehicles and robots with autonomous or intelligent driving capabilities. Vehicles include: passenger vehicles (e.g., cars, buses, coaches, minibuses, etc.), cargo vehicles (e.g., regular trucks, box trucks, trailer trucks, enclosed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, special-structure trucks), special vehicles (e.g., logistics delivery vehicles, automated guided vehicles (AGVs), patrol vehicles, cranes, excavators, bulldozers, loaders, road rollers, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor scrubbers, water sprinkler trucks, sweeping robots, food delivery robots, shopping guide robots, lawnmowers, golf carts, etc.), recreational vehicles (e.g., amusement park vehicles, amusement park autonomous driving devices, balance bikes, etc.), and rescue vehicles (e.g., fire trucks, ambulances, power repair vehicles, engineering emergency vehicles, etc.). Robots can be, for example, sweeping robots or food delivery robots.

[0093] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0094] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0095] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0097] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the solution provided in this embodiment, depending on actual needs.

[0098] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0099] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A target detection method integrating vision and lidar applied to embodied intelligent agents, characterized in that, include: Acquire multi-view image data and LiDAR point cloud data of the target scene; Visual features are extracted from the multi-view image data to obtain multi-scale visual features; LiDAR feature extraction is performed on the LiDAR point cloud data to obtain LiDAR bird's-eye view features; Based on the multi-scale visual features, all initial instances are initially updated to obtain multiple candidate instances, wherein each candidate instance includes: corresponding candidate anchor points and candidate instance features; According to the candidate anchor points, the features of the LiDAR bird's-eye view are fused into the features of the candidate instances to obtain the fused instance features; Based on the fused instance features, inference is performed to determine the 3D target detection results in the target scene.

2. The method according to claim 1, characterized in that, The process involves initially updating all initial instances based on the multi-scale visual features to obtain multiple candidate instances, including: Using each initial anchor point as a condition, camera deformable feature aggregation is performed on the multi-scale visual features to obtain the visual features corresponding to each initial anchor point in the multi-scale visual features, wherein the initial anchor point is the anchor point contained in the initial instance; The visual features corresponding to each initial anchor point are fused with the initial instance features corresponding to each initial anchor point to obtain preliminary fused instance features; Each initial instance is adjusted based on the preliminary fusion instance features corresponding to each initial anchor point to obtain the multiple candidate instances.

3. The method according to claim 1, characterized in that, The step of fusing the LiDAR bird's-eye view features with the candidate instance features according to the candidate anchor points to obtain the fused instance features includes: For each candidate instance, the feature points of the candidate anchor points in the candidate instance are used as reference positions, and conditional sampling and weighted aggregation are performed on the LiDAR bird's-eye view features based on the candidate instance features in the candidate instance to obtain the fused instance features in each candidate instance.

4. The method according to claim 3, characterized in that, For each candidate instance, the feature points of the candidate anchor points in the candidate instance are used as reference positions, and conditional sampling and weighted aggregation are performed on the features of the LiDAR bird's-eye view based on the candidate instance features in the candidate instance to obtain the fused instance features corresponding to each candidate instance. This includes: performing the fusion operation N times in a loop, and taking the instance features in the fusion result obtained from the last fusion operation as the fused instance features, where N is an integer greater than or equal to 2. The fusion operation includes: Temporal graph attention processing is performed on each target instance to obtain each temporally enhanced instance in which historical motion information is incorporated into the features of the target instance. In the case of the first execution of the fusion operation, the target instance is the candidate instance. In the case of the non-first execution of the fusion operation, the target instance is the fusion result obtained from the previous fusion operation. Each of the time-series enhancement instances is subjected to self-attention processing to obtain a global logic instance corresponding to each of the time-series enhancement instances; By using the multi-scale visual features and adjusting the instance features in each global logical instance according to the candidate anchor points in each global logical instance, an enhanced instance corresponding to each global logical instance is obtained after enhancing the semantic description of the appearance. Using the feature points of the candidate anchor points in each enhancement instance as reference positions, and based on the enhancement instance features in each enhancement instance, conditional sampling and weighted aggregation are performed on the LiDAR bird's-eye view features to obtain point cloud aggregation features. Based on the candidate anchor points, the enhancement instance features in each enhancement instance, and the point cloud aggregation features, a multimodal instance corresponding to each enhancement instance is obtained. Each multimodal instance is processed sequentially through an FFN network and a refinement layer to obtain the fusion result corresponding to the features of each multimodal instance.

5. The method according to claim 4, characterized in that, The process of sequentially passing each multimodal instance through an FFN network and a refinement layer to obtain the fusion result corresponding to the features of each multimodal instance includes: By using the FFN network, the enhanced instance features and the point cloud aggregation features in each multimodal instance are fused to obtain the multimodal fused instance features corresponding to each multimodal instance. The fusion result is obtained by adjusting the multimodal fusion instance features corresponding to each multimodal instance through the refinement layer.

6. The method according to claim 1, characterized in that, The step of extracting lidar features from the lidar point cloud data to obtain lidar bird's-eye view features includes: The lidar point cloud data is processed sequentially through a voxelization module, a columnar feature encoding module, a bird's-eye view scattering module, a point cloud backbone network, and a point cloud neck network to obtain the processed features. Based on the dimensions of multi-scale visual features, channel compression is performed on the processed features to obtain the LiDAR bird's-eye view features that are compatible with the multi-scale visual features.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The three-dimensional target detection results are transmitted to the embodied intelligent decision-making module; The embodied intelligent decision-making module plans the planned path in the target scene and the motion control commands for the embodied intelligent agent based on the three-dimensional target detection results, wherein the embodied intelligent agent is the entity in which the embodied intelligent decision-making module is located.

8. A target detection device that fuses vision and lidar for use in embodied intelligent agents, characterized in that, include: The acquisition module is used to acquire multi-view image data and LiDAR point cloud data of the target scene; The extraction module is used to extract visual features from the multi-view image data to obtain multi-scale visual features; LiDAR feature extraction is performed on the LiDAR point cloud data to obtain LiDAR bird's-eye view features; The initial update module is used to perform an initial update on all initial instances according to the multi-scale visual features to obtain multiple candidate instances, wherein each candidate instance includes: corresponding candidate anchor points and candidate instance features; The fusion module is used to fuse the features of the LiDAR bird's-eye view into the features of the candidate instance according to the candidate anchor points, so as to obtain the fused instance features; The inference module is used to perform inference based on the fused instance features to determine the 3D target detection results in the target scene.

9. An electronic device comprising a processor, a communication interface, a memory, and a communication bus, wherein, The processor, the communication interface, and the memory communicate with each other via the communication bus, characterized in that... The memory is used to store computer programs; The processor is configured to perform the method of any one of claims 1 to 7 by running the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 7 when run on a processor.

11. An embodied intelligent agent, characterized in that, Includes the electronic device as described in claim 9.

Images