Indoor robot perception method based on dynamic fusion of monocular vision and radar data

Abstract

The application discloses an indoor robot perception method based on dynamic fusion of monocular vision and radar data. It belongs to the technical field of image data processing. The method comprises the following steps: acquiring image data collected by a monocular camera and point cloud data collected by a laser radar; inputting the image data into a target detection model to output a 2D target frame, and calculating a first 3D target frame information of a target object in a robot coordinate system according to the 2D target frame; performing clustering processing on the point cloud data to obtain at least one point cloud cluster, and converting the point cloud cluster to the robot coordinate system; matching each converted point cloud cluster with the first 3D target frame information, and when there is a matching successful point cloud cluster, updating the depth information and the deflection angle in the first 3D target frame information by using the point cloud data in the matching successful point cloud cluster to obtain second 3D target frame information; and generating a control instruction for controlling the motion of the indoor robot according to the second 3D target frame information.

Description

Technical Field

[0001] This application relates to the field of image data processing technology, and in particular to an indoor robot perception method based on the dynamic fusion of monocular vision and radar data. Background Technology

[0002] With the intelligent evolution of urban transportation systems, indoor robots, as the core carriers of future smart mobility and logistics delivery, are expanding their operating scenarios from traditional outdoor roads to more complex indoor environments, such as underground parking lots, indoor storage spaces, and logistics centers. Compared to open roads, indoor environments are characterized by complex lighting conditions, compact spatial structures, lack of GPS signals, and diverse obstacles, placing higher demands on the environmental perception and safety control capabilities of indoor robots. In these environments, the emergency braking and precise obstacle avoidance capabilities of indoor robots are crucial; if the system fails to respond in time, collisions or property damage can easily occur.

[0003] In indoor robot autonomous driving and emergency braking systems, high-precision, low-latency environmental perception capabilities are crucial for ensuring safe operation. Especially in confined indoor spaces or unexpected scenarios, the perception system must possess real-time ranging and multi-target recognition capabilities. Currently, visual perception, due to its advantages such as low cost, ease of deployment, and rich structural information, has become an important component of indoor robot perception systems.

[0004] In particular, 2D monocular detectors perform exceptionally well in target detection tasks, especially in recognizing long-tail categories (such as rare objects and small obstacles). They also do not rely on expensive 3D data annotation and offer good scalability and adaptability. This advantage is particularly evident in indoor environments, where the variety of targets is vast and the training samples are limited, making long-tail category recognition crucial for the performance of the perception system. However, monocular vision suffers from scale ambiguity and localization errors due to the lack of depth information. Especially in scenes with varying lighting or severe occlusion, its ranging performance is easily affected, making it difficult to meet high-precision requirements.

[0005] While lidar can provide accurate 3D spatial information, it has limitations in target category recognition, and relying solely on lidar is insufficient to handle the need for multi-target differentiation in complex scenes. Therefore, effectively integrating monocular vision and lidar data to fully leverage the advantages of both has become a key research direction for improving the perception performance of indoor robots.

[0006] To address the technical problem that traditional single-sensor-based indoor robot perception methods in the aforementioned technologies cannot simultaneously achieve ranging accuracy and target recognition capability, and thus cannot meet the high-precision and high-robust perception requirements in complex indoor environments, no effective solution has yet been proposed. Summary of the Invention

[0007] The embodiments of this disclosure provide an indoor robot perception method and apparatus based on the dynamic fusion of monocular vision and radar data, so as to at least solve the technical problem that the existing indoor robot perception methods based on a single sensor are difficult to balance ranging accuracy and target recognition capability, and cannot meet the high-precision and high-robust perception requirements in complex indoor environments.

[0008] According to one aspect of the present disclosure, an indoor robot perception method based on dynamic fusion of monocular vision and radar data is provided, comprising: acquiring image data collected by a monocular camera and point cloud data collected by a lidar; wherein the monocular camera and the lidar are mounted on an indoor robot; inputting the image data into a pre-trained target detection model, outputting a 2D target bounding box, and calculating a first 3D target bounding box information of the target object in the robot's own coordinate system based on the 2D target bounding box; the first 3D target bounding box information includes at least the three-dimensional position coordinates, size parameters, and deflection angle of the target object; the three... The X-direction component in the 3D position coordinates represents the depth information of the target object; the point cloud data is clustered to obtain at least one point cloud cluster, and the point cloud cluster is transformed to the robot's own coordinate system; each transformed point cloud cluster is matched with the first 3D target bounding box information, and when a successfully matched point cloud cluster exists, the depth information and deflection angle in the first 3D target bounding box information are updated using the point cloud data in the successfully matched point cloud cluster to obtain the second 3D target bounding box information; based on the second 3D target bounding box information, control commands for controlling the movement of the indoor robot are generated.

[0009] According to another aspect of the present disclosure, a storage medium is also provided, the storage medium including a stored program, wherein, when the program is executed, a processor performs any of the methods described above.

[0010] According to another aspect of the present disclosure, an indoor robot perception device based on dynamic fusion of monocular vision and radar data is also provided, comprising: a multimodal data acquisition module for acquiring image data acquired by a monocular camera and point cloud data acquired by a lidar; wherein the monocular camera and the lidar are mounted on an indoor robot; and an image data processing module for inputting the image data into a pre-trained target detection model, outputting a 2D target bounding box, and calculating a first 3D target bounding box information of the target object in the robot's own coordinate system based on the 2D target bounding box; wherein the first 3D target bounding box information includes at least the three-dimensional position coordinates, size parameters, and deflection angle of the target object. The X-direction component in the three-dimensional position coordinates represents the depth information of the target object; the point cloud data processing module is used to perform clustering processing on the point cloud data to obtain at least one point cloud cluster, and transform the point cloud cluster to the robot's own coordinate system; the information update module is used to match the transformed point cloud cluster with the first 3D target box information, and when the match is successful, use the point cloud data in the point cloud cluster to update the depth information and deflection angle in the first 3D target box information to obtain the second 3D target box information; and the control command generation module is used to generate control commands for controlling the movement of the indoor robot based on the second 3D target box information.

[0011] According to another aspect of the present disclosure, an indoor robot perception device based on dynamic fusion of monocular vision and radar data is also provided, comprising: a processor; and a memory connected to the processor, for providing the processor with instructions to process the following steps: acquiring image data acquired by a monocular camera and point cloud data acquired by a lidar; wherein the monocular camera and the lidar are mounted on an indoor robot; inputting the image data into a pre-trained target detection model, outputting a 2D target bounding box, and calculating a first 3D target bounding box information of the target object in the robot's own coordinate system based on the 2D target bounding box; wherein the first 3D target bounding box information includes at least the target object's... The system comprises three-dimensional position coordinates, size parameters, and deflection angles; the X-direction component in the three-dimensional position coordinates represents the depth information of the target object; the point cloud data is clustered to obtain at least one point cloud cluster, and the point cloud clusters are transformed to the robot's own coordinate system; each transformed point cloud cluster is matched with the first 3D target bounding box information, and when a successfully matched point cloud cluster exists, the depth information and deflection angle in the first 3D target bounding box information are updated using the point cloud data in the successfully matched point cloud cluster to obtain the second 3D target bounding box information; based on the second 3D target bounding box information, control commands for controlling the movement of the indoor robot are generated.

[0012] In the context of indoor robot environmental perception, this application first uses a monocular camera and LiDAR mounted on the indoor robot to simultaneously acquire image data and point cloud data from the environment, serving as the initial multimodal input for subsequent fusion processing. Then, a pre-trained target detection model is used to process the image data, outputting 2D bounding boxes. Based on this, the first 3D bounding box information of the target object in the robot's own coordinate system is calculated. This information includes at least three-dimensional position coordinates carrying depth information, size parameters, and deflection angles, thereby achieving preliminary target localization and category recognition in the scene, providing initial target hypotheses based on visual understanding for subsequent fusion. Next, the point cloud data acquired by the LiDAR is clustered to obtain at least one point cloud cluster, which is then transformed to the same robot's own coordinate system, thus obtaining an independent set of object point clouds based on precise spatial measurements. Next, each converted point cloud cluster is matched one by one with the first 3D target bounding box information. When a successfully matched point cloud cluster exists, the depth information and deflection angle in the first 3D target bounding box information obtained from visual calculation are updated using the precise 3D spatial point cloud data in that cluster. This results in a second 3D target bounding box information that integrates visual semantic information and precise radar spatial information, achieving deep coupling of the advantages of the two sensors. Finally, based on the updated and more accurate second 3D target bounding box information, control commands for controlling the movement of the indoor robot are generated, providing a reliable environmental perception basis for the robot's safe obstacle avoidance and autonomous navigation. Thus, this application, through the dynamic fusion of vision and radar, fully leverages the advantages of monocular vision target recognition capabilities while using radar point cloud data to accurately correct the depth information and deflection angle obtained from visual calculation. This achieves a balance between ranging accuracy and target recognition capability, effectively improving the accuracy and robustness of the indoor robot's environmental perception in complex environments. This solves the technical problem that existing indoor robot perception methods based on a single sensor cannot balance ranging accuracy and target recognition capability, and cannot meet the high-precision and high-robust perception requirements in complex indoor environments. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this application, illustrate exemplary embodiments of this disclosure and are used to explain this disclosure, but do not constitute an undue limitation of this disclosure. In the drawings: Figure 1 This is a hardware structure block diagram of a computing device for implementing the method described in Embodiment 1 of this disclosure; Figure 2 This is a flowchart illustrating the indoor robot perception method based on dynamic fusion of monocular vision and radar data according to Embodiment 1 of this disclosure. Figure 3This is a complete flowchart of the indoor robot perception method based on dynamic fusion of monocular vision and radar data according to Embodiment 1 of this disclosure; Figure 4 This is a schematic diagram of the architecture of the improved YOLOv8 model according to Embodiment 1 of this disclosure; Figure 5 This is a schematic diagram of the structure of an indoor robot perception device based on the dynamic fusion of monocular vision and radar data, according to Embodiment 2 of this disclosure; and Figure 6 This is a schematic diagram of the structure of an indoor robot perception device based on the dynamic fusion of monocular vision and radar data, according to Embodiment 3 of this disclosure. Detailed Implementation

[0014] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1

[0017] According to this embodiment, an embodiment of a method for indoor robot perception based on dynamic fusion of monocular vision and radar data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0018] The method embodiments provided in this example can be executed on mobile terminals, computer terminals, servers, or similar computing devices. Figure 1 A hardware block diagram of a computing device for realizing an indoor robot perception method and apparatus based on the dynamic fusion of monocular vision and radar data is shown. Figure 1 As shown, a computing device may include one or more processors (processors may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory for storing data, a transmission device for communication functions, and an input / output interface. The memory, transmission device, and input / output interface are connected to the processor via a bus. In addition, it may also include a display, keyboard, and cursor control device connected to the input / output interface. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, a computing device may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0019] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element in a computing device. As involved in the embodiments of this disclosure, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0020] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the indoor robot perception method and device based on dynamic fusion of monocular vision and radar data in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the above-mentioned application of the indoor robot perception method and device based on dynamic fusion of monocular vision and radar data. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the computing device via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0021] The transmission device is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the computing device's communication provider. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0022] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows users to interact with the user interface of the computing device.

[0023] It should be noted here that, in some optional embodiments, the above... Figure 1 The computing device shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computing devices.

[0024] Under the aforementioned operating environment, according to the first aspect of this embodiment, an indoor robot perception method based on the dynamic fusion of monocular vision and radar data is provided. This method comprises... Figure 1 The processor implementation shown. Figure 2 A flowchart illustrating the method is shown below. (Refer to...) Figure 2 As shown, the method includes: S202: Acquire image data from a monocular camera and point cloud data from a lidar; the monocular camera and the lidar are mounted on an indoor robot; S204: Input the image data into a pre-trained target detection model, output a 2D target bounding box, and calculate the first 3D target bounding box information of the target object in the robot's own coordinate system based on the 2D target bounding box; the first 3D target bounding box information includes at least the three-dimensional position coordinates, size parameters, and deflection angle of the target object; the X-direction component in the three-dimensional position coordinates represents the depth information of the target object; S206: Perform clustering processing on the point cloud data to obtain at least one point cloud cluster, and transform the point cloud cluster to the robot's own coordinate system; S208: Match each converted point cloud cluster with the first 3D target box information, and when there is a successfully matched point cloud cluster, use the point cloud data in the successfully matched point cloud cluster to update the depth information and deflection angle in the first 3D target box information to obtain the second 3D target box information. S210: Based on the second 3D target box information, generate control commands for controlling the movement of the indoor robot.

[0025] Specifically, in indoor robot environmental perception scenarios, such as complex environments like underground parking lots, indoor warehouses, or logistics centers, indoor robots need to perceive their surroundings in real time to complete autonomous navigation and obstacle avoidance tasks. First, the indoor robot uses its onboard monocular camera and LiDAR to simultaneously acquire image data and point cloud data of the environment (corresponding to step S202). Specifically, the monocular camera acquires color images of the robot's forward direction in real time at a certain frame rate (e.g., 30fps), while the LiDAR performs a 360° scan at a corresponding frequency (e.g., 10Hz) to obtain point cloud data of the surrounding environment. To ensure the accuracy of subsequent fusion processing, the monocular camera and LiDAR can achieve frame-level alignment in time through hardware triggering or software synchronization, and their relative pose relationship in space can be determined through pre-calibrated joint calibration. This provides spatiotemporally aligned multimodal raw input data for subsequent processing.

[0026] Then, the processor inputs the acquired image data into a pre-trained target detection model. This model, trained on a large amount of labeled data, is capable of recognizing target objects in the image and outputting corresponding 2D bounding boxes. Based on the output 2D bounding boxes, the processor combines camera intrinsic parameters and preset target object size information to calculate the first 3D bounding box information of the target object in the robot's own coordinate system. This first 3D bounding box information includes at least the target object's three-dimensional position coordinates, size parameters, and deflection angle, where the X-direction component in the three-dimensional position coordinates is the depth information of the target object relative to the robot (corresponding to step S204). For example, in the robot's own coordinate system, the forward direction is usually defined as the positive X-axis, the left side as the positive Y-axis, and the top as the positive Z-axis. Based on the position and size of the 2D bounding box in the image, the processor calculates the distance of the target object in the camera coordinate system using the principle of similar triangles, and then, combined with the camera's extrinsic parameters (rotation matrix and translation matrix) relative to the robot, converts this distance into X-direction coordinate values ​​in the robot's own coordinate system. Simultaneously, by combining the lateral position of the 2D bounding box and the preset target object size, the Y and Z coordinate values ​​and the physical dimensions (length, width, and height) of the target object are calculated. The deflection angle can be initialized to 0 or further determined in subsequent processing. Thus, a preliminary 3D target representation with semantic category information is obtained, providing initial target hypotheses based on visual understanding for subsequent fusion.

[0027] Secondly, the processor performs clustering processing on the point cloud data acquired by the LiDAR, dividing spatially adjacent point clouds into different clusters, each representing a potential target object. After clustering, based on the pre-calibrated rotation and translation matrices of the LiDAR relative to the robot, the point cloud coordinates in each cluster are transformed from the LiDAR coordinate system to the unified robot coordinate system (corresponding to step S206). This ensures that all point cloud data and image processing results are in the same spatial reference system, laying the foundation for subsequent matching and fusion.

[0028] Next, the processor matches each converted point cloud cluster with the first 3D target bounding box information. Specifically, for the currently traversed point cloud cluster, it determines how many point cloud coordinates in the cluster are located within the three-dimensional spatial region defined by the first 3D target bounding box information. When the proportion of points in a point cloud cluster located within this region exceeds a preset threshold (e.g., 60%), the point cloud cluster is determined to be successfully matched with the first 3D target bounding box information, meaning they correspond to the same physical target object. When a successfully matched point cloud cluster exists, the processor uses the precise three-dimensional spatial point cloud data from that cluster to update the depth information and deflection angle in the first 3D target bounding box information, obtaining second 3D target bounding box information that integrates visual semantic information and precise radar spatial information (corresponding to step S208). Thus, while preserving the visual target recognition capability, radar point cloud data is used to correct potential errors in visual calculation.

[0029] Finally, the processor sends the updated second 3D target bounding box information to the robot's motion control module. Based on the precise position, size, and orientation of the target object contained in the second 3D target bounding box information, combined with the robot's current speed, size, and preset safety model, the motion control module performs collision risk assessment and obstacle avoidance path planning, and generates corresponding control commands, such as acceleration, deceleration, steering, or emergency braking, to drive the robot to perform safe and reliable motion (corresponding to step S210).

[0030] As described in the background section, high-precision, low-latency environmental perception is crucial for ensuring safe operation in indoor robot autonomous driving and emergency braking systems. This is especially true in confined indoor spaces or unexpected scenarios, where the perception system must possess real-time ranging and multi-target recognition capabilities. Currently, visual perception has become a vital component of indoor robot perception systems due to its advantages such as low cost, ease of deployment, and rich structural information. However, monocular vision lacks depth information, suffers from scale ambiguity and positioning errors, and its ranging performance is easily affected by changes in lighting or severe occlusion, making it difficult to meet high-precision requirements. While lidar can provide accurate three-dimensional spatial information, it has limitations in target category recognition, and relying solely on lidar is insufficient to address the multi-target differentiation needs in complex scenarios. Therefore, effectively integrating monocular vision and lidar data to fully leverage their respective advantages has become a key research direction for improving the perception performance of indoor robots.

[0031] In view of this, this application first uses a monocular camera and LiDAR mounted on an indoor robot to simultaneously acquire image data and point cloud data from the environment, which serve as the original multimodal input for subsequent fusion processing. Then, a pre-trained target detection model is used to process the image data, outputting 2D target bounding boxes, and based on this, the first 3D target bounding box information of the target object in the robot's own coordinate system is calculated. This information includes at least the three-dimensional position coordinates carrying depth information, size parameters, and deflection angle, thereby achieving preliminary localization and category recognition of the target in the scene, providing an initial target hypothesis based on visual understanding for subsequent fusion. Secondly, the point cloud data acquired by the LiDAR is clustered to obtain at least one point cloud cluster, which is then transformed to the same robot's own coordinate system, thereby obtaining an independent set of object point clouds based on precise spatial measurements. Next, each converted point cloud cluster is matched one by one with the first 3D target bounding box information. When a successfully matched point cloud cluster exists, the depth information and deflection angle in the first 3D target bounding box information obtained from visual calculation are updated using the precise 3D spatial point cloud data in that cluster. This results in a second 3D target bounding box information that integrates visual semantic information and precise radar spatial information, achieving deep coupling of the advantages of the two sensors. Finally, based on the updated and more accurate second 3D target bounding box information, control commands for controlling the movement of the indoor robot are generated, providing a reliable environmental perception basis for the robot's safe obstacle avoidance and autonomous navigation. Thus, this application, through the dynamic fusion of vision and radar, fully leverages the advantages of monocular vision target recognition capabilities while using radar point cloud data to accurately correct the depth information and deflection angle obtained from visual calculation. This achieves a balance between ranging accuracy and target recognition capability, effectively improving the accuracy and robustness of the indoor robot's environmental perception in complex environments. This solves the technical problem that existing indoor robot perception methods based on a single sensor cannot balance ranging accuracy and target recognition capability, and cannot meet the high-precision and high-robust perception requirements in complex indoor environments.

[0032] In addition, such as Figure 3As shown, before executing step S204, the image data acquired by the monocular camera can be preprocessed to remove distortion, and the point cloud data acquired by the LiDAR can be filtered and denoised. Specifically, for the raw image data acquired by the monocular camera, radial and tangential distortions can occur due to lens manufacturing and assembly errors, causing straight lines in the actual scene to appear as curves in the image, directly affecting the positioning accuracy of subsequent 2D target boxes and the accuracy of depth information calculation based on 2D boxes. Therefore, the processor first uses the pre-calibrated camera distortion coefficient matrix to perform distortion removal processing on the raw image, mapping each pixel in the image to its ideal position and restoring the true geometric relationship. For the raw point cloud data acquired by the LiDAR, due to the influence of environmental noise, multipath reflection, and other factors, there are often a large number of outliers and noise points in the point cloud. These points can interfere with subsequent clustering processing, leading to false targets or missed detection of real targets. Therefore, the processor first performs histogram filtering or statistical filtering on the original point cloud to remove outliers that significantly deviate from the main distribution. Subsequently, voxel filtering or bilateral filtering can be used to downsample and smooth the point cloud, reducing the amount of data and computational complexity while preserving the main geometric structure information. These preprocessing operations provide higher-quality and more reliable input data for subsequent image object detection and point cloud clustering.

[0033] Optionally, the target detection model is an improved YOLOv8 model. The improved YOLOv8 model introduces an adaptive illumination normalization layer at the input of the backbone network, adds a multi-scale feature enhancement module in the neck network, and introduces a channel attention mechanism module before the detection head. Furthermore, the operation of inputting the image data into the pre-trained target detection model and outputting 2D target boxes includes: inputting the image data into the adaptive illumination normalization layer to perform illumination normalization processing on the image and outputting an illumination-normalized image; inputting the illumination-normalized image into the backbone network of the improved YOLOv8 model for feature extraction and outputting feature maps of different scales; inputting the feature maps of different scales into the multi-scale feature enhancement module, performing enhancement processing on the feature maps of different scales through parallel convolutional structures, and adaptively weighting and fusing the output feature maps of each branch to output a fused feature map; inputting the fused feature map into the channel attention mechanism module to output an enhanced feature map; and inputting the enhanced feature map into the head network of the improved YOLOv8 model for target detection and outputting the corresponding 2D target boxes.

[0034] Specifically, to address the impact of complex and variable lighting conditions in indoor environments (such as alternating light and dark in underground parking lots and localized strong light in warehouse spaces) on target detection accuracy, this embodiment makes targeted improvements to the YOLOv8 model, and its network structure is as follows: Figure 4 As shown. Reference Figure 4 As shown, the improved YOLOv8 model introduces an adaptive illumination normalization layer at the input. In actual processing, the processor first inputs the acquired raw image data into this adaptive illumination normalization layer. This layer performs real-time illumination equalization on the image, enabling subsequent modules to receive input images with relatively stable lighting conditions, thereby improving the model's robustness in complex lighting scenarios. The image after illumination normalization is input into the backbone network for feature extraction. The backbone network outputs feature maps at different scales at multiple depths through layer-by-layer convolution and downsampling operations. For example, there are fifth-level feature maps (1 / 8 scale), seventh-level feature maps (1 / 16 scale), and tenth-level feature maps (1 / 32 scale) extracted from different levels. These feature maps preserve multi-level information of the image, from detailed texture to high-level semantics.

[0035] Subsequently, these feature maps at different scales are fed into the multi-scale feature enhancement module in the neck network. This module employs a parallel convolutional structure to independently enhance feature maps at scales of 1 / 8, 1 / 16, and 1 / 32, with each branch strengthening the feature representation at its corresponding scale through specific convolutional operations. This parallel convolutional structure includes three branches: a 1 / 8 scale branch, a 1 / 16 scale branch, and a 1 / 32 scale branch. The 1 / 8 scale branch processes high-resolution feature maps, preserving more detailed information. The 1 / 16 scale branch processes medium-resolution feature maps, balancing detail and semantic information. The 1 / 32 scale branch processes low-resolution feature maps, acquiring richer semantic information. The enhanced feature maps from the three branches are integrated through an adaptive weighted fusion mechanism. The fusion weights are automatically learned by the network during training, enabling the network to dynamically adjust its dependence on features at different scales based on the current scene, effectively addressing the challenges of diverse target scales in indoor environments (such as pedestrians at close range and vehicles at a distance). The fusion formula is as follows: ; in, , , For adaptive learning fusion weights, satisfying ; The fused feature map; The feature map is at a 1 / 8 scale; The feature map is at a scale of 1 / 16. The feature map is at a scale of 1 / 32.

[0036] The fused feature map is input into the channel attention mechanism module. This module recalibrates the feature map by learning the importance weights of each channel, enhancing the response of key channel features (such as target contours and edges) and suppressing redundant information. Finally, the enhanced feature map is fed into the detection head for multi-scale target detection, outputting 2D bounding boxes containing target category and location information. Through the above improvements, the YOLOv8 model in this embodiment achieves significantly improved detection accuracy in complex indoor lighting and multi-scale target scenes, effectively improving the accuracy of the final output 2D bounding boxes.

[0037] Optionally, the operation of inputting the image data into the adaptive illumination normalization layer to perform illumination normalization processing on the image and outputting the illumination-normalized image includes: performing histogram statistics on the RGB three channels of the image data respectively, calculating the brightness distribution characteristics of each channel, wherein the brightness distribution characteristics include mean, variance, and peak position; dynamically generating a brightness gain coefficient and a brightness offset based on the brightness distribution characteristics; and performing illumination normalization processing on the image data according to the following formula to obtain the illumination-normalized image, wherein the color saturation of the image remains unchanged while adjusting the brightness: ; in, The input image data, The image is after illumination normalization. The brightness gain coefficient is... This refers to the brightness offset.

[0038] Specifically, the core of the adaptive illumination normalization layer lies in dynamically adjusting illumination parameters based on the brightness characteristics of the image itself. When a frame of image data is input into this layer, the processor first performs histogram statistical analysis on the three RGB channels of the image, calculating the mean brightness (reflecting the overall brightness level), variance (reflecting the contrast strength), and peak position (reflecting the main brightness distribution range) for each channel. Based on these statistical characteristics, the processor evaluates the current image's illumination status in real time: for example, if the mean and peak position are low, it indicates that the image is generally too dark; if the variance is too small, it indicates insufficient image contrast. Based on the evaluation results, the processor dynamically generates a brightness gain coefficient suitable for the current frame. and brightness offset For example, for a dark image, you can appropriately increase the size. Increase the overall brightness and adjust the value. The value avoids overexposure. Then, the processor follows the formula... The input image undergoes a pixel-by-pixel linear transformation to achieve illumination normalization. Throughout the process, the transformation parameters of each channel are kept consistent to ensure that the color saturation of the image does not change significantly, thus avoiding color distortion introduced by illumination adjustments.

[0039] Using the above method, the adaptive illumination normalization layer can quickly complete the illumination normalization processing of a single frame image with minimal impact on computational performance, while effectively improving the target detection accuracy in scenarios with changing illumination.

[0040] Optionally, the operation of inputting the fused feature map into the channel attention mechanism module and outputting the enhanced feature map includes: after inputting the fused feature map into the channel attention mechanism module, compressing the two-dimensional features of each channel of the fused feature map into a scalar through global average pooling to obtain a channel-level global feature description; learning the importance weight of each channel in the channel-level global feature description through two fully connected layers and a sigmoid activation function; and multiplying the importance weight of each channel with the fused feature map channel by channel to obtain the enhanced feature map.

[0041] Specifically, the channel attention mechanism module employs a lightweight Squeeze-and-Excitation (SE) structure, designed to adaptively recalibrate the feature responses along the channel dimensions. This is achieved when the fused feature map... After inputting this module, a compression (Squeeze) operation is first performed: the two-dimensional spatial features of each channel are compressed into a real number through global average pooling, generating a channel-level statistical vector. This vector aggregates global spatial information, enabling subsequent layers to utilize the overall receptive field. The c-th element of the vector is calculated as: ; in, Let H be the feature value of the c-th channel at position (i,j), and let H and W be the height and width of the feature map, respectively.

[0042] The excitation operation then proceeds: a nonlinear transformation of the vector z is performed through two fully connected layers to learn the normalized weights for each channel. The first fully connected layer compresses the number of channels to C / r (typically r=16), and after ReLU activation, the second fully connected layer restores it to the original number of channels C. Finally, the output is mapped to the (0,1) interval using the Sigmoid function to obtain the channel weight vector. The entire process can be represented as: ; in, and This is the weight matrix of the fully connected layer. It is the ReLU activation function. This is the Sigmoid activation function. The weight vector reflects the importance of different channels to the current task; for example, channels related to the target contour and edges will be given higher weights.

[0043] Finally, a reweighting operation is performed: the original feature map F is multiplied channel by channel by the weight vector s to obtain the enhanced feature map. ,Right now .in, This is the feature map after recalibration of the c-th channel. Let be the weight of the c-th channel.

[0044] Through this mechanism, the network can dynamically enhance its response to key features and suppress irrelevant or redundant information, thereby improving detection accuracy. This module has low computational cost and is easily embedded into existing network structures, effectively improving the feature representation capabilities for complex backgrounds and multi-scale targets in indoor robot target detection tasks.

[0045] Optionally, the operation of calculating the first 3D target bounding box information of the target object in the robot's own coordinate system based on the 2D target bounding box includes: determining the ground center point of the 2D target bounding box, and calculating the depth information of the target object in the camera coordinate system based on the pixel coordinates of the ground center point and camera intrinsic parameters; calculating the three-dimensional position coordinates and deflection angle of the target object in the camera coordinate system based on the depth information, the camera intrinsic parameters, and a preset target object size, and obtaining the 3D target bounding box information of the target object in the camera coordinate system by combining the preset target object size; and transforming the 3D target bounding box information in the camera coordinate system to the robot's own coordinate system based on the rotation matrix and translation matrix between the camera coordinate system and the robot's own coordinate system to obtain the first 3D target bounding box information.

[0046] Specifically, after obtaining the 2D target bounding box, it needs to be mapped into 3D space to obtain a preliminary 3D target bounding box. First, the ground center point of the target bounding box is determined: assuming the target object is in contact with the ground (e.g., the bottom of an obstacle), the midpoint of the bottom edge of the target detection bounding box is taken as the ground center point. The bounding box information of the target detection is as follows: , Represents the two-dimensional coordinates of the top left corner of the rectangle. This represents the two-dimensional coordinates of the bottom right corner of the rectangle, with the pixel coordinates of the ground center point being (x3, y3). , .

[0047] Known The focal length of the monocular camera is given, and the coordinates of the image center point are given. , and Indicates each pixel in shaft and Physical dimensions along the axis; dimensions of the indoor robot are Preset the size of the object in front based on the type of object in front. , This indicates the width (left-right dimension) of the object in front. Indicates the height (vertical dimension) of the object in front. This indicates the length (length in the front-to-back direction) of the object in front.

[0048] Since the coordinates (x3, y3) in the pixel coordinate system need to be converted to coordinates in the image coordinate system The units are converted to millimeters (mm), and the corresponding relationships are as follows: ; Among them, after conversion, the height of the foreground object image Using similar triangles to satisfy From the detected bounding box information, the straight-line distance (i.e., depth information) between the object in front and the robot can be calculated as D. The depth information D is the distance of the target object along the optical axis in the camera coordinate system.

[0049] After measuring the straight-line distance D, a coordinate transformation is performed to the camera coordinate system, and the point information in the image coordinate system is obtained. Points in the camera coordinate system The correspondence is as follows: ; in, Here are the depth coordinates of the target object in the camera coordinate system. The horizontal axis is... The coordinates are in the vertical direction; The camera mounting height (vertical distance from the ground to the camera's optical center). The physical meaning of the calculation formula is: the Z coordinate of the target point is equal to the camera installation height minus half the height of the object in front, that is, the vertical position of the bottom center point of the object in front in the camera coordinate system.

[0050] at this time, This refers to the three-dimensional coordinates of the target object's ground center point in the camera coordinate system. This is combined with the preset target object dimensions. and initial deflection angle This allows us to obtain the 3D bounding box information of the target object in the camera coordinate system. .

[0051] To fuse with radar data, the 3D bounding box information in the camera coordinate system needs to be transformed to the robot's own coordinate system. Given that the rotation matrix from the camera to the robot is R and the translation matrix is ​​T, the transformed first 3D bounding box information is... ,Right now .

[0052] in, Let be the three-dimensional position coordinates of the target object in the robot's own coordinate system. The directional component represents the depth information along the robot's forward direction. This completes the process of converting the 2D target bounding box into the first 3D target bounding box in the robot's own coordinate system. This aligns the visual perception results with the robot's own spatial reference frame, laying the foundation for subsequent fusion with radar data.

[0053] Optionally, the operation of clustering the point cloud data to obtain at least one point cloud cluster and transforming the point cloud cluster to the robot's own coordinate system includes: performing clustering processing on the point cloud data using a clustering algorithm to obtain at least one point cloud cluster; for each point cloud cluster, transforming the coordinates of each point cloud in the point cloud cluster from the radar coordinate system to the robot's own coordinate system according to the rotation matrix and translation matrix between the lidar and the robot's own coordinate system.

[0054] Specifically, the raw point cloud acquired by lidar contains a large number of three-dimensional points, each composed of its spatial coordinates (x, y, z) and attributes such as reflection intensity. To extract independent object candidates from the discrete point cloud, clustering is required. First, the point cloud is preprocessed, such as removing ground points and isolated noise points, to reduce computation and improve clustering accuracy. Then, a clustering algorithm based on Euclidean distance is used: a distance threshold is set. For any point, search its radius. If any neighboring points exist within a given cluster, they are grouped into the same cluster. Then, the cluster is recursively expanded until no new points are added. This results in several point cloud clusters, each representing an independent object or obstacle in space.

[0055] After clustering, all point cloud coordinates need to be unified to the robot's own coordinate system. Given the installation position and orientation of the LiDAR relative to the robot, the rotation matrix can be obtained through calibration. Translation vector For each point in each point cloud cluster. The conversion formula is: The transformed point coordinates This means that the robot is located in its own coordinate system, sharing the same spatial reference as the image processing result. This ensures that subsequent matching and fusion operations can be performed within the same coordinate system, guaranteeing spatial consistency.

[0056] Optionally, the operation of updating the depth information and deflection angle in the first 3D target bounding box information using point cloud data from the successfully matched point cloud clusters to obtain the second 3D target bounding box information includes: calculating the distance from all point clouds in the successfully matched point cloud clusters to the origin of the robot's own coordinate system, and using the maximum distance value as the updated depth information; calculating the updated deflection angle of the target object relative to the robot's own coordinate system based on the two points with the smallest and largest x-coordinates in the successfully matched point cloud clusters; replacing the depth information and deflection angle in the first 3D target bounding box information with the updated depth information and the updated deflection angle, and combining them with the unupdated three-dimensional position coordinates and size parameters in the first 3D target bounding box information to obtain the second 3D target bounding box information.

[0057] Specifically, when a point cloud cluster successfully matches the first 3D target bounding box calculated by vision, it indicates that the cluster corresponds to the same physical target, and the point cloud in the cluster provides more accurate spatial location information than the visual estimation. Therefore, the point cloud data in this cluster can be used to correct the depth information and yaw angle in the first 3D target bounding box.

[0058] The depth information update operation is as follows: Depth information is defined as the farthest distance of the target object along the robot's forward direction (X-axis) to ensure the safety of obstacle avoidance decisions. For successfully matched point cloud clusters, the depth information is calculated for each point. Distance to the origin of the robot's own coordinate system And find the maximum distance among all points. Take this maximum value as the updated depth information. Since the farthest point in point cloud clustering represents the farthest extension of the target object in the depth direction, using this value can ensure a safety margin.

[0059] The deflection angle update operation is as follows: deflection angle This describes the orientation of the target object relative to the robot's direction of travel. The deflection angle is estimated using the two points with the smallest and largest x-coordinates (i.e., the leftmost and rightmost points) from the point cloud cluster. Let the leftmost point be... The rightmost point is Calculate the difference between these two points on the horizontal (Y-axis). and the difference in depth (X-axis). The deflection angle can then be approximated as... This angle reflects the orientation of the target object relative to the robot's X-axis; a positive value indicates a rightward deflection, and a negative value indicates a leftward deflection.

[0060] Finally, use the updated depth information and deflection angle Replace the corresponding original values ​​in the first 3D target bounding box information, while keeping other parameters (such as the Y and Z components in the 3D position coordinates and the size parameters) unchanged, to obtain the second 3D target bounding box information. .

[0061] This ensures that the final output 3D target box retains both the semantic category and approximate size obtained from visual recognition, and integrates the precise spatial information provided by LiDAR, significantly improving the overall accuracy and robustness of the perception system.

[0062] Furthermore, based on the second 3D target bounding box information, the operation of generating control commands for controlling the movement of the indoor robot includes the following steps: 1. Input information: 3D target detection bounding box information for each detected object. ,in For depth information, The horizontal distance. This is the deflection angle.

[0063] 2. Safe distance calculation: Calculate the safe distance based on the type of the target object and the current indoor robot speed. When the target is a stationary obstacle, Where v is the current driving speed, This is the maximum braking acceleration. When the target is a moving object... ,in For relative velocity, This represents the system response time.

[0064] 3. Collision Risk Assessment: Compare with current depth information and safe distance : like If the condition is deemed safe, the current driving strategy will be maintained. like If the warning status is not met, a deceleration command will be sent. like If the situation is deemed an emergency, an emergency braking command will be sent.

[0065] 4. Lateral obstacle avoidance decision: based on lateral distance and deflection angle Calculate the obstacle avoidance path: like If the lateral space is sufficient, the current driving direction can be maintained. like Then calculate the obstacle avoidance path and send a turning command; in, For the width of the indoor robot, The width of the target object. A safety buffer distance is reserved.

[0066] 5. Command Generation: The control module converts the decision results into speed and direction commands, which are then sent to the robot's power module to control the robot's speed and direction.

[0067] In summary, this application proposes a depth matching method that integrates monocular vision and LiDAR point cloud data. This method breaks through the limitations of traditional single-sensor approaches by fusing monocular images and LiDAR point cloud data, fully leveraging the advantages of both to achieve accurate ranging and blind spot elimination, effectively improving the accuracy and robustness of indoor robot perception. Furthermore, it innovatively projects the 2D detection box from the monocular camera onto a 3D coordinate space and matches it with the point cloud data, effectively avoiding the multi-target occlusion mismatches caused by traditional 3D point cloud projection onto 2D images, thus improving the accuracy and stability of target recognition.

[0068] Furthermore, according to this embodiment, a storage medium is also provided. The storage medium includes a stored program, wherein, during program execution, a processor executes any of the methods described above.

[0069] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0070] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0071] Example 2

[0072] Figure 5 An indoor robot perception device based on the dynamic fusion of monocular vision and radar data according to this embodiment is shown, which corresponds to the method described in Embodiment 1. (Reference) Figure 5 As shown, the device includes: a multimodal data acquisition module 510, used to acquire image data collected by a monocular camera and point cloud data collected by a lidar; the monocular camera and the lidar are mounted on an indoor robot; an image data processing module 520, used to input the image data into a pre-trained target detection model, output a 2D target bounding box, and calculate the first 3D target bounding box information of the target object in the robot's own coordinate system based on the 2D target bounding box; the first 3D target bounding box information includes at least the three-dimensional position coordinates, size parameters, and deflection angle of the target object; the X-direction component in the three-dimensional position coordinates represents the target... The system includes: depth information of the object; a point cloud data processing module 530, used to perform clustering processing on the point cloud data to obtain at least one point cloud cluster, and to convert the point cloud cluster to the robot's own coordinate system; an information update module 540, used to match the converted point cloud cluster with the first 3D target box information, and when the match is successful, to update the depth information and deflection angle in the first 3D target box information using the point cloud data in the point cloud cluster to obtain the second 3D target box information; and a control command generation module 550, used to generate control commands for controlling the movement of the indoor robot based on the second 3D target box information.

[0073] It should be noted that the indoor robot perception device based on the dynamic fusion of monocular vision and radar data provided in this embodiment can realize all the functions and steps in the above method embodiments, solve the same technical problems, and achieve the same technical effects. The similarities will not be repeated here.

[0074] Therefore, according to this embodiment, this application achieves a balance between ranging accuracy and target recognition capability by dynamically fusing vision and radar. While fully leveraging the advantages of monocular vision target recognition, it utilizes radar point cloud data to precisely correct the depth information and deflection angle calculated by vision, effectively improving the accuracy and robustness of indoor robot environmental perception in complex environments. This solves the technical problem in existing technologies where single-sensor-based indoor robot perception methods struggle to balance ranging accuracy and target recognition capability, failing to meet the high-precision, high-robustness perception requirements in complex indoor environments.

[0075] Example 3

[0076] Figure 6An indoor robot perception device based on the dynamic fusion of monocular vision and radar data according to this embodiment is shown, which corresponds to the method described in Embodiment 1. (Reference) Figure 6 As shown, the device includes: a processor 610; and a memory 620 connected to the processor 610, used to provide the processor 610 with instructions for processing the following steps: acquiring image data collected by a monocular camera and point cloud data collected by a lidar; the monocular camera and the lidar are mounted on an indoor robot; inputting the image data into a pre-trained target detection model, outputting a 2D target bounding box, and calculating a first 3D target bounding box information of the target object in the robot's own coordinate system based on the 2D target bounding box; the first 3D target bounding box information includes at least the three-dimensional position coordinates and size of the target object. The parameters and deflection angle; the X-direction component in the three-dimensional position coordinates represents the depth information of the target object; the point cloud data is clustered to obtain at least one point cloud cluster, and the point cloud cluster is transformed to the robot's own coordinate system; the transformed point cloud cluster is matched with the first 3D target bounding box information, and when the match is successful, the depth information and deflection angle in the first 3D target bounding box information are updated using the point cloud data in the point cloud cluster to obtain the second 3D target bounding box information; and control commands for controlling the movement of the indoor robot are generated based on the second 3D target bounding box information.

[0077] It should be noted that the indoor robot perception device based on the dynamic fusion of monocular vision and radar data provided in this embodiment can realize all the functions and steps in the above method embodiments, solve the same technical problems, and achieve the same technical effects. The similarities will not be repeated here.

[0078] Therefore, according to this embodiment, this application achieves a balance between ranging accuracy and target recognition capability by dynamically fusing vision and radar. While fully leveraging the advantages of monocular vision target recognition, it utilizes radar point cloud data to precisely correct the depth information and deflection angle calculated by vision, effectively improving the accuracy and robustness of indoor robot environmental perception in complex environments. This solves the technical problem in existing technologies where single-sensor-based indoor robot perception methods struggle to balance ranging accuracy and target recognition capability, failing to meet the high-precision, high-robustness perception requirements in complex indoor environments.

[0079] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0080] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0081] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0082] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0084] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0085] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An indoor robot perception method based on dynamic fusion of monocular vision and radar data, characterized in that, include: The system acquires image data from a monocular camera and point cloud data from a lidar; the monocular camera and the lidar are mounted on an indoor robot. The image data is input into a pre-trained target detection model, which outputs a 2D bounding box. Based on the 2D bounding box, the first 3D bounding box information of the target object in the robot's own coordinate system is calculated. The first 3D bounding box information includes at least the three-dimensional position coordinates, size parameters, and deflection angle of the target object. The X-direction component in the three-dimensional position coordinates represents the depth information of the target object. The point cloud data is clustered to obtain at least one point cloud cluster, and the point cloud cluster is transformed to the robot's own coordinate system. Each point cloud cluster after conversion is matched with the first 3D target box information. When there is a successfully matched point cloud cluster, the depth information and deflection angle in the first 3D target box information are updated using the point cloud data in the successfully matched point cloud cluster to obtain the second 3D target box information. Based on the second 3D target bounding box information, control commands for controlling the movement of the indoor robot are generated.

2. The method according to claim 1, characterized in that, The target detection model is an improved YOLOv8 model. The improved YOLOv8 model introduces an adaptive illumination normalization layer at the input of the backbone network, adds a multi-scale feature enhancement module in the neck network, and introduces a channel attention mechanism module in front of the detection head. Furthermore, the operation of inputting the image data into a pre-trained object detection model and outputting a 2D object bounding box includes: The image data is input into the adaptive illumination normalization layer to perform illumination normalization processing on the image, and the illumination-normalized image is output. The image after illumination normalization is input into the backbone network of the improved YOLOv8 model for feature extraction, and feature maps of different scales are output. The feature maps of different scales are input into the multi-scale feature enhancement module. The feature maps of different scales are enhanced by parallel convolutional structures, and the output feature maps of each branch are adaptively weighted and fused to output the fused feature map. The fused feature map is input into the channel attention mechanism module, and the enhanced feature map is output. The enhanced feature map is input into the head network of the improved YOLOv8 model for object detection, and the corresponding 2D object bounding box is output.

3. The method according to claim 2, characterized in that, The operation of inputting the image data into the adaptive illumination normalization layer, performing illumination normalization processing on the image, and outputting the illumination-normalized image includes: Histogram statistics are performed on the three RGB channels of the image data to calculate the brightness distribution characteristics of each channel, including the mean, variance, and peak position. Based on the brightness distribution characteristics, the brightness gain coefficient and brightness offset are dynamically generated; The image data is subjected to illumination normalization processing according to the following formula to obtain an illumination-normalized image, wherein the color saturation of the image is kept unchanged while adjusting the brightness: ； in, The input image data, The image is after illumination normalization. The brightness gain coefficient is... This refers to the brightness offset.

4. The method according to claim 2, characterized in that, The operation of inputting the fused feature map into the channel attention mechanism module and outputting the enhanced feature map includes: After the fused feature map is input into the channel attention mechanism module, the two-dimensional features of each channel of the fused feature map are compressed into a scalar by global average pooling to obtain a channel-level global feature description. The importance weights of each channel in the channel-level global feature description are learned through two fully connected layers and a sigmoid activation function. The importance weight of each channel is multiplied channel by channel with the fused feature map to obtain the enhanced feature map.

5. The method according to claim 1, characterized in that, The operation of obtaining the first 3D target bounding box information of the target object in the robot's own coordinate system based on the 2D target bounding box includes: Determine the ground center point of the 2D target bounding box, and calculate the depth information of the target object in the camera coordinate system based on the pixel coordinates of the ground center point and the camera intrinsic parameters; Based on the depth information, the camera intrinsic parameters, and the preset target object size, the three-dimensional position coordinates and deflection angle of the target object in the camera coordinate system are calculated, and combined with the preset target object size, the 3D target bounding box information of the target object in the camera coordinate system is obtained. Based on the rotation and translation matrices between the camera coordinate system and the robot's own coordinate system, the 3D target bounding box information in the camera coordinate system is transformed to the robot's own coordinate system to obtain the first 3D target bounding box information.

6. The method according to claim 1, characterized in that, The operation of clustering the point cloud data to obtain at least one point cloud cluster and converting the point cloud cluster to the robot's own coordinate system includes: The point cloud data is clustered using a clustering algorithm to obtain at least one point cloud cluster; For each point cloud cluster, the coordinates of each point cloud in the cluster are transformed from the radar coordinate system to the robot's own coordinate system based on the rotation and translation matrices between the LiDAR and the robot's own coordinate system.

7. The method according to claim 1, characterized in that, The operation of updating the depth information and yaw angle in the first 3D target bounding box information using the point cloud data from the successfully matched point cloud clusters to obtain the second 3D target bounding box information includes: Calculate the distance from all point clouds in the successfully matched point cloud cluster to the origin of the robot's own coordinate system, and use the maximum distance value as the updated depth information; Based on the two points with the smallest and largest x-coordinates in the successfully matched point cloud cluster, calculate the updated deflection angle of the target object relative to the robot's own coordinate system; Using the updated depth information and the updated deflection angle, the depth information and deflection angle in the first 3D target bounding box information are replaced, and combined with the unupdated three-dimensional position coordinates and size parameters in the first 3D target bounding box information, the second 3D target bounding box information is obtained.

8. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, the method described in any one of claims 1 to 7 is performed by a processor.

9. An indoor robot perception device based on dynamic fusion of monocular vision and radar data, characterized in that, include: A multimodal data acquisition module is used to acquire image data collected by a monocular camera and point cloud data collected by a lidar; the monocular camera and the lidar are mounted on an indoor robot. The image data processing module is used to input the image data into a pre-trained target detection model, output a 2D target bounding box, and calculate the first 3D target bounding box information of the target object in the robot's own coordinate system based on the 2D target bounding box. The first 3D target bounding box information includes at least the three-dimensional position coordinates, size parameters, and deflection angle of the target object. The X-direction component in the three-dimensional position coordinates represents the depth information of the target object. The point cloud data processing module is used to perform clustering processing on the point cloud data to obtain at least one point cloud cluster, and to transform the point cloud cluster to the robot's own coordinate system. The information update module is used to match the converted point cloud cluster with the first 3D target box information, and when the match is successful, to update the depth information and deflection angle in the first 3D target box information using the point cloud data in the point cloud cluster to obtain the second 3D target box information. as well as The control command generation module is used to generate control commands for controlling the movement of the indoor robot based on the second 3D target box information.

10. An indoor robot perception device based on dynamic fusion of monocular vision and radar data, characterized in that, include: processor; as well as A memory, connected to the processor, for providing the processor with instructions to perform the following processing steps: The system acquires image data from a monocular camera and point cloud data from a lidar; the monocular camera and the lidar are mounted on an indoor robot. The image data is input into a pre-trained target detection model, which outputs a 2D bounding box. Based on the 2D bounding box, the first 3D bounding box information of the target object in the robot's own coordinate system is calculated. The first 3D bounding box information includes at least the three-dimensional position coordinates, size parameters, and deflection angle of the target object. The X-direction component in the three-dimensional position coordinates represents the depth information of the target object. The point cloud data is clustered to obtain at least one point cloud cluster, and the point cloud cluster is transformed to the robot's own coordinate system. The converted point cloud clusters are matched with the first 3D target bounding box information. When the match is successful, the depth information and deflection angle in the first 3D target bounding box information are updated using the point cloud data in the point cloud clusters to obtain the second 3D target bounding box information. as well as Based on the second 3D target bounding box information, control commands for controlling the movement of the indoor robot are generated.

Images