A robot complex scene understanding and recognition method, system, device and medium

Abstract

The present disclosure relates to a robot complex scene understanding and recognition method, system, device and medium, and belongs to the technical field of robots. The robot complex scene understanding and recognition method comprises the following steps: acquiring visual data of a camera, three-dimensional point cloud distribution data of a laser radar and existing historical test information in a complex scene; determining initial weights of the laser radar and the camera according to the existing historical test information, and performing credibility calculation on the three-dimensional point cloud distribution data and the visual data to obtain corresponding credibility weights; calculating comprehensive weights according to the initial weights and the credibility weights, selecting a scene recognition model based on the comprehensive weights, and identifying object properties in a complex scene through the scene recognition model. The present disclosure effectively improves the recognition accuracy of robots for complex scenes by performing credibility calculation on laser radar data and visual data and selecting a scene recognition model according to the analysis results.

Description

Technical Field

[0001] This disclosure belongs to the field of robotics technology, and in particular relates to a method, system, device and medium for understanding and recognizing complex scenes in robots. Background Technology

[0002] With the increasing application of robots in the industrial and transportation sectors, improving robots' ability to understand and recognize complex scenarios has become a major direction in the field of robotics research. Currently, robot scene recognition capabilities mainly rely on LiDAR and visual data analysis. LiDAR generates dot matrix cloud maps to identify the distance information of objects in the scene, while visual data is used to collect information about objects in the scene. The fusion of these two methods can enhance the richness and accuracy of scene understanding.

[0003] Existing scene analysis techniques often train scenes by inputting a large amount of existing visual data, and build a large number of different scene models through pre-training. In practical applications, the existing models are combined with the dot matrix cloud map generated by LiDAR to improve the accuracy of scene recognition.

[0004] The existing technical solutions have the following drawbacks:

[0005] In existing technologies, the dot matrix cloud images generated by LiDAR and visual data have completely different descriptive forms and complex coupling correspondences, resulting in redundancy and inconsistency in scene recognition feature information. LiDAR has low detection accuracy for small objects at a distance, and visual data is easily affected by environmental conditions such as lighting, leading to a decrease in detection accuracy. Averaging multimodal data to calculate the shape of objects can reduce scene recognition accuracy due to the decrease in accuracy of a single modality.

[0006] Therefore, it is necessary to provide a new method, system, device, and medium for understanding and recognizing complex scenes in robots to solve the above-mentioned technical problems. Summary of the Invention

[0007] The purpose of this disclosure is to provide a method, system, device, and medium for understanding and recognizing complex scenes in robots in order to solve the above-mentioned problems.

[0008] This disclosure achieves the above objectives through the following technical solutions:

[0009] A method for robot understanding and recognition of complex scenes includes the following steps:

[0010] Acquire visual data from cameras, 3D point cloud distribution data from LiDAR, and existing historical test information in complex scenes;

[0011] The initial weights of the lidar and camera are determined based on the existing historical test information, and the credibility of the three-dimensional point cloud distribution data and the visual data is calculated to obtain the corresponding credibility weights.

[0012] A comprehensive weight is calculated based on the initial weight and the confidence weight. A scene recognition model is selected based on the comprehensive weight, and the object characteristics in complex scenes are identified through the scene recognition model.

[0013] As a further optimization of this disclosure, the visual data includes video frame rate, video bit rate, and RGB channel values ​​based on RAW format;

[0014] The three-dimensional point cloud distribution data includes the number of lidar beams, scanning frequency, number of points per loop per beam, installation height, vertical field of view, and travel speed.

[0015] The existing historical test information includes initial judgment information from lidar and cameras, which is determined based on robot testing and basic equipment parameters.

[0016] As a further optimization of this disclosure, the existing historical test information is combined with the device basic parameters of the lidar and camera to allocate initial weights for the lidar and camera.

[0017] As a further optimization of this disclosure, the credibility of the visual data is calculated to obtain its credibility weight, including:

[0018] The visual data is normalized.

[0019] Calculate the average saturation of the normalized visual data;

[0020] The confidence weight factor of the camera is calculated based on the average saturation.

[0021] As a further optimization of this disclosure, the credibility of the three-dimensional point cloud distribution data is calculated to obtain its credibility weight, including:

[0022] The point frequency is calculated based on the number of lidar beams, scanning frequency, and number of points per loop per beam in the three-dimensional point cloud distribution data.

[0023] The point cloud density is calculated based on the point frequency and the lidar installation height, vertical field of view, and travel speed in the three-dimensional point cloud distribution data.

[0024] The reliability weighting factor of the lidar is calculated based on the density of the point cloud map.

[0025] As a further optimization of this disclosure, a comprehensive weight is calculated based on the initial weight and the confidence weight, and a scene recognition model is selected based on the comprehensive weight, including:

[0026] Calculate the corresponding comprehensive weights based on the initial weights and the confidence weights of the camera and LiDAR, respectively;

[0027] A priority decision rule is used to determine the selection of the pre-trained model. When the absolute value of the difference between the comprehensive weight of the camera and the comprehensive weight of the LiDAR is greater than a preset threshold, the high-weight mode is selected as the main data source. When the absolute value of the difference between the comprehensive weight of the camera and the comprehensive weight of the LiDAR is not greater than the preset threshold, feature-level geometric semantic alignment technology is used to project and match the 3D bounding box generated by the LiDAR with the 2D bounding box. If the intersection-union ratio of the two in space is not less than a preset percentage, feature-level fusion is used to project the point cloud map onto the video image plane and comprehensively judge the object shape with visual features, and the corresponding pre-trained model is selected.

[0028] As a further optimization of this disclosure, the scene recognition model identifies object properties in complex scenes, including:

[0029] By using an existing multimodal joint pre-trained model and calling a cascaded fusion architecture, object candidate boxes are quickly generated using LiDAR. Within these candidate boxes, visual data is used to detect object categories. Finally, Kalman filtering is used to fuse multiple frames of data to predict object characteristics, enabling robots to analyze and understand complex scenes using LiDAR data and video data.

[0030] A robot complex scene understanding and recognition system, comprising:

[0031] The data collection module acquires visual data from cameras, 3D point cloud distribution data from LiDAR, and existing historical test information in complex scenes.

[0032] The preprocessing module determines the initial weights of the lidar and camera based on the existing historical test information, and performs credibility calculations on the three-dimensional point cloud distribution data and the visual data to obtain the corresponding credibility weights.

[0033] The scene judgment and understanding module calculates a comprehensive weight based on the initial weight and the confidence weight, selects a scene recognition model based on the comprehensive weight, and identifies the object characteristics in complex scenes through the scene recognition model.

[0034] As a further optimization of this disclosure, the visual data includes video frame rate, video bit rate, and RGB channel values ​​based on RAW format;

[0035] The three-dimensional point cloud distribution data includes the number of lidar beams, scanning frequency, number of points per loop per beam, installation height, vertical field of view, and travel speed.

[0036] The existing historical test information includes initial judgment information from lidar and cameras, which is determined based on robot testing and basic equipment parameters.

[0037] As a further optimization of this disclosure, the existing historical test information is combined with the device basic parameters of the lidar and camera to allocate initial weights for the lidar and camera.

[0038] As a further optimization of this disclosure, the preprocessing module performs credibility calculations on the visual data to obtain its credibility weights, including:

[0039] The visual data is normalized.

[0040] Calculate the average saturation of the normalized visual data;

[0041] The confidence weight factor of the camera is calculated based on the average saturation.

[0042] As a further optimization of this disclosure, the preprocessing module performs a credibility calculation on the three-dimensional point cloud distribution data to obtain its credibility weight, including:

[0043] The point frequency is calculated based on the number of lidar beams, scanning frequency, and number of points per loop per beam in the three-dimensional point cloud distribution data.

[0044] The point cloud density is calculated based on the point frequency and the lidar installation height, vertical field of view, and travel speed in the three-dimensional point cloud distribution data.

[0045] The reliability weighting factor of the lidar is calculated based on the density of the point cloud map.

[0046] As a further optimization of this disclosure, the scene judgment and understanding module calculates a comprehensive weight based on the initial weight and the confidence weight, and selects a scene recognition model based on the comprehensive weight, including:

[0047] Calculate the corresponding comprehensive weights based on the initial weights and the confidence weights of the camera and LiDAR, respectively;

[0048] A priority decision rule is used to determine the selection of the pre-trained model. When the absolute value of the difference between the comprehensive weight of the camera and the comprehensive weight of the LiDAR is greater than a preset threshold, the high-weight mode is selected as the main data source. When the absolute value of the difference between the comprehensive weight of the camera and the comprehensive weight of the LiDAR is not greater than the preset threshold, feature-level geometric semantic alignment technology is used to project and match the 3D bounding box generated by the LiDAR with the 2D bounding box. If the intersection-union ratio of the two in space is not less than a preset percentage, feature-level fusion is used to project the point cloud map onto the video image plane and comprehensively judge the object shape with visual features, and the corresponding pre-trained model is selected.

[0049] As a further optimization of this disclosure, the scene judgment and understanding module identifies object properties in complex scenes through the scene recognition model, including:

[0050] By using an existing multimodal joint pre-trained model and calling a cascaded fusion architecture, object candidate boxes are quickly generated using LiDAR. Within these candidate boxes, visual data is used to detect object categories. Finally, Kalman filtering is used to fuse multiple frames of data to predict object characteristics, enabling robots to analyze and understand complex scenes using LiDAR data and video data.

[0051] An electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0052] Memory, used to store computer programs;

[0053] The processor is used to execute the program stored in the memory to implement the robot's complex scene understanding and recognition method.

[0054] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the robot complex scene understanding and recognition method.

[0055] The beneficial effects of this disclosure are as follows:

[0056] This disclosure calculates the credibility weights of collected LiDAR and video data based on LiDAR point cloud density and video data frame saturation through a preprocessing module. This enables precise control of the credibility of multimodal data, provides data credibility support for multimodal fusion understanding and recognition of complex scenes, improves the accuracy of environmental object recognition model selection in complex scenes, and avoids the overall model selection accuracy being reduced due to the decrease in accuracy of a single modality.

[0057] This disclosure uses a scene judgment and understanding module to comprehensively calculate the initial weights and confidence weights of LiDAR and camera, accurately determine the weight of a single modality for model recognition, achieve more accurate and efficient model matching, reduce model matching time, and improve the overall system information utilization rate. Attached Figure Description

[0058] Figure 1 This is a flowchart of a method in an embodiment of this disclosure;

[0059] Figure 2 This is a flowchart illustrating a specific method in an embodiment of this disclosure;

[0060] Figure 3 This is a system structure block diagram of an embodiment of this disclosure;

[0061] Figure 4 This is a block diagram of the device structure in an embodiment of this disclosure. Detailed Implementation

[0062] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0063] like Figure 1 As shown, a method for robot understanding and recognition of complex scenes includes the following steps:

[0064] S1. Acquire visual data from cameras, 3D point cloud distribution data from LiDAR, and existing historical test information in complex scenes.

[0065] S2. Determine the initial weights of the lidar and camera based on the existing historical test information, and calculate the credibility of the three-dimensional point cloud distribution data and the visual data to obtain the corresponding credibility weights.

[0066] S3. Calculate the comprehensive weight based on the initial weight and the confidence weight, select the scene recognition model based on the comprehensive weight, and identify the object characteristics in the complex scene through the scene recognition model.

[0067] like Figure 2 In this embodiment, step S1 specifically includes:

[0068] Acquire 3D point cloud distribution data in complex scenes, acquire visual data in complex scenes, and obtain initial judgment information for LiDAR and cameras from existing historical test information, including the data itself, data timestamps, data change rate, and data type.

[0069] By monitoring changes in data sources in real time, the system acquires 3D point cloud distribution data in complex environments generated by LiDAR and visual data in complex scenes collected by cameras. Collection rules are set, and the collection frequency increases as the data change rate increases. When data changes, the data collection operation is automatically triggered.

[0070] The visual data includes video frame rate, video bit rate, and RGB channel values ​​based on RAW format;

[0071] The lidar data includes the number of lidar beams, scanning frequency, number of points per loop per beam, installation height, vertical field of view, and travel speed.

[0072] The initial judgment information from the lidar and camera is determined based on robot testing and basic equipment parameters.

[0073] Step S2 specifically includes:

[0074] The collected historical test information is evaluated to determine the initial weights of the LiDAR and visual data. These weights are then used to calculate the credibility of the collected 3D point cloud distribution data from the LiDAR and the visual data from the camera, and their corresponding credibility weights are calculated based on their credibility.

[0075] Based on existing historical test information and the hardware conditions of the LiDAR and camera, the initial weights are calculated as follows:

[0076] The formula for calculating the initial weights of a lidar system is as follows:

[0077] ;

[0078] in, Indicates the initial weights of the lidar. Indicates the actual detection distance. This represents the theoretical maximum detection range. This represents the theoretical minimum detection range. This indicates the actual point frequency, which is the number of points scanned per second by the lidar. Represents the theoretical maximum point frequency. Expressed as signal-to-noise ratio. This is expressed as the signal-to-noise ratio threshold, where the signal-to-noise ratio is the ratio of usable signal points to total signal points in the lidar point cloud image.

[0079] The formula for calculating the initial weight of the camera is as follows:

[0080] ;

[0081] in, This represents the initial weight of the camera. This is expressed as the camera resolution value. This represents the maximum resolution value of the camera under current technology. Indicates the dynamic range of the camera. This represents the maximum dynamic range of the camera under current technology. Indicates the camera frame rate. This represents the maximum frame rate value of the camera under current technology. This is represented as the camera's perspective. This represents the maximum field of view of the camera under current technology.

[0082] If the parameter information of the lidar and camera exists in the historical database, then their different initial weights are calculated by combining their previous test information;

[0083] If the parameter information of LiDAR and camera does not exist in the historical database, they are given the same initial weight, and their initial weights are gradually adjusted through subsequent tests.

[0084] The data credibility calculation unit obtains data credibility by performing point cloud density calculation and video data frame average saturation calculation on the three-dimensional point cloud distribution data collected by lidar and the visual data collected by camera, respectively.

[0085] The credibility weight calculation unit calculates the credibility of the data obtained by the data credibility calculation unit to obtain the corresponding credibility weight of the lidar and visual data in scene model recognition.

[0086] The process of calculating the credibility of visual data is as follows:

[0087] Let the collected video frame data sample set be... Where S represents the sample set, Indicates the first A set of samples, representing the RGB values ​​of the i-th pixel. Indicates the first The red channel value of each sample. Indicates the first The green channel value for each sample. Indicates the first Blue channel values ​​for each sample For sample index;

[0088] First, the data is subjected to relative scaling normalization, mapping the values ​​in the red, green, and blue channel parameter sets to the [0,1] interval. For the red channel values... The normalization formula is:

[0089] ;

[0090] in, Minimum value This is used to prevent the denominator from being zero. The superscript "norm" means the data after normalization. Similarly, for green channel values... The normalization formula is:

[0091] ;

[0092] Similarly, for the blue channel value The normalization formula is:

[0093] ;

[0094] After the data processing is complete, for the sample set The average saturation is calculated using the following formula:

[0095]

[0096] in, ; ;

[0097] N is the sample set The number of samples in the middle.

[0098] The reliability calculation process for lidar is as follows:

[0099] Let M be the number of lidar beams, F be the scanning frequency, and C be the number of points per loop per beam. Then the formula for calculating the point frequency A is as follows:

[0100] ;

[0101] Let the installation height of the lidar point be H, the vertical field of view be J, and the travel speed be V. Then the formula for calculating the point cloud density is as follows:

[0102] ;

[0103] Where D is the density of the point cloud map generated by the lidar.

[0104] Visual data credibility weighting factor The calculation formula is as follows:

[0105] ;

[0106] in The average saturation calculated for the current frame. , As an empirical threshold, when < At that time, forced To avoid interference from low saturation;

[0107] LiDAR point cloud density reliability weighting factor The calculation formula is as follows:

[0108] ;

[0109] Where D is the current point cloud density. , These are the minimum and maximum values ​​for point cloud density in the lidar parameter settings.

[0110] Step S3 specifically includes:

[0111] Let the initial weights of the lidar and the camera be respectively. , The formula for calculating the overall weight of lidar is as follows:

[0112] ;

[0113] The formula for calculating the overall weight of visual data is as follows:

[0114] ;

[0115] The overall credibility of the LiDAR and video data is calculated, and a priority selection strategy is applied when... The selection of pre-trained models is determined by forcibly choosing high-weight modes as the primary data source. The feature-level geometric semantic alignment technology is used to project and match the 3D bounding box generated by the LiDAR with the 2D bounding box. If the intersection-union ratio (IOU) of the two in space is ≥70%, feature-level fusion is used to project the point cloud map onto the video image plane and comprehensively judge the object shape with visual features, and select the corresponding pre-trained model.

[0116] The credibility of LiDAR and video data can be calculated through the above process. By incorporating the credibility into the scene model recognition process, the impact of the accuracy of multi-source data on the scene model recognition process can be effectively improved, thereby better recognizing scene models in practical applications.

[0117] The scene understanding unit is used to combine the pre-trained model selected by the scene judgment unit with visual data and LiDAR data to predict object characteristics. By using the existing multimodal joint pre-trained model and calling the cascaded fusion architecture, firstly, LiDAR is used to quickly generate object candidate boxes, then visual data is called within the candidate boxes to perform object category detection, and finally, Kalman filtering is used to fuse multi-frame data to predict object characteristics.

[0118] This approach effectively solves the problem that the reduced reliability of single-modal recognition in traditional multimodal recognition scenarios leads to a decrease in the robot's ability to understand and recognize complex scenes. It greatly improves the reliability of the robot's understanding and recognition of complex scenes, providing a reliable foundation for the robot's autonomous operation.

[0119] like Figure 3 As shown, embodiments of this disclosure provide a robot complex scene understanding and recognition system, including:

[0120] The data collection module includes a camera and a lidar. The lidar is used to acquire three-dimensional point cloud distribution data in complex scenes, and the camera is used to acquire visual data in complex scenes. At the same time, the initial weight judgment information of lidar and camera is obtained from existing historical test information, including the data itself, data timestamp, data change rate, and data type.

[0121] The preprocessing module is connected to the data collection module and is used to judge the historical test information collected by the data collection module, thereby determining the initial weights of the lidar and visual data. It is also used to calculate the credibility of the lidar three-dimensional point cloud distribution data collected by the data collection module and the visual data collected by the camera, and calculate their corresponding credibility weights based on their credibility.

[0122] The scene judgment and understanding module is connected to the preprocessing module. It receives the initial weights of the LiDAR and camera determined from the preprocessing module, as well as the calculated confidence weights of the LiDAR and camera. Based on the calculated comprehensive weights, it makes corresponding judgments to obtain the scene judgment strategy. The given scene judgment strategy determines the corresponding analysis and recognition model for different objects, and then comprehensively judges the object characteristics.

[0123] In this embodiment, a power maintenance company needs to carry out operations in a complex power scenario, but at the same time, it needs to perform live operations to ensure stable power supply. To ensure the safety of the operators, robots are used to replace manual operations. However, the accuracy of traditional robots in predicting the shape of objects by visual data analysis or LiDAR recognition is not high. Therefore, this disclosure proposes a robot complex scene understanding and recognition system based on LiDAR and visual data analysis.

[0124] In this scenario, the robot's data collection module collects 3D point cloud distribution data generated by the LiDAR in the complex environment and video data of the complex scene captured by the visual camera. It also collects data from the historical test database. The 3D point cloud distribution data includes the number of LiDAR beams, scanning frequency, number of points per circle per beam, installation height, traveling speed, and vertical field of view. The camera collects RGB channel values ​​in RAW format. The historical test data includes parameter information of the LiDAR and camera and historical test data. The collection rule is set: the collection frequency gradually increases as the rate of change of the LiDAR and camera reported data increases.

[0125] Next, the preprocessing module on the robot will perform calculations on the data acquired by the data collection module, which includes three parts: the initial weight allocation unit, the data credibility calculation unit, and the credibility weight recalculation unit.

[0126] The initial weight allocation unit calculates its initial weights based on existing historical test information and the hardware conditions of the LiDAR and camera. The calculation process is as follows:

[0127] The formula for calculating the initial weights of a lidar system is as follows:

[0128] ;

[0129] in, Indicates the initial weights of the lidar. Indicates the actual detection distance. This represents the theoretical maximum detection range. This represents the theoretical minimum detection range. This indicates the actual point frequency, which is the number of points scanned per second by the lidar. Represents the theoretical maximum point frequency. Expressed as signal-to-noise ratio. This is expressed as the signal-to-noise ratio threshold, where the signal-to-noise ratio is the ratio of usable signal points to total signal points in the lidar point cloud image.

[0130] The formula for calculating the initial weight of the camera is as follows:

[0131] ;

[0132] in, This represents the initial weight of the camera. This is expressed as the camera resolution value. This represents the maximum resolution value of the camera under current technology. Indicates the dynamic range of the camera. This represents the maximum dynamic range of the camera under current technology. Indicates the camera frame rate. This represents the maximum frame rate value of the camera under current technology. This is represented as the camera's perspective. This represents the maximum field of view of the camera under current technology.

[0133] The above calculation process is based on a standardized evaluation of the core performance parameters of the sensors. The initial weights for the LiDAR primarily measure the effectiveness of the detection range, reflecting detection capability through the ratio of the actual detection range to the theoretical maximum detection range. Simultaneously, the ratio of the actual point frequency to the theoretical maximum point frequency is used to evaluate scanning efficiency, adapting to actual signal attenuation scenarios. The initial weights for the camera start from resolution fidelity, quantifying imaging quality through the ratio of the resolution value to the current technological limit. This is combined with dynamic range, the ratio of the actual frame rate to the maximum frame rate, and the ratio of the actual viewing angle to the maximum viewing angle, thus forming a multi-dimensional perception evaluation system. This calculation scheme effectively eliminates the superposition interference of parameters with different dimensions through ratio normalization and introduces theoretical maximum values ​​(such as maximum detection range, maximum frame rate, etc.) to constrain the range of calculation results, ensuring that the weights conform to technological boundaries. The empirical coefficients in the scheme are based on industry testing standards.

[0134] If the parameter information of the lidar and camera exists in the historical database, then their different initial weights are calculated by combining their previous test information;

[0135] If the parameter information of LiDAR and camera does not exist in the historical database, they are given the same initial weight, and their initial weights are gradually adjusted through subsequent tests.

[0136] The initial weight allocation unit described above can effectively determine the initial weights of LiDAR and camera in scene recognition based on the performance of the initial hardware, thereby improving the accuracy of scene model judgment.

[0137] The data credibility calculation unit calculates the average saturation of the video frame and the point cloud density of the lidar based on the video frame data collected in the data collection module and the number of lidar beams, the number of points per coil, the scanning frequency, and the vertical field of view data.

[0138] Let the collected video frame data sample set be... Where S represents the sample set, Indicates the first A set of samples, representing the RGB values ​​of the i-th pixel. Indicates the first The red channel value of each sample. Indicates the first The green channel value for each sample. Indicates the first Blue channel values ​​for each sample For sample index;

[0139] First, the data is subjected to relative scaling normalization, mapping the values ​​in the red, green, and blue channel parameter sets to the [0,1] interval. For the red channel values... The normalization formula is:

[0140] ;

[0141] in, Minimum value This is used to prevent the denominator from being zero. The superscript "norm" means the data after normalization. Similarly, for green channel values... The normalization formula is:

[0142] ;

[0143] Similarly, for the blue channel value The normalization formula is:

[0144] ;

[0145] After the data processing is complete, for the sample set The average saturation is calculated using the following formula:

[0146] ;

[0147] in, , ;

[0148] N is the sample set The number of samples is used to calculate the average saturation of the video data frames.

[0149] Next, we calculate the point cloud density of the lidar. Let the number of lidar beams be M, the scanning frequency be F, and the number of points per circle per beam be C. Then, the formula for calculating the point frequency A is as follows:

[0150] ;

[0151] Let the installation height of the lidar point be H, the vertical field of view be J, and the travel speed be V. Then the formula for calculating the point cloud density is as follows:

[0152] ;

[0153] Where D is the density of the point cloud map generated by the lidar.

[0154] This disclosure proposes a robot complex scene understanding and recognition system based on LiDAR and visual data analysis. The method quantifies the reliability index of LiDAR point cloud data and visual image data by establishing a mathematical model, thereby achieving reliability assessment of different sensor data sources. Compared to existing technologies, this method fully utilizes the advantages of both LiDAR and visual image data by integrating them, achieving a deep understanding of complex scenes. The high-precision spatial information provided by LiDAR, combined with the rich texture and color features of visual images, enables the system to quickly and accurately capture key information in complex and ever-changing scenes. Furthermore, by quantifying the reliability index, this method assigns weights to different sensor data sources, further improving the accuracy and reliability of the judgment, and providing strong support for robot scene recognition in complex environments.

[0155] Based on the evaluation results, the system can dynamically adjust the weight distribution of each modality data in the target recognition model, thereby optimizing the selection mechanism of the model input features and improving the accuracy and stability of object prediction compared with existing technologies.

[0156] The credibility weight calculation unit receives data and calculates the density of the lidar point cloud and the saturation of the video data frame to perform weight calculation.

[0157] The process of calculating the credibility weight of video data is as follows:

[0158] ;

[0159] in The average saturation calculated for the current frame. , As an empirical threshold, when < At that time, forced To avoid interference from low saturation;

[0160] The calculation process for the weights in the lidar credibility calculation is as follows:

[0161] The reliability weighting factor of lidar point cloud density The calculation formula is as follows:

[0162] ;

[0163] Where D is the current point cloud density. , These are the minimum and maximum values ​​for point cloud density in the lidar parameter settings.

[0164] The scene judgment and understanding module receives the initial weights and confidence weights calculated by the preprocessing module, and performs comprehensive calculations to judge the scene recognition model. It includes a scene judgment unit and a scene understanding unit.

[0165] The scene judgment unit combines the initial judgment weight and confidence weight output by the preprocessing module, and calculates the comprehensive weight of the scene model judgment using a calculation formula. The calculation process is as follows:

[0166] Let the initial weights of the lidar and the camera be respectively. , The formula for calculating the overall weight of lidar is as follows:

[0167] ;

[0168] The formula for calculating the overall weight of visual data is as follows:

[0169] ;

[0170] The overall credibility of the LiDAR and video data is calculated, and a priority selection strategy is applied when... The selection of pre-trained models is determined by forcibly choosing high-weight modes as the primary data source. The feature-level geometric semantic alignment technology is used to project and match the 3D bounding box generated by the LiDAR with the 2D bounding box. If the intersection-union ratio (IOU) of the two in space is ≥70%, feature-level fusion is used to project the point cloud map onto the video image plane and comprehensively judge the object shape with visual features, and select the corresponding pre-trained model.

[0171] The credibility of LiDAR and video data can be calculated through the above process. By incorporating the credibility into the scene model recognition process, the impact of the accuracy of multi-source data on the scene model recognition process can be effectively improved, thereby better recognizing scene models in practical applications.

[0172] The scene understanding unit is used to combine the pre-trained model selected by the scene judgment unit with visual data and LiDAR data to predict object characteristics. By using the existing multimodal joint pre-trained model and calling the cascaded fusion architecture, firstly, LiDAR is used to quickly generate object candidate boxes, then visual data is called within the candidate boxes to perform object category detection, and finally, Kalman filtering is used to fuse multi-frame data to predict object characteristics.

[0173] In this way, the system effectively solves the problem that the reliability of single-modal recognition is reduced in traditional multimodal recognition scenarios, which in turn reduces the robot's ability to understand and recognize complex scenarios. It greatly improves the reliability of the robot's understanding and recognition of complex scenarios and provides a reliable recognition capability foundation for the realization of autonomous robot operation.

[0174] The implementation process of the functions and roles of each module in the above system is detailed in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0175] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0176] In the above embodiments, any number of modules can be combined into one module, or any one module can be split into multiple modules. Alternatively, at least some of the functionality of one or more of these modules can be combined with at least some of the functionality of other modules and implemented in one module. At least one of all modules can be at least partially implemented as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of all modules can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0177] See Figure 4 The electronic device provided in the embodiments of this disclosure includes a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120 and the memory 1130 communicate with each other through the communication bus 1140.

[0178] Memory 1130 is used to store computer programs;

[0179] The processor 1110, when executing the program stored in the memory 1130, implements the above-described robot complex scene understanding and recognition method.

[0180] The aforementioned communication bus 1140 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.

[0181] The communication interface 1120 is used for communication between the above-mentioned electronic device and other devices.

[0182] The memory 1130 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1130 may also be at least one storage device located remotely from the aforementioned processor 1110.

[0183] The processor 1110 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0184] Embodiments of this disclosure also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the robot complex scene understanding and recognition method described above.

[0185] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or it may exist independently and not assembled into the device / apparatus. The computer-readable storage medium carries one or more programs, which, when executed, implement the robot complex scene understanding and recognition method according to the embodiments of this disclosure.

[0186] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0187] The embodiments described above are merely examples of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these modifications and improvements all fall within the protection scope of this disclosure.

Claims

1. A method for robot complex scene understanding and recognition, characterized in that, The method comprises the following steps: acquiring visual data of a camera, three-dimensional point cloud distribution data of a laser radar and existing historical test information in a complex scene; the existing historical test information comprises initial judgment information of the laser radar and the camera, and the initial judgment information is determined based on robot testing and device basic parameters; determining initial weights of the laser radar and the camera according to the existing historical test information, and performing credibility calculation on the three-dimensional point cloud distribution data and the visual data to obtain corresponding credibility weights; calculating comprehensive weights according to the initial weights and the credibility weights, and selecting a scene recognition model based on the comprehensive weights, comprising: calculating corresponding comprehensive weights according to the initial weights and the credibility weights of the camera and the laser radar respectively; judging by using a priority decision rule, when an absolute value of a difference between the comprehensive weight of the camera and the comprehensive weight of the laser radar is greater than a preset threshold, selecting a high-weight mode as a main data source to determine a pre-trained model selection; when the absolute value of the difference between the comprehensive weight of the camera and the comprehensive weight of the laser radar is not greater than the preset threshold, using a feature-level geometric semantic alignment technology to project and match a 3D bounding box generated by the laser radar with a 2D bounding box, if a spatial intersection-over-union of the two is not less than a preset percentage, using feature-level fusion to project a point cloud image to a video image plane and comprehensively judge an object shape based on visual features, and selecting a corresponding pre-trained model; and identifying object properties in the complex scene through the scene recognition model. 2.The method of claim 1, wherein, The visual data comprises a video frame rate, a video code rate and RGB channel values based on a RAW format. The three-dimensional point cloud distribution data comprises a laser radar beam number, a scanning frequency, a point number per beam single circle, an installation height, a vertical field of view angle and a travel speed. 3.The method of claim 2, wherein, The existing historical test information combines device basic parameters of the laser radar and the camera to allocate initial weights of the laser radar and the camera. 4.The method of claim 1, wherein, The credibility calculation on the visual data comprises: normalizing the visual data; performing average saturation calculation on the normalized visual data; calculating a credibility weight factor of the camera based on the average saturation.

5. The method of claim 1, wherein, The credibility calculation on the three-dimensional point cloud distribution data comprises: calculating a point frequency based on the laser radar beam number, the scanning frequency and the point number per beam single circle in the three-dimensional point cloud distribution data; calculating a point cloud image density based on the point frequency and the laser radar installation height, the vertical field of view angle and the travel speed in the three-dimensional point cloud distribution data; calculating a credibility weight factor of the laser radar based on the point cloud image density. 6.The method of claim 1, wherein, The identification of object properties in the complex scene through the scene recognition model comprises: using an existing multi-modal joint pre-trained model to call a cascaded fusion architecture, using the laser radar to quickly generate an object candidate box, calling visual data in the object candidate box to detect an object category, and finally predicting object properties by fusing multiple frames of data through Kalman filtering, so as to enable a robot to analyze and understand a complex scene through laser radar data and video data.

7. A robotic complex scene understanding recognition system, comprising: The method comprises: The data collection module acquires visual data of a camera, three-dimensional point cloud distribution data of a laser radar and existing historical test information in a complex scene; the existing historical test information includes initial judgment information of the laser radar and the camera, and the initial judgment information is determined based on robot testing and device basic parameters; The preprocessing module determines initial weights of the laser radar and the camera according to the existing historical test information, and calculates the reliability of the three-dimensional point cloud distribution data and the visual data to obtain corresponding reliability weights; The scene judgment understanding module calculates comprehensive weights according to the initial weights and the reliability weights, and selects a scene recognition model based on the comprehensive weights, including: calculating corresponding comprehensive weights according to the initial weights and the reliability weights of the camera and the laser radar respectively; The priority decision rule is adopted for judgment, when the absolute value of the difference between the comprehensive weight of the camera and the comprehensive weight of the laser radar is greater than a preset threshold, a high weight mode is selected as a main data source to determine a pre-trained model selection; when the absolute value of the difference between the comprehensive weight of the camera and the comprehensive weight of the laser radar is not greater than the preset threshold, a feature-level geometric semantic alignment technology is adopted, a 3D bounding box generated by the laser radar is projected and matched with a 2D enclosing box, if the spatial intersection ratio of the two is not less than a preset percentage, a feature-level fusion is adopted to project a point cloud graph to a video image plane and a visual feature to comprehensively judge the shape of an object, and a corresponding pre-trained model is selected; and an object property in a complex scene is identified through the scene recognition model. 8.The robot complex scene understanding and recognition system of claim 7, wherein, The visual data includes a video frame rate, a video code rate and an RGB channel value based on a RAW format; The three-dimensional point cloud distribution data includes a laser radar beam number, a scanning frequency, a single circle point number per beam, an installation height, a vertical field of view angle and a travel speed. 9.The robot complex scene understanding and recognition system of claim 8, wherein, The existing historical test information combines device basic parameters of the laser radar and the camera to allocate initial weights of the laser radar and the camera. 10.The robot complex scene understanding and recognition system of claim 7, wherein, The preprocessing module calculates the reliability of the visual data to obtain its reliability weight, including: The visual data is normalized; The normalized visual data is calculated for average saturation; The reliability weight factor of the camera is calculated based on the average saturation. 11.The robot complex scene understanding and recognition system of claim 7, wherein, The preprocessing module calculates the reliability of the three-dimensional point cloud distribution data to obtain its reliability weight, including: The point frequency is calculated based on the laser radar beam number, the scanning frequency and the single circle point number per beam in the three-dimensional point cloud distribution data; The point cloud graph density is calculated based on the point frequency and the laser radar installation height, the vertical field of view angle and the travel speed in the three-dimensional point cloud distribution data; The reliability weight factor of the laser radar is calculated based on the point cloud graph density. 12.The robot complex scene understanding and recognition system of claim 7, wherein, The scene judgment understanding module identifies an object property in a complex scene through the scene recognition model, including: By using an existing multi-modal joint pre-training model, a cascaded fusion architecture is called, laser radar is used to quickly generate object candidate boxes, visual data is called in the object candidate boxes for object category detection, and finally multi-frame data is fused through Kalman filtering to predict object properties, so as to realize that a robot analyzes and understands a complex scene through laser radar data and video data.

13. An electronic device, comprising: The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; The memory is used for storing a computer program; The processor is used for executing the program stored in the memory, and realizes the robot complex scene understanding and recognition method in any one of claims 1-6.

14. A computer readable storage medium storing a computer program, wherein the computer program comprises program instructions configured to cause a processor to perform the method according to any one of claims 1 to 13. The computer program is executed by the processor to realize the robot complex scene understanding and recognition method in any one of claims 1-6.

Images