A method for modeling and positioning an augmented reality environment for a mobile robot

Abstract

The present invention relates to the field of mobile robot positioning processing technology, and in particular to an augmented reality environment modeling and positioning processing method for a mobile robot. The method comprises the following steps: obtaining sensor configuration monitoring parameters; collecting environmental image data of the monitoring sensor according to the sensor configuration monitoring parameters to generate environmental image data; modeling and processing an environmental measurement physical model based on the environmental image data to obtain an environmental measurement physical model; performing fused point cloud prior map analysis based on the environmental measurement physical model to obtain generated fused point cloud prior map data; analyzing inter-frame moving feature point matching data through fused point cloud prior map data; and performing mapping and positioning operations of the mobile robot based on the fused point cloud prior map data and inter-frame moving feature point matching data. The present invention realizes accurate mapping and positioning of the mobile robot in the dynamic process by expanding the monitoring information of the mobile robot.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile robot positioning processing, and in particular to an augmented reality environment modeling and positioning processing method for a mobile robot. Background Art

[0002] After decades of research, the problem of simultaneous localization and mapping (SLAM) of mobile robots has made great progress. SLAM (Simultaneous Localization and Mapping) is to study and solve the problem of simultaneous localization and mapping of robots in unfamiliar environments. It mainly studies three aspects, namely robot positioning, mapping and robot motion path planning. It is a hot issue in the current research of mobile and autonomous robots. In recent years, with the continuous development of computer technology and artificial intelligence technology, new development directions have emerged in the problem of SLAM of mobile robots. Sensors with higher accuracy and lower prices are used for SLAM problems, and multi-sensor fusion SLAM solutions are booming. However, the traditional mobile robot environment modeling and positioning processing methods are insufficient in image information collected by monitoring equipment, resulting in the fact that visual SLAM technology faces the problem of feature points in the actual environment, and the insufficient data information provided causes the interruption of image and feature point analysis, resulting in the failure of mapping and positioning of mobile robots. Summary of the invention

[0003] Based on this, the present invention provides an augmented reality environment modeling and positioning processing method for a mobile robot to solve at least one of the above technical problems.

[0004] To achieve the above object, a method for modeling and positioning an augmented reality environment of a mobile robot comprises the following steps:

[0005] Step S1: acquiring sensor configuration monitoring parameters; configuring monitoring parameters of the monitoring sensor according to the sensor configuration monitoring parameters, and collecting environmental image data through the monitoring sensor after the monitoring parameter configuration to generate environmental image data, wherein the monitoring sensor is a dual Kinect sensor and a laser radar sensor built into the mobile robot;

[0006] Step S2: synchronously process the image spatiotemporal features of the environmental image data to generate synchronous environmental image data; perform fitting matching and verification processing of the environmental physical centroid of the heterogeneous image based on the synchronous environmental image data to generate verification environmental physical centroid fitting matching data; perform environmental measurement physical model modeling processing based on the verification environmental physical centroid fitting matching data and the synchronous environmental image data to obtain the environmental measurement physical model;

[0007] Step S3: performing cross-modal environmental point cloud data fusion processing on the environmental measurement physical model to generate fused environmental point cloud data; performing fused point cloud prior map analysis based on the fused environmental point cloud data to obtain generated fused point cloud prior map data;

[0008] Step S4: Based on the preset RGBD-SLAM algorithm, the map color and depth data of the fused point cloud prior map data are analyzed to generate the fused point cloud prior map color data and the fused point cloud prior map depth data; according to the fused point cloud prior map color data, the inter-frame moving feature point matching analysis and processing of the time difference is performed to generate the inter-frame moving feature point matching data; based on the fused point cloud prior map depth data and the inter-frame moving feature point matching data, the fused point cloud prior map data is modeled to generate the inter-frame environment map model; according to the fused point cloud prior map depth data and the inter-frame moving feature point matching data, the mobile robot inter-frame motion feature analysis and processing is performed to generate the mobile robot inter-frame motion feature data; according to the mobile robot inter-frame motion feature data and the inter-frame environment map model, the mobile robot positioning data is analyzed to generate the mobile robot positioning data.

[0009] The present invention ensures that the dual Kinect sensors and laser radar sensors built into the mobile robot can operate with the most optimized settings when collecting environmental image data by acquiring sensor configuration monitoring parameters, adjusts the working parameters of the sensors according to specific scenarios and requirements, and improves the accuracy and stability of environmental data collection. The monitoring sensor configured according to the monitoring parameters can collect environmental image data at a specific collection frequency, resolution, and sensitivity, improves the efficiency and consistency of sensor data collection, ensures the quality and reliability of subsequent data processing, collects environmental image data through the configured monitoring sensor, obtains complete environmental data including RGB images of the dual Kinect sensors and image depth information of the laser sensor, can obtain multimodal data of the environment in real time when the mobile robot is running, and provides necessary input information for subsequent environmental modeling and positioning processing. The collected environmental image data is processed synchronously with time and space features to ensure that the data collected by different sensors are consistent in time and space, reduce the time sequence deviation and spatial difference between the data, and provide a consistent data basis for subsequent data fusion and analysis. Based on the synchronized environmental image data, the environmental physical centroid is fitted and matched and verified to ensure the accuracy and reliability of the fitting results. Through accurate centroid matching, the understanding and modeling capabilities of the physical structure of the environment are improved, laying the foundation for the subsequent establishment of the environmental physical model. Based on the verified environmental physical centroid fitting and matching data, the environmental measurement physical model is modeled and processed to generate an accurate environmental measurement model, including key information such as the position, shape and size of the object, providing an accurate description of the environment for the subsequent positioning and navigation tasks of the mobile robot. The environmental point cloud data of the environmental measurement physical model is fused across modalities, and the environmental point cloud data collected by different sensors is fused across modalities to integrate information of different perception modalities, such as RGB and depth, to improve the accuracy and completeness of environmental modeling and to more comprehensively understand the structure and characteristics of the environment. Generate a priori map based on fused environmental point cloud data, analyze and extract key environmental features and structures, provide prior knowledge of mobile robots in unknown environments, and help improve the accuracy and efficiency of path planning, obstacle avoidance and positioning. Perform color and depth data analysis on the fused point cloud priori map data to obtain the visual information and spatial structure of the map, provide the necessary visual reference and depth information for the subsequent SLAM algorithm, and improve the accuracy of positioning and navigation. Based on the inter-frame moving feature point matching data, analyze the motion relationship and changes between different frames, detect environmental changes and robot movement trends, and provide support for environmental perception and dynamic obstacle detection. Based on the inter-frame moving feature point matching data, perform dynamic environmental modeling on the fused point cloud priori map, update and optimize the environmental map in real time, so that the mobile robot can accurately locate and navigate in complex environments.According to the inter-frame motion data of the mobile robot, the robot's motion mode and behavior characteristics are analyzed, providing the real-time status and trend of the robot's motion, which provides an important basis for path planning and dynamic obstacle avoidance. Based on the inter-frame environmental map model and the robot's inter-frame motion data, the precise positioning data of the mobile robot is analyzed and updated, which improves the accuracy and stability of the robot's positioning and supports application scenarios with long-term operation in complex environments.

[0010] The beneficial effect of the present application lies in that the augmented reality environment modeling and positioning processing method of a mobile robot of the present invention targets the situation where visual SIAM technology faces the lack of feature points in the actual environment, and combines dual Kinect visual sensors with laser radars. On the one hand, the number of visual sensors is increased, which in turn expands the detection range of the visual sensors and increases the number of detected visual SLAM feature points in the actual environment. On the other hand, the laser radar has the characteristics of high measurement accuracy, large measurement range and its point cloud data is not affected by light conditions. The point cloud data of the laser that is not affected by visible light is used as prior map information to provide visual positioning and mapping, and then the visual SLAM process is carried out. At the same time, when feature points are scarce in the visual SLAM process due to the lack of light intensity and environmental texture, the robustness of the lidar point cloud to environmental texture and light is used to fuse the visual point cloud with the lidar point cloud to form fused point cloud data. The fused point cloud data is analyzed through the color and depth information of the point cloud, and the robust feature point data in the environment is used as a reference node to match the key data frame point cloud information to complete the positioning and mapping process, thereby overcoming the problem of positioning and mapping failure caused by insufficient or changed environmental conditions in visual SIAM technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of the steps of a method for modeling and positioning an augmented reality environment of a mobile robot according to the present invention;

[0012] Figure 2 for Figure 1 Detailed implementation steps of step S2 in the flowchart;

[0013] Figure 3 for Figure 1 Detailed implementation steps of step S4 in FIG.

[0014] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0015] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0016] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0017] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0018] To achieve this, please refer to Figures 1 to 3 The present invention provides an augmented reality environment modeling and positioning processing method for a mobile robot, comprising the following steps:

[0019] Step S1: acquiring sensor configuration monitoring parameters; configuring monitoring parameters of the monitoring sensor according to the sensor configuration monitoring parameters, and collecting environmental image data through the monitoring sensor after the monitoring parameter configuration to generate environmental image data, wherein the monitoring sensor is a dual Kinect sensor and a laser radar sensor built into the mobile robot;

[0020] Step S2: synchronously process the image spatiotemporal features of the environmental image data to generate synchronous environmental image data; perform fitting matching and verification processing of the environmental physical centroid of the heterogeneous image based on the synchronous environmental image data to generate verification environmental physical centroid fitting matching data; perform environmental measurement physical model modeling processing based on the verification environmental physical centroid fitting matching data and the synchronous environmental image data to obtain the environmental measurement physical model;

[0021] Step S3: performing cross-modal environmental point cloud data fusion processing on the environmental measurement physical model to generate fused environmental point cloud data; performing fused point cloud prior map analysis based on the fused environmental point cloud data to obtain generated fused point cloud prior map data;

[0022] Step S4: Based on the preset RGBD-SLAM algorithm, the map color and depth data of the fused point cloud prior map data are analyzed to generate the fused point cloud prior map color data and the fused point cloud prior map depth data; according to the fused point cloud prior map color data, the inter-frame moving feature point matching analysis and processing of the time difference is performed to generate the inter-frame moving feature point matching data; based on the fused point cloud prior map depth data and the inter-frame moving feature point matching data, the fused point cloud prior map data is modeled to generate the inter-frame environment map model; according to the fused point cloud prior map depth data and the inter-frame moving feature point matching data, the mobile robot inter-frame motion feature analysis and processing is performed to generate the mobile robot inter-frame motion feature data; according to the mobile robot inter-frame motion feature data and the inter-frame environment map model, the mobile robot positioning data is analyzed to generate the mobile robot positioning data.

[0023] The present invention ensures that the dual Kinect sensors and laser radar sensors built into the mobile robot can operate with the most optimized settings when collecting environmental image data by acquiring sensor configuration monitoring parameters, adjusts the working parameters of the sensors according to specific scenarios and requirements, and improves the accuracy and stability of environmental data collection. The monitoring sensor configured according to the monitoring parameters can collect environmental image data at a specific collection frequency, resolution, and sensitivity, improves the efficiency and consistency of sensor data collection, ensures the quality and reliability of subsequent data processing, collects environmental image data through the configured monitoring sensor, obtains complete environmental data including RGB images of the dual Kinect sensors and image depth information of the laser sensor, can obtain multimodal data of the environment in real time when the mobile robot is running, and provides necessary input information for subsequent environmental modeling and positioning processing. The collected environmental image data is processed synchronously with time and space features to ensure that the data collected by different sensors are consistent in time and space, reduce the time sequence deviation and spatial difference between the data, and provide a consistent data basis for subsequent data fusion and analysis. Based on the synchronized environmental image data, the environmental physical centroid is fitted and matched and verified to ensure the accuracy and reliability of the fitting results. Through accurate centroid matching, the understanding and modeling capabilities of the physical structure of the environment are improved, laying the foundation for the subsequent establishment of the environmental physical model. Based on the verified environmental physical centroid fitting and matching data, the environmental measurement physical model is modeled and processed to generate an accurate environmental measurement model, including key information such as the position, shape and size of the object, providing an accurate description of the environment for the subsequent positioning and navigation tasks of the mobile robot. The environmental point cloud data of the environmental measurement physical model is fused across modalities, and the environmental point cloud data collected by different sensors is fused across modalities to integrate information of different perception modalities, such as RGB and depth, to improve the accuracy and completeness of environmental modeling and to more comprehensively understand the structure and characteristics of the environment. Generate a priori map based on fused environmental point cloud data, analyze and extract key environmental features and structures, provide prior knowledge of mobile robots in unknown environments, and help improve the accuracy and efficiency of path planning, obstacle avoidance and positioning. Perform color and depth data analysis on the fused point cloud priori map data to obtain the visual information and spatial structure of the map, provide the necessary visual reference and depth information for the subsequent SLAM algorithm, and improve the accuracy of positioning and navigation. Based on the inter-frame moving feature point matching data, analyze the motion relationship and changes between different frames, detect environmental changes and robot movement trends, and provide support for environmental perception and dynamic obstacle detection. Based on the inter-frame moving feature point matching data, perform dynamic environmental modeling on the fused point cloud priori map, update and optimize the environmental map in real time, so that the mobile robot can accurately locate and navigate in complex environments.According to the inter-frame motion data of the mobile robot, the robot's motion mode and behavior characteristics are analyzed, providing the real-time status and trend of the robot's motion, which provides an important basis for path planning and dynamic obstacle avoidance. Based on the inter-frame environmental map model and the robot's inter-frame motion data, the precise positioning data of the mobile robot is analyzed and updated, which improves the accuracy and stability of the robot's positioning and supports application scenarios with long-term operation in complex environments.

[0024] As an embodiment of the present invention, refer to Figure 1 The above is a schematic diagram of the steps of an augmented reality environment modeling and positioning processing method for a mobile robot of the present invention. In this embodiment, the augmented reality environment modeling and positioning processing method for a mobile robot includes the following steps:

[0025] Step S1: acquiring sensor configuration monitoring parameters; configuring monitoring parameters of the monitoring sensor according to the sensor configuration monitoring parameters, and collecting environmental image data through the monitoring sensor after the monitoring parameter configuration to generate environmental image data, wherein the monitoring sensor is a dual Kinect sensor and a laser radar sensor built into the mobile robot;

[0026] In the embodiment of the present invention, the sensor configuration monitoring parameters are obtained, including the configuration parameters of the dual Kinect sensors and the configuration parameters of the laser radar sensor. The data collected by the Kinect camera is active ranging. It emits a near-infrared beam outward through its own infrared speckle emitter. When the beam contacts the obstacle, it is reflected back and received by the infrared receiver. Finally, the distance is calculated by calculating the collective relationship between the speckles reflected back. If there is an overlapping field of view between the two cameras, the infrared beams before the two cameras will interfere with each other when the depth distance is measured, resulting in unreliable environmental depth information. Therefore, the fields of view between the two Kinect cameras are designed to be non-overlapping and non-interfering. The position between the two cameras is fixed, firstly to ensure the non-overlapping field of view of the cameras, and secondly, the relative position of the cameras is fixed, which makes it easier for the system to complete mapping and determine its own posture. That is, the configuration parameters of the dual Kinect sensors are used to collect non-overlapping areas and complementary visual data, while the configuration parameters of the laser radar sensor are used to cover the environmental ranging information, prior map information, etc. of the area where the visual data collected by the dual Kinect sensors are located. The monitoring parameters of the monitoring sensors are configured according to the sensor configuration monitoring parameters. The monitoring sensors are the dual Kinect sensors and lidar sensors built into the mobile robot. The configured dual Kinect sensors and lidar sensors are used to collect environmental image data. Through real-time scanning and collection of sensors, the visual and lidar data in the environment are obtained and integrated into environmental image data.

[0027] Step S2: synchronously process the image spatiotemporal features of the environmental image data to generate synchronous environmental image data; perform fitting matching and verification processing of the environmental physical centroid of the heterogeneous image based on the synchronous environmental image data to generate verification environmental physical centroid fitting matching data; perform environmental measurement physical model modeling processing based on the verification environmental physical centroid fitting matching data and the synchronous environmental image data to obtain the environmental measurement physical model;

[0028] In an embodiment of the present invention, the environmental image data obtained from the dual Kinect sensors and the laser radar sensor are preprocessed, including noise removal, image correction, coordinate system conversion, etc., and spatiotemporal features are extracted from the preprocessed environmental image data. The spatiotemporal features may include dynamic changes in time series, visual feature points in space, motion trajectories of objects, etc. The environmental image data from different sensors are synchronously processed to ensure that the spatiotemporal features are aligned in time and space to generate synchronized environmental image data. The physical centroid points of objects in the environment are extracted from the synchronized environmental image data using image processing and computational geometry methods. These centroid points may correspond to the center position or important feature points of the object. The centroid point data obtained from the dual Kinect sensors and the laser radar sensor are fitted and matched, and the centroid point fitting matching is performed by computational geometry methods, such as pixel information between point clouds, i.e., Euclidean distance, the centroid points corresponding to different sensors are matched and fitted, and the corresponding integration is performed as initial centroid point fitting data, and the fitted and matched centroid point data are verified, such as unmatched centroid point outlier detection and geometric shape matching by RANSAC, so as to ensure the accuracy and reliability of the fitting. The verified centroid fitting matching data is integrated with the synchronized environmental image data to form a complete environmental measurement data set. Based on the integrated data set, the physical characteristics of the environment are modeled using mathematical modeling methods (such as statistical modeling, machine learning models, etc.). This may include modeling the shape, size, position, and movement trend of objects in the environment to obtain an environmental measurement physical model, including the physical model of the environment established under each sensor and the corresponding centroid matching information.

[0029] Step S3: performing cross-modal environmental point cloud data fusion processing on the environmental measurement physical model to generate fused environmental point cloud data; performing fused point cloud prior map analysis based on the fused environmental point cloud data to obtain generated fused point cloud prior map data;

[0030] In an embodiment of the present invention, corresponding environmental point cloud data is obtained from the environmental measurement physical model corresponding to the dual Kinect sensors and the laser radar sensor for preprocessing, including realizing the conversion of the coordinate system, removing abnormal points, aligning data and other operations through the external reference relationship between each sensor to ensure data quality and consistency. The environmental point cloud data from different sensors are fused. The fusion can be based on the geometric features of the point cloud, such as the position, normal direction, color and other information of the point, for example, the point cloud data is fused based on a feature matching fusion algorithm, such as the ICP (Iterative Closest Point) algorithm or a grid-based fusion method. Generate fused environmental point cloud data, which contains the fusion information of the point cloud data from the dual Kinect sensors and the laser radar sensor, providing a more comprehensive and accurate description of the environment. The analysis of the generated fused environment point cloud data involves operations such as geometric structure analysis of the prior map, feature point extraction, and classification of map areas. According to the analysis results, a prior map of the fused point cloud is constructed to provide spatial distribution information and appearance information of each object and structure in the environment, including the location of obstacles, the shape of walls, the layout of the room, etc. The generated fused point cloud prior map data provides a high-precision model of the environment and provides basic data support for subsequent robot navigation, path planning, target recognition and other tasks.

[0031] Step S4: Based on the preset RGBD-SLAM algorithm, the map color and depth data of the fused point cloud prior map data are analyzed to generate the fused point cloud prior map color data and the fused point cloud prior map depth data; according to the fused point cloud prior map color data, the inter-frame moving feature point matching analysis and processing of the time difference is performed to generate the inter-frame moving feature point matching data; based on the fused point cloud prior map depth data and the inter-frame moving feature point matching data, the fused point cloud prior map data is modeled to generate the inter-frame environment map model; according to the fused point cloud prior map depth data and the inter-frame moving feature point matching data, the mobile robot inter-frame motion feature analysis and processing is performed to generate the mobile robot inter-frame motion feature data; according to the mobile robot inter-frame motion feature data and the inter-frame environment map model, the mobile robot positioning data is analyzed to generate the mobile robot positioning data.

[0032] In the embodiment of the present invention, a preset RGBD-SLAM algorithm is applied to process the fused point cloud prior map data, and the algorithm simultaneously considers the color information (RGB color data) and depth information (distance data) of the map to extract the structure and features in the map, and obtains the processed map color data from the RGBD-SLAM algorithm, which reflects the color information of different areas in the environment, and also obtains the processed map depth data from the RGBD-SLAM algorithm, which represents the distance and position information of objects and structures in the environment. Based on the generated fused point cloud prior map color data, feature points moving between frames are extracted, and these feature points include key points of significant structures or objects in the environment, and matching analysis is performed on the feature points moving between frames using a time difference algorithm (such as an optical flow method) to determine their movement and changes between different frames. Based on the fused point cloud prior map depth data and the inter-frame moving feature point matching data, the fused point cloud prior map data is modeled for the inter-frame environment map, including integrating the motion and position information of the inter-frame moving feature points and the changes in the depth information reflected by the inter-frame moving feature point matching data into the fused point cloud prior map data to implement the update modeling of the environment map and form an inter-frame environment map model. According to the fused point cloud prior map depth data and the inter-frame moving feature point matching data, the inter-frame motion feature analysis of the mobile robot is performed, including the changes between the inter-frame moving feature points and the fused point cloud prior map depth data, so as to analyze the movement trajectory, speed change and movement direction of the robot in the environment. Combined with the inter-frame motion feature data of the mobile robot and the inter-frame environment map model, the positioning data of the mobile robot is analyzed, the positioning data of the mobile robot is generated, and the current position, posture and motion path of the robot are determined, so as to accurately obtain the continuous positioning information of the mobile robot and provide a basis for navigation and decision-making.

[0033] Preferably, step S1 comprises the following steps:

[0034] Step S11: Obtain dual Kinect sensor debugging monitoring parameters;

[0035] Step S12: performing dual Kinect visual overlap area correction configuration processing on the dual Kinect sensor debugging monitoring parameters to obtain dual Kinect sensor configuration monitoring parameters;

[0036] Step S13: performing dual Kinect visual coverage analysis according to dual Kinect sensor configuration monitoring parameters to generate dual Kinect visual coverage data;

[0037] Step S14: analyzing the configuration monitoring parameters of the laser radar sensor according to the dual Kinect visual coverage data to obtain the configuration monitoring parameters of the laser radar sensor;

[0038] Step S15: integrating the dual Kinect sensor configuration monitoring parameters and the laser radar sensor configuration monitoring parameters to obtain the sensor configuration monitoring parameters;

[0039] Step S16: configuring monitoring parameters of the monitoring sensor according to the sensor configuration monitoring parameters, and collecting environmental image data through the monitoring sensor after the monitoring parameter configuration to generate environmental image data.

[0040] The present invention obtains the debugging monitoring parameters of the dual Kinect sensor, and uses the debugging monitoring parameters to subsequently optimize the working performance of the sensor to ensure that it can operate effectively under various environmental conditions. The dual Kinect sensor debugging monitoring parameters are processed by dual Kinect visual overlap area correction configuration processing, and the visual overlap area correction configuration processing is performed to ensure that there is no visual overlap area between the dual Kinect sensors, and the optimal installation position and angle of each dual Kinect sensor are determined to maximize its visual coverage. Determining a suitable configuration can avoid visual overlap between the dual Kinect sensors, thereby ensuring that the overall visual system has a wider coverage range without repeated acquisition areas. Dual Kinect visual coverage analysis is performed according to the dual Kinect sensor configuration monitoring parameters. Through the visual coverage analysis, the visual coverage of the dual Kinect sensors in the working area is fully understood, which helps to optimize the arrangement and layout of sensors to maximize data acquisition efficiency. According to the dual Kinect visual coverage data, the lidar sensor configuration monitoring parameters are analyzed. After analyzing the dual Kinect visual coverage, the position and parameters of the lidar sensor can be optimized to ensure that the area monitored by the lidar sensor can cover the dual Kinect visual data to supplement the depth information of the dual Kinect visual data, realize effective collaboration between different sensors, improve overall perception capabilities, and maximize the advantages of different sensors, such as the long-range detection of lidar and the high-precision vision of Kinect, to comprehensively improve the comprehensiveness and accuracy of environmental data collection. The dual Kinect sensor configuration monitoring parameters and the lidar sensor configuration monitoring parameters are integrated into the sensor configuration monitoring parameters. The integration of the monitoring parameters of different sensors can comprehensively optimize the configuration of the overall sensor network, achieve the best data collection and processing effects, and dynamically adjust the sensor configuration in different work scenarios to cope with data collection challenges in complex environments. The monitoring parameters of the monitoring sensor are configured according to the sensor configuration monitoring parameters, and the environmental image data is collected through the monitoring sensor after the monitoring parameters are configured. By configuring the monitoring parameters, it is ensured that the monitoring sensor of the mobile robot can work as expected, and the quality and reliability of environmental image data collection are improved. Through intelligent analysis and automated processing, the configuration process of the monitoring sensor is simplified, the efficiency and real-time performance of data collection are improved, and the environmental image data can be used for subsequent analysis and processing in a timely manner. Compared with traditional single-sensor monitoring equipment, the images monitored in this part provide monitoring images with richer information, ensure the supplement of feature points when visual SLAM technology is missing, and expand the detection range of the sensor in disguise. The limitations of the field of view of a single Kinect sensor can be well solved.

[0041] In an embodiment of the present invention, the debugging and monitoring parameters of the dual Kinect sensor are obtained, including the internal settings and calibration parameters of the dual Kinect sensor. The dual Kinect sensor is connected to the debugging and monitoring device to ensure that the device can communicate with the sensor, and the corresponding software tool or API is used to read the current debugging and monitoring parameters from the dual Kinect sensor, including the resolution, depth detection range, frame rate, etc. of the camera, and the obtained parameters are recorded to ensure that the subsequent configuration and analysis steps can accurately use these parameters. According to the visual overlap area of ​​the dual Kinect sensor, the correction configuration is performed to optimize the visual coverage and data quality of the sensor, and the visual overlap area of ​​the dual Kinect sensor is analyzed using a visual analysis tool or algorithm, including identifying and quantifying the geometry, coverage and overlap of the overlap area. According to the analysis results of the overlap area, the internal parameters and configuration of the dual Kinect sensor are adjusted to maximize the coverage and reduce the influence of the overlapped part, which involves adjusting the installation angle, field of view setting or depth calibration of the sensor, and updating the configuration file or setting of the sensor to obtain the dual Kinect sensor configuration monitoring parameters, ensuring that the sensor can accurately capture the visual data in the environment and avoid the influence of the overlap area. Analyze the visual range covered by the configured dual Kinect sensor parameters, use the configured dual Kinect sensor to scan the environment to ensure that the entire area of ​​interest is covered, collect environmental image data from the dual Kinect sensor, use computer vision technology or image processing algorithms to analyze and quantify the visual coverage of the dual Kinect sensor, and generate dual Kinect visual coverage data, including generating a coverage map, coverage statistics, and boundary identification of the coverage area. Based on the analysis data of the dual Kinect visual coverage, analyze and optimize the configuration parameters of the lidar sensor to ensure that each sensor complements each other when working and improves the overall data collection effect. Compare and analyze the visual coverage data of the dual Kinect sensor with the scanning data of the lidar sensor to find overlapping and non-overlapping areas. According to the results of the comparative analysis, adjust the scanning angle, resolution or other related parameters of the lidar sensor to ensure that the area monitored by the dual Kinect sensor is covered and maximize the acquisition and coverage of environmental data. Update the configuration file or settings of the lidar sensor to ensure that the sensor can accurately capture the required environmental information in real-time monitoring to obtain the lidar sensor configuration monitoring parameters. Integrate the configuration monitoring parameters of the dual Kinect sensors and the LiDAR sensor to form a unified sensor configuration file or parameter set for subsequent environmental image data collection and monitoring operations. Combine the visual coverage data of the dual Kinect sensors and the monitoring parameter analysis results of the LiDAR sensor to ensure that the integrated parameters can be effectively coordinated and cooperated between different sensors.Generate the final sensor configuration file to obtain the sensor configuration monitoring parameters, including the detailed settings, calibration and monitoring parameters of all sensors, so as to be used in the subsequent real-time monitoring and data collection tasks. According to the sensor configuration monitoring parameters, configure the monitoring parameters of the monitoring sensor, and collect environmental image data through the monitoring sensor after the monitoring parameters are configured to generate environmental image data. According to the integrated sensor configuration monitoring parameters, configure the parameters of the dual Kinect sensor and the lidar sensor, including setting the working mode, resolution, scanning frequency, etc. of each sensor, to ensure that the sensor can work stably and accurately collect environmental image data in real-time monitoring. Apply the configured monitoring parameters to the actual dual Kinect sensor and lidar sensor to ensure that the sensor can work and collect environmental image data as expected. Use the configured sensor to collect environmental image data in real time. The sensor generates visual data and depth information containing objects, structures and scenes in the environment through scanning and detection. The collected environmental image data is sorted and stored to form a data set that can be used for subsequent analysis and application.

[0042] Preferably, step S12 comprises the following steps:

[0043] Step S121: performing dual Kinect vision test processing according to dual Kinect sensor debugging monitoring parameters to generate dual Kinect vision test data;

[0044] Step S122: performing dual Kinect vision test overlapping area analysis on the dual Kinect vision test data to generate dual Kinect vision test overlapping area data;

[0045] Step S123: performing dual Kinect visual overlapping area correction configuration processing on the dual Kinect sensor debugging monitoring parameters according to the dual Kinect visual test overlapping area data to obtain the dual Kinect sensor configuration monitoring parameters.

[0046] The present invention performs dual Kinect visual test processing according to dual Kinect sensor debugging monitoring parameters, evaluates the visual coverage of dual Kinect sensors at different positions and angles, determines the corresponding parameters of each monitoring parameter and the actual visual picture, facilitates subsequent adjustment, and generates visual test data that provides the visual effect of each dual Kinect sensor in a real environment, including information such as field of view angle, depth range and image quality. The dual Kinect visual test overlapping area analysis is performed on the dual Kinect visual test data to determine the visual overlapping area that may exist between the dual Kinect sensors, avoid repeated collection of the same environmental information, and identify the overlapping area to guide the adjustment of the installation position and viewing angle of the sensor to maximize the coverage of the overall visual system, while ensuring that each sensor is responsible for a unique visual task. According to the dual Kinect visual test overlapping area data, the dual Kinect visual overlapping area correction configuration processing is performed on the dual Kinect sensor debugging monitoring parameters to ensure that the dual Kinect sensors complement each other rather than overlap when working, effectively expand the coverage capacity of the overall visual system, and optimize the configuration parameters to improve the overall coordination efficiency of the system, ensure the information complementarity between sensors, and improve the stability and data accuracy of the system.

[0047] In the embodiment of the present invention, a dual Kinect visual test process is performed according to the dual Kinect sensor debugging monitoring parameters, the dual Kinect sensor is installed at a predetermined position, the overlapping area of ​​the field of view of the two sensors is analyzed, and in a stable environment, a marker or a test pattern is placed in the test environment, and these patterns should have obvious feature points and contrast, which is convenient for visual testing, and the dual Kinect sensor is started to collect real-time images and depth data. Use a corresponding driver or software tool to obtain the visual data of each sensor, store the collected image data in a computer, and generate dual Kinect visual test data, which includes the data of the respective images of the two Kinect sensors and their overlapping parts. Analyze the visual test data of the dual Kinect sensor to determine and quantify the overlapping area between the two sensors, use image processing algorithms, such as image matching and image stitching technology, to identify the overlapping area in the field of view of the dual Kinect sensor, and achieve it by detecting the same feature points or patterns in the two fields of view. Analyze the identified overlapping area, and calculate the size, shape and position of the overlapping area. Generate a data report of the overlapping area, including the coordinates and area of ​​the overlapping area and its impact on the image quality, and organize the analysis results into the overlapping area data of the dual Kinect visual test, including a visual overlapping area map and related statistical information. Adjust the dual Kinect sensor debugging parameters according to the overlapping area data of the dual Kinect visual test to optimize the visual coverage and reduce the impact of the overlapping part, analyze the nature and influencing factors of the overlapping area, such as the angle, position or field of view setting of the sensor, adjust the configuration parameters of the dual Kinect sensor, such as the viewing angle, distance, exposure setting, etc., reduce or optimize the overlapping area, which involves adjusting the physical installation position of the sensor or adjusting it through software settings. Re-perform the visual test process to verify whether the adjusted configuration improves the processing of the overlapping area, ensure that the visual coverage under the new parameters meets the requirements, integrate the corrected parameters, and generate the dual Kinect sensor configuration monitoring parameters for subsequent data collection and analysis.

[0048] Preferably, step S2 comprises the following steps:

[0049] Step S21: performing time sequence node and space node analysis according to the environment image data to obtain environment image time sequence node data and environment image space node data respectively;

[0050] Step S22: using the environment image temporal node data and the environment image spatial node data to perform image temporal and spatial feature synchronization processing on the environment image data to generate synchronized environment image data;

[0051] Step S23: performing environment real material centroid fitting and matching processing on the heterogeneous images based on the synchronous environment image data to generate environment real material centroid fitting and matching data;

[0052] Step S24: performing abnormal physical centroid point verification and elimination on the environment physical centroid point fitting matching data based on a preset RANSAC algorithm to obtain verification environment physical centroid point fitting matching data;

[0053] Step S25: Performing environment measurement physical model building processing based on the calibration environment physical centroid point fitting matching data and the synchronous environment image data to obtain the environment measurement physical model.

[0054] The present invention performs time series node and space node analysis based on environmental image data. Through time series analysis, the changing trend of environmental image data in time series can be captured to help understand the dynamic changes of environmental status. Spatial analysis provides the location and distribution information of environmental image data in three-dimensional space, which is used for modeling and analysis of spatial relationships. The environmental image data is synchronously processed with the time series node data and the space node data of the environmental image, and the time series and space node data are synchronously processed to ensure the consistency of the environmental image data in time and space, improve the accuracy and availability of the data, and the synchronous processing eliminates the existing time deviation and space dislocation, so that the environmental image data is analyzed and processed at the same latitude. Based on the synchronous environmental image data, the environmental physical centroid point fitting and matching processing of heterogeneous images is performed to generate environmental physical centroid point fitting and matching data, and the environmental physical centroid point data from different sensors are fitted and matched, which enhances the integrity and credibility of the data. By matching the physical centroid points, the location and shape information of important objects in the environment and the spatial correspondence of the physical centroid points monitored by each sensor can be accurately obtained, providing basic data for subsequent analysis and modeling. Based on the preset RANSAC algorithm, the environment physical centroid fitting matching data is checked and eliminated for abnormal physical centroid points. The use of the RANSAC algorithm can effectively identify and eliminate abnormal values ​​in the environment physical centroid data, and improve the accuracy and reliability of the data. That is, the RANSAC algorithm is used to analyze whether the physical centroids under different sensors have corresponding relationships. When there is no corresponding relationship, the abnormal physical centroids are eliminated. The cleaned verification data is more reliable and can accurately reflect the real position and shape of important objects in the environment under each sensor. Based on the verification of the environment physical centroid fitting matching data and the synchronous environment image data, the environment measurement physical model is modeled and processed to establish the corresponding environment measurement physical model under each sensor, including the geometric shape of important objects in the environment and the relative position information with each sensor. The physical model can be used in application fields such as environmental monitoring, path planning, virtual simulation, etc., providing important support for robot navigation and decision-making.

[0055] As an embodiment of the present invention, refer to Figure 2 As shown, Figure 1 FIG. 1 is a flow chart of detailed implementation steps of step S2 in FIG. 1 . In this embodiment, step S2 includes:

[0056] Step S21: performing time sequence node and space node analysis according to the environment image data to obtain environment image time sequence node data and environment image space node data respectively;

[0057] In an embodiment of the present invention, time series node and space node analysis are performed based on the environmental image data. First, the original image sequence is extracted from the environmental image data, including image information collected by dual Kinect sensors and lidar sensors. Time series analysis is performed on the continuous image sequence to detect key points or features of changes in each frame of the image, including temporal change trends, motion changes, etc. Time series node data, such as key frames or time series feature points, are extracted by calculating the differences between consecutive frames. Spatial node analysis is performed on each frame of the image through the time series node data to identify the spatial change features in the image, and the positions and intensities of these feature points are recorded to generate spatial node data, and finally the environmental image time series node data and environmental image spatial node data are obtained.

[0058] Step S22: using the environment image temporal node data and the environment image spatial node data to perform image temporal and spatial feature synchronization processing on the environment image data to generate synchronized environment image data;

[0059] In an embodiment of the present invention, the time series and spatial node data are combined to achieve synchronization of the spatiotemporal characteristics of the image data, ensuring the consistency of the image data in time and space, that is, the collected environmental image data are aligned at the same latitude, the events of the environmental image data are synchronized, and the spatial changes of the environmental image data monitored by each sensor correspond to each other, so as to generate synchronized environmental image data.

[0060] Step S23: performing environment real material centroid fitting and matching processing on the heterogeneous images based on the synchronous environment image data to generate environment real material centroid fitting and matching data;

[0061] In the embodiment of the present invention, heterogeneous image data from different sensors (Kinect and LiDAR) are fitted and matched to each other, feature points of environmental objects are extracted from synchronized environmental image data, and feature extraction is performed on the data of each sensor (RGB and depth images of Kinect, point cloud data of LiDAR), such as edge detection, corner point detection, etc. The centroid point is calculated for each environmental object. The centroid point is the center point of a feature area of ​​a physical object in the environment, which is usually obtained by weighted averaging of feature points. The centroid points of different sensors are matched using a fitting algorithm (such as the least squares method). The position of the centroid point in each sensor in the global coordinate system is calculated, and they are aligned. The matched centroid point data is sorted out to generate the environmental physical centroid point fitting matching data, which will contain the matching results of the centroid points from different sensors.

[0062] Step S24: performing abnormal physical centroid point verification and elimination on the environment physical centroid point fitting matching data based on a preset RANSAC algorithm to obtain verification environment physical centroid point fitting matching data;

[0063] In an embodiment of the present invention, the parameters of the RANSAC algorithm are set, such as the number of iterations, the inlier threshold, etc. The RANSAC algorithm fits the environmental physical centroid point fitting matching data by randomly selecting a data subset, and identifies and removes abnormal points by comparing the residuals of all data points in the environmental physical centroid point fitting matching data. The RANSAC algorithm is applied to the environmental physical centroid point fitting matching data to identify and remove abnormal centroid points that do not conform to the fitting model. Abnormal points usually appear inconsistent with most data points. According to the output of the RANSAC algorithm, the verified environmental physical centroid point matching data is updated. This data will only contain accurate centroid points that conform to the fitting model to obtain verified environmental physical centroid point fitting matching data, that is, environmental physical centroid point fitting matching data after removing environmental physical centroid points that have no matching information.

[0064] Step S25: Performing environment measurement physical model building processing based on the calibration environment physical centroid point fitting matching data and the synchronous environment image data to obtain the environment measurement physical model.

[0065] In the embodiment of the present invention, the verification environment physical centroid fitting matching data and the synchronized environment image data are combined as the basic data for modeling, and an appropriate modeling algorithm (such as 3D reconstruction, point cloud modeling) is selected. The environment measurement physical model is constructed by verifying the environment physical centroid fitting matching data and the synchronized environment image data, that is, the map geometry information of the environment image collected by each sensor. The initially constructed environment model is optimized, and the model parameters are adjusted to improve the accuracy, including smoothing, detail enhancement, etc. The constructed environment measurement physical model is verified to ensure the consistency between the model and the actual environment, and necessary adjustments are made to generate the final environment measurement physical model, including the environment physical model established under each sensor and the corresponding centroid matching information.

[0066] Preferably, step S23 includes the following steps:

[0067] Step S231: performing environmental object laser point cloud feature analysis according to the synchronized environmental image data to obtain environmental object laser point cloud feature data;

[0068] Step S232: performing an environmental object visual point cloud feature analysis based on the synchronized environmental image data to obtain environmental object visual point cloud feature data;

[0069] Step S233: performing pixel matching analysis of point cloud features on the synchronized environment image data according to the environment object laser point cloud feature data and the environment object visual point cloud feature data to generate pixel matching data;

[0070] Step S234: performing Euclidean distance matching analysis of point cloud features according to the pixel point matching data to generate Euclidean distance matching data;

[0071] Step S235: performing environmental real-matter centroid fitting matching processing on the heterogeneous image according to the pixel point matching data and the Euclidean distance matching data to generate environmental real-matter centroid fitting matching data.

[0072] The present invention performs laser point cloud feature analysis of environmental objects based on synchronized environmental image data, and obtains geometric shapes, surface features, and three-dimensional position information of objects in the environment by analyzing the laser point cloud features of environmental objects. The laser point cloud feature data enriches the precise position dimension of environmental data, and enhances the accuracy and reliability of subsequent data processing and model building. Performing visual point cloud feature analysis of environmental objects based on synchronized environmental image data, analyzing the visual point cloud features of environmental objects, can capture information such as the visual appearance, color, and texture of objects. Visual point cloud features and laser point cloud features provide feature information of environmental objects in different modes, such as the texture and visual appearance of environmental objects of visual point cloud features, and the position and geometric form of laser point clouds, providing a more comprehensive and multi-angle description of the environment, and enhancing the integrity and expression ability of the data. According to the laser point cloud feature data of the physical environment and the visual point cloud feature data of the physical environment, the pixel matching analysis of the point cloud features of the synchronized environment image data is performed, and the features of the laser point cloud and the visual point cloud are matched and analyzed with the environment image data, which can accurately associate the point cloud data with the image pixels. The pixel matching data provides the spatial relationship and consistency between the point cloud features and the image data, providing a basis for subsequent data processing and modeling. According to the pixel matching data, the Euclidean distance matching analysis of the point cloud features is performed, and the Euclidean distance is used to match the point cloud features. The similarity and difference between the point cloud data are quantified. The Euclidean distance matching data provides more accurate and quantifiable point cloud feature matching results, which helps to deeply understand the spatial distribution and morphology of objects in the environment. The environment real matter centroid fitting and matching processing of heterogeneous images is performed according to the pixel matching data and the Euclidean distance matching data. The real matter centroid in the environment is accurately fitted and matched by combining the pixel matching and the Euclidean distance matching data. The fitted environment real matter centroid data covers the observation results of multiple sensors (dual Kinect and lidar), and provides a comprehensive perspective on the position and shape of important objects in the environment. The generated fitted environment real matter centroid fitting data is the matching information between the environment real matter centroids corresponding to the first Kinect sensor, the second Kinect sensor and the lidar sensor.

[0073] In an embodiment of the present invention, the point cloud data generated by the laser radar is extracted from the synchronous environmental image data, and the point cloud processing algorithm (such as the feature estimation algorithm in the PCL library) is used to extract features from the point cloud data. Common features include normal vectors, curvature, surface flatness, etc. The extracted feature data (such as the position of feature points, normal vectors, etc.) are organized into a structured data format as environmental physical laser point cloud feature data. The visual image data of the dual Kinect sensors is extracted from the synchronous environmental image data, and the RGB features of the visual image are used to identify key points and descriptors using a feature extraction algorithm (such as SIFT, SURF or ORB) to generate visual point cloud data. The point cloud processing algorithm is used to extract visual feature data, such as the color of the points, and the extracted visual point cloud feature data (such as the position of the feature points, color information, etc.) is organized into a structured data format as environmental physical visual point cloud feature data. Use feature matching algorithms (such as descriptor-based matching) to align the laser point cloud feature data and the visual point cloud feature data, calculate the correspondence between each feature point in the two data sources, and match the feature points in the laser point cloud and the visual point cloud with the pixels in the synchronized environment image data based on the feature alignment results. Apply image registration technology to perform pixel matching, record the pixel data of each matching pair and its corresponding feature point data, and generate pixel matching data. Use the Euclidean distance algorithm to calculate the Euclidean distance matching information between each corresponding pixel point, evaluate the matching quality based on the calculated distance, select matching pairs with distances within the preset threshold, record the Euclidean distance of each matching pair, and generate Euclidean distance matching data. Based on the pixel matching data and the Euclidean distance matching data, the centroid point of each environmental object is calculated. The centroid point is the center point obtained by weighted averaging the matched feature points. A fitting algorithm, such as the least squares method, is used to fit the centroid points in the data of different sensors to reduce errors. The centroid point data of all sensors are integrated, and the matching results of the fitting information of the centroid point of each environmental object in the data of different sensors are recorded to generate the environmental object centroid point fitting matching data.

[0074] Preferably, step S3 comprises the following steps:

[0075] Step S31: performing cross-modal environmental point cloud data fusion processing on the environmental measurement physical model to generate fused environmental point cloud data, wherein the fused environmental point cloud data includes laser radar prior point cloud data and dual Kinect visual feature data;

[0076] Step S32: performing a laser radar prior map analysis based on the laser radar prior point cloud data to generate laser radar point cloud prior map data;

[0077] Step S33: performing map visual detail fusion processing on the laser radar point cloud prior map data according to the dual Kinect visual feature data to generate fused point cloud prior map data.

[0078] The present invention performs cross-modal environmental point cloud data fusion processing on the physical model of environmental measurement, integrates the visual feature data of laser radar and dual Kinect, and provides a multi-angle and multi-dimensional description of the environment, including geometric shapes, visual features and surface textures, etc. Cross-modal data fusion can comprehensively utilize the advantages of different sensors, make up for the shortcomings of their respective data, and improve the comprehensiveness and accuracy of environmental measurement. According to the laser radar prior point cloud data, the laser radar prior map analysis is performed, and based on the laser radar prior point cloud data, an accurate laser radar point cloud prior map is established, including detailed information such as the position, shape and size of objects in the environment. The laser radar point cloud prior map data provides high-precision spatial information, which provides a reliable foundation for subsequent tasks such as navigation, positioning and obstacle avoidance. The laser radar point cloud prior map is processed by using the visual feature data of dual Kinect to perform map visual detail fusion processing, improve the visual expression ability and detail of the map, and the fusion point cloud prior map data combines the accuracy of the laser radar and the visual information of the dual Kinect, making the map more vivid and comprehensive, which helps the intelligent system understand and respond to the environment.

[0079] In an embodiment of the present invention, point cloud data is obtained from a laser radar and a dual Kinect sensor. The laser radar provides high-precision image depth data, and the dual Kinect sensor provides an RGB image. The point clouds of the two data sources are preprocessed, including denoising, downsampling, etc., to improve data quality. The point cloud data of the laser radar and the dual Kinect sensor are aligned using calibration data, and coordinate transformation is performed to convert all point cloud data into a unified coordinate system. Features (such as edges and textures) are extracted from the visual data of the dual Kinect sensors, and these features are combined with the geometric information in the laser radar data. The laser radar point cloud is aligned with the Kinect point cloud using a feature fusion algorithm (such as an ICP algorithm), and the aligned point cloud data are merged to generate fused environment point cloud data. A point cloud merging algorithm (such as a Voxel Grid filter) is used to remove redundant data and generate a final point cloud model. Perform LiDAR prior map analysis based on LiDAR prior point cloud data, extract environmental feature points from LiDAR point cloud data, use LiDAR data to create a 3D point cloud map of the environment, apply map modeling algorithms (such as raster map generation) to convert point cloud data into a visual map, and analyze the generated map, including obstacle detection, environmental feature extraction, etc. Generate LiDAR point cloud prior map data, verify and correct the generated prior map to ensure its accuracy and consistency. Use dual Kinect visual feature data to enhance the details of the LiDAR prior map, generate a more comprehensive fused point cloud map, extract visual features such as texture, color, edges, etc. from dual Kinect RGB images and depth images, apply visual features to the LiDAR prior map, and increase the details of the map through texture mapping, color enhancement, etc. Use image processing techniques (such as image segmentation and edge detection) to identify and enhance map details, use image fusion algorithms (such as multi-view stereo vision) to combine the visual features of dual Kinect with the lidar prior map to ensure that the visual details and point cloud data are consistent in the same coordinate system, and merge the processed lidar prior map with the visual enhancement data to generate fused point cloud prior map data.

[0080] Preferably, step S31 includes the following steps:

[0081] Step S311: performing dual-Kinect vision external parameter relationship analysis based on the calibration environment physical centroid point fitting matching data and the environment measurement physical model to generate dual-Kinect vision external parameter relationship data;

[0082] Step S312: performing visual and laser radar external parameter relationship analysis based on dual Kinect visual external parameter relationship data and the environmental measurement physical model to generate visual-radar external parameter relationship data;

[0083] Step S313: performing cross-modal environmental point cloud data fusion processing on the environmental measurement physical model according to the dual-Kinect visual external parameter relationship data and the visual-radar external parameter relationship data to generate fused environmental point cloud data.

[0084] The present invention performs external parameter relationship analysis of dual Kinect vision according to the calibration environment physical centroid fitting matching data and the environment measurement physical model. By analyzing the calibration environment physical centroid fitting matching data, the external parameter relationship between the dual Kinect vision sensors is accurately calculated to ensure the accuracy and consistency during data fusion. The dual Kinect vision external parameter relationship data ensures that data collected from different viewing angles can be correctly corresponded to the same coordinate system, eliminating the data inconsistency caused by different viewing angles. The external parameter relationship analysis of vision and laser radar is performed according to the dual Kinect vision external parameter relationship data and the environment measurement physical model. The external parameter relationship between vision and laser radar is analyzed in combination with the appearance coefficient of the dual Kinect sensor and the environment measurement physical model, ensuring that the data collected by different sensors can be effectively fused. The vision-radar external parameter relationship data enables the visual data and the laser radar data to correspond accurately in space, thereby improving the accuracy and integrity of the environment point cloud data fusion. According to the dual-Kinect visual extrinsic parameter relationship data and the visual-radar extrinsic parameter relationship data, the cross-modal environmental point cloud data fusion processing of the environmental measurement physical model is performed. The cross-modal environmental point cloud data fusion processing utilizes the dual-Kinect visual extrinsic parameter relationship data and the visual-radar extrinsic parameter relationship data to ensure that the data collected by different sensors can be completely and seamlessly integrated. The fused environmental point cloud data combines the advantages of vision and lidar, providing richer and more comprehensive environmental information, which is helpful for the subsequent intelligent system's precise positioning, path planning and obstacle avoidance.

[0085] In an embodiment of the present invention, the calibration environment physical centroid fitting matching data, that is, the centroid data after calibration and matching is used as a reference point, including the actual position and direction of the dual Kinect sensors in the environment, and the external parameter relationship model of the dual Kinect visual system is established according to the environmental measurement physical model (that is, the environmental data obtained from the sensor) and the centroid matching data. The external parameter relationship includes position offset (translation) and direction difference (rotation), etc. The external parameter relationship of the dual Kinect sensor is calculated by an analytical geometry method (such as the least squares method). The dual Kinect visual external parameter relationship data is generated using these calculation results to describe the relative position and direction of the two Kinect sensors in three-dimensional space. The dual Kinect visual external parameter relationship data (such as the focal length of the camera, the principal point position and other parameters) and the environmental measurement physical model are used for analysis, and the calibrated dual Kinect sensor coefficients are analyzed based on the dual Kinect visual external parameter relationship data, and the actual layout and posture of the laser radar sensor in the environment are provided by the environmental measurement physical model. According to the calibrated dual Kinect sensor coefficients and the actual layout and posture of the LiDAR sensor in the environment, the external parameter relationship model of vision and LiDAR is established, including the relative position and direction relationship between vision and LiDAR, and the external parameter relationship between vision and LiDAR is calculated by applying calibration algorithms (such as calibration plate calibration) and geometric calculation methods. Generate vision-radar external parameter relationship data to describe the relative position and direction between the two sensors. Obtain external parameter information from dual Kinect vision external parameter relationship data and vision-radar external parameter relationship data. These data are used to align data from different sensors, convert the point cloud data of dual Kinect and the point cloud data of LiDAR into a unified coordinate system, use the external parameter relationship data for coordinate conversion and alignment, ensure that the point clouds from different data sources are consistent in the same space, and apply point cloud fusion algorithms (such as ICP algorithm, Iterative Closest Point) to merge the aligned dual Kinect point cloud data and LiDAR point cloud data. The fusion processing includes steps such as point cloud merging, denoising, and smoothing. The fused data is used to generate the final fused environment point cloud data. This data set combines the high-precision geometric information of the lidar and the rich visual details of Kinect to provide comprehensive information for environmental modeling.

[0086] Preferably, step S4 comprises the following steps:

[0087] Step S41: performing map color and depth data analysis on the fused point cloud prior map data based on a preset RGBD-SLAM algorithm to generate fused point cloud prior map color data and fused point cloud prior map depth data;

[0088] Step S42: performing inter-frame moving feature point analysis based on the fused point cloud prior map color data to obtain inter-frame moving feature point data;

[0089] Step S43: performing inter-frame moving feature point matching processing of the time sequence difference on the inter-frame moving feature point data to generate inter-frame moving feature point matching data;

[0090] Step S44: performing inter-frame environment map modeling processing on the fused point cloud prior map data based on the fused point cloud prior map depth data and the inter-frame moving feature point matching data to generate an inter-frame environment map model;

[0091] Step S45: performing inter-frame pose estimation of the mobile robot according to the fused point cloud priori map depth data and the inter-frame moving feature point matching data to obtain inter-frame pose data of the mobile robot;

[0092] Step S46: performing inter-frame motion feature analysis and processing on the mobile robot according to the inter-frame pose data of the mobile robot to generate inter-frame motion feature data of the mobile robot;

[0093] Step S47: Analyze the mobile robot positioning data based on the inter-frame motion feature data of the mobile robot and the inter-frame environment map model to generate the mobile robot positioning data.

[0094] The present invention performs map color and depth data analysis on the fused point cloud prior map data based on a preset RGBD-SLAM algorithm. The RGBD-SLAM algorithm accurately analyzes the color and depth data of the fused point cloud prior map, thereby improving the accuracy and detail of the map. The fused point cloud prior map color data and depth data provide rich environmental information, including the surface color, texture and depth distribution of the object, which is helpful for subsequent environmental perception and path planning. Inter-frame moving feature point analysis is performed based on the fused point cloud prior map color data. Inter-frame moving feature points are used to accurately capture the temporal relationship of motion features, that is, the corresponding relationship between the changes in the inter-frame moving feature points during the motion of the mobile robot. The movement distance and orientation of the mobile robot can be inferred through the changes in the inter-frame moving feature points, which is helpful for real-time environmental monitoring and motion control, and provides key information for the intelligent positioning decision of the mobile robot. The inter-frame moving feature point data is processed by inter-frame moving feature point matching with time sequence difference. The inter-frame moving feature point matching processing with time sequence difference can ensure the continuity of the motion trajectory, especially in a dynamic environment, which helps to identify and track the motion trajectory of the object. The inter-frame moving feature point matching data provides detailed information on the motion mode, including the mode, frequency and periodic changes of the motion, and provides support for robot behavior planning and positioning. The inter-frame environment map modeling is processed based on the fused point cloud prior map depth data and the inter-frame moving feature point matching data. The inter-frame environment map modeling based on the fused point cloud prior map depth data can accurately express the structure and topological relationship of the environment, including the position, size and spatial distribution of the object, and analyzes the changes in the map of the mobile robot during the movement through the inter-frame moving feature point matching data. The inter-frame environment map model is dynamically updated and optimized according to the inter-frame robot's movement correspondence, capturing real-time changes in the environment and providing real-time environmental information for navigation and obstacle avoidance. The inter-frame pose estimation of the mobile robot is performed based on the fused point cloud prior map depth data and the inter-frame moving feature point matching data. The inter-frame pose estimation based on the fused point cloud prior map depth data and the inter-frame moving feature point matching data improves the positioning accuracy and stability of the mobile robot and ensures accurate navigation in complex environments. The acquisition and analysis of the inter-frame pose data of the mobile robot optimizes the real-time performance and adapts to the rapidly changing environmental conditions and the robot's own motion requirements. The inter-frame motion feature analysis and processing of the mobile robot is performed based on the inter-frame pose data of the mobile robot. The inter-frame motion feature analysis of the mobile robot is used to identify the robot's motion mode and behavior, including the travel speed, steering angle and acceleration changes, so as to analyze the positioning information of the mobile robot and provide data support for the robot's path planning and dynamic obstacle avoidance.The mobile robot positioning data analysis is performed based on the mobile robot's inter-frame motion feature data and the inter-frame environment map model. The mobile robot positioning data analysis combines the mobile robot's inter-frame motion feature data and the inter-frame environment map model to improve the robot's positioning accuracy and stability in complex environments. The mobile robot positioning data provides key information for path planning and navigation, ensuring that the robot can complete the task effectively and safely.

[0095] As an embodiment of the present invention, refer to Figure 3 As shown, Figure 1 FIG. 4 is a flow chart of detailed implementation steps of step S4 in FIG. 4 . In this embodiment, step S4 includes:

[0096] Step S41: performing map color and depth data analysis on the fused point cloud prior map data based on a preset RGBD-SLAM algorithm to generate fused point cloud prior map color data and fused point cloud prior map depth data;

[0097] In the embodiment of the present invention, a preset RGBD-SLAM algorithm is applied to process the RGB (color) information in the fused point cloud prior map data to generate fused point cloud prior map color data. The algorithm analyzes the color changes in the image and maps it to the three-dimensional point cloud. The RGBD-SLAM algorithm is used to analyze the depth information in the point cloud, and the geometric features of the map are analyzed based on the depth information in the point cloud to generate the fused point cloud prior map depth data, which involves extracting distance information from the depth map and applying it to the point cloud data.

[0098] Step S42: performing inter-frame moving feature point analysis based on the fused point cloud prior map color data to obtain inter-frame moving feature point data;

[0099] In an embodiment of the present invention, nodes with large brightness differences in the color data of the fused point cloud prior map are extracted. These feature points should show significant visual differences between different frames, and features are extracted from these nodes to generate inter-frame moving feature point data, which includes the coordinates and description information of all identified feature points.

[0100] Step S43: performing inter-frame moving feature point matching processing of the time sequence difference on the inter-frame moving feature point data to generate inter-frame moving feature point matching data;

[0101] In an embodiment of the present invention, a transfer matrix matching algorithm is used to match the feature points of the current frame with the transfer matrix corresponding to the feature points of the previous frame. The matching process includes calculating the transfer matrix constructed by the distance between the feature points. Through the matching results between the transfer matrices, such as the closed-loop correspondence between the transfer matrix of the previous moment and the current transfer matrix, the matching results are output, that is, the inter-frame moving feature points corresponding to different frames, and the inter-frame moving feature point matching data is generated.

[0102] Step S44: performing inter-frame environment map modeling processing on the fused point cloud prior map data based on the fused point cloud prior map depth data and the inter-frame moving feature point matching data to generate an inter-frame environment map model;

[0103] In an embodiment of the present invention, the geometric distribution and distance movement of the environment map are analyzed based on the fused point cloud prior map depth data, and the details of the visual changes of the mobile robot are supplemented through the inter-frame moving feature point matching data. The fused point cloud prior map depth data and the inter-frame moving feature point matching data are fused using a modeling algorithm (such as sparse reconstruction and dense modeling), so as to analyze the dynamic changes of the map of the mobile robot during the movement process, reflect the path and structural changes of the mobile robot in the environment, and perform dynamic map modeling through modeling technology to generate an inter-frame environment map model.

[0104] Step S45: performing inter-frame pose estimation of the mobile robot according to the fused point cloud priori map depth data and the inter-frame moving feature point matching data to obtain inter-frame pose data of the mobile robot;

[0105] In an embodiment of the present invention, a pose estimation algorithm (such as pose graph optimization and visual odometry) is applied to estimate the pose of the robot between different frames, and the fused point cloud prior map depth data and the inter-frame moving feature point matching data are transmitted to the pose estimation algorithm for pose change analysis, and the rotation and displacement of the robot at different time points are calculated to generate the inter-frame pose data of the mobile robot, which includes the pose information of the robot in each frame.

[0106] Step S46: performing inter-frame motion feature analysis and processing on the mobile robot according to the inter-frame pose data of the mobile robot to generate inter-frame motion feature data of the mobile robot;

[0107] In an embodiment of the present invention, the motion characteristics of the robot between frames are calculated based on the inter-frame posture data of the mobile robot, including changes in speed, acceleration and motion path, and the motion characteristics are calculated using a motion analysis algorithm (such as motion smoothing and path tracking) to generate the inter-frame motion characteristic data of the mobile robot to describe the dynamic behavior of the robot in the environment.

[0108] Step S47: Analyze the mobile robot positioning data based on the inter-frame motion feature data of the mobile robot and the inter-frame environment map model to generate the mobile robot positioning data.

[0109] In an embodiment of the present invention, preliminary mobile robot positioning data is analyzed by analyzing the inter-frame motion feature data of the mobile robot, that is, the preliminary position of the robot is determined by a positioning algorithm, such as a particle filter and a Kalman filter, and mapped accordingly with the inter-frame environment map model to obtain accurate positioning information of the mobile robot and generate mobile robot positioning data.

[0110] Preferably, step S42 includes the following steps:

[0111] Step S421: performing image pixel point traversal processing of the robot moving between frames according to the fused point cloud prior map color data to obtain inter-frame traversal image pixel point data;

[0112] Step S422: performing inter-frame moving feature point analysis on the inter-frame traversal image pixel point data to obtain inter-frame moving feature point data.

[0113] The present invention performs image pixel point traversal processing of the robot's inter-frame movement based on the fused point cloud prior map color data, and obtains the image pixel point data of the robot between frames during the movement. The inter-frame traversal image pixel point data provides detailed spatial information, including the position, shape and spatial distribution of the environment of the object, and provides basic data for subsequent motion feature analysis. Inter-frame moving feature point analysis is performed on the inter-frame traversal image pixel point data. The inter-frame moving feature point analysis effectively extracts the robot's motion feature points from the inter-frame traversal image pixel point data, including features such as reference points in the path. The inter-frame moving feature point data is used to detect and analyze dynamic changes in the environment, such as the movement of objects and changes in environmental structures, and provides support for the robot's real-time perception and positioning.

[0114] In the embodiment of the present invention, the fused point cloud prior map color data is divided into pixels in units of pixels, and the divided pixels are traversed, that is, each pixel of the fused point cloud prior map color data is identified to obtain inter-frame traversal image pixel data. Inter-frame moving feature point analysis is performed on the inter-frame traversal image pixel data, and pixels with large visual effects differences are analyzed through the brightness difference between the inter-frame traversal image pixel data and the neighboring grids, and are marked to obtain inter-frame moving feature point data.

[0115] Preferably, step S422 includes the following steps:

[0116] Step S4221: select pixel points of the inter-frame traversal image pixel point data one by one as the central node, perform a central node neighborhood pixel point brightness analysis on the central node through the preset central node neighborhood selection range data to obtain the central node neighborhood pixel point brightness data; perform a brightness gradient analysis on the central node neighborhood pixel point brightness data to generate the central node neighborhood pixel point brightness gradient data;

[0117] Step S4222: When the brightness gradient data of the neighboring pixels of the central node meets the preset brightness gradient difference judgment condition, the central node is marked as an inter-frame moving feature point mark to obtain the inter-frame moving feature point data; when the brightness gradient data of the neighboring pixels of the central node does not meet the preset brightness gradient difference judgment condition, return to step S4221.

[0118] The present invention selects the pixel point of the inter-frame traversal image pixel point data as the central node, performs the central node neighbor pixel point brightness analysis on the central node through the preset central node neighbor pixel point selection range data, reflects the lighting conditions and brightness distribution of the selected pixel node corresponding area, analyzes the brightness gradient of the central node neighbor pixel point brightness data, reveals the change of lighting in the surrounding area, and thus identifies the lighting changes, shadows and light intensity in the environment with strong nodes as control points, that is, control feature nodes in the mobile robot's movement process. By determining whether the central node neighbor pixel point brightness gradient data meets the preset brightness gradient difference condition, the dynamic feature points in the inter-frame movement are effectively detected. These feature points reflect the nodes that have undergone significant changes in the environment. When the central node neighbor pixel point brightness gradient data does not meet the condition, return to step S4221 to continue analysis. This iterative processing helps to optimize the data processing flow, ensure that sufficiently accurate inter-frame moving feature points are detected, and obtain enough feature nodes for analyzing the movement of the mobile robot.

[0119] In the embodiment of the present invention, each image pixel is first traversed, and each selected pixel is recorded as P. The brightness of each selected pixel is set as IPK. With the selected pixel P as the center of the circle and a radius of 3 pixels, 16 neighboring pixels are selected, and the brightness of each selected neighboring pixel is recorded as IPW. T is recorded as the pre-designed pixel brightness gradient difference, such as T = 0.3*IPK. M = 0, and M is when IPW is greater than or equal to IPK+T among the 16 neighboring pixels, or IPW is less than or equal to IPK-T, and when 9 or more neighboring pixels meet the condition, the pixel P is an inter-frame moving feature point, and all the pixels in the fused point cloud prior map color data are traversed to obtain the final inter-frame moving feature point set, and generate inter-frame moving feature point data.

[0120] The beneficial effect of the present application lies in that the augmented reality environment modeling and positioning processing method of a mobile robot of the present invention targets the situation where visual SIAM technology faces the lack of feature points in the actual environment, and combines dual Kinect visual sensors with laser radars. On the one hand, the number of visual sensors is increased, which in turn expands the detection range of the visual sensors and increases the number of detected visual SLAM feature points in the actual environment. On the other hand, the laser radar has the characteristics of high measurement accuracy, large measurement range and its point cloud data is not affected by light conditions. The point cloud data of the laser that is not affected by visible light is used as prior map information to provide visual positioning and mapping, and then the visual SLAM process is carried out. At the same time, when feature points are scarce in the visual SLAM process due to the lack of light intensity and environmental texture, the robustness of the lidar point cloud to environmental texture and light is used to fuse the visual point cloud with the lidar point cloud to form fused point cloud data. The fused point cloud data is analyzed through the color and depth information of the point cloud, and the robust feature point data in the environment is used as a reference node to match the key data frame point cloud information to complete the positioning and mapping process, thereby overcoming the problem of positioning and mapping failure caused by insufficient or changed environmental conditions in visual SIAM technology.

[0121] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0122] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A method for modeling and positioning an augmented reality environment of a mobile robot, characterized in that: The following steps are involved: Step S1: acquiring sensor configuration monitoring parameters; configuring monitoring parameters of the monitoring sensor according to the sensor configuration monitoring parameters, and collecting environmental image data through the monitoring sensor after the monitoring parameter configuration to generate environmental image data, wherein the monitoring sensor is a dual Kinect sensor and a laser radar sensor built into the mobile robot; Wherein, step S1 comprises: Step S11: Obtain dual Kinect sensor debugging monitoring parameters; Step S12: performing dual Kinect visual overlap area correction configuration processing on the dual Kinect sensor debugging monitoring parameters to obtain dual Kinect sensor configuration monitoring parameters; Step S12 includes the following steps: Step S121: performing dual Kinect vision test processing according to dual Kinect sensor debugging monitoring parameters to generate dual Kinect vision test data; Step S122: performing dual Kinect vision test overlapping area analysis on the dual Kinect vision test data to generate dual Kinect vision test overlapping area data; Step S123: performing dual Kinect visual overlapping area correction configuration processing on dual Kinect sensor debugging monitoring parameters according to dual Kinect visual test overlapping area data to obtain dual Kinect sensor configuration monitoring parameters; Step S13: performing dual Kinect visual coverage analysis according to dual Kinect sensor configuration monitoring parameters to generate dual Kinect visual coverage data; Step S14: analyzing the configuration monitoring parameters of the laser radar sensor according to the dual Kinect visual coverage data to obtain the configuration monitoring parameters of the laser radar sensor; Step S15: integrating the dual Kinect sensor configuration monitoring parameters and the laser radar sensor configuration monitoring parameters to obtain the sensor configuration monitoring parameters; Step S16: configuring monitoring parameters of the monitoring sensor according to the sensor configuration monitoring parameters, and collecting environmental image data through the monitoring sensor after the monitoring parameter configuration to generate environmental image data; Step S2: synchronously process the image spatiotemporal features of the environmental image data to generate synchronous environmental image data; perform fitting matching and verification processing of the environmental physical centroid of the heterogeneous image based on the synchronous environmental image data to generate verification environmental physical centroid fitting matching data; perform environmental measurement physical model modeling processing based on the verification environmental physical centroid fitting matching data and the synchronous environmental image data to obtain the environmental measurement physical model; Step S3: performing cross-modal environmental point cloud data fusion processing on the environmental measurement physical model to generate fused environmental point cloud data; performing fused point cloud prior map analysis based on the fused environmental point cloud data to obtain generated fused point cloud prior map data; Step S4: Based on the preset RGBD-SLAM algorithm, the map color and depth data of the fused point cloud prior map data are analyzed to generate the fused point cloud prior map color data and the fused point cloud prior map depth data; according to the fused point cloud prior map color data, the inter-frame moving feature point matching analysis and processing of the time difference is performed to generate the inter-frame moving feature point matching data; based on the fused point cloud prior map depth data and the inter-frame moving feature point matching data, the fused point cloud prior map data is modeled to generate the inter-frame environment map model; according to the fused point cloud prior map depth data and the inter-frame moving feature point matching data, the mobile robot inter-frame motion feature analysis and processing is performed to generate the mobile robot inter-frame motion feature data; according to the mobile robot inter-frame motion feature data and the inter-frame environment map model, the mobile robot positioning data is analyzed to generate the mobile robot positioning data.

2. The method for modeling and positioning an augmented reality environment of a mobile robot according to claim 1, characterized in that: Step S2 includes the following steps: Step S21: performing time sequence node and space node analysis according to the environment image data to obtain environment image time sequence node data and environment image space node data respectively; Step S22: using the environment image temporal node data and the environment image spatial node data to perform image temporal and spatial feature synchronization processing on the environment image data to generate synchronized environment image data; Step S23: performing environment real material centroid fitting and matching processing on the heterogeneous images based on the synchronous environment image data to generate environment real material centroid fitting and matching data; Step S24: performing abnormal physical centroid point verification and elimination on the environment physical centroid point fitting matching data based on a preset RANSAC algorithm to obtain verification environment physical centroid point fitting matching data; Step S25: Performing environment measurement physical model building processing based on the calibration environment physical centroid point fitting matching data and the synchronous environment image data to obtain the environment measurement physical model.

3. The method for modeling and positioning an augmented reality environment of a mobile robot according to claim 2, characterized in that: Step S23 includes the following steps: Step S231: performing environmental object laser point cloud feature analysis according to the synchronized environmental image data to obtain environmental object laser point cloud feature data; Step S232: performing an environmental object visual point cloud feature analysis based on the synchronized environmental image data to obtain environmental object visual point cloud feature data; Step S233: performing pixel matching analysis of point cloud features on the synchronized environment image data according to the environment object laser point cloud feature data and the environment object visual point cloud feature data to generate pixel matching data; Step S234: performing Euclidean distance matching analysis of point cloud features according to the pixel point matching data to generate Euclidean distance matching data; Step S235: performing environmental real-matter centroid fitting matching processing on the heterogeneous image according to the pixel point matching data and the Euclidean distance matching data to generate environmental real-matter centroid fitting matching data.

4. The method for modeling and positioning an augmented reality environment of a mobile robot according to claim 1, characterized in that: Step S3 includes the following steps: Step S31: performing cross-modal environmental point cloud data fusion processing on the environmental measurement physical model to generate fused environmental point cloud data, wherein the fused environmental point cloud data includes laser radar prior point cloud data and dual Kinect visual feature data; Step S32: performing a laser radar prior map analysis based on the laser radar prior point cloud data to generate laser radar point cloud prior map data; Step S33: performing map visual detail fusion processing on the laser radar point cloud prior map data according to the dual Kinect visual feature data to generate fused point cloud prior map data.

5. The method for modeling and positioning an augmented reality environment of a mobile robot according to claim 4, characterized in that: Step S31 includes the following steps: Step S311: performing dual-Kinect vision external parameter relationship analysis based on the calibration environment physical centroid point fitting matching data and the environment measurement physical model to generate dual-Kinect vision external parameter relationship data; Step S312: performing visual and laser radar external parameter relationship analysis based on dual Kinect visual external parameter relationship data and the environmental measurement physical model to generate visual-radar external parameter relationship data; Step S313: performing cross-modal environmental point cloud data fusion processing on the environmental measurement physical model according to the dual-Kinect visual external parameter relationship data and the visual-radar external parameter relationship data to generate fused environmental point cloud data.

6. The method for modeling and positioning an augmented reality environment of a mobile robot according to claim 1, characterized in that: Step S4 includes the following steps: Step S41: performing map color and depth data analysis on the fused point cloud prior map data based on a preset RGBD-SLAM algorithm to generate fused point cloud prior map color data and fused point cloud prior map depth data; Step S42: performing inter-frame moving feature point analysis based on the fused point cloud prior map color data to obtain inter-frame moving feature point data; Step S43: performing inter-frame moving feature point matching processing of the time sequence difference on the inter-frame moving feature point data to generate inter-frame moving feature point matching data; Step S44: performing inter-frame environment map modeling processing on the fused point cloud prior map data based on the fused point cloud prior map depth data and the inter-frame moving feature point matching data to generate an inter-frame environment map model; Step S45: performing inter-frame pose estimation of the mobile robot according to the fused point cloud priori map depth data and the inter-frame moving feature point matching data to obtain inter-frame pose data of the mobile robot; Step S46: performing inter-frame motion feature analysis and processing on the mobile robot according to the inter-frame pose data of the mobile robot to generate inter-frame motion feature data of the mobile robot; Step S47: Analyze the mobile robot positioning data based on the inter-frame motion feature data of the mobile robot and the inter-frame environment map model to generate the mobile robot positioning data.

7. The method for modeling and positioning an augmented reality environment of a mobile robot according to claim 6, characterized in that: Step S42 includes the following steps: Step S421: performing image pixel point traversal processing of the robot moving between frames according to the fused point cloud prior map color data to obtain inter-frame traversal image pixel point data; Step S422: performing inter-frame moving feature point analysis on the inter-frame traversal image pixel point data to obtain inter-frame moving feature point data.

8. The method for modeling and positioning an augmented reality environment of a mobile robot according to claim 7, characterized in that: Step S422 includes the following steps: Step S4221: select pixel points of the inter-frame traversal image pixel point data one by one as the central node, perform a central node neighborhood pixel point brightness analysis on the central node through the preset central node neighborhood selection range data to obtain the central node neighborhood pixel point brightness data; perform a brightness gradient analysis on the central node neighborhood pixel point brightness data to generate the central node neighborhood pixel point brightness gradient data; Step S4222: When the brightness gradient data of the neighboring pixels of the central node meets the preset brightness gradient difference judgment condition, the central node is marked as an inter-frame moving feature point mark to obtain the inter-frame moving feature point data; when the brightness gradient data of the neighboring pixels of the central node does not meet the preset brightness gradient difference judgment condition, return to step S4221.

Images