Indoor mobile robot adaptive fusion positioning mapping voice interaction method and system

Abstract

The invention discloses an indoor mobile robot adaptive fusion positioning mapping voice interaction method and system, and belongs to the technical field of indoor mobile robot multi-sensor fusion positioning mapping and voice interaction, and the method comprises the following steps: S1, multi-source sensor data collection; s2, environment state sensing and feature parameter extraction; s3, dynamic distribution of master and slave SLAM weights; s4, performing weighted positioning mapping calculation; s5, constructing spatial semantic association; s6, voice instruction receiving and semantic analysis; s7, semantic instruction conversion and control parameter generation; and S8, executing and feedback closed loop. According to the indoor mobile robot adaptive fusion positioning mapping voice interaction method and system, stability and reliability of positioning mapping in a complex environment are improved, low-quality data interference is reduced, deep fusion of voice and positioning mapping is realized, and man-machine collaborative exploration efficiency and interaction flexibility are improved.

Description

Technical Field

[0001] This invention belongs to the field of multi-sensor fusion localization mapping and voice interaction technology for indoor mobile robots, specifically involving an adaptive fusion localization mapping and voice interaction method and system for indoor mobile robots. Background Technology

[0002] Service robots, inspection robots, and rescue robots are being used more and more widely. In closed or semi-closed environments without satellite signal coverage, such as indoor spaces, basements, and underground tunnels, autonomous simultaneous localization and mapping (SLAM) has become a core technical requirement for robots to achieve autonomous operation, and this requirement is becoming more and more urgent as the application scenarios of robots expand.

[0003] However, existing technologies have the following shortcomings: most adopt fixed fusion strategies or only dynamically weight data, without dynamically evaluating environmental conditions (such as lighting), and cannot flexibly adjust the master-slave relationship of visual / laser SLAM. In complex environments, localization drift and mapping failure are prone to occur. Voice is only used to trigger simple actions such as moving forward and stopping, and is independent of the localization and mapping module. It does not participate in spatial understanding and mapping decisions, and manual intervention in unknown environments is inefficient and lacks flexibility. Visual SLAM is greatly affected by lighting, and features are insufficient in low light / no light environments. Laser SLAM lacks semantic information, and mapping accuracy decreases in narrow spaces and strong reflection scenes.

[0004] Therefore, a new method is urgently needed. Summary of the Invention

[0005] The purpose of this invention is to provide an adaptive fusion localization mapping and voice interaction method and system for indoor mobile robots. This method improves the stability and reliability of localization mapping in complex environments, reduces interference from low-quality data, achieves deep fusion of voice and localization mapping, and significantly improves the efficiency and flexibility of human-machine collaborative exploration.

[0006] To achieve the above objectives, this invention provides an adaptive fusion localization, mapping, and voice interaction method and system for indoor mobile robots, comprising the following steps: S1. Through the multi-source sensor data acquisition module, synchronously acquire the raw image data from the visual sensor and the raw point cloud data from the lidar; transmit the raw image data to S2, and synchronously transmit the raw image data and the raw point cloud data to S4. S2, the environmental state perception module receives the raw image data transmitted by S1, extracts environmental feature parameters through analysis, and transmits the environmental feature parameters to S3; S3, the master-slave sensor weight allocation module calculates the illumination confidence level based on the environmental feature parameters extracted by S2 through linear normalization or piecewise mapping, and constructs and outputs dynamic weight parameters based on the illumination confidence level; S4, the localization and mapping module receives the raw image data and point cloud data from S1 and the dynamic weight parameters from S3. It performs weighted modeling on the visual observation information and the laser observation information. After weighted processing of the two types of observation residual terms, it performs pose estimation and environmental map construction, and outputs robot pose information and environmental map information. S5. Divide the environmental map information output by S4 into spatial regions and assign semantic labels, and match the robot pose information of S4 with the spatial regions to establish and store pose-semantic label association data. S6. Receive voice commands input by the user, the voice commands including spatial location query and exploration control; perform speech recognition and semantic parsing on the commands to obtain parsing results, the parsing results including spatial semantic information and exploration control semantic information; S7. Based on the parsing results of S6, if it is a spatial location query, call the pose-semantic label association data of S5 to determine the spatial semantic label data corresponding to the current pose; if it is an exploration control, generate exploration control parameters. S8: Receive the exploration control parameters transmitted by S7, adjust the exploration strategy, and update the pose and map update data by combining the original image data from S1. It receives spatial semantic label data transmitted by S7, converts it into voice feedback data and feeds it back to the user, thus completing the human-computer interaction closed loop.

[0007] Preferably, in S2, the extracted environmental feature parameters include the mean and variance of image brightness, contrast, number of visual feature points, and changes in camera exposure parameters; the original image data is converted into a single-channel grayscale image through grayscale processing, and the grayscale value calculation formula is: ; In the formula, For the three color components of a pixel; These are pixel coordinates.

[0008] Preferably, in S2, the formula for calculating the average image brightness is: ; In the formula, The average brightness of the image; Image height; Image width; The row index of the pixel; The column index for pixels; Image brightness variance, the formula is: ; In the formula, The brightness variance of the image; Image contrast is calculated using the following formula: ; In the formula, Image contrast; This represents the maximum value of the grayscale image. ( I () represents the minimum value of the grayscale image; The formula for calculating the change in camera exposure parameters is: ; In the formula, This represents the change in camera exposure parameters; This is the current exposure time; This represents the average exposure time of the first 10 frames.

[0009] Preferably, in S3, the illumination confidence level The value range is [0,1], and the formula for the linear normalization model is: ; In the formula, For illumination confidence level; , , As a weighting factor, ; This is the normalized value of the average brightness. This is the normalized value of the image contrast. This is the normalized value of the number of feature points; The formula for the piecewise mapping model is: ; The linear distribution formula is: ; ; In the formula, The weights for visual SLAM; The weights of the laser SLAM are satisfied; ; The formula for segmented allocation is: .

[0010] Preferably, in S4, the weighted localization mapping uses factor graph optimization and extended Kalman filter modeling; when performing factor graph optimization, a factor graph is constructed. , For pose nodes, For observation factors; the formula for calculating weighted observation factors is: ; ; In the formula, The weighted visual observation factors; The weighted laser observation factor; The original visual observation factor; The original laser observation factor; When using extended Kalman filtering, the formula for calculating weighted fused observations is: ; In the formula, For the first Weighted fused observations at different times; These are visual observations; These are laser observation values.

[0011] Preferably, in S7, the control parameters are explored. The formula for the target heading angle is: ; In the formula, The target heading angle; This is the output of the direction mapping function; This refers to the direction semantic field within the self-control class semantic information; This is the conversion factor from degrees to radians; Generate range parameters ;in, The minimum coordinate value in the X-axis direction; The maximum coordinate value in the X-axis direction; The minimum coordinate value in the Y-axis direction; The maximum coordinate value in the Y-axis direction; Mapping weighting coefficients The correspondence is as follows: Priority "Preferred": ; Priority "Normal": ; Priority "Secondary": .

[0012] Preferably, in S8, the path planning cost function is: ; In the formula, To provide comprehensive value; As a cost of distance; Score the regional value.

[0013] This invention also provides an indoor mobile robot adaptive fusion localization mapping voice interaction system, comprising: The multi-source sensor data acquisition module is used to execute S1, synchronously acquire raw image data from the visual sensor and raw point cloud data from the lidar, and distribute them to the corresponding modules. An environmental state perception module, connected to the multi-source sensor data acquisition module, is used to execute S2, receive raw image data and extract environmental feature parameters; The master-slave sensor weight allocation module is connected to the environmental state perception module and is used to execute S3, calculate the illumination confidence based on environmental feature parameters and output dynamic weight parameters. The localization and mapping module is connected to the multi-source sensor data acquisition module and the master-slave sensor weight allocation module, respectively, and is used to execute S4 to realize weighted localization and mapping and output pose and map information; The spatial semantic association construction module is connected to the localization and mapping module and is used to execute S5 to divide spatial regions, assign semantic labels, and establish pose-semantic label associations. The voice command processing module is used to execute S6, receive voice commands, and perform recognition and semantic parsing; The semantic conversion and control parameter generation module is connected to the spatial semantic association construction module and the voice command processing module, respectively, and is used to execute S7 to generate spatial semantic label data or explore control parameters. The execution and feedback module is connected to the semantic conversion and control parameter generation module and the multi-source sensor data acquisition module, respectively, and is used to execute S8, adjust the exploration strategy, update the pose and map, and realize voice feedback.

[0014] Therefore, the present invention employs the above-mentioned adaptive fusion localization mapping and voice interaction method and system for indoor mobile robots. Compared with the prior art, the technical solution of the present invention has the following beneficial effects: (1) By adopting an environmental state perception and master-slave weight adaptive mechanism, the visual / laser SLAM weights are dynamically adjusted, which overcomes the technical problems of poor environmental adaptability of a single sensor, fixed fusion redundant calculation and error accumulation, thereby achieving the technical effect of improving the stability and reliability of localization and mapping in complex environments and reducing low-quality data interference. (2) By adopting the technical means of converting speech semantics into spatial semantic association information and exploration control parameters, the separation between speech and localization mapping is broken, and the technical defects of speech being able to only trigger simple actions and unable to participate in spatial cognition and mapping decision-making are overcome, thereby achieving the technical effect of deep integration of speech and localization mapping, improving the efficiency of human-machine collaborative exploration and the flexibility of interaction. (3) The illumination confidence quantification and weighted modeling technology are adopted, taking into account the core advantages of visual SLAM and laser SLAM, overcoming the inherent technical deficiencies of visual SLAM failure in weak light and laser SLAM lacking semantics, thereby achieving the technical effect of significantly improving the stability of localization and mapping in complex environments.

[0015] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0016] Figure 1 This is an architecture diagram of an embodiment of the indoor mobile robot adaptive fusion localization mapping voice interaction method and system of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used in the present invention should have the ordinary meaning understood by those skilled in the art.

[0018] Example 1 like Figure 1 As shown, this embodiment provides an adaptive fusion localization mapping and voice interaction method and system for indoor mobile robots. It should be understood that the specific parameters, models and protocols mentioned in this embodiment are only examples to help those skilled in the art understand the present invention, and are not intended to limit the present invention.

[0019] The indoor mobile robot adaptive fusion localization mapping and voice interaction method and system of the present invention includes the following steps: S1. Through the multi-source sensor data acquisition module, the raw image data of the vision sensor (camera) and the raw point cloud data of the lidar are acquired simultaneously; the raw image data is transmitted to S2 (environmental state perception module), and the raw image data and the raw point cloud data are transmitted to S4 (localization and mapping module). S2 receives the raw image data transmitted by S1, extracts environmental feature parameters (including image brightness mean and variance, contrast, number of visual feature points, etc.) through analysis, and transmits the environmental feature parameters to S3 (master-slave sensor weight allocation module) for the quantification of ambient light status. In this step, the original image data is converted into a single-channel grayscale image through grayscale processing. The formula for calculating the grayscale value is: ; In the formula, For the three color components of a pixel; These are pixel coordinates; Extracting environmental feature parameters, specifically: The average brightness of the image is calculated using the following formula: ; In the formula, The average brightness of the image; Image height; Image width; The row index of the pixel; The column index for pixels; Image brightness variance, the formula is: ; In the formula, The brightness variance of the image; Image contrast is calculated using the following formula: ; In the formula, Image contrast; This represents the maximum value of the grayscale image. ( I () represents the minimum value of the grayscale image; The number of feature points that can be stably tracked is ≥500 in the initial detection and ≥3 in the tracking frames. ; The formula for calculating the change in camera exposure parameters is: ; In the formula, This represents the change in camera exposure parameters; This is the current exposure time; This is the average exposure time of the first 10 frames; The extracted parameters are preprocessed by normalization (mapped to the [0,1] interval) to form an environmental feature parameter vector, represented as follows: ; In the formula, This is a vector of environmental feature parameters; This is the normalized value of the average brightness. This is the normalized value of the image brightness variance; This is the normalized value of the image contrast. This is the normalized value of the number of feature points; This is the normalized value of the camera exposure parameter changes; S3, the master-slave sensor weight allocation module calculates the illumination confidence L (value [0,1]) based on the environmental feature parameters extracted by S2 through linear normalization or piecewise mapping, constructs and outputs the visual / laser SLAM dynamic weight parameters based on L, and passes the weight parameters to S4; In this step, the normalized value of the average brightness is selected. As a core indicator, it is combined with the normalized value of image contrast. The value after normalization of the number of feature points Construct the illumination confidence level L; For general scenarios, a linear normalized model is constructed, with the following formula: ; In the formula, For illumination confidence level; , , As a weighting factor, In this embodiment, , , ; For complex lighting scenes, a piecewise mapping model is constructed, with the following formula: ; based on Construct visual SLAM weights and laser SLAM weights, dynamically allocate them using a model, and use the following linear allocation formula: ; ; In the formula, The weights for visual SLAM; The weights of the laser SLAM are satisfied; ; The formula for segmented allocation is: ; Output weight parameter pair And store the historical weight sequence for smooth transition; S4 receives the raw image data and point cloud data from S1, as well as the dynamic weight parameters from S3. When environmental conditions are favorable for visual perception, visual SLAM has a larger weight in the modeling process; when environmental conditions are unfavorable for visual perception, laser SLAM has a larger weight in the modeling process. The visual observation information and laser observation information are weighted and modeled. After weighting the two types of observation residual terms, pose estimation and environmental map construction are performed, and the robot pose information and environmental map information are output. This achieves adaptive adjustment of the master-slave relationship between visual SLAM and laser SLAM. This step also includes SLAM modeling based on factor graph optimization, specifically: Constructing factor graphs , pose node , For observation factors; This corresponds to the robot's pose at frame 0, frame 1, ..., frame k, respectively. The weighted observation factor is calculated using the following formula: ; ; In the formula, The weighted visual observation factors; The weighted laser observation factor; The original visual observation factor; The original laser observation factor; The formula for calculating the visual factor residual is: ; In the formula, These are visual observations; For visual observation models; For visual observation residuals; The laser factor residual is calculated using the following formula: ; In the formula, These are laser observation values; For laser observation models; For laser observation residuals; The objective function, calculated using the following formula: ; In the formula, This is the transpose of the visual observation residual vector; This is the transpose of the laser observation residual vector; This is the inverse of the covariance matrix of the visual observation noise; Let be the inverse of the covariance matrix of the laser observation noise; iterative optimization is performed until the residual converges; SLAM modeling based on extended Kalman filtering is as follows: State prediction, the calculation formula is: ; In the formula, For robot motion control; For the first Prior state estimation at time step; This is the state transition function; Weighted observation update, calculated using the following formula: ; In the formula, For the first Weighted fused observations at different times; Kalman gain, calculated using the following formula: ; In the formula, For the first Kalman gain matrix at time step; For the first The prior state covariance matrix at time t; The observation matrix; This is the transpose of the observation matrix; State correction, calculated using the following formula: ; In the formula, For the first Posterior state estimation at time t; Output robot pose information With environmental map information; among which, These are the position coordinates in a three-dimensional rectangular coordinate system; The angle by which the robot rotates around the Y-axis; The angle by which the robot rotates around the Z-axis; Let X be the angle by which the robot rotates around the X-axis. S5. Divide the environmental map information output by S4 into several spatial regions and assign semantic labels. Match the robot pose information of S4 with the spatial regions to establish and store pose-semantic label association data. In this step, the laser point cloud map is segmented to extract obstacle contours; based on the contours' connectivity, clustering is performed, and a distance threshold is set. ,when When they are classified as belonging to the same area, , Two points on the outline; Supports importing from a pre-set tag library or user-defined tags. The tag storage format is as follows: ; In the formula, For spatial region semantic labels; A unique identifier for the label; The semantic name of a spatial region; For the type attribute of the spatial region; Calculate robot pose With each regional center European distance ,when At that time, associate the region label and output the association table. ); S6. Receive voice commands input by the user, including: spatial location query and exploration control; perform speech recognition and semantic parsing on the commands to obtain the parsing results; the parsing results include: spatial semantic information generated from parsing the spatial location query scenario, and exploration control semantic information generated from parsing the exploration control scenario; In this step, keywords such as "location" and "where" are parsed from the spatial location query scenario to generate spatial semantic information, represented as: ; In the formula, For querying semantic information objects; This indicates that the type of semantic information is a query; The core keyword for the query is location; The keywords "exploration," "priority," and "between devices" are parsed from the exploration control scenario to generate exploration control semantic information, represented as follows: ; In the formula, For control class semantic information objects; The type of semantic information is control; The target direction of the control command is none; The target area for control commands is the equipment room; The control commands have a high priority; Semantic information is output in JSON format; S7. Based on the parsing results of S6, if it is a spatial location query, call the pose-semantic label association data of S5 to determine the spatial semantic label data corresponding to the current pose; if it is an exploration control, generate exploration control parameters. In this step, for spatial location queries, the S5 relational table is called. Input robot pose Match the corresponding semantic tags Output ; For exploration control, the keywords "east / south / west / north" are analyzed and converted into angle parameters. The formula is: ; In the formula, The target heading angle; This is the output of the direction mapping function; This refers to the direction semantic field within the self-control class semantic information; This is the conversion factor from degrees to radians; Generate range parameters based on the region boundaries queried by labels. ;in, The minimum coordinate value in the X-axis direction; The maximum coordinate value in the X-axis direction; The minimum coordinate value in the Y-axis direction; The maximum coordinate value in the Y-axis direction; Mapping weighting coefficients The specific correspondence is as follows: Priority "Preferred": ; Priority "Normal": ; Priority "Secondary": ; Output exploration control parameters ; S8. Receive the exploration control parameters transmitted by S7, adjust the exploration strategy, update the pose and map update data by combining the original image data of S1, and send the updated data back to S5. It receives spatial semantic label data transmitted by S7, converts the spatial semantic label data into voice feedback data, and feeds it back to the user, thus completing the human-computer interaction closed loop. In this step, the pose and map update data are updated, specifically as follows: S4 receives exploration control parameters from S7. ,Adjustment The cost function for path planning is expressed as: ; In the formula, To provide comprehensive value; As a cost of distance; Score the regional value; These are weighting coefficients; The robot is controlled to move along the adjusted path, and SLAM mapping is performed synchronously. The pose update frequency is 10Hz and the map update frequency is 5Hz. The pose error is checked after each frame update, and relocalization is triggered if the error exceeds the threshold. Updated pose The map is sent back to S5, and the relationship table is updated. ; Voice feedback, specifically: S8 receives from S7 The text is converted into a speech signal by the text-to-speech module and output through the speaker; after the feedback is completed, a closed-loop confirmation signal is output to indicate the end of the single interaction.

[0020] Therefore, the present invention adopts the above-mentioned indoor mobile robot adaptive fusion localization mapping voice interaction method and system. This method improves the stability and reliability of localization mapping in complex environments, reduces low-quality data interference, realizes deep fusion of voice and localization mapping, and significantly improves the efficiency and flexibility of human-machine collaborative exploration.

[0021] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0022] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An adaptive fusion localization, mapping, and voice interaction method for indoor mobile robots, characterized in that, Includes the following steps: S1. Through the multi-source sensor data acquisition module, synchronously acquire the raw image data from the visual sensor and the raw point cloud data from the lidar; transmit the raw image data to S2, and synchronously transmit the raw image data and the raw point cloud data to S4. S2, the environmental state perception module receives the raw image data transmitted by S1, extracts environmental feature parameters through analysis, and transmits the environmental feature parameters to S3; S3, the master-slave sensor weight allocation module calculates the illumination confidence level based on the environmental feature parameters extracted by S2 through linear normalization or piecewise mapping, and constructs and outputs dynamic weight parameters based on the illumination confidence level; S4, the localization and mapping module receives the raw image data and point cloud data from S1 and the dynamic weight parameters from S3. It performs weighted modeling on the visual observation information and the laser observation information. After weighted processing of the two types of observation residual terms, it performs pose estimation and environmental map construction, and outputs robot pose information and environmental map information. S5. Divide the environmental map information output by S4 into spatial regions and assign semantic labels, and match the robot pose information of S4 with the spatial regions to establish and store pose-semantic label association data. S6. Receive voice commands input by the user, the voice commands including spatial location query and exploration control; perform speech recognition and semantic parsing on the commands to obtain parsing results, the parsing results including spatial semantic information and exploration control semantic information; S7. Based on the parsing results of S6, if it is a spatial location query, call the pose-semantic label association data of S5 to determine the spatial semantic label data corresponding to the current pose; if it is an exploration control, generate exploration control parameters. S8: Receive the exploration control parameters transmitted by S7, adjust the exploration strategy, and update the pose and map update data by combining the original image data from S1. It receives spatial semantic label data transmitted by S7, converts it into voice feedback data and feeds it back to the user, thus completing the human-computer interaction closed loop.

2. The indoor mobile robot adaptive fusion localization mapping and voice interaction method according to claim 1, characterized in that, In S2, the extracted environmental feature parameters include the mean and variance of image brightness, contrast, number of visual feature points, and changes in camera exposure parameters; the original image data is converted into a single-channel grayscale image through grayscale processing, and the grayscale value calculation formula is: ； In the formula, For the three color components of a pixel; These are pixel coordinates.

3. The indoor mobile robot adaptive fusion localization mapping and voice interaction method according to claim 2, characterized in that, In S2, the formula for calculating the average image brightness is: ； In the formula, The average brightness of the image; Image height; Image width; The row index of the pixel; The column index for pixels; Image brightness variance, the formula is: ； In the formula, The brightness variance of the image; Image contrast is calculated using the following formula: ； In the formula, Image contrast; This represents the maximum value of the grayscale image. ( I () represents the minimum value of the grayscale image; Camera exposure parameters The change is calculated using the following formula: ； In the formula, This represents the change in camera exposure parameters; This is the current exposure time; This represents the average exposure time of the first 10 frames.

4. The indoor mobile robot adaptive fusion localization mapping and voice interaction method according to claim 3, characterized in that, In S3, illumination confidence The value range is [0,1], and the formula for the linear normalization model is: ； In the formula, For illumination confidence level; , , As a weighting factor, ; This is the normalized value of the average brightness. This is the normalized value of the image contrast. This is the normalized value of the number of feature points; The formula for the piecewise mapping model is: ； The linear distribution formula is: ； ； In the formula, The weights for visual SLAM; The weights of the laser SLAM are satisfied; ; The formula for segmented allocation is: 。 5. The indoor mobile robot adaptive fusion localization mapping and voice interaction method according to claim 4, characterized in that, In S4, weighted localization mapping employs factor graph optimization and extended Kalman filtering modeling; when using factor graph optimization, factor graphs are constructed. , For pose nodes, For observation factors; The formula for calculating the weighted observation factor is: ； ； In the formula, The weighted visual observation factors; The weighted laser observation factor; The original visual observation factor; The original laser observation factor; When using extended Kalman filtering, the formula for calculating weighted fused observations is: ； In the formula, For the first Weighted fusion of observations at different times; These are visual observations; These are laser observation values.

6. The indoor mobile robot adaptive fusion localization mapping and voice interaction method according to claim 5, characterized in that, In S7, explore control parameters The formula for the target heading angle is: ； In the formula, The target heading angle; This is the output of the direction mapping function; This refers to the direction semantic field within the self-control class semantic information; This is the conversion factor from degrees to radians; Generate range parameters ;in, This represents the minimum coordinate value along the X-axis. The maximum coordinate value in the X-axis direction; The minimum coordinate value in the Y-axis direction; The maximum coordinate value in the Y-axis direction; Mapping weighting coefficients The correspondence is as follows: Priority "Preferred": ; Priority "Normal": ; Priority "Secondary": .

7. The indoor mobile robot adaptive fusion localization mapping and voice interaction method according to claim 6, characterized in that, In S8, the path planning cost function is: ； In the formula, To provide comprehensive value; As a cost of distance; Score the regional value.

8. An indoor mobile robot adaptive fusion localization and mapping voice interaction system, applied to the indoor mobile robot adaptive fusion localization and mapping voice interaction method according to any one of claims 1-7, characterized in that, include: The multi-source sensor data acquisition module is used to execute S1, synchronously acquire raw image data from the visual sensor and raw point cloud data from the lidar, and distribute them to the corresponding modules. An environmental state perception module, connected to the multi-source sensor data acquisition module, is used to execute S2, receive raw image data and extract environmental feature parameters; The master-slave sensor weight allocation module is connected to the environmental state perception module and is used to execute S3, calculate the illumination confidence based on environmental feature parameters and output dynamic weight parameters. The localization and mapping module is connected to the multi-source sensor data acquisition module and the master-slave sensor weight allocation module, respectively, and is used to execute S4 to realize weighted localization and mapping and output pose and map information; The spatial semantic association construction module is connected to the localization and mapping module and is used to execute S5 to divide the spatial region, assign semantic labels and establish pose-semantic label association; The voice command processing module is used to execute S6, receive voice commands, and perform recognition and semantic parsing; The semantic conversion and control parameter generation module is connected to the spatial semantic association construction module and the voice command processing module, respectively, and is used to execute S7 to generate spatial semantic label data or explore control parameters. The execution and feedback module is connected to the semantic conversion and control parameter generation module and the multi-source sensor data acquisition module, respectively, and is used to execute S8, adjust the exploration strategy, update the pose and map, and realize voice feedback.

9. A computer device, characterized in that, include: A processor configured to be coupled to a memory, read and execute instructions and / or program code in the memory to perform the method as described in any one of claims 1-7.

10. A computer-readable medium, characterized in that, The computer-readable medium stores computer program code that, when executed on a computer, causes the computer to perform the method as described in any one of claims 1-7.

Images