A home service robot navigation method and system

Abstract

The application is suitable for the field of robot technology, and provides a household service robot navigation method and system. The method comprises: respectively performing feature extraction on multi-modal environment data perceived by a household service robot to obtain a plurality of modal feature vectors. Through an improved attention mechanism, the plurality of modal feature vectors are subjected to spatio-temporal alignment and weight fusion processing to generate a comprehensive semantic environment vector. According to a preset multi-association feature, a user state label in the comprehensive semantic environment vector is identified to determine a navigation adaptation strategy matched with the user state label. The household service robot is controlled to navigate according to an individualized navigation path generated by the comprehensive semantic environment vector and the navigation adaptation strategy. Through the method, a navigation system of semantic understanding, dynamic decision-making of double special groups and precise execution can be formed, safe adaptation and service navigation of double special groups in a household scene are realized, and thus the navigation accuracy of the household service robot in a complex household environment is improved.

Description

Technical Field

[0001] This application belongs to the field of robotics technology, and in particular relates to a navigation method and system for a home service robot. Background Technology

[0002] With the accelerating aging of the population, the proportion of elderly people in households is constantly rising, while the smart home ecosystem is gradually improving, leading to increasingly diverse application scenarios for home service robots. In the home environment, the elderly and children, as special groups, have unique behavioral characteristics and needs. The elderly walk slowly and tire easily, while children move frequently and have unpredictable activity ranges. This requires home service robot navigation systems to not only recognize the outlines of objects in the environment but also accurately identify the functional attributes of objects (such as armchairs commonly used by the elderly and fragile toys for children) to provide more considerate and safe services for the elderly and children. At the same time, the home environment is a dynamically changing scenario, with frequent occurrences of furniture movement, lighting changes, and temporary obstacles (such as clothes drying or scattered toys). Home service robots need to possess strong robustness to dynamic scenarios, able to respond to these changes in real time, and continuously optimize navigation strategies through physical interaction feedback to better adapt to the core needs of the home environment: "actively adapting to people and dynamically responding to the environment."

[0003] However, current home service robots suffer from shortcomings such as insufficient semantic perception, lack of collaborative adaptation for dual special groups, and weak robustness in dynamic scenarios, resulting in problems such as low navigation accuracy and poor personalization and long-term adaptability in complex home environments. Summary of the Invention

[0004] This application provides a navigation method and system for home service robots, which can solve the technical problems of low navigation accuracy and poor personalization and long-term adaptability in complex home environments caused by the shortcomings of existing home service robots, such as insufficient semantic perception, lack of collaborative adaptation between two special groups, and weak robustness in dynamic scenes.

[0005] In a first aspect, embodiments of this application provide a navigation method for a home service robot, the method comprising: Feature extraction is performed on the multimodal environmental data perceived by the home service robot to obtain multiple modal feature vectors; wherein, the multimodal environmental data includes visual data, voice data, tactile data, and inertial data; An improved attention mechanism is used to perform spatiotemporal alignment and weight fusion processing on the multiple modal feature vectors to generate a comprehensive semantic environment vector. The comprehensive semantic environment vector is used to represent the environmental information perceived by the home service robot. The improved attention mechanism is used to realize the timestamp alignment and spatial coordinate alignment of multiple modal feature vectors, and to dynamically allocate the fusion weights of multiple modal feature vectors according to the environmental scene. The user state labels in the comprehensive semantic environment vector are identified based on preset multi-link features, and a navigation adaptation strategy matching the user state labels is determined; wherein, the preset multi-link features are pre-set features used to characterize typical user states and behavior patterns; the navigation adaptation strategy is used to adjust the navigation adaptation parameters of the home service robot; The home service robot is controlled to navigate according to the personalized navigation path generated by the comprehensive semantic environment vector and the navigation adaptation strategy.

[0006] In one possible implementation of the first aspect, before extracting features from the multimodal environmental data perceived by the home service robot to obtain multiple modal feature vectors, the method includes: The visual data, which includes scene images and user images, is acquired through the color depth camera in the semantic vision unit. The speech data, which includes ambient speech and user commands, is acquired through the microphone array in the speech semantic unit. The tactile force sensing unit collects tactile force data containing obstacle contact information and ground material information; The inertial data of the home service robot is acquired through the inertial measurement unit; wherein, the inertial data is used to correct the spatiotemporal alignment deviation between the visual target pose and the multimodal environment data.

[0007] In one possible implementation of the first aspect, the step of generating a comprehensive semantic environment vector by performing spatiotemporal alignment and weight fusion processing on the multiple modal feature vectors through an improved attention mechanism includes: The multiple modal feature vectors are preprocessed to obtain preprocessed multiple modal feature vectors; Based on the spatiotemporal calibration module in the improved attention mechanism, the preprocessed multiple modal feature vectors are spatiotemporally aligned to obtain aligned visual feature vectors, aligned speech feature vectors, aligned tactile feature vectors, and aligned inertial feature vectors. Based on adaptive weight values, the aligned visual feature vector, the aligned speech feature vector, the aligned tactile feature vector, and the aligned inertial feature vector are subjected to weight fusion processing to obtain a fused feature matrix; wherein, the adaptive weight values ​​are determined according to the confidence values ​​and scene modes corresponding to the multiple modal feature vectors; The fused feature matrix is ​​mapped into a low-dimensional dense vector through a fully connected network to obtain the comprehensive semantic environment vector.

[0008] In one possible implementation of the first aspect, the step of identifying user state labels in the comprehensive semantic environment vector based on preset multi-link features and determining a navigation adaptation strategy matching the user state labels includes: Based on the preset multi-linked features, visual features, speech features, and behavioral features are extracted from the comprehensive semantic environment vector; The visual features, voice features, and behavioral features are matched with a preset crowd feature template to obtain user status tags in the environment; the user status tags include at least user identity, user behavioral features, and user location. Based on the user status tags and the preset adaptation template, determine the navigation adaptation strategy that matches the user status tags.

[0009] In one possible implementation of the first aspect, before the control of the home service robot to navigate according to the personalized navigation path generated based on the integrated semantic environment vector and the navigation adaptation strategy, the method includes: Based on the comprehensive semantic environment vector and the incremental synchronous localization and mapping algorithm, a two-layer dynamic semantic map is constructed; wherein, the two-layer dynamic semantic map is used to represent the static fixed information and dynamic change information of the environment in which the home service robot is located; Based on the two-layer dynamic semantic map and the navigation adaptation strategy, the personalized navigation path is generated through a semantic reinforcement learning path planning hybrid algorithm.

[0010] In one possible implementation of the first aspect, the construction of a two-layer dynamic semantic map based on the integrated semantic environment vector and the incremental simultaneous localization and mapping (SMR) algorithm includes: Static semantic features are extracted from the visual data using a semantic segmentation model. The static semantic features, the inertial data, and the ground material data in the tactile data are fused together to construct the static semantic layer data; The spatial coordinates of dynamic target features in real-time visual data collected by the semantic visual unit are predicted by the dual-time long short-term memory trajectory prediction algorithm, and the spatial coordinates of the dynamic target features are corrected by the incremental synchronous positioning and map building algorithm to construct dynamic semantic layer data. The object semantic attributes in the comprehensive semantic environment vector, the user status labels, and the spatial coordinate values ​​of static semantic features in the static semantic layer data, as well as the spatial coordinate values ​​of dynamic target features in the dynamic semantic layer data, are bound together to obtain the two-layer dynamic semantic map.

[0011] In one possible implementation of the first aspect, generating the personalized navigation path using a semantic reinforcement learning path planning hybrid algorithm based on the two-layer dynamic semantic map and the navigation adaptation strategy includes: Based on the semantic attributes of objects and the user status tags in the dual-layer dynamic semantic map, the environmental location is risk-classified to obtain the scene risk classification result. According to the navigation adaptation strategy, initialize the navigation adaptation parameters of the home service robot respectively; Based on the dual-layer dynamic semantic map, the scene risk classification results, and the navigation adaptation parameters, an initial navigation path is generated using the semantic reinforcement learning path planning hybrid algorithm. The initial navigation path is iteratively optimized based on a preset reward function to obtain the personalized navigation path.

[0012] In one possible implementation of the first aspect, the method includes, during the process of controlling the home service robot to navigate according to the personalized navigation path generated by the integrated semantic environment vector and the navigation adaptation strategy, the method includes: Determine whether the home service robot has experienced a collision during navigation; If a collision occurs, the semantic attributes of the collided obstacle and the navigation adaptation deviation are recorded, and the obstacle avoidance strategy update function is called to adjust the risk classification threshold and navigation adaptation parameters to obtain the optimized navigation adaptation parameters. If no collision occurs, the personalized navigation path will continue to be executed and the navigation adaptation parameters will be retained.

[0013] In one possible implementation of the first aspect, after the home service robot completes navigation, the method further includes: The execution feedback data generated by the home service robot during navigation is obtained, wherein the execution feedback data includes at least: collision information, path deviation information, and user feedback data; The execution feedback data, the comprehensive semantic environment vector, and the personalized navigation path are integrated to obtain the local navigation full-process data of the home service robot; Based on the anonymized local navigation process data, the local semantic model of the home service robot is trained, and the model parameters of the local semantic model are optimized. The optimized local semantic model parameters are encrypted and then uploaded to the cloud server, so that the cloud server can aggregate the model parameters of multiple local semantic models through a federated averaging algorithm to generate a globally optimized model. Based on the global optimization model and user habit data issued by the cloud server, the model parameters of the local semantic model are fine-tuned to obtain the optimized local semantic model.

[0014] Secondly, embodiments of this application provide a home service robot navigation system, including: a modal semantic perception layer, an embodied intelligence decision-making layer, and a personalized execution layer; wherein, The modal semantic perception layer, connected to the embodied intelligent decision-making layer, is used to extract features from the multimodal environmental data perceived by the home service robot to obtain multiple modal feature vectors. Furthermore, through an improved attention mechanism, the multiple modal feature vectors are spatiotemporally aligned and weighted to generate a comprehensive semantic environment vector. The multimodal environmental data includes visual data, speech data, tactile data, and inertial data. The comprehensive semantic environment vector represents the environmental information perceived by the home service robot. The improved attention mechanism aligns the timestamps and spatial coordinates of the multiple modal feature vectors and dynamically allocates fusion weights based on the environmental scene. The embodied intelligent decision-making layer, connected to the personalized execution layer, is used to identify the user state in the comprehensive semantic environment vector based on preset multi-link features, and determine a navigation adaptation strategy matching the user state. The preset multi-link features are pre-defined features used to characterize typical user states and behavioral patterns. The navigation adaptation strategy is used to adjust the navigation adaptation parameters of the home service robot during navigation. Furthermore, it generates a personalized navigation path based on the comprehensive semantic environment vector and the navigation adaptation strategy. The personalized execution layer is used to control the home service robot to navigate according to the personalized navigation path and generate execution feedback data.

[0015] In one possible implementation of the second aspect, the system includes: a federated learning optimization layer; The federated learning optimization layer establishes communication connections with the embodied intelligent decision-making layer and the personalized execution layer, respectively. It is used to train the local semantic model of the home service robot based on the received personalized navigation path, execution feedback data, and comprehensive semantic environment vector, optimizing the model parameters of the local semantic model; and to encrypt and upload the optimized local semantic model parameters to a cloud server, enabling the cloud server to perform federated aggregation to obtain a globally optimized model; and to fine-tune the model parameters of the local semantic model based on the globally optimized model and user habit data, obtaining an optimized local semantic model.

[0016] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the home service robot navigation method described in any of the above claims.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the home service robot navigation method described in any of the preceding claims.

[0018] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the home service robot navigation method described in any one of the first aspects.

[0019] The beneficial effects of the embodiments in this application compared with the prior art are: This application provides a navigation method for a home service robot. The method includes: First, extracting features from the multimodal environmental data perceived by the home service robot to obtain multiple modal feature vectors; wherein the multimodal environmental data includes visual data, voice data, tactile data, and inertial data. Then, using an improved attention mechanism, spatiotemporal alignment and weight fusion processing are performed on the multiple modal feature vectors to generate a comprehensive semantic environment vector; wherein the comprehensive semantic environment vector represents the environmental information perceived by the home service robot; the improved attention mechanism is used to align the timestamps and spatial coordinates of the multiple modal feature vectors and dynamically allocate the fusion weights of the multiple modal feature vectors according to the environmental scene. User state labels in the comprehensive semantic environment vector are identified based on preset multi-linked features to determine a navigation adaptation strategy matching the user state labels; wherein the preset multi-linked features are pre-defined features used to represent typical user states and behavioral patterns; the navigation adaptation strategy is used to adjust the navigation adaptation parameters of the home service robot. Finally, the home service robot is controlled to navigate according to a personalized navigation path generated by the comprehensive semantic environment vector and the navigation adaptation strategy. This method enables the formation of a navigation system that combines semantic understanding, dynamic decision-making for two special groups, and precise execution. It achieves safe adaptation and service-oriented navigation for two special groups in home scenarios, thereby improving the navigation accuracy of home service robots in complex home environments. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the structure of a home service robot navigation system provided in one embodiment of this application; Figure 2 This is a flowchart illustrating a navigation method for a home service robot according to an embodiment of this application; Figure 3 This is a schematic diagram of the navigation timing process for a home service robot provided in one embodiment of this application; Figure 4 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0023] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0024] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0026] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0028] With the aging population and the improvement of the smart home ecosystem, the navigation requirements of home care robots have been upgraded from "basic obstacle avoidance and positioning" to higher-level requirements such as "semantic environmental understanding + personalized adaptation for dual special groups + physical interaction feedback". The core requirements are reflected in three aspects: First, semantic environmental perception, which requires accurate identification of the functional attributes of objects in the home environment (such as armchairs commonly used by the elderly and fragile toys for children) rather than just recognizing outlines; Second, adaptation for dual special groups, which requires simultaneous adaptation to the slow gait and fatigue characteristics of the elderly and the sudden movement and uncertain activity range of children to achieve predictive safe navigation; Third, robustness to dynamic scenes, which requires coping with high-frequency dynamic scenes in the home such as furniture movement, changes in lighting, and temporary obstacles (clothes drying, scattered toys), while optimizing navigation strategies by combining physical interaction feedback.

[0029] Embodied intelligence, as a cutting-edge field at the intersection of artificial intelligence and robotics, is based on the integration of perception, cognition, decision-making and action through the dynamic interaction between physical entities and the environment. It emphasizes that intelligence needs to continuously evolve through real-time interaction between the "body" and the environment.

[0030] Current navigation technology for home service robots faces the following significant bottlenecks: (1) Lack of semantic perception dimension and insufficient service targeting. Existing technologies can only identify the geometric outline of obstacles (such as rectangular objects) and cannot understand their functional attributes and semantic relationships (such as walking aids for the elderly or toy cars for children). This results in a lack of service orientation in navigation strategies. For example, it may mistakenly treat a commonly used armchair for the elderly as an ordinary obstacle and bypass it, affecting the convenience of the service. According to tests, the accuracy rate of existing technologies in semantic recognition of the home environment is less than 65%, and the accuracy rate in recognizing special items is less than 60%.

[0031] (2) Lack of adaptation to dual special populations, resulting in significant safety hazards. Existing technologies either lack population adaptation or only adapt to the elderly or children, failing to achieve collaborative adaptation to dual special populations; when accompanying the elderly, there is a lack of gait matching and fatigue warning association, and when accompanying children, it is impossible to predict sudden movement trajectories within 1.5 seconds, with obstacle avoidance response delay exceeding 380ms, resulting in a high risk of collision; it is also impossible to dynamically adjust navigation priority based on user behavior characteristics (such as the elderly bending over or children running).

[0032] (3) The dynamic scene is not robust and the physical interaction is not adapted. The existing navigation relies on the preset environment map, which is slow to respond to dynamic changes such as furniture movement and temporary obstacles (such as scattered toys and clothes drying). The map update error exceeds 10cm. In the case of sudden changes in lighting (such as sudden lights off or direct sunlight), the failure of the visual sensor causes the positioning accuracy to drop by more than 50%, and the navigation success rate is less than 72%. It lacks tactile and force perception and cannot adjust the movement parameters according to the ground material (carpet, wooden floor), which makes it easy to slip or bump.

[0033] (4) The lack of a closed-loop embodied intelligence results in poor long-term adaptability. Existing technologies lack a closed-loop mechanism of "perception-action-feedback-optimization" and cannot optimize navigation strategies through their own action effects. For example, they may continue to use the same obstacle avoidance method after colliding with a certain type of obstacle multiple times. Even if federated learning is used, it is not combined with semantic perception and human adaptation, and cannot achieve personalized and continuous iteration of navigation strategies.

[0034] Therefore, this application introduces embodied intelligence technology and integrates four modal semantic information of vision, voice, touch and force to construct a personalized closed-loop navigation system for dual special groups. This solves the three major pain points of existing technologies in semantic perception, dual special group adaptation and dynamic robustness, and fills the application gap of embodied intelligence in the field of navigation for dual special groups in home service robots.

[0035] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of a home service robot navigation system according to an embodiment of this application. The home service robot navigation system 100 includes: a modal semantic perception layer 101, an embodied intelligence decision-making layer 102, and a personalized execution layer 103.

[0036] The modal semantic perception layer 101, connected to the embodied intelligence decision layer 102, is used to extract features from the multimodal environmental data perceived by the home service robot, obtaining multiple modal feature vectors. Furthermore, an improved attention mechanism is used to perform spatiotemporal alignment and weight fusion processing on the multiple modal feature vectors to generate a comprehensive semantic environment vector. The multimodal environmental data includes visual data, speech data, tactile data, and inertial data; the comprehensive semantic environment vector represents the environmental information perceived by the home service robot; and the improved attention mechanism is used to align the timestamps and spatial coordinates of the multiple modal feature vectors and dynamically allocate fusion weights based on the environmental scene.

[0037] The embodied intelligent decision-making layer 102, connected to the personalized execution layer 103, is used to identify the user state in the comprehensive semantic environment vector based on preset multi-linked features, and determine a navigation adaptation strategy matching the user state. The preset multi-linked features are pre-defined features used to characterize typical user states and behavioral patterns. The navigation adaptation strategy is used to adjust the navigation adaptation parameters of the home service robot during navigation. Furthermore, a personalized navigation path is generated based on the comprehensive semantic environment vector and the navigation adaptation strategy.

[0038] The personalized execution layer 103 is used to control the home service robot to navigate according to the personalized navigation path and generate execution feedback data.

[0039] In the home service robot navigation system 100, the modal semantic perception layer 101, the embodied intelligent decision-making layer 102, and the personalized execution layer 103 are sequentially connected in communication.

[0040] Furthermore, such as Figure 1 As shown, the home service robot navigation system 100 includes a federated learning optimization layer 104.

[0041] The federated learning optimization layer 104 establishes communication connections with the embodied intelligence decision-making layer 102 and the personalized execution layer 103, respectively. It is used to train the local semantic model of the home service robot based on the received personalized navigation path, execution feedback data, and comprehensive semantic environment vector, and optimize the model parameters of the local semantic model. It also encrypts the model parameters of the optimized local semantic model and uploads them to the cloud server so that the cloud server can perform federated aggregation to obtain a globally optimized model. Finally, it fine-tunes the model parameters of the local semantic model based on the globally optimized model and user habit data to obtain the optimized local semantic model.

[0042] Understandably, this home service robot navigation system is based on the core concept of embodied intelligence and constructs four major modules: "modal semantic perception layer, embodied intelligence decision-making layer, personalized execution layer, and federated learning optimization layer". This forms a closed-loop navigation system that goes through the entire process of "semantic understanding - dynamic decision-making for dual special groups - precise execution - physical feedback - continuous optimization", enabling safe adaptation and service-oriented navigation for dual special groups in home scenarios.

[0043] Please see Figure 2 , Figure 2 This is a flowchart illustrating a navigation method for a home service robot according to an embodiment of this application. As an example and not a limitation, this method can be applied to or run in terminal devices such as home service robots. The method includes: S11. Extract features from the multimodal environmental data perceived by the home service robot to obtain multiple modal feature vectors.

[0044] Multimodal environmental data includes visual data, voice data, tactile data, and inertial data.

[0045] S12. Through an improved attention mechanism, spatiotemporal alignment and weight fusion processing are performed on multiple modal feature vectors to generate a comprehensive semantic environment vector.

[0046] Among them, the comprehensive semantic environment vector is used to represent the environmental information perceived by the home service robot; the improved attention mechanism is used to realize the timestamp alignment, spatial coordinate alignment and dynamic allocation of fusion weights of multiple modal feature vectors according to the environmental scene.

[0047] S13. Identify user status labels in the comprehensive semantic environment vector based on preset multi-link features, and determine the navigation adaptation strategy that matches the user status labels.

[0048] Among them, the preset multi-connection features are pre-defined features used to characterize typical user states and behavioral patterns; the navigation adaptation strategy is used to adjust the navigation adaptation parameters of the home service robot.

[0049] S14. Control the home service robot to navigate according to the personalized navigation path generated by the comprehensive semantic environment vector and navigation adaptation strategy.

[0050] It should be noted that home service robots are intelligent robots that provide various services to families, such as cleaning, companionship, and moving items. They have the ability to perceive the environment, make decisions, and perform actions, and can also be called family nanny robots.

[0051] Multimodal environmental data can be understood as data from different sensory channels. This data exists in various forms and can more comprehensively reflect environmental information. In this embodiment, multimodal environmental data may include visual data, voice data, tactile data, and inertial data.

[0052] The visual data can include scene images and user images to help the home service robot understand the layout and structure of its surrounding environment, facilitating better interaction and service delivery. Scene images mainly consist of overall views of various objects and areas within the robot's environment, including visual information about environmental elements such as furniture, walls, doors, and windows. User images primarily depict the user within the robot's environment, used to identify the user's posture, actions, and location. Voice data can include environmental voice and user commands. User commands are voice instructions given by the user to the robot, instructing it to perform corresponding tasks, such as cleaning the living room or fetching a water glass. Environmental voice includes various sounds in the environment other than user commands, such as television sounds, appliance sounds, and conversations among family members. This environmental voice helps the robot understand the dynamics of its surroundings. Tactile data can include obstacle contact information and ground material information. Obstacle contact information can be understood as the information collected by the tactile force sensing unit regarding the magnitude, direction, and location of the contact force when the home service robot comes into contact with an obstacle during movement. This information helps the robot determine the nature and location of obstacles, allowing it to adjust its movement strategy in a timely manner to avoid collisions or achieve more flexible interactions. Ground material information can be understood as information about the ground characteristics perceived by the tactile and force sensing units through contact with the ground, such as the hardness, smoothness, and friction of the ground. Different ground materials can have different effects on the mobility of home service robots. Understanding ground material information can help the robot optimize its movement methods, ensuring stable and safe operation. Inertial data refers to parameter information about the robot's speed, acceleration, and displacement, collected through the inertial measurement unit.

[0053] Modal feature vectors are vectors obtained by extracting features from environmental data of different modalities (such as vision, speech, tactile sensation, inertia, etc.). In this embodiment, the multiple modal feature vectors corresponding to multimodal environmental data are semantic vision feature vectors, speech semantic feature vectors, tactile sensation feature vectors, and inertial feature vectors. These vectors contain key feature information of the corresponding modal data and are the basis for subsequent processing and analysis.

[0054] The improved attention mechanism enhances upon the traditional attention mechanism by ensuring consistency in temporal and spatial dimensions across different modalities, and by assigning different weights to each modality based on its importance in different scenarios. The comprehensive semantic environment vector integrates information from multiple modal feature vectors to represent the environmental information perceived by the home service robot; that is, it includes semantic information about various elements in the environment. The comprehensive semantic environment vector can include at least information such as object semantic attributes, user state, and terrain physical features.

[0055] The preset multi-features are a pre-defined set of features used to characterize typical user states and behavioral patterns. Learning from and analyzing large amounts of user behavior data helps identify user state labels. User state labels identify the user's current state, such as whether the user is elderly or a child, resting or exercising, etc. These labels reflect the user's category, behavioral patterns, and needs. The navigation adaptation strategy is a strategy determined based on the user state labels to adjust the navigation adaptation parameters of the home service robot. The navigation adaptation strategy can include adjustments to navigation speed, obstacle avoidance methods, etc. Different user states may require the robot to adopt different navigation methods; different navigation adaptation strategies allow the home service robot to make adaptive adjustments. For example, when the user is resting, the robot needs to navigate more quietly and slowly. The personalized navigation path is a generated navigation path suitable for the current environment and user state, better meeting the navigation needs of the home service robot in the home environment.

[0056] Specifically, firstly, the home service robot acquires multimodal environmental data from its surroundings using various sensors (such as color depth cameras, microphones, tactile sensors, and inertial measurement units). Each modality of data is then processed using a corresponding feature extraction algorithm. For example, convolutional neural networks may be used to extract image features from visual data, while Mel-frequency cepstral coefficients and other methods may be used to extract speech features from speech data, ultimately resulting in multiple modal feature vectors.

[0057] Next, using an improved attention mechanism, firstly, multiple modal feature vectors are aligned in terms of timestamps and spatial coordinates to ensure accurate correspondence between different modal data in both time and space, eliminating inconsistencies caused by differences in data acquisition time or spatial location. Then, based on the current environment, corresponding fusion weights are dynamically assigned to each modal feature vector. For example, the basic weights for each modal feature vector are: semantic visual feature vector 0.4, speech semantic feature vector 0.3, tactile-force feature vector 0.2, and inertial feature vector 0.1. When the current environment is a children's scenario, the weight of the semantic visual feature vector is increased to 0.5; when the current environment is an elderly scenario, the weight of the speech semantic feature vector is increased to 0.4. Finally, the aligned and weighted multiple modal feature vectors are fused to generate a comprehensive semantic environment vector, which more comprehensively and accurately reflects the environmental information of the robot.

[0058] Then, the fused comprehensive semantic environment vector is compared and analyzed with preset multi-linked features. These preset multi-linked features contain feature information about various typical user states and behavioral patterns; by comparing them, the user state label corresponding to the current comprehensive semantic environment vector can be determined. Based on the identified user state label, a matching navigation adaptation strategy is selected from a pre-set navigation adaptation strategy library. For example, if the user is identified as being in a resting state, a corresponding quiet, slow navigation adaptation strategy is selected to adjust the navigation adaptation parameters of the home service robot, such as speed and obstacle avoidance sensitivity.

[0059] Finally, by combining a comprehensive semantic environment vector (containing environmental information) and a navigation adaptation strategy (reflecting the navigation parameter adjustment requirements corresponding to user needs), a personalized navigation path is generated through a path planning algorithm. The home service robot is then controlled to move according to the generated personalized navigation path, achieving navigation functionality that better suits the current environment and user state, thereby improving the robot's service quality and user experience in the home environment.

[0060] It's worth noting that after generating a personalized navigation path, the home service robot can achieve precise execution through the adaptive drive module and the safety interaction module. The adaptive drive module uses Mecanum wheels for omnidirectional drive (12cm diameter, maximum speed 0.8m / s), combined with rapid response data from pressure sensors (less than 3 microseconds), and adjusts wheel speed and torque in real time using PID and fuzzy control algorithms. For example, when driving on carpet, torque is increased by 20% and wheel speed is reduced by 15% to prevent slippage; when driving on tile, wheel speed is increased to 0.5m / s to ensure efficiency; when encountering a threshold less than 2cm high, the robot uses a linked elastic suspension system (5-20mm shock absorption travel) to adjust its height, achieving a success rate of approximately 98%. The safety interaction module deploys infrared proximity sensors (range 0-50cm, accuracy ±1cm, response delay less than 30ms), which decelerate when the robot is less than 30cm from the user and stop abruptly when the distance is less than 10cm, reducing the collision rate to below 0.5%. In addition, it also integrates voice and light level feedback (such as slow flashing green light in elderly mode, constant yellow light in children mode, and breathing blue light in coexistence mode), emergency voice commands (such as "stop" and "get out of the way", with the highest priority and a response delay of less than 100ms, adapted to the emergency needs of the elderly and children), etc.

[0061] It is understood that this application provides a navigation method for a home service robot. The method includes: First, extracting features from the multimodal environmental data perceived by the home service robot to obtain multiple modal feature vectors; wherein the multimodal environmental data includes visual data, voice data, tactile data, and inertial data. Then, through an improved attention mechanism, spatiotemporal alignment and weight fusion processing are performed on the multiple modal feature vectors to generate a comprehensive semantic environment vector; wherein the comprehensive semantic environment vector is used to represent the environmental information perceived by the home service robot; the improved attention mechanism is used to achieve timestamp alignment, spatial coordinate alignment, and dynamic allocation of fusion weights for the multiple modal feature vectors according to the environmental scene. User state labels in the comprehensive semantic environment vector are identified based on preset multi-linked features to determine a navigation adaptation strategy matching the user state labels; wherein the preset multi-linked features are pre-defined features used to represent typical user states and behavioral patterns; the navigation adaptation strategy is used to adjust the navigation adaptation parameters of the home service robot. Finally, the home service robot is controlled to navigate according to the personalized navigation path generated by the comprehensive semantic environment vector and the navigation adaptation strategy. This method enables the formation of a navigation system that combines semantic understanding, dynamic decision-making for two special groups, and precise execution. It achieves safe adaptation and service-oriented navigation for two special groups in home scenarios, thereby improving the navigation accuracy of home service robots in complex home environments.

[0062] In one possible implementation, before extracting features from the multimodal environmental data perceived by the home service robot to obtain multiple modal feature vectors, the method includes: Visual data, including scene images and user images, is acquired using a color depth camera in the semantic vision unit.

[0063] Voice data, including ambient speech and user commands, is acquired through the microphone array in the speech semantic unit.

[0064] The tactile force sensing unit collects tactile data containing obstacle contact information and ground material information.

[0065] Inertial data of the home service robot is collected through the inertial measurement unit; the inertial data is used to correct the spatiotemporal alignment deviation between the visual target pose and the multimodal environment data.

[0066] The modal semantic perception layer of the home service robot navigation system includes a semantic vision unit, a speech semantic unit, a tactile force perception unit, and an inertial measurement unit.

[0067] The semantic vision unit (SVI) utilizes a color depth camera to acquire visual data, including scene and user images. The SVI is a module on the home service robot used for visual perception and related semantic understanding. It integrates devices capable of acquiring visual information and performing semantic processing, allowing the robot to acquire visual data to support its recognition and analysis of the surrounding environment and users. For example, the SVI integrates an RGB-D color depth camera (core parameters: resolution 1280×720@30fps, ranging range 0.1-5m, depth accuracy ±2cm; integrated IMU module and infrared illumination unit) for visual data acquisition.

[0068] Meanwhile, the semantic vision unit is also equipped with an improved YOLOv8 semantic segmentation model (such as a new 120-category family-specific semantic label library, including special item labels such as elderly walking aids and children's fragile toys, with label weights dynamically adapted to family scenarios), and innovatively introduces a dual-temporal long short-term memory network LSTM prediction module to predict the movement status of the elderly (gait trend) and children (sudden movement trajectory) respectively. The prediction error can be less than 5cm. In low-light environments (illuminance <10 lux), it is linked with infrared supplementary lighting, and the semantic recognition accuracy can reach about 95%, and the special item recognition accuracy can reach about 98%.

[0069] The microphone array in the speech semantic unit can also collect speech data containing environmental speech and user commands. The speech semantic unit is the module on the home service robot responsible for speech acquisition and semantic understanding. The microphone array in the speech semantic unit can effectively collect sound signals, and through processing and analysis of this speech data, the robot can understand the speech content in the environment and the user's commands.

[0070] For example, in this embodiment, the speech semantic unit integrates a microphone array (4 microphones, 3-meter voice capture, signal-to-noise ratio ≥60dB, supporting VAD detection and DOA positioning). Combined with a speech semantic understanding model and sentiment analysis algorithm, it parses natural language commands such as "go to the living room to get Grandpa's water cup" and "avoid the baby's toys," realizing the association and matching of speech features with the identity of the people. It distinguishes between the elderly, children, and adults through voiceprint and speech rate, and dynamically improves navigation priority through speech emotions (such as the elderly's hurried voice and the child's crying).

[0071] Furthermore, tactile and force sensing units can collect tactile and force data containing obstacle contact information and ground material information. These tactile and force sensing units are modules on the home service robot used to perceive tactile and force information. They can detect various mechanical information when the robot comes into contact with surrounding objects, providing feedback to the robot about obstacles and ground conditions. For example, in this embodiment, the home service robot has 16 digital capacitive touch sensors deployed around its body (detection range 0-5N, accuracy ±0.1N) to sense the softness / hardness of obstacles (distinguishing between plush toys and hard furniture) and collision force, triggering a graded obstacle avoidance strategy. The robot's wheels integrate pressure sensor modules (pressure range 0.1N-10N, rise time <3 microseconds) to collect ground contact pressure in real time, accurately determining the ground material (carpet, wood flooring, tile) and flatness. The collaborative data from the digital capacitive touch sensors and pressure sensors provides a physical interaction basis for adjusting movement parameters, filling the gap in physical perception in existing technologies.

[0072] Inertial data of the home service robot can also be acquired through an inertial measurement unit (IMU). An IMU is a sensor device that measures the motion state of an object, typically including components such as accelerometers and gyroscopes. The IMU can obtain inertial data such as the robot's velocity, acceleration, and displacement. This IMU can be integrated with an RGB-D color depth camera to collect the home service robot's own angular velocity, acceleration, and other inertial data in real time, helping to correct the spatiotemporal alignment deviation between the visual target pose and the multimodal environment data. The visual target pose can be understood as the position and orientation information of the target object in the visual coordinate system, used to describe the spatial state of the target object relative to the camera, determined during visual data processing. Spatiotemporal alignment deviation can be understood as the time and space mismatch that occurs when fusing multimodal environment data due to differences in the time and spatial location of data acquisition from different modalities. For example, visual data and voice data may not be acquired at the same time, or different sensors may be installed in different spatial locations, making it impossible for the information reflected by the data to accurately correspond in the time and spatial dimensions.

[0073] Inertial data, by accurately reflecting the robot's motion state, can be used to correct the pose of visual targets, making it more accurate in reflecting the actual position and attitude of the target object during the robot's movement. Simultaneously, by establishing temporal correlations and spatial mappings between different modal data and inertial data, inertial data can be used to perform spatiotemporal alignment of multiple modal data, eliminating mismatches caused by differences in acquisition time and spatial location. This provides a more accurate foundation for subsequent modal feature extraction and multimodal data fusion.

[0074] In one possible implementation, an improved attention mechanism is used to perform spatiotemporal alignment and weight fusion on multiple modal feature vectors to generate a comprehensive semantic environment vector, including: Multiple modal feature vectors are preprocessed to obtain preprocessed modal feature vectors.

[0075] Based on the spatiotemporal calibration module in the improved attention mechanism, the preprocessed multiple modal feature vectors are spatiotemporally aligned to obtain aligned visual feature vectors, aligned speech feature vectors, aligned tactile feature vectors, and aligned inertial feature vectors.

[0076] Based on adaptive weight values, the aligned visual feature vector, aligned speech feature vector, aligned tactile feature vector, and aligned inertial feature vector are weighted and fused to obtain a fused feature matrix; where the adaptive weight values ​​are determined based on the confidence values ​​and scene modes corresponding to multiple modal feature vectors.

[0077] By mapping the fused feature matrix into a low-dimensional dense vector through a fully connected network, a comprehensive semantic environment vector is obtained.

[0078] Specifically, since feature vectors from different modalities may have different data distributions, scales, and ranges, direct processing may affect the subsequent spatiotemporal alignment and weight fusion effects. First, it is necessary to preprocess the feature vectors of multiple modalities. Common preprocessing methods include normalization and standardization. Preprocessing can make the feature vectors of each modality have similar data characteristics, providing a good foundation for subsequent processing.

[0079] Subsequently, based on the spatiotemporal calibration module in the improved attention mechanism, the preprocessed multiple modal feature vectors are spatiotemporally aligned to obtain aligned visual feature vectors, aligned speech feature vectors, aligned tactile feature vectors, and aligned inertial feature vectors.

[0080] The spatiotemporal calibration module is a key component of the improved attention mechanism, addressing the alignment of multiple modal feature vectors across time and space. Since the acquisition time and spatial location of data from different modalities may vary—for example, visual and speech data may not be acquired at the same time, or different sensors may be installed in different spatial positions—this spatiotemporal calibration module aligns these preprocessed modal feature vectors in both time and space. For time alignment, methods such as timestamp matching and interpolation are used to ensure that the feature vectors correspond consistently on the time axis. For spatial alignment, the feature vectors are mapped to the same spatial coordinate system based on the relative positions and coordinate transformations between sensors. After spatiotemporal alignment, aligned visual, speech, tactile, and inertial feature vectors are obtained, consistent in both time and space. This spatiotemporal calibration module eliminates these differences, ensuring spatiotemporal consistency among the feature vectors, facilitating subsequent fusion processing.

[0081] Then, based on the confidence values ​​corresponding to multiple modal feature vectors and the current scene mode, an adaptive weight value is dynamically determined for each modal feature vector. Following these adaptive weight values, the aligned visual, speech, tactile, and inertial feature vectors are weighted and fused to obtain a fused feature matrix. This matrix integrates the information from each modal feature vector, and the contribution of each modal information is reasonably adjusted according to its importance and reliability.

[0082] The adaptive weight values ​​are dynamically determined based on the confidence values ​​of multiple modal feature vectors and the scene mode. In the weight fusion process, different weights are assigned to different modal feature vectors according to their importance and reliability in the current scene, making the fusion result more accurately reflect environmental information. The confidence value is an indicator of the reliability of a modal feature vector, reflecting the accuracy and credibility of the information contained within it. The confidence value is usually determined based on factors such as data acquisition quality and the stability of the feature extraction algorithm; a higher confidence value indicates more reliable information from the modal feature vector. The scene mode is a classification description of the environment in which the home service robot operates. Different scene modes represent different environmental characteristics and task requirements. For example, scene modes can be divided into elderly mode, child mode, and elderly-child coexistence mode. The scene mode affects the importance of each modal feature vector; for example, in elderly mode, voice feature vectors may be relatively more important, while in child mode, visual feature vectors may be relatively more important, thus affecting the determination of the adaptive weight values. The fusion feature matrix is ​​the matrix obtained by fusing multiple modal feature vectors after spatiotemporal alignment processing according to the adaptive weight values. This matrix integrates information from the feature vectors of various modalities, containing more comprehensive and richer environmental features.

[0083] Finally, the fused feature matrix is ​​mapped to a low-dimensional dense vector through a fully connected network, resulting in a comprehensive semantic environment vector. The fully connected network is a common artificial neural network structure where each layer is fully connected to the next. The low-dimensional dense vector is a vector with lower dimensionality and denser element distribution after processing by the fully connected network. Compared to the high-dimensional fused feature matrix, the low-dimensional dense vector is easier to store, compute, and process, while retaining key information from the fused feature matrix. Although the fused feature matrix integrates multimodal information, it may have a high dimensionality and contain redundant information, which is not conducive to subsequent processing and storage. The fully connected network can perform nonlinear transformations and feature extraction on the fused feature matrix. Through the computation of multiple layers of neurons and the action of activation functions, the high-dimensional fused feature matrix is ​​mapped to a low-dimensional dense vector, ultimately obtaining a comprehensive semantic environment vector that can represent the environment perceived by the home service robot. This comprehensive semantic environment vector contains three core types of information: object semantic attributes (e.g., fragile toys for children), user status (e.g., elderly people walking slowly), and terrain physical features (e.g., carpet material), providing a unified scene representation basis for subsequent decision-making.

[0084] It should be understood that by integrating visual, speech, tactile, and force-sensory data and using an improved attention mechanism to achieve spatiotemporal alignment and dynamic weight allocation, a comprehensive semantic environment vector containing object semantic attributes, user state, and terrain physical features is generated, which solves the problem of missing semantic perception and physical interaction in existing technologies.

[0085] In one possible implementation, user state labels in the comprehensive semantic environment vector are identified based on preset multi-linked features, and a navigation adaptation strategy matching the user state labels is determined, including: Based on the preset multi-connected features, visual features, speech features, and behavioral features are extracted from the comprehensive semantic environment vector.

[0086] Visual features, voice features, and behavioral features are matched with preset population feature templates to obtain user status tags in the environment; user status tags include at least user identity, user behavioral features, and user location.

[0087] Based on user status tags and preset adaptation templates, determine the navigation adaptation strategy that matches the user status tags.

[0088] The comprehensive semantic environment vector contains rich information fused from multiple modalities, but this information is mixed together. To accurately identify user state labels, visual features, speech features, and behavioral features need to be extracted from the comprehensive semantic environment vector according to various feature extraction rules defined in the pre-defined multi-linked features. Visual features are those extracted from the comprehensive semantic environment vector that relate to visual information. These features reflect visual aspects such as the appearance of objects and the layout of the scene, including the color, shape, size, and positional relationships of objects, as well as lighting conditions and background features. Speech features are those extracted from the comprehensive semantic environment vector that relate to speech information, including acoustic features such as pitch, volume, speech rate, and timbre, as well as linguistic features such as vocabulary, grammar, and semantics. Speech features help the robot understand the user's language commands and emotional state. Behavioral features are those extracted from the comprehensive semantic environment vector that relate to user behavior. These features describe the user's action patterns, movement trajectories, and posture changes, such as walking speed, gait, and gestures. Behavioral features reflect the user's current activity and intentions.

[0089] For example, during feature extraction, visual features can be extracted using image processing algorithms to obtain features such as the object's color histogram and shape contour; speech features can be extracted using speech signal processing techniques to obtain acoustic features such as pitch and speech rate, and speech recognition can be used to obtain lexical semantic features; behavioral features can be extracted by analyzing the user's motion trajectory data to extract features such as action patterns and movement speed. Through this step, the comprehensive semantic environment vector is decomposed into more specific and targeted features, providing a foundation for subsequent matching operations.

[0090] After extracting visual, voice, and behavioral features, these features are compared and matched with corresponding features in a pre-defined user feature template. The template feature combination most similar to the extracted features is found by calculating the similarity between features or using other matching algorithms. Based on the matched template feature combination, the user's status label in the environment is determined, including the user's identity category (e.g., elderly, child), the user's current behavior (e.g., walking, sitting), and the user's specific location (e.g., near the sofa in the living room, next to the dining table in the kitchen). The extracted features are then transformed into specific user status information through a matching operation.

[0091] The preset user characteristic templates are a set of feature templates pre-established based on the characteristics and common behavioral patterns of different user groups. These templates cover various combinations of possible user identities (such as elderly, children, adults, etc.), user behavioral characteristics (such as standing, walking, sitting, etc.), and user locations (such as in a corner of a room, near furniture, etc.). User status tags are identifying information describing the user's current state in the environment, including at least three aspects: user identity, user behavioral characteristics, and user location. User identity clarifies which group the user belongs to; user behavioral characteristics reflect the user's ongoing activities or actions; and user location indicates the user's specific location in the environment. These tags help home service robots better understand user needs and intentions, thereby providing more accurate services.

[0092] After determining the user's status tags in the environment, the robot searches and matches these tags against a pre-defined adaptation template to find the corresponding navigation adaptation strategy. The pre-defined adaptation template is a set of rules that determine the navigation adaptation strategy based on the user's status tags. It defines the corresponding navigation strategies the home service robot should adopt under different combinations of user status tags. For example, if the user's status tag indicates an elderly user walking slowly, the pre-defined adaptation template might instruct the robot to reduce its navigation speed, maintain a safe distance from the user, and choose a flatter, wider path. If the user is in a narrow passageway in the room and performing some operation, the robot may need to adjust its navigation route to avoid disturbing the user. Through this step, user status information is transformed into specific navigation behavior guidance, enabling the home service robot to provide personalized navigation services based on the user's different statuses.

[0093] It should be noted that user behavior characteristics are mainly obtained through the collaborative acquisition and joint analysis of semantic visual units and speech semantic units, as follows: The RGB-D depth camera of the semantic visual unit is used as the main acquisition carrier to capture visual behavioral data such as the user's movement speed, movement amplitude, gait (step frequency, stride length), and body posture (e.g., the elderly bending over, the children running) in real time; the speech data collected by the speech semantic unit (e.g., the rapid speech of the elderly, the crying of the children) serves as an auxiliary supplement to the behavioral characteristics, used to determine the user's emergency state (e.g., the rapid movements and rapid speech of the elderly before falling); the visually captured behavioral data is combined with the speech auxiliary data, and the feature fusion module of the three-feature recognition model is used to extract core behavioral feature parameters such as movement speed threshold, movement amplitude level, and gait feature vector, which are then matched with preset group feature templates to finally complete the identification and classification of user behavior characteristics, providing a basis for population adaptation strategies.

[0094] For example, in elderly mode, the movement speed matches the elderly person's gait (0.2-0.3 m / s), prioritizes navigation along the handrail side, reserves a safe distance of more than 50 cm, actively prompts and stops when encountering steps or thresholds, and adjusts the navigation rhythm based on the elderly person's fatigue state; in children's mode, based on dual-temporal LSTM, it predicts the movement trajectory within 1.5 seconds, decelerates 1.5 m in advance (≤0.2 m / s), avoids areas where toys are scattered, prohibits entry into dangerous areas such as balconies and kitchens, and dynamically adjusts the path by tracking the child's position in real time; in elderly and children coexisting mode, it automatically identifies the relative positions of the elderly and children in the scene, plans the optimal path that "takes into account the elderly person's gait and stays away from the child's activity trajectory", and increases the safe distance to 0.8 m.

[0095] In this embodiment, user identity is identified through multiple features, including visual features (height, gait), voice features (voiceprint, speech rate), and behavioral features (movement speed, amplitude of movement), achieving a classification accuracy of 99%.

[0096] In one possible implementation, the method includes the following steps before controlling the home service robot to navigate using a personalized navigation path generated based on a comprehensive semantic environment vector and a navigation adaptation strategy: Based on the integrated semantic environment vector and the incremental simultaneous localization and mapping algorithm, a two-layer dynamic semantic map is constructed. The two-layer dynamic semantic map is used to represent the static fixed information and dynamic change information of the environment in which the home service robot is located.

[0097] Based on a two-layer dynamic semantic map and a navigation adaptation strategy, a personalized navigation path is generated through a hybrid algorithm of semantic reinforcement learning path planning.

[0098] Incremental Simultaneous Localization and Mapping (Incremental SLAM) is an algorithm for robots to simultaneously determine their own position and build a map of their environment. Unlike traditional SLAM algorithms, Incremental SLAM does not rebuild the entire map each time. Instead, it incrementally updates and expands the map based on previously built maps by continuously acquiring new environmental information (such as new sensor data). This approach can handle dynamically changing environments more efficiently, reduce computational load, and improve the real-time performance of map building.

[0099] The dual-layer dynamic semantic map is constructed using an incremental simultaneous localization and mapping (SMR) algorithm based on comprehensive semantic environment vectors. This dual-layer dynamic semantic map employs a two-layer structure to represent information about the environment in which the home service robot operates. One layer (static semantic layer) stores static, fixed information about the environment, such as fixed furniture, room functional areas, and areas frequently used by the elderly (e.g., sofas, armchairs), which remains relatively stable over a period of time. The other layer (dynamic semantic layer) stores dynamic information about the environment, such as temporary obstacles and the user's movement, reflecting changes in the environment in real time. This dual-layer structure allows the map to describe the environment more comprehensively and accurately, providing richer information for the robot's navigation.

[0100] The Semantic Reinforcement Learning Path Planning Hybrid Algorithm (DQN-Semantic-RRT) is a path planning algorithm that combines semantic information and reinforcement learning techniques. Semantic information provides a high-level understanding of the environment for path planning, such as the semantic categories of objects and the user's behavioral intentions. Reinforcement learning, on the other hand, allows the robot to learn the optimal path planning strategy by continuously trying and exploring the environment and learning from the rewards or penalties obtained. The Semantic Reinforcement Learning Path Planning Hybrid Algorithm can leverage semantic information to guide the reinforcement learning process, enabling the robot to learn personalized navigation path planning strategies adapted to different environments more quickly, thus improving the efficiency and accuracy of path planning.

[0101] Specifically, during operation, the home service robot acquires environmental data through multiple sensors and processes it to generate a comprehensive semantic environment vector. The incremental simultaneous localization and mapping (MS / MR) algorithm uses this comprehensive semantic environment vector as input. First, it uses sensor data to determine the robot's current position in the environment, achieving localization. Then, based on newly acquired environmental information, it incrementally updates the previously constructed map. During the update process, the algorithm divides environmental information into two parts: static fixed information and dynamically changing information. Static fixed information is stored in the static semantic layer of the map; dynamically changing information is updated in real-time to the dynamic semantic layer. In this way, a two-layer dynamic semantic map that accurately represents both static and dynamic environmental information is gradually constructed, providing a comprehensive environmental model for subsequent navigation path planning.

[0102] Then, a semantic reinforcement learning path planning hybrid algorithm combines semantic information from a two-layer dynamic semantic map with the requirements of navigation adaptation strategies, allowing the robot to perform simulated or actual navigation attempts in the environment. During navigation, the robot receives corresponding rewards or penalties based on different behaviors, continuously adjusting its path planning strategy through reinforcement learning. Simultaneously, it utilizes semantic information to achieve a higher-level understanding of the environment, such as identifying areas frequently visited by the user and prioritizing avoidance or adopting more cautious navigation methods. After multiple learning and trials, the algorithm can generate a personalized navigation path that conforms to both the actual environmental conditions and the user's needs, enabling the home service robot to complete navigation tasks more efficiently and safely.

[0103] In one possible implementation, a two-layer dynamic semantic map is constructed based on a comprehensive semantic environment vector and an incremental simultaneous localization and mapping (SMR) algorithm, including: Static semantic features are extracted from visual data using a semantic segmentation model.

[0104] The ground material data from static semantic features, inertial data, and tactile data are fused together to construct the static semantic layer data.

[0105] The spatial coordinates of dynamic target features in real-time visual data collected by the semantic visual unit are predicted by the dual-temporal long short-term memory trajectory prediction algorithm. The spatial coordinates of dynamic target features are then corrected by the incremental synchronous localization and map building algorithm to construct dynamic semantic layer data.

[0106] By binding the object semantic attributes and user status labels in the comprehensive semantic environment vector with the spatial coordinate values ​​of static semantic features in the static semantic layer data and the spatial coordinate values ​​of dynamic target features in the dynamic semantic layer data, a two-layer dynamic semantic map is obtained.

[0107] Semantic segmentation models are deep learning-based image processing models. They divide input visual data into multiple regions with different semantic meanings and assign corresponding semantic labels to each region, such as identifying different objects like walls, furniture, and people in an image, thereby extracting static semantic features from the visual data. Dual-temporal Long Short-Term Memory (LSTM) trajectory prediction algorithms are algorithms that combine two different time-series information. LSTM is a special type of recurrent neural network (RNN) that can handle long-term dependencies in sequential data. Dual-temporal LSTM trajectory prediction algorithms can use historical and current visual data information to predict the spatial coordinates of dynamic target features, thus estimating the future position of the dynamic target.

[0108] Specifically, after the semantic vision unit collects visual data from the environment, it inputs it into the semantic segmentation model. The semantic segmentation model uses pre-trained model parameters to analyze and classify each pixel in the visual data, extracting static semantic features such as fixed furniture and room functional areas. In this way, static semantic features are extracted from the visual data. These features describe the category and location information of fixed objects in the environment, providing a foundation for constructing static semantic layer data.

[0109] Next, different types of data—static semantic features extracted by the semantic segmentation model, inertial data collected by the inertial measurement unit, and ground material data collected by the tactile sensor—are fused. Through data association and filtering, noise and errors in the data are eliminated, and the position and attitude of static semantic features in the environment are determined more accurately, thus constructing static semantic layer data. Inertial data helps determine the robot's motion trajectory and attitude changes in the environment, while ground material data provides information on the characteristics of the environmental ground. The static semantic layer data contains semantic information and precise positions of fixed objects in the environment, providing a static environmental reference for the robot's navigation.

[0110] Then, the semantic vision unit continuously collects real-time visual data, including information on dynamic targets (such as moving users, pets, etc.). A dual-temporal long short-term memory trajectory prediction algorithm uses historical and current visual data to analyze the motion patterns of dynamic targets over time and predict the spatial coordinates of dynamic target features at future moments. However, due to potential errors in the prediction process, an incremental simultaneous localization and mapping (SLAM) algorithm is needed to correct the prediction results. The incremental SLAM algorithm combines the home service robot's own localization information with new sensor data (such as detection data of temporary obstacles like scattered toys and clothes drying) to update and optimize the position of dynamic targets in real time (e.g., updating the dynamic semantic layer every 100ms) to improve the accuracy of dynamic target position prediction. In this way, dynamic semantic layer data is constructed, reflecting the positional changes of dynamic targets in the environment, enabling the home service robot to respond promptly to dynamic changes in the environment.

[0111] Finally, the semantic attributes of objects (such as elderly walking aids and children's toys) and user status tags (such as elderly people staying or children playing) in the comprehensive semantic environment vector are bound to the spatial coordinate values ​​of static semantic features in the static semantic layer data and the spatial coordinate values ​​of dynamic target features in the dynamic semantic layer data. This means assigning richer semantic information to each object and dynamic target in the environment and associating this information with their specific locations within the environment, resulting in a two-layer dynamic semantic map. This achieves a three-dimensional map representation of "spatial location + semantic attributes + crowd association," providing support for subsequent path planning and crowd adaptation. For example, the semantic attribute of the object "sofa" is bound to the spatial coordinate values ​​of the sofa in the static semantic layer, while the user status tag "user currently walking" is bound to the spatial coordinate values ​​of the user in the dynamic semantic layer.

[0112] In one possible implementation, a personalized navigation path is generated based on a two-layer dynamic semantic map and a navigation adaptation strategy, using a semantic reinforcement learning path planning hybrid algorithm, including: Based on the semantic attributes of objects and user status tags in the two-layer dynamic semantic map, the risk classification of the environment is carried out to obtain the scene risk classification result.

[0113] According to the navigation adaptation strategy, initialize the navigation adaptation parameters of the home service robots respectively.

[0114] Based on the two-layer dynamic semantic map, scene risk classification results, and navigation adaptation parameters, an initial navigation path is generated through a semantic reinforcement learning path planning hybrid algorithm.

[0115] The initial navigation path is iteratively optimized based on a preset reward function to obtain a personalized navigation path.

[0116] In this embodiment, the semantic reinforcement learning path planning hybrid algorithm DQN-Semantic-RRT is used for path generation and optimization for home service robot navigation. The specific application scenario is that "after the robot receives navigation instructions (such as going to the living room to accompany the elderly), based on the current comprehensive semantic environment vector and the identified user status (elderly / child / coexisting), the algorithm generates a personalized optimized path that takes into account safety, efficiency, and user adaptability. At the same time, during the path execution process, the path is adjusted in real time according to physical interaction feedback to achieve closed-loop optimization." It is suitable for all home navigation scenarios (elderly care, child care, item delivery, etc.).

[0117] The scenario risk classification result is obtained by assessing the risk of the environment based on the semantic attributes of objects and user status labels in a two-layer dynamic semantic map. Different risk levels correspond to different potential danger levels; for example, toys near children are considered high-risk, and steps near the elderly are considered extremely high-risk. Navigation adaptation parameters are initialized according to the navigation adaptation strategy and are used to control various behaviors of the home service robot during navigation, such as navigation speed, acceleration, obstacle avoidance distance, and path selection weight. These parameters are adjusted according to user status and environmental conditions. The preset reward function is a function pre-set in reinforcement learning to evaluate the quality of the robot's navigation behavior; for example, semantic adaptation weight 20, dual-person safety weight 25, collision-free rate weight 10, and collision penalty weight -50.

[0118] Specifically, semantic attributes of objects and user status labels are obtained from a two-layer dynamic semantic map. Based on this information, risk classification standards are formulated, taking into account factors such as the danger level of objects, the activity status of users, and their positions in the environment. The entire environment is assessed and classified according to the risk classification standards (e.g., toys near children are high-risk, steps near the elderly are extremely high-risk, ordinary furniture is medium-risk, and safe areas for the elderly are low-risk). High-risk areas that need to be avoided in path planning are identified, and the scene risk classification results are obtained. This provides risk reference for subsequent path planning, enabling the robot to avoid high-risk areas and ensuring navigation safety.

[0119] Next, the navigation adaptation strategy was analyzed to clarify the required navigation behaviors of the robot under different user states and environmental conditions. For example, when the user is elderly or a child, the robot should reduce its navigation speed; when the user is busy operating items, the robot should maintain a larger obstacle avoidance distance. Based on these requirements, the robot's navigation adaptation parameters were initialized. These parameters will serve as the initial input to the semantic reinforcement learning path planning hybrid algorithm, affecting the generation of the robot's initial navigation path. For example, in elderly mode, the moving speed was set to 0.25 m / s, the safety distance to 0.5 m, and the navigation priority to 1 (highest); in child mode, the moving speed was set to 0.2 m / s, the safety distance to 0.8 m, and a dual-temporal LSTM trajectory prediction function was called to predict the child's sudden movement trajectory within 1.5 seconds; if both elderly and children were detected in the scene (coexistence mode), the global parameters were optimized, setting the global moving speed to 0.22 m / s and the global safety distance to 0.8 m to ensure the safety of both groups.

[0120] Then, the static and dynamic semantic information from the two-layer dynamic semantic map, the scene risk classification results, and the initialized navigation adaptation parameters are input into the semantic reinforcement learning path planning hybrid algorithm. The semantic information is combined to perform a high-level understanding of the environment, such as identifying which areas are passable and which areas have obstacles or high risks. At the same time, the requirements of navigation adaptation parameters are considered, such as adjusting the navigation speed and obstacle avoidance method according to the user's status, to generate an initial navigation path that initially meets the environmental constraints and user needs.

[0121] Subsequently, a preset reward function is invoked, and based on the initial navigation path, the robot continuously attempts navigation in the environment. After each attempt, a reward value is calculated according to the preset reward function. Based on the magnitude of the reward value, the robot's path planning strategy is adjusted, and the navigation path is iteratively optimized. After multiple iterations, the robot learns a navigation path that can obtain the highest reward value, namely a personalized navigation path that takes into account "semantic adaptation, crowd safety, and navigation efficiency." This path can meet user needs and environmental requirements to the greatest extent, achieving safe, efficient, and personalized navigation.

[0122] In one possible implementation, the method includes the following steps during the process of controlling a home service robot to navigate according to a personalized navigation path generated based on a comprehensive semantic environment vector and a navigation adaptation strategy: Determine whether the home service robot has experienced a collision during navigation.

[0123] If a collision occurs, the semantic attributes of the collided obstacle and the navigation adaptation deviation are recorded, and the obstacle avoidance strategy update function is called to adjust the risk classification threshold and navigation adaptation parameters to obtain the optimized navigation adaptation parameters.

[0124] If no collision occurs, the personalized navigation path will continue to be executed and the navigation adaptation parameters will be retained.

[0125] During the process of controlling the home service robot to navigate according to the personalized navigation path generated by the comprehensive semantic environment vector and navigation adaptation strategy, it is determined whether the home service robot has collided: if a collision occurs, the semantic attributes of the collision obstacle (such as soft toys, hard furniture) and navigation adaptation deviation (such as insufficient distance to the elderly, failure to anticipate the sudden movement of children) are immediately recorded, the obstacle avoidance strategy update function is called, the risk classification threshold and navigation adaptation parameters are adjusted, and the optimized navigation adaptation parameters are output to provide an optimization basis for the next path planning; if no collision occurs, the personalized navigation path and navigation adaptation parameters are directly output to guide the movement operation of the execution layer.

[0126] In one possible implementation, after the home service robot has completed navigation, the method further includes: Acquire the execution feedback data generated by the home service robot during navigation. The execution feedback data includes at least: collision information, path deviation information, and user feedback data.

[0127] By integrating execution feedback data, comprehensive semantic environment vectors, and personalized navigation paths, we obtain the full-process local navigation data for the home service robot.

[0128] Based on the anonymized local navigation process data, the local semantic model of the home service robot is trained and the model parameters of the local semantic model are optimized.

[0129] The optimized local semantic model parameters are encrypted and then uploaded to the cloud server. The cloud server then uses a federated averaging algorithm to aggregate the model parameters of multiple local semantic models to generate a globally optimized model.

[0130] Based on the global optimization model and user habit data distributed by the cloud server, the model parameters of the local semantic model are fine-tuned to obtain the optimized local semantic model.

[0131] Execution feedback data refers to the data generated by the home service robot during navigation, reflecting its navigation performance. This data may include collision information (such as whether a collision occurred, the location and frequency of collisions), path deviation (the degree of deviation between the actual path and the planned path), and user feedback data (user evaluations and suggestions regarding the navigation process). This data helps to understand the robot's actual navigation performance and user satisfaction.

[0132] Specifically, firstly, after completing the navigation task, the home service robot collects relevant information during the navigation process—namely, execution feedback data—through its onboard sensors (such as collision sensors and positioning sensors) and the user interaction interface. The collision information, path deviation information, and user feedback data recorded in the execution feedback data are then correlated and integrated with the environmental object semantic information, user state information, and personalized navigation path planning information contained in the comprehensive semantic environment vector. Through data fusion technology, these different types of data are unified into a single data framework, forming local navigation end-to-end data. This local navigation end-to-end data covers the entire process of robot navigation, from environmental perception and path planning to execution feedback, providing a comprehensive data foundation for the subsequent training of the local semantic model.

[0133] Next, the entire local navigation process data undergoes anonymization, removing or encrypting sensitive information to protect user privacy. The anonymized data is then input into a local semantic model, where machine learning algorithms (such as deep learning) allow the model to learn patterns and rules within the data. For example, the model can learn how to avoid collisions, reduce path deviations, and improve user satisfaction under different environmental semantics and navigation conditions. Based on the learning results, the model parameters of the local semantic model are adjusted to optimize its performance, enabling it to more accurately understand and process environmental semantic information, thereby generating more reasonable navigation decisions. The local semantic model is a model that runs locally on the home service robot to process and understand environmental semantic information. It can learn and optimize based on input environmental data (such as a comprehensive semantic environment vector) and navigation-related data (such as execution feedback data and personalized navigation paths) to improve the robot's environmental perception and navigation decision-making capabilities.

[0134] Then, the optimized local semantic model parameters are encrypted using an encryption algorithm (such as symmetric or asymmetric encryption) to convert the parameters into ciphertext, preventing them from being stolen or tampered with during transmission. The encrypted model parameters are then uploaded to a cloud server. The cloud server collects local semantic model parameters from multiple home service robots and aggregates these parameters using a federated averaging algorithm. The federated averaging algorithm generates a global optimized model parameter by calculating the average or weighted average of the individual local model parameters. This parameter incorporates the navigation experience and learning outcomes of multiple home service robots. The cloud server is a remote server cluster with powerful computing and storage capabilities, used to receive the local semantic model parameters uploaded by multiple home service robots and aggregate them using the federated averaging algorithm to generate the global optimized model. The federated averaging algorithm is a distributed machine learning algorithm used to train models on multiple devices (such as home service robots) and aggregate the model parameters trained on each device to generate a global optimized model. This algorithm can fully utilize data from multiple devices for model optimization while protecting user data privacy.

[0135] Subsequently, the cloud server distributes the generated global optimization model parameters to each home service robot. Upon receiving these parameters, each robot fine-tunes its local semantic model parameters by incorporating locally stored user habit data. This user habit data reflects users' personalized needs and preferences (such as frequently used navigation routes, frequency of visits to different areas, and required robot navigation speed). By combining this data with the global optimization model parameters, the local semantic model can better adapt to the specific user's habits while maintaining global optimization performance.

[0136] Finally, after fine-tuning, an optimized local semantic model is obtained. This optimized local semantic model can provide more accurate and personalized navigation services for home service robots, thereby improving the practicality of home service robots and user satisfaction.

[0137] It should be noted that during the training of the local semantic model, two optimization items, "population adaptation satisfaction" and "physical interaction smoothness", were set to ensure that the global model generated by federated learning and subsequent local fine-tuning always evolve in the two core directions of improving the safety service experience for the elderly / children and enhancing the environmental adaptability based on physical interaction. This is fundamentally different from the approach of general robot navigation models that only optimize path length or obstacle avoidance success rate.

[0138] Furthermore, this application embodiment also provides a family-specific fine-tuning strategy, specifically as follows: A family-specific sample library is established, covering the specific apartment layout, furniture placement, frequently used areas by the elderly / children, and behavioral habit data (such as the daily activity trajectory of the elderly, the preferred play areas of children, etc.). The sample library supports real-time incremental updates without manual intervention. An incremental learning algorithm is used to update only the model parameters related to the family scenario (such as family-specific semantic labels, population adaptation parameters, etc.), without performing full network retraining. Experimental verification shows that the fine-tuning time is reduced by 75% compared to full network retraining, significantly improving model iteration efficiency. Simultaneously, three families with different apartment layouts were selected for scenario fine-tuning tests. Experimental data shows that after fine-tuning, the navigation accuracy of the family service robot can be improved by 35%, and the task completion efficiency can be improved by 52%, better meeting the personalized navigation and service needs of families.

[0139] Please see Figure 3 , Figure 3 This is a schematic diagram of the navigation timing process for a home service robot provided in one embodiment of this application. Figure 3 In this system, the navigation process of the home service robot includes a perception phase, a decision-making phase, an execution phase, and an optimization phase. Specifically: In the perception phase, four modal sensors simultaneously collect visual data, voice data, tactile data, and inertial data, which are then fused using an improved attention mechanism to generate a comprehensive semantic environment vector. In the decision-making phase, a two-layer dynamic semantic map is constructed, user status is identified, and a navigation adaptation strategy is matched. A personalized navigation path is generated using a DQN-Semantic-RRT hybrid algorithm. In the execution phase, the adaptive drive module moves according to the personalized navigation path, adjusting wheel speed and torque based on tactile feedback, while the safety interaction module monitors and provides feedback in real time. In the optimization phase, the entire navigation process data is recorded locally and a local semantic model is trained. Global optimization parameters are aggregated through federated learning, fine-tuned, and the local semantic model is updated to complete a closed-loop iteration.

[0140] It is understood that the home service robot navigation method provided in this application has the following advantages: (1) Significantly improved semantic recognition accuracy. Due to the adoption of an improved attention mechanism of modal semantic perception layer, which integrates multi-dimensional data such as vision, speech, and tactile sensation, and is equipped with a family-specific semantic tag library, the semantic recognition accuracy of the home environment can be greater than 95%, and the recognition accuracy of special items related to the elderly / children can be greater than 98%, which is more than 46% higher than the existing technology (<65%), thus solving the core problem of insufficient service targeting of the existing technology.

[0141] (2) Industry-leading adaptability to dual special populations. Thanks to the synergistic effect of the dual special population collaborative adaptation decision-making method and the dual temporal LSTM prediction in the modal semantic perception layer, the success rate of elderly caregivers can reach 99%, the response delay for children's sudden movement obstacle avoidance is <100ms (73% improvement compared to the existing technology of 380ms), the obstacle avoidance success rate reaches 99%, and the collision rate is reduced to below 0.5%, filling the industry gap in dual special population collaborative adaptation.

[0142] (3) The robustness of dynamic scenes is greatly enhanced. Based on the collaborative optimization of the dual-layer dynamic semantic map and the incremental semantic SLAM algorithm, the map update error can be less than 3cm (70% improvement compared to the existing technology of 10cm), and the navigation success rate for furniture movement and temporary obstacles is greater than 97%. At the same time, relying on the infrared supplementary lighting function of the semantic vision unit and the physical feedback of tactile perception, the semantic recognition accuracy in low light / sudden change of lighting scenes can be greater than 90%. Due to the closed-loop linkage of the tactile perception unit and the adaptive drive module, the slip rate of the home service robot on different floor materials (carpet, tile) is less than 2%, which is an order of magnitude improvement compared to the solution without physical feedback (slip rate > 15%), and can be adapted to dynamic scenes of the whole family.

[0143] (4) Balancing privacy protection and continuous optimization. The unique combination of federated learning optimization layer and embodied intelligence closed loop enables collaborative optimization of multi-family data, improving navigation efficiency by more than 15% iteratively, while avoiding the uploading of raw data, meeting the needs of family privacy; embodied closed loop feedback enables navigation strategy to be continuously optimized over time, improving user satisfaction by 80%.

[0144] (5) Wide applicability. It supports more than 90% of family apartment types, including small apartments, large single-level apartments, and duplex ground floors, and is suitable for multiple scenarios such as elderly care, child care, and item delivery.

[0145] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0146] It should be noted that the information interaction and execution process between the modules in the above-mentioned home service robot navigation system are based on the same concept as the method embodiment of this application. For details on their specific functions and technical effects, please refer to the method embodiment section, which will not be repeated here.

[0147] This application also provides a terminal device, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. (Refer to...) Figure 4 The terminal device 4 in this embodiment includes a memory 41, a processor 42, and a computer program stored in the memory 41 and executable on the processor 42. When the processor 42 executes the computer program, it implements the steps in the home service robot navigation method embodiment described above.

[0148] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0149] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the various method embodiments.

[0150] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0151] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0152] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0153] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A navigation method for a home service robot, characterized in that, include: Feature extraction is performed on the multimodal environmental data perceived by the home service robot to obtain multiple modal feature vectors; wherein, the multimodal environmental data includes visual data, voice data, tactile data, and inertial data; An improved attention mechanism is used to perform spatiotemporal alignment and weight fusion processing on the multiple modal feature vectors to generate a comprehensive semantic environment vector. The comprehensive semantic environment vector is used to represent the environmental information perceived by the home service robot. The improved attention mechanism is used to realize the timestamp alignment and spatial coordinate alignment of multiple modal feature vectors, and to dynamically allocate the fusion weights of multiple modal feature vectors according to the environmental scene. The user state labels in the comprehensive semantic environment vector are identified based on preset multi-link features, and a navigation adaptation strategy matching the user state labels is determined; wherein, the preset multi-link features are pre-set features used to characterize typical user states and behavior patterns; the navigation adaptation strategy is used to adjust the navigation adaptation parameters of the home service robot; The home service robot is controlled to navigate according to the personalized navigation path generated by the comprehensive semantic environment vector and the navigation adaptation strategy.

2. The home service robot navigation method as described in claim 1, characterized in that, Before extracting features from the multimodal environmental data perceived by the home service robot to obtain multiple modal feature vectors, the method includes: The visual data, which includes scene images and user images, is acquired through the color depth camera in the semantic vision unit. The speech data, which includes ambient speech and user commands, is acquired through the microphone array in the speech semantic unit. The tactile force sensing unit collects tactile force data containing obstacle contact information and ground material information; The inertial data of the home service robot is acquired through the inertial measurement unit; wherein, the inertial data is used to correct the spatiotemporal alignment deviation between the visual target pose and the multimodal environment data.

3. The home service robot navigation method as described in claim 2, characterized in that, The improved attention mechanism performs spatiotemporal alignment and weight fusion processing on the multiple modal feature vectors to generate a comprehensive semantic environment vector, including: The multiple modal feature vectors are preprocessed to obtain preprocessed multiple modal feature vectors; Based on the spatiotemporal calibration module in the improved attention mechanism, the preprocessed multiple modal feature vectors are spatiotemporally aligned to obtain aligned visual feature vectors, aligned speech feature vectors, aligned tactile feature vectors, and aligned inertial feature vectors. Based on adaptive weight values, the aligned visual feature vector, the aligned speech feature vector, the aligned tactile feature vector, and the aligned inertial feature vector are subjected to weight fusion processing to obtain a fused feature matrix; wherein, the adaptive weight values ​​are determined according to the confidence values ​​and scene modes corresponding to the multiple modal feature vectors; The fused feature matrix is ​​mapped into a low-dimensional dense vector through a fully connected network to obtain the comprehensive semantic environment vector.

4. The home service robot navigation method as described in claim 1, characterized in that, The step of identifying user state labels in the comprehensive semantic environment vector based on preset multi-link features and determining a navigation adaptation strategy matching the user state labels includes: Based on the preset multi-linked features, visual features, speech features, and behavioral features are extracted from the comprehensive semantic environment vector; The visual features, voice features, and behavioral features are matched with a preset crowd feature template to obtain user status tags in the environment; the user status tags include at least user identity, user behavioral features, and user location; Based on the user status tags and the preset adaptation template, determine the navigation adaptation strategy that matches the user status tags.

5. The home service robot navigation method as described in claim 1, characterized in that, Before the home service robot navigates according to the personalized navigation path generated by the integrated semantic environment vector and the navigation adaptation strategy, the method includes: Based on the comprehensive semantic environment vector and the incremental synchronous localization and mapping algorithm, a two-layer dynamic semantic map is constructed; wherein, the two-layer dynamic semantic map is used to represent the static fixed information and dynamic change information of the environment in which the home service robot is located; Based on the two-layer dynamic semantic map and the navigation adaptation strategy, the personalized navigation path is generated through a semantic reinforcement learning path planning hybrid algorithm.

6. The home service robot navigation method as described in claim 5, characterized in that, The process of constructing a two-layer dynamic semantic map based on the comprehensive semantic environment vector and the incremental simultaneous localization and mapping (SMR) algorithm includes: Static semantic features are extracted from the visual data using a semantic segmentation model. The static semantic features, the inertial data, and the ground material data in the tactile data are fused together to construct the static semantic layer data; The spatial coordinates of dynamic target features in real-time visual data collected by the semantic visual unit are predicted by the dual-time long short-term memory trajectory prediction algorithm, and the spatial coordinates of the dynamic target features are corrected by the incremental synchronous positioning and map building algorithm to construct dynamic semantic layer data. The object semantic attributes in the comprehensive semantic environment vector, the user status labels, and the spatial coordinate values ​​of static semantic features in the static semantic layer data, as well as the spatial coordinate values ​​of dynamic target features in the dynamic semantic layer data, are bound together to obtain the two-layer dynamic semantic map.

7. The home service robot navigation method as described in claim 6, characterized in that, The step of generating the personalized navigation path based on the two-layer dynamic semantic map and the navigation adaptation strategy, using a semantic reinforcement learning path planning hybrid algorithm, includes: Based on the semantic attributes of objects and the user status tags in the dual-layer dynamic semantic map, the environmental location is risk-classified to obtain the scene risk classification result. According to the navigation adaptation strategy, initialize the navigation adaptation parameters of the home service robot respectively; Based on the dual-layer dynamic semantic map, the scene risk classification results, and the navigation adaptation parameters, an initial navigation path is generated using the semantic reinforcement learning path planning hybrid algorithm. The initial navigation path is iteratively optimized based on a preset reward function to obtain the personalized navigation path.

8. The home service robot navigation method as described in claim 1, characterized in that, In the process of controlling the home service robot to navigate according to the personalized navigation path generated by the comprehensive semantic environment vector and the navigation adaptation strategy, the method includes: Determine whether the home service robot has experienced a collision during navigation; If a collision occurs, the semantic attributes of the collided obstacle and the navigation adaptation deviation are recorded, and the obstacle avoidance strategy update function is called to adjust the risk classification threshold and navigation adaptation parameters to obtain the optimized navigation adaptation parameters. If no collision occurs, the personalized navigation path will continue to be executed and the navigation adaptation parameters will be retained.

9. The home service robot navigation method as described in claim 1, characterized in that, After the home service robot completes navigation, the method further includes: The execution feedback data generated by the home service robot during navigation is obtained, wherein the execution feedback data includes at least: collision information, path deviation information, and user feedback data; The execution feedback data, the comprehensive semantic environment vector, and the personalized navigation path are integrated to obtain the local navigation full-process data of the home service robot; Based on the anonymized local navigation process data, the local semantic model of the home service robot is trained, and the model parameters of the local semantic model are optimized. The optimized local semantic model parameters are encrypted and then uploaded to the cloud server, so that the cloud server can aggregate the model parameters of multiple local semantic models through a federated averaging algorithm to generate a globally optimized model. Based on the global optimization model and user habit data issued by the cloud server, the model parameters of the local semantic model are fine-tuned to obtain the optimized local semantic model.

10. A navigation system for a home service robot, characterized in that, include: Modal semantic perception layer, embodied intelligent decision-making layer, and personalized execution layer; among them, The modal semantic perception layer, connected to the embodied intelligent decision-making layer, is used to extract features from the multimodal environmental data perceived by the home service robot to obtain multiple modal feature vectors. Furthermore, through an improved attention mechanism, the multiple modal feature vectors are spatiotemporally aligned and weighted to generate a comprehensive semantic environment vector. The multimodal environmental data includes visual data, speech data, tactile data, and inertial data. The comprehensive semantic environment vector represents the environmental information perceived by the home service robot. The improved attention mechanism aligns the timestamps and spatial coordinates of the multiple modal feature vectors and dynamically allocates fusion weights based on the environmental scene. The embodied intelligent decision-making layer, connected to the personalized execution layer, is used to identify the user state in the comprehensive semantic environment vector based on preset multi-link features, and determine a navigation adaptation strategy matching the user state. The preset multi-link features are pre-defined features used to characterize typical user states and behavioral patterns. The navigation adaptation strategy is used to adjust the navigation adaptation parameters of the home service robot during navigation. Furthermore, it generates a personalized navigation path based on the comprehensive semantic environment vector and the navigation adaptation strategy. The personalized execution layer is used to control the home service robot to navigate according to the personalized navigation path and generate execution feedback data.

11. The home service robot navigation system as described in claim 10, characterized in that, The system includes: a federated learning optimization layer; The federated learning optimization layer establishes communication connections with the embodied intelligent decision-making layer and the personalized execution layer, respectively. It is used to train the local semantic model of the home service robot based on the received personalized navigation path, execution feedback data, and comprehensive semantic environment vector, optimizing the model parameters of the local semantic model; and to encrypt and upload the optimized local semantic model parameters to a cloud server, enabling the cloud server to perform federated aggregation to obtain a globally optimized model; and to fine-tune the model parameters of the local semantic model based on the globally optimized model and user habit data, obtaining an optimized local semantic model.

Images