Multi-robot voice command response method and system

Abstract

This application discloses a multi-robot voice command response method and system. The method includes: a first robot responding to a user's voice command by acquiring a first environmental image and determining the location of a first user; creating a first broadcast data packet based on the first user's location, the first robot's local state information, and the first environmental image; receiving second broadcast data packets from other robots to obtain multiple broadcast data packets; obtaining a target robot number based on the multiple broadcast data packets, the user's voice command, the historical dialogue record between the first robot and the first user in the current dialogue cycle, and an attribution robot determination model; and continuing to respond to the service request indicated by the user's voice command when a matching number is detected. This application can effectively avoid multi-robot competitive response conflicts and improve the accuracy and intelligence of multi-robot voice command response.

Description

Technical Field

[0001] This application relates to the field of general control or regulation system technology, or the field of speech processing technology, or the field of computer system technology based on a specific computing model, and in particular to a method and system for responding to voice commands of multiple robots. Background Technology

[0002] Currently, embodied intelligent robots all adopt a hardware design of one machine per board. The edge model deployed on the motherboard needs to be trained in the cloud before being deployed locally, resulting in poor adaptability between model inference and hardware computing power. Traditional voice interaction solutions for embodied intelligent robots generally adopt a working mode of independent perception, local recognition, and autonomous response. Each robot collects environmental voice signals through its own sound pickup device, completes voice recognition and intent parsing locally or in the cloud, and then autonomously executes response actions. This mode is designed based on the independent working scenario of a single robot and is no longer suitable for complex service environments where multiple robots coexist and multiple users interact concurrently.

[0003] In real-world service scenarios involving multiple individuals and robots, there is a problem of competitive response among multiple robots. A single user's spoken voice command can be simultaneously collected by multiple robots within the effective pickup range. Each robot independently determines the validity of the command and then initiates the execution action. This can easily lead to execution conflicts where multiple robots respond to the same command simultaneously. This cannot meet the collaborative service needs of multiple users and multiple tasks in parallel in multi-robot service scenarios. It can also result in a waste of robot computing resources, low service efficiency, and even safety hazards such as collisions and interference caused by multiple robots operating simultaneously. It is difficult to adapt to real-world application scenarios such as elderly care, which have high requirements for service accuracy and collaboration. Summary of the Invention

[0004] In view of this, this application provides a multi-robot voice command response method and system, applied to the training and promotion system-on-a-chip motherboard of a robot in a multi-robot service system. By responding to user voice commands, collecting environmental images, determining the user's location, and combining the robot's local state information to generate robot broadcast data packets and multi-robot interaction integrated data packet information, and combining historical dialogue records with a pre-trained attribution robot determination model, the unique command response target robot is accurately determined. This achieves the unique determination of the command response subject in multi-robot coexistence scenarios, effectively avoiding the conflict problem of multi-robot competitive response. Moreover, relying on the training and promotion integrated hardware architecture, the robot end can complete model inference and task processing, adapting to the complex service environment of multi-robot and multi-user concurrent interaction.

[0005] In a first aspect, embodiments of this application provide a multi-robot voice command response method, applied to a system-on-a-chip motherboard with integrated training and push capabilities for the first robot in a multi-robot service system; the method includes:

[0006] In response to a user's voice command, the robot acquires a first environmental image and determines the location of the first user; and creates a first broadcast data packet for the first robot based on the first user's location, the robot's local status information, and the first environmental image; and receives second broadcast data packets from other robots to obtain multiple broadcast data packets including the first broadcast data packet.

[0007] Based on the multiple broadcast data packets, the user's voice command, the historical dialogue records between the first robot and the first user in the current dialogue cycle, and the pre-trained attribution robot determination model, the target robot number responding to the user's voice command is obtained.

[0008] When the target robot number is detected to be consistent with the device number of the first robot, the system continues to respond to the service requests indicated by the user's voice command.

[0009] Secondly, this application also provides a multi-robot service system, the system including multiple robots, the multiple robots including a first robot, each robot including a system-on-a-chip motherboard with integrated training and propulsion capabilities, the system-on-a-chip motherboard including a main controller, a main computing card and an expansion computing card communicatively connected to the main controller, and a memory communicatively connected to the main controller, wherein the system-on-a-chip motherboard of the first robot with integrated training and propulsion capabilities is used to execute the steps in the first aspect of the embodiments of this application.

[0010] Thirdly, embodiments of this application provide an electronic device, including a processing module, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processing module, and the programs include instructions for performing the steps in the first aspect of embodiments of this application.

[0011] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in the first aspect of embodiments of this application.

[0012] Fifthly, embodiments of this application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of embodiments of this application. The computer program product may be a software installation package.

[0013] As can be seen, through the multi-robot voice command response method and system provided in this application, the system-on-a-chip motherboard of the first robot, which has integrated training and push capabilities, responds to the user's voice command, acquires the first environmental image, and determines the location of the first user; and creates the first broadcast data packet of the first robot based on the location of the first user, the local state information of the first robot, and the first environmental image; and receives the second broadcast data packets of other robots, obtaining multiple broadcast data packets including the first broadcast data packet; and obtains the target robot number responding to the user's voice command based on the multiple broadcast data packets, the user's voice command, the historical dialogue record between the first robot and the first user in the current dialogue cycle, and the pre-trained attribution robot determination model; when the target robot number is detected to be consistent with the device number of the first robot, it continues to respond to the service request indicated by the user's voice command. Thus, compared with the existing single-robot voice interaction scheme with independent perception, local recognition, and autonomous response, this application achieves the unique determination of the responding subject in a multi-robot scenario through the interaction and fusion of multi-robot broadcast data packets, avoiding multi-robot competitive response conflicts. At the same time, relying on the integrated training and push hardware architecture, the edge side can independently complete model inference and task processing, adapting to the complex service environment of concurrent interaction of multiple robots and multiple users. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of the architecture of a multi-robot service system provided in an embodiment of this application;

[0016] Figure 2 This is a flowchart illustrating the steps of a multi-robot voice command response method provided in an embodiment of this application;

[0017] Figure 3 This is a schematic diagram of a process for determining the target robot number provided in an embodiment of this application;

[0018] Figure 4 This is a schematic diagram of a scenario where the user space is empty, provided in an embodiment of this application;

[0019] Figure 5 This is a schematic diagram of a scenario where the user space is not empty, provided in an embodiment of this application;

[0020] Figure 6This is a schematic diagram of another scenario where the user space occupancy state is not empty, provided in an embodiment of this application;

[0021] Figure 7 This is a schematic diagram of an attribution robot determination model provided in an embodiment of this application;

[0022] Figure 8 This is a schematic diagram of a process for performing intent recognition on user voice commands according to an embodiment of this application;

[0023] Figure 9 This is a schematic diagram illustrating a scenario where a robot responds to a user's voice command, as provided in an embodiment of this application.

[0024] Figure 10 This is a block diagram of the functional units of a multi-robot service system provided in an embodiment of this application;

[0025] Figure 11 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0027] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0028] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0029] In this application's embodiments, "multiple" refers to two or more. In this application's embodiments, "connection" refers to various connection methods, such as direct or indirect connections, to achieve communication between devices; this application's embodiments do not impose any limitations on this.

[0030] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0031] To address the aforementioned issues, embodiments of this application provide a multi-robot voice command response method and system.

[0032] First, combined Figure 1 The multi-robot service system in the embodiments of this application will be described. Figure 1 This is a schematic diagram of the architecture of a multi-robot service system provided in an embodiment of this application, such as... Figure 1 As shown, the system includes multiple robots 110. Each robot 110 includes a system-on-a-chip motherboard 111 with integrated training and propulsion capabilities. The system-on-a-chip motherboard 111 includes a main controller 112, a main computing card 113 and an expansion computing card 114 that are communicatively connected to the main controller 112, and a memory 115 that is communicatively connected to the main controller 112. The multiple robots 110 communicate with each other through a regional ad hoc network. The multiple robots 110 serve multiple users 120, and are used to determine the individual robot 110 to respond to the user voice commands of the individual user 120.

[0033] Among them, the system-on-a-chip motherboard 111, which has integrated training and inference capabilities, is the core hardware carrier for the robot 110 to realize voice command response, multi-machine collaborative interaction and intelligent task execution. It integrates the main controller 112, main computing power card 113, expansion computing power card 114 and memory 115, and realizes end-side collaboration of model training and inference through integrated architecture, breaking through the limitations of traditional motherboard computing power fixation and model dependence on the cloud.

[0034] Specifically, the main controller 112, as the core scheduling unit of the motherboard, is responsible for coordinating and managing the entire motherboard's workflow. It receives user voice commands, environmental images, and other input data; coordinates task allocation between the main computing card 113 and the extended computing card 114; controls data read / write operations in the memory 115; and interacts with other robots 110 via a self-organizing network to broadcast data packets, ensuring the stable operation of the collaborative logic of multiple robots 110. The main computing card 113 communicates with the main controller 112 and undertakes computationally intensive tasks, primarily responsible for high-computational-consumption operations such as multi-machine vision fusion, sound source localization, spatial position calculation, and inference of the attribution robot determination model. The instruction processing provides core computing capabilities; the extended computing card 114 is also connected to the main controller 112 and is responsible for parallel and lightweight task processing, such as semantic recognition, voiceprint feature extraction, and historical dialogue data retrieval, forming a division of computing power with the main computing card 113 to improve system processing efficiency and real-time performance; the memory 115 is connected to the main controller 112 and is used to store key data such as user voiceprint feature sets and historical dialogue state datasets; neither the main computing card 113 nor the extended computing card 114 directly accesses the memory 115, and all data read and write requests are uniformly forwarded and scheduled by the main controller 112 to ensure data access consistency and system stability.

[0035] The system comprises multiple robots (110 in total) that form a self-organizing regional network without a central node, utilizing WiFi, Bluetooth Mesh, or UWB. Within this network, low-latency, highly reliable packet broadcasting and point-to-point transmission are supported. Each robot (110 in total) acts as both a communication terminal and a relay node, autonomously constructing network topology and forwarding data. It supports dynamic network entry, disconnection, and multi-hop relaying, and automatically reselects paths in the event of network interruption. The protocol employs dynamic routing, combined with anti-collision mechanisms and time slot allocation, to ensure concurrent communication among multiple nodes. At the transmission layer, modulation and multiple antennas enhance anti-interference capabilities, supplemented by encryption, ensuring stable, secure, and real-time communication between clusters.

[0036] As can be seen, in this embodiment, by configuring each robot 110 with a system-on-a-chip motherboard 111 that integrates training and inference, and relying on the hardware architecture of the main controller 112 for unified scheduling of dual computing cards and centralized storage of memory 115, efficient collaboration between edge model training and inference is achieved, breaking through the technical limitations of traditional motherboard computing power fixation and reliance on the cloud; at the same time, combined with the regional self-organizing network, secure and real-time interaction of broadcast data packets between multiple robots 110 is realized, providing a stable hardware and communication foundation for multiple robots 110 to collaboratively determine the subject of instruction response and efficiently execute voice command tasks, which can accurately adapt to the robot collaborative service needs in multi-user scenarios.

[0037] The following is combined Figure 2 The multi-robot voice command response method provided in the embodiments of this application will be further described below. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the steps of a multi-robot voice command response method provided in an embodiment of this application, applied to a system-on-a-chip motherboard of a first robot with integrated training and propagation capabilities, such as... Figure 2 As shown, the method includes the following steps:

[0038] Step S210: In response to a user's voice command, acquire a first environmental image and determine the location of the first user; and create a first broadcast data packet for the first robot based on the first user's location, the local status information of the first robot, and the first environmental image; and receive second broadcast data packets from other robots to obtain multiple broadcast data packets including the first broadcast data packet.

[0039] Among them, user voice commands are interactive requests initiated by users in the form of voice, which are collected in real time by the first robot through its own onboard sound pickup device (such as a microphone array) in the service scenario. The collected voice commands are transmitted in real time to the main controller of the system-on-a-chip motherboard, where the main controller coordinates and allocates computing resources for subsequent processing to ensure the real-time processing of commands.

[0040] The first environmental image is visual data captured in real time by the first robot using its own visual acquisition devices (such as cameras and visual sensors) in the same spatiotemporal scene as the user's voice commands. It includes single or multiple consecutive frames of scene images, clearly presenting visual information such as the environmental layout, user position, and object distribution within the robot's field of view. After acquisition, the first environmental image is synchronously transmitted to the system-on-a-chip motherboard, where the main controller schedules the main computing card for image processing according to task requirements, ensuring the synchronization and correlation between visual and voice data.

[0041] Specifically, to avoid false responses, after collecting voice commands, the first robot will first filter and judge through multiple dimensions such as voice volume threshold, voice duration, and voiceprint matching degree. Only valid voice commands that meet the wake-up conditions will be executed for subsequent environmental image acquisition and location determination operations. Non-target voices such as user talking to themselves or normal conversations with others will be automatically ignored to ensure the accuracy of command response and efficient use of system resources.

[0042] Furthermore, sound source localization is performed on the user's voice command to obtain the first user's position. In a specific embodiment, sound source localization can be achieved using a microphone array delay estimation method. Taking a four-element microphone array mounted on the first robot as an example, when the user issues a voice command, different microphones will receive the same voice signal with a slight time difference due to differences in spatial distance from the user. The main computing card of the motherboard will perform time-domain and frequency-domain analysis on the audio signals collected by each microphone. By calculating the time delay difference of the signal reaching each microphone, and combining the geometric arrangement parameters of the microphone array, the direction of arrival (DOA) estimation algorithm is used to calculate the incident direction of the voice signal. Then, combined with the real-time spatial positioning coordinates of the first robot (such as its own three-dimensional coordinates obtained based on SLAM or LiDAR), the absolute spatial position of the user in the global coordinate system is further calculated, i.e., the first user's position. If it is a small indoor scene, the relative polar coordinate position with the first robot as the origin can also be output (such as 5 meters away from the robot and an azimuth angle of 30°). This position data will be accurately quantified into digital coordinate information.

[0043] Understandably, while the first robot creates the first broadcast data packet, other robots in the current environment simultaneously collect the user's voice command and complete sound source localization to obtain the user's location. Combining their own status information and the environmental images they have collected, they generate and broadcast their own second broadcast data packets. Then, the first robot receives the second broadcast data packets sent by all other robots in real time through the regional ad hoc network, integrates and summarizes them with its own first broadcast data packet, so that the first robot can obtain multiple broadcast data packets covering the multi-dimensional perception and status data of all participating collaborative robots in the current environment.

[0044] Step S220: Based on the multiple broadcast data packets, the user voice command, the historical dialogue records between the first robot and the first user in the current dialogue cycle, and the pre-trained attribution robot determination model, obtain the target robot number that responds to the user voice command.

[0045] The historical dialogue record for the current dialogue cycle is a collection of all interaction data generated between the first robot and the first user in the same dialogue scenario before this voice command interaction. This record covers key information such as the content of the voice dialogue between the two parties, command interaction nodes, and robot response feedback. It is stored and updated in real time by the first robot's memory and is limited to valid interaction data within the current dialogue cycle.

[0046] Among them, the pre-trained attribution robot determination model is an intelligent decision-making model and the core algorithm carrier for determining the subject of instruction response. After being pre-trained with a large amount of sample data from multi-robot interaction scenarios, this model is deployed in the robot's training and propagation integrated system-on-a-chip motherboard and can complete efficient edge-side inference by relying on the main computing card.

[0047] Step S230: When it is detected that the target robot number is consistent with the device number of the first robot, continue to respond to the service request indicated by the user's voice command.

[0048] Specifically, when the target robot number matches the device number of the first robot, it indicates that the first robot has been selected as the sole responder to the user's voice command. At this time, the first robot will rely on the computing power and data resources of its own training and push integrated motherboard to officially start the intent recognition and task sequence generation process for the voice command in order to meet the service requirements indicated by the user's voice command. If the two do not match, the first robot will stop the subsequent processing of the command and remain in a state of waiting for response.

[0049] As can be seen, in this embodiment, by real-time interaction of broadcast data packets between multiple robots based on ad hoc networks, combined with voice filtering, sound source localization, historical dialogue records and pre-trained robot attribution determination models, the target robot responding to user voice commands is accurately determined, achieving efficient and accurate command response and reasonable resource allocation under multi-robot collaboration. At the same time, relying on the integrated training and induction system-on-a-chip to ensure edge-side inference efficiency and decision reliability, the intelligence and collaboration of command response in multi-robot service scenarios are significantly improved.

[0050] Please refer to details. Figure 3 , Figure 3 This is a flowchart illustrating a process for determining a target robot number, as provided in an embodiment of this application. Figure 3 As shown, obtaining the target robot number responding to the user's voice command based on the multiple broadcast data packets, the user's voice command, the historical dialogue records between the first robot and the first user in the current dialogue cycle, and the pre-trained attribution robot determination model includes the following steps:

[0051] S301, the second broadcast data packet is parsed to obtain the second environmental image collected by the other robot in response to the user's voice command and the local status information of the other robot.

[0052] The robot's local status information includes the timestamp of the current processing time window, spatial location, device number, and computing resource utilization rate. Additionally, it may include status information such as remaining battery power, network connection status, hardware operating temperature, remaining storage space, motion mode, and task execution progress, comprehensively reflecting the robot's real-time operating parameters and resource status.

[0053] S302, perform fusion processing on the first environmental image and the plurality of second environmental images to obtain a multi-machine vision fusion image.

[0054] Among them, multi-robot vision fusion images integrate environmental visual information captured by each robot from different perspectives and positions, making up for the problems of perspective occlusion, limited field of view, and distance limitation that exist in single-robot vision acquisition, forming a global environmental visual representation with wider coverage, more complete information, and richer details.

[0055] Specifically, the fusion processing of multiple environmental images can be achieved through techniques such as image spatial registration, feature extraction and matching, pixel-level / feature-level fusion, noise suppression and detail enhancement, and coordinate system mapping. By combining the pose information of multiple robots and the environmental features involved in the camera, images acquired from different perspectives and at different times are aligned, fused and optimized to generate a global environmental image with more complete information and richer details.

[0056] For example, after receiving the global state dataset of the entire robot (including the intrinsic and extrinsic parameters, position, and field of view of each robot's camera), the set of image frames (environmental images) synchronously acquired by each robot, and the global spatial position coordinates of the first user, the first robot end-side main computing card performs the following calculation steps to generate a multi-machine visual fusion image:

[0057] First, a camera projection model is constructed based on the intrinsic parameter matrices of each robot's camera. extrinsic parameter matrix The projection transformation relationship is constructed, and the projection matrix formula is:

[0058] ;

[0059] in,( , () represents the image pixel coordinates. K represents the depth value in the camera coordinate system. For the camera intrinsic parameter matrix ( , Focal length , (Primary point coordinates) For the camera extrinsic matrix ( Rotation matrix, Translation vector), ( , , () represents the three-dimensional coordinates in the global world coordinate system;

[0060] Next, multi-view image homography transformation is performed, and the homography matrix is ​​calculated based on the projection matrix and the ground plane equation. By using inverse projection transformation, the 2D image frames acquired by each robot are mapped to the global ground plane (BEV viewpoint) to achieve spatial alignment. Homography transformation formula:

[0061] ;

[0062] in,( , ) represents the pixel coordinates of the original image, ( , () represents the pixel coordinates of the BEV viewpoint. It is obtained by solving the ground plane equation using camera intrinsic and extrinsic parameters.

[0063] Then, multi-machine image weighted fusion calculation is performed. The spatially aligned multi-machine BEV feature maps are used to generate a global multi-machine visual fusion image according to the weighted fusion formula. :

[0064] ;

[0065] in, For global multi-machine visual fusion images exist( , The pixel value at () For the first The BEV feature map of robot No. 1 after transformation is in ( , The pixel value at () The total number of robots participating in the integration. For the first The image fusion weights of robot number 1 are calculated using the following formula:

[0066] ;

[0067] in, For the first Confidence level of image acquisition by robot No. 1 The Euclidean distance between the robot and the first user. This is the distance attenuation coefficient. For the first The global spatial position coordinates of robot number 1 Let be the global spatial coordinates of the first user, and exp() be the natural exponential function;

[0068] Finally, based on the fusion Complete static obstacle detection and semantic segmentation, generate a global scene static obstacle map and global spatial semantic segmentation results, and construct a relative position mapping matrix between the user and the robot. Finally, output a global BEV fusion image, a global scene map, a relative position mapping matrix and semantic segmentation results for reuse in subsequent steps.

[0069] S303, determine the robot call detection result based on the user's voice command and the historical dialogue record between the first robot and the first user in the current dialogue cycle.

[0070] The robot call detection result includes the device number of the specific robot or a preset anomaly value, wherein the preset anomaly value indicates that there is no specific robot in the current dialogue cycle.

[0071] In one possible embodiment, determining the robot call detection result based on the user's voice command and the historical dialogue records between the first robot and the first user in the current dialogue cycle includes: determining the voice text content based on the user's voice command; identifying whether the voice text content contains preset robot call feature words, the robot call feature words including robot device number, robot unique name, and robot location identifier; if yes, generating the robot call detection result based on the device number of the specific robot pointed to by the robot call feature words; and if no, querying and identifying whether the historical dialogue records of the first user in the current dialogue cycle contain the robot call feature words; if yes, generating the robot call detection result based on the device number of the specific robot pointed to by the robot call feature words; and if no, generating the robot call detection result based on the preset outlier.

[0072] Among them, robot call feature words are pre-stored identification information used to identify specific robot devices in user voice commands. These include the robot's unique number, unique name, location identifier, and function type identifier. For example, robot device number R001, robot's unique name "Xiao A", location identifier "Construction Site No. 3", and function type identifier "lifting operation".

[0073] For example, if a user says, "Robot A in the living room, please check the temperature in the living room," the system will parse the voice text, identify the feature words "A" and the location marker "living room," and directly generate a robot call detection result, marking "Robot A" as the specific robot. If the user first asks, "Can the devices in the living room help me check the temperature?" and then issues the voice command "Help me operate it," and the system does not find any robot call feature words in the voice text corresponding to the current voice command, it will review the historical dialogue, identify the location marker "living room" in the history, and then generate a non-empty robot call detection result, marking the robot A located in the object as the specific robot. If the user only says, "How's the weather today?", there are no matching robot call feature words, and a robot call detection result will be generated based on preset outliers, determining the detection result to be empty.

[0074] Furthermore, if no historical dialogue record of the first user in the current dialogue cycle is found, the system directly determines that the robot call detection result is empty, does not mark any specific robot, only retains the basic voice interaction state, and waits for the user to supplement the complete robot call feature words before re-performing call recognition and response matching.

[0075] S304, the robot call detection result, the multi-machine visual fusion image, the first user location, and the multiple local status information of multiple robots corresponding to the multiple broadcast data packets are input into the home robot determination model to obtain the target robot number output by the home robot determination model.

[0076] Specifically, the robot attribution determination model takes multi-dimensional heterogeneous data as input and can integrate multiple data features such as spatial location, robot computing power status, environmental visual information, user voice features and historical dialogue scene information. Through preset determination logic and algorithm model, it performs comprehensive analysis and decision-making, and outputs robot number results that adapt to the user's voice command. It has the characteristics of multi-feature fusion, efficient edge-side reasoning and accurate determination results.

[0077] In one possible embodiment, the attribution robot determination model is configured to perform the following operations: if the robot call detection result is detected to include the device number of the specific robot, then the device number of the specific robot is determined as the target robot number; if the robot call detection result is detected to include the preset outlier, then a reference position range mapped by the multi-machine visual fusion image is determined, the reference position range representing a spatial area that can be observed by multiple robots in the current environment; and, the spatial occupancy status of the first user's position relative to the reference position range is determined, the spatial occupancy status being empty indicates that the first user is outside the field of view of the multiple robots, and the spatial occupancy status being non-empty indicates that the first user is within the field of view of at least one robot; and, the target robot number is determined based on the spatial occupancy status, multiple local state information of the multiple robots, and the first user's position.

[0078] Among them, the reference position range is a spatial area that can be observed by multiple robots through multi-machine vision fusion image mapping. It is the core basis for determining whether the user's position is within the robot's field of view. The spatial occupancy status is used to identify the user's position relative to the reference range. It is divided into two categories: outside the field of view and within the field of view. It directly determines the decision logic of the robot's subsequent selection.

[0079] In one possible embodiment, determining the spatial occupancy state of the first user's position relative to the reference position range includes: determining the spatial boundary of the reference position range in the global coordinate system; determining whether the first user's position falls within the spatial boundary of the reference position range; if it is determined that the first user's position falls within the spatial boundary of the reference position range, then determining that the spatial occupancy state is non-empty; if it is determined that the first user's position exceeds the spatial boundary of the reference position range, then determining that the spatial occupancy state is empty.

[0080] Specifically, the user's spatial occupancy status determination is based on the fusion of machine perception images and global semantic segmentation results to detect the spatial occupancy area of ​​the user's human body target in the global scene and determine the user's global position. The formula for determining whether something is within the effective field of view of any robot's camera is as follows:

[0081] ;

[0082] in, M represents the effective field of view of the camera of robot m in the three-dimensional space of the global coordinate system; M is the total number of robots participating in the fusion. This represents the user's position in the viewfinder. A non-empty position indicates that the user is within the viewfinder of at least one robot, while an empty position indicates that the user is outside the viewfinder of all robots.

[0083] It should be noted that this application does not limit the specific construction method of the global coordinate system, the algorithm for dividing the spatial boundary of the reference position range, or the positioning technology for the first user position, including but not limited to obtaining user position information based on satellite positioning, indoor base station positioning, visual positioning, inertial navigation positioning, etc.

[0084] Specifically, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating a scenario where the user space is empty, as provided in an embodiment of this application. Figure 4 As shown, the first user's position is outside the spatial boundary of the reference position range. After the system completes the spatial boundary anchoring and position matching judgment through the global coordinate system, it determines that the spatial occupancy state is empty, that is, the first user is outside the field of view of multiple robots. The robots will maintain the normal inspection or standby state and will not trigger the targeted interactive response task.

[0085] In one possible embodiment, determining the target robot number based on the space occupancy status, multiple local state information of the multiple robots, and the first user's location includes: if it is determined that the space occupancy status is empty, then determining the target robot number based on the multiple local state information of the multiple robots and the first user's location; if it is determined that the space occupancy status is not empty, then determining the occupancy area image of the first user in the multi-machine visual fusion image based on the first user's location; and detecting whether there is a robot within a preset range corresponding to the first user's facial orientation based on the occupancy area image; if a robot is detected, then determining the device number of the detected robot as the target robot number; and if no robot is detected, then determining the target robot number based on the multiple local state information of the multiple robots and the first user's location.

[0086] Among them, the occupancy area image is the imaging result of the area where the first user is located in the multi-machine vision fusion image. This image is generated by fusing the image information collected by vision devices on multiple robots, and accurately maps the occupancy area of ​​the first user in the global coordinate system. When determining this image, the system will first combine the first user's location information to delineate the corresponding area in the multi-machine vision fusion image, and then integrate the images of multiple cameras through image fusion algorithms, remove redundant background information, highlight the image features of the user's occupancy area, and form exclusive image data that can be used for subsequent detection.

[0087] The preset range is defined by taking the first user's facial orientation as the core and combining it with the spatial range of the robot deployment. Its size and shape are determined according to the robot's visual detection angle, coverage radius and environmental layout, and it is usually a fan-shaped or circular area in the direction of the user's facial orientation.

[0088] For example, user body and facial keypoint detection includes: if the user's space occupancy is not empty, based on a locally deployed human keypoint detection model, detecting 21 human body keypoints and 106 facial keypoints of the user in a robot image frame containing the user, obtaining a set of facial keypoint coordinates for the user's facial region; and then, calculating the three-dimensional Euler angles of the facial orientation, including: based on the facial keypoint coordinates and combined with a standard 3D face model, calculating the three-dimensional Euler angles of the user's facial orientation using the Perspective-n-Point (PnP) algorithm, including yaw (left-right orientation), pitch (up-down orientation), and roll. The Euler angle calculation formula is:

[0089] ;

[0090] in, These are the pixel coordinates of two-dimensional facial key points. These are the coordinates of the three-dimensional key points corresponding to the 3D standard human face model. This is a rotation matrix, which can be converted to Euler angles (yaw, pitch, roll). It is a translation vector. This is the camera intrinsic parameter matrix;

[0091] Next, robot detection is performed within the orientation range, including: determining a preset angle range (e.g., ±30°) directly in front of the user's face based on the calculated facial yaw angle. In the global coordinate system, construct a ray and a fan-shaped detection region representing the user's facial orientation, and determine whether a robot exists within this region. The calculation formula is as follows:

[0092] ;

[0093] in, The global azimuth angle of the user's face; Let m be the azimuth angle of the robot relative to the user; Let be the angle between the two. If ≤30 If the robot is within the range of the user's face orientation, it is determined to be included in the candidate robot set;

[0094] Then, the confidence level of the results is verified, including: verifying the confidence level of the detection results. If the confidence level of the facial key point detection is lower than the threshold such as 0.7, the orientation detection result is determined to be invalid and processed as no orientation matching result.

[0095] Finally, the system outputs the first user's spatial occupancy status (empty / non-empty), the user's facial orientation in 3D Euler angles, the set of candidate robots within the orientation range, and the facial orientation detection confidence.

[0096] Specifically, please refer to Figure 5 , Figure 6 , Figure 5 This is a schematic diagram illustrating a scenario where the user space is not empty, as provided in an embodiment of this application. Figure 6 This is a schematic diagram of another scenario where the user space is not empty, as provided in the embodiments of this application. Figure 5 , Figure 6 As shown, the spatial occupancy state of the first user's position relative to the reference position range is not empty, that is, the first user is located within the framing range of multiple robots.

[0097] like Figure 5As shown, the first user is within the reference position range of the multi-machine visual fusion image mapping, and the space occupancy state is non-empty. The preset range corresponding to the user's face orientation covers the first robot next to the table. After the model detects the robot, it directly determines its device number as the target robot number, thus achieving a precise response.

[0098] like Figure 6 As shown, the first user is also within the reference position range, and the space occupancy status is non-empty. The preset range corresponding to the user's face orientation does not cover any robot. After the model does not detect the target robot, it selects and determines the optimal other robot as the target robot number based on the multiple local state information of multiple robots and the position of the first user.

[0099] In one possible embodiment, the multiple local state information of the multiple robots includes multiple spatial locations and multiple computing resource occupancy rates corresponding to each of the multiple robots; determining the target robot number based on the multiple local state information of the multiple robots and the first user location includes: determining the relative distances between the multiple spatial locations and the first user location to obtain multiple relative distances corresponding to each of the multiple robots; scoring the relative distance of each of the multiple robots according to a first preset scoring rule to obtain multiple first scores, wherein the scores of the first scores are negatively correlated with the distance of the relative distance; scoring the computing resource occupancy rate of each of the multiple robots according to a second preset scoring rule to obtain multiple second scores, wherein the scores of the second scores are negatively correlated with the computing resource occupancy rate; weighting and summing the first scores and second scores corresponding to each robot according to a first preset weight and a second preset weight to obtain multiple comprehensive scores; and determining the device number of the robot with the highest comprehensive score among the multiple robots as the target robot number.

[0100] Specifically, the closer the distance and the lower the computing power consumption, the higher the score; the comprehensive score is calculated by combining the distance weight and computing power weight to select the optimal robot, taking into account both efficiency and resource utilization.

[0101] For example, a nursing home matches users with service robots, setting a weight of 0.6 for distance score and 0.4 for computing power score. Candidate robot A: distance 30 meters, computing power utilization 60%, distance score 90, computing power score 80, overall score = 90 × 0.6 + 80 × 0.4 = 86; Candidate robot B: distance 40 meters, computing power utilization 50%, distance score 80, computing power score 90, overall score = 80 × 0.6 + 90 × 0.4 = 84; Candidate robot C: distance 60 meters, computing power utilization 65%, distance score 70, computing power score 85, overall score = 70 × 0.6 + 85 × 0.4 = 76. Ultimately, robot A with the highest overall score is selected as the target robot.

[0102] It should be noted that this application only provides one embodiment for determining the target robot number based on multiple relative distances and multiple computing resource occupancy rates, including but not limited to a weighted scoring method based on relative distance and computing resource occupancy rate, a screening and sorting method based on relative distance priority, a dynamic allocation method based on computing resource occupancy rate priority, and a multi-dimensional optimization allocation method that comprehensively considers spatial location and computing resources.

[0103] As can be seen, in this embodiment, through multi-level collaborative mechanisms such as multi-machine vision fusion, robot call detection, spatial occupancy status determination, facial orientation matching, and multi-dimensional scoring and filtering, the accurate matching of user voice commands and target robots is achieved. This not only ensures the accurate response of specific robots, but also dynamically selects the optimal robot by combining visual, location, and computing power information when there is no clear direction. At the same time, it is compatible with various positioning and algorithm implementation methods, which greatly improves the accuracy, real-time performance, and resource utilization efficiency of interaction in multi-robot collaborative scenarios.

[0104] Specifically, please refer to Figure 7 , Figure 7 This is a schematic diagram of an attribution robot determination model provided in an embodiment of this application, such as... Figure 7 As shown, the robot attribution determination model is a decision model based on the fusion of multi-source heterogeneous data. Its layers, from front to back, are the input layer, data preprocessing layer, multi-feature fusion layer, decision reasoning layer, and output layer.

[0105] The input layer serves as the unified input interface and front-end adaptation unit for the model. The input consists of multi-source heterogeneous raw data, including robot call detection results, multi-machine vision fusion images, robot spatial location / device number / computing power utilization, and first user location data. The core processing involves receiving data through a unified interface, performing multi-source data temporal alignment, format adaptation, and tensor conversion, while simultaneously performing basic legality checks and filtering invalid data packets. The output is a temporally aligned, uniformly formatted raw input structured dataset, which is then passed to the data preprocessing layer.

[0106] The data preprocessing layer is the standardized processing unit of the model. The input is the original structured dataset output by the input layer. The core processing is to perform customized preprocessing for different types of data, complete the normalization of numerical data, global alignment of spatial coordinates, and optimization of visual image noise reduction, and simultaneously perform outlier removal and missing value filling of the entire dataset. The output is a standardized preprocessed dataset with uniform dimensions and no outliers, which is passed to the multi-feature fusion layer.

[0107] The multi-feature fusion layer is the core feature processing unit of the model. Its input is the standardized preprocessed dataset output by the data preprocessing layer. The core processing involves extracting four types of core features—semantic, visual, spatial, and device status—through a multi-branch structure. These features are then combined through feature concatenation and attention-weighted fusion to achieve multi-modal feature integration. Redundant features are then removed using a dimensionality reduction algorithm. The output is a unified high-dimensional fusion feature vector used for attribution determination, which is then passed to the decision reasoning layer.

[0108] Specifically, the multi-feature fusion layer integrates core features through multimodal feature concatenation and attention-weighted fusion. Let the four types of core feature vectors be: semantic features... ∈R Ms The dimension Ms represents the semantic information visual features extracted from user voice commands and historical dialogue records. ∈R Mv The dimension is Mv, representing the visual and spatial features extracted from multi-machine visual fusion images, facial orientation, and environmental visual information; ∈R Msp The dimension is Msp, which represents spatial information such as the relative distance between the robot and the user, global spatial coordinates, position mapping relationship, and device status characteristics. ∈R Md The dimension Md represents hardware-related features such as robot computing power utilization, device number, and operating status, while R represents the rotation matrix.

[0109] Initial fusion vector after feature concatenation:

[0110] ;

[0111] in, This is the initial fused feature vector. This indicates the feature vector concatenation operation.

[0112] A channel attention mechanism is introduced to weight the concatenated features. The attention weights are calculated as follows:

[0113] ;

[0114] in, Let W be the number of feature channels, and W and b be the weight matrix and bias of the attention layer, respectively. The attention weight for the j-th feature channel is used to highlight key features and suppress redundant features. For the initial fused feature vector The eigenvalue of the j-th feature channel; For the initial fused feature vector The Middle The eigenvalues ​​of each feature channel; The activation function is a linear correction unit, and the formula is: =max(0,x) is used to introduce nonlinear characteristics and enhance the model's feature representation ability.

[0115] Redundant features are removed using dimensionality reduction algorithms (such as Principal Component Analysis (PCA) to obtain a unified high-dimensional fused feature vector. ∈R D D is the feature dimension after dimensionality reduction.

[0116] The decision reasoning layer is the core logic judgment unit of the model. Its input is a unified high-dimensional fusion feature vector output by the multi-feature fusion layer. The core processing is to perform reasoning according to hierarchical logic. First, it completes the specific robot call matching judgment. If there is no specific robot, it sequentially completes the user space occupancy status judgment and facial orientation matching judgment. If there is no matching robot, it completes the candidate robot ranking and optimization through weighted scoring. Simultaneously, it calculates the reasoning confidence and matching basis. The output is the final reasoning result of the target robot number, with the reasoning confidence and matching basis labels, which is passed to the output layer.

[0117] The output layer is the unified output interface and result adaptation unit of the model. The input is the reasoning result, reasoning confidence and matching basis label output by the decision reasoning layer. The core processing is to complete the standardized conversion of the reasoning result format, the structured encapsulation of the judgment information and the output legality verification. The output is the standardized target robot number, which is accompanied by structured reasoning information and judgment log, and sent to the downstream task scheduling system and the upper-level management and control platform.

[0118] Specifically, the model is trained using the cross-entropy loss function, which is:

[0119] ;

[0120] in, This is the total loss value of the model, used to measure the error between the predicted result and the true label. The training objective is to minimize this loss. N represents the total number of training samples, i.e., the number of multi-robot command response scenario samples used in this model training. The label (one-hot encoded) of robot k corresponding to sample i. The model predicts the probability that sample i belongs to robot k, where K is the total number of robots.

[0121] As can be seen, in this embodiment, the attribution robot determination model, through a clear five-layer hierarchical architecture and standardized training process, achieves orderly processing of multi-dimensional input data, efficient feature fusion, and accurate decision-making reasoning. This not only ensures the accuracy and real-time performance of target robot determination, but also adapts to multi-robot collaboration scenarios through a continuous iteration mechanism, ensuring that the determination results are feasible and optimizable, effectively supporting the collaborative service needs of multiple users and multiple robots.

[0122] Please refer to details. Figure 8 , Figure 8 This is a flowchart illustrating the intent recognition of user voice commands provided in an embodiment of this application, such as... Figure 8 As shown, the step of performing intent recognition on the user's voice command based on the historical dialogue state dataset corresponding to the first user to obtain the target intent includes the following steps:

[0123] S801, based on the historical dialogue state dataset corresponding to the first user, perform intent recognition on the user's voice command to obtain the target intent.

[0124] The historical dialogue state dataset for the first user is a collection of all-dimensional interaction and association state data stored long-term by the first robot for that user. This dataset is stored in the robot's memory and can be retrieved as needed by the main controller. The historical dialogue state dataset includes historical dialogue data, the state of historical events in which the user participated, the state data of items associated with the user, and user profile data representing the user's activity habits.

[0125] In one possible embodiment, the step of performing intent recognition on the user's voice command based on the historical dialogue state dataset corresponding to the first user to obtain the target intent includes: performing intent recognition on the voice text content corresponding to the user's voice command to obtain an initial intent; performing voiceprint recognition on the user's voice command to determine a first voiceprint feature set; querying a pre-stored historical dialogue state database based on the first voiceprint feature set to obtain a historical dialogue state dataset corresponding to the first user, wherein the historical dialogue state database pre-stores multiple voiceprint feature sets and multiple historical dialogue state datasets corresponding to multiple users; and adjusting the initial intent based on the historical dialogue state dataset to obtain the target intent.

[0126] In a specific embodiment, natural language processing technology is used to perform intent recognition on the speech text content. First, preprocessing operations such as word segmentation, part-of-speech tagging, and named entity recognition are performed to extract core text features. Then, rule-based matching, text classification models (such as CNN and BERT), or intent classification algorithms are combined to map the speech text to the corresponding initial intent, thereby achieving a preliminary determination of the core purpose of the user's command. This application does not impose any limitations on this.

[0127] The first voiceprint feature set is a set of features obtained by digitally extracting voiceprint information from user voice commands. It contains feature parameters in the speech signal that are strongly related to the speaker's identity, such as fundamental frequency, formants, vocal tract spectrum, Mel frequency cepstral coefficients (MFCC), linear predictive coding (LPC) coefficients, and other core features.

[0128] For example, based on the feature vector of the extracted voiceprint features, it is matched with the user voiceprint templates pre-stored in the local voiceprint feature database, and the matching degree is calculated using cosine similarity, as shown in the formula:

[0129] ;

[0130] in, The similarity score between the extracted voiceprint features and the voiceprint of the k-th user is calculated, with a value ranging from -1 to 1. A larger value indicates a higher voiceprint matching degree. Let k be the voiceprint template feature vector of the k-th user. The feature vector of the Mel-frequency cepstral coefficients (MFCC) of the user's voice commands;

[0131] Set the matching threshold Tvoice=0.75, if the maximum similarity If the similarity is greater than or equal to Tvoice, the user is determined to be a registered user in the database, and a unique user ID is output. If the similarity is below the threshold, the user is determined to be a new user, a temporary user ID is generated, and a new voiceprint template is created and stored in the database. Finally, the unique ID of the first user is output, and the historical dialogue state dataset that uniquely corresponds to that ID is found.

[0132] Furthermore, the historical dialogue state dataset integrates multi-dimensional data, which can compensate for the information gaps in single-turn voice commands and enable dynamic calibration and correction of intent. This includes, but is not limited to, using technologies such as natural language understanding, contextual semantic association analysis, and historical behavior pattern mining, combined with rule matching and machine learning models, to verify, supplement, or correct the initial intent, ultimately generating a target intent that fits the user's real needs, thereby improving the interactive accuracy and personalized service capabilities of the intelligent service system.

[0133] For example, when a senior care service robot receives a voice command from an elderly person saying "Please get me my water cup," it first converts the voice into text and performs intent recognition, initially determining the initial intent to be for item delivery. Then, it performs voiceprint recognition on the user's voice, extracting a first voiceprint feature set containing the user's voiceprint characteristics. Based on this feature set, it queries the historical dialogue state database, retrieving the elderly person's previous interaction records (e.g., the elderly person's habit of obtaining items at 9 am, frequently using the water cup from the bedside table), the status of historical events participated in (e.g., the elderly person's fixed daily rehabilitation activity time), the water cup status data associated with the user (the water cup is placed on the coffee table in the living room), and the elderly person's daily activity habits (the elderly person often sits in the sofa area of ​​the living room). Finally, combining the aforementioned historical dialogue state dataset, it adjusts the initial item delivery intent to the target intent of "taking the water cup from the coffee table in the living room and delivering it to the elderly person's sofa," thus completing accurate intent recognition and task matching.

[0134] As can be seen, in this embodiment, by accurately matching the user's identity through voiceprint recognition and retrieving their full-dimensional historical dialogue state dataset, and combining it with natural language processing technology to complete the initial intent recognition and dynamic calibration, it not only makes up for the information limitations of single-turn voice commands, but also realizes personalized correction of intent, significantly improving the accuracy of user voice command intent recognition and the level of intelligent interaction in elderly care service scenarios, and better meeting the personalized service needs of elderly users.

[0135] S802, Generate a sequence of subtasks to achieve the target intent based on the historical dialogue state dataset.

[0136] Furthermore, after determining the target intent corresponding to the user's voice command, the system will decompose the complex intent into a series of basic sub-tasks that are executed sequentially, logically related, and can be completed independently, based on the historical dialogue state dataset and through task decomposition algorithms (such as Markov decision process models). The system will clarify the execution subject, operation parameters, and execution conditions of each sub-task, and at the same time generate the dependency relationships and result verification rules between sub-tasks, ultimately forming a complete and implementable sub-task sequence to drive the robot to complete the corresponding operation.

[0137] Further, in one possible embodiment, generating a subtask sequence to achieve the target intent based on the historical dialogue state dataset includes: determining the intent type of the target intent, the intent type including item acquisition, navigation guidance, information query, emotional support, safety reminder, and device control; calling a pre-stored subtask template library to query a subtask sequence template corresponding to the intent type, the subtask template library including multiple intent types and their corresponding multiple subtask sequence templates; obtaining an initial subtask sequence of the target intent based on the subtask sequence templates; determining task constraint rules based on the historical dialogue state dataset, the task constraint rules including movement path adjustment rules, task type priority, subtask execution order, and executor calling parameters, the executor including the robot's robotic arm and mobile chassis; and adjusting the initial subtask sequence according to the task constraint rules to obtain a subtask sequence to achieve the target intent.

[0138] Among these, the intents for item retrieval can generate delivery sub-tasks by querying the location of the target item; the intents for navigation guidance are used to plan the path from the current location to a designated location; the intents for information query can respond to user inquiries about device status and environmental parameters; the intents for emotional support can interact with the user through preset dialogue content; the intents for safety reminders can trigger warning prompts when an anomaly is detected; and the intents for device control can enable the switching on and off and parameter adjustment of hardware such as robot actuators and sensors. This application does not limit the specific intent types.

[0139] Among them, the task constraint rules are rules that regulate and constrain the entire process of robot task execution. The movement path adjustment rules are used to clarify the robot's movement trajectory, obstacle avoidance strategy, docking point and other movement requirements. The task type priority is used to divide the execution order of different types of tasks. When multiple tasks are concurrent, the higher priority task is determined according to the preset rules to respond first and avoid task conflicts. The task types can include safety tasks, operation tasks and navigation tasks. The subtask execution order specifies the execution order logic of the subtasks after a single task is broken down. The actuator call parameters define the parameter standards such as the start threshold of hardware devices such as robotic arms, sensors and actuators.

[0140] S803, execute the subtask sequence.

[0141] Specifically, when executing a sequence of subtasks, the system prioritizes and allocates resources based on the computing power card, breaking down the overall intent into independently executable basic subtasks, which are then issued sequentially according to a preset timeline and dependencies. The main controller coordinates the robotic arm to perform grasping, placing, and rotating actions based on the subtask type, controls the chassis to move to the target position, and provides real-time status feedback through the sensing module. The computing power card is responsible for real-time calculation of motion trajectories, obstacle avoidance paths, and force control parameters to ensure stable coordination among all execution units. For example, when the user command is "Put the water glass on the table to the second shelf of the bookshelf," the system first breaks it down into a sequence of subtasks such as "Move to the tabletop," "robotic arm grasps the water glass," "Move to the second shelf of the bookshelf," and "Place the water glass." The computing power card plans the robotic arm's end effector trajectory and the chassis's obstacle avoidance path. The robotic arm performs precise grasping and placing according to the command, and the chassis moves smoothly with computing power support, ultimately achieving the target intent.

[0142] As can be seen, in this embodiment, by accurately decomposing the target intent, calling the appropriate sub-task template, and optimizing the sub-task sequence by combining the task constraint rules determined by the historical dialogue state dataset, and then through the coordinated scheduling of the main controller and the computing card, the robot's execution units are driven to execute the sub-tasks in an orderly manner. This achieves accurate judgment and execution of the target intent, ensuring the standardization, stability and efficiency of task execution, and further improving the accuracy and personalization of robot services by adapting to user habits based on personalized constraint rules.

[0143] Please see Figure 9 , Figure 9 This is a schematic diagram illustrating a scenario where a robot responds to user voice commands, as provided in an embodiment of this application. Figure 9 As shown, the first robot, based on recognizing the target intent of the first user (elderly person), analyzes the task intent and breaks down the overall task into multiple ordered sub-task sequences. The first robot completes each sub-task sequentially according to a preset execution order. Following a planned path, the first robot moves to the target location via its mobile chassis, then uses its robotic arm to grasp the water cup. It then returns to the first user (elderly person) along the same path, hands the water cup to the user on the recliner, and says in a friendly tone, "Sorry to keep you waiting, here's your water cup. Please call me anytime if you have any problems," thus completing the sub-task sequence and achieving the target intent. In the background, other robots stand by, serving several other elderly people.

[0144] As can be seen, in this embodiment, by accurately identifying the user's target intent, breaking down and executing the sub-task sequence in an orderly manner, and combining the planned movement path with the operation of the robotic arm, safe, efficient and considerate personalized services for multiple users are achieved. At the same time, the collaborative standby of multiple robots ensures the continuity of services, significantly improving the intelligence of multi-user services.

[0145] This application embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0146] Please see Figure 10 , Figure 10 This is a functional unit block diagram of a multi-robot service system provided in this application embodiment. The multi-robot service system includes: a response unit 101 and a processing unit 102; wherein, the response unit 101 is used to respond to a user's voice command, acquire a first environmental image, determine the location of a first user; and create a first broadcast data packet for the first robot based on the first user's location, the local status information of the first robot, and the first environmental image; and receive second broadcast data packets from other robots to obtain multiple broadcast data packets including the first broadcast data packet; the processing unit 102 is used to obtain the target robot number responding to the user's voice command based on the multiple broadcast data packets, the user's voice command, the historical dialogue record between the first robot and the first user in the current dialogue cycle, and a pre-trained attribution robot determination model; when it is detected that the target robot number is consistent with the device number of the first robot, it continues to respond to the service request indicated by the user's voice command.

[0147] In one possible embodiment, based on the multiple broadcast data packets, the user voice command, the historical dialogue records between the first robot and the first user in the current dialogue cycle, and a pre-trained attribution robot determination model, the target robot number responding to the user voice command is obtained. The processing unit 102 is specifically configured to: parse the second broadcast data packets to obtain the second environmental image collected by the other robots in response to the user voice command and the local state information of the other robots, the robot's local state information including the timestamp, spatial location, device number, and computing resource utilization rate of the current processing time window; perform fusion processing on the first environmental image and the second environmental image to obtain a multi-machine visual fusion image; determine the robot call detection result based on the user voice command and the historical dialogue records between the first robot and the first user in the current dialogue cycle, the robot call detection result including the device number of a specific robot or a preset anomaly value, the preset anomaly value indicating that there is no specific robot in the current dialogue cycle; input the robot call detection result, the multi-machine visual fusion image, the first user's location, and the multiple local state information of the multiple robots corresponding to the multiple broadcast data packets into the attribution robot determination model to obtain the target robot number output by the attribution robot determination model.

[0148] In one possible embodiment, the robot call detection result is determined based on the user's voice command and the historical dialogue records between the first robot and the first user in the current dialogue cycle. The processing unit 102 is specifically configured to: determine the voice text content based on the user's voice command; identify whether the voice text content contains preset robot call feature words, the robot call feature words including robot device number, robot unique name, and robot location identifier; if yes, generate the robot call detection result based on the device number of the specific robot pointed to by the robot call feature words; and if no, query and identify whether the historical dialogue records of the first user in the current dialogue cycle contain the robot call feature words; if yes, generate the robot call detection result based on the device number of the specific robot pointed to by the robot call feature words; and if no, generate the robot call detection result based on the preset outlier.

[0149] In one possible embodiment, the attribution robot determination model performs the following operations: if the robot call detection result is detected to include the device number of the specific robot, then the device number of the specific robot is determined as the target robot number; if the robot call detection result is detected to include the preset outlier, then a reference position range mapped by the multi-machine visual fusion image is determined, the reference position range representing a spatial area that can be observed by multiple robots in the current environment; and, the spatial occupancy state of the first user position relative to the reference position range is determined, the spatial occupancy state being empty indicates that the first user is outside the field of view of the multiple robots, and the spatial occupancy state being non-empty indicates that the first user is within the field of view of at least one robot; and, the target robot number is determined based on the spatial occupancy state, multiple local state information of the multiple robots, and the first user position.

[0150] In one possible embodiment, determining the spatial occupancy status of the first user's position relative to the reference position range includes: determining the spatial boundary of the reference position range in the global coordinate system; determining whether the first user's position falls within the spatial boundary of the reference position range; if it is determined that the first user's position falls within the spatial boundary of the reference position range, then determining that the spatial occupancy status is non-empty; if it is determined that the first user's position exceeds the spatial boundary of the reference position range, then determining that the spatial occupancy status is empty.

[0151] In one possible embodiment, determining the target robot number based on the space occupancy status, multiple local state information of the multiple robots, and the first user's location includes: if it is determined that the space occupancy status is empty, then determining the target robot number based on the multiple local state information of the multiple robots and the first user's location; if it is determined that the space occupancy status is not empty, then determining the occupancy area image of the first user in the multi-machine visual fusion image based on the first user's location; and detecting whether there is a robot within a preset range corresponding to the first user's facial orientation based on the occupancy area image; if a robot is detected, then determining the device number of the detected robot as the target robot number; and if no robot is detected, then determining the target robot number based on the multiple local state information of the multiple robots and the first user's location.

[0152] In one possible embodiment, the multiple local state information of the multiple robots includes multiple spatial locations and multiple computing resource occupancy rates corresponding to each of the multiple robots; determining the target robot number based on the multiple local state information of the multiple robots and the first user location includes: determining the relative distances between the multiple spatial locations and the first user location to obtain multiple relative distances corresponding to each of the multiple robots; scoring the relative distance of each of the multiple robots according to a first preset scoring rule to obtain multiple first scores, wherein the scores of the first scores are negatively correlated with the distance of the relative distance; scoring the computing resource occupancy rate of each of the multiple robots according to a second preset scoring rule to obtain multiple second scores, wherein the scores of the second scores are negatively correlated with the computing resource occupancy rate; weighting and summing the first scores and second scores corresponding to each robot according to a first preset weight and a second preset weight to obtain multiple comprehensive scores; and determining the device number of the robot with the highest comprehensive score among the multiple robots as the target robot number.

[0153] In one possible embodiment, in response to the intent task requirement of the user's voice command, the processing unit 102 is specifically configured to: perform intent recognition on the user's voice command based on the historical dialogue state dataset corresponding to the first user to obtain the target intent, wherein the historical dialogue state dataset includes historical dialogue data, the state of historical events in which the user participated, the state data of items associated with the user, and user profile data representing the user's activity habits; generate a sub-task sequence to achieve the target intent based on the historical dialogue state dataset; and execute the sub-task sequence.

[0154] In one possible embodiment, the processing unit 102 performs intent recognition on the user's voice command based on the historical dialogue state dataset corresponding to the first user to obtain the target intent. Specifically, the processing unit 102 is used to: perform intent recognition on the voice text content corresponding to the user's voice command to obtain an initial intent; perform voiceprint recognition on the user's voice command to determine a first voiceprint feature set; query a pre-stored historical dialogue state database based on the first voiceprint feature set to obtain a historical dialogue state dataset corresponding to the first user, wherein the historical dialogue state database pre-stores multiple voiceprint feature sets and multiple historical dialogue state datasets corresponding to multiple users; and adjust the initial intent based on the historical dialogue state dataset to obtain the target intent.

[0155] As can be seen, in this embodiment, by integrating broadcast data packet information through multi-machine interaction, and combining historical dialogue records with a pre-trained attribution robot determination model, the unique target robot for the instruction response is accurately determined. Then, for the target robot, the intent recognition of the voice instruction and the generation and execution of sub-task sequences are completed. This achieves the unique determination of the subject of the instruction response in a multi-robot coexistence scenario, effectively avoiding the conflict problem of multi-machine competitive response. At the same time, by combining scene information, the intent disambiguation of ambiguous voice instructions is completed, improving the accuracy of instruction understanding and execution.

[0156] It is understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the method embodiment section in this application should be adapted to the device embodiment section in a synchronous manner, and will not be repeated here.

[0157] Figure 11 This is a structural block diagram of an electronic device provided in an embodiment of this application. For example... Figure 11 As shown, the electronic device 1100 may include one or more of the following components: a processing module 1101 and a memory 1102 coupled to the processing module 1101, wherein the memory 1102 may store one or more computer programs, which may be configured to implement the methods described in the examples above when executed by one or more processing modules 1101.

[0158] The processing module 1101 may include one or more processing cores. The processing module 1101 connects to various parts within the electronic device 1100 using various interfaces and lines. It executes various functions and processes data of the electronic device 1100 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1102, and by calling data stored in the memory 1102. Optionally, the processing module 1101 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processing module 1101 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. It is understood that the aforementioned modem may also not be integrated into the processing module 1101 and may be implemented separately through a communication chip.

[0159] The memory 1102 may include random access memory (RAM) or read-only memory (ROM). The memory 1102 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1102 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method examples described above. The data storage area may also store data created during the use of the electronic device 1100.

[0160] It is understood that the electronic device 1100 may include more or fewer structural elements than those shown in the above block diagram, such as a power module, physical buttons, WiFi (Wireless Fidelity) module, speaker, Bluetooth module, sensor, etc., without limitation.

[0161] This application also provides a computer storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements some or all of the steps of any of the methods described in the above method embodiments.

[0162] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments.

[0163] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0165] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0167] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute partial steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, volatile memory, or non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM), etc., which are various media capable of storing program code.

[0168] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can easily conceive of variations or substitutions without departing from the spirit and scope of the present invention, and various modifications and alterations can be made, including combinations of the different functions and implementation steps described above, as well as software and hardware implementation methods, all of which are within the protection scope of the present invention.

Claims

1. A method for responding to voice commands from multiple robots, characterized in that, A system-on-a-chip motherboard with integrated training and propulsion capabilities for the first robot in a multi-robot service system; the method includes: In response to a user's voice command, the robot acquires a first environmental image and determines the location of the first user; and creates a first broadcast data packet for the first robot based on the first user's location, the robot's local status information, and the first environmental image; and receives second broadcast data packets from other robots to obtain multiple broadcast data packets including the first broadcast data packet. The second broadcast data packet is parsed to obtain the second environmental image collected by the other robots in response to the user's voice command and the local status information of the other robots. The local status information of the robots includes the timestamp of the current processing time window, spatial location, device number, and computing resource utilization rate. The first environmental image and the second environmental image are fused to obtain a multi-machine vision fused image; The robot call detection result is determined based on the user's voice command and the historical dialogue record between the first robot and the first user in the current dialogue cycle. The robot call detection result includes the device number of the specific robot or a preset anomaly value. The preset anomaly value indicates that there is no specific robot in the current dialogue cycle. The robot call detection result, the multi-machine visual fusion image, the first user location, and the local state information of multiple robots corresponding to the multiple broadcast data packets are input into the pre-trained attribution robot determination model to obtain the target robot number output by the attribution robot determination model. When the target robot number is detected to match the device number of the first robot, the system continues to respond to the service request indicated by the user's voice command; wherein... The robot attribution determination model is used to perform the following operations: If the robot call detection result includes the device number of the specific robot, then the device number of the specific robot is determined as the target robot number; If the robot call detection result includes the preset outlier, then the reference position range mapped by the multi-machine visual fusion image is determined, whereby the reference position range characterizes the spatial region that multiple robots can jointly observe in the current environment; and, Determine the spatial occupancy state of the first user's position relative to the reference position range, wherein an empty spatial occupancy state indicates that the first user is outside the framing range of the plurality of robots, and a non-empty spatial occupancy state indicates that the first user is within the framing range of at least one robot; and, If it is determined that the space occupancy status is empty, the target robot number is determined based on the multiple local status information of the multiple robots and the first user's position. If it is determined that the space occupancy status is not empty, then the occupancy area image of the first user in the multi-machine visual fusion image is determined according to the first user's position; and, the presence of a robot is detected within a preset range corresponding to the first user's facial orientation based on the occupancy area image. If a robot is detected, the device number of the detected robot is determined as the target robot number; and, If no robot is detected, the target robot number is determined based on the multiple local status information of the multiple robots and the first user's location.

2. The method according to claim 1, characterized in that, The step of determining the robot call detection result based on the user's voice command and the historical dialogue records between the first robot and the first user in the current dialogue cycle includes: The voice text content is determined based on the user's voice command; The system identifies whether the voice text content contains preset robot call feature words, which include robot device number, robot unique name, and robot location identifier. If so, then the robot call detection result is generated based on the device number of the specific robot pointed to by the robot call feature words; and, If not, query and identify whether the first user's historical dialogue records in the current dialogue cycle contain the robot's call feature words; If so, then the robot call detection result is generated based on the device number of the specific robot pointed to by the robot call feature words; and, If not, the robot call detection result is generated based on the preset outlier value.

3. The method according to claim 1, characterized in that, Determining the spatial occupancy status of the first user's location relative to the reference location range includes: Determine the spatial boundaries of the reference position range in the global coordinate system; Determine whether the first user's location falls within the spatial boundary of the reference location range; If it is determined that the first user's position falls within the spatial boundary of the reference position range, then the spatial occupancy status is determined to be non-empty. If it is determined that the first user's position exceeds the spatial boundary of the reference position range, then the spatial occupancy status is determined to be empty.

4. The method according to claim 1, characterized in that, The multiple local status information of the multiple robots includes multiple spatial locations and multiple computing resource occupancy rates corresponding to each of the multiple robots; determining the target robot number based on the multiple local status information of the multiple robots and the first user location includes: The relative distances between the plurality of spatial locations and the first user's location are determined respectively, thereby obtaining a plurality of relative distances corresponding to the plurality of robots; The relative distance of each of the plurality of robots is scored according to the first preset scoring rule to obtain a plurality of first scores, wherein the score of the first score is negatively correlated with the relative distance. The computing power resource utilization rate of each of the plurality of robots is scored according to the second preset scoring rule to obtain a plurality of second scores. The score of the second score is negatively correlated with the computing power resource utilization rate. The first score and the second score corresponding to each robot are weighted and summed according to the first preset weight and the second preset weight respectively to obtain multiple comprehensive scores; The device number of the robot with the highest overall score among the multiple robots is determined as the target robot number.

5. The method according to any one of claims 1-4, characterized in that, The continued response to the service request indicated by the user's voice command includes: The user's voice command is identified based on the historical dialogue state dataset corresponding to the first user to obtain the target intent. The historical dialogue state dataset includes historical dialogue data, the state of historical events in which the user participated, the state data of items associated with the user, and user profile data representing the user's activity habits. Generate a sequence of subtasks to achieve the target intent based on the historical dialogue state dataset; Execute the subtask sequence.

6. The method according to claim 5, characterized in that, The step of performing intent recognition on the user's voice command based on the historical dialogue state dataset corresponding to the first user to obtain the target intent includes: The initial intent is obtained by performing intent recognition on the voice text content corresponding to the user's voice command; The user's voice commands are subjected to voiceprint recognition to determine the first voiceprint feature set; Based on the first voiceprint feature set, query the pre-stored historical dialogue state database to obtain the historical dialogue state dataset corresponding to the first user. The historical dialogue state database pre-stores multiple voiceprint feature sets and multiple historical dialogue state datasets corresponding to multiple users. The initial intent is adjusted based on the historical dialogue state dataset to obtain the target intent.

7. A multi-robot service system, characterized in that, The system includes multiple robots, including a first robot. Each robot includes a system-on-a-chip motherboard with integrated training and propulsion capabilities. The system-on-a-chip motherboard includes a main controller, a main computing card and an expansion computing card that are communicatively connected to the main controller, and a memory that is communicatively connected to the main controller. The system-on-a-chip motherboard of the first robot with integrated training and propulsion capabilities is used to execute the steps in the method as described in any one of claims 1-6.

Images