Object recognition and processing method and system in robot voice service

Abstract

The application discloses a method and system for object recognition and processing in a robot voice service, comprising: collecting image frames, depth point cloud frames and audio data streams of a current interaction environment; determining N interaction objects in the current interaction environment according to the image frames, and performing instance segmentation on the N interaction objects to obtain a human body mask of each interaction object; determining N first spatial positions of the N interaction objects, and a visual perception preprocessing result based on the depth point cloud frames and the human body mask; determining a second spatial position of each sound source based on the audio data stream, performing nearest neighbor matching between the second spatial position of each sound source and the N first spatial positions to complete binding of the sound source and the human body object, and obtaining a bound interaction object; and generating an active tracking strategy of the robot based on a set of interaction intention prediction results.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method and system for object recognition and processing in robot voice services. Background Technology

[0002] In guided tours in noisy and complex environments with multiple users, the robot needs to accurately separate real interactive speech from non-real interactive speech. Real interactive speech refers to the sound data of real interactive environmental objects that have a genuine intention to have a dialogue with the robot, while non-real interactive speech refers to the sound data generated by environmental objects other than real interactive environmental objects.

[0003] Current conventional methods are based on the robot's microphone module and conventional filtering algorithms for sound data dimensions. However, the actual results are not particularly good. For example, if there are three natural persons in front of the robot, and all three are talking, but the first and second persons are chatting with each other, and only the third person is speaking to the robot, the robot will likely collect and process the voices of all three persons, resulting in low interaction accuracy.

[0004] Furthermore, because the robot's microphone has a relatively narrow pickup angle, such as 30 degrees or 50 degrees, the effective pickup direction range may not adequately cover the location of the third person's voice source, which will also affect the comprehensiveness and accuracy of effective interactive voice acquisition. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method and system for object recognition and processing in robot voice services, which identifies and processes interactive objects in the robot's complex environment, solves the problem of the comprehensiveness and accuracy of the robot's voice processing in complex environments, and improves the robot's ability and accuracy in voice processing in complex environments.

[0006] In a first aspect, embodiments of this application provide a method for object recognition and processing in a robot voice service, including: Collect image frames, depth point cloud frames, and audio data streams of the current interactive environment; Based on the image frames, N interactive objects in the current interactive environment are determined, and instance segmentation is performed on the N interactive objects to obtain the human body mask for each interactive object, where N is a positive integer. Based on the depth point cloud frames and the human body masks, N first spatial positions and visual perception preprocessing results for the N interactive objects are determined. Based on the audio data stream, the second spatial position of each sound source is determined, and nearest neighbor matching is performed between the second spatial position of each sound source and the N first spatial positions to complete the binding of the sound source and the human body object, resulting in a bound interactive object. Speech recognition and semantic feature extraction are performed on the bound interactive objects to obtain preprocessing results for audio information. Based on the visual perception preprocessing results, the preprocessing results for audio information, the interactive object state dictionary from the previous moment, and the historical execution action list, the interactive object state dictionary and key object list from the current moment are obtained. Using the interactive object state dictionary and the key object list, an interactive intent prediction result set is calculated through a pre-trained dialogue intent prediction model. A robot's active tracking strategy is generated based on the set of interaction intent prediction results.

[0007] Secondly, embodiments of this application provide an object recognition and processing system for robot voice services, including: The acquisition unit is used to acquire image frames, depth point cloud frames, and audio data streams of the current interactive environment; The detection and processing unit is configured to: determine N interactive objects in the current interactive environment based on the image frame; perform instance segmentation on the N interactive objects to obtain a human mask for each interactive object, where N is a positive integer; determine N first spatial positions and visual perception preprocessing results for the N interactive objects based on the depth point cloud frame and the human mask; determine the second spatial position of each sound source based on the audio data stream; perform nearest neighbor matching between the second spatial position of each sound source and the N first spatial positions to complete the binding of the sound source and the human object, thus obtaining the bound interactive object; perform speech recognition and semantic feature extraction on the bound interactive object to obtain the preprocessing results of the audio information; obtain the current interactive object state dictionary and key object list based on the visual perception preprocessing results, the preprocessing results of the audio information, the interactive object state dictionary of the previous moment, and the historical execution action list; and use the interactive object state dictionary and the key object list to infer the interactive intent prediction result set through a pre-trained dialogue intent prediction model. The tracking unit is used to generate an active tracking strategy for the robot based on the set of interaction intent prediction results.

[0008] Thirdly, embodiments of this application provide a robot, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing the steps in the first aspect of embodiments of this application.

[0009] Fourthly, embodiments of this application provide an electronic device, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing the steps in the first aspect of embodiments of this application.

[0010] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in the first aspect of embodiments of this application.

[0011] Sixthly, embodiments of this application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of embodiments of this application. The computer program product may be a software installation package.

[0012] As can be seen, compared with the existing robot voice service systems which suffer from insufficient accuracy and real-time performance in recognizing real interactive objects in complex multi-person environments, the robot in this application first collects image frames, depth point cloud frames, and audio data streams of the current interactive environment; then, it determines N interactive objects in the current interactive environment based on the image frames, and performs instance segmentation on the N interactive objects to obtain the human mask of each interactive object, where N is a positive integer; based on the depth point cloud frames and human mask, it determines N first spatial positions of the N interactive objects and visual perception preprocessing results; based on the audio data stream, it determines the second spatial position of each sound source, and compares the second spatial position of each sound source with the N first spatial positions of the N interactive objects. The robot uses a spatial location nearest neighbor matching to bind the sound source to the human object, thus obtaining the bound interaction object. Speech recognition and semantic feature extraction are performed on the bound interaction object to obtain preprocessed audio information. Based on the visual perception preprocessing result, the audio information preprocessing result, the interaction object state dictionary from the previous moment, and the historical action list, the interaction object state dictionary and key object list for the current moment are obtained. Using the interaction object state dictionary and key object list, a pre-trained dialogue intent prediction model is used to infer a set of interaction intent prediction results. Finally, an active tracking strategy for the robot is generated based on the set of interaction intent prediction results. In this way, the robot can determine the object to be interacted with through image frames, then determine the first spatial location of the natural person user through depth point cloud frames, determine the second spatial location of the sound source through audio data streams, and determine the real interaction object through double matching of image-derived location and sound source-derived location. Then, dynamic tracking is performed through audio processing of the real interaction object, thereby improving the accuracy and real-time performance of the robot in multi-person, noisy, and complex environments for recognizing real interaction objects and providing voice services. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram illustrating a scenario of object recognition and processing in a robot voice service provided in an embodiment of this application; Figure 2 This is a schematic diagram of the hardware architecture of a robot provided in an embodiment of this application; Figure 3 A schematic diagram of the software architecture of a robot provided in this application; Figure 4 This is a schematic diagram illustrating the specific process of an object recognition and processing method in a robot voice service provided in an embodiment of this application; Figure 5 A flowchart of an object recognition and processing method in a robot voice service provided in this application; Figure 6 This is a schematic diagram of the structure of an object recognition and processing system in a robot voice service provided in an embodiment of this application; Figure 7 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0016] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0017] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0018] In this application's embodiments, "multiple" refers to two or more. In this application's embodiments, "connection" refers to various connection methods, such as direct or indirect connections, to achieve communication between devices; this application's embodiments do not impose any limitations on this.

[0019] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] The following describes the relevant content, concepts, meanings, technical issues, technical solutions, and beneficial effects involved in the embodiments of this application.

[0021] Object Detection is an open-vocabulary object detection model. Its core is to use natural language descriptions to detect any object in an image, without being limited to a fixed category during training.

[0022] Segment Anything Model (SAM) is a general image segmentation basic model. Its core is to segment any object in an image using simple cues (points / boxes / text), and it has a powerful zero-shot generalization ability.

[0023] Particle-based object representation refers to a technique that uses a large set of basic particles (points / metaspheres / voxels) with independent properties to describe or simulate the geometry, appearance, and dynamic behavior of 3D objects. It does not rely on a fixed-topology mesh and is particularly suitable for representing irregular, dynamic, and ambiguous objects (such as smoke, fluids, hair, and deformable bodies). It is also commonly used for the discretization representation and simulation of models.

[0024] MUSIC (Multiple Signal Classification) is an ultra-high resolution parameter estimation algorithm based on subspace decomposition. Its core is to use the orthogonality between the signal subspace and the noise subspace to estimate signal parameters.

[0025] LLM (Large Language Model) is a large-scale autoregressive language model based on the Transformer architecture and pre-trained using massive amounts of text data.

[0026] DiPac-Online (Differentiable Particles Online) is a real-time model calibration method for manipulating deformable objects. Its core is to update model parameters online through microparticle dynamics simulation and gradient descent, so as to quickly adapt to sudden changes in the physical properties of objects or the environment and narrow the gap between simulation and reality.

[0027] To address the aforementioned issues, this application provides a method and system for object recognition and processing in robot voice services. This method identifies and processes interactive objects in complex robot environments, solving the problems of comprehensiveness and accuracy in robot voice processing in complex environments, and improving the robot's voice processing capabilities and accuracy in complex environments.

[0028] First, please refer to the details. Figure 1 , Figure 1This is a schematic diagram illustrating a scenario of object recognition and processing in a robot voice service provided in an embodiment of this application, such as... Figure 1 As shown, the above scenario includes a robot and multiple users.

[0029] Based on the locational relationships between multiple users and the robot, the environment of voice services can be divided into various scenarios, such as... Figure 1 The ring-shaped scene shown is in the image. Figure 1 The ring shown surrounds the scene (such as) Figure 1 The users A, B, C, and D shown are generally suitable for situations where multiple people interact with the robot and ask questions simultaneously. They are often adapted to technical scenarios with a large number of people and concentrated inquiries, such as shopping malls and airports.

[0030] Of course, the aforementioned multiple users and robots can also form a face-to-face group interaction scenario, where users are present on both the front and back of the robot. These multiple users and robots can also form a C-shaped or L-shaped group interaction scenario, where the robot faces multiple users. The scenario is categorized as L-shaped or C-shaped depending on the angles between the users and the robot. Specifically, if the angle between the lines connecting the multiple users and the robot is a right angle, it is defined as an L-shaped group interaction scenario; if the angle is an acute angle, it is defined as a C-shaped group interaction scenario. In other application scenarios, the robot and multiple users can have various other positional relationships, which will not be elaborated upon here.

[0031] Please refer to details. Figure 2 , Figure 2 This is a schematic diagram of the hardware architecture of a robot provided in an embodiment of this application, such as... Figure 2 The robot shown can be adapted for applications such as Figure 1 The ring shape shown can be applied to other complex environments as well. Figure 2 As shown, the hardware architecture of the aforementioned robot specifically includes: Environmental perception layer 20 specifically includes: The visual perception unit 201 employs an RGB-D depth camera (paired with a 360° panoramic gimbal, supporting environmental scanning from 6 different angles, and a multi-view RGB acquisition scheme) and a facial feature capture infrared camera (frame rate ≥30fps, resolution ≥1920×1080) to acquire RGB images, depth point clouds, facial key points, and eye gaze direction data of interactive objects in the environment; it is also equipped with a Grounding DINO target detection module and a Segment Anything segmentation model to achieve real-time detection, instance segmentation, and spatial localization of human targets in the environment.

[0032] Audio sensing unit 202: It adopts a microphone array with ≥6 channels (supporting digital beamforming on the basis of omnidirectional sound pickup, with an initial horizontal sound pickup angle range of ±30°, which can be dynamically adjusted to ±180° by pan and tilt) and a high-precision audio codec chip, which supports multi-source direction recognition, time-domain / frequency-domain audio data acquisition, sampling rate ≥48kHz, quantization bit depth ≥16bit, and realizes high-fidelity acquisition of omnidirectional speech data and DOA (direction of arrival) estimation of sound sources in the environment.

[0033] Pose perception unit 203: Equipped with robot body IMU inertial measurement unit, wheel odometry and laser SLAM module, it calculates the pose, orientation and spatial position of robot body in world coordinate system in real time, and provides a reference coordinate system for spatial alignment calculation of interactive objects.

[0034] The core computing and decision-making layer 21 uses an embedded industrial computer equipped with an NVIDIA RTX 4090 GPU, an Intel Xeon series CPU, ≥32GB of memory, ≥24GB of video memory, and supports highly parallel computing of the JAX framework (refer to the DaXBench parallel computing architecture). It is used to run multimodal data preprocessing algorithms, dialogue intent prediction models, LLM state estimation modules, and motion planning algorithms, meeting the computing power requirements of real-time inference on the edge side, with a single-round full-process inference cycle of ≤200ms.

[0035] Execution and feedback layer 22, specifically includes: The 221 gimbal execution unit is a 2-DOF high-precision servo gimbal equipped with a microphone array and a vision camera. It has a repeatability accuracy of ≤0.1° and a response time of ≤100ms. It is used to dynamically adjust the camera shooting angle and the main lobe direction of the microphone array's pickup beam according to decision commands.

[0036] Voice interaction execution unit 222: high-fidelity speaker and end-side speech synthesis module, used to realize voice dialogue interaction with real interaction objects and synchronously collect historical interaction data during the dialogue process.

[0037] Safety feedback unit 223: emergency stop module and collision detection sensor, to ensure the robot's motion safety and environmental adaptability during active adjustment.

[0038] For example Figure 2 The robot hardware architecture shown also needs to be adapted to the corresponding software architecture, such as... Figure 3 As shown, Figure 3 This application provides a schematic diagram of the software architecture of a robot, such as... Figure 3 As shown, the software architecture of the aforementioned robot specifically includes: The hardware driver and adaptation layer 30 is responsible for driver adaptation, data acquisition, and command issuance of the underlying hardware, including camera driver, microphone array driver, gimbal servo driver, IMU and odometer driver. It realizes real-time acquisition and timestamp synchronization of raw image data, audio data, and robot pose data. At the same time, it converts the upper-layer motion control commands into hardware-executable electrical signals to ensure real-time data interaction between the hardware and software layers, with a time synchronization error of ≤10ms.

[0039] Multimodal data preprocessing layer 31 specifically includes: It is divided into a visual preprocessing submodule and an audio preprocessing submodule, corresponding to the preprocessing of visual / audio information in this application. The visual preprocessing submodule implements human detection, instance segmentation, facial key point extraction, 3D spatial coordinate reconstruction, and facial orientation and gaze direction calculation; the audio preprocessing submodule implements sound source DOA estimation, speech activity detection (VAD), noise reduction filtering, speech recognition (ASR), semantic extraction, and historical dialogue context association.

[0040] Interactive object state representation and management layer 32: Applying the LLM-State open world state representation method, a dual-track state representation system of structured object entries + unstructured review summary is constructed. A dynamically expandable state dictionary is built for each detected environmental interactive object, and the spatial attributes, visual features, speech and semantic features, historical interaction records and intent prediction results of the object are continuously updated to achieve full life cycle state tracking of multiple interactive objects in the open world.

[0041] The multimodal fusion and intent reasoning layer 33 is based on a pre-trained dialogue intent prediction model and a corresponding LLM as State Estimator module. This layer performs cross-modal fusion of preprocessed visual features, audio semantic features, and object state data, inputs them into the dialogue intent prediction model, and outputs the true dialogue intent probability prediction result for each interactive object. At the same time, through unstructured review summarization, it performs causal reasoning and feature supplementation for cases of interaction failure and intent misjudgment, thereby optimizing the model's inference accuracy.

[0042] The decision-making and motion planning layer 34 is built on the LLM as Policy module. Its core functions include two parts: first, generating processing strategies for target speech filtering and non-interactive speech masking based on the intent prediction results; second, generating beamforming parameters for the microphone array and motion control commands for the pan-tilt unit based on the sound source location and spatial pose of the real interactive object, so as to realize active optimization and dynamic tracking of the sound pickup direction.

[0043] The closed-loop feedback and online optimization layer 35 applies the DiPac online model calibration method to continuously update the state dictionary of the interactive object based on the success / failure results of the voice interaction and the state changes of the interactive object. At the same time, it performs online fine-tuning and parameter calibration of the dialogue intent prediction model. Based on the feedback of the sound pickup effect, it optimizes the parameters of the microphone beamforming algorithm, forming a fully closed-loop software process of "perception-reasoning-execution-feedback-optimization".

[0044] In one possible embodiment, regarding the voice service, the robot specifically performs the following: Figure 4 Chinese method, Figure 4 This is a schematic diagram illustrating the specific process of an object recognition and processing method in a robot voice service provided in this application embodiment, such as... Figure 4 As shown, the robot performs the following steps: Step S401: Acquisition and time synchronization of multimodal raw data.

[0045] Input: Target instructions for the robot's guided tour task The raw data collected by each hardware component in the environmental perception layer, and the robot's pose data; specific operations: Hardware-triggered synchronization: Synchronization is achieved through the time synchronization mechanism of the robot's operating system ROS, synchronizing the RGB-D image frames acquired by the vision camera. Depth point cloud frames Multi-channel audio data stream acquired by microphone array Robot pose data output by IMU Unified allocation of global timestamps The time synchronization error is ≤10ms.

[0046] Global coordinate system definition: Establish the robot's base coordinate system with the center of the robot's face as the origin. The X-axis points directly in front of the robot, the Y-axis points to the left of the robot, and the Z-axis is perpendicular to the ground and pointing upwards (i.e., the height direction); a world coordinate system is established simultaneously using the SLAM module. And through homogeneous transformation matrix accomplish and The coordinate transformation formula is as follows: .

[0047] Explanation of formula parameters: : A three-dimensional coordinate vector in the world coordinate system, with dimensions of 4×1 (homogeneous coordinates); : A three-dimensional coordinate vector in the robot's base coordinate system, with a dimension of 4×1 (homogeneous coordinates); : The 4×4 homogeneous transformation matrix from the world coordinate system to the robot base coordinate system, including the rotation matrix and translation vector, is calculated and output in real time by the robot SLAM module and IMU.

[0048] in, .

[0049] Rotation matrix, describing attitude transformation; p =[x,y,z] T Translation vector, describing position; 0 T =[0,0,0]: Zero vector; The bottom right corner is always 1.

[0050] Raw data caching: Synchronized image, audio, and pose data are stored in a circular buffer with timestamps. The buffer duration is ≥10 seconds, providing data support for subsequent historical data backtracking and contextual reasoning.

[0051] Output: Time-synchronized original multimodal dataset Robot base coordinate system Global timestamp baseline .

[0052] Step S402: Visual perception information preprocessing and spatial interaction feature calculation.

[0053] Input: Output of step S401 , , Robot base coordinate system ; Specific operations: Environmental Human Target Detection and Instance Segmentation: The Grounding DINO target detection model is used for... Inference is performed to detect the bounding boxes and class confidence scores of all human targets in the environment. Human targets with a confidence score ≥ 0.7 are then selected, resulting in... An interactive object, denoted as Then, using the Segment Anything model, each Perform instance segmentation to obtain the human body mask. , .

[0054] 3D point cloud reconstruction of interactive objects: combining depth point clouds With human body mask Extract each interaction object 3D point cloud Each particle For object Upper The three-dimensional coordinates (coordinates in the robot's base coordinate system) of each key point. The total number of point cloud particles; using a particle-based object representation method, high-density particle sampling is performed on the human face, eyeballs, and torso regions (≥68 particles in the face region and ≥16 particles in the eyeball region).

[0055] Calculation of facial orientation and gaze direction: (1) Based on 68 key point particles on the face, principal component analysis (PCA) is used to solve the object. Facial normal vector (i.e., the facial orientation vector), the formula is as follows: .

[0056] Explanation of formula parameters: : object The centroid coordinates of facial key point particles. ; : Facial normal vector, i.e., object The unit vector representing the face orientation has a dimension of 3×1; : The i-th person, the j-th 68 facial key points (3D vectors).

[0057] (2) Based on 16 key point particles in the eye region, the solution object eye gaze direction vector And calculate the spatial deviation between the gaze point and the center of the robot's face; where, .

[0058] Explanation of formula parameters: : object The particle coordinates of the j-th 3D keypoint in the eye region; Centroids (mean points) of 16 key points on the eyeball; : The projection of the key point in the gaze direction.

[0059] (3) Calculate the core parameter of spatial interaction willingness: facial orientation alignment Alignment of gaze direction The formula is as follows: ; .

[0060] Explanation of formula parameters: : The unit vector of the X-axis in the robot's base coordinate system, i.e., the direction the robot is facing; : object The coordinates of the intersection of the line of sight with the plane of the robot's face; : Coordinates of the center of the robot's face (origin in the robot's base coordinate system); Gaussian kernel bandwidth, set to a fixed value of 0.15m, is used to control the sensitive range of the gaze direction; : Facial orientation alignment, with a value range of [0,1]. The larger the value, the more directly the object's face is facing the robot. : Gaze direction alignment, with a value range of [0,1]. The larger the value, the more focused the object's gaze is on the robot's face.

[0061] Visual perception preprocessing result encapsulation: for each interactive object Generate visual feature vectors ,in This represents the spatial location of the object, used for subsequent sound source orientation matching.

[0062] Output: A collection of interactive objects in the environment Visual feature vectors of each object Visual perception preprocessing results .

[0063] Step S403: Preprocessing of speech reception information and extraction of semantic interaction features.

[0064] Input: Multi-channel audio data stream output from step S401 Global timestamp The set of interactive objects output in step S402 Spatial location ; Specific operations: Sound source localization and speech activity detection: (1) Based on the phase difference of the multi-channel microphone array, the DOA of the sound source is estimated by the MUSIC algorithm, and the DOA of the sound source in the environment is calculated. Azimuth angle of an effective sound source Pitch angle and the spatial location of each sound source. (in robot base coordinate system) .

[0065] (2) By using energy threshold-based speech activity detection (VAD), endpoint detection is performed on the audio data of each sound source to filter out effective speech segments. Remove silent and noisy segments.

[0066] Sound source-object space matching: Based on the spatial location of the interactive object obtained in step S402 By using nearest neighbor matching, each sound source is... Match the corresponding interaction object The matching rules are as follows: .

[0067] Definitions of each parameter: : The final matched target object number; : The coordinates of the m-th sound source (obtained using the MUSIC (Multiple Signal Classification) algorithm); : The coordinates of the center of the face of the i-th person.

[0068] When the matching distance is ≤0.3m (of course, different distances can be configured depending on the scenario, such as 0.2m, 0.4m, etc.), the sound source is determined. From the interaction object The system completes the one-to-one binding of the sound source and the human body object; if the matching distance is >0.3m, it is determined to be an environmental noise source and is directly eliminated.

[0069] Speech recognition and semantic feature extraction: (1) Binding to the interaction object Valid speech segments Converted into text content using an Automatic Speech Recognition (ASR) model. That is, basic semantics.

[0070] (2) Based on the contextual reasoning capability of LLM, introduce objects Historical dialogue records Through LLM Context optimization, referential resolution, and ambiguity correction are performed to obtain the target semantic text. .

[0071] (3) Using a pre-trained text encoder (BERT), Encoded as a fixed-dimensional semantic feature vector The dimensions are 768×1.

[0072] Semantic interaction intention calculation: Calculate the semantic relevance between the target semantic text and the robot's interaction commands. The formula is as follows: .

[0073] Explanation of formula parameters: : The feature vector of the instruction text encoding for the robot's current navigation task, with dimensions and Consistent; Cosine similarity calculation function; : Semantic relevance, with a value range of [-1, 1]. The larger the value, the higher the relevance between the speech content and the robot task, and the stronger the semantic interaction intention.

[0074] Encapsulation of preprocessed radio information results: for each interactive object Generate audio feature vectors .

[0075] Output: Audio feature vector for each interactive object Preprocessing results of radio reception information Valid speech segment set .

[0076] Step S404: Construction and update of interactive object state based on LLM-State.

[0077] Input: Output of step S402 The output of step S403 The dictionary of the interaction objects from the previous moment List of historical actions .

[0078] Specific operations: This step fully utilizes the three core modules of LLM (LLM as Attention, LLM as StateEstimator) to achieve dynamic state tracking of interactive objects in an open world, specifically divided into two sub-steps: LLM as Attention: Filtering and expanding key interaction objects; With current task instructions The currently observed set of interactive objects As input, LLM identifies key objects relevant to the task and updates the list of key objects. The updated formula is: .

[0079] Explanation of formula parameters: The LLM as Attention module is used to filter interaction objects that are relevant to the current voice interaction task. The list of key objects from the previous moment uses a recursive mechanism of "adding only and not subtracting" to ensure that tracked objects are not lost. The current list of key objects is used to update the state and infer intent only for objects in the list, reducing the computational overhead of irrelevant objects and avoiding exceeding the LLM context length limit.

[0080] LLM as State Estimator: Dual-track state representation update; With a list of key objects Historical action list Current observations and As input, the LLM-State of each key object is constructed and updated using LLM. The state representation is divided into two parts: structured object entries and unstructured review summary entries, which are fully aligned with the state representation system. (1) Updating structured object entries: for each key object Construct a dynamically expandable attribute dictionary, and generate attribute update instructions using LLM. The update formula is as follows: .

[0081] Explanation of formula parameters: : The object at the current moment The structured state has an object ID as the key and a list of attributes as the value, which includes three main categories: spatial attributes, semantic attributes, and historical interaction attributes. It supports dynamic addition and modification of attributes. The attribute update function defined in LLM is used to update the object's state dictionary with features obtained from visual and audio preprocessing.

[0082] (2) Unstructured review summary item update: Through chain thinking (CoT), historical actions, interaction results, and feature changes are summarized step by step to generate summary text. The update formula is: .

[0083] Explanation of formula parameters: The LLM chain reasoning process focuses on analyzing the reasons for misjudgment of intent, speech recognition failure, and abnormal action execution in historical interactions. The LLM-defined summary generation function outputs unstructured review summary text. This provides contextual support for subsequent intent prediction and solves the problem of error accumulation in long-term interactions.

[0084] Final state representation generation: Integrating structured object entries with unstructured review summaries yields the global state representation at the current moment. .

[0085] Output: The global LLM-State representation at the current moment. Key object list .

[0086] Step S405: Dialogue intent prediction model inference and real interaction object identification.

[0087] Input: The global state representation output from step S404 Key object list Pre-trained dialogue intent prediction model ; Specific operations: Cross-modal feature fusion: for each key object The visual and audio features in its structured state are concatenated and normalized to obtain a fused feature vector. The formula is as follows: .

[0088] Explanation of formula parameters: Feature concatenation function; Layer normalization function: used to unify the numerical distribution of features in different dimensions and improve the stability of model inference.

[0089] Intent probability prediction: fusing features of each object Unstructured summary of global state Input dialogue intent prediction model The model outputs the probability that the object has a genuine intention to interact. The formula is as follows:

[0090] Model A three-layer fully connected neural network is constructed with ReLU activation function and Sigmoid function in the output layer. The core loss function is Chamfer-α loss function to address the imbalance between positive and negative samples. The loss function formula is as follows: .

[0091] Explanation of formula parameters: Positive sample feature set, that is, the set of sample fusion features that contain genuine interaction intentions; The predicted feature set output by the model; Points in the prediction point set; The points where real points are concentrated; Deviation factor, with a value range of [0,1]. In this scheme, a fixed value of 0.5 is used to balance coverage loss and deviation loss. Chamfer-α loss value, used for offline training and online fine-tuning of the model.

[0092] Determining the real interaction object: Setting an intent probability threshold ,when When determining the object For the actual interactive object, denoted as ;when When determining the object It is a non-interactive object.

[0093] Output encapsulation: Generates the interaction intent prediction results for each interactive object. The labels are "real interactive objects / non-interactive objects".

[0094] Output: Set of interactive intent prediction results Real interactive objects Non-interactive object collection .

[0095] Step S406: Non-interactive speech masking and target speech enhancement processing.

[0096] Input: Output of step S405 and The set of valid speech segments output in step S403 Raw audio data stream .

[0097] Specific operations: Target speech filtering: Extracting real interaction objects Corresponding valid speech segments As the sole input for subsequent voice interaction processing, it is fed into the robot dialogue system and the voice understanding module.

[0098] Non-interactive voice masking: for a collection of non-interactive objects For all corresponding sound sources, a spatial notch filter is generated in the frequency domain using a microphone array beamforming algorithm. This filter adaptively reduces noise and masks the directions of non-interactive sound sources. The masking formula is as follows: .

[0099] Explanation of formula parameters: Spatial filtering weights for microphone arrays; : The azimuth angle of the sound source of the actual interactive object; : Azimuth angle of the sound source for all non-interactive objects.

[0100] The filtering weights are applied to multi-channel audio data to achieve spatial shielding of non-interactive speech while enhancing the speech signal in the target direction.

[0101] Target speech enhancement: Secondary noise reduction is performed using the Wiener filtering algorithm to improve the speech signal-to-noise ratio, ultimately yielding the enhanced target speech. .

[0102] Output: Enhanced target speech Audio data stream after non-interactive voice masking, and the execution results of voice processing strategies.

[0103] Step S407: Actively optimize microphone pickup direction based on intent prediction results.

[0104] Input: The actual interactive object output from step S405 The output of step S402 Spatial location The output of step S403 Sound source location The robot's current gimbal pose; Specific operations: This step is based on the LLM as Policy module and uses trajectory tree optimization and online adjustment methods to achieve closed-loop tracking of the microphone pickup direction. The specific operations are as follows: Target sound pickup direction calculation: based on real interactive objects spatial location Calculate the horizontal azimuth angle that the microphone array needs to be adjusted. With pitch angle The formula is as follows: ; .

[0105] Explanation of formula parameters: Target horizontal azimuth angle, in rad, with a range of [-π, π]. Target pitch angle, in rad, with a value range of [-π / 2, π / 2].

[0106] Gimbal motion planning and beamforming parameter generation: (1) Using the LLM as Policy module, based on the current gimbal pose and the target pose It generates motion control commands for the pan-tilt unit, controls the pan-tilt unit to rotate the microphone array, and aligns the main pickup lobe with the target sound source, with a control accuracy of ≤0.5°.

[0107] (2) The beamforming parameters of the microphone array are updated synchronously, the main lobe width is set to ±15°, the sound source position of the real interactive object is focused, and the main lobe gain is increased by 3dB to further enhance the acquisition effect of the target speech.

[0108] Active tracking strategy: When the spatial position of the real interactive object changes, repeat the above calculation and planning process to achieve real-time dynamic tracking of the sound pickup direction to the real interactive object, ensuring that the effective sound pickup range always covers the target sound source position.

[0109] Output: Gimbal motion control commands, microphone array beamforming parameters, and sound pickup direction optimization execution results.

[0110] Step S408: Closed-loop state update and online model calibration.

[0111] Inputs: Speech processing results of this round of interaction, sound pickup optimization execution results, success / failure labels of the dialogue interaction, and the current state output from step S404. Historical action list ; Specific operations: Historical Action List Update: Add the voice processing actions, gimbal movement actions, and action execution results (success / failure) executed in this round to the historical action list. This provides data support for the next round of status updates.

[0112] Online model calibration: Based on the DiPac online model calibration method, the error loss of the model prediction is calculated according to the success / failure result of the current round of dialogue interaction, and the dialogue intent prediction model is calibrated. Perform online fine-tuning and update model parameters. The updated formula is as follows: .

[0113] Explanation of formula parameters: The learning rate is fine-tuned online, taking a fixed value of 1e-5; The task loss in this round of interaction, when the interaction is successful. When interaction fails ; The gradient of the loss function with respect to the model parameters.

[0114] Next round of state initialization: Initialize the state after this round of updates. Historical action list The fine-tuned model parameters serve as the initial input for the next round of algorithm execution, completing the entire closed loop of "perception-reasoning-execution-feedback-optimization".

[0115] Output: Updated list of historical actions calibrated dialogue intent prediction model The initial state parameters for the next round of the algorithm.

[0116] The technical solution provided in this application involves acquiring image frames. Depth point cloud frames Audio data stream Pose data This is done by determining the object to be interacted with through image frames, and then through depth point cloud frames. To determine spatial location, through audio data streams The location of the sound source is determined by matching the video location with the sound source location to identify the real interactive object. Then, the audio of the real interactive object is processed for dynamic tracking, thereby improving the accuracy and real-time performance of the robot in recognizing real interactive objects and providing voice services in multi-person, noisy and complex environments.

[0117] See Figure 5 , Figure 5 A flowchart of an object recognition and processing method in a robot voice service provided in this application includes: Step S501: Acquire image frames, depth point cloud frames, and audio data streams of the current interactive environment; Step S502: Determine N interactive objects in the current interactive environment based on the image frame, and perform instance segmentation on the N interactive objects to obtain the human body mask for each interactive object, where N is a positive integer; determine N first spatial positions and visual perception preprocessing results for the N interactive objects based on the depth point cloud frame and the human body mask; determine the second spatial position of each sound source based on the audio data stream, and perform nearest neighbor matching between the second spatial position of each sound source and the N first spatial positions to complete the binding of the sound source and the human body object, thus obtaining the bound interactive object; perform speech recognition and semantic feature extraction on the bound interactive object to obtain the preprocessing results of the audio information; based on the visual perception preprocessing results, the preprocessing results of the audio information, the interactive object state dictionary of the previous moment, and the historical execution action list, obtain the interactive object state dictionary and the key object list of the current moment; use the interactive object state dictionary and the key object list to infer the interactive intent prediction result set through a pre-trained dialogue intent prediction model; Step S503: Generate the robot's active tracking strategy based on the set of interaction intent prediction results.

[0118] In contrast to existing robot voice service systems that suffer from insufficient accuracy and real-time performance in recognizing real interactive objects in complex multi-person environments, this application's robot first acquires image frames, depth point cloud frames, and audio data streams of the current interactive environment. Then, based on the image frames, it determines N interactive objects in the current environment and performs instance segmentation on these N objects to obtain a human mask for each object, where N is a positive integer. Based on the depth point cloud frames and the human mask, it determines N first spatial positions of the N interactive objects and visual perception preprocessing results. Based on the audio data stream, it determines the second spatial position of each sound source and connects the second spatial position of each sound source with the N first spatial positions of the N interactive objects. The robot uses a spatial location nearest neighbor matching to bind the sound source to the human object, thus obtaining the bound interaction object. Speech recognition and semantic feature extraction are performed on the bound interaction object to obtain preprocessed audio information. Based on the visual perception preprocessing result, the preprocessed audio information result, the interaction object state dictionary from the previous moment, and the historical action list, the interaction object state dictionary and key object list for the current moment are obtained. Using the interaction object state dictionary and key object list, a pre-trained dialogue intent prediction model is used to infer the interaction intent prediction result set. Finally, the robot's active tracking strategy is generated based on the interaction intent prediction result set. In this way, the robot can determine the object to be interacted with through image frames, then determine the first spatial location of the natural person user through depth point cloud frames, determine the second spatial location of the sound source through audio data streams, and determine the real interaction object through double matching of image-derived location and sound source-derived location. Then, dynamic tracking is performed through audio processing of the real interaction object, thereby improving the accuracy and real-time performance of the robot in multi-person, noisy, and complex environments for recognizing real interaction objects and providing voice services.

[0119] In one optional scheme, the determination of N first spatial locations of N interactive objects based on the depth point cloud frame and the human body mask, and the visual perception preprocessing results specifically include: The depth point cloud frame is sampled to obtain facial key point particles and eye key point particles. The facial normal vector is obtained based on the facial key point particles, and the eye gaze direction vector is obtained based on the eye key point particles. The facial orientation alignment and gaze direction alignment are calculated based on the facial normal vector and the eye gaze direction vector. Based on the facial orientation alignment, gaze direction alignment, N first spatial positions, facial normal vector, and eye gaze direction vector, visual feature vectors and visual perception preprocessing results are generated.

[0120] In one optional scheme, the step of determining the second spatial position of each sound source based on the audio data stream, and performing nearest neighbor matching between the second spatial position of each sound source and N first spatial positions to complete the binding of the sound source and the human body object, resulting in the bound interactive object, specifically includes: Based on the audio data stream and the phase difference of the multi-channel microphone array, the azimuth angle, elevation angle and second spatial position of each of the m effective sound sources are estimated and calculated. Calculate the matching distance between the second spatial position of each sound source and the first spatial position of the interactive object. Match the interactive object whose matching distance is lower than the distance threshold with the spatial position of the sound source to obtain the bound interactive object.

[0121] In one optional approach, the preprocessing of the audio information obtained by performing speech recognition and semantic feature extraction on the bound interactive object specifically includes: Energy threshold-based speech activity detection performs endpoint detection on the audio data of each sound source to filter out valid speech segments; valid speech segments bound to interactive objects are converted into text content through an automatic speech recognition model. The text content is optimized for context, resolved for pronouns, and corrected for ambiguity using a large language model to obtain the target semantic text. The target semantic text is then encoded into a semantic feature vector of a preset dimension using a text encoder. The semantic relevance is obtained by calculating the similarity between the instruction text encoded feature vector of the robot's current navigation task and the semantic feature vector. For each interactive object, the semantic relevance, the second spatial location of the sound source, the semantic feature vector, and the target semantic text are used to obtain the audio feature vector. The preprocessed result is obtained from the set of audio feature vectors of N interactive objects.

[0122] In one optional scheme, obtaining the current interactive object state dictionary and key object list based on the visual perception preprocessing result, the audio information preprocessing result, the interactive object state dictionary of the previous moment, and the historical execution action list specifically includes: The key object list is obtained by identifying interactive objects related to the current task instructions through a large language model and updating the key object list from the previous moment. Based on the list of key objects, the list of historical actions, the current observation visual perception preprocessing results, and the preprocessing results, the state of the large language model for each key object is constructed and updated through the large language model, and the interaction object state dictionary of the previous moment is updated to obtain the interaction object state dictionary of the current moment.

[0123] In one optional approach, the step of using the interaction object state dictionary and the key object list to infer the interaction intent prediction result set through a pre-trained dialogue intent prediction model specifically includes: For each key object, the visual feature vector and audio feature vector in its structured state are concatenated and normalized to obtain a fused feature vector; The fused features and unstructured summary of the global state of each object are input into the pre-trained dialogue intent prediction model to calculate the probability that the object has a real interaction intent. When the probability is greater than or equal to the probability threshold, it is marked as a real interaction object; when the probability is less than the probability threshold, it is marked as a non-interaction object. The probabilities of the real interaction intent of all key objects are collected to obtain the interaction intent prediction result set.

[0124] In an alternative embodiment, the method further includes: Extract the set of valid speech segments corresponding to the real interactive objects, and perform a filtering algorithm to reduce noise to obtain the enhanced target speech.

[0125] In an alternative embodiment, the method further includes: Extract the spatial location of the real interactive object from N first spatial locations, and extract the sound source location of the real interactive object from the azimuth angles of m effective sound sources; Based on the spatial location, sound source orientation, pose, and target pose of the real interactive object, robot control commands are generated to make the microphone array align with the sound source orientation.

[0126] In an alternative embodiment, the method further includes: Obtain the current dialogue execution result of the interactive object, and perform training calibration on the pre-trained dialogue intent prediction model based on the execution result.

[0127] Figure 6 This is a schematic diagram of the structure of an object recognition and processing system in a robot voice service provided in an embodiment of this application, including: The acquisition unit 601 is used to acquire image frames, depth point cloud frames, and audio data streams of the current interactive environment; The detection and processing unit 602 is configured to: determine N interactive objects in the current interactive environment based on the image frame; perform instance segmentation on the N interactive objects to obtain a human mask for each interactive object, where N is a positive integer; determine N first spatial positions and visual perception preprocessing results for the N interactive objects based on the depth point cloud frame and the human mask; determine the second spatial position of each sound source based on the audio data stream; perform nearest neighbor matching between the second spatial position of each sound source and the N first spatial positions to complete the binding of the sound source and the human object, thus obtaining the bound interactive object; perform speech recognition and semantic feature extraction on the bound interactive object to obtain the preprocessing results of the audio information; obtain the current interactive object state dictionary and key object list based on the visual perception preprocessing results, the preprocessing results of the audio information, the interactive object state dictionary of the previous moment, and the historical execution action list; and use the interactive object state dictionary and the key object list to infer the interactive intent prediction result set through a pre-trained dialogue intent prediction model. The tracking unit 603 is used to generate an active tracking strategy for the robot based on the set of interaction intent prediction results.

[0128] In some alternative implementations, the detection processing unit 602 described above can achieve the following: Figure 4 or Figure 5 The detailed schemes in the method embodiments shown will not be elaborated here.

[0129] Figure 7 This is a structural block diagram of an electronic device provided in an embodiment of this application. For example... Figure 7 As shown, the electronic device 700 may include one or more of the following components: a processor 701, and a memory 702 coupled to the processor 701, wherein the memory 702 may store one or more computer programs, which may be configured to implement the methods described in the examples above when executed by one or more processors 701. Figure 7 The electronic device 700 shown can be a robot.

[0130] Processor 701 may include one or more processing cores. Processor 701 connects to various parts within the electronic device 700 using various interfaces and lines, and performs various functions and processes data of the electronic device 700 by running or executing instructions, programs, code sets, or instruction sets stored in memory 702, and by calling data stored in memory 702. Optionally, processor 701 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 701 may integrate one or more of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. It is understood that the aforementioned modem may also not be integrated into processor 701, but may be implemented separately through a communication chip.

[0131] The memory 702 may include random access memory (RAM) or read-only memory (ROM). The memory 702 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 702 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method examples described above. The data storage area may also store data created during the use of the electronic device 700.

[0132] It is understood that the electronic device 700 may include more or fewer structural elements than those shown in the above block diagram, such as a power module, physical buttons, WiFi (Wireless Fidelity) module, speaker, Bluetooth module, sensor, etc., without limitation.

[0133] This application also provides a computer storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements some or all of the steps of any of the methods described in the above method embodiments.

[0134] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments.

[0135] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0137] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0139] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute partial steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, volatile memory, or non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM), etc., which are various media capable of storing program code.

[0140] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can easily conceive of variations or substitutions without departing from the spirit and scope of the present invention, and various modifications and alterations can be made, including combinations of the different functions and implementation steps described above, as well as software and hardware implementation methods, all of which are within the protection scope of the present invention.

Claims

1. A method for object recognition and processing in robot voice services, characterized in that, include: Collect image frames, depth point cloud frames, and audio data streams of the current interactive environment; Based on the image frame, determine N interactive objects in the current interactive environment, and perform instance segmentation on the N interactive objects to obtain the human body mask for each interactive object, where N is a positive integer; Based on the depth point cloud frame and the human body mask, determine the N first spatial positions of N interactive objects and the visual perception preprocessing results; Based on the audio data stream, the second spatial position of each sound source is determined. The second spatial position of each sound source is then matched with the nearest neighbor of N first spatial positions to complete the binding of the sound source and the human body object, thus obtaining the bound interactive object. Speech recognition and semantic feature extraction are then performed on the bound interactive object to obtain the preprocessing result of the audio information. Based on the visual perception preprocessing results, the audio information preprocessing results, the interaction object state dictionary of the previous moment, and the historical execution action list, the interaction object state dictionary and the key object list of the current moment are obtained. Using the interaction object state dictionary and the key object list, a set of interaction intent prediction results is obtained by inferring through a pre-trained dialogue intent prediction model; A robot's active tracking strategy is generated based on the set of interaction intent prediction results.

2. The object recognition and processing method in robot voice service according to claim 1, characterized in that, The determination of N first spatial locations of N interactive objects based on the depth point cloud frame and the human body mask, and the visual perception preprocessing results specifically include: The depth point cloud frame is sampled to obtain facial key point particles and eye key point particles. The facial normal vector is obtained based on the facial key point particles, and the eye gaze direction vector is obtained based on the eye key point particles. The facial orientation alignment and gaze direction alignment are calculated based on the facial normal vector and the eye gaze direction vector. Based on the facial orientation alignment, gaze direction alignment, N first spatial positions, facial normal vector, and eye gaze direction vector, visual feature vectors and visual perception preprocessing results are generated.

3. The object recognition and processing method in robot voice service according to claim 1, characterized in that, The process of determining the second spatial position of each sound source based on the audio data stream, and performing nearest neighbor matching between the second spatial position of each sound source and N first spatial positions to complete the binding of the sound source and the human body object, resulting in the bound interactive object, specifically includes: Based on the audio data stream and the phase difference of the multi-channel microphone array, the azimuth angle, elevation angle and second spatial position of each of the m effective sound sources are estimated and calculated. Calculate the matching distance between the second spatial position of each sound source and the first spatial position of the interactive object. Match the interactive object whose matching distance is lower than the distance threshold with the spatial position of the sound source to obtain the bound interactive object.

4. The object recognition and processing method in robot voice service according to claim 3, characterized in that, The specific steps of performing speech recognition and semantic feature extraction on the bound interactive object to obtain the preprocessing result of the audio information include: Energy threshold-based speech activity detection performs endpoint detection on the audio data of each sound source to filter out valid speech segments; valid speech segments bound to interactive objects are converted into text content through an automatic speech recognition model. The text content is optimized for context, resolved for pronouns, and corrected for ambiguity using a large language model to obtain the target semantic text. The target semantic text is then encoded into a semantic feature vector of a preset dimension using a text encoder. The semantic relevance is obtained by calculating the similarity between the instruction text encoded feature vector of the robot's current navigation task and the semantic feature vector. For each interactive object, the semantic relevance, the second spatial location of the sound source, the semantic feature vector, and the target semantic text are used to obtain the audio feature vector. The preprocessed result is obtained from the set of audio feature vectors of N interactive objects.

5. The object recognition and processing method in robot voice service according to claim 4, characterized in that, The process of obtaining the current interaction object state dictionary and key object list based on visual perception preprocessing results, audio information preprocessing results, the interaction object state dictionary from the previous moment, and the historical execution action list specifically includes: The key object list is obtained by identifying interactive objects related to the current task instructions through a large language model and updating the key object list from the previous moment. Based on the list of key objects, the list of historical actions, the current observation visual perception preprocessing results, and the preprocessing results, the state of the large language model for each key object is constructed and updated through the large language model, and the interaction object state dictionary of the previous moment is updated to obtain the interaction object state dictionary of the current moment.

6. The object recognition and processing method in robot voice service according to claim 5, characterized in that, The step of using the interaction object state dictionary and the key object list to infer the interaction intent prediction result set through a pre-trained dialogue intent prediction model specifically includes: For each key object, the visual feature vector and audio feature vector in its structured state are concatenated and normalized to obtain a fused feature vector; The fused features and unstructured summary of the global state of each object are input into the pre-trained dialogue intent prediction model to calculate the probability that the object has a real interaction intent. When the probability is greater than or equal to the probability threshold, it is marked as a real interaction object; when the probability is less than the probability threshold, it is marked as a non-interaction object. The probabilities of the real interaction intent of all key objects are collected to obtain the interaction intent prediction result set.

7. The object recognition and processing method in robot voice service according to claim 4, characterized in that, The method further includes: Extract the set of valid speech segments corresponding to the real interactive objects, and perform a filtering algorithm to reduce noise to obtain the enhanced target speech.

8. The object recognition and processing method in robot voice service according to claim 6, characterized in that, The method further includes: Extract the spatial location of the real interactive object from N first spatial locations, and extract the sound source location of the real interactive object from the azimuth angles of m effective sound sources; Based on the spatial location, sound source orientation, pose, and target pose of the real interactive object, robot control commands are generated to make the microphone array align with the sound source orientation.

9. The object recognition and processing method in robot voice service according to claim 7, characterized in that, The method further includes: Obtain the current dialogue execution result of the interactive object, and perform training calibration on the pre-trained dialogue intent prediction model based on the execution result.

10. An object recognition and processing system for robot voice services, characterized in that, include: The acquisition unit is used to acquire image frames, depth point cloud frames, and audio data streams of the current interactive environment; The detection and processing unit is used to determine N interactive objects in the current interactive environment based on the image frame, and to perform instance segmentation on the N interactive objects to obtain the human body mask of each interactive object, where N is a positive integer; Based on the depth point cloud frame and the human body mask, determine the N first spatial positions of N interactive objects and the visual perception preprocessing results; Based on the audio data stream, the second spatial position of each sound source is determined. The second spatial position of each sound source is then matched with the nearest neighbor of N first spatial positions to complete the binding of the sound source and the human body object, thus obtaining the bound interactive object. Speech recognition and semantic feature extraction are then performed on the bound interactive object to obtain the preprocessing result of the audio information. Based on the visual perception preprocessing results, the audio information preprocessing results, the interaction object state dictionary of the previous moment, and the historical execution action list, the interaction object state dictionary and the key object list of the current moment are obtained. Using the interaction object state dictionary and the key object list, a set of interaction intent prediction results is obtained by inferring through a pre-trained dialogue intent prediction model; The tracking unit is used to generate an active tracking strategy for the robot based on the set of interaction intent prediction results.

Images