Intelligent interaction method and system of cloud-based collaborative desktop robot

Abstract

The application discloses an intelligent interaction method and system of a desktop robot based on cloud cooperation, and relates to the technical field of cloud cooperation. The desktop robot collects multi-modal interaction data under an enterprise workstation scene, separates the voice instructions of a user from environmental noise, and constructs the working state context of the user at the current moment in combination with the expression of the user. The working state context and corresponding visual features are subjected to feature alignment and dimension reduction, and then corresponding edge feature vectors are generated, and the working interaction intention of the user is determined in combination with the sensitive preliminary screening result of the multi-modal interaction data, thereby improving the accuracy of the working interaction intention of the user. The corresponding user interaction scheme is determined according to the tracing of each task path, and the intelligent interaction event of the desktop robot is determined in combination with the reply content of the user and the current working state of the desktop robot, thereby improving the accuracy of the intelligent interaction event of the desktop robot.

Description

Technical Field

[0001] This invention relates to the technical field of cloud collaboration, and in particular to an intelligent interaction method and system for a desktop robot based on cloud collaboration. Background Technology

[0002] As enterprises accelerate their digital transformation, intelligent office equipment is gradually evolving from simple command execution tools into intelligent assistants with deep perception and collaborative capabilities. In open office environments, desktop robots, as an emerging intelligent terminal, are widely used to assist employees in handling daily tasks, retrieving information, and controlling peripheral devices.

[0003] Existing desktop robots typically rely solely on simple speech recognition technology for command parsing. In open-plan offices, ambient noise easily interferes with microphone arrays, leading to a significant drop in speech command recognition rates. Furthermore, existing technologies often overlook the user's non-explicit states and lack the construction of contextual understanding of the user's work state. This means that the robot cannot understand the deeper background of the user's commands, affecting the accuracy of the user's work interaction intentions and resulting in low accuracy of intelligent interaction events in desktop robots. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art. This invention provides an intelligent interaction method and system for desktop robots based on cloud collaboration.

[0005] This invention provides an intelligent interaction method for a cloud-based collaborative desktop robot, comprising: By collecting multimodal interaction data in enterprise workstation scenarios through desktop robots, separating user voice commands from environmental noise, and combining user facial expressions to construct the user's current work status context; The working state context and corresponding visual features are aligned and dimensionality reduced to generate corresponding edge feature vectors; the cloud collaboration server receives the edge feature vectors and combines them with the sensitive initial screening results of multimodal interaction data to determine the user's working interaction intent; Based on the identification of work interaction intent, multiple work interaction items of different dimensions are determined, and multiple task paths are determined by matching with the enterprise's knowledge graph. Based on the tracing of each task path, the corresponding user interaction scheme is determined, and the intelligent interaction events of the desktop robot are determined by combining the user's reply content and the current working status of the desktop robot. Extract multiple work feedback data from the user in the intelligent interaction event, and combine them with the user's work sequence to determine the user's work status in each work interaction project. Based on the work status of each work interaction project and the project progress marked by the desktop robot, determine the corresponding intelligent interaction report.

[0006] This invention provides an intelligent interaction system for a cloud-based collaborative desktop robot, which is applied to the aforementioned intelligent interaction method for a cloud-based collaborative desktop robot. The intelligent interaction system for the cloud-based collaborative desktop robot includes: The work status module is used to collect multimodal interaction data in enterprise workstation scenarios through desktop robots, separate user voice commands from environmental noise, and combine user facial expressions to construct the user's work status context at the current moment. The interaction intent module is used to align and reduce the dimensions of the working state context and the corresponding visual features, thereby generating the corresponding edge feature vector. The cloud collaboration server receives the edge feature vector and combines it with the sensitive preliminary screening results of multimodal interaction data to determine the user's working interaction intent. The intelligent interaction event module is used to identify multiple work interaction items in different dimensions based on the recognition of work interaction intentions, and to determine multiple task paths by matching with the enterprise's knowledge graph. Based on the tracing of each task path, the corresponding user interaction scheme is determined, and the intelligent interaction events of the desktop robot are determined by combining the user's reply content and the current working status of the desktop robot. The intelligent interaction report module is used to extract multiple work feedback data from the user in the intelligent interaction event, and combine them with the user's work sequence to determine the user's work status in each work interaction project. Based on the work status of each work interaction project and the project progress marked by the desktop robot, the corresponding intelligent interaction report is determined.

[0007] Compared with the prior art, the beneficial effects of the present invention are: (1) Multimodal interaction data in the enterprise workstation scenario is collected by desktop robot, the user's voice command and environmental noise are separated, and the user's expression is combined to construct the user's current work state context; the work state context and the corresponding visual features are aligned and dimensionality reduced to generate the corresponding edge feature vector; the cloud collaborative server receives the edge feature vector and determines the user's work interaction intention by combining the sensitive screening results of multimodal interaction data. The introduction of work state context further controls the edge feature vector and the sensitive screening results of multimodal interaction data, and improves the accuracy of the user's work interaction intention.

[0008] (2) Based on the identification of work interaction intentions, multiple work interaction items of different dimensions are determined, and multiple task paths are determined by matching the enterprise's knowledge graph. The corresponding user interaction scheme is determined by tracing each task path, and the intelligent interaction event of the desktop robot is determined by combining the user's reply content and the current working status of the desktop robot. The multiple task paths are further controlled, and the user interaction scheme, the user's reply content and the current working status of the desktop robot are fully considered, which improves the accuracy of the intelligent interaction event of the desktop robot.

[0009] (3) Extract multiple work feedback data of the user in the intelligent interaction event, and determine the user's work status in each work interaction project by combining the user's work sequence. Determine the corresponding intelligent interaction report based on the work status of each work interaction project and the project progress marked by the desktop robot. This fully considers the work status of each work interaction project and the project progress marked by the desktop robot, thus improving the accuracy of the intelligent interaction report. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the intelligent interaction method of a cloud-based collaborative desktop robot in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structural composition of the intelligent interactive system for a cloud-based collaborative desktop robot in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0012] Please see Figures 1 to 2 A cloud-based collaborative desktop robot intelligent interaction method is proposed and applied to cloud-based collaborative scenarios. The cloud-based collaborative desktop robot intelligent interaction method includes: Step S11: Collect multimodal interaction data in the enterprise workstation scenario through a desktop robot, separate the user's voice commands from environmental noise, and combine the user's facial expressions to construct the user's current work status context; Step S12: Align and reduce the dimensions of the working state context and the corresponding visual features to generate the corresponding edge feature vector; the cloud collaboration server receives the edge feature vector and determines the user's working interaction intent by combining the sensitive preliminary screening results of multimodal interaction data. Step S13: Based on the identification of work interaction intent, determine multiple work interaction items in different dimensions, and determine multiple task paths by matching the enterprise's knowledge graph. Based on the tracing of each task path, determine the corresponding user interaction scheme, and determine the intelligent interaction event of the desktop robot by combining the user's reply content and the current working status of the desktop robot. Step S14: Extract multiple work feedback data from the user in the intelligent interaction event, and determine the user's work status in each work interaction item by combining the user's work sequence. Determine the corresponding intelligent interaction report based on the work status of each work interaction item and the project progress marked by the desktop robot.

[0013] In step S11, the specific steps are as follows: S111: A multi-dimensional perception array is deployed on the desktop robot. This multi-dimensional perception array includes a multi-channel microphone array and a depth camera. The multi-dimensional perception array synchronously collects audio streams and visual images in the enterprise workstation scene, and then extracts the corresponding visual features, including lip movement sequences, facial changes and gaze points. S112: Using lip movement features in visual features as an auxiliary condition, temporal alignment and cross-modal fusion are performed with the audio stream. During the fusion process, spatial beamforming and masking filtering techniques are used to separate the user's voice commands from environmental noise, which includes keyboard typing and ambient voices. S113: Obtain the user's facial image sequence, further extract multiple facial expression features, and combine them with the user's voice commands to perform temporal fusion in the long short-term memory network to dynamically construct the user's working state context, which includes the user's current attention concentration, emotional valence, and fatigue level.

[0014] In the embodiments of this application, a multi-dimensional perception array is deployed on the desktop robot. The multi-dimensional perception array includes a multi-channel microphone array and a depth camera. The multi-dimensional perception array synchronously collects audio streams and visual images in the enterprise workstation scene, and then extracts the corresponding visual features, including lip movement sequences, facial changes and gaze points, thus introducing visual features.

[0015] At this point, a multi-dimensional sensing array is deployed on the top or front panel of the desktop robot. This array includes a four-channel MEMS microphone array arranged in a ring and a binocular depth camera equipped with RGB and infrared sensors.

[0016] Based on a hardware clock synchronization mechanism, using an FPGA (Field Programmable Gate Array) or a high-precision timer as a unified clock source, the audio sampling clock of the microphone array and the frame exposure trigger signal of the depth camera are locked in a phase-locked loop to ensure that the audio stream and the video stream are aligned at the millisecond level in timestamps, eliminating asynchronous deviations caused by different acquisition rates.

[0017] The microphone array continuously samples the ambient sound field at a sampling rate of 16kHz or 48kHz, generating a raw audio pulse-code modulation stream containing multi-channel sound pressure information. Simultaneously, the depth camera continuously captures images of a fan-shaped area directly in front of the robot at a frame rate of 30 frames per second, generating an RGB image stream containing pixel brightness information and a depth image stream containing object distance information. The system encapsulates the aforementioned audio stream and visual image stream with a unified timestamp into a data frame queue and stores it in the edge buffer for processing.

[0018] The face region in the acquired RGB image stream is detected and key points are located; a face alignment algorithm based on deep convolutional neural networks (such as ResNet or MobileNet variants) is used to accurately locate 68 or more facial key points in each frame of the image.

[0019] Based on this, a subset of lip key points is identified from facial key points, typically including 20 feature points such as the upper lip, lower lip, and corners of the mouth. The spatial coordinate displacement vectors of these key points between consecutive frames, the "opening degree: distance between the midpoints of the upper and lower lips", and the rate of change of the mouth corner width are calculated. These geometric change parameters are combined with time axis information to construct a high-dimensional lip movement geometric feature vector. This vector sequence can accurately describe the user's mouth opening and closing, rounding and spreading, and tongue position auxiliary movements during pronunciation, forming a visualized lip movement sequence, which serves as the core visual basis for subsequent comparison with audio signals.

[0020] To capture subtle emotional fluctuations and changes in user state, the system performs dynamic texture analysis on the RGB image stream, extracting minute changes in facial color components and deformation information of facial muscle movement units. Specifically, optical flow is used to calculate pixel motion vectors between adjacent frames for key facial regions, such as the forehead, between the eyebrows, and sides of the nose, generating a facial micro-variation feature map. Simultaneously, based on a facial expression recognition model, the system extracts the degree of eyebrow raising and lowering, the degree of eyelid opening and closing, and the puffing or stretching of the cheeks. These non-rigid deformation features are encoded to generate a facial feature vector describing the overall changes in the user's face. This feature vector reflects subtle facial changes from calm to frowning, from relaxation to tension.

[0021] Using depth information provided by a depth camera and combined with RGB images, a 3D geometric model of the user's head and eyes is constructed. A head pose estimation algorithm is used to calculate the pitch, yaw, and roll angles of the user's head in 3D space to determine the approximate orientation of the head. Next, eye gaze vectors are estimated using eye geometric features, such as the relative position of the iris center and the corner of the eye. The head pose vector and the eye gaze vector are then weighted and fused to construct a 3D gaze ray originating from the user's eyes.

[0022] The user's gaze ray is projected onto a pre-built "workstation 3D spatial plane model, including the computer monitor area, desktop keyboard area, document area, and robot body area" using ray projection calculation. By solving the coordinates of the intersection point of the ray and the plane, the location of the user's current gaze point in physical space is accurately determined, generating gaze point coordinate data. This data is used to quantify whether the user is looking at the screen for deep work or looking at the robot waiting for interaction.

[0023] Specifically, desktop robots are used in enterprises; the enterprise is an internet technology company, and employees use desktop robots as intelligent office assistants in an open office area; in the current scenario, employee user A is working on urgent code at his workstation. The surrounding environment is noisy, with colleagues talking and keyboard typing. At this time, user A needs to use voice commands to ask the desktop robot to query the server logs.

[0024] The circular microphone array on top of the desktop robot is activated, capturing user A's voice and surrounding background noise at a sampling rate of 48kHz; at the same time, the depth camera at the front of the robot locks onto user A's face in real time at a frame rate of 30fps; the hardware-level synchronization mechanism ensures that the audio data captured at second t and the facial image of user A acquired at second t are strictly one-to-one corresponding on the timeline, with no audio-visual misalignment.

[0025] When user A issues the command "Check the server error log", the depth camera captures the opening and closing of user A's lips; the algorithm inside the robot extracts the changes in lip contours when user A pronounces words such as "check" and "server" in real time, generating continuous lip movement geometric feature vectors; for example, the algorithm accurately captures the plosive opening of the lips when pronouncing "check" and the rounding of the lips when pronouncing "service". This lip movement sequence provides a visual matching basis for subsequent auxiliary recognition of speech content in noisy environments.

[0026] During the data collection process, due to the urgency of the project, the depth camera captured geometric deformation features of user A's furrowed brows and shortened distance between the eyebrows. At the same time, the optical flow method detected that user A's facial muscles were in a state of tension. The robot encoded these features into facial change feature vectors, identifying that the user's current facial state was anxious or focused, rather than relaxed or casual, thus providing a basis for subsequent judgment of the urgency of the user's work.

[0027] During the process of user A issuing commands, the depth camera calculated from the depth data that user A's head was slightly tilted forward, and the coordinates of the intersection point of the eye gaze vector and the workstation plane were displayed in the center area of ​​the monitor screen. This indicates that user A's gaze was mainly focused on the screen code, rather than staring at the desktop robot. This gaze characteristic suggests to the desktop robot that although the user is engaging in voice interaction, their attention is still highly focused on the task on the computer screen. Therefore, when providing feedback, the robot should use concise voice broadcasts or side screen prompts as much as possible, and avoid performing mechanical actions that significantly obstruct the screen, in order to conform to the user's current workflow habits.

[0028] Furthermore, lip movement features from visual characteristics are used as an auxiliary condition for temporal alignment and cross-modal fusion with the audio stream. During the fusion process, spatial beamforming and masking filtering techniques are used to separate the user's voice commands from environmental noise, which includes keyboard typing and ambient voices, thus fully considering both voice commands and environmental noise.

[0029] At this point, due to the significant difference in sampling rate between "visual signal: video frame" and "auditory signal: audio sampling point", it is necessary to perform precise temporal calibration on the lip movement features extracted in step S111 and the audio stream. Simultaneously, an interpolation alignment mechanism based on a unified timestamp is established; the high-frequency sampled audio stream is resampled or segmented into audio frames corresponding to the video frame rate, for example, one audio frame every 30ms; for each video time point, the audio features within the corresponding time window are calculated; and the discrete lip keypoint coordinate sequence is transformed into continuous lip movement trajectory parameters to ensure that each frame of audio signal can find a unique corresponding lip visual state. During this process, a Dynamic Time Warping (DTW) algorithm is introduced to correct local time offsets caused by head micro-movements or system processing delays, thereby generating a strictly synchronized "audiovisual synchronized data stream," providing a temporal basis for subsequent cross-modal fusion.

[0030] Based on temporal alignment, lip movement features are used as prior auxiliary conditions to guide the microphone array in spatial beamforming, constructing an enhanced target speech signal. Using the position of the lip movement features in the image coordinate system, combined with distance information measured by the depth camera, the precise azimuth and pitch angles of the user's mouth in the three-dimensional spatial coordinate system of the microphone array are calculated through a coordinate transformation matrix. Guided by these spatial coordinates, the filter weight coefficients of the multi-channel microphone array are adjusted to construct an adaptive beam pointing towards the user's mouth position.

[0031] This beamformer preserves incident sound waves from the user's direction to the greatest extent possible while spatially suppressing sound waves from "non-user directions, such as the side, rear, or distance." In addition, at the output of the beamformer, a cross-modal attention mechanism based on lip visual features is introduced. When the lips are detected to be in a static closed state but the audio input is strong, it is determined to be non-speech noise, and the beam output gain is reduced. Conversely, when the lip movement amplitude is positively correlated with the audio energy intensity, a high gain output is maintained, thereby achieving visual-assisted spatial filtering enhancement.

[0032] To address residual noise in the same direction as or near the user after beamforming, such as keyboard clicks, a masking filter generated by a deep learning network is used for fine separation in the time and frequency domains, ultimately outputting clean voice commands. The beamformed audio signal is converted into a time-frequency representation, such as a logarithmic power spectrum, and input together with synchronized lip movement feature vectors into a pre-trained audiovisual fusion denoising network (such as AV-Conformer or Dual-PathRNN).

[0033] This audiovisual fusion denoising network uses lip visual features as auxiliary information to predict the probability value of each time-frequency unit belonging to the "target speech," generating an ideal binary mask or an ideal ratio mask. This mask "illuminates" the speech-dominated region on the time-frequency graph while "masking" the noise-dominated region. The time-frequency coefficients of the original mixed signal are filtered using this mask, and the time-domain waveform is reconstructed through inverse transform. In this process, the algorithm specifically models and suppresses "keyboard tapping sounds: characterized by high-frequency, transient impact" and "ambient human voices: characterized by steady-state or non-steady-state spectral overlap," thereby completely separating the user's voice commands from the background noise.

[0034] Specifically, the processor inside the desktop robot receives image and sound data from user A; the system detects that user A's lips begin to open in preparation for pronunciation at second 0 and immediately locks the corresponding audio segment at that moment; although user A's head shakes slightly while speaking, the dynamic time warping algorithm still ensures that the lip shape change trajectory when user A pronounces the word "xiao" is precisely aligned with the starting plosive sound in the audio waveform, generating a strict audiovisual synchronization frame, thus eliminating time misalignment errors for subsequent processing.

[0035] The desktop robot uses lip visual features to locate user A's mouth at a distance of 50cm, 30 degrees to the left and in front of it. The microphone array quickly adjusts its weights to form a beam with the main lobe pointing towards user A's mouth. At this time, the voice of engineer Wang next to him is located in the direction of the side lobe of the beam and is suppressed by about 15 decibels. More importantly, when user A pauses between instructions: lips closed, although the environment is noisy, the visual auxiliary signal prompts the system that there is no target speech at this time. The beamformer adaptively reduces the overall gain, further filtering out useless background noise.

[0036] The beamformed signal still contains the crisp "click" sound of User A's mechanical keyboard. The system inputs the processed audio features and the visual features of User A's rapid lip opening and closing when speaking into the denoising network. The network generates a fine time-frequency mask that accurately identifies the instantaneous peak features of the keyboard click sound in the spectrum and masks them as noise, setting them to zero. At the same time, it enhances the audio segments that perfectly match the rhythm of User A's lip movements.

[0037] The output successfully separated a clear voice command stream: "Query the error log of the Backend-01 server"; the voice of Engineer Wang and the sound of User A typing on the keyboard, which were originally mixed in, were significantly removed, and the signal-to-noise ratio (SNR) was improved to the effective recognition range, ensuring that the desktop robot could accurately understand User A's urgent instructions.

[0038] Therefore, by acquiring the user's facial image sequence, further extracting multiple facial expression features, and combining them with the user's voice commands in the long short-term memory network, the user's working state context is dynamically constructed. This working state context includes the user's current attention concentration, emotional valence, and fatigue level, thus introducing the working state context.

[0039] At this point, the acquired user facial image sequence is analyzed frame by frame, and a pre-trained facial action unit detection network is used to extract high-dimensional expression feature vectors. The system performs facial keypoint detection and optical flow calculation on the input image sequence to capture subtle deformations of facial muscles; based on the Facial Action Coding System (FACS), the algorithm decomposes facial movements into multiple independent action units, such as the raising of the inner eyebrow (AU1), the frowning of the eyebrow (AU4), the degree of eyelid closure (AU43), and the raising or lowering of the corners of the mouth (AU12 / AU15).

[0040] Simultaneously, the system calculates the activation intensity, duration, and rate of change of these action units; in addition, it includes the analysis of micro-changes in skin texture, such as changes in redness caused by blood flow. These geometric deformation features are combined with texture features to construct a sequence of facial expression feature vectors containing multiple dimensions, which are used to characterize the user's instantaneous emotional fluctuations and mental state.

[0041] Acoustic features are extracted from the clean speech instructions separated in step S112, and then deeply fused with the facial expression feature sequence on the time axis. Speech signal processing algorithms are used to extract paralinguistic acoustic features from the speech instructions, in addition to the textual semantics. These include the fundamental frequency (F0) variation curve (reflecting pitch fluctuations), formant frequency distribution (reflecting vocal tract shape changes), short-time energy (reflecting volume strength), and speech rate rhythm. The system aligns and concatenates the extracted acoustic feature vectors with the facial expression feature vectors based on timestamps. For example, the acoustic features of "high fundamental frequency, fast speech rate" in a certain time period are combined with the facial expression features of "tight eyebrows and downturned mouth" in the same period to form a multimodal joint feature vector, which serves as the input tensor for subsequent networks.

[0042] The multimodal joint feature vector is input into a Long Short-Term Memory (LSTM) network or its variants (such as Bi-LSTM) for deep feature learning and state inference of time series data. The LSTM network handles long-range dependencies in sequence data through its unique gating mechanism: input gate, forget gate, and output gate. The network can "memorize" the user's state trends over a period of time and combine them with the input features at the current moment to suppress transient noise interference and smooth state judgment. The network output layer connects the fully connected layer and the classification and regression head, mapping and quantifying three core indicators in real time: "Attention Concentration: numerical range 0-1, reflecting the degree of focus of vision and cognition", "Emotional Valence: numerical range -1 to 1, reflecting negative to positive emotional tendencies", and "Fatigue Level: numerical range 0-10, reflecting the physiological state from wakefulness to extreme drowsiness". The system encapsulates these three indicators into structured data objects, namely the working state context, and continuously updates this context over time, forming a dynamic state flow.

[0043] Specifically, the desktop robot acquires a sequence of facial images of user A during the period of issuing commands and waiting; the algorithm detects that the inner side of user A's eyebrows (AU1) and the main body of the eyebrows (AU4) show a continuous and high-intensity state of lifting and tightening, forming obvious "frown lines"; at the same time, the corners of the mouth (AU15) show a slight downward pull, and the lips are tightly closed; the system combines these specific muscle movements into a high-dimensional facial expression feature vector, which is marked as a feature pattern of "extreme focus" and "stress".

[0044] The desktop robot analyzes the voice command "Query the error log of the Backend-01 server" just issued by user A. In addition to recognizing the text content, the acoustic feature extractor captures that user A's speech rate is significantly faster than his / her daily average, for example, 30% faster, and the "fundamental frequency: pitch" is in a high position, with a rapid drop in pitch at the end, indicating urgency. The system strictly aligns these acoustic features reflecting anxiety and urgency with the aforementioned facial expression features on a millisecond-level time axis to construct a multimodal joint feature of user A's "urgent voice and serious expression".

[0045] Multimodal joint features are input into the desktop robot's local LSTM inference network; the network combines this with historical data showing that user A has been frequently typing on the keyboard and has not left the computer in the past 10 minutes to make a comprehensive judgment: Attention concentration assessment: Based on the stability of eye contact with the screen and the persistence of brow tension, the network outputs an attention concentration score of 0.95, which is extremely high.

[0046] Emotional valence assessment: Based on the frowning feature and rapid tone of voice, the network assessed the emotional valence as -0.6, which is biased towards negative anxiety.

[0047] Fatigue level assessment: Despite the high pressure, User A's eye opening and closing is normal, and there are no signs of frequent blinking or yawning. Based on the network and historical data, the fatigue level is determined to be 2.0, indicating mild fatigue and being in a state of heightened work intensity.

[0048] The desktop robot dynamically constructs the current work state context of user A: {Attention: Very High, Emotion: Anxiety / Urgent, Fatigue: Low}. Based on this context, the robot decides to use concise and efficient voice broadcast when returning log query results, and to highlight the error lines on the screen, without engaging in any irrelevant chatter or animations, in order to match user A's current urgent work state.

[0049] In step S12, the specific steps are as follows: S121: Mark the edge computing end of the desktop robot, project the working state context vector and the corresponding visual features to the same high-dimensional latent space, and then perform cosine similarity maximization alignment in the high-dimensional latent space to output the aligned high-dimensional tensor. The high-dimensional tensor is then subjected to nonlinear dimensionality reduction and feature compression through a variational autoencoder to generate edge feature vectors with low bandwidth requirements. S122: Before transmitting to the cloud collaborative server, the edge computing terminal first triggers the sensitive information screening mechanism to output the sensitive screening results of multimodal interaction data, and uses the local lightweight named entity recognition model to perform entity desensitization and blurring of the business secrets and personal privacy data in the text and visual features of the voice command, thereby outputting the privacy-protected edge feature vector. S123: After receiving the privacy-protected edge feature vector, the cloud collaboration server uses the privacy-protected edge feature vector as a prompt word and couples it with the input layer of the multimodal large language space. It also performs contextual analysis of the question and answer in conjunction with the preset enterprise interaction scenario ontology library, and then outputs the user's work interaction intent, which includes information retrieval, task assignment, progress reporting or emotion adjustment.

[0050] In the embodiments of this application, the edge computing end of the labeled desktop robot projects the working state context vector and the corresponding visual features to the same high-dimensional latent space, thereby maximizing the alignment of cosine similarity in the high-dimensional latent space to output the aligned high-dimensional tensor. The high-dimensional tensor is then subjected to nonlinear dimensionality reduction and feature compression by a variational autoencoder to generate edge feature vectors with low bandwidth requirements, thus introducing edge feature vectors.

[0051] At this time, the system establishes a resource monitoring and environmental awareness thread in the "embedded computing unit: edge computing end" of the desktop robot; the edge computing end marks the current CPU, GPU and memory usage in real time, and at the same time obtains the working state context vector generated in step S113 and the original visual features extracted in step S111.

[0052] The system loads "working state context vectors; such as attention values, emotional valence, etc., scalar or low-dimensional vectors" and "visual features, such as facial key point heatmaps, gaze vectors, etc., high-dimensional features" in parallel. In order to eliminate the differences in numerical magnitude and physical dimension between different modal data, the system uses a normalization layer to standardize the input data, so that the abstract semantic information of the working state context and the low-level pixel information of the visual features are mapped to a unified numerical range, thereby generating a standardized input tensor for subsequent cross-modal fusion processing.

[0053] The standardized multimodal features are projected into a shared high-dimensional latent space using a deep neural network, and consistency constraints across modal features are enforced within this space. A two-stream encoder network is obtained to process the working state context and visual features separately. The working state context vector is mapped to a semantic embedding vector in the latent space, and the visual features are mapped to a visual embedding vector in the latent space. A contrastive loss function is introduced during the training or inference phase, with the optimization objective being to maximize the cosine similarity between the two types of embedding vectors in the latent space. By calculating the dot product of the two vectors and dividing by the product of their magnitudes, the system continuously adjusts the encoder parameters so that the semantic vector representing the user's state at the same moment and the visual vector tend to be consistent in direction. The system outputs an aligned high-dimensional tensor that integrates visual appearance information and internal state information. This tensor achieves semantic alignment between modalities while preserving complementary multimodal information.

[0054] To adapt to the limited uplink bandwidth of the network at the edge, a variational autoencoder is used to probabilistically generate and model high-dimensional tensors, achieving extreme data compression and feature denoising. The high-dimensional tensor is input into the encoder part of a pre-trained variational autoencoder; the encoder does not directly output compressed features, but predicts the probability distribution parameters of the features in the latent space, namely the mean vector and variance vector.

[0055] The system employs a reparameterization technique to sample and generate latent variables from the distribution. This process forces the features to learn the manifold structure of the data rather than simply copying pixels, thereby achieving excellent nonlinear dimensionality reduction. The system encodes the low-dimensional latent variables into a compact binary stream or floating-point array to generate edge feature vectors. These vectors not only have extremely low dimensionality, typically 1% or even less of the original image data, significantly reducing transmission bandwidth requirements, but also possess extremely high anti-interference capabilities and semantic integrity because they contain the essential features after denoising.

[0056] Specifically, the desktop robot's local chip detects that the current network is under high load, such as downloading a large file, and therefore automatically triggers a high compression ratio mode. The system loads the user A's current work status context values, such as "attention 0.95, high; mood -0.6; anxiety and fatigue 2.0, low," as well as the visual feature matrix captured by the camera, such as "user A's furrowed brow and dilated pupils," into memory. These data are standardized to floating-point numbers between 0 and 1, forming two sets of parallel input data streams.

[0057] The system's internal neural network projects the two sets of data streams into a 512-dimensional high-dimensional latent space. In this space, the originally distinct numerical attributes of "anxiety value" and "frowning image feature" are transformed into two high-dimensional vectors. The computation module calculates the cosine similarity of these two vectors and finds that they highly overlap in spatial direction, both pointing to the semantic direction of "urgent work". Based on this, the system aligns the two vectors and fuses them to generate a high-dimensional tensor that integrates "user A not only looks anxious, but physiological indicators do show anxiety", correcting the errors that may exist in a single modality, such as visual feature deviation caused by changes in lighting, but with stable context.

[0058] To quickly upload this complex state to the cloud, the 512-dimensional high-dimensional tensor was input into a variational autoencoder. The encoder ignored irrelevant visual noise such as colleagues moving in the background, extracted the most core feature distribution parameters, and resampled to generate an edge feature vector containing only 64 values. Compared to the original video stream of several megabytes, this data packet of 64 values ​​is extremely small. The desktop robot instantly sends this tiny data packet to the cloud via Wi-Fi. Even without seeing the original video, the cloud can perfectly reconstruct the core information that user A is "in an extremely focused and anxious emergency work state" from these 64 values, thus providing efficient and accurate data support for subsequent intent recognition.

[0059] Furthermore, before transmitting data to the cloud-based collaborative server, the edge computing terminal first triggers a sensitive information screening mechanism to output the initial screening results of the multimodal interaction data. It then utilizes a local lightweight named entity recognition model to desensitize and obfuscate trade secrets and personal privacy data within the text and visual features of the voice commands, thereby outputting a privacy-protected edge feature vector. This process fully considers the edge feature vector... At this point, before the edge computing end prepares to send data, the system proactively triggers a sensitive information screening mechanism. This mechanism, acting as a daemon process, performs a full-link scan of the data stream to be packaged and transmitted. The system reverse-maps and parses the edge feature vectors output in step S121, and combines them with the original voice command text and key visual features in the buffer to construct a queue of data to be inspected. The screening mechanism, based on a preset sensitive rule base, including a keyword blacklist and a list of high-frequency privacy attributes, performs a rapid matching scan of the data queue.

[0060] At the same time, the system detects whether the data contains sensitive signals of a specific pattern, such as a string of numbers of the length of an ID card number, an email address in a specific format, or an area of ​​ID photos or contract text appearing in the screen image. The scanning process outputs a preliminary sensitivity screening result, which marks whether there are potential sensitive items in the data and their modal channels, audio or visual, providing a location index for subsequent fine processing.

[0061] For suspicious data flagged by the initial screening mechanism, a locally deployed lightweight Named Entity Recognition (NER) model is loaded at the edge computing end. This model typically employs a pruned and quantized BERT-Tiny or BiLSTM-CRF architecture to adapt to the computing power limitations of edge devices. During execution, the model performs sequence labeling on the text converted from voice commands, identifying entity types such as person names (PER), organization names (ORG), geographic locations (LOC), and specific business entities such as project codes and IP addresses. Simultaneously, the model performs semantic analysis on the optical character recognition (OCR) results and saliency map regions in the visual features to determine whether the image content contains "trade secrets, such as financial statements on the screen or core algorithm fragments of source code" or "personal privacy, such as private chat logs." The model outputs specific entity labels and their location coordinates in the data stream, precisely locating sensitive information particles that need protection.

[0062] Based on the output of the named entity recognition model, corresponding data transformation operations are performed on the locked sensitive information to generate irreversible or difficult-to-reverse desensitized data, which is then repackaged into a secure edge feature vector. For text-based voice commands, the system employs entity desensitization technology: recognizing proper nouns is replaced with predefined generic placeholders, such as replacing "User A" with "...". <user>Replace "Backend-01" with "<SERVER_ID> Alternatively, a hash algorithm can be used to replace the original text. For visual features, the system employs blurring technology: using a Gaussian blur kernel or pixelation algorithm, the feature components corresponding to "sensitive areas, such as lines of code on the screen or private objects" in the image feature vector are smoothed or set to zero, thereby erasing specific visual details and retaining only the outline or existence features of the object. The system then re-integrates the desensitized text features with the blurred visual features, encapsulates them to generate a privacy-protected edge feature vector, which, while retaining the semantic structure required for user interaction intent, completely strips away specific sensitive content, ensuring that the data transmitted to the cloud complies with the enterprise's data security and privacy standards.

[0063] Specifically, before the desktop robot is ready to upload user A's interaction data, the edge-side sensitive information screening mechanism is activated; the system quickly scans user A's voice text "query the error log of the Backend-01 server" and the visual features of the current workstation; the screening rule base finds that "Backend-01" conforms to the internal server naming format, and the visual features capture high-density text characters on the screen, which are suspected to be code, and then outputs the sensitive screening result, marking "voice stream contains business entities, visual stream contains high-density text information", which requires in-depth inspection.

[0064] The robot activates its local lightweight named entity recognition model. On the speech side, the model analyzes the text sequence and identifies "Backend-01" as an entity belonging to the "Business Secrets - Server Equipment" category. On the visual side, the model combines OCR technology with scene classification to determine that the screen feature vector contains a large number of "programming language keywords" (such as class, def, import) and "internal project codes". The model outputs precise labels, indicating the specific word locations in the speech and the coordinates of the screen area in the visual features as highly sensitive regions.

[0065] Based on the model's recognition results, the desktop robot immediately performs desensitization operations. At the speech feature level, "Backend-01" is replaced with the general placeholder feature "[TARGET_SERVER]", and the acoustic fingerprint of the original word is scrambled, so that the cloud only knows that user A is querying a certain server, but does not know which one. At the visual feature level, the Gaussian blur algorithm is used to strongly smooth the feature vector of the screen code area, erasing the specific syntax logic and variable name features of the code, and only retaining the state feature that "the user is looking at the screen".

[0066] The desktop robot generates and uploads a privacy-protected edge feature vector. After receiving the vector, the cloud collaboration server can accurately understand user A's intention to "urgently query technical logs" and perceive their "anxious and focused" state, but it cannot obtain the specific asset information "Backend-01" or the code content on the screen. This perfectly balances the convenience of intelligent interaction with the security of enterprise-level data.

[0067] Therefore, after receiving the privacy-protected edge feature vector, the cloud-based collaborative server uses the privacy-protected edge feature vector as a prompt word and couples it with the input layer of the multimodal large language space. It also performs contextual analysis of the context and the pre-set enterprise interaction scenario ontology library to output the user's work interaction intent. This work interaction intent includes information retrieval, task assignment, progress reporting, or emotion adjustment. The introduction of work state context further controls the sensitive initial screening results of edge feature vector and multimodal interaction data, improving the accuracy of the user's work interaction intent.

[0068] At this point, the cloud-based collaborative server receives the privacy-preserving edge feature vector, maps and couples this vector to the input layer of the multimodal large language model, realizing the transformation from the low-level feature space to the high-level semantic space. The cloud-deployed "multimodal large language model, such as a vision-language model based on the Transformer architecture," has a unified input interface for processing text, audio, and image features. The system treats the edge feature vector as a special type of "soft cue word" or conditional vector, and through a learnable feature projection matrix, it upscales the low-dimensional edge feature vector and aligns it to the model's word embedding dimension or feature token dimension.

[0069] At the input layer, the vector is tensor-concatenated or cross-attention computed with text prompts used for task guidance, such as "analyze the user's intent based on the following user status". Through this coupling mechanism, the edge features that originally contained compressed audiovisual information are "translated" into deep semantic representations that the model can understand, enabling the large language model to "see" and "understand" the audiovisual behavior uploaded by the user at the edge and its corresponding desensitized speech content.

[0070] Within the model, a pre-defined enterprise interaction scenario ontology library is used to perform contextual constraints and relational parsing on the semantic representation of the input, thereby eliminating ambiguities in the understanding of vertical domains by the general model. At this point, the enterprise interaction scenario ontology library constructs a structured knowledge network containing "entity-relationship-attribute", covering the enterprise's internal business processes, organizational structure, commonly used terminology, and operational norms.

[0071] The system inputs the coupled semantic representation into the inference layer and uses Retrieval Augmentation Generation (RAG) technology to retrieve the most relevant context fragments to the current user state and desensitized instructions from the ontology. For example, if the model detects that the user input contains verbs like "query" and emotional features like "anxiety," the ontology will activate context branches related to "operation and maintenance fault handling" or "data permission application." Based on this contextual information, the large language model disambiguates and completes the user's ambiguous instructions, analyzing the potential purpose behind the user's behavior rather than simply performing literal matching.

[0072] Based on the results of the association analysis, the model calculates the probability distribution of the user's goal at the output layer and maps it to predefined interaction intent categories, thereby outputting structured work interaction intents. At this point, the model's classification head performs multi-classification processing on the semantic features, calculating the confidence level for each preset intent category. These categories cover the main needs of enterprise office work, including: "Information retrieval: querying data and documents," "Task assignment: creating work orders and sending notifications," "Progress reporting: updating status and summarizing work," and "Emotional regulation: relieving stress and relaxing through casual conversation." The system selects the category with the highest confidence level as the final output, along with a corresponding confidence score. This output guides subsequent steps in selecting the appropriate task processing module, ensuring that the robot's response actions highly match the user's actual needs.

[0073] Specifically, the cloud-based collaborative server receives the edge feature vector uploaded by the desktop robot. This vector contains the anonymized voice command feature "query the log of [SERVER_ID]" and the visual and state features of user A, such as "frowning and speaking rapidly". The cloud-based multimodal large language model receives this vector and maps it into a high-dimensional semantic token through a projection layer. This token is injected into the model's input layer and merged with the system's preset text prompt "analyze user requests", allowing the model to instantly "fill in" the complete scene of user A anxiously querying data at their workstation.

[0074] When analyzing semantic tokens, the model calls the enterprise's "enterprise interaction scenario ontology library." The ontology library defines knowledge such as "online fault handling process" and "server operation and maintenance specifications." When the model recognizes the feature combination of "query log" and "high focus / anxiety," the ontology library quickly associates it with the context branch of "emergency fault troubleshooting." Based on this, the model infers that user A is not performing routine data browsing or simple log recording, but is dealing with a sudden technical problem and needs to quickly obtain key information to resolve the error.

[0075] Based on the above in-depth analysis, the model's output layer calculates the probability of each intent category. The results show that the confidence score of the "information retrieval" intent, specifically for fault information retrieval, is as high as 0.92, far higher than "task delegation" (0.05) or "emotion regulation" (0.03). Although user A exhibits anxiety, based on their specific behavior and log query, the model determines that their core purpose is to solve the problem rather than seek emotional relief. The cloud collaboration server outputs a clear work interaction intent: {Type: Information Retrieval, Subclass: Fault Log Query, Priority: High}. This accurate intent recognition result will drive subsequent steps to call the corresponding enterprise knowledge graph interface to prioritize obtaining the server's key error data for user A.

[0076] In step S13, the specific steps are as follows: S131: The cloud-based collaborative server inputs the work interaction intent into a multi-dimensional space and deconstructs multiple work interaction items within the multi-dimensional space; it calls the enterprise's knowledge graph, uses a graph neural network to take the work interaction items as target nodes, performs graph walking and subgraph matching based on meta-paths in the enterprise's knowledge graph, and plans multiple task paths, including required documents, contacts and approval processes. S132: By performing reverse logic tracing and conflict detection on each task path, a corresponding user interaction scheme is generated based on the principle of shortest path and lowest interaction cost. The user interaction scheme includes a multi-turn dialogue tree and action execution sequence. The user interaction scheme is then sent to the desktop robot. S133: When the desktop robot executes this user interaction scheme, it analyzes the user's reply and the desktop robot's current working status in real time, and performs adaptive jumps between nodes in the dialogue tree to determine the most suitable intelligent interaction event for the current situation. Intelligent interaction events include actively pushing files, silent standby, screen projection display, or voice Q&A.

[0077] In the embodiments of this application, the cloud collaboration server inputs the work interaction intent into a multi-dimensional space and deconstructs multiple work interaction items in the multi-dimensional space; it calls the enterprise's knowledge graph, uses a graph neural network to take the work interaction items as target nodes, performs graph walking and subgraph matching based on meta-paths in the enterprise's knowledge graph, and plans multiple task paths. The task paths include required documents, contacts and approval processes, thus introducing task paths.

[0078] At this point, the cloud-based collaboration server receives the identified "work interaction intent" and inputs it into a preset multi-dimensional task deconstruction space. This space defines multiple core dimensions required for task execution in the enterprise's business logic, including but not limited to "time dimension," "resource dimension," "functional dimension," and "process dimension."

[0079] The system employs a semantic parsing-based intent decomposition algorithm to map the input interaction intent to the aforementioned dimensions. In the resource dimension, the system identifies the information assets required to complete the task, such as databases and document libraries. In the functional dimension, the system identifies the relevant role permissions, such as operations personnel and approval managers. In the process dimension, the system identifies the required operational steps, such as querying, applying, and reporting. The system combines the identified elements from each dimension to deconstruct and generate a set of specific, atomic, or aggregate-level work interaction items. Each work interaction item represents an independent execution unit, such as "obtaining log access permissions," "retrieving error logs for a specific time period," or "generating a fault analysis report."

[0080] The system invokes the enterprise's knowledge graph, which stores the people, events, and things within the enterprise and their complex relationships. Simultaneously, it loads a pre-trained Graph Neural Network (GNN) model, such as GraphSAGE or GAT (GraphAttention Network). The system treats the work interaction items deconstructed in the previous step as target nodes or query nodes and maps them to the topology of the knowledge graph. Utilizing the node embedding capability of GNN, the system initializes the vector representation of the target node. The system defines meta-path patterns based on business logic; meta-paths define the reasonable connection logic between nodes, such as "user - (with permissions) - role - (accessible) - server" or "user - (belongs to) - department - (manages) - project". Based on these meta-paths, the GNN model performs graph walks and neighbor aggregation operations in the graph. Through multi-layer graph convolution propagation, it aggregates feature information from one-hop, two-hop, and even multi-hop neighbor nodes, thereby locating a set of potential nodes highly semantically related to the target node within the vast graph.

[0081] After feature aggregation, the system uses a subgraph matching algorithm to search the knowledge graph for the optimal topology that connects the target node with related resource nodes. Based on the node embedding similarity output by the graph neural network, the system filters out candidate nodes with the highest relevance to the work interaction project, such as the specific "Backend-01 server" node, "DevOps team" node, and "fault handling SOP document" node. The system further constructs a logical link connecting "starting point: user" to "end point: task target" based on edge attributes in the graph, such as "contains", "depends on", "belongs to", and "requires approval". This link covers the "required documents, such as operation and maintenance manual", "contact persons, such as on-duty operation and maintenance experts", and "approval processes, such as emergency access authorization flow" required to complete the task. The system evaluates the weights of possible links, such as path length, response time, and success rate, and finally plans one or more feasible task paths and outputs them as a structured execution sequence.

[0082] Specifically, the cloud-based collaborative server inputs the intent of "emergency fault information retrieval" into a multi-dimensional space. In the resource dimension, the system identifies that user A needs access to "server log files." In the function dimension, the system identifies user A as a "R&D engineer" who may require assistance from an "Operations administrator." In the process dimension, the system identifies this as a "read-only query" action. Based on this, the system deconstructs three specific work interaction projects: Project 1: Verify log read permissions; Project 2: Locate the error logs of the Backend-01 server; Project 3: Extract key error stack information.

[0083] The system invokes the enterprise's knowledge graph, which includes nodes such as "User A," "Backend-01 Server," "Log Database," "Operations Document A," and "On-Duty Engineer User B." The system maps these work interaction items to target nodes in the graph and starts a graph neural network. Following the meta-path pattern "User A - (team) > R&D Group - (project in charge) > e-commerce platform project - (running on) > Backend-01 Server," the GNN traverses the graph. The model aggregates the features of nodes along the path and finds that the Backend-01 server is in a "highly sensitive" level, and its log data is constrained by the "data security policy" edge. The weight of the edge directly connecting User A node and the log node is extremely low.

[0084] Based on the aggregation results of GNN, the system performs subgraph matching and finds that connecting "User A" and "Error Log" requires passing through an intermediate node. Required document: Matched node "Emergency Troubleshooting Guide V2.0". Contact person: Matched node "On-Duty Maintenance Engineer - User B". Approval process: Due to the involvement of sensitive logs, a subgraph path is matched: "User A > Request temporary access > (Automatic approval: due to emergency status + User A's job level) > Authorization approved". The system plans a complete task path: Path: Obtain "Emergency Troubleshooting Guide" > Trigger emergency automatic approval > Establish secure connection with Backend-01 > Retrieve error log > (If failed) Contact User B; The cloud sends this path to the desktop robot, guiding it to assist User A according to this logic.

[0085] Furthermore, by performing reverse logic tracing and conflict detection on each task path, a corresponding user interaction scheme is generated based on the principle of shortest path and lowest interaction cost. This user interaction scheme includes a multi-turn dialogue tree and action execution sequence. The user interaction scheme is then distributed to the desktop robot, fully taking into account the user interaction scheme.

[0086] At this point, for each task path generated in step S131, the system performs reverse logical tracing from the target node back to the starting user to verify the integrity and compliance of the path. The system constructs a virtual backtracking stack, starting from the end of the task path, such as "get logs," and tracing backwards to check the fulfillment of each precondition. During this process, the system calls a logic detection engine to check step-by-step whether dependencies form a closed loop, whether required resources are locked, and whether permissions are mutually exclusive. For example, if the path includes "execute sensitive operations," the backtracking will check whether the preceding node "administrator approval" has been completed. If it is detected that the user is only required to provide the identity credentials they should have at the beginning of the path at the end, it is determined as a logical breakpoint. Simultaneously, the system performs conflict detection, analyzing whether there is "resource contention" between different task paths, such as both paths needing exclusive microphone access, or mutual exclusion of states, such as a conflict between "mute mode" and "voice broadcast," and marks paths with logical deadlocks or resource conflicts as infeasible and removes them.

[0087] Among the feasible paths that pass conflict detection, the system, based on a multi-objective optimization strategy, comprehensively considers the principles of "shortest path: fewest physical node hops" and "lowest interaction cost: fewest user operation steps and lowest cognitive load" to select the optimal path and compile it into a structured user interaction scheme. At this point, the system calculates a comprehensive weight for each feasible path. This weight calculation includes not only the graph topological distance but also the interaction friction coefficient. For example, paths requiring manual user input of verification codes are assigned a high-cost weight, while paths based on biometrics or context-based implicit verification are assigned a low-cost weight. After selecting the optimal path, the system uses Natural Language Generation (NLG) technology to construct a multi-turn dialogue tree, defining the queries, confirmations, and feedback scripts that the robot should initiate at different nodes. Simultaneously, it generates action execution sequences, defining the API interfaces the robot needs to call, the screen display layout, and mechanical movement instructions. This interaction scheme not only includes the logical flow but also specifies the tone and rhythm of the interaction, ensuring the scheme's efficiency and user-friendliness.

[0088] The cloud-based collaborative server serializes and encodes the generated user interaction scheme, producing standard control instruction data packets, which are then sent to the edge computing terminal of the desktop robot via a secure communication channel. The system parses the multi-turn dialogue tree into a state machine transition table and the action execution sequence into event triggers on the timeline, encapsulating both into JSON or binary format data packets conforming to the robot communication protocol. Before sending, the system performs adaptive compression on the data packets based on the edge terminal state flags obtained in step S121. If network bandwidth is insufficient, only the first round of interaction instructions is sent, and subsequent instructions are sent in a streaming manner. This data packet containing complete interaction logic is pushed to the desktop robot, guiding its local system to load the corresponding voice model and UI interface, preparing to begin specific interaction with the user.

[0089] Specifically, the cloud performs reverse testing on the two paths. For the first path (direct query): backtracking reveals that directly querying logs requires "Debug privileges." The current user A's role is "R&D Engineer," and their basic privileges do not include direct access to this sensitive log, creating a logical breakpoint as the precondition is not met. However, further testing of the enterprise rule base reveals a "temporary privilege escalation for emergency faults" logic. If the user is in an "emergency state," S123 has identified this and can trigger automatic approval. After completing the logic, the first path becomes feasible. For the second path (contacting human assistance): the logic is completely feasible and conflict-free.

[0090] System analysis revealed that the second path requires establishing a voice call channel, but user A is talking to a colleague next to him. The ambient human voices detected by S112 would introduce interference if a call were introduced at this time, which is a high interaction cost scenario.

[0091] The system compares two paths; the first path, although it requires temporary privilege escalation logic, is completed automatically in the system background and has a short physical path, with the user being greater than the log system; the second path, although simple in logic, has a long interaction chain and requires interrupting user A's current work.

[0092] Therefore, based on the principle of lowest interaction cost, the system selects the optimized first path as the optimal solution; the system generates a user interaction plan: the first round of dialogue is defined as "Emergency query permission has been requested for you, and the Backend-01 log is being retrieved, please wait"; if the query fails, the branch node "Do you need to transfer to the on-duty maintenance user B" is started; instruction 1 activates the green breathing light to indicate that it is being processed, instruction 2 displays the "Permission acquisition in progress..." progress bar on the screen, instruction 3 displays the code in a split screen after the log is retrieved.

[0093] The cloud encapsulates the above solution into a command data packet and sends it to the desktop robot via the Wi-Fi channel. After receiving the data packet, the desktop robot immediately parses the state machine, lights up the green breathing light, and prepares to report to user A according to the preset script after the log is retrieved.

[0094] Therefore, when executing this user interaction scheme, the desktop robot analyzes the user's response and its own current working status in real time, and adaptively jumps between nodes in the dialogue tree to determine the most suitable intelligent interaction event for the current situation. Intelligent interaction events include actively pushing files, silent standby, screen projection display, or voice Q&A. It further controls multiple task paths, fully considers the user interaction scheme, the user's response, and the current working status of the desktop robot, and improves the accuracy of the desktop robot's intelligent interaction events.

[0095] At this time, during the execution of the user interaction plan, the edge computing end of the desktop robot continuously runs a multimodal listening thread to capture and semantically parse the user's reply content in real time, and simultaneously monitor the robot's own current working status.

[0096] The system utilizes an Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) pipeline to analyze user speech feedback during interaction intervals sentence by sentence, extracting intent slots and emotional polarities, such as negative words like "no need," affirmative words like "okay," or interruptive commands like "wait a moment." Simultaneously, the system's state monitoring module polls the robot's hardware and software internal registers in real time to obtain current operating status parameters, including battery level, speaker occupancy status, screen display activity level, network latency, and the current position of the robotic arm. The system aligns the parsed "semantic information: user dimension" with "state parameters: machine dimension" to construct a dynamic contextual feature vector, providing real-time input for subsequent path decisions.

[0097] Based on the constructed contextual feature vector, the system performs node traversal and jump logic judgments within a multi-turn dialogue tree structure loaded in memory, breaking the preset linear execution order. Each node in the dialogue tree has predefined trigger conditions, jump rules, and child node pointers. The system performs fuzzy matching between the current contextual feature vector and the trigger conditions of the current node. If the system detects that the user's response contains negative intent or negative emotions, and the robot itself is in a low-battery state, it triggers "interruption" or "simplification" logic, directly jumping to the end node or power-saving node. If the system detects that the user's response contains specific detailed needs, such as "only view error messages," it skips the general information display node and directly jumps to the specific information filtering node. If the system detects that the user issues a new command during the robot's broadcast, it forcibly terminates the current dialogue tree branch and jumps back to the root node or the starting node corresponding to the new intent. Through this adaptive jump mechanism, the system ensures that the interaction flow always matches the user's real-time needs.

[0098] Based on the final termination or execution node determined by the dialogue tree, the system maps and activates the corresponding intelligent interaction event, and calls the corresponding hardware resources to complete the interaction action. The system maintains an event mapping table, associating the final node index of the dialogue tree with the specific interaction event type; intelligent interaction events include, but are not limited to: if the user requests a file and the transmission conditions are met, an active file push event is triggered, activating the file transfer protocol; if the user does not respond for a long time and the ambient noise is extremely high, a silent standby event is triggered, turning off audio output and retaining only visual indicators; if the system detects that the user's gaze is focused on the robot screen and the content is rich, a screen projection display event is triggered, rendering a high-resolution UI to the desktop; if the user asks a clear question and the environment is relatively quiet, a voice question-and-answer event is triggered, activating the TTS engine; after determining the event type, the system generates a specific set of control instructions to drive the corresponding actuators to complete the human-computer interaction action.

[0099] Specifically, the desktop robot began executing the plan, but when it was about to perform the "voice broadcast log summary" step, User A suddenly said loudly to the colleague next to him, "Wait a minute, let me check this error message;" The voice response was either due to environmental interference or was not directed at the robot, and the robot's microphone detected that the ambient noise level had increased again; at the same time, User A leaned forward, staring intently at the computer screen, with his head turned away from the robot. The robot judged the current working state as: the user's attention was extremely low and the screen was occupied.

[0100] The robot's edge computing terminal analyzed in real time that user A's reply was not a response to the log query, and combined with the fact that the "high focus" state in S113 remained unchanged, the system determined that forcibly broadcasting "original planned node" would seriously interfere with user A.

[0101] Therefore, the system performs adaptive navigation in the dialogue tree: skipping the "voice broadcast log summary" node, skipping the "ask if detailed analysis is needed" node, and directly jumping along the branch to the "silent assistance" node, which is logically defined as: when the user is focused and there is noise, only visual assistance is provided, without auditory disturbance.

[0102] Based on the "Silent Assistance" node after the redirection, the system determines that the current intelligent interaction event is "screen projection display," rather than the originally planned voice Q&A. The desktop robot remains silent and does not trigger a voice event, but its built-in projector or vertical screen immediately lights up, projecting the critical error logs of the Backend-01 server in high-contrast text directly onto the desktop or robot side screen within user A's line of sight. A flashing red light at the bottom of the screen prompts user A that "the result is out." After glancing at it out of the corner of their eye, user A can quickly scan and obtain the information without having to listen to the voice, perfectly matching the current urgent and noisy work scenario.

[0103] In step S14, the specific steps are as follows: S141: During the execution cycle of the intelligent interaction event, the desktop robot extracts multiple work feedback data from the user. The work feedback data includes voice confirmation commands, mouse / keyboard response operation latency, and gaze duration. A temporal convolutional network is used to correlate and compare the work feedback data with the user's preset work sequence, and an anomaly detection mechanism is combined to evaluate the user's work performance on each work interaction item. S142: The edge computing end of the desktop robot maintains a lightweight state machine and updates the process nodes of each work interaction project in real time to form the project process marked by the desktop robot. By combining the work status of each work interaction project with the corresponding project process, the cloud collaboration server calls a report generation template based on a preset template and performs data visualization mapping and natural language association to generate a structured intelligent interactive report. The report not only includes a panoramic display of the current task progress, but also further embeds rule-based and data-driven work alerts and workflow optimization suggestions.

[0104] In the embodiments of this application, during the execution cycle of the intelligent interaction event, the desktop robot extracts multiple work feedback data from the user. The work feedback data includes voice confirmation commands, mouse / keyboard response operation latency, and gaze duration. A temporal convolutional network is used to correlate and compare the work feedback data with the user's preset work sequence, and an anomaly detection mechanism is combined to evaluate the user's work status on each work interaction item, thus introducing work status.

[0105] At this time, during the execution cycle of "intelligent interactive events, such as screen projection displays or voice Q&A," the edge computing terminal of the desktop robot simultaneously launches multi-source data acquisition threads to capture user feedback data from all angles. The system acquires data through the following channels: Auditory channel: Continuous listening using a microphone array, capturing user voice confirmation commands such as "received", "okay", "check again" or negative feedback through keyword detection technology, while recording the intonation and pause characteristics of the voice.

[0106] Peripheral input channel: By listening to the user's host operating system low-level interface (such as Hook technology) or USB protocol, capture the response operation latency of the user's mouse / keyboard; specifically record the time difference from the robot output information, such as the display of logs to the user's first key press or mouse movement, as well as the subsequent click frequency and typing speed.

[0107] Visual Channel: Utilizing the eye-tracking module of a depth camera, the system calculates the duration of the user's gaze on the robot's screen or projection area, as well as the gaze trajectory. The system performs timestamp alignment and noise reduction on this raw data, eliminating noise data caused by environmental vibrations or unconscious movements, and constructs a structured feedback data stream.

[0108] The system employs a temporal convolutional network (TCN) as its core analysis model, performing deep spatiotemporal correlation comparisons between extracted work feedback data and user-preset work sequences distributed from the cloud. The user-preset work sequences define standard temporal behavior templates for completing specific tasks, such as "checking logs," for example: reading a summary for 5 seconds, frequently typing on the keyboard, and finally issuing a confirmation command. The system maps the work feedback data into multivariate time series vectors and inputs them into the TCN network. The TCN network, utilizing its dilated causal convolutional kernels, can capture the temporal dependencies of long sequences and calculate the similarity matrix between the current feedback sequence and the preset sequence in parallel. The model compares the user's actual behavior flow with the standard template flow frame by frame: for example, detecting whether "gaze duration" matches reading expectations, and detecting whether "keyboard operation latency" involves excessively long pauses. By calculating Hamming distance or Dynamic Time Warping (DTW) loss, the network outputs alignment error features for each time step, quantifying the degree of deviation between user behavior and the expected process.

[0109] Based on the alignment error features output by TCN, the system combines an anomaly detection mechanism to qualitatively and quantitatively evaluate the user's performance on each work interaction item. Specifically, the system constructs an anomaly detection module containing either an isolated forest or a single-class support vector machine (OCSVM). This module establishes the boundary of normal behavior distribution based on historical normal interaction data. The system inputs the current alignment error features into the detector to calculate its anomaly score. If a user exhibits excessively long response latency, cluttered eye movements, or a lack of expected voice confirmation on a particular interaction item, and the anomaly score exceeds a preset threshold, then the item is determined to have an anomaly such as "execution obstruction" or "comprehension difficulty." Conversely, if the temporal characteristics of the feedback data highly match the preset sequence, and the anomaly score remains at a low level, the evaluation result is "smooth execution." The system ultimately generates an evaluation report containing the completion, fluency, and anomaly type of each interaction item, providing a basis for subsequent interaction strategy adjustments. Specifically, during the 30-second log display period displayed by the robot, the edge computing terminal began recording data. Visually: User A's gaze lingered on the "NullPointerException" line in the projection area for 3.5 seconds, indicating a prolonged gaze, followed by a rapid scan of the code editor. Peripherally: Within 0.5 seconds of seeing the log line, User A began rapidly typing on the keyboard and copied a piece of code; the keyboard / mouse response latency was extremely short, indicating a rapid reaction. Auditorily: User A muttered "As expected, that's the error," providing voice feedback; although no direct command was given to the robot, it contained confirmation semantics.

[0110] The TCN network inside the desktop robot compares user A's behavior flow with the preset "troubleshooting" standard sequence. The standard sequence is expected to be: user pays attention to error information > quickly locates the code > begins repair. TCN analysis found that user A's "gaze-action" timing curve highly overlaps with the waveform of the standard template, only 0.5 seconds longer than expected in the specific gaze duration, possibly due to the complexity of the error. The network outputs a low value of sequence error characteristics, indicating that user A's behavioral logic fully conforms to the expected process of the current task.

[0111] The anomaly detection mechanism receives the error characteristics of the TCN and calculates that they fall within the confidence interval of the normal behavior distribution. Despite the noisy environment, User A's keyboard feedback did not hesitate or pause, and no "stuttering" or "confusion" anomaly alarms were triggered. The system ultimately evaluated User A's performance on the "view logs" task as: {Status: Efficient processing, Anomaly level: None, Confirmation: High}. This means that the robot does not need to provide additional explanation or assistance and can remain silent or end the current interaction task to avoid disturbing User A's repair work.

[0112] Furthermore, the edge computing end of the desktop robot maintains a lightweight state machine and updates the process nodes of each work interaction project in real time to form the project process marked by the desktop robot. Combining the work status of each work interaction project with the corresponding project process, the cloud collaboration server calls a report generation template based on a preset template and performs data visualization mapping and natural language association to generate a structured intelligent interactive report. The report not only includes a panoramic display of the current task progress, but also embeds rule-based and data-driven work alerts and workflow optimization suggestions. It fully considers the work status of each work interaction project and the project process marked by the desktop robot, thus improving the accuracy of the intelligent interactive report.

[0113] At this point, the edge computing end of the desktop robot constructs and maintains a finite state machine in memory. The state set of this state machine corresponds one-to-one with the work interaction items planned in step S131. The state machine defines the lifecycle states of each item, including "not started", "in execution", "suspended", "completed", and "abnormal interruption". The edge end drives the state machine to perform state transitions in real time based on the trigger signals of the intelligent interaction events determined in step S133 and the user feedback data extracted in step S141. For example, when the user completes the voice confirmation command, the state of the corresponding item is updated from "in execution" to "completed". If no operation is detected for a long time or negative feedback is received from the user, the state transitions to "abnormal interruption". The system saves the snapshot sequence of the state machine at different timestamps to form a project process data stream with time axis markings, thereby accurately recording the progress trajectory of the task in the micro-time dimension.

[0114] The cloud-based collaborative server receives project progress data uploaded from the edge device and the work status assessment generated in step S141, including anomaly scores and execution efficiency, and merges and aligns the two in terms of time. At this point, the server loads a preset report generation template, which includes a metadata header, time, user, device ID, task summary area, detailed data analysis area, and suggestion area. The system matches the corresponding layout structure from the template library according to the task type, such as "information retrieval" or "troubleshooting". The server fills the merged data into each logical unit of the template, for example, marking the status of "Backend-01 log query" as "success" and filling the time consumption data into the efficiency statistics column. At the same time, the system highlights high-risk events marked by the anomaly detection module, forming structured semi-finished report data.

[0115] Based on the template, the system performs multi-dimensional semantic enhancement processing to ultimately generate a visual, intelligent, and interactive report. Specifically, the system calls a chart rendering engine to map numerical "project progress data, such as the time spent on each stage," into time-series bar charts or process Gantt charts, and "work performance evaluations, such as attention concentration," into heat maps or radar charts, thus achieving a graphical presentation of the data.

[0116] By using Natural Language Generation (NLG) technology, structured data and charts are transformed into human-readable text descriptions; for example, "takes 5 seconds, anomaly score 0.1" is transformed into "users respond quickly in the log query process, with no obvious lag".

[0117] The system combines rule-based expert systems, such as "triggering an alert if the error rate exceeds a threshold," with data-driven predictive models, such as "predicting potential bottlenecks in the current operation based on historical data." It automatically embeds work alerts into reports, such as "detecting that the user has been working at high intensity for an extended period of time and suggesting a break," as well as workflow optimization suggestions, such as "suggesting that the Backend-01 log query permission be preset to a regular permission to reduce approval time." The final generated report includes a panoramic view of the current task progress, in-depth performance analysis, and forward-looking improvement suggestions, and is output to users or managers through interface display or document format.

[0118] Specifically, on the desktop robot, the state machine at the edge computing end runs in real time. When the log is projected and displayed, the state machine marks the "Log Query" item as "Executing". When it detects that user A stares for 3.5 seconds and starts typing on the keyboard to fix the code, the state machine determines that this step has been completed and updates the state to "Completed". As user A stops interacting with the robot and focuses on programming, the state machine marks the entire interaction task as "Suspended / Waiting for Result". These state changes are packaged into a project progress data stream in real time, ready for uploading.

[0119] The cloud-based collaborative server receives project progress data and, combined with the work performance of "User A responds quickly and has extremely high attention" assessed by S141, retrieves a "troubleshooting task report template" from the template library. The server automatically populates the data: task name "Backend-01 Log Query", start time "14:30:00", end time "14:30:35", and status "Success". At the same time, the extremely low anomaly score calculated by S141 is filled into the "Health" indicator column, forming the skeleton of the report.

[0120] The cloud engine processes the data to generate a Gantt chart containing four nodes: "Command Issuance - Permission Request - Log Display - User Confirmation." This chart visually demonstrates the total time taken: only 35 seconds. The permission request stage, which took 2 seconds, is highlighted as a highly efficient stage. The generated text description reads: "In this interaction, User A demonstrated extremely high response efficiency during the log retrieval phase. The transition from information presentation to code repair was seamless, and the task objective has been achieved."

[0121] The system detected that user A has been working continuously for more than 3 hours and is currently under high stress. Based on the emotional valence of S123, a yellow warning was embedded in the report: "The user is detected to be in a state of 'high fatigue but high excitement'. It is recommended to take a mandatory 15-minute break after the code is fixed."

[0122] Based on the experience of completing this query without manual approval, the report recommends: "Given that Backend-01 is a high-frequency failure node, it is recommended that the IT department modify the security policy to automatically enable read-only log permissions for developers on this server, which is expected to save approximately 1.5 seconds of approval handshake time each time." This intelligent interactive report, complete with graphics and text, popped up on the desktop robot's screen. After fixing the code, User A could immediately see that the task was closed and received a helpful reminder and process optimization suggestions from the system.

[0123] Please see Figure 2 The cloud-based collaborative desktop robot intelligent interaction system is applied to the aforementioned cloud-based collaborative desktop robot intelligent interaction method; the cloud-based collaborative desktop robot intelligent interaction system includes: The work status module 21 is used to collect multimodal interaction data in the enterprise workstation scenario through the desktop robot, separate the user's voice commands from the environmental noise, and combine the user's facial expressions to construct the user's work status context at the current moment. The interaction intent module 22 is used to perform feature alignment and dimensionality reduction on the working state context and corresponding visual features, thereby generating the corresponding edge feature vector; the cloud collaboration server receives the edge feature vector and determines the user's working interaction intent by combining the sensitive preliminary screening results of multimodal interaction data. The intelligent interaction event module 23 is used to identify multiple work interaction items in different dimensions based on the recognition of work interaction intent, and to determine multiple task paths by matching with the enterprise's knowledge graph. Based on the tracing of each task path, the corresponding user interaction scheme is determined, and the intelligent interaction event of the desktop robot is determined by combining the user's reply content and the current working status of the desktop robot. The intelligent interaction report module 24 is used to extract multiple work feedback data of the user in the intelligent interaction event, and combine the user's work sequence to determine the user's work status in each work interaction project. Based on the work status of each work interaction project and the project progress marked by the desktop robot, the corresponding intelligent interaction report is determined.

[0124] It should be noted that although multiple modules are mentioned in the detailed description above, this division is not mandatory; in fact, according to the embodiments of this disclosure, the features and functions of two or more modules or described above can be embodied in one module; conversely, the features and functions of one module described above can be further divided into multiple modules to be embodied.< / user>

Claims

1. A method for intelligent interaction of a desktop robot based on cloud-based collaboration, characterized in that, include: By collecting multimodal interaction data in enterprise workstation scenarios through desktop robots, separating user voice commands from environmental noise, and combining user facial expressions to construct the user's current work status context; The working state context and corresponding visual features are aligned and dimensionality reduced to generate the corresponding edge feature vectors. The cloud-based collaborative server receives edge feature vectors and combines them with the sensitive initial screening results of multimodal interaction data to determine the user's work interaction intent; Based on the identification of work interaction intent, multiple work interaction items of different dimensions are determined, and multiple task paths are determined by matching with the enterprise's knowledge graph. Based on the tracing of each task path, the corresponding user interaction scheme is determined, and the intelligent interaction events of the desktop robot are determined by combining the user's reply content and the current working status of the desktop robot. Extract multiple work feedback data from the user in the intelligent interaction event, and combine them with the user's work sequence to determine the user's work status in each work interaction project. Based on the work status of each work interaction project and the project progress marked by the desktop robot, determine the corresponding intelligent interaction report.

2. The intelligent interaction method for a cloud-based collaborative desktop robot according to claim 1, characterized in that, The process involves collecting multimodal interaction data in enterprise workstation scenarios using a desktop robot, separating user voice commands from environmental noise, and combining this with the user's facial expressions to construct the user's current work state context, including: A multi-dimensional perception array is deployed on the desktop robot. This multi-dimensional perception array includes a multi-channel microphone array and a depth camera. The multi-dimensional perception array synchronously collects audio streams and visual images in the enterprise workstation scene, and then extracts the corresponding visual features, including lip movement sequences, facial changes and gaze points. Lip movement features from visual characteristics are used as an auxiliary condition to perform temporal alignment and cross-modal fusion with the audio stream. During the fusion process, spatial beamforming and masking filtering techniques are used to separate the user's voice commands from environmental noise, which includes keyboard typing and ambient voices.

3. The intelligent interaction method for a cloud-based collaborative desktop robot according to claim 2, characterized in that, The method of collecting multimodal interaction data in enterprise workstation scenarios through desktop robots, separating user voice commands from environmental noise, and constructing the user's current work state context by combining user facial expressions also includes: The system acquires the user's facial image sequence, further extracts multiple facial expression features, and combines them with the user's voice commands to perform temporal fusion in the Long Short-Term Memory network to dynamically construct the user's working state context, which includes the user's current attention concentration, emotional valence, and fatigue level.

4. The intelligent interaction method for a desktop robot based on cloud collaboration according to claim 1, characterized in that, The process involves aligning and reducing the dimensions of the working state context and corresponding visual features to generate the corresponding edge feature vector. The cloud-based collaboration server receives edge feature vectors and, combined with the sensitive initial screening results of multimodal interaction data, determines the user's work interaction intent, including: The edge computing end of the labeled desktop robot projects the working state context vector and the corresponding visual features into the same high-dimensional latent space, thereby aligning them by maximizing cosine similarity in the high-dimensional latent space to output the aligned high-dimensional tensor. The high-dimensional tensor is then nonlinearly reduced in dimensionality and compressed in feature by a variational autoencoder, thereby generating edge feature vectors with low bandwidth requirements. Before transmitting to the cloud-based collaborative server, the edge computing terminal first triggers a sensitive information screening mechanism to output the sensitive screening results of multimodal interaction data. It then uses a local lightweight named entity recognition model to perform entity desensitization and blurring of trade secrets and personal privacy data in the text and visual features of voice commands, thereby outputting privacy-protected edge feature vectors.

5. The intelligent interaction method for a cloud-based collaborative desktop robot according to claim 4, characterized in that, The process involves aligning and reducing the dimensions of the working state context and corresponding visual features to generate the corresponding edge feature vector. The cloud-based collaborative server receives edge feature vectors and, combined with the sensitive initial screening results of multimodal interaction data, determines the user's work interaction intent. This also includes: After receiving the privacy-protected edge feature vector, the cloud-based collaborative server uses the privacy-protected edge feature vector as a prompt word and couples it with the input layer of the multimodal large language space. It also performs contextual analysis of the context and related questions by combining the preset enterprise interaction scenario ontology library, and then outputs the user's work interaction intent, which includes information retrieval, task assignment, progress reporting or emotion adjustment.

6. The intelligent interaction method for a cloud-based collaborative desktop robot according to claim 1, characterized in that, The process involves identifying multiple work interaction items across different dimensions based on the recognition of work interaction intentions, determining multiple task paths by matching with the enterprise's knowledge graph, determining corresponding user interaction schemes based on the tracing of each task path, and determining intelligent interaction events for the desktop robot by combining the user's response content and the current working status of the desktop robot, including: The cloud-based collaborative server inputs the work interaction intent into a multi-dimensional space and deconstructs multiple work interaction items within that space. It then calls upon the enterprise's knowledge graph and uses a graph neural network to select the work interaction items as target nodes. The server performs graph walking and subgraph matching based on meta-paths within the enterprise's knowledge graph to plan multiple task paths, which include required documents, contacts, and approval processes.

7. The intelligent interaction method for a cloud-based collaborative desktop robot according to claim 6, characterized in that, The process of identifying multiple work interaction items across different dimensions based on the recognition of work interaction intentions, determining multiple task paths by matching with the enterprise's knowledge graph, determining corresponding user interaction schemes based on the tracing of each task path, and determining intelligent interaction events for the desktop robot by combining the user's response content and the current working status of the desktop robot, also includes: By performing reverse logic tracing and conflict detection on each task path, a corresponding user interaction scheme is generated based on the principle of shortest path and lowest interaction cost. The user interaction scheme includes a multi-turn dialogue tree and action execution sequence, and the user interaction scheme is sent to the desktop robot. When executing this user interaction scheme, the desktop robot analyzes the user's reply and its own current working status in real time, and adaptively jumps between nodes in the dialogue tree to determine the most suitable intelligent interaction event for the current situation. Intelligent interaction events include actively pushing files, silent standby, screen projection display, or voice Q&A.

8. The intelligent interaction method for a cloud-based collaborative desktop robot according to claim 1, characterized in that, The process involves extracting multiple work feedback data from the user during the intelligent interaction event, combining this data with the user's work sequence to determine the user's work status for each work interaction item, and determining the corresponding intelligent interaction report based on the work status of each work interaction item and the project progress marked by the desktop robot. This report includes: During the execution cycle of intelligent interactive events, the desktop robot extracts multiple work feedback data from the user, including voice confirmation commands, mouse / keyboard response latency, and gaze duration. A temporal convolutional network is used to correlate and compare the work feedback data with the user's preset work sequence, and an anomaly detection mechanism is combined to evaluate the user's work performance on each work interaction item.

9. The intelligent interaction method for a cloud-based collaborative desktop robot according to claim 8, characterized in that, The process of extracting multiple work feedback data from the user in the intelligent interaction event, determining the user's work status in each work interaction item by combining the user's work sequence, and determining the corresponding intelligent interaction report based on the work status of each work interaction item and the project progress marked by the desktop robot also includes: The edge computing end of the desktop robot maintains a lightweight state machine and updates the process nodes of each work interaction project in real time to form the project process marked by the desktop robot. By combining the work status of each work interaction project with the corresponding project process, the cloud collaboration server calls a report generation template based on a preset template and performs data visualization mapping and natural language association to generate a structured intelligent interactive report. The report not only includes a panoramic display of the current task progress, but also further embeds rule-based and data-driven work alerts and workflow optimization suggestions.

10. An intelligent interactive system for a desktop robot based on cloud collaboration, characterized in that, The cloud-based collaborative desktop robot intelligent interaction system is applied to the cloud-based collaborative desktop robot intelligent interaction method as described in any one of claims 1-9; the cloud-based collaborative desktop robot intelligent interaction system includes: The work status module is used to collect multimodal interaction data in enterprise workstation scenarios through desktop robots, separate user voice commands from environmental noise, and combine user facial expressions to construct the user's work status context at the current moment. The interaction intent module is used to align and reduce the dimensions of the working state context and the corresponding visual features, thereby generating the corresponding edge feature vector. The cloud collaboration server receives the edge feature vector and combines it with the sensitive preliminary screening results of multimodal interaction data to determine the user's working interaction intent. The intelligent interaction event module is used to identify multiple work interaction items in different dimensions based on the recognition of work interaction intentions, and to determine multiple task paths by matching with the enterprise's knowledge graph. Based on the tracing of each task path, the corresponding user interaction scheme is determined, and the intelligent interaction events of the desktop robot are determined by combining the user's reply content and the current working status of the desktop robot. The intelligent interaction report module is used to extract multiple work feedback data from the user in the intelligent interaction event, and combine them with the user's work sequence to determine the user's work status in each work interaction project. Based on the work status of each work interaction project and the project progress marked by the desktop robot, the corresponding intelligent interaction report is determined.

Images