Data processing method, apparatus, medium, program product, and robotic system

Abstract

The embodiment of the disclosure provides a kind of data processing method, device, medium, program product and robot system, it is related to robot technical field.Therein, the data processing method comprises: obtaining the first audio input data that the audio receiver of robot collects;In the case where it is determined that there is biological speech information in the first audio input data, the first audio input data is processed to generate the first text encoding data for representing biological speech information, and the pre-generated original image encoding data is extracted from the target cache as target image encoding data, the original image encoding data is the data generated and stored in the target cache based on the original image data collected by the image sensor of robot;According to the first text encoding data and the target image encoding data, the first audio output data for replying biological speech information is generated.In this way, the delay of multi-modal data processing can be reduced, and the delay of robot multi-modal voice interaction can be reduced.

Description

Technical Field

[0001] This disclosure relates to the field of robotics, and more specifically, to a data processing method, apparatus, medium, program product, and robot system. Background Technology

[0002] With the continuous development of robotics technology, robots are being widely used in an increasing number of fields such as industry, healthcare, and homes. To enhance interactive intelligence and user experience, robots offer multimodal voice interaction capabilities that integrate and process both speech and visual information. However, multimodal voice interaction technologies suffer from significant speech response delays, impacting the user experience. Summary of the Invention

[0003] In view of this, the present disclosure proposes a new technical solution for data processing.

[0004] According to a first aspect of the present disclosure, a data processing method is provided, the method comprising:

[0005] Acquire the first audio input data collected by the robot's audio receiver;

[0006] If it is determined that there is biological speech information in the first audio input data, the first audio input data is processed to generate first text encoded data to characterize the biological speech information, and pre-generated raw image encoded data is extracted from the target cache as target image encoded data; wherein, the raw image encoded data is data generated and stored in the target cache after encoding the raw image data collected by the robot's image sensor;

[0007] Based on the first text encoding data and the target image encoding data, first audio output data for responding to the biological speech information is generated.

[0008] Optionally, the original image encoded data in the target cache is generated in the following manner:

[0009] Acquire the raw image data collected by the robot's image sensor;

[0010] The original image data is image encoded to generate original image encoded data;

[0011] If the target cache does not store the original image encoded data, then the original image encoded data is stored in the target cache; or, if the target cache already stores the original image encoded data, then the original image encoded data stored in the target cache is replaced with the newly generated original image encoded data.

[0012] Optionally,

[0013] The first process of encoding the original image data and the second process of processing the first audio input data are different processes;

[0014] The first process encodes each frame of raw image data acquired by the robot's image sensor and stores the newly generated raw image encoded data in the target cache.

[0015] Optionally, the first process encodes each frame of raw image data acquired by the robot's image sensor in the following ways:

[0016] Each frame of raw image data is stored in the video stream buffer in the order of acquisition time;

[0017] The original image data in the video stream buffer is encoded frame by frame in the order of acquisition time from early to late, and the newly generated original image encoded data is stored in the target buffer.

[0018] Optionally, the method further includes:

[0019] Acquire the second audio input data collected by the robot's audio receiver;

[0020] If it is determined that there is biological speech information in the second audio input data and the first audio output data for responding to the biological speech information in the first audio input data has not been generated, the generation of the first audio output data is stopped, and the processing of the second audio input data is started to generate the second audio output data for responding to the biological speech information in the second audio input data.

[0021] Optionally, the method is executed by a server that receives audio input data and raw image data sent by the robot based on a real-time communication protocol.

[0022] Optionally, the method further includes:

[0023] During the server initialization phase, preload speech-to-text models, text-to-speech models, and visual language models.

[0024] After confirming robot initialization, a data processing pipeline corresponding to the robot is established. The data processing pipeline is used to process the audio input data and raw image data received from the robot.

[0025] According to a second aspect of the present disclosure, a robot system is provided, the robot system comprising a robot and a server, wherein:

[0026] The robot is used to collect first audio input data through an audio receiver, collect raw image data through an image sensor, and send the raw image data and the first audio input data to the server.

[0027] The server is configured to receive the first audio input data and the original image data to execute the data processing method described in the first aspect.

[0028] According to a third aspect of the present disclosure, a data processing apparatus is provided, the data processing apparatus including a memory and a processor, the memory being used to store computer instructions, and the processor being used to invoke the computer instructions from the memory to perform the method described in the first aspect.

[0029] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, the computer program implementing the method described in the first aspect when executed by a processor.

[0030] According to a fifth aspect of the present disclosure, a computer program product is provided, which may include a computer program that, when executed by a processor, can implement the method described in the first aspect.

[0031] The data processing method of this disclosure extracts pre-generated raw image encoding data from the target cache during voice interaction, without waiting for image encoding processing. This reduces the latency caused by image encoding during voice interaction, lowers multimodal data processing latency, reduces the latency of robot multimodal voice interaction, and improves user experience.

[0032] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the present disclosure.

[0034] Figure 1 This is a block diagram of a robot system that implements a data processing method according to an embodiment of the present disclosure.

[0035] Figure 2 This is a schematic flowchart of a data processing method provided in an embodiment of this disclosure.

[0036] Figure 3 This is a schematic flowchart of a data processing method provided in an embodiment of this disclosure.

[0037] Figure 4 This is a schematic flowchart of a data processing method provided in an embodiment of this disclosure.

[0038] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this disclosure. Detailed Implementation

[0039] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0040] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0041] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0042] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0043] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0044] It should be noted that the actions involved in the collection, storage, use, processing, transmission, provision, disclosure, and deletion of data in this disclosure embodiment are all carried out in compliance with the relevant data protection laws and regulations of the country or region where the data is located, and with the full authorization of the corresponding data owner.

[0045] Figure 1 This is a block diagram of a robot system implementing a data processing method according to an embodiment of this disclosure. Figure 1 As shown, the robot system 100 may include one or more robots 110, which may be humanoid robots, wheeled robots, tracked robots or other types of robots.

[0046] Each robot 110 may include an audio receiver 111, an image sensor 112, an audio player 113, and a controller 114. The audio receiver 111 may be a sensor such as a microphone, used to collect audio input data, such as ambient audio or biological voice audio (e.g., human voice, animal sounds). The image sensor 112 may be a sensor such as a camera, used to collect image data. The audio player 113 may be a device such as a speaker, used to play audio to the outside world. The controller 114 can be used to control the robot's audio receiver 111, image sensor 112, audio player 113, or other components to perform corresponding tasks. For example, the controller 114 can control the audio receiver to collect audio input data, control the image sensor to collect raw image data, and control the audio player to play audio to the outside world.

[0047] In some examples, the robot system 100 may further include a server 120. The robot may also include a communication component. The robot's controller may send audio input data and / or raw image data to the server 120 via the communication component. After processing the audio input data and / or raw image data, the server generates audio output data and sends it to the robot. Upon receiving the audio output data, the robot's controller can control an audio player to play the audio output data. In this way, multimodal voice interaction can be achieved through the robot and the server. In this example, the server may deploy an artificial intelligence model to process the audio input data and / or raw image data.

[0048] In other examples, the robot itself can process audio input data and / or raw image data to generate audio output data, and then play this audio output data through an audio player, thereby achieving multimodal voice interaction. In this example, the robot can deploy an edge-side artificial intelligence model to process the audio input data and / or raw image data.

[0049] While related technologies offer multimodal voice interaction capabilities based on artificial intelligence models that fuse audio and image data, this functionality suffers from significant voice response latency, impacting user experience. For example, when a speech segment is detected in the audio, encoding and decoding of both audio and image are initiated. In practical applications, image encoding is often slower than audio encoding, resulting in substantial delays in multimodal data encoding and processing, thus affecting voice response latency.

[0050] Therefore, it is necessary to provide a new data processing method to reduce the latency of multimodal data processing in order to reduce the latency of multimodal voice interaction in robots.

[0051] Figure 2 This is a schematic flowchart of a data processing method provided in an embodiment of this disclosure. The data processing method of this embodiment can be... Figure 1 The robot or server in the illustrated robot system performs the execution. For example... Figure 2 As shown, the data processing method may include the following steps S210 to S230.

[0052] Step S210: Obtain the first audio input data collected by the robot's audio receiver.

[0053] For example, the audio receiver may include a microphone, through which the robot can acquire the first audio input data.

[0054] Step S220: If it is determined that there is biological speech information in the first audio input data, the first audio input data is processed to generate first text encoding data for representing biological speech information, and the pre-generated original image encoding data is extracted from the target cache as target image encoding data.

[0055] Among them, the raw image encoded data is generated by encoding the raw image data collected by the robot's image sensor and stored in the target cache.

[0056] In some examples, after the robot initializes and initiates multimodal voice interaction, it can acquire raw image data via an image sensor and audio input data via an audio receiver. At this point, regardless of whether voice interaction occurs, the raw image data can be encoded in real time to generate raw image encoded data, which is then stored in a target cache. If bio-voice information is present in the acquired first audio input data, a voice interaction process can be triggered. During this process, the raw image encoded data stored in the target cache can be used directly, thus reducing the latency caused by encoding images during voice interaction.

[0057] In some examples, the time difference between the generation time of the target image encoded data and the generation time of the first text encoded data can be less than or equal to a preset time threshold, which can be a pre-set time, such as 1 second, 5 seconds, 10 seconds, or 60 seconds. This avoids excessively large time differences between the image and audio, preventing them from mismatching and improving the reliability of multimodal voice interaction.

[0058] In some examples, the aforementioned bio-voice information may include human voice feature information or animal sound feature information. The robot can interact with humans through dialogue or respond to and interact with animal sounds. For example, a Voice Activity Detection (VAD) module can be used to identify whether bio-voice information exists in the first audio input data. As a preprocessing module, VAD can identify whether bio-voice information exists in the original audio and segment and extract valid speech segments containing bio-voice information. The methods by which the VAD module detects whether bio-voice information exists in the first audio input data may include: energy thresholding (e.g., the energy of speech frames is usually significantly higher than background noise), statistical model methods (e.g., Gaussian Mixture Model, GMM), machine learning models (e.g., Support Vector Machine, SVM), etc.

[0059] It should be noted that if the first audio input data collected does not contain biological speech information, then there is no need to process the first audio input data, that is, steps S220 and S230 can be omitted.

[0060] Step S230: Generate first audio output data for responding to biological speech information based on the first text encoding data and the target image encoding data.

[0061] For example, the first text-encoded data and the target image-encoded data can be input into a decoder to obtain the first text information used to respond to biological speech information. The first text information can then be input into a text-to-speech model to obtain the first audio output data used to respond to biological speech information. This decoder can be a decoder for a Vision Language Model (VLM) that can simultaneously process and understand visual (image) and linguistic (text) information, enabling cross-modal joint reasoning and generation tasks.

[0062] In some examples, the initial audio data can be played back using the robot's audio player to respond with the biological voice information, enabling multimodal voice dialogue.

[0063] Using the above method, first audio input data is acquired from the robot's audio receiver. If bio-speech information is found in the first audio input data, it is processed to generate first text-encoded data representing the bio-speech information. Pre-generated raw image-encoded data is extracted from the target cache as target image-encoded data. This raw image-encoded data is generated by encoding raw image data acquired by the robot's image sensor and stored in the target cache. Based on the first text-encoded data and the target image-encoded data, first audio output data for responding to the bio-speech information is generated. In this way, pre-generated raw image-encoded data is extracted from the target cache during voice interaction, eliminating the need to wait for image encoding processing. This reduces the latency caused by image encoding during voice interaction, lowers multimodal data processing latency, reduces the latency of robot multimodal voice interaction, and improves the user experience.

[0064] In some embodiments, the raw image encoded data in the target cache can be generated by: acquiring raw image data collected by the robot's image sensor; performing image encoding on the raw image data to generate raw image encoded data; if the raw image encoded data is not stored in the target cache, storing the raw image encoded data in the target cache; or, if the raw image encoded data is already stored in the target cache, replacing the raw image encoded data stored in the target cache with the newly generated raw image encoded data.

[0065] In this way, image encoding, as a computationally intensive operation, is performed in advance and the results are cached. When voice interaction needs to combine visual information, the latest encoded image data can be directly retrieved from the target cache, completely avoiding time-consuming real-time image encoding processing during voice interaction and reducing the latency of robot multimodal voice interaction.

[0066] In some examples, this method can be used to continuously or periodically acquire raw image data and update the raw image encoding data in the target cache. This ensures the real-time nature of the cache, meaning that the target cache always stores the encoded data obtained from the most recently processed and up-to-date raw image data. This allows for multimodal voice interaction that incorporates the latest visual scenarios, improving the real-time performance and reliability of multimodal voice interaction.

[0067] In some examples, after the robot is started and initialized, the raw image data can be continuously or periodically acquired and the raw image encoding data in the target cache can be updated in this way to further reduce the latency of multimodal voice interaction.

[0068] In some embodiments, the first process of encoding the raw image data and the second process of processing the first audio input data are different processes.

[0069] For example, the first process can encode each frame of raw image data acquired by the robot's image sensor and store the newly generated raw image encoded data in the target cache.

[0070] By processing images and audio separately through different processes, multi-process parallel processing can be achieved, further reducing the latency of multimodal voice interaction.

[0071] In some examples, the way the first process encodes each frame of raw image data acquired by the robot's image sensor may include: storing each frame of raw image data acquired in the order of acquisition time in a video stream buffer; encoding the raw image data in the video stream buffer frame by frame in the order of acquisition time from earliest to latest, and storing the newly generated raw image encoded data in a target buffer.

[0072] Thus, by using video stream buffering to buffer the original image data during the first process of encoding, data loss due to untimely video processing can be avoided. Furthermore, buffering by capturing the data in chronological order can prevent out-of-order image data during transmission, ensuring the timing accuracy of the encoded images.

[0073] In some embodiments, the data processing method may further include the following steps:

[0074] Step S240: Obtain the second audio input data collected by the robot's audio receiver.

[0075] Step S250: If it is determined that there is biological speech information in the second audio input data and the first audio output data for responding to the biological speech information in the first audio input data has not been generated, stop generating the first audio output data and start processing the second audio input data to generate the second audio output data for responding to the biological speech information in the second audio input data.

[0076] It should be noted that steps S240 and S250 can be performed at any time. Figure 1 The steps S220 and S230 shown are executed during the execution process. For example, they can be executed during the execution of step S220, after step S220, or during the execution of step S230.

[0077] The first audio input data and the second audio input data mentioned above correspond to different biological speech information. For example, the first audio input data and the second audio input data can be sound segments made by different people, or different sound segments made by the same person at certain intervals.

[0078] In this way, the latest audio segment can interrupt the output of historical audio segments, thereby realizing a real-time interruption mechanism for voice interaction and improving the real-time performance of voice interaction.

[0079] In some examples, this real-time interruption mechanism can be implemented based on tasks. For instance, each time it is determined that there is biological speech information (such as human voice segments) in the audio input data, a first task is initialized and generated, executing functional modules such as Speech-to-Text (STT) model, Vision Language Models (VLM), and Text-to-Speech (TTS) model. If a second human voice segment enters before the first task is completed, the original first task is immediately stopped. For example, the first task can be discarded and destroyed, and a second task can be created to process the second human voice segment. This enables a sound-based real-time interruption function, improving the real-time performance of voice interaction.

[0080] In some embodiments, the data processing method may be provided by Figure 1 The server in the robot system shown can receive audio input data and raw image data sent by the robot based on the Real-Time Communication protocol.

[0081] For example, a robot can acquire audio input data using an audio receiver and raw image data using an image sensor, and then send the audio input data and raw image data to a server via the RTC protocol. Upon receiving the audio input data and raw image data via the RTC protocol, the server can either process them directly or store them in a data cache for sequential processing.

[0082] It should be noted that WebSocket technology can be used to transmit audio and image data in related technologies. WebSocket technology is a full-duplex communication protocol based on TCP. However, this embodiment uses RTC technology. The robot transmits audio and images directly to the server in the form of streaming media. Compared with the WebSocket technology used in related technologies, this can save about 200 milliseconds of latency.

[0083] In some examples, the robot can send audio input data and raw image data to the server separately based on different RTC streams. For instance, it can send audio input data to the server via a first RTC stream and raw image data via a second RTC stream. After receiving the audio input data and raw image data separately based on the different RTC streams, the server can store them in different data buffers. This allows audio and image to be transmitted separately, avoiding mutual interference and improving the reliability of data transmission.

[0084] In some examples, the server can preload a speech-to-text (STT) model, a text-to-speech (TTS) model, and a visual language model (VLM) during the server initialization phase; after confirming that the robot initialization is successful, a data processing pipeline corresponding to the robot is established, which can be used to process the audio input data and raw image data received from the robot.

[0085] The STT model described above can be used to convert input audio data into text data, the TTS model described above can be used to convert text data into audio data, and the VLM described above can be used to achieve cross-modal understanding and generation by fusing image and text information.

[0086] This data processing pipeline can be a modular, programmable task processing pipeline that connects multiple independent algorithm steps in a logical sequence to achieve an end-to-end data processing flow. This data processing pipeline can also be called a pipeline. It can call models such as STT, VLM, and TTS for data processing.

[0087] In this example, the algorithm model and data processing pipeline are initialized separately. The server initialization implements the initialization of algorithm models such as VLM, STT, and TTS, and these models can reside permanently on the server. Establishing the data processing pipeline after robot initialization improves the efficiency of data processing initialization and shortens the startup time for a better user experience.

[0088] In some embodiments, Figure 2 The data processing method shown can also be performed by a robot. For example, an edge AI model can be deployed on the robot to process audio input data and / or raw image data. In this way, the robot itself can process the audio input data and / or raw image data to generate audio output data, and play the audio output data through an audio player, thereby realizing multimodal voice interaction and further reducing the latency of multimodal voice interaction.

[0089] Figure 3This is a schematic flowchart of a data processing method provided in an embodiment of this disclosure. Figure 3 As shown, the data processing method may include the following steps 301 to 309.

[0090] Step 301: The server loads the speech-to-text (STT) model, text-to-speech (TTS) model, and visual language model (VLM).

[0091] Step 302: After confirming that the robot initialization is successful, establish the data processing pipeline corresponding to the robot.

[0092] This data processing pipeline can be used to process audio input data and raw image data received from the robot.

[0093] For example, after any robot is successfully initialized, the code of the corresponding pipeline section can be pulled up to initialize the multimodal voice interaction function. This initialization process takes about 2 seconds.

[0094] Step 303: The robot collects audio input data through the audio receiver, collects raw image data through the image sensor, and transmits the audio input data and raw image data to the server in real time through the RTC protocol.

[0095] Step 304: The server processes the raw image data based on the first process.

[0096] For example, the server can store the received raw image data in the video stream buffer, encode the raw image data based on the VLM image encoder, and store the newly generated raw image encoded data in the target buffer.

[0097] It should be noted that the target cache can retain only the most recently generated raw image encoding data. Each time new raw image encoding data is generated and stored, the historically generated raw image encoding data in the target cache can be replaced.

[0098] Step 305: The robot collects first audio input data through an audio receiver and transmits the first audio input data to the server in real time via the RTC protocol. The first audio input data may contain biological speech information.

[0099] For example, on the robot's end, a user asks, "What do you see?" At this time, the first audio input data collected by the robot contains biological speech information (such as human speech characteristics).

[0100] Step 306: The server's Voice Activity Detection (VAD) module detects the biological voice information, can initialize and generate the first task, extract the valid voice segments containing the biological voice information, and output the valid voice segments to the Speech-to-Text (STT) model.

[0101] Step 307: The STT module converts the valid speech segment into first text (What do you see?), and inputs the first text into the VLM text encoder.

[0102] Step 308: The VLM text encoder encodes the first text to obtain the first text encoded data. At the same time, it pulls the pre-generated original image encoded data from the target cache as the target image encoded data. Both the first text encoded data and the target image encoded data are input into the VLM decoder to obtain the first response text used to reply to the first text. The first response text is then input into the text-to-speech (TTS) model.

[0103] Step 309: The TTS model processes the first response text to generate the first audio output data. This first audio output data can be transmitted to the robot based on the RTC protocol to complete the first task.

[0104] For example, the first audio output data generated by TTS can be processed into an audio stream and transmitted to the robot via the RTC protocol. After receiving the first audio output data based on the RTC protocol, the robot can play the first audio output data through an audio player to complete the voice interaction.

[0105] This method extracts pre-generated raw image encoding data from the target cache during voice interaction, eliminating the need to wait for image encoding. This reduces the latency caused by image encoding during voice interaction, lowers multimodal data processing latency, reduces the latency of robot multimodal voice interaction, and improves user experience.

[0106] Figure 4 This is a schematic flowchart of a data processing method provided in an embodiment of this disclosure. Figure 4 As shown, the data processing method may include steps 401 to 406.

[0107] Step 401: The server loads the speech-to-text (STT) model, text-to-speech (TTS) model, and visual language model (VLM).

[0108] Step 402: After confirming that the robot initialization is successful, establish the data processing pipeline corresponding to the robot.

[0109] This data processing pipeline can be used to process audio input data and raw image data received from the robot.

[0110] For example, after any robot is successfully initialized, the code of the corresponding pipeline section can be pulled up to initialize the multimodal voice interaction function. This initialization process takes about 2 seconds.

[0111] In some examples, the robot collects audio input data through an audio receiver, acquires raw image data through an image sensor, and transmits the audio input data and raw image data to the server in real time via the RTC protocol.

[0112] The server can process the raw image data based on the first process. For example, the server can store the received raw image data in the video stream buffer, encode the raw image data based on the VLM image encoder, and store the newly generated raw image encoded data in the target buffer.

[0113] It should be noted that the target cache can retain only the most recently generated raw image encoding data. Each time new raw image encoding data is generated and stored, the historically generated raw image encoding data in the target cache can be replaced.

[0114] Step 403: The robot collects the first audio input data through the audio receiver and transmits the first audio input data to the server in real time through the RTC protocol. The first audio input data may contain biological speech information.

[0115] For example, on the robot's end, a user asks, "What do you see?" At this time, the first audio input data collected by the robot contains the human voice feature (which is a type of biological voice information).

[0116] It should be noted that, in order to achieve data transmission based on the RTC protocol, an RTC client can be deployed on the robot and an RTC server can be deployed on the server. Data transmission based on the RTC protocol can be achieved through the RTC client and the RTC server.

[0117] Step 404: The server's Voice Activity Detection (VAD) module detects the biological voice information (human voice features), can initialize and generate the first task, extract the valid voice segments containing the biological voice information, and output the valid voice segments to the speech-to-text (STT) model for data processing based on STT, VLM, and TTS.

[0118] Step 405: During the execution of the first task, if the server receives the second audio input data sent by the robot, determines that there is biological speech information in the second audio input data, and the first audio output data used to respond to the biological speech information in the first audio input data has not been generated, the server stops generating the first audio output data and initializes the generation of the second task. The second task is used to start processing the second audio input data to generate the second audio output data used to respond to the biological speech information in the second audio input data.

[0119] For example, regardless of the step the first task has reached, if a user suddenly asks "How's the weather today?" through the chatbot, the server's VAD module detects the human voice, creates a second task, extracts the human voice segment, and inputs it into the STT (Speed-to-Speech) model. At this point, the first task can be suspended and destroyed, and the second task can be started. For instance, the second audio input data can be processed into speech-to-text to obtain the second text (How's the weather today?). This second text is then encoded into text to obtain the second text encoded data. Simultaneously, pre-generated raw image encoded data is retrieved from the target cache as the target image encoded data. Both the second text encoded data and the target image encoded data are input into the VLM decoder to obtain the second response text. This second response text is then input into the text-to-speech (TTS) model.

[0120] Step 406: The TTS model processes the second response text to generate second audio output data. This second audio output data can be transmitted to the robot based on the RTC protocol to complete the second task.

[0121] After receiving the second audio output data based on the RTC protocol, the robot can play the second audio output data through an audio player.

[0122] This method enables real-time interruption based on sound during voice interaction, thereby improving the real-time performance of voice interaction.

[0123] This disclosure also provides a robot system, such as Figure 1 As shown, the robot system may include a robot and a server, wherein: the robot can be used to acquire first audio input data via an audio receiver, acquire raw image data via an image sensor, and send the raw image data and the first audio input data to the server; the server can be used to receive the first audio input data and the raw image data to perform actions such as... Figure 2 The data processing method shown.

[0124] In some examples, the robotic system can perform actions such as Figure 3 or Figure 4 The data processing method shown.

[0125] Figure 5 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this disclosure. Figure 5 As shown, the data processing apparatus 900 may include a memory 910 and a processor 920. The memory 910 may be used to store computer instructions, and the processor 920 may be used to retrieve computer instructions from the memory 910 to execute all or part of the steps of any of the methods in the foregoing embodiments of this disclosure. It should be noted that the processor 920 may include one or more processors to execute instructions, and the memory 910 may also include one or more memories to store computer instructions.

[0126] In some embodiments, the processor 920 can be used to control the overall operation of the data processing device 900. For example, the processor 920 can execute instructions to implement all or part of the steps of the methods in any of the foregoing embodiments of this disclosure, thereby enabling one or more of operations such as voice communication, data communication, database operation, display control, component control, and multimedia processing. The aforementioned components can be internal components of the data processing device itself, or external components connected to the data processing device wirelessly or wiredly. For example, the components may include sensors, cameras, headphones, input / output devices, etc. The aforementioned multimedia may include one or more of voice, images, video, and text.

[0127] In some embodiments, the memory 910 may include one or more memories, the contents of which may be the same or different. The memory 910 may be configured to store various types of data to support the operation of the data processing device 900. Examples of such data include instructions for any application or method operating on the data processing device 900, contact data, phonebook data, messages, pictures, videos, etc. The memory 910 may be implemented by any type of temporary or non-temporary storage device or a combination thereof, such as a portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), compact disc-read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanical encoding device, such as a punch card or recessed protrusion structure storing instructions thereon, and any combination thereof.

[0128] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods in the foregoing embodiments of this disclosure. Optionally, the computer-readable storage medium may be a non-transitory storage medium, but is not limited thereto; it may also be a temporary storage medium.

[0129] This disclosure also provides a computer program product that may include a computer program that, when executed by a processor, can implement any of the methods described in the foregoing embodiments of this disclosure.

[0130] This disclosure may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement any of the methods in the foregoing embodiments of this disclosure.

[0131] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media may include, for example, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0132] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0133] The computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" language or similar programming languages). The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network (such as a local area network or a wide area network), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this disclosure.

[0134] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0135] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0136] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It should be noted that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are all equivalent.

[0138] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Acquire the first audio input data collected by the robot's audio receiver; If it is determined that there is biological speech information in the first audio input data, the first audio input data is processed to generate first text encoded data to characterize the biological speech information, and pre-generated raw image encoded data is extracted from the target cache as target image encoded data; wherein, the raw image encoded data is data generated and stored in the target cache after encoding the raw image data collected by the robot's image sensor after the robot is initialized and the multimodal voice interaction function is started. The first text encoding data and the target image encoding data are input into the decoder to obtain the first text information used to respond to the biological speech information. The first text information is then input into the text-to-speech model to generate the first audio output data used to respond to the biological speech information.

2. The method according to claim 1, characterized in that, in, The raw image encoded data in the target cache is generated in the following manner: Acquire the raw image data collected by the robot's image sensor; The original image data is image encoded to generate original image encoded data; If the target cache does not store the original image encoded data, then the original image encoded data is stored in the target cache; or, if the target cache already stores the original image encoded data, then the original image encoded data stored in the target cache is replaced with the newly generated original image encoded data.

3. The method according to claim 2, characterized in that, The first process of encoding the original image data and the second process of processing the first audio input data are different processes; The first process encodes each frame of raw image data acquired by the robot's image sensor and stores the newly generated raw image encoded data in the target cache.

4. The method according to claim 3, characterized in that, The first process encodes each frame of raw image data acquired by the robot's image sensor in the following ways: Each frame of raw image data is stored in the video stream buffer in the order of acquisition time; The original image data in the video stream buffer is encoded frame by frame in the order of acquisition time from early to late, and the newly generated original image encoded data is stored in the target buffer.

5. The method according to claim 1, characterized in that, The method further includes: Acquire the second audio input data collected by the robot's audio receiver; If it is determined that there is biological speech information in the second audio input data and the first audio output data for responding to the biological speech information in the first audio input data has not been generated, the generation of the first audio output data is stopped, and the processing of the second audio input data is started to generate the second audio output data for responding to the biological speech information in the second audio input data.

6. The method according to claim 1, characterized in that, The method is executed by a server, which receives audio input data and raw image data sent by the robot based on a real-time communication protocol.

7. The method according to claim 6, characterized in that, The method further includes: During the server initialization phase, preload speech-to-text models, text-to-speech models, and visual language models. After confirming robot initialization, a data processing pipeline corresponding to the robot is established. The data processing pipeline is used to process the audio input data and raw image data received from the robot.

8. A robot system, characterized in that, The robotic system includes a robot and a server, wherein: The robot is used to collect first audio input data through an audio receiver, collect raw image data through an image sensor, and send the raw image data and the first audio input data to the server. The server is configured to receive the first audio input data and the original image data to perform the data processing method according to any one of claims 1 to 7.

9. A data processing apparatus, characterized in that, The data processing apparatus includes a memory and a processor, the memory being used to store computer instructions, and the processor being used to retrieve the computer instructions from the memory to execute the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1 to 7.

Images