Human-computer interaction method, system and equipment based on voice differential processing and medium

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly discloses a human-computer interaction method, system and device based on voice differential processing and a medium, and the method comprises the steps: extracting a voice data sequence and scene features, constructing a background noise database, collecting user voice information, carrying out voiceprint recognition model training, obtaining a noise reducer, and carrying out noise reduction analysis. Extracting a time interval of the noise reduction voice data sequence, performing dialogue segmentation, determining a plurality of semantic data subsequences, performing differential semantic analysis on the plurality of semantic data subsequences, determining a plurality of interaction instructions and interaction time, performing action execution scheme analysis based on the interaction instructions and the interaction time, and determining a robot action execution scheme. Performing interaction control on the robot according to the action execution scheme; the response degree of the robot to the voice instruction of the user can be better improved, the individual requirements of the user are met, and therefore efficient interaction application is conducted in the family environment.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a human-computer interaction method, system, device and medium based on speech differential processing. Background Art

[0002] Artificial intelligence technology is developing rapidly. In terms of technology, breakthroughs in advanced algorithms such as deep learning have greatly improved the learning ability of machines. In terms of data, massive amounts of network and IoT data provide rich materials for AI training. In terms of policy, countries have supported and invested in AI research, accelerating the maturity of technology. The growing demand for intelligent services in society has promoted the application of AI in many fields. In addition, advances in hardware such as GPUs have provided powerful power for complex AI calculations. These have not only promoted the development of artificial intelligence, but also brought about a series of problems and challenges such as employment changes, privacy protection, and ethics.

[0003] Currently, in a home environment, users want to control the robot to turn left through voice commands, but due to the interference of the sound of electronic devices and other sound sources, the traditional system often makes mistakes in recognition, often misidentifying as "forward" or missing recognition, which leads to response failure. When there are multiple guests visiting the home environment, and the user sends voice commands to the robot at the same time, the multiple guests have loud conversations, causing the user's voice to overlap with the guests' voices. The robot cannot accurately distinguish the overlapping voices and determine the voice command to be executed, and thus cannot execute the actions required by the user's voice command.

[0004] In addition, in a home environment, when users are accustomed to issuing multiple commands in succession (such as "forward", "stop", "turn right") to control the robot to complete corresponding continuous actions, the traditional system is easily confused by too many commands and cannot accurately judge the order, which is prone to misrecognition and response delays, making it impossible for the robot to achieve intelligent and efficient voice interaction experience in different scenarios.

[0005] In view of this, the present invention proposes a human-computer interaction method, system, device and medium based on speech differential processing. Summary of the invention

[0006] The purpose of the present invention is to provide a human-computer interaction method, system, device and medium based on voice differential processing to solve the technical problems in the prior art that robots have high levels of misrecognition, missed recognition and response failures in recognizing voice control commands.

[0007] To achieve the above object, the present invention provides the following technical solution: a human-computer interaction method based on speech difference processing, comprising the following steps:

[0008] S1: extracting a speech data sequence received by a target robot within a preset time window, and performing scene feature extraction on an application scene of the target robot to obtain a scene feature set, and constructing a background noise database based on the scene feature set;

[0009] S2: Collect the user's voice information, combine it with the electronic device sound information and the guest sound information in the background noise database, perform voiceprint recognition model training, obtain a background voiceprint recognition denoiser, and use the background voiceprint recognition denoiser to perform noise reduction analysis on the scene feature set and the voice data sequence to obtain a noise-reduced voice data sequence;

[0010] S3: extracting the time intervals between adjacent noise-reduced speech data in the noise-reduced speech data sequence, performing dialogue segmentation on the noise-reduced speech data sequence according to the extraction result, and determining a plurality of noise-reduced semantic data subsequences;

[0011] S4: performing differential semantic analysis on the multiple noise reduction semantic data subsequences to determine multiple interaction instructions, where the multiple interaction instructions include multiple interaction times;

[0012] S5: analyzing the action execution plan of the target robot based on the multiple interaction instructions and the multiple interaction times, and determining the action execution plan of the target robot;

[0013] S6: interactively controlling the target robot according to the target robot action execution plan.

[0014] As a preferred technical solution of the first aspect of the present invention, a preset time window is configured according to the actual application requirements of the robot, the preset time window is used as a collection period for voice data, and voice samples are obtained based on the preset time window.

[0015] As a preferred technical solution of the first aspect of the present invention, the logic for obtaining the scene feature set is:

[0016] Within the preset time window, the microphone is enabled to collect the user's voice data in real time, and the collected audio data is transmitted in the form of digital signals;

[0017] At the same time, the robot's environmental sensor is activated to obtain environmental feature data of the current application scenario based on the environmental sensor, and the microphone is activated to collect background noise data in the home scenario. The background noise data includes the sound of electronic equipment and the sound information of guests, so as to obtain scene data;

[0018] The collected scene data is preprocessed, including removing outliers and normalizing, filtering out speech recognition scene features based on a specific threshold, and combining the scene features into a scene feature set.

[0019] As a preferred technical solution of the first aspect of the present invention, the construction process of the background voiceprint recognition denoiser is as follows:

[0020] A1: noise identification is performed on the sound information of the electronic device and the sound information of the guest in the background noise database to obtain a noise sample set, the voice information of the user is collected to perform a command to issue a sound identification to obtain a recognition sample set, and the noise sample set and the recognition sample set are mixed according to a preset ratio to obtain a mixed sample set;

[0021] A2: Use audio processing technology to traverse the mixed sample set to perform voiceprint feature analysis, and obtain a voiceprint feature sample set to perform initial training on the discriminator, wherein the discriminator is used to identify user voice information from voice data containing noise. During the training, the training process is supervised by the recognition voiceprint feature set corresponding to the recognition sample set with the command sound identification, so as to help the discriminator learn how to distinguish the user's command sound from the background noise, and the training process is analyzed by using the loss function, and the hyperparameters of the discriminator are continuously adjusted, including the learning rate, batch size and number of network layers; until the loss converges, a trained voiceprint denoising recognizer is obtained;

[0022] A3: The loss function is the cross entropy loss function, and the function formula is as follows:

[0023]

[0024] in, is the loss function value, N is the number of samples in the mixed sample set, which is an integer greater than or equal to 1, and y i is the true label of the i-th voiceprint feature in the mixed sample set (1 means that the voiceprint feature comes from the noise sample set, and 0 means that the voiceprint feature comes from the recognition sample set), is the predicted probability of the discriminator for the i-th recognition voiceprint feature from the recognition sample set.

[0025] As a preferred technical solution of the first aspect of the present invention, the acquisition logic of the noise reduction semantic data subsequence;

[0026] In the acquired noise reduction speech data sequence, the start and end time of each speech segment is extracted and recorded to ensure that the subsequent time interval calculation has an accurate start and end point, and the time interval of each two adjacent speech data segments in the noise reduction speech data sequence is calculated, and the time intervals of all adjacent segments are stored in an interval list;

[0027] Set time interval thresholds based on scenarios and user habits, and distinguish between continuous commands and conversation pauses based on the time interval thresholds;

[0028] Traverse the time interval list and compare each interval with the set time interval threshold. If the time interval exceeds the threshold, the point is regarded as a dialogue segmentation point and an independent semantic data subsequence is generated; if it does not exceed the threshold, the speech data is classified into the same subsequence;

[0029] According to the segmentation results, the divided voice data segments are organized into multiple noise-reduced semantic data subsequences, each of which contains a complete voice command and records the corresponding timestamp.

[0030] As a preferred technical solution of the first aspect of the present invention, the acquisition logic of the interaction instruction is:

[0031] Perform semantic analysis on each denoised semantic data subsequence, extract the main speech content and convert it into a processable text form;

[0032] Generate a timestamp for each subsequence, record the time when the voice command is issued, and store it in the corresponding semantic subsequence;

[0033] The interaction time uses the start time or end time of the subsequence, and the appropriate time point is selected according to the needs;

[0034] Compare the semantic contents of adjacent subsequences. If the subsequence contents are repeated, remove the repeated instructions, analyze the content differences of adjacent subsequences through differential algorithms, and extract the new or unique semantic information in each subsequence.

[0035] By using a semantic feature extraction network layer to extract semantic features from the multiple denoised semantic data subsequences, a multiple denoised semantic feature subsequences are obtained, and differential semantic analysis is performed on the multiple denoised semantic data subsequences:

[0036] When the semantic difference is greater than the preset semantic difference, two interaction instructions are respectively generated according to two user intentions reflected by two denoised semantic feature subsequences corresponding to two adjacent denoised semantic data subsequences;

[0037] When the semantic difference is greater than the preset semantic difference, the two user intentions reflected by the two denoised semantic feature subsequences corresponding to the two adjacent denoised semantic data subsequences are fused to generate an interactive instruction, thereby achieving the goal of removing duplicate instructions.

[0038] Determine the differences between instructions, identify each new instruction of the user, organize the independent instructions obtained after differential analysis into an interactive instruction set, and attach the corresponding interaction time. Each instruction corresponds to its timestamp to form a set containing interactive instructions and interaction time.

[0039] In the interactive instruction set, the execution order or priority of the instructions is optimized according to the instruction content and interaction time, and the interactive instruction set after differential semantic analysis and optimization is output to the action execution module to provide clear instruction input for the robot's action plan.

[0040] As a preferred technical solution of the first aspect of the present invention, the determination logic of the target robot action execution scheme is:

[0041] The interactive instructions are arranged in the order of interaction time to form a time-serialized instruction chain. Each instruction corresponds to its interaction time one by one, indicating that the execution follows the order of user intention;

[0042] Perform dependency analysis on adjacent instructions in the instruction sequence. If there is a conflict, retain the instruction with higher priority, or determine the execution instruction based on the time sequence; generate a specific action execution plan based on the classification, priority and dependency analysis results;

[0043] The execution time of each instruction is corrected according to the interaction time, and the execution plan is simulated within the system to detect whether there are unreasonable or unexecutable instructions in the plan, and the plan is adjusted or re-planned; the final action execution plan is output to the robot's control module and prepared for execution.

[0044] In a second aspect, the present invention provides a human-computer interaction system based on voice differential processing, based on the implementation of the first aspect, comprising a background noise extraction module, a voice information extraction module, a voice data processing module, an interactive instruction extraction module, an execution scheme analysis module and a control module, wherein each module is connected via wired or wireless communication:

[0045] A background noise extraction module extracts the speech data sequence received by the target robot within a preset time window, and performs scene feature extraction on the application scene of the target robot to obtain a scene feature set, and constructs a background noise database based on the scene feature set;

[0046] A voice information extraction module collects the user's voice information, combines the electronic device sound information and the guest sound information in the background noise database, performs voiceprint recognition model training, obtains a background voiceprint recognition denoiser, uses the background voiceprint recognition denoiser to perform noise reduction analysis on the scene feature set and the voice data sequence, and obtains a noise-reduced voice data sequence;

[0047] A voice data processing module extracts the time intervals between adjacent noise-reduced voice data in the noise-reduced voice data sequence, performs dialogue segmentation on the noise-reduced voice data sequence according to the extraction result, and determines a plurality of noise-reduced semantic data subsequences;

[0048] An interaction instruction extraction module performs differential semantic analysis on the multiple noise reduction semantic data subsequences to determine multiple interaction instructions, wherein the multiple interaction instructions include multiple interaction times;

[0049] An execution plan analysis module performs an action execution plan analysis on the target robot based on the multiple interaction instructions and the multiple interaction times to determine the action execution plan of the target robot;

[0050] A control module interactively controls the target robot according to the target robot action execution plan.

[0051] In a third aspect, the present invention provides an electronic device, comprising: a processor and a memory, wherein the memory stores a computer program that can be called by the processor;

[0052] The processor executes the first aspect by calling a computer program stored in the memory.

[0053] In a fourth aspect, the present invention provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the first aspect.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The present invention collects voice data and scene features in a specific time period, uses a background voiceprint recognition denoiser to improve the accuracy of voice command recognition in noisy environments, improves the system's processing capability and response efficiency to continuous commands through dialogue segmentation and differential semantic analysis, and simultaneously considers the timing and content of the commands to form a reasonable action execution plan. The intelligence and interactivity of the system are enhanced through real-time interactive control, thereby achieving efficient and precise control of robots in complex environments, significantly improving the adaptability and control accuracy of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of a human-computer interaction method based on speech difference processing in an embodiment of the present invention;

[0057] Figure 2 A human-computer interaction system framework diagram based on speech differential processing in an embodiment of the present invention;

[0058] Figure 3 The figure is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In the description of the present invention, it should be noted that the terms "vertical", "up", "down", "horizontal", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0061] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0062] Example 1

[0063] like Figure 1 As shown, the present invention provides a technical solution: a human-computer interaction method based on speech difference processing, comprising the following steps:

[0064] S1: extracting a speech data sequence received by a target robot within a preset time window, and performing scene feature extraction on an application scene of the target robot to obtain a scene feature set, and constructing a background noise database based on the scene feature set;

[0065] Specifically, according to the actual application needs of the robot (such as home, etc.), set a reasonable time window (for example, 3 seconds or 5 seconds) as the collection period of voice data. In each time window, start the voice collection module and determine the frequency of data collection to obtain enough voice samples in this window. By limiting the duration and frequency of data collection, it is ensured that complete voice data is collected within a fixed time to adapt to a variety of application scenarios.

[0066] Within the preset time window, the microphone is enabled to collect the user's voice data in real time, and the collected audio data is transmitted to the system in the form of digital signals. The collected voice data will be used as the original input for subsequent processing.

[0067] At the same time, the robot's environmental sensors (such as noise sensors, light sensors, distance sensors, etc.) are activated to collect environmental feature data of the current application scenario, and the microphone is activated to collect various background noise data in the home scenario (such as electronic equipment sounds, guest voice information of guests and users, etc.) to obtain a scene feature set. These data include but are not limited to environmental noise levels, background sound features (such as pitch, frequency), light intensity, whether there are obstacles, etc., as well as guest voiceprint information. By capturing the environmental features around the robot, more background information is provided for subsequent noise reduction, distinguishing and identifying user voices, and command analysis, so that the system can adapt to environmental changes in different scenarios.

[0068] Then, the collected scene data is preprocessed, such as removing outliers and normalizing, to ensure the stability of the data. At the same time, based on a specific threshold (such as noise exceeding a certain decibel), features that have a significant impact on speech recognition are screened out. The processed data constitutes a scene feature set.

[0069] Through the above steps, the speech data and scene feature set of the target robot are obtained in each time window, which provides comprehensive data support for subsequent noise reduction processing, command segmentation and differential analysis, enabling the robot to better adapt to user needs and environmental changes in different scenarios.

[0070] S2: Collect the user's voice information, combine it with the electronic device sound information and the guest sound information in the background noise database, perform voiceprint recognition model training, obtain a background voiceprint recognition denoiser, and use the background voiceprint recognition denoiser to perform noise reduction analysis on the scene feature set and the voice data sequence to obtain a noise-reduced voice data sequence;

[0071] It should be noted that S2 uses the background voiceprint recognition noise reducer to effectively filter out environmental noise and the noise caused by conversations between guests and users, ensuring that the robot's voice data processing is accurate in different scenarios, thereby improving its response to user commands.

[0072] Specifically, the construction process of the background voiceprint recognition denoiser is as follows:

[0073] A1: Perform noise identification on the sound information of electronic devices and the sound information of guests in the background noise database to obtain a noise sample set, collect the user's voice information to issue a command for sound identification, obtain a recognition sample set, and mix the noise sample set and the recognition sample set according to a preset ratio (such as 1:1 mixing, 1:2 mixing, etc.) to obtain a mixed sample set.

[0074] More specifically, the mixing ratio is 1:1 mixing or 1:2 mixing;

[0075] A2: Use audio processing technology to traverse the mixed sample set to perform voiceprint feature analysis, and obtain a voiceprint feature sample set to perform initial training on the discriminator, wherein the discriminator is used to identify user voice information from voice data containing noise. During training, the training process is supervised by the recognition voiceprint feature set corresponding to the recognition sample set with the command sound identification, to help the discriminator learn how to distinguish the user's command sound from background noise, and the loss function is used to analyze the training process, and the hyperparameters of the discriminator are continuously adjusted, including the learning rate, batch size and number of network layers; until the loss converges, a trained voiceprint denoising recognizer is obtained.

[0076] A3: The loss function is the cross entropy loss function, and the function formula is as follows:

[0077]

[0078] in, is the loss function value, N is the number of samples in the mixed sample set, which is an integer greater than or equal to 1, and y i is the true label of the i-th voiceprint feature in the mixed sample set (1 means that the voiceprint feature comes from the noise sample set, and 0 means that the voiceprint feature comes from the recognition sample set), is the predicted probability of the discriminator for the i-th recognition voiceprint feature from the recognition sample set.

[0079] Through the above steps, the neural network-based background noise reducer can learn and eliminate background noise from scene features and voice data, provide high-quality audio input for the robot's voice command system, improve the accuracy of the robot's execution of actions, and enhance the quality of human-computer interaction.

[0080] S3: extracting the time intervals between adjacent noise-reduced speech data in the noise-reduced speech data sequence, performing dialogue segmentation on the noise-reduced speech data sequence according to the extraction result, and determining a plurality of noise-reduced semantic data subsequences;

[0081] Specifically, in the acquired noise reduction speech data sequence, each segment of speech data has a timestamp. First, extract and record the start and end time of each segment of speech to ensure that the calculation of subsequent time intervals has an accurate start and end point. Thus, basic data is provided for the calculation of subsequent time intervals, so that the system can accurately determine the intervals between speech segments. The time interval is calculated for every two adjacent segments of speech data in the noise reduction speech data sequence. The calculation formula is: time interval = start time of the current speech segment - end time of the previous speech segment. The time intervals of all adjacent segments are stored in an interval list for subsequent processing.

[0082] Set the time interval threshold (for example, 0.5 seconds or 1 second) according to the scenario and user habits. This threshold is used to distinguish continuous commands from pauses in the conversation. The threshold can be adjusted dynamically according to the application scenario. For example, a shorter threshold may be set in a noisy environment to reduce mis-segmentation. Thus, the segmentation criteria for voice commands are determined, and the accuracy of segmentation is ensured by setting the threshold to avoid mis-segmentation of continuous commands.

[0083] Traverse the time interval list and compare each interval with the set threshold. If the time interval exceeds the threshold, the point is regarded as a dialogue segmentation point and an independent semantic data subsequence is generated; if it does not exceed the threshold, the voice data is classified into the same subsequence. In this way, independent voice command segments are segmented to ensure that each subsequence corresponds to a complete instruction or sentence, laying the foundation for subsequent semantic analysis. According to the segmentation results, the divided voice data segments are organized into multiple noise-reduced semantic data subsequences. Each subsequence contains a complete voice command and records the corresponding timestamp for subsequent time sequence processing.

[0084] Verify the initially segmented subsequence set to ensure that the segmentation points match the natural pauses of the voice command. If too many or insufficient segmentations are found during the verification process, the time interval threshold can be adjusted to further improve the rationality of the dialogue segmentation.

[0085] Through the above steps, the user's voice pause points can be accurately identified in the noise reduction voice data sequence, and the voice data can be reasonably divided into independent semantic subsequences to ensure the robot's accurate response to each command, thereby improving the system's multi-command processing capabilities and user experience.

[0086] S4: performing differential semantic analysis on the multiple noise reduction semantic data subsequences to determine multiple interaction instructions, wherein the multiple interaction instructions include multiple interaction times;

[0087] Semantic analysis is performed on each denoised semantic data subsequence to extract the main speech content and convert it into a processable text form. A natural language processing (NLP) model is used to convert the speech data in the subsequence into text instructions for subsequent semantic differential analysis.

[0088] Generate a timestamp for each subsequence, record the time when the voice command is issued, and store it in the corresponding semantic subsequence. The interaction time can be the start time or end time of the subsequence, and the appropriate time point can be selected according to the needs. Compare the semantic content of adjacent subsequences to check whether the content is the same or highly similar. If the subsequence content is repeated, remove the repeated instructions (such as the user's slip of the tongue, resulting in the robot being given the same instruction twice) to ensure that the system only processes independent instructions that are meaningful. Reduce the error of issuing the same instruction multiple times, improve system efficiency, and avoid erroneous operations caused by repeated instructions.

[0089] The difference algorithm is used to analyze the content differences of adjacent subsequences to extract new or unique semantic information in each subsequence. The semantic features (word vectors or sentence vectors) of the multiple denoised semantic data subsequences are extracted by using a semantic feature extraction network layer to obtain multiple denoised semantic feature subsequences.

[0090] Furthermore, differential semantic analysis is performed on the multiple denoised semantic data subsequences in order to identify and compare the semantic differences of the text or speech elements (such as words, phrases or sentences) in two adjacent denoised semantic data subsequences. This processing method usually involves comparing different inputs to find their differences in meaning, emotion or other characteristics. This makes it easier to understand the user's intention based on the sequential semantic information and improve the accuracy of the robot's execution of actions.

[0091] Preferably, a differential processing formula is used to perform differential semantic analysis on the multiple noise reduction semantic feature subsequences.

[0092] The differential processing formula is: ΔI = I n -I n-1 ΔI is the semantic difference between two adjacent denoised semantic data subsequences, I n is the semantic feature of the nth denoised semantic data subsequence, I n-1 is the semantic feature of the n-1th denoised semantic data subsequence.

[0093] When the semantic difference is greater than the preset semantic difference, two interaction instructions are respectively generated according to two user intentions reflected by two denoised semantic feature subsequences corresponding to two adjacent denoised semantic data subsequences;

[0094] When the semantic difference is greater than the preset semantic difference, the two user intentions reflected by the two denoised semantic feature subsequences corresponding to the two adjacent denoised semantic data subsequences are fused to generate an interactive instruction, thereby achieving the goal of removing duplicate instructions.

[0095] Determine the differences between instructions and identify each new instruction from the user to ensure that the system accurately captures changes in user needs. Organize the independent instructions obtained after differential analysis into an interactive instruction set and attach the corresponding interaction time. Each instruction corresponds to its timestamp to form a set of interactive instructions and interaction time. In this way, the independence and time consistency of each instruction are ensured, providing a clear and complete instruction chain for subsequent robot action execution.

[0096] In the interactive instruction set, the execution order or priority of the instructions is optimized according to the instruction content and interaction time. For example, if the user issues a "stop" instruction, the system will increase the priority of the "stop" instruction to ensure timely response to emergency instructions. The interactive instruction set after differential semantic analysis and optimization is output to the action execution module to provide clear instruction input for the robot's action plan.

[0097] Through the above steps, the system can accurately extract the user's independent instructions from multiple noise-reduced semantic data subsequences, remove repeated or invalid instructions, and form a time-ordered interactive instruction set, so that the robot can quickly respond to diverse voice needs and enhance the intelligent interactive experience.

[0098] S5: analyzing the action execution plan of the target robot based on the multiple interaction instructions and the multiple interaction times, and determining the action execution plan of the target robot;

[0099] Arrange the interactive instructions in the order of interaction time to form a time-serialized instruction chain. Each instruction corresponds to its interaction time one by one to ensure sequential consistency so that the system follows the order of user intent when executing. Categorize each instruction in the instruction sequence (such as move, stop, turn, etc.) and assign different priorities based on the content. For example, the "stop" instruction usually has the highest priority, and the "forward" or "turn" instructions have the next priority. Prioritize the instructions according to their importance to handle conflicting or urgent instructions in complex scenarios.

[0100] Perform dependency analysis on adjacent instructions in the instruction sequence. For example, "forward" followed by "stop" are consecutive instructions, but "forward" and "backward" need to be detected for conflicts. If there is a conflict, keep the instruction with higher priority, or determine the execution instruction based on the time sequence. Generate a specific action execution plan based on the classification, priority and dependency analysis results. For example, if the instruction sequence is "forward->turn->stop", it is converted into the robot's motion path planning and executed in the order of "forward->turn->stop".

[0101] Correct the execution time of each instruction according to the interaction time. For example, if the time interval between two instructions is short, the robot needs to speed up the execution; if the interval is long, the system can delay appropriately during execution. This process ensures that the rhythm of instruction execution is synchronized with the frequency of user instructions. Simulate the execution plan within the system to detect whether there are unreasonable or unexecutable instructions in the plan, such as route intersections, time overlaps, etc. If there are problems, adjust or re-plan the plan. Output the final generated action execution plan to the robot's control module and prepare for execution. The output plan should include the content, execution order and time interval information of each instruction. Provide clear operating instructions for the robot control system to ensure that the robot completes the corresponding actions according to user requirements.

[0102] Through the above steps, the system can generate a logically coherent and prioritized action execution plan based on user interaction instructions and time arrangements, ensuring that the robot can efficiently respond to user needs in complex environments and provide a smooth intelligent interaction experience.

[0103] S6: interactively controlling the target robot according to the target robot action execution plan.

[0104] Before executing the action plan, start the robot's control module, including the drive motor, direction control module, sensor monitoring module, etc. Ensure that all hardware components are working properly and establish a real-time communication connection with the central control system. From the generated action execution plan, read the first action instruction and its corresponding time parameters in sequence, load them into the execution queue of the control module, and prepare to start execution. The control module executes the specific operations of the robot according to the loaded action instructions, such as moving forward, turning, stopping, etc. During the execution process, the robot's sensors will feedback position information, direction, speed and other status data to the central control system in real time.

[0105] During the execution of the action, the robot's operating status is continuously monitored to check for abnormal conditions (such as obstacles, tilt or direction deviation, etc.). If an abnormality is detected, the interrupt mechanism or adjustment plan is triggered immediately. According to the time parameters in the execution plan, the next instruction is loaded to the control module at the instruction interval to keep the execution rhythm consistent with the frequency of user instructions. If the previous instruction is not completed on time, the next instruction is loaded or the execution speed is adjusted. After each instruction is completed, the control module sends a completion signal to the central control system to mark the completion of the instruction. After receiving the signal, the system loads the next instruction and continues to control the robot to execute until all instructions in the action execution plan are completed.

[0106] When all instructions are executed, the system generates summary feedback, including the robot's running trajectory, instruction response, etc., and notifies the user through voice or screen that the instruction has been successfully executed. Feedback the execution results to the user to improve the interactive experience and let the user understand the robot's execution status and completion. After all action instructions are completed, the control module automatically returns to standby mode and waits for the user's next instruction input. If there is a next task instruction, the system will automatically enter a new control process. Ensure that the system is on standby at all times to improve user convenience. Through the above steps, the system can accurately interactively control the robot based on the action execution plan, so that the robot can complete continuous actions under the user's instructions, providing an efficient and stable intelligent control experience.

[0107] Example 2

[0108] See also Figure 2 The human-computer interaction system based on speech differential processing includes a background noise extraction module, a speech information extraction module, a speech data processing module, an interaction instruction extraction module, an execution scheme analysis module and a control module. Each module is connected by wire or wireless:

[0109] A background noise extraction module extracts the speech data sequence received by the target robot within a preset time window, and performs scene feature extraction on the application scene of the target robot to obtain a scene feature set, and constructs a background noise database based on the scene feature set;

[0110] A voice information extraction module collects the user's voice information, combines the electronic device sound information and the guest sound information in the background noise database, performs voiceprint recognition model training, obtains a background voiceprint recognition denoiser, uses the background voiceprint recognition denoiser to perform noise reduction analysis on the scene feature set and the voice data sequence, and obtains a noise-reduced voice data sequence;

[0111] A voice data processing module extracts the time intervals between adjacent noise-reduced voice data in the noise-reduced voice data sequence, performs dialogue segmentation on the noise-reduced voice data sequence according to the extraction result, and determines a plurality of noise-reduced semantic data subsequences;

[0112] An interaction instruction extraction module performs differential semantic analysis on the multiple noise reduction semantic data subsequences to determine multiple interaction instructions, wherein the multiple interaction instructions include multiple interaction times;

[0113] An execution plan analysis module performs an action execution plan analysis on the target robot based on the multiple interaction instructions and the multiple interaction times to determine the action execution plan of the target robot;

[0114] A control module interactively controls the target robot according to the target robot action execution plan.

[0115] Preferably, a preset time window is configured according to the actual application requirements of the robot, the preset time window is used as a collection period for voice data, and voice samples are obtained based on the preset time window.

[0116] Preferably, the logic for acquiring the scene feature set is:

[0117] Within the preset time window, the microphone is enabled to collect the user's voice data in real time, and the collected audio data is transmitted in the form of digital signals;

[0118] At the same time, the robot's environmental sensor is activated to obtain environmental feature data of the current application scenario based on the environmental sensor, and the microphone is activated to collect background noise data in the home scenario. The background noise data includes the sound of electronic equipment and the sound information of guests, so as to obtain scene data;

[0119] The collected scene data is preprocessed, including removing outliers and normalizing, filtering out speech recognition scene features based on a specific threshold, and combining the scene features into a scene feature set.

[0120] Preferably, the construction process of the background voiceprint recognition noise reducer is as follows:

[0121] A1: noise identification is performed on the sound information of the electronic device and the sound information of the guest in the background noise database to obtain a noise sample set, the voice information of the user is collected to perform a command to issue a sound identification to obtain a recognition sample set, and the noise sample set and the recognition sample set are mixed according to a preset ratio to obtain a mixed sample set;

[0122] A2: Use audio processing technology to traverse the mixed sample set to perform voiceprint feature analysis, and obtain a voiceprint feature sample set to perform initial training on the discriminator, wherein the discriminator is used to identify user voice information from voice data containing noise. During the training, the training process is supervised by the recognition voiceprint feature set corresponding to the recognition sample set with the command sound identification, so as to help the discriminator learn how to distinguish the user's command sound from the background noise, and the training process is analyzed by using the loss function, and the hyperparameters of the discriminator are continuously adjusted, including the learning rate, batch size and number of network layers; until the loss converges, a trained voiceprint denoising recognizer is obtained;

[0123] A3: The loss function is the cross entropy loss function, and the function formula is as follows:

[0124]

[0125] in, is the loss function value, N is the number of samples in the mixed sample set, which is an integer greater than or equal to 1, and yi is the true label of the i-th voiceprint feature in the mixed sample set (1 means that the voiceprint feature comes from the noise sample set, and 0 means that the voiceprint feature comes from the recognition sample set), is the predicted probability of the discriminator for the i-th recognition voiceprint feature from the recognition sample set.

[0126] Preferably, the acquisition logic of the noise-reduced semantic data subsequence;

[0127] In the acquired noise reduction speech data sequence, the start and end time of each speech segment is extracted and recorded to ensure that the subsequent time interval calculation has an accurate start and end point, and the time interval of each two adjacent speech data segments in the noise reduction speech data sequence is calculated, and the time intervals of all adjacent segments are stored in an interval list;

[0128] Set time interval thresholds based on scenarios and user habits, and distinguish between continuous commands and conversation pauses based on the time interval thresholds;

[0129] Traverse the time interval list and compare each interval with the set time interval threshold. If the time interval exceeds the threshold, the point is regarded as a dialogue segmentation point and an independent semantic data subsequence is generated; if it does not exceed the threshold, the speech data is classified into the same subsequence;

[0130] According to the segmentation results, the divided voice data segments are organized into multiple noise-reduced semantic data subsequences, each of which contains a complete voice command and records the corresponding timestamp.

[0131] Preferably, the acquisition logic of the interaction instruction:

[0132] Perform semantic analysis on each denoised semantic data subsequence, extract the main speech content and convert it into a processable text form;

[0133] Generate a timestamp for each subsequence, record the time when the voice command is issued, and store it in the corresponding semantic subsequence;

[0134] The interaction time uses the start time or end time of the subsequence, and the appropriate time point is selected according to the needs;

[0135] Compare the semantic contents of adjacent subsequences. If the subsequence contents are repeated, remove the repeated instructions, analyze the content differences of adjacent subsequences through differential algorithms, and extract the new or unique semantic information in each subsequence.

[0136] By using a semantic feature extraction network layer to extract semantic features from the multiple denoised semantic data subsequences, a multiple denoised semantic feature subsequences are obtained, and differential semantic analysis is performed on the multiple denoised semantic data subsequences:

[0137] When the semantic difference is greater than the preset semantic difference, two interaction instructions are respectively generated according to two user intentions reflected by two denoised semantic feature subsequences corresponding to two adjacent denoised semantic data subsequences;

[0138] When the semantic difference is greater than the preset semantic difference, the two user intentions reflected by the two denoised semantic feature subsequences corresponding to the two adjacent denoised semantic data subsequences are fused to generate an interactive instruction, thereby achieving the goal of removing duplicate instructions.

[0139] Determine the differences between instructions, identify each new instruction of the user, organize the independent instructions obtained after differential analysis into an interactive instruction set, and attach the corresponding interaction time. Each instruction corresponds to its timestamp to form a set containing interactive instructions and interaction time.

[0140] In the interactive instruction set, the execution order or priority of the instructions is optimized according to the instruction content and interaction time, and the interactive instruction set after differential semantic analysis and optimization is output to the action execution module to provide clear instruction input for the robot's action plan.

[0141] Preferably, the determination logic of the target robot action execution scheme is:

[0142] The interactive instructions are arranged in the order of interaction time to form a time-serialized instruction chain. Each instruction corresponds to its interaction time one by one, indicating that the execution follows the order of user intention;

[0143] Perform dependency analysis on adjacent instructions in the instruction sequence. If there is a conflict, retain the instruction with higher priority, or determine the execution instruction based on the time sequence; generate a specific action execution plan based on the classification, priority and dependency analysis results;

[0144] The execution time of each instruction is corrected according to the interaction time, and the execution plan is simulated within the system to detect whether there are unreasonable or unexecutable instructions in the plan, and the plan is adjusted or re-planned; the final action execution plan is output to the robot's control module and prepared for execution.

[0145] Example 3

[0146] An electronic device according to an exemplary embodiment includes: a processor and a memory, wherein the memory stores a computer program that can be called by the processor;

[0147] The processor executes the above-mentioned human-computer interaction method based on speech difference processing by calling the computer program stored in the memory.

[0148] Figure 3It is a structural diagram of an electronic device provided by an embodiment of the present application. The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) and one or more memories, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the human-computer interaction method based on voice differential processing provided by the above-mentioned various method embodiments. The electronic device may also include other components for realizing the functions of the device. For example, the electronic device may also have components such as a wired or wireless network interface and an input and output interface for input and output. The embodiments of the present application are not described in detail here.

[0149] Example 4

[0150] According to an exemplary embodiment, a computer-readable storage medium is shown, on which a rewritable computer program is stored;

[0151] When the computer program is executed on a computer device, the computer device is enabled to execute the above-mentioned human-computer interaction method based on speech difference processing.

[0152] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer program, and the at least one computer program is executable by a processor to complete the human-computer interaction method based on speech difference processing in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0153] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program includes one or more program codes, and the one or more program codes are stored in a computer-readable storage medium. One or more processors of an electronic device can read the one or more program codes from the computer-readable storage medium, and the one or more processors execute the one or more program codes, so that the electronic device can perform the above-mentioned human-computer interaction method based on speech differential processing.

[0154] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0155] It should be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.

[0156] A person of ordinary skill in the art will understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0157] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0158] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only one, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0159] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0161] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A human-computer interaction method based on speech differential processing, characterized in that: The following steps are involved: S1: extracting a speech data sequence received by a target robot within a preset time window, and performing scene feature extraction on an application scene of the target robot to obtain a scene feature set, and constructing a background noise database based on the scene feature set; S2: Collect the user's voice information, combine it with the electronic device sound information and the guest sound information in the background noise database, perform voiceprint recognition model training, obtain a background voiceprint recognition denoiser, and use the background voiceprint recognition denoiser to perform noise reduction analysis on the scene feature set and the voice data sequence to obtain a noise-reduced voice data sequence; S3: extracting the time intervals between adjacent noise-reduced speech data in the noise-reduced speech data sequence, performing dialogue segmentation on the noise-reduced speech data sequence according to the extraction result, and determining a plurality of noise-reduced semantic data subsequences; S4: performing differential semantic analysis on the multiple noise reduction semantic data subsequences to determine multiple interaction instructions, where the multiple interaction instructions include multiple interaction times; S5: analyzing the action execution plan of the target robot based on the multiple interaction instructions and the multiple interaction times, and determining the action execution plan of the target robot; S6: interactively controlling the target robot according to the target robot action execution plan.

2. The human-computer interaction method based on speech differential processing according to claim 1, characterized in that: Configure a preset time window according to the actual application requirements of the robot, use the preset time window as the collection period of voice data, and obtain voice samples based on the preset time window.

3. The human-computer interaction method based on speech differential processing according to claim 1, characterized in that: The logic for obtaining the scene feature set is as follows: Within the preset time window, the microphone is enabled to collect the user's voice data in real time, and the collected audio data is transmitted in the form of digital signals; At the same time, the robot's environmental sensor is activated to obtain environmental feature data of the current application scenario based on the environmental sensor, and the microphone is activated to collect background noise data in the home scenario. The background noise data includes the sound of electronic equipment and the sound information of guests, so as to obtain scene data; The collected scene data is preprocessed, including removing outliers and normalizing, filtering out speech recognition scene features based on a specific threshold, and combining the scene features into a scene feature set.

4. The human-computer interaction method based on speech differential processing according to claim 1, characterized in that: The construction process of the background voiceprint recognition denoiser is as follows: A1: noise identification is performed on the sound information of the electronic device and the sound information of the guest in the background noise database to obtain a noise sample set, the voice information of the user is collected to perform a command to issue a sound identification to obtain a recognition sample set, and the noise sample set and the recognition sample set are mixed according to a preset ratio to obtain a mixed sample set; A2: Use audio processing technology to traverse the mixed sample set to perform voiceprint feature analysis, and obtain a voiceprint feature sample set to perform initial training on the discriminator, wherein the discriminator is used to identify user voice information from voice data containing noise. During the training, the training process is supervised by the recognition voiceprint feature set corresponding to the recognition sample set with the command sound identification, so as to help the discriminator learn how to distinguish the user's command sound from the background noise, and the training process is analyzed by using the loss function, and the hyperparameters of the discriminator are continuously adjusted, including the learning rate, batch size and number of network layers; until the loss converges, a trained voiceprint denoising recognizer is obtained; A3: The loss function is the cross entropy loss function, and the function formula is as follows: in, is the loss function value, N is the number of samples in the mixed sample set, which is an integer greater than or equal to 1, and y i is the true label of the i-th voiceprint feature in the mixed sample set (1 means that the voiceprint feature comes from the noise sample set, and 0 means that the voiceprint feature comes from the recognition sample set), is the predicted probability of the discriminator for the i-th recognition voiceprint feature from the recognition sample set.

5. The human-computer interaction method based on speech difference processing according to claim 1, characterized in that: The acquisition logic of the noise reduction semantic data subsequence; In the acquired noise reduction speech data sequence, the start and end time of each speech segment is extracted and recorded to ensure that the subsequent time interval calculation has an accurate start and end point, and the time interval of each two adjacent speech data segments in the noise reduction speech data sequence is calculated, and the time intervals of all adjacent segments are stored in an interval list; Set time interval thresholds based on scenarios and user habits, and distinguish between continuous commands and conversation pauses based on the time interval thresholds; Traverse the time interval list and compare each interval with the set time interval threshold. If the time interval exceeds the threshold, the point is regarded as a dialogue segmentation point and an independent semantic data subsequence is generated; if it does not exceed the threshold, the speech data is classified into the same subsequence; According to the segmentation results, the divided voice data segments are organized into multiple noise-reduced semantic data subsequences, each of which contains a complete voice command and records the corresponding timestamp.

6. The human-computer interaction method based on speech differential processing according to claim 2, characterized in that: The acquisition logic of the interaction instruction: Perform semantic analysis on each denoised semantic data subsequence, extract the main speech content and convert it into a processable text form; Generate a timestamp for each subsequence, record the time when the voice command is issued, and store it in the corresponding semantic subsequence; The interaction time uses the start time or end time of the subsequence, and the appropriate time point is selected according to the needs; Compare the semantic contents of adjacent subsequences. If the subsequence contents are repeated, remove the repeated instructions, analyze the content differences of adjacent subsequences through differential algorithms, and extract the new or unique semantic information in each subsequence. By using a semantic feature extraction network layer to extract semantic features from the multiple denoised semantic data subsequences, a multiple denoised semantic feature subsequences are obtained, and differential semantic analysis is performed on the multiple denoised semantic data subsequences: When the semantic difference is greater than the preset semantic difference, two interaction instructions are respectively generated according to two user intentions reflected by two denoised semantic feature subsequences corresponding to two adjacent denoised semantic data subsequences; When the semantic difference is greater than the preset semantic difference, the two user intentions reflected by the two denoised semantic feature subsequences corresponding to the two adjacent denoised semantic data subsequences are fused to generate an interactive instruction, thereby achieving the goal of removing duplicate instructions. Determine the differences between instructions, identify each new instruction of the user, organize the independent instructions obtained after differential analysis into an interactive instruction set, and attach the corresponding interaction time. Each instruction corresponds to its timestamp to form a set containing interactive instructions and interaction time. In the interactive instruction set, the execution order or priority of the instructions is optimized according to the instruction content and interaction time, and the interactive instruction set after differential semantic analysis and optimization is output to the action execution module to provide clear instruction input for the robot's action plan.

7. The human-computer interaction method based on speech difference processing according to claim 4 is characterized in that: The determination logic of the target robot action execution plan: The interactive instructions are arranged in the order of interaction time to form a time-serialized instruction chain. Each instruction corresponds to its interaction time one by one, indicating that the execution follows the order of user intention; Perform dependency analysis on adjacent instructions in the instruction sequence. If there is a conflict, retain the instruction with higher priority, or determine the execution instruction based on the time sequence; generate a specific action execution plan based on the classification, priority and dependency analysis results; The execution time of each instruction is corrected according to the interaction time, and the execution plan is simulated within the system to detect whether there are unreasonable or unexecutable instructions in the plan, and the plan is adjusted or re-planned; the final action execution plan is output to the robot's control module and prepared for execution.

8. A human-computer interaction system based on speech differential processing, based on the implementation of the human-computer interaction method based on speech differential processing according to any one of claims 1 to 7, characterized in that: It includes background noise extraction module, voice information extraction module, voice data processing module, interactive instruction extraction module, execution plan analysis module and control module. Each module is connected by wire or wireless: A background noise extraction module extracts the speech data sequence received by the target robot within a preset time window, and performs scene feature extraction on the application scene of the target robot to obtain a scene feature set, and constructs a background noise database based on the scene feature set; A voice information extraction module collects the user's voice information, combines the electronic device sound information and the guest sound information in the background noise database, performs voiceprint recognition model training, obtains a background voiceprint recognition denoiser, uses the background voiceprint recognition denoiser to perform noise reduction analysis on the scene feature set and the voice data sequence, and obtains a noise-reduced voice data sequence; A voice data processing module extracts the time intervals between adjacent noise-reduced voice data in the noise-reduced voice data sequence, performs dialogue segmentation on the noise-reduced voice data sequence according to the extraction result, and determines a plurality of noise-reduced semantic data subsequences; An interaction instruction extraction module performs differential semantic analysis on the multiple noise reduction semantic data subsequences to determine multiple interaction instructions, wherein the multiple interaction instructions include multiple interaction times; An execution plan analysis module performs an action execution plan analysis on the target robot based on the multiple interaction instructions and the multiple interaction times to determine the action execution plan of the target robot; A control module interactively controls the target robot according to the target robot action execution plan.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores a computer program that can be called by the processor; The processor executes the human-computer interaction method based on speech difference processing as described in any one of claims 1 to 7 by calling the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: Instructions are stored, and when the instructions are executed on a computer, the computer executes the human-computer interaction method based on speech difference processing as described in any one of claims 1 to 7.

Images