Action determination method, device, computer device, readable storage medium and program product

Abstract

The present application relates to an action determination method, apparatus, computer device, computer-readable storage medium, and computer program product. The method comprises: receiving an audio signal; converting the audio signal into text information; if the text information does not include an action instruction, generating a response message corresponding to the text information, and determining a target action corresponding to the response message from a preset action library; executing the target action and outputting the response message. This method can accurately determine the target action.

Description

Technical Field

[0001] The present application relates to the field of large model technology, and in particular to an action determination method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of artificial intelligence technology, the application of robots in different scenarios is gradually increasing, especially in service, education, medical and other scenarios. Robots not only need to meet basic task execution, but also need to have flexible and personalized response behaviors that meet user expectations to determine the corresponding user needs.

[0003] Currently, corresponding actions are executed through a predefined action library. How to select the appropriate action from the action library is a problem to be solved. Summary of the Invention

[0004] Based on this, it is necessary to provide an action determination method, device, computer equipment, computer-readable storage medium and computer program product that can accurately determine the robot's target execution action from the dialogue to address the above technical problems.

[0005] In a first aspect, the present application provides an action determination method, the method comprising:

[0006] receiving audio signals;

[0007] Converting the audio signal into text information;

[0008] If the text message does not include an action instruction, generating a response message corresponding to the text message, and determining a target execution action corresponding to the response message from a preset action library;

[0009] The target execution action is executed, and the response information is output.

[0010] In one embodiment, the process of determining whether the text information includes the action instruction includes:

[0011] If the text information does not conform to the preset sentence structure, analyzing the text information to identify verbs or task-related words in the text information;

[0012] Filtering historical interaction information through a preset window to obtain target interaction information related to the verb or the task-related word;

[0013] The user intention is obtained according to the verb or the task-related word and the target interaction information; the user intention is used to determine whether the text information includes the action instruction.

[0014] In one embodiment, the method further comprises:

[0015] If the text information includes an action instruction, the target execution action is determined by matching the verb in the preset action library.

[0016] In one embodiment, generating answer information corresponding to the text information and determining a target execution action corresponding to the answer information from a preset action library includes:

[0017] generating the answer information according to the user intention;

[0018] Inputting the answer information into a pre-trained multi-classification model to obtain a classification result;

[0019] The target execution action is determined by matching the classification result with the preset action library.

[0020] In one embodiment, the training process of the multi-classification model includes:

[0021] Acquire first sample data; the first sample data carries an action instruction tag;

[0022] Preprocessing the first sample data to obtain multiple word segments;

[0023] The first sample data is input into a first initial model for training to obtain a prediction classification result; the first initial model includes multiple instruction classification heads and a fully connected layer, each instruction classification head includes an input layer, an encoding layer, and a pooling layer; the input layer converts the word segmentation into a word vector and passes the word vector to the encoding layer; the encoding layer obtains the initial features of each word vector through a self-attention mechanism and passes the initial features to the pooling layer; the pooling layer compresses the initial features to obtain target features; the fully connected layer merges the target features corresponding to each classification head to obtain the prediction classification result;

[0024] The predicted classification result is compared with the action instruction label to obtain a first difference, and the parameters of the first initial model are adjusted according to the first difference until the first initial model is trained to obtain the multi-classification model.

[0025] In one embodiment, the conversion of the audio signal into text information is performed by a pre-trained speech recognition model; the training process of the speech recognition model includes:

[0026] Acquire second sample data; the second sample data carries a text label;

[0027] adding background noise to the second sample data; the background noise includes one or more of television playback sound, kitchen sound, and conversation sound;

[0028] Inputting the second sample data carrying the background noise into a second initial model for training to obtain predicted text data;

[0029] A second difference between the predicted text data and the text label is compared, and parameters of the second initial model are adjusted according to the second difference until the second initial model is trained to obtain the speech recognition model.

[0030] In a second aspect, the present application provides an action determination device, the device comprising:

[0031] A receiving module, configured to receive an audio signal;

[0032] A text conversion module, configured to convert the audio signal into text information;

[0033] an action determination module, configured to generate answer information corresponding to the text information if the text information does not include an action instruction, and determine a target execution action corresponding to the answer information;

[0034] The execution module is used to execute the target execution action and output the response information.

[0035] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0036] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0037] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0038] The above-mentioned action determination method, apparatus, computer device, computer-readable storage medium and computer program product, when determining that the text information does not include action instructions, infer user needs based on semantic analysis and context understanding, and generate corresponding answer information, and efficiently match the corresponding target execution action from the preset action library based on the answer information. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 1 is a flow chart of an action determination method according to an embodiment;

[0041] Figure 2 is a flowchart of an action determination method in another embodiment;

[0042] Figure 3 is a structural block diagram of an action determination device in one embodiment;

[0043] Figure 4 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0045] In one embodiment, Figure 1 As shown, a method for determining an action is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0046] Step S102: receiving an audio signal.

[0047] Optionally, an audio signal from the external environment may be received by an audio acquisition device, wherein the audio signal may be a user's voice command, background noise, or other environmental sounds.

[0048] Furthermore, to ensure the system can accurately interpret voice commands from mixed information, the system is typically equipped with a noise suppression module to remove or reduce the interference of ambient noise on speech recognition. This module uses microphone array technology and beamforming algorithms to focus on receiving sound signals from a specific direction, thereby reducing the interference of background noise.

[0049] Optionally, the robot can be equipped with a digital signal processor (DSP) or a dedicated speech recognition chip. These hardware modules are responsible for preprocessing and noise reduction, ensuring that the quality of the incoming speech signal meets the requirements of speech recognition.

[0050] Optionally, the audio signal is first collected by a microphone array, amplified by a preamplifier, and converted to a digital signal by an ADC (analog-to-digital converter). The digital signal then enters a processing unit (such as a DSP chip) for preliminary noise cancellation and echo suppression. Finally, the processed signal is transmitted to the speech recognition module to be converted into text information.

[0051] Step S104: convert the audio signal into text information.

[0052] Among them, text information is text data converted from audio signals through voice recognition technology, including the content of voice commands.

[0053] Optionally, the received audio signal is processed by a speech recognition model and converted into text information. The speech recognition model can be trained based on a deep learning model, such as a convolutional neural network or a recurrent neural network, to recognize different pronunciations and speech features.

[0054] Optionally, before the audio signal enters the speech recognition model, a series of preprocessing steps will be performed, such as denoising, speech segmentation, feature extraction, etc.

[0055] Optionally, the speech recognition model can extract features of the audio signal through methods such as MFCC (Mel-Frequency Cepstral Coefficients), then use a long short-term memory (LSTM) or Transformer model for language modeling, and finally output text content.

[0056] Alternatively, assuming the audio signal is "Turn on the living room light," the speech recognition module first converts the frequency and time domain features of the speech signal into Mel-Frequency Cepstral Coefficients (MFCC) features and inputs them into a deep neural network. The neural network extracts high-dimensional features through convolutional layers, then performs sequence modeling using a recurrent neural network (RNN) or Transformer, ultimately outputting the corresponding text: "Turn on the living room light."

[0057] Step S106: If the text information does not include an action instruction, generate answer information corresponding to the text information, and determine a target execution action corresponding to the answer information.

[0058] An action command is a user-generated instruction, issued through voice, text, or other input methods, that instructs the robot to perform a specific task. An action command not only specifies the action the robot needs to perform, such as "open," "close," or "move," but also includes detailed task information related to the action, such as the object, location, and time.

[0059] Among them, the target execution action refers to the robot determining the specific tasks or actions that need to be performed based on the analysis of text information, such as controlling the robotic arm, making sounds, etc.

[0060] Alternatively, whether the action instruction is included may be determined by a pre-trained judgment model, wherein the judgment model is a binary classification model, and its output result is yes or no.

[0061] Furthermore, the judgment model can judge whether the text information includes action instructions through context semantics. For details, please refer to the description in the following embodiments.

[0062] Step S108: Execute the target action and output response information.

[0063] Optionally, once the target action is determined, the system will execute the specific task through a mechanical control system, such as motor drive, sensor feedback, etc. For example, if the target action is to "turn on the light," the system will control the power switch through a relay; if it is to "move an object," it will be completed by a robotic arm.

[0064] Furthermore, when executing a target action, a robot typically includes multiple modules, such as motion control, speech, and perception. Each module works together to ensure the smooth execution of the task. For example, the motion control module adjusts the robot arm's movements according to instructions, while the speech module is responsible for generating and outputting response information.

[0065] Alternatively, tasks may need to be split into multiple subtasks that can be executed in parallel. For example, if a robot needs to answer a question and perform a target action simultaneously, the system may need to send the answer portion of the task to the language generation module and the action execution portion to the control module, completing the tasks in parallel.

[0066] In the above-mentioned action determination method, when it is determined that the text information does not include action instructions, the user needs are inferred based on semantic analysis and context understanding, and corresponding answer information is generated, and the corresponding target execution action is efficiently matched from the preset action library based on the answer information.

[0067] Furthermore, in one embodiment, the process of determining whether the above-mentioned text information includes action instructions includes: if the text information does not conform to a preset sentence structure, analyzing the text information to identify verbs or task-related words in the text information; filtering historical interaction information through a preset window to obtain target interaction information; obtaining user intention based on verbs or task-related words, and target interaction information; and using user intention to determine whether the text information includes action instructions.

[0068] The preset sentence structure refers to a predefined sentence structure template based on a previously trained model or rule library, which is used to identify core elements and action instructions in the text.

[0069] For example, if the scenario is family, the preset sentence structure can be as follows:

[0070] Sentence structure 1: please [verb] [task], such as "Please open the door"

[0071] Sentence structure 2: [verb] [object], such as "close the window"

[0072] Task-related words are keywords directly related to the task being performed. They describe the object, goal, or context of the action. For example, in a smart home scenario, if a user issues a command like "turn up the temperature," task-related words would include "turn up" and "temperature." Because users have engaged in multiple conversations or established context, the context can be used to infer the object being turned up.

[0073] The preset window refers to the time range within which the system filters historical interaction information when determining whether a text message contains an action instruction. This window helps the system better understand the user's intent, especially in multi-turn conversation scenarios. Due to the time limit, the system only considers historical interaction information within a certain time range. For example, if a user recently requested "turn on the air conditioner" and then said "the temperature is a bit low" or "raise the temperature," the system will filter the target interaction information "turn on the air conditioner" from the historical interaction information based on the preset window and infer that the user may want to adjust the air conditioner temperature. The target interaction information refers to the key information related to the current task that is filtered from the historical interactions. Target interaction information can help the system accurately understand the current user's intent and perform appropriate actions. The system records every interaction between the user and the robot and continuously updates the context.

[0074] Finally, by combining the verbs in the text, task-related words, and historical interaction information, the system can comprehensively judge the user's intention. In multi-round conversations, the system not only relies on the current instructions, but also infers the user's needs based on previous interactions. For example, if the user says "turn on the air conditioner" in the first interaction and then mentions "the temperature is a bit low", the system will infer that the user's need is to adjust the temperature by analyzing the "air conditioner" and "low temperature" information in the historical interactions. Therefore, the system can accurately determine whether the text information contains action instructions based on this comprehensive information and perform the corresponding action.

[0075] Furthermore, the judgment process of whether the text information in this embodiment includes action instructions can be obtained through a pre-trained judgment model, and the initial model of the judgment model is a large language model, such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pretrained Transformer).

[0076] Exemplarily, after receiving text information, the judgment model converts the text into vector representations through the input layer. These vectors are then fed into the model's encoding layer. In the encoding layer, the model uses a self-attention mechanism to match these vectors with pre-set sentence structure vectors to determine whether the text conforms to the predetermined sentence template. The pre-set sentence structure is converted into a fixed vector embedding and stored in the model's parameters. When the input text vector is compared with the pre-set sentence structure vector, the model can determine whether the input text conforms to a specific sentence structure. If the text does not conform to the pre-set sentence structure, the model proceeds to the next step of extracting verbs and task-related words. Exemplarily, the model extracts verbs and task-related words from the text through named entity recognition (NER) or dependency parsing. The initial model uses its built-in self-attention mechanism to analyze the relationships between words in the text to determine which words are verbs and which words are task-related nouns. The model not only relies on syntactic rules but also utilizes contextual information to enhance its understanding of verbs and task-related words. For example, in the phrase "the temperature is a bit low," the model identifies "temperature" and "low" as task-related terms, inferring that the user wants to adjust the air conditioner temperature. Next, the model identifies the user's intent by combining verbs, task-related terms, and historical interaction information. For multi-turn conversations, the system utilizes a memory network or context encoder to extract and store key information from historical interactions. This information is filtered using a preset window, retaining only relevant historical interactions within the timeframe. For example, when a user inputs "the temperature is a bit low," the model infers that the user wants to adjust the air conditioner temperature based on historical interactions within the window, such as "turn on the air conditioner." Combining the current task-related terms "temperature" and "low" with the historical information about "air conditioner," the model accurately identifies the user's intent to adjust the temperature. Finally, the model utilizes its deep multi-classification decision layer to determine whether the text message contains action instructions based on the identified intent and task.

[0077] In one embodiment, the method further includes: if the text information includes action instructions, matching the verbs with a preset action library to determine the target execution action.

[0078] The preset action class is a predefined database containing various actions that the robot can perform and their corresponding instructions. For example, the following is an example of a preset action library.

[0079]

[0080] Action descriptions are standardized expressions of robot motion instructions within the system. Each action description corresponds to a specific behavior, such as "robot_wave" for waving a hand or "robot_move_forward" for moving one step forward. Action descriptions not only clarify robot behavior but also facilitate the system's scheduling and management of different actions, ensuring accurate and consistent execution.

[0081] Among them, the action description is a standardized string or identifier used to refer to a specific behavior of the robot. Each action description should correspond one-to-one to the actual action of the robot to ensure that the system can clearly know how to perform a certain task.

[0082] For example, using robot_wave as an example, an action description might include a target module, an action angle, and an action amplitude. The target module determines which part of the body performs the action; the action angle can be "from 0° to 45°," which specifies the angular range of the arm from its initial position to its swing position; and the action amplitude is the amplitude of the command hand's movement; the amplitude of the arm's swing determines its range of motion.

[0083] The standardization of action descriptions facilitates the system's scheduling and management of actions, ensuring that there are no conflicts between different tasks and that the system can process different tasks sequentially or in parallel.

[0084] Optionally, in conjunction with the above embodiments, the judgment model determines whether the text message contains an action instruction. If so, the judgment model outputs "yes" along with the corresponding verb. The verb is then matched against a preset action library to obtain the target action.

[0085] For example, a string comparison method can be used to determine whether an exact match exists. A match is determined only when the keyword is exactly the same as an instruction in the preset action library or the difference is less than a preset threshold.

[0086] In this way, when the keyword matches the preset action library, the target execution action can be quickly determined.

[0087] Furthermore, in one embodiment, the above-mentioned generation of answer information corresponding to the text information and determination of the target execution action corresponding to the answer information include: generating answer information according to the user's intention; inputting the answer information into a pre-trained multi-classification model to obtain a classification result; and matching the classification result with a preset action library to determine the target execution action.

[0088] When the text information does not include action instructions, the judgment model will output "no" and the corresponding user intention.

[0089] After obtaining the user's intention, the system will generate answer information based on the intention, and the answer information includes the system's response to the user's needs.

[0090] Next, the generated response is fed into a pre-trained multi-classification model. This model categorizes the response according to pre-defined categories, determining which response the user's request falls into. The multi-classification model has been extensively trained to identify the correct intent category based on the input.

[0091] The classification result will indicate the type of response the user expects, such as whether a certain action needs to be performed. The system will then search for actions related to that category from the preset action library based on the classification result and match them. If the classification result indicates that the user intends to perform a certain action, the system will search the preset action library for the corresponding execution action. For example, if the user intends to adjust the air conditioning temperature, the system will find the corresponding action from the action library, such as "Raise the air conditioning temperature", and perform the action.

[0092] Furthermore, in one embodiment, the training process of the above-mentioned multi-classification model includes: obtaining first sample data; the first sample data carries an action instruction label; preprocessing the first sample data to obtain multiple word segmentations; inputting the first sample data into the first initial model for training to obtain a predicted classification result; the first initial model includes multiple instruction classification heads and a fully connected layer, each instruction classification head includes an input layer, an encoding layer and a pooling layer; the input layer converts the word segmentations into word vectors, and passes the word vectors to the encoding layer; the encoding layer obtains the initial features of each word vector through a self-attention mechanism, and sends the initial features to the pooling layer; the pooling layer compresses the initial features to obtain target features; the fully connected layer merges the target features corresponding to each classification head to obtain a predicted classification result; the predicted classification result is compared with the action instruction label to obtain a first difference, and the parameters of the first initial model are adjusted according to the first difference until the first initial model is trained to obtain a multi-classification model.

[0093] First, we obtain first sample data, where each sample data contains user-entered text and a corresponding action instruction label. The action instruction label identifies the action type or task category corresponding to the text information and provides supervisory information for the training process. The first sample data requires preprocessing. This preprocessing step includes word segmentation, which breaks down the text information into multiple word units to enable the model to better understand the structure and semantics of the text.

[0094] Next, the segmented data is fed into the first initial model for training. This initial model structure consists of multiple instruction classification heads and a fully connected layer. Each instruction classification head primarily comprises an input layer, an encoding layer, and a pooling layer. The input layer converts the segmented text into word vectors, which represent the semantic information of each word in the text. The word vectors are passed as input to the encoding layer, where the model processes the word vectors using a self-attention mechanism to obtain initial feature representations for each word.

[0095] The self-attention mechanism enables the model to dynamically adjust the weights of individual words based on contextual relationships, thereby better capturing dependencies between words. For example, when a word depends on the context of another word, the model will enhance the relationship between the words, thereby taking this contextual information into account when generating features. The initial features output by the encoding layer are passed to the pooling layer, which compresses these features and integrates the feature information of each word into a more concise and expressive target feature. This target feature represents the semantic information of the entire sentence or text.

[0096] The fully connected layer then merges the target features from each classification head. These merged features are passed to the classification module, which ultimately generates a predicted classification result. This result represents the model's predicted category for the input text information, i.e., the corresponding action instruction type.

[0097] Next, the model's predictions are compared with the actual action command labels to generate a first difference. This difference represents the difference between the model's predictions and the actual labels, reflecting the model's accuracy. Based on this difference, the backpropagation algorithm is used to adjust the parameters of the initial model and optimize the model weights to reduce prediction error. The training process continues until the difference reaches a predetermined tolerance or the model training cycle reaches a set threshold.

[0098] After training is complete, the initial model undergoes a preset number of parameter adjustments, ultimately resulting in a fully trained multi-classification model that can accurately predict corresponding action commands based on new text information and provide a basis for subsequent task execution.

[0099] In one embodiment, the above-mentioned conversion of the audio signal into text information is obtained through a pre-trained speech recognition model; the training process of the speech recognition model includes: obtaining second sample data; the second sample data carries a text label; adding background noise to the second sample data; the background noise includes one or more of the sound of TV playback, kitchen audio, and conversation; inputting the second sample data carrying background noise into the second initial model for training to obtain predicted text data; comparing the second difference between the predicted text data and the text label, and adjusting the parameters of the second initial model according to the second difference, until the second initial model training is completed to obtain the speech recognition model.

[0100] In this embodiment, the conversion of audio signals into text is accomplished using a pre-trained speech recognition model. This model converts audio signals into text, helping the system understand and process voice commands. The model training process involves acquiring sample data, adding noise to enhance robustness, feeding in training data, and adjusting model parameters based on errors.

[0101] The training process begins by acquiring secondary sample data. This secondary sample data consists of an audio signal and its corresponding text label. The text label is an accurate transcription of the audio signal, representing the actual textual information corresponding to the audio. This secondary sample data serves as supervisory data for training the model. To enhance the model's adaptability and robustness in real-world environments, this data is preprocessed during training, including adding background noise to the audio signal.

[0102] Background noise is added to simulate everyday noise environments and enhance the speech recognition model's ability to resist interference. Background noise can include one or more noise sources, such as television sound, kitchen speakers, and conversations. This allows the model to maintain good recognition accuracy even in noisy environments. Adding background noise changes the characteristics of the audio signal, allowing the model to learn how to perform speech recognition in complex environments.

[0103] After adding background noise, the noisy second sample data is fed into the second initial model for training. This second initial model typically consists of an acoustic feature extraction layer, an acoustic model layer, and a decoding layer. First, the acoustic feature extraction layer extracts useful feature information from the audio signal, such as Mel-Frequency Cepstral Coefficients (MFCCs) and filter bank features. This feature information characterizes the frequency and time domain characteristics of the audio signal, providing basic data for subsequent speech recognition.

[0104] Next, the acoustic model layer processes this feature information and builds a model using deep neural networks (such as RNNs and LSTMs) to capture the temporal information and speech patterns in the audio signal. The decoding layer maps the processed features into text output, namely the predicted text data. This predicted text data represents the model's transcription of the input audio signal.

[0105] The model then compares the predicted text data with the actual text labels and calculates a second difference, or the error between the predicted result and the actual label. This difference reflects the model's prediction accuracy; smaller errors indicate better model performance. Based on this second difference, the backpropagation algorithm is used to adjust the parameters of the second initial model. Backpropagation optimizes the model's recognition capabilities by calculating gradients and updating the model's weights.

[0106] This training process continues until the error of the second initial model reaches a predetermined tolerance, or until the model's predictions gradually approach the actual text labels after a predetermined number of training cycles. Ultimately, after multiple adjustments to the second initial model, a trained speech recognition model is obtained. This speech recognition model can accurately convert audio signals into text information under various environmental and noise conditions, providing text input for subsequent tasks.

[0107] For example, combined Figure 2 , Figure 2 The present invention is a flowchart of the steps of an action determination method in one embodiment.

[0108] After receiving textual information, the system directly inputs it into a binary classification model, the judgment model in the aforementioned embodiment, to determine whether the textual information contains action instructions. If so, the system directly matches the textual information with a preset action library to determine the target action to be performed. If not, the system interprets the response and outputs a free dialogue. The dialogue is then input into a multi-classification model to determine the target action to be performed from the preset action library.

[0109] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0110] Based on the same inventive concept, the embodiments of the present application further provide an action determination device for implementing the aforementioned action determination method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more action determination device embodiments provided below can be found in the above-mentioned limitations of the action determination method and will not be repeated here.

[0111] In an exemplary embodiment, Figure 4 As shown, an action determination device is provided, comprising: a receiving module 100, a text conversion module 200, an action determination module 300 and an execution module 400, wherein:

[0112] The receiving module 100 is configured to receive an audio signal.

[0113] The text conversion module 200 is configured to convert the audio signal into text information.

[0114] The action determination module 300 is configured to generate answer information corresponding to the text information if the text information does not include an action instruction, and determine a target execution action corresponding to the answer information.

[0115] The execution module 400 is used to execute the target execution action and output response information.

[0116] In one embodiment, the apparatus further comprises:

[0117] The keyword extraction module is used to extract keywords from the text information if the text information includes action instructions.

[0118] The matching module is used to match keywords with the preset action library to determine the target execution action.

[0119] In one embodiment, the action determination module includes:

[0120] The analysis unit is used to analyze the text information if it does not conform to the preset sentence structure and identify verbs or task-related words in the text information.

[0121] The screening unit is used to screen the historical interaction information through a preset window to obtain target interaction information.

[0122] The intention extraction unit is used to obtain user intention based on verbs or task-related words and target interaction information; user intention is used to determine whether the text information includes action instructions.

[0123] In one embodiment, the apparatus further includes a first training module, the first training module including:

[0124] The first sample acquisition unit is used to acquire first sample data; the first sample data carries an action instruction tag.

[0125] The first preprocessing unit is used to preprocess the first sample data to obtain a plurality of segmented words.

[0126] The first training unit is used to input the first sample data into the first initial model for training to obtain a predicted classification result; the first initial model includes multiple instruction classification heads and a fully connected layer, each instruction classification head includes an input layer, an encoding layer and a pooling layer; the input layer converts word segmentation into word vectors and passes the word vectors to the encoding layer; the encoding layer obtains the initial features of each word vector through the self-attention mechanism, and sends the initial features to the pooling layer; the pooling layer compresses the initial features to obtain the target features; the fully connected layer merges the target features corresponding to each classification head to obtain a predicted classification result.

[0127] The first model adjustment unit is used to compare the predicted classification result with the action instruction label to obtain a first difference, and adjust the parameters of the first initial model according to the first difference until the first initial model training is completed to obtain a multi-classification model.

[0128] In one embodiment, the apparatus further includes a second training module, the second training module including:

[0129] The second sample acquisition unit is used to acquire second sample data; the second sample data carries a text label.

[0130] The second preprocessing unit is used to add background noise to the second sample data; the background noise includes one or more of TV playback sound, kitchen audio sound and conversation sound.

[0131] The second training unit is used to input the second sample data carrying background noise into the second initial model for training to obtain predicted text data.

[0132] The second model adjustment unit is used to compare the second difference between the predicted text data and the text label, and adjust the parameters of the second initial model according to the second difference until the second initial model training is completed to obtain a speech recognition model.

[0133] Each module in the above-mentioned action determination device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0134] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 4As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store preset actions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an action determination method is implemented.

[0135] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0136] In an exemplary embodiment, a computer device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: receiving an audio signal; converting the audio signal into text information; if the text information does not include an action instruction, generating answer information corresponding to the text information, and determining a target execution action corresponding to the answer information from a preset action library; executing the target execution action, and outputting the answer information.

[0137] In one embodiment, when the processor executes the computer program, the following steps are also implemented: if the text information does not conform to the preset sentence structure, the text information is analyzed to identify the verbs or task-related words in the text information; historical interaction information is filtered through a preset window to obtain target interaction information related to the verbs or task-related words.

[0138] In one embodiment, when the processor executes the computer program, the following steps are further implemented: if the text information includes action instructions, matching is performed in a preset action library according to the verb to determine the target execution action.

[0139] In one embodiment, when the processor executes the computer program, it also implements the following steps: generating answer information based on the user's intention; inputting the answer information into a pre-trained multi-classification model to obtain a classification result; matching the classification result with a preset action library to determine the target execution action.

[0140] In one embodiment, when the processor executes the computer program, it also implements the following steps: obtaining first sample data; the first sample data carries an action instruction label; preprocessing the first sample data to obtain multiple word segmentations; inputting the first sample data into the first initial model for training to obtain a predicted classification result; the first initial model includes multiple instruction classification heads and a fully connected layer, each instruction classification head includes an input layer, an encoding layer and a pooling layer; the input layer converts the word segmentations into word vectors and passes the word vectors to the encoding layer; the encoding layer obtains the initial features of each word vector through a self-attention mechanism, and sends the initial features to the pooling layer; the pooling layer compresses the initial features to obtain target features; the fully connected layer merges the target features corresponding to each classification head to obtain a predicted classification result; the predicted classification result is compared with the action instruction label to obtain a first difference, and the parameters of the first initial model are adjusted according to the first difference until the first initial model is trained to obtain a multi-classification model.

[0141] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining second sample data; the second sample data carries a text label; adding background noise to the second sample data; the background noise includes one or more of the sound of a TV playing, the sound of a kitchen speaker, and the sound of a conversation; inputting the second sample data carrying the background noise into a second initial model for training to obtain predicted text data; comparing a second difference between the predicted text data and the text label, and adjusting the parameters of the second initial model according to the second difference, until the second initial model is trained to obtain a speech recognition model.

[0142] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: receiving an audio signal; converting the audio signal into text information; if the text information does not include an action instruction, generating answer information corresponding to the text information, and determining a target execution action corresponding to the answer information from a preset action library; executing the target execution action, and outputting the answer information.

[0143] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: if the text information does not conform to the preset sentence structure, the text information is analyzed to identify the verbs or task-related words in the text information; historical interaction information is filtered through a preset window to obtain target interaction information related to the verbs or task-related words; user intention is obtained based on the verbs or task-related words and the target interaction information; user intention is used to determine whether the text information includes action instructions.

[0144] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: if the text information includes action instructions, matching is performed in a preset action library according to the verb to determine the target execution action.

[0145] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: generating answer information based on the user's intention; inputting the answer information into a pre-trained multi-classification model to obtain a classification result; matching the classification result with a preset action library to determine the target execution action.

[0146] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining first sample data; the first sample data carries an action instruction label; preprocessing the first sample data to obtain multiple word segmentations; inputting the first sample data into the first initial model for training to obtain a predicted classification result; the first initial model includes multiple instruction classification heads and a fully connected layer, each instruction classification head includes an input layer, an encoding layer and a pooling layer; the input layer converts the word segmentations into word vectors and passes the word vectors to the encoding layer; the encoding layer obtains the initial features of each word vector through a self-attention mechanism, and sends the initial features to the pooling layer; the pooling layer compresses the initial features to obtain target features; the fully connected layer merges the target features corresponding to each classification head to obtain a predicted classification result; the predicted classification result is compared with the action instruction label to obtain a first difference, and the parameters of the first initial model are adjusted according to the first difference until the first initial model is trained to obtain a multi-classification model.

[0147] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining second sample data; the second sample data carries a text label; adding background noise to the second sample data; the background noise includes one or more of the sound of TV playback, kitchen audio, and conversation; inputting the second sample data carrying the background noise into the second initial model for training to obtain predicted text data; comparing a second difference between the predicted text data and the text label, and adjusting the parameters of the second initial model according to the second difference, until the second initial model is trained to obtain a speech recognition model.

[0148] In one embodiment, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the following steps: receiving an audio signal; converting the audio signal into text information; if the text information does not include an action instruction, generating answer information corresponding to the text information, and determining a target execution action corresponding to the answer information from a preset action library; executing the target execution action, and outputting the answer information.

[0149] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: if the text information does not conform to the preset sentence structure, the text information is analyzed to identify the verbs or task-related words in the text information; historical interaction information is filtered through a preset window to obtain target interaction information related to the verbs or task-related words; user intention is obtained based on the verbs or task-related words and the target interaction information; user intention is used to determine whether the text information includes action instructions.

[0150] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: if the text information includes action instructions, matching is performed in a preset action library according to the verb to determine the target execution action.

[0151] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: generating answer information based on the user's intention; inputting the answer information into a pre-trained multi-classification model to obtain a classification result; matching the classification result with a preset action library to determine the target execution action.

[0152] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining first sample data; the first sample data carries an action instruction label; preprocessing the first sample data to obtain multiple word segmentations; inputting the first sample data into the first initial model for training to obtain a predicted classification result; the first initial model includes multiple instruction classification heads and a fully connected layer, each instruction classification head includes an input layer, an encoding layer and a pooling layer; the input layer converts the word segmentations into word vectors and passes the word vectors to the encoding layer; the encoding layer obtains the initial features of each word vector through a self-attention mechanism, and sends the initial features to the pooling layer; the pooling layer compresses the initial features to obtain target features; the fully connected layer merges the target features corresponding to each classification head to obtain a predicted classification result; the predicted classification result is compared with the action instruction label to obtain a first difference, and the parameters of the first initial model are adjusted according to the first difference until the first initial model is trained to obtain a multi-classification model.

[0153] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining second sample data; the second sample data carries a text label; adding background noise to the second sample data; the background noise includes one or more of the sound of TV playback, kitchen audio, and conversation; inputting the second sample data carrying the background noise into the second initial model for training to obtain predicted text data; comparing a second difference between the predicted text data and the text label, and adjusting the parameters of the second initial model according to the second difference, until the second initial model is trained to obtain a speech recognition model.

[0154] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0155] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0156] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for determining an action, characterized in that: The method comprises: receiving audio signals; Converting the audio signal into text information; Determining whether the text information includes an action instruction by using a judgment model; if the text information does not include an action instruction, generating answer information corresponding to the text information, and determining a target execution action corresponding to the answer information from a preset action library; executing the target execution action and outputting the response information; The process of determining whether the text information includes the action instruction includes: If the text information does not conform to the preset sentence structure, analyzing the text information to identify verbs or task-related words in the text information; Filtering historical interaction information through a preset window to obtain target interaction information related to the verb or the task-related word; Obtaining a user intention based on the verb or the task-related word and the target interaction information; the user intention is used to determine whether the text information includes the action instruction; The process of determining whether the text information conforms to the preset sentence structure includes: the judgment model converts the text into a vector representation through an input layer, and feeds the vector into an encoding layer of the judgment model; at the encoding layer, the judgment model uses a self-attention mechanism to match the vector with a preset sentence vector to determine whether the text conforms to the preset sentence structure; the preset sentence structure is converted into a fixed vector embedding and stored in the parameters of the judgment model; The generating of answer information corresponding to the text information and determining a target execution action corresponding to the answer information from the preset action library includes: Inputting the answer information into a multi-classification model to obtain an action instruction type; Matching the action instruction type in the preset action library to determine the target execution action corresponding to the answer information; The multi-classification model includes multiple instruction classification heads and a fully connected layer, each instruction classification head includes an input layer, an encoding layer and a pooling layer; the self-attention mechanism of the encoding layer dynamically adjusts the weight of each word in the text information according to the context relationship to obtain the initial features of each word; the pooling layer compresses each initial feature to obtain the target feature; the fully connected layer merges the target features corresponding to each instruction classification head to obtain the action instruction type.

2. The method according to claim 1, characterized in that The method further comprises: If the text information includes an action instruction, the target execution action is determined by matching the verb in the preset action library.

3. The method according to claim 1, characterized in that The generating of answer information corresponding to the text information and determining a target execution action corresponding to the answer information from a preset action library includes: generating the answer information according to the user intention; Inputting the answer information into a pre-trained multi-classification model to obtain a classification result; The target execution action is determined by matching the classification result with the preset action library.

4. The method according to claim 3, characterized in that The training process of the multi-classification model includes: Acquire first sample data; the first sample data carries an action instruction tag; Preprocessing the first sample data to obtain multiple word segments; The first sample data is input into a first initial model for training to obtain a prediction classification result; the first initial model includes multiple instruction classification heads and a fully connected layer, each instruction classification head includes an input layer, an encoding layer, and a pooling layer; the input layer converts the word segmentation into a word vector and passes the word vector to the encoding layer; the encoding layer obtains the initial features of each word vector through a self-attention mechanism and passes the initial features to the pooling layer; the pooling layer compresses the initial features to obtain target features; the fully connected layer merges the target features corresponding to each classification head to obtain the prediction classification result; The predicted classification result is compared with the action instruction label to obtain a first difference, and the parameters of the first initial model are adjusted according to the first difference until the first initial model is trained to obtain the multi-classification model.

5. The method according to claim 1, wherein The conversion of the audio signal into text information is obtained by a pre-trained speech recognition model; The training process of the speech recognition model includes: Acquire second sample data; the second sample data carries a text label; adding background noise to the second sample data; the background noise includes one or more of television playback sound, kitchen sound, and conversation sound; Inputting the second sample data carrying the background noise into a second initial model for training to obtain predicted text data; A second difference between the predicted text data and the text label is compared, and parameters of the second initial model are adjusted according to the second difference until the second initial model is trained to obtain the speech recognition model.

6. An action determination device, characterized in that: The device comprises: A receiving module, configured to receive an audio signal; A text conversion module, configured to convert the audio signal into text information; an action determination module, configured to determine whether the text information includes an action instruction by using a judgment model; if the text information does not include an action instruction, generate answer information corresponding to the text information, and determine a target execution action corresponding to the answer information from a preset action library; an execution module, configured to execute the target execution action and output the response information; The action determination module includes: An analyzing unit, configured to analyze the text information and identify verbs or task-related words in the text information if the text information does not conform to a preset sentence structure; a screening unit, configured to screen the historical interaction information through a preset window to obtain target interaction information related to the verb or the task-related word; An intention extraction unit is configured to obtain a user intention based on the verb or the task-related word and the target interaction information; the user intention is used to determine whether the text information includes the action instruction; the process of determining whether the text information conforms to the preset sentence structure includes: the judgment model converts the text into a vector representation through an input layer, and feeds the vector into an encoding layer of the judgment model; at the encoding layer, the judgment model uses a self-attention mechanism to match the vector with a preset sentence vector to determine whether the text conforms to the preset sentence structure; the preset sentence structure is converted into a fixed vector embedding and stored in the parameters of the judgment model; The action determination module is also used to input the answer information into a multi-classification model to obtain an action instruction type; match the action instruction type in the preset action library to determine the target execution action corresponding to the answer information; the multi-classification model includes multiple instruction classification heads and a fully connected layer, each instruction classification head includes an input layer, an encoding layer and a pooling layer; the self-attention mechanism of the encoding layer dynamically adjusts the weight of each word in the text information according to the context relationship to obtain the initial features of each word; the pooling layer compresses each initial feature to obtain the target feature; the fully connected layer merges the target features corresponding to each instruction classification head to obtain the action instruction type.

7. The device according to claim 6, characterized in that The device further comprises: The keyword extraction module is used to match the verb in the preset action library if the text information includes an action instruction, and determine the target execution action.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images