Variable extraction method based on large model and related equipment

Abstract

The invention discloses a variable extraction method based on a large model and related equipment. The method comprises the steps that a cloud receives recording data uploaded by a robot to generate to-be-processed recording data; the to-be-processed recording data is input into a large language model to obtain first text information, and the large language model is configured to perform sentence splitting on the to-be-processed recording data to obtain multiple pieces of audio data and perform voice activity detection on the to-be-processed recording data to obtain respective detection results of all voice activity data; performing semantic recognition on each piece of audio data to obtain a semantic recognition result, and converting the to-be-processed recording data into first text information based on the semantic recognition result and the detection result; inputting the first text information into an AI large model for variable extraction to obtain second text information; and inputting the second text information into the large language model to generate voice data, and issuing the voice data and the second text information to the robot, so that the precision of variable extraction can be improved through end-cloud cooperative processing between the robot and the cloud.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a variable extraction method and related equipment based on a large model. Background Technology

[0002] Variable extraction refers to matching and extracting information with a specific structure from text in order to extract the desired information from the text. For example, time information (year, month, day, hour, minute, second, etc.) can be one of the variables extracted.

[0003] In related technologies, if it is necessary to extract variables from audio, it is usually necessary to first convert the audio into text information and then extract variables from the text information. However, since the variable extraction process is highly dependent on the accuracy of the text information, the accuracy of variable extraction is also severely limited by the conversion accuracy between audio and text. Summary of the Invention

[0004] To address the aforementioned technical issues, this application proposes a variable extraction method and related equipment based on a large model, which can improve the conversion accuracy between audio and text, thereby facilitating improved variable extraction accuracy through end-to-cloud collaborative processing between robots and the cloud.

[0005] Firstly, embodiments of this application provide a variable extraction method based on a large model, suitable for cloud computing, the method comprising:

[0006] It receives pre-processed audio data uploaded by the robot on the edge in real time to generate audio data to be processed;

[0007] The audio recording data to be processed is input into a language model, and the first text information output by the language model is obtained. The language model is configured to: perform sentence segmentation on the input audio recording data to be processed to obtain multiple audio data; perform speech activity detection on the input audio recording data to be processed to obtain the detection results of each speech activity data in the audio recording data to be processed; perform semantic recognition on each audio data to obtain the corresponding semantic recognition result; and, based on the semantic recognition result and the detection result, convert the input audio recording data to be processed into the first text information.

[0008] The first text information is input into the AI ​​model, and the second text information is obtained and output by the AI ​​model after the first text information is processed by variable extraction.

[0009] The second text information is input into the language model to generate speech data, and the acquired speech data and the second text information are sent to the robot.

[0010] Optionally, converting the input recording data to be processed into the first text information based on the semantic recognition result and the detection result includes:

[0011] From the detection results of each of the aforementioned voice activity data, a detection result matching each of the aforementioned audio data is determined;

[0012] Based on the multiple audio data and their respective detection results and semantic recognition results, the input recording data to be processed is converted into the first text information.

[0013] Optionally, converting the input recording data to be processed into the first text information based on the plurality of audio data and their respective corresponding detection results and semantic recognition results includes:

[0014] For each voice activity data in all voice activity data in the recording data to be processed, select the audio data corresponding to the voice activity data from the plurality of audio data, and convert the voice activity data into third text information based on the audio data corresponding to the voice activity data, the detection result and the semantic recognition result.

[0015] The first text information is determined based on the third text information corresponding to each of the aforementioned voice activity data.

[0016] Optionally, the large AI model is configured as follows:

[0017] Determine the text representation vector of the first text information;

[0018] Based on the correlation between multiple preset variable elements and the text representation vector, the element information of each variable element contained in the first text information is determined.

[0019] Based on the element information of each variable element and the text representation vector, determine the set of variable elements corresponding to the first text information;

[0020] Based on the set of variable elements, the first text information is subjected to variable extraction to obtain the second text information.

[0021] Optionally, determining the element information of each variable element contained in the first text information based on the correlation between multiple preset variable elements and the text representation vector includes:

[0022] Based on the correlation corresponding to each preset variable element, determine the probability distribution corresponding to each preset variable element;

[0023] Based on the semantic matching degree between the multiple preset variable elements, each probability distribution is classified into multiple probability distribution sets;

[0024] Based on the multiple probability distribution sets, the element information of each variable element contained in the first text information is determined.

[0025] Optionally, determining the element information of each variable element contained in the first text information based on the plurality of probability distribution sets includes:

[0026] Determine multiple filters corresponding to the multiple probability distribution sets, wherein the multiple filters are uncorrelated with each other;

[0027] For each set of probability distributions, the corresponding filter is invoked to filter each probability distribution contained in the set.

[0028] Based on the multiple probability distribution sets after filtering, the element information of each variable element contained in the first text information is determined.

[0029] Optionally, determining the set of variable elements corresponding to the first text information based on the element information of each variable element and the text representation vector includes:

[0030] Based on the textual representation vector, generate a set of variable elements to be adjusted;

[0031] The set of variable elements to be adjusted is adjusted based on the element information of each variable element.

[0032] Based on the adjusted set of variable elements to be adjusted, determine the set of variable elements corresponding to the first text information.

[0033] Secondly, embodiments of this application provide a variable extraction device based on a large model, suitable for cloud computing, the device comprising:

[0034] The audio recording data processing module is used to receive pre-processed audio recording data uploaded by the robot on the edge in real time to generate audio recording data to be processed.

[0035] A language large model processing module is used to input the recording data to be processed into a language large model and obtain the first text information output by the language large model. The language large model is configured to: perform sentence segmentation on the input recording data to be processed to obtain multiple audio data; perform speech activity detection on the input recording data to be processed to obtain the detection results of each speech activity data in the recording data to be processed; perform semantic recognition on each audio data to obtain the corresponding semantic recognition result; and, based on the semantic recognition result and the detection result, convert the input recording data to be processed into the first text information.

[0036] The AI ​​large model processing module is used to input the first text information into the AI ​​large model and obtain the second text information obtained and output by the AI ​​large model after extracting variables from the first text information.

[0037] The voice data processing module is used to input the second text information into the language model to generate voice data, and to send the acquired voice data and the second text information to the robot.

[0038] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above-mentioned embodiments.

[0039] Fourthly, embodiments of this application provide a computer device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the preceding claims.

[0040] In summary, the embodiments of this application have at least the following beneficial effects:

[0041] In this embodiment, pre-processed audio data uploaded by a robot on the edge is received in real time to generate audio data to be processed. The audio data to be processed is input into a language model to obtain the first text information output by the language model. The language model is configured to: perform sentence segmentation on the input audio data to be processed to obtain multiple audio data; perform speech activity detection on the input audio data to obtain the detection results of each speech activity data in the audio data to be processed; perform semantic recognition on each audio data to obtain the corresponding semantic recognition result; and, based on the semantic recognition result and the detection result, convert the input audio data to be processed into the first text information. The first text information is input into an AI model to obtain the second text information obtained and output by the AI ​​model after variable extraction of the first text information. The second text information is input into the language model to generate speech data, and the obtained speech data and the second text information are sent to the robot. This improves the conversion accuracy between audio and text, and facilitates improved variable extraction accuracy through edge-cloud collaborative processing between the robot and the cloud. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating the variable extraction method based on a large model provided in an embodiment of this application;

[0043] Figure 2 This is a schematic diagram of the structure of the variable extraction device based on a large model provided in an embodiment of this application;

[0044] Figure 3 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0046] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more. In the description of this application, the term "comprising" and its variations are open-ended, meaning "including but not limited to." The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments."

[0047] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0048] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the application. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0049] Firstly, see [the following] Figure 1 The diagram shows a flowchart of a variable extraction method based on a large model provided in an embodiment of this application. The method is applicable to the cloud and includes steps S101-S104, as detailed below.

[0050] S101 receives pre-processed audio data uploaded by the robot on the edge in real time to generate audio data to be processed.

[0051] In some examples, the preprocessed audio data mentioned above may refer to audio data obtained by the robot in real time and after preprocessing. The preprocessing may include data cleaning, which may include at least one of the following: missing value handling (filling missing data with the mean, median, mode, etc., or deleting records containing missing values), outlier handling, and duplicate data handling (checking and deleting duplicate data entries).

[0052] In some examples, the robot can be referred to as a Verbatim robot.

[0053] In some examples, the robot described above can be of any form, such as a humanoid robot, a virtual robot on a terminal device, or an industrial robot. The robot has a recording function, used to respond to user-input recording commands (equivalent to obtaining the user's recording permission to ensure the recording process complies with relevant user privacy regulations), collect sound data from its surroundings to generate recording data, and upload it to the cloud in real time. For example, the robot itself may not store the recorded sound data; instead, it may immediately upload the collected sound data to the cloud.

[0054] In some examples, the aforementioned recording data to be processed can refer to the recording data received by the cloud within a certain period of time. For example, the recording data received within each preset time period can be used as the recording data to be processed. The robot can also be configured to upload a recording identifier simultaneously when uploading the recording data. The recording identifier can include a corresponding start recording identifier and an end recording identifier. Whenever the cloud parses and obtains a set of corresponding start recording identifiers and end recording identifiers, the recording data received within the time points corresponding to the start recording identifier and the end recording identifier will be used as the recording data to be processed.

[0055] In some examples, the robots described above can communicate and interact with the cloud via Wi-Fi, 4G networks, and / or 5G networks.

[0056] In some examples, the aforementioned robot can upload recording data to the aforementioned cloud via the UDP (User Datagram Protocol) protocol.

[0057] S102, the recording data to be processed is input into a language model, and the first text information output by the language model is obtained. The language model is configured to: perform sentence segmentation on the input recording data to be processed to obtain multiple audio data; perform speech activity detection on the input recording data to be processed to obtain the detection results of each speech activity data in the recording data to be processed; perform semantic recognition on each audio data to obtain the corresponding semantic recognition result; and, based on the semantic recognition result and the detection result, convert the input recording data to be processed into the first text information.

[0058] In this embodiment, the semantic recognition results and the speech activity detection results can be combined to perform more accurate speech-to-text conversion processing. For example, the non-background noise part of the recording data to be processed can be converted into initial text information based on the speech activity detection results. Then, the semantic recognition results are used to perform semantic adjustment on the initial text information (for example, semantic recognition is performed on the initial text information, and the recognition result of the initial text information is compared with the above semantic recognition results to complete the semantic adjustment according to the comparison result), thereby obtaining the first text information after semantic adjustment. In this way, the recording data to be processed can be converted into the first text information based solely on the semantic recognition results and the speech activity detection results.

[0059] In some examples, the aforementioned large language model may include a general large language model, such as Doubao or other publicly available general large language models, which are not specifically limited here.

[0060] In some examples, the general speech recognition and / or natural language processing functions inherent in the language big model can be invoked to perform semantic analysis on the recording data to be processed. This allows for sentence segmentation of the recording data based on the semantic analysis results, resulting in multiple audio data sets, each of which can represent a semantically complete sentence.

[0061] In some examples, speech activity detection can be used to identify speech and non-speech components (such as silence or background noise) in an audio signal (i.e., the recording data to be processed), thereby enabling accurate differentiation from a continuous audio stream of which segments contain speech and which are merely background noise or silence.

[0062] In some examples, speech activity detection is implemented in various ways. For instance, the energy level of the speech signal in the recording data to be processed can be used to determine whether it is speech or background noise or silence (speech usually has higher energy than background noise or silence); it can also be analyzed how many times the speech signal in the recording data to be processed crosses the zero-crossing line per unit time (different types of audio signals have different zero-crossing rate characteristics). In this embodiment, the data corresponding to the identified speech in the recording data to be processed is audio data, while the data corresponding to background noise or silence in the recording data to be processed is not considered audio data.

[0063] In some examples, since speech and background noise often exhibit different characteristics in the frequency domain, multiple audio data can be identified by analyzing the frequency values ​​of the speech signal in the recording data to be processed.

[0064] In some examples, the general semantic recognition and / or natural language processing functions inherent in the language big model itself can be invoked to perform semantic recognition on audio data.

[0065] S103, input the first text information into the AI ​​big model, and obtain the second text information obtained and output by the AI ​​big model after extracting variables from the first text information.

[0066] In some examples, the aforementioned AI (Artificial Intelligence) big model may include the DeepSeek AI big model.

[0067] In some examples, the aforementioned large AI model has the ability to extract variables from textual information. This large AI model can be a model that has been trained to have the ability to predict using first textual information as model input and second textual information obtained through variable extraction as model output. In specific training, the first textual sample can be used as sample data (the sample data also carries the expected corresponding variable label, which represents the target variable corresponding to the first textual sample), and a general training algorithm (such as gradient descent) can be used to train the model so that the trained model can have the above-mentioned ability.

[0068] S104, input the second text information into the language model to generate speech data, and send the obtained speech data and the second text information to the robot.

[0069] In some examples, the robot described above may have a streaming media playback function. The robot includes a display screen and / or a speaker. The robot can achieve the streaming media playback function by performing the following steps: in response to receiving second text information, controlling the display screen to display the received second text information, and / or, in response to receiving voice data, controlling the speaker to play the received voice data.

[0070] In some examples, the robot can play the received voice data directly without storing it. Alternatively, it can store the received second text information in its own RAM (Random Access Memory) for direct output to the display screen.

[0071] In some examples, the robot may have motors, and the second text information received by the robot may be used to instruct the robot to control the motors to perform corresponding operations. For example, the second text information may represent at least one action variable, which is a set of action variables that can be performed by the humanoid robot. There may be at least one motor, and the at least one action variable may correspond to at least one motor (it may be a one-to-one correspondence, or one action variable may correspond to at least two of the at least one motors, or one motor may correspond to at least two of the at least one action variable, which is not specifically limited here). Each action variable is used to instruct its corresponding motor to perform an action, so that the at least one action variable can be used to instruct the robot to complete the set of actions by controlling the motors to perform corresponding actions.

[0072] In one optional implementation, the step of converting the input recording data to be processed into the first text information based on the semantic recognition result and the detection result includes:

[0073] From the detection results of each of the aforementioned voice activity data, a detection result matching each of the aforementioned audio data is determined;

[0074] Based on the multiple audio data and their respective detection results and semantic recognition results, the input recording data to be processed is converted into the first text information.

[0075] In one optional implementation, the step of converting the input recording data to be processed into the first text information based on the plurality of audio data and their respective corresponding detection results and semantic recognition results includes:

[0076] For each voice activity data in all voice activity data in the recording data to be processed, select the audio data corresponding to the voice activity data from the plurality of audio data, and convert the voice activity data into third text information based on the audio data corresponding to the voice activity data, the detection result and the semantic recognition result.

[0077] The first text information is determined based on the third text information corresponding to each of the aforementioned voice activity data.

[0078] In some examples, the conversion of the speech activity data into third-party text information based on the corresponding audio data, detection results, and semantic recognition results can include: calling the speech recognition and / or natural language processing functions of the language model itself and / or external speech recognition models (such as Whisper or Paraformer) to perform speech recognition on the audio data, obtaining preliminary speech recognition results; combining the preliminary speech recognition results, the detection results, and the semantic recognition results into a prompt word, and enabling the language model to reason about the speech activity data under the guidance of the prompt word, thereby converting the speech activity data into third-party text information. In this embodiment, a context-rich prompt word can be formed to guide the language model to semantically understand and re-express the original speech activity data, ultimately outputting high-quality third-party text information, thereby improving the quality of subsequent first-party text information generation.

[0079] In some examples, determining the first text information based on the third text information corresponding to each of the voice activity data may include: combining / concatenating the various third text information according to the time sequence of all the voice activity data to obtain the first text information.

[0080] In one alternative implementation, the large AI model is configured as follows:

[0081] Determine the text representation vector of the first text information;

[0082] Based on the correlation between multiple preset variable elements and the text representation vector, the element information of each variable element contained in the first text information is determined.

[0083] Based on the element information of each variable element and the text representation vector, determine the set of variable elements corresponding to the first text information;

[0084] Based on the set of variable elements, the first text information is subjected to variable extraction to obtain the second text information.

[0085] In some examples, the aforementioned text representation vector can be obtained by encoding the first text information using an encoder.

[0086] In one optional implementation, determining the element information of each variable element contained in the first text information based on the correlation between multiple preset variable elements and the text representation vector includes:

[0087] Based on the correlation corresponding to each preset variable element, determine the probability distribution corresponding to each preset variable element;

[0088] Based on the semantic matching degree between the multiple preset variable elements, each probability distribution is classified into multiple probability distribution sets;

[0089] Based on the multiple probability distribution sets, the element information of each variable element contained in the first text information is determined.

[0090] In some examples, the above probability distribution is suitable for indicating the probability that the first text information contains a corresponding preset variable element. Thus, each probability distribution in each probability distribution set can be used to indicate the probability of a corresponding preset variable element. By comparing this probability with a preset probability threshold, preset variable elements with probabilities greater than the preset probability threshold are considered as variable elements contained in the first text information, and their numbers are counted (it is easy to understand that the same preset variable element may appear more than once in the first text information, so there can be multiple counts of that preset variable element), thereby obtaining the element information for each variable element. In other words, the element information for each variable element includes the number of each variable element.

[0091] In some examples, the probability distribution described above may include a Gaussian distribution.

[0092] In some examples, the semantic matching degree mentioned above can be obtained by calculating the semantic similarity between different preset variable elements, which may include cosine similarity and / or Euclidean distance, etc.

[0093] In some examples, multiple preset variable elements can be clustered based on semantic matching degree to obtain multiple cluster centers, and each cluster center is associated with a set of probability distributions; for each probability distribution, it is classified into the set of probability distributions associated with the cluster center that is closest to the corresponding preset variable element.

[0094] In one optional implementation, determining the element information of each variable element contained in the first text information based on the plurality of probability distribution sets includes:

[0095] Determine multiple filters corresponding to the multiple probability distribution sets, wherein the multiple filters are uncorrelated with each other;

[0096] For each set of probability distributions, the corresponding filter is invoked to filter each probability distribution contained in the set.

[0097] Based on the multiple probability distribution sets after filtering, the element information of each variable element contained in the first text information is determined.

[0098] In some cases, probabilistic modeling often involves noise (such as outliers or fluctuations deviating from the true distribution, uncertainties introduced by model estimation errors, and spurious patterns appearing in low-sample regions or high-dimensional spaces. This noise can make the generated probability distribution less smooth and stable, potentially misleading subsequent decisions). This noise can easily interfere with the accuracy of the subsequently determined element information. Therefore, this embodiment uses filtering to remove noise from the probability distribution, thereby improving the spatial continuity of the probability distribution and thus enhancing its reliability and accuracy.

[0099] Furthermore, if the same filter is directly and repeatedly used to filter different sets of probability distributions, the filter, which approximates the training process during filtering, can learn the common regularities among different preset variable elements, even if the semantics of these preset variable elements are very different. This could lead to misidentification by the filter, for example, classifying preset variable elements with vastly different semantics as the same case and applying the same filtering process, thus affecting the accuracy of subsequent operations. Therefore, this embodiment first classifies different probability distributions into multiple probability distribution sets according to semantic matching degree, ensuring that the preset variable elements corresponding to the various probability distributions within the same probability distribution set are semantically similar. Then, a corresponding filter is independently called for filtering each probability distribution set to avoid misidentification by the filter.

[0100] In some examples, the aforementioned probability distribution is suitable for indicating the probability that the first text information contains a corresponding preset variable element. Thus, each probability distribution in each set of filtered probability distributions can be used to indicate the filtered probability of the corresponding preset variable element. By comparing this filtered probability with a preset probability threshold, preset variable elements with a filtered probability greater than the preset probability threshold can be included as variable elements in the first text information, and their numbers can be counted (it's easy to understand that the same preset variable element may appear more than once in the first text information, so there can be multiple counts of this preset variable element). This yields the element information for each variable element. In other words, the element information for each variable element includes the number of each variable element.

[0101] In one optional implementation, determining the set of variable elements corresponding to the first text information based on the element information of each variable element and the text representation vector includes:

[0102] Based on the textual representation vector, generate a set of variable elements to be adjusted;

[0103] The set of variable elements to be adjusted is adjusted based on the element information of each variable element.

[0104] Based on the adjusted set of variable elements to be adjusted, determine the set of variable elements corresponding to the first text information.

[0105] In some examples, the aforementioned large AI model may contain a pre-trained encoder and decoder. The encoder encodes the first text information to obtain a text representation vector, and the decoder decodes the text representation vector to obtain a set of variable elements to be adjusted.

[0106] In some examples, the above element information may include the quantity of each variable element. The above set of variable elements to be adjusted can be used to characterize the probability of the existence of each variable element. Thus, the more a variable element has, the higher the probability of its existence in the set of variable elements to be adjusted. Conversely, the fewer the number, the lower the probability of its existence in the set of variable elements to be adjusted. By performing the above operation on all variable elements in the set of variable elements to be adjusted, the adjusted set of variable elements to be adjusted is obtained.

[0107] Secondly, correspondingly, this application also provides a variable extraction device based on a large model, which can implement all the processes of the variable extraction method based on a large model provided in the above embodiments.

[0108] See Figure 2 The diagram illustrates the structure of a variable extraction device based on a large model provided in an embodiment of this application. This device is suitable for cloud computing and includes:

[0109] The recording data processing module 201 is used to receive pre-processed recording data uploaded by the robot on the edge in real time to generate recording data to be processed.

[0110] The language large model processing module 202 is used to input the recording data to be processed into the language large model and obtain the first text information output by the language large model. The language large model is configured to: perform sentence segmentation on the input recording data to be processed to obtain multiple audio data; perform speech activity detection on the input recording data to be processed to obtain the detection results of each speech activity data in the recording data to be processed; perform semantic recognition on each audio data to obtain the corresponding semantic recognition result; and, based on the semantic recognition result and the detection result, convert the input recording data to be processed into the first text information.

[0111] AI large model processing module 203 is used to input the first text information into the AI ​​large model and obtain the second text information obtained and output by the AI ​​large model after the first text information is extracted by variables;

[0112] The voice data processing module 204 is used to input the second text information into the language model to generate voice data, and to send the acquired voice data and the second text information to the robot.

[0113] In one optional implementation, the step of converting the input recording data to be processed into the first text information based on the semantic recognition result and the detection result includes:

[0114] From the detection results of each of the aforementioned voice activity data, a detection result matching each of the aforementioned audio data is determined;

[0115] Based on the multiple audio data and their respective detection results and semantic recognition results, the input recording data to be processed is converted into the first text information.

[0116] In one optional implementation, the step of converting the input recording data to be processed into the first text information based on the plurality of audio data and their respective corresponding detection results and semantic recognition results includes:

[0117] For each voice activity data in all voice activity data in the recording data to be processed, select the audio data corresponding to the voice activity data from the plurality of audio data, and convert the voice activity data into third text information based on the audio data corresponding to the voice activity data, the detection result and the semantic recognition result.

[0118] The first text information is determined based on the third text information corresponding to each of the aforementioned voice activity data.

[0119] In one alternative implementation, the large AI model is configured as follows:

[0120] Determine the text representation vector of the first text information;

[0121] Based on the correlation between multiple preset variable elements and the text representation vector, the element information of each variable element contained in the first text information is determined.

[0122] Based on the element information of each variable element and the text representation vector, determine the set of variable elements corresponding to the first text information;

[0123] Based on the set of variable elements, the first text information is subjected to variable extraction to obtain the second text information.

[0124] In one optional implementation, determining the element information of each variable element contained in the first text information based on the correlation between multiple preset variable elements and the text representation vector includes:

[0125] Based on the correlation corresponding to each preset variable element, determine the probability distribution corresponding to each preset variable element;

[0126] Based on the semantic matching degree between the multiple preset variable elements, each probability distribution is classified into multiple probability distribution sets;

[0127] Based on the multiple probability distribution sets, the element information of each variable element contained in the first text information is determined.

[0128] In one optional implementation, determining the element information of each variable element contained in the first text information based on the plurality of probability distribution sets includes:

[0129] Determine multiple filters corresponding to the multiple probability distribution sets, wherein the multiple filters are uncorrelated with each other;

[0130] For each set of probability distributions, the corresponding filter is invoked to filter each probability distribution contained in the set.

[0131] Based on the multiple probability distribution sets after filtering, the element information of each variable element contained in the first text information is determined.

[0132] In one optional implementation, determining the set of variable elements corresponding to the first text information based on the element information of each variable element and the text representation vector includes:

[0133] Based on the textual representation vector, generate a set of variable elements to be adjusted;

[0134] The set of variable elements to be adjusted is adjusted based on the element information of each variable element.

[0135] Based on the adjusted set of variable elements to be adjusted, determine the set of variable elements corresponding to the first text information.

[0136] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above-mentioned embodiments.

[0137] Fourthly, embodiments of this application provide a computer device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the steps of the method described in any of the preceding claims.

[0138] See Figure 3 The computer device in this embodiment includes a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301, such as a variable extraction program based on a large model. When the processor 301 executes the computer program, it implements the steps in the various variable extraction method embodiments based on large models described above, for example... Figure 1 The steps S101-S104 are shown.

[0139] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 302 and executed by the processor 301 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0140] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will understand that the schematic diagram is merely an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0141] The processor 301 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 301 can be any conventional processor. The processor 301 is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0142] The memory 302 can be used to store the computer programs and / or modules. The processor 301 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 302 and calling the data stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0143] Wherein, if the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 301, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0144] In summary, the embodiments of this application have at least the following beneficial effects:

[0145] In this embodiment, pre-processed audio data uploaded by a robot on the edge is received in real time to generate audio data to be processed. The audio data to be processed is input into a language model to obtain the first text information output by the language model. The language model is configured to: perform sentence segmentation on the input audio data to be processed to obtain multiple audio data; perform speech activity detection on the input audio data to obtain the detection results of each speech activity data in the audio data to be processed; perform semantic recognition on each audio data to obtain the corresponding semantic recognition result; and, based on the semantic recognition result and the detection result, convert the input audio data to be processed into the first text information. The first text information is input into an AI model to obtain the second text information obtained and output by the AI ​​model after variable extraction of the first text information. The second text information is input into the language model to generate speech data, and the obtained speech data and the second text information are sent to the robot. This improves the conversion accuracy between audio and text, and facilitates improved variable extraction accuracy through edge-cloud collaborative processing between the robot and the cloud.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware platforms, or it can be implemented entirely by hardware. Based on this understanding, all or part of the technical solutions of this application that contribute to the background technology can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0147] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A variable extraction method based on a large model, characterized in that, Applicable to the cloud, the method includes: It receives pre-processed audio data uploaded by the robot on the edge in real time to generate audio data to be processed; The audio recording data to be processed is input into a language model, and the first text information output by the language model is obtained. The language model is configured to: perform sentence segmentation on the input audio recording data to be processed to obtain multiple audio data; perform speech activity detection on the input audio recording data to be processed to obtain the detection results of each speech activity data in the audio recording data to be processed; perform semantic recognition on each audio data to obtain the corresponding semantic recognition result; and, based on the semantic recognition result and the detection result, convert the input audio recording data to be processed into the first text information. The first text information is input into the AI ​​model, and the second text information is obtained and output by the AI ​​model after the first text information is extracted by variables. The second text information is input into the language model to generate speech data, and the acquired speech data and the second text information are sent to the robot.

2. The method according to claim 1, characterized in that, The step of converting the input recording data to be processed into the first text information based on the semantic recognition result and the detection result includes: From the detection results of each of the aforementioned voice activity data, a detection result matching each of the aforementioned audio data is determined; Based on the multiple audio data and their respective detection results and semantic recognition results, the input recording data to be processed is converted into the first text information.

3. The method according to claim 2, characterized in that, The step of converting the input recording data to be processed into the first text information based on the multiple audio data and their respective corresponding detection results and semantic recognition results includes: For each voice activity data in all voice activity data in the recording data to be processed, select the audio data corresponding to the voice activity data from the plurality of audio data, and convert the voice activity data into third text information based on the audio data corresponding to the voice activity data, the detection result and the semantic recognition result. The first text information is determined based on the third text information corresponding to each of the aforementioned voice activity data.

4. The method according to claim 1, characterized in that, The large AI model is configured as follows: Determine the text representation vector of the first text information; Based on the correlation between multiple preset variable elements and the text representation vector, the element information of each variable element contained in the first text information is determined. Based on the element information of each variable element and the text representation vector, determine the set of variable elements corresponding to the first text information; Based on the set of variable elements, the first text information is subjected to variable extraction to obtain the second text information.

5. The method according to claim 4, characterized in that, The step of determining the element information of each variable element contained in the first text information based on the correlation between multiple preset variable elements and the text representation vector includes: Based on the correlation corresponding to each preset variable element, determine the probability distribution corresponding to each preset variable element; Based on the semantic matching degree between the multiple preset variable elements, each probability distribution is classified into multiple probability distribution sets; Based on the multiple probability distribution sets, the element information of each variable element contained in the first text information is determined.

6. The method according to claim 5, characterized in that, The step of determining the element information of each variable element contained in the first text information based on the multiple probability distribution sets includes: Determine multiple filters corresponding to the multiple probability distribution sets, wherein the multiple filters are uncorrelated with each other; For each set of probability distributions, the corresponding filter is invoked to filter each probability distribution contained in the set. Based on the multiple probability distribution sets after filtering, the element information of each variable element contained in the first text information is determined.

7. The method according to claim 4, characterized in that, The step of determining the set of variable elements corresponding to the first text information based on the element information of each variable element and the text representation vector includes: Based on the textual representation vector, generate a set of variable elements to be adjusted; The set of variable elements to be adjusted is adjusted based on the element information of each variable element. Based on the adjusted set of variable elements to be adjusted, determine the set of variable elements corresponding to the first text information.

8. A variable extraction device based on a large model, characterized in that, Suitable for cloud environments, the device includes: The audio recording data processing module is used to receive pre-processed audio recording data uploaded by the robot on the edge in real time to generate audio recording data to be processed. A language large model processing module is used to input the recording data to be processed into a language large model and obtain the first text information output by the language large model. The language large model is configured to: perform sentence segmentation on the input recording data to be processed to obtain multiple audio data; perform speech activity detection on the input recording data to be processed to obtain the detection results of each speech activity data in the recording data to be processed; perform semantic recognition on each audio data to obtain the corresponding semantic recognition result; and, based on the semantic recognition result and the detection result, convert the input recording data to be processed into the first text information. The AI ​​large model processing module is used to input the first text information into the AI ​​large model and obtain the second text information obtained and output by the AI ​​large model after extracting variables from the first text information. The voice data processing module is used to input the second text information into the language model to generate voice data, and to send the acquired voice data and the second text information to the robot.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

10. A computer device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-7.

Images