Data processing method, data processing device, electronic equipment and medium

Abstract

The invention discloses a data processing method, a data processing device, electronic equipment and a medium, and belongs to the technical field of artificial intelligence. The data processing method comprises the steps of obtaining first data; second data is input into a network layer corresponding to the data type of the second data to be processed, a first processing result of the network layer is obtained, and the second data is data of any data type in the first data; and obtaining a second processing result of the first data based on the first processing result.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a data processing method, a data processing device, an electronic device and a medium. Background Art

[0002] A multimodal model is a deep learning model that can process multiple different types of input data. Multimodal models can integrate these different types of input data together for processing to achieve more comprehensive and integrated reasoning.

[0003] In the related art, when a multimodal model processes different types of input data, it is necessary to convert the different types of input data into data of the same dimension before processing, so as to realize the reasoning of the model. For example, when a multimodal model processes text-type and voice-type input data, it is necessary to convert the voice-type data into the same dimension as the text-type data before processing.

[0004] However, after converting different types of input data into data of the same dimension, the multimodal model may process different types of input data as the same type of data, which may reduce the accuracy of model reasoning. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a data processing method, a data processing device, an electronic device and a medium, which can improve the accuracy of model reasoning.

[0006] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0007] Acquire first data;

[0008] Inputting the first data into a network layer corresponding to a data type of the first data for processing, and obtaining a first processing result of the network layer;

[0009] Based on the first processing result, a second processing result of the first data is obtained.

[0010] In a second aspect, an embodiment of the present application provides a data processing device, the device comprising: an acquisition module and a processing module;

[0011] An acquisition module, used for acquiring first data;

[0012] A processing module, configured to input the first data into a network layer corresponding to a data type of the first data for processing, and obtain a first processing result of the network layer;

[0013] The processing module is further used to obtain a second processing result of the first data based on the processing result of the network layer.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the data processing method described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the data processing method described in the first aspect are implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the data processing method as described in the first aspect.

[0017] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the data processing method as described in the first aspect.

[0018] In the embodiment of the present application, the first data is obtained; the first data is input into the network layer corresponding to the data type of the first data for processing, and the first processing result of the network layer is obtained; based on the first processing result, the second processing result of the first data is obtained. In this way, the first data is processed by the network layer corresponding to the data type, and when the data type of the first data is different, the first processing result corresponding to the different data type can be obtained, so that when the first processing result is processed, the processing results corresponding to the different data types can be obtained, thereby improving the accuracy of model reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of a data processing method provided in an embodiment of the present application. Figure 1 ;

[0020] Figure 2 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 2 ;

[0021] Figure 3 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 3 ;

[0022] Figure 4 is a schematic diagram of a first interface provided in an embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of a data processing process provided by an embodiment of the present application. Figure 2 ;

[0024] Figure 6 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 4 ;

[0025] Figure 7 This is a schematic diagram of the structure of a first multimodal model provided in an embodiment of the present application. Figure 4 ;

[0026] Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 1 ;

[0027] Fig. 9 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 2 ;

[0028] Fig.10 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0029] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0031] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0032] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0033] The data processing method, data processing device, electronic device and medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0034] The data processing method, data processing device, electronic device and medium provided in the embodiments of the present application can be applied to multimodal data processing scenarios.

[0035] In an application scenario, the data processing method provided in an embodiment of the present application can be applied to electronic devices to process voice-type input data and text-type input data through a large language model to achieve a human-computer communication scenario.

[0036] In the human-computer communication scenario, the electronic device can not only receive the text input by the user, but also process the text input by the user through the large language model to realize information exchange with the user. At the same time, it can receive the voice input by the user, and process the voice input by the user through the large language model to realize information exchange with the user. In this way, the electronic device can realize real-time communication between the user and the large language model, making communication more convenient and efficient.

[0037] For example, when the user inputs the voice "How is the weather today" and the input text is "Voice broadcast of weather conditions", the electronic device can use the large voice model to process the input voice "How is the weather today" and the input text "Voice broadcast of weather conditions" and then voice broadcast the result as "Today will be sunny to cloudy".

[0038] The embodiment of the present application provides a data processing method, a data processing device, an electronic device and a medium, which obtains first data; inputs the first data into a network layer corresponding to the data type of the first data for processing, and obtains a first processing result of the network layer; based on the first processing result, obtains a second processing result of the first data. In this way, the first data is processed by the network layer corresponding to the data type, and when the data type of the first data is different, the first processing result corresponding to the different data types can be obtained, so that when the first processing result is processed, the processing results corresponding to the different data types can be obtained, thereby improving the accuracy of model reasoning.

[0039] The execution subject of the data processing method provided in the embodiment of the present application may be a data processing device. Exemplarily, the data processing device may be an electronic device, or a functional component or functional entity in the electronic device. The data processing method provided in the embodiment of the present application will be exemplarily described below by taking the execution subject as an electronic device as an example.

[0040] Figure 1 is a flow chart of a data processing method provided in an embodiment of the present application, such as Figure 1 As shown, the data processing method provided in the embodiment of the present application may include the following steps 201 to 203.

[0041] Step 201: An electronic device obtains first data.

[0042] In some embodiments of the present application, the first data is obtained by encoding target data, and the data type of the target data is the target data type.

[0043] In some embodiments of the present application, the above-mentioned target data can be multimodal data, for example, the target data can include text data, image data, audio data, and video data, etc., or it can be a single type of data, for example, the target data can be any one of text data, image data, audio data, and video data.

[0044] In some embodiments of the present application, the above-mentioned target data type includes at least two data types;

[0045] Combination Figure 1 ,like Figure 2 As shown, the above step 201 can be implemented through the following steps 201a to 201c:

[0046] Step 201a: The electronic device obtains first coded data and second coded data.

[0047] In some embodiments of the present application, the first encoded data is obtained by encoding data of a first data type in the first data, and the second encoded data is obtained by encoding data of a second data type in the first data, and the first data type and the second data type are different data types among the at least two types mentioned above.

[0048] In some embodiments of the present application, the above-mentioned first data type may be a text type, the above-mentioned first data may be at least one first text data, the above-mentioned second data type may be a voice type, the above-mentioned first data may be at least one first voice data, the above-mentioned first encoded data may be text encoded data obtained by the electronic device encoding the at least one first text data through a text encoder, and the second encoded data may be voice encoded data obtained by the electronic device encoding the at least one first voice data through voice encoding.

[0049] In some embodiments of the present application, the electronic device may first perform audio feature extraction on the at least one first voice data to generate a first voice spectrum. The mathematical representation of the first voice spectrum may be a two-dimensional matrix M(i,j), and the size may be (1280,80), which respectively represents the time length and feature dimension after the feature extraction of the at least one first voice data.

[0050] In some embodiments of the present application, after generating the first speech spectrum, the electronic device may use a speech encoder to encode the first speech spectrum to obtain speech encoding data H(i, j), and the speech encoding data H(i, j) may represent high-dimensional information after at least one first speech data is compressed, and its size may be (320, 80). Wherein '320' represents the length of the speech encoding data, and '80' represents the dimension of the speech encoding data.

[0051] In some embodiments of the present application, the dimension of text encoded data obtained by the electronic device encoding the at least one first text data through a text encoder can be 522817.

[0052] For example, when at least one first text data is "identify the content of this audio", its data size is (1,9). The size of the text encoding data T(i,j) obtained by the electronic device through the text encoder for "identify the content of this audio" is (9,522817). Among them, '9' represents the length of the text encoding data T(i,j), and '522817' represents the dimension of the text encoding data T(i,j).

[0053] Step 201b: The electronic device maps the data type of the second encoded data to the first data type to obtain mapping data.

[0054] In some embodiments of the present application, the electronic device can use a mapper to map the data type of the second encoded data to the first data type to obtain the mapping data, wherein the length of the mapping data is the same as the length of the second encoded data, and the data dimension of the mapping data is the same as the dimension of the first encoded data.

[0055] In some embodiments of the present application, when the first encoded data is text encoded data and the second encoded data is voice encoded data, the electronic device may use a voice mapper to map the voice encoded data to a text domain, that is, to map the data type of the voice encoded data to a text type to obtain mapping data, wherein the length of the mapping data is the same as the length of the first voice encoded data, and the data dimension of the mapping data is the same as the dimension of the text encoded data.

[0056] For example, the electronic device maps the data type mapped from the speech coding data to the text type, and the data size of the mapping data S(i,j) obtained can be (320,522817), where '320' represents the length of the mapping data, which is the same as the length of the speech coding data, and '522817' represents the dimension of the mapping data, which is the same as the dimension of the text coding data.

[0057] Step 201c: The electronic device concatenates the first coded data and the mapping data to obtain first data.

[0058] In some embodiments of the present application, the electronic device may concatenate the lengths of the first encoded data and the mapped data, keeping the dimension unchanged, to obtain the first data.

[0059] For example, the electronic device can concatenate the lengths of the mapping data S(i,j) and the text encoding data T(i,j) to obtain the above-mentioned first data. In the above-mentioned first data, the mapping data S(i,j) is in front and the text encoding data S(i,j) is in the back, and the size of the above-mentioned first data can be (329,522817). Among them, '329' represents the length of the first data, which is the sum of the length of the speech encoding data and the length of the text encoding data T(i,j), and '522817' represents the dimension of the mapping data, which is the same as the dimension of the text encoding data.

[0060] It should be noted that, if the target data is a single type of data, the first data is data obtained after the electronic device encodes the single type of data.

[0061] For example, when the target data is the at least one first text data “identify the content of the audio”, the first data may be text encoding data T(i, j) obtained by the electronic device through a text encoder for “identify the content of the audio”.

[0062] For another example, when the target data is the at least one first voice data, the first data may be mapping data S(i, j) corresponding to voice coding data obtained by encoding a first voice spectrum generated by an electronic device through a voice encoder to extract audio features of at least one first voice data.

[0063] In this way, the electronic device obtains the first encoded data and the second encoded data; maps the data type of the second encoded data to the first data type to obtain mapping data; and splices the first encoded data and the mapping data to obtain the first data, thereby being able to obtain encoded data of the same dimension, thereby facilitating the model to process data of different forms.

[0064] Step 202: The electronic device inputs the first data into a network layer corresponding to a data type of the first data for processing, and obtains a first processing result of the network layer.

[0065] In some embodiments of the present application, the data type of the first data is the data type of the target data, that is, the data type of the first data is the target data type.

[0066] In some embodiments of the present application, the structure of the above-mentioned network layer may be a multimodal expert system network structure.

[0067] In some embodiments of the present application, when the above-mentioned target data type includes at least one of a text data type and a voice data type, the structure of the above-mentioned network layer can be a speech text-modal expert system (ST-MES) network structure, which can realize mixed encoding of mapping data corresponding to text encoding data and voice encoding data.

[0068] In some embodiments of the present application, when the data type of the first data is voice data, the network layer includes an attention module, a first normalization module, a first linear transformation module corresponding to the voice data, and a second normalization module; Figure 1 ,like Figure 3 As shown, the above step 202 can be implemented through the following steps 202a1 to 202d1:

[0069] Step 202a1: The electronic device inputs the first data into an attention module to obtain an attention distribution result of the first data.

[0070] In some embodiments of the present application, when the data type of the above-mentioned first data is voice data, the electronic device can first increase the length of the above-mentioned first data to a preset length to obtain new first data, and then input the new first data into the above-mentioned attention module to obtain the attention distribution result of the first data. The above-mentioned preset length is the length of data that the first multimodal model can process.

[0071] In some embodiments of the present application, when the data type of the first data is voice data, the first data may be mapping data S(i, j).

[0072] For example, when the above-mentioned preset length is 1024, when the above-mentioned first data is the mapping data S(i,j), the electronic device can add 0 at the end of S(i,j) to update its data size from (320,522817) to (1024,522817), that is, update the length of the mapping data S(i,j) from 320 to 1024, and obtain the new first data I(i,j).

[0073] In some embodiments of the present application, the above-mentioned attention module can be a multi-head attention (Multi-HeadAttention) module. The Multi-HeadAttention module is an extended form of the attention mechanism widely used in the Transformer model. It obtains the attention distribution of different subspaces of the new first data by running multiple independent attention mechanisms in parallel, thereby more comprehensively capturing the potential multiple semantic associations in the new first data.

[0074] Step 202b1: The electronic device inputs the attention distribution result into a first normalization module to obtain a first normalized result of the attention distribution result.

[0075] In some embodiments of the present application, the first normalization module can be a residual connection and normalization Add&Norm module to obtain a first normalized result of the attention distribution result. The main function of the Add&Norm module is to transmit information deeper and enhance the fitting ability of the model. The Add&Norm module includes a residual connection submodule and a normalization submodule.

[0076] In some embodiments of the present application, the electronic device may first input the above-mentioned attention distribution result into the residual connection submodule to increase the depth of the attention distribution result without losing the initial feature information of the above-mentioned attention distribution result. The residual connection submodule can enable the above-mentioned first multimodal model to learn more complex feature representations in the attention distribution result during the training process, thereby improving the expression ability of the model, and can enable the original gradient information in the attention distribution result to be transmitted to the deeper layer of the network, thereby alleviating the problem of the gradient disappearance of the attention distribution result.

[0077] In some embodiments of the present application, the electronic device may input the attention distribution result output by the residual connection submodule into the normalization submodule to normalize the attention distribution result to obtain the above-mentioned first normalized result. The electronic device normalizes the input data of the first multimodal model through the normalization submodule, which can reduce the problem of internal covariate shift of the first multimodal model, so that the input data of each layer in the first multimodal model is kept on a similar scale, which is beneficial to the training and optimization of the model. In addition, normalization operation through the normalization submodule can help the model converge faster and improve the stability of training.

[0078] Step 202c1: The electronic device inputs the first normalization result into a first linear transformation module to obtain a linear transformation result of the first normalization result.

[0079] In some embodiments of the present application, the structure of the first linear transformation module may be a feedforward neural network FFN structure, including a first fully connected layer, an activation layer, and a second fully connected layer, and the formula thereof may be:

[0080] FFN(x)=max(0,x*w1+b1)*w2+b2; (1)

[0081] Among them, x*w1+b1 is the calculation formula of the first fully connected layer, max(0,x*w1+b1) is the calculation formula of the activation layer, max(0,x*w1+b1)*w2+b2 is the calculation formula of the second fully connected layer, x represents the first normalization result, w1 represents the weight of the first fully connected layer, b1 represents the bias vector of the first fully connected layer, w2 represents the weight of the second fully connected layer, and b2 represents the bias vector of the second fully connected layer.

[0082] In some embodiments of the present application, when the data type of the first data is speech data, that is, the target data type is speech data type, that is, the target data is the at least one first speech data, the linear transformation module is a speech FFN corresponding to the speech data. The speech FFN can be an FFN in the first multimodal model trained with speech coding data.

[0083] It should be noted that when the target data type is different, the structures of the linear transformation modules corresponding to the different data types are the same, all of which are FFN structures, but the corresponding weight parameters and bias vectors of the linear transformation modules corresponding to the different data types are different.

[0084] In this way, when the data type of the first data is voice data, the electronic device processes the first data through the first linear transformation module corresponding to the voice data, so that the first data can obtain the best encoding effect, thereby improving the accuracy of model reasoning.

[0085] Step 202d1: The electronic device inputs the linear transformation result into the second normalization module to obtain a first processing result.

[0086] It should be noted that the implementation process of the above step 202d is similar to that of the above step 202b. To avoid repetition, this embodiment will not be repeated here.

[0087] It should be noted that the output of each module in the above-mentioned first multimodal model is consistent with the size of the above-mentioned first data.

[0088] For example, if the size of the first data is (1024, 522817), then the sizes of the attention distribution result, the first normalization result, the linear transformation result and the first processing result are all (1024, 522817), which is consistent with the size of the first data.

[0089] In this way, when the data type of the first data is voice data, the electronic device can input the first data into the attention module to obtain the attention distribution result of the first data, thereby further refining the attention of the first data and expanding the model's ability to focus on different positions; input the attention distribution result into the first normalization module to obtain the first normalized result of the attention distribution result, thereby transmitting information more deeply and enhancing the fitting ability of the model; input the first normalized result into the first linear transformation module to obtain the linear transformation result of the first normalized result, thereby capturing features of the first normalized result.

[0090] In some embodiments of the present application, when the data type of the first data is text data, the network layer includes an attention module, a first normalization module, a second linear transformation module corresponding to the text data, and a second normalization module; the step 202 can be implemented by the following steps 202a2 to 202d2:

[0091] Step 202a2: The electronic device inputs the first data into an attention module to obtain an attention distribution result of the first data.

[0092] It should be noted that the implementation process of step 202a2 is similar to that of step 202a1. To avoid repetition, the embodiment of the present application will not be repeated here.

[0093] Step 202b2: The electronic device inputs the attention distribution result into a first normalization module to obtain a first normalized result of the attention distribution result.

[0094] It should be noted that the implementation process of step 202b2 is similar to that of step 202b1. To avoid repetition, the embodiment of the present application will not be repeated here.

[0095] Step 202c2: The electronic device inputs the first normalization result into the second linear transformation module to obtain a linear transformation result of the first normalization result.

[0096] It should be noted that the implementation process of step 202c2 is similar to that of step 202c1. To avoid repetition, the embodiment of the present application will not be repeated here.

[0097] In some embodiments of the present application, the structure of the second linear transformation module is similar to that of the first linear transformation module. To avoid repetition, this embodiment will not be described again.

[0098] It should be noted that the implementation process of step 202c2 is similar to that of step 202c1. To avoid repetition, the embodiment of the present application will not be repeated here.

[0099] In some embodiments of the present application, when the data type of the first data is text data, that is, the target data type is a text data type, that is, the target data is the at least one first text data, the second linear transformation module is a text FFN corresponding to the text data. The text FFN can be an FFN in the first multimodal model trained with text encoding data.

[0100] Step 202d2: The electronic device inputs the linear transformation result into the second normalization module to obtain a first processing result.

[0101] It should be noted that the implementation process of step 202c2 is similar to that of step 202c1. To avoid repetition, the embodiment of the present application will not be repeated here.

[0102] It should be noted that the data type of the above-mentioned first data can also be other single data types, for example, the first data can be video data, or the quantity type of the above-mentioned first data can also include at least two data types, for example, the first data can be voice data and text data. However, regardless of whether the data type of the first data is other single data types including at least two data types, the network layer corresponding to the first data includes an attention module, a first normalization module, a linear transformation module corresponding to the data type of the first data, and a second normalization module, and the processing process of the first data by the network layer corresponding to the first data is similar to the processing process of the first data by the network layer corresponding to the voice data and the processing process of the first data by the network layer corresponding to the text data. To avoid repetition, this embodiment will not be repeated here.

[0103] Step 203: The electronic device obtains a second processing result of the first data based on the first processing result.

[0104] In some embodiments of the present application, the above step 203 may be implemented by the following step 203a:

[0105] Step 203a: The electronic device inputs the first processing result into the multimodal model for decoding to obtain a second processing result.

[0106] In some embodiments of the present application, the multimodal model may be a large speech model. When the target data includes at least one of the at least one first text data and the at least one first speech data, the electronic device may input the first processing result into the large speech model, and decode the first processing result through the large language model to obtain a second processing result of the first data.

[0107] In some embodiments of the present application, the size of the second processing result is consistent with the size of the first data. For example, if the size of the first data is (1024, 522817), the size of the second processing result is (1024, 522817), which is consistent with the size of the first data.

[0108] In the data processing method provided in the present application, first data is obtained; the first data is input into a network layer corresponding to the data type of the first data for processing, and a first processing result of the network layer is obtained; based on the first processing result, a second processing result of the first data is obtained. In this way, the first data is processed by the network layer corresponding to the data type, and when the data type of the first data is different, the first processing result corresponding to the different data types can be obtained, so that when the first processing result is processed, the processing results corresponding to the different data types can be obtained, thereby improving the accuracy of model reasoning.

[0109] In some embodiments of the present application, the target data includes at least one first voice data;

[0110] Before the above step 201, the data processing method provided by the present application further includes the following step 204:

[0111] Step 204: The electronic device obtains at least one first voice data collected by the microphone in response to the user's first input to the sound receiving control on the first interface.

[0112] In some embodiments of the present application, the above-mentioned first input may include any of the following: user's click input, sliding input, pressing input, voice input, gesture input, or other feasible input, which is not limited in the embodiments of the present application.

[0113] In some embodiments of the present application, the above-mentioned gesture input may include but is not limited to at least one of the following: click gesture, sliding gesture, drag gesture, pressure recognition gesture, long press gesture, area change gesture, double press gesture, double click gesture, specific gesture input or other possible gesture inputs. The specific gesture input form can be determined according to actual needs and is not limited to this in some embodiments.

[0114] In some embodiments of the present application, the above-mentioned click input can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long-press input or a short-press input, which is not limited in some embodiments.

[0115] In some embodiments of the present application, the above-mentioned sliding input can be a sliding input in any direction, such as sliding up, sliding down, sliding to the left or sliding to the right, etc., which is not limited in some embodiments.

[0116] In some embodiments of the present application, the first input is used to trigger the electronic device to collect the at least one first voice data through a speaker.

[0117] In some embodiments of the present application, the at least one first voice data may be a segment of user's voice collected by a speaker.

[0118] For example, combined with Figure 4 As shown, when the electronic device is a mobile phone, and the first interface 60 is a voice chat interface for human-computer communication on the mobile phone, the user can long-press the sound receiving control 601 on the first interface 60 and speak to the microphone of the mobile phone. The mobile phone can respond to the long-press input of the sound receiving control 601 on the first interface 60 by the user, and collect the user's speaking voice through the microphone. When the user finishes speaking, the long-press input of the sound receiving control 601 can be stopped, and the microphone of the mobile phone stops collecting the user's speaking voice at the same time.

[0119] In this way, the electronic device obtains at least one first voice data collected by the microphone in response to the user's first input to the sound receiving control on the first interface, and can quickly obtain the input data.

[0120] In some embodiments of the present application, the target data includes at least one first text data;

[0121] Before the above step 201, the data processing method provided by the present application further includes the following step 205:

[0122] Step 205: The electronic device obtains at least one first text data input by the user on the first interface in response to the user's second input to the text control on the first interface.

[0123] In some embodiments of the present application, the second input may include any one of the following: user click input, sliding input, pressing input, voice input, gesture input, or other feasible input, which is not limited in the embodiments of the present application.

[0124] In some embodiments of the present application, the second input is used to trigger the electronic device to obtain the at least one first text data.

[0125] For example, combined with Figure 4 As shown, when the electronic device is a mobile phone, and the first interface 60 is a voice chat interface for human-computer communication on the mobile phone, the user can perform at least one pressing input on the text control 602 on the first interface 60 to input at least one text data on the first interface 60. The mobile phone can obtain at least one text data of the user input value in response to the user's at least one pressing input on the text control 602 on the first interface 60.

[0126] In this way, the electronic device obtains at least one first text data input by the user on the first interface in response to the user's first input to the text control on the first interface, and can quickly obtain the input data.

[0127] In some embodiments of the present application, after the above step 203, the data processing method provided in the embodiment of the present application may further include the following step 206a and the following step 206c:

[0128] Step 206a: In response to the third input from the user, the electronic device uses a text encoder to decode the second processing result to obtain second text data corresponding to the second processing result.

[0129] In some embodiments of the present application, the third input is used to trigger the display of the processing result in text form.

[0130] In some embodiments of the present application, the third input may include any one of the following: user click input, sliding input, pressing input, voice input, gesture input, or other feasible input, which is not limited in the embodiments of the present application.

[0131] In some embodiments of the present application, when the electronic device uses a text encoder to decode the second processing result, the size of the processing result can be converted into the size corresponding to the second text data to obtain the second text data. For example, if the size of the second processing result is (1024, 522817), the size of the second text data can be (26668, 31375) after decoding the processing result through the text encoder, and the corresponding text is "Beijing".

[0132] In this way, the electronic device decodes the second processing result using a text encoder in response to the user's third input to obtain second text data corresponding to the second processing result, and can display the corresponding processing result according to the user's needs.

[0133] Step 206b: The electronic device displays the second text data on the first interface.

[0134] In some embodiments of the present application, the electronic device may display the second text data in text form on the first interface.

[0135] In some embodiments of the present application, after the above step 203, the data processing method provided in the embodiment of the present application may further include the following steps 207a to 207c:

[0136] Step 207a: The electronic device maps the second processing result into second voice data in response to the fourth input of the user.

[0137] In some embodiments of the present application, the fourth input is used to trigger the playing of the second processing result in voice mode.

[0138] In some embodiments of the present application, the fourth input may include any one of the following: user click input, sliding input, pressing input, voice input, gesture input, or other feasible input, which is not limited in the embodiments of the present application.

[0139] In some implementations of the present application, the electronic device may use a speech mapper to compress and map the above-mentioned second processing result column by column, compress the dimension of the second processing result, and map the second processing result into second text data in the speech domain, that is, map the second processing result into a speech spectrum.

[0140] For example, the electronic device may use a voice mapper to compress the dimension of the second processing node from 522817 to 80, so that its data dimension is reduced from (1024, 522817) to (1024, 80).

[0141] Step 207b: The electronic device decodes the second voice data to obtain third voice data corresponding to the processing result.

[0142] In some embodiments of the present application, the electronic device may use a vocoder to decode the second voice data to obtain third voice data corresponding to the processing result from the second voice data.

[0143] Step 207c: the electronic device plays the third voice data.

[0144] In some embodiments of the present application, the electronic device can play the third voice data through a speaker.

[0145] In this way, the electronic device maps the second processing result to second voice data in response to the user's fourth input; decodes the second voice data to obtain third voice data corresponding to the second processing result; and plays the third voice data to display the corresponding processing result according to the user's needs.

[0146] Next, we take the electronic device as a mobile phone, the network layer structure as the ST-ME network, and the multimodal model as a large language model as an example. Figure 5, the data processing method provided in this application is further described in detail. Figure 6 It is a flowchart of the data processing method provided in an embodiment of the present application. The method may include the following steps 801 to 814.

[0147] Step 801: A mobile phone receives at least one first voice data input by a user.

[0148] For example, combined with Figure 4 As shown, when the user clicks the sound receiving control 601 on the first interface 60, the mobile phone will automatically call the microphone device to record the sound of the user's oral call, that is, the at least one first voice data mentioned above.

[0149] Step 802: The mobile phone extracts audio features from the received first voice data.

[0150] Exemplarily, the mobile phone performs audio feature extraction on at least one first voice data to generate a voice spectrum, which is mathematically represented as a two-dimensional matrix M(i,j) with a size of (1280,80), which respectively represents the time length and feature dimension of the voice spectrum extracted from the audio feature of at least one first voice data.

[0151] Step 803: The mobile phone uses a speech encoder to encode the speech spectrum to obtain speech encoding data.

[0152] Exemplarily, the mobile phone can pass the voice spectrum M(i,j) into the voice encoder 701, compress and encode the voice spectrum M(i,j) to obtain the voice encoding data H(i,j). The voice encoding data H(i,j) can represent the high-dimensional information after the voice spectrum is compressed, and its size is (320, 80).

[0153] Step 804: The mobile phone uses a voice mapper to map the encoded voice data to a text domain to generate mapping data.

[0154] Exemplarily, the mobile phone sends the speech coding data H(i,j) to the speech mapper 702, which maps it to the text domain range to generate mapping data S(i,j), and the size of the mapping data S(i,j) changes from (320,80) of H(i,j) to (320,522817).

[0155] Step 805: The mobile phone receives at least one first text data input by the user.

[0156] For example, in combination Figure 4 As shown, the mobile phone can receive data from the user on the text control 602 on the first interface 60, such as the keyboard control, and obtain at least one first text data input by the user, such as when the at least one first text data is "the data size of at least one first text data is (1,9).

[0157] Step 806: The mobile phone encodes the received at least one first text data to obtain text encoding data.

[0158] Exemplarily, the mobile phone can input at least one first text data such as "identify the content of this audio" into the text encoder 703, and the text encoder 703 encodes the at least one first text data to obtain text encoding data T(i,j). The data size of the text encoding data T(i,j) changes from (1,9) to (9,522817).

[0159] Step 807: The mobile phone performs hybrid coding conversion processing on the mapping data and / or the text coding data through the first multimodal model of the ST-MES network structure to obtain a first processing result.

[0160] For example, the mobile phone can concatenate the data S(i,j) and T(i,j) to obtain new data I(i,j). The specific operation is as follows:

[0161] When there is only mapping data, add 0 to the end of S(i,j) to increase its data dimension from (320,522817) to (1024,522817) and name it I(i,j);

[0162] When there is only text encoding data, add 0 to the end of T(i,j) to increase its data dimension from (9,522817) to (1024,522817) and name it I(i,j);

[0163] When there are both mapping data and text encoding data, the S(i,j) and T(i,j) data are concatenated on the horizontal axis, with the mapping data in front and the text encoding data in the back. The data size changes from (320,522817) and (9,522817) to (329,522817), and the model is padded with 0 to increase its dimension to (1024,522817), and named I(i,j). Then I(i,j) is used as the input of the ST-MES network structure for encoding conversion.

[0164] Exemplarily, the structure of the first multimodal model 704 of the ST-MES network structure is as follows: Figure 7As shown, it consists of 6 submodules, namely, Multi-HeadAttention module 7041, first Add&Norm module 7042, speech FFN module 7043, text FFN module 7044, speech-text FFN module 7045 and second Add&Norm module 7046. Among them, the speech FFN module 7043 corresponds to the speech data type, the text FFN module 7044 corresponds to the text data type, and the speech-text FFN module 7045 corresponds to the speech data type and the text data type.

[0165] For example, the mobile phone can first input I(i, j) into the Multi-HeadAttention module and the first Add&Norm module, and use the attention mechanism to extract key features to obtain data H i , the data size remains unchanged and is always (1024,522817).

[0166] For example, when the user only inputs at least one first voice data, the mobile phone can i Input voice FFN module 7043; when the user only inputs at least one first text data, H i Input text FFN module 7044; when the user simultaneously inputs at least one first voice data and at least one first text data, H i Input speech text to FFN module 7045.

[0167] The first multimodal model of the ST-MES network structure can assign separate network parameters for learning to different types of inputs when processing input data of different data types, making the model more professional when processing the modal data.

[0168] Exemplarily, the result output by the FFN module corresponding to each data type is passed through the second Add&Norm module 7046 to obtain the output of the second Add&Norm module 7046, that is, the above-mentioned first processing result, and the data size of the first processing result remains unchanged.

[0169] Step 808: The mobile phone inputs the first processing result into the large language model for decoding processing to obtain a decoding result (ie, the above-mentioned second processing result).

[0170] Exemplarily, the mobile phone may input the first processing result generated in the above step 807 into the large language model 705 for decoding, wherein the large language model is a conventional transformer structure model, and obtain the decoding result J through decoding with the large language model.

[0171] Step 809: The mobile phone receives the third input from the user.

[0172] Exemplarily, the third input is used to trigger the mobile phone to play the decoding result in language.

[0173] Step 810: The mobile phone maps the decoding result in response to the third input to obtain second voice data.

[0174] Exemplarily, the mobile phone can compress and map the decoding result J column by column through the speech mapper 702 to compress the dimension of the decoding result J from 522817 to 80. The purpose is to switch the data from the text domain to the speech domain to obtain the second speech data, which is the speech spectrum.

[0175] Step 811: The mobile phone encodes the second voice data to obtain third voice data corresponding to the decoding result.

[0176] Exemplarily, the electronic device may encode the second voice data through the vocoder 706 and restore the voice from the second voice data, namely, the third voice data.

[0177] Step 812: The mobile phone plays the third voice data through the speaker.

[0178] Step 813: The mobile phone encodes the decoding result in response to the fourth input to obtain second text data corresponding to the decoding result.

[0179] Exemplarily, the fourth input is used to trigger the mobile phone to play the above decoding result in voice mode.

[0180] Exemplarily, the mobile phone can decode the decoding result through the text encoder 703, convert the data size of the decoding result, and obtain the second text data. For example, the mobile phone can convert the decoding result from (1024,522817) to [26668,31375] through the text encoder 707, and the corresponding text data is "Beijing".

[0181] Step 814: The mobile phone displays the second text data to the user through the mobile phone screen.

[0182] Exemplarily, the mobile phone may display the second text data on the first interface.

[0183] The data processing method provided in the embodiment of the present application can adapt to the input forms of different data types through the first multimodal model of the ST-ME network structure, so as to provide targeted decoding results of different data types to the user. If the user only inputs text, only the text FFN module is called without calling the voice FFN module and the voice-text FFN network, thereby realizing seamless switching of the T-ME network structure at the user level and enabling the model to achieve the best output effect.

[0184] It should be noted that the specific implementation process of the data processing method in the above steps can refer to the relevant description of the above embodiment. To avoid repetition, this embodiment will not be repeated here.

[0185] It should be noted that the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or any two or more can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not impose any restrictions on this.

[0186] The data processing method provided in the embodiment of the present application can be executed by a data processing device. In the embodiment of the present application, the data processing device provided in the embodiment of the present application is described by taking the data processing method executed by the data processing device as an example.

[0187] Figure 8 1 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application, wherein the data processing device 1000 includes an acquisition module 1001 and a processing module 1002;

[0188] The acquisition module 1001 is used to acquire first data;

[0189] The processing module 1002 is used to input the first data into a network layer corresponding to a data type of the first data for processing, so as to obtain a first processing result of the network layer;

[0190] The processing module 1002 is further configured to obtain a second processing result of the first data based on the processing result of the network layer.

[0191] In some embodiments of the present application, when the data type of the first data is voice data, the network layer includes an attention module, a first normalization module, a first linear transformation module corresponding to the voice data, and a second normalization module;

[0192] The processing module 1002 is specifically used for:

[0193] Inputting the first data into the attention module to obtain an attention distribution result of the first data;

[0194] Inputting the attention distribution result into the first normalization module to obtain a first normalized result of the attention distribution result;

[0195] Inputting the first normalization result into the linear transformation module to obtain a linear transformation result of the first normalization result;

[0196] The linear transformation result is input into the second normalization module to obtain the first processing result.

[0197] In some embodiments of the present application, when the data type of the first data is text data, the network layer includes an attention module, a first normalization module, a second linear transformation module corresponding to the text data, and a second normalization module;

[0198] The processing module 1002 is specifically used for:

[0199] Inputting the first data into the attention module to obtain an attention distribution result of the first data;

[0200] Inputting the attention distribution result into the first normalization module to obtain a first normalized result of the attention distribution result;

[0201] Inputting the first normalization result into the second linear transformation module to obtain a linear transformation result of the first normalization result;

[0202] The linear transformation result is input into the second normalization module to obtain the first processing result.

[0203] In some embodiments of the present application, the processing module 1002 is specifically used to:

[0204] The first processing result is input into a multimodal model for decoding to obtain the second processing result.

[0205] In some embodiments of the present application, the acquisition module 1001 is specifically used to:

[0206] Obtaining first coded data and second coded data, wherein the first coded data is obtained by encoding data of a first data type in the target data, and the second coded data is obtained by encoding data of a second data type in the target data, and the first data type and the second data type are different;

[0207] Mapping the data type of the second encoded data to the first data type to obtain mapping data;

[0208] The first encoded data and the mapping data are concatenated to obtain the first data.

[0209] In some embodiments of the present application, the target data includes at least one first voice data;

[0210] The acquisition module 1001 is further used for:

[0211] Before acquiring the first data, in response to a first input of a user to a sound receiving control on the first interface, the at least one first voice data collected by the microphone is acquired.

[0212] In some embodiments of the present application, the target data includes at least one first text data;

[0213] The acquisition module 1001 is further used for:

[0214] Before acquiring the first data, in response to the user's second input to the text control on the first interface, the at least one first text data input by the user on the first interface is acquired.

[0215] In some embodiments of the present application, Figure 8 ,like Fig. 9 As shown, the device 1000 further includes: a decoding module 1003, which is used to obtain a second processing result of the first data based on the first processing result, and then, in response to a third input of a user, decode the second processing result using a text encoder to obtain second text data corresponding to the second processing result, wherein the third input is used to trigger the display of the second processing result in text form;

[0216] The device 1000 further includes: a display module 1004, configured to display the second text data on a first interface.

[0217] In some embodiments of the present application, Figure 8 ,like Fig. 9 As shown, the device 1000 further includes: a decoding module 1003, configured to obtain a second processing result of the first data based on the first processing result, and map the second processing result to second voice data in response to a fourth input of a user, wherein the fourth input is used to trigger the playing of the second processing result in voice mode;

[0218] Decoding the second voice data to obtain third voice data corresponding to the second processing result;

[0219] The device 1000 further includes: a playing module 1005, configured to play the third voice data.

[0220] In the data processing device provided in the present application, the data processing device obtains first data; inputs the first data into a network layer corresponding to the data type of the first data for processing, and obtains a first processing result of the network layer; based on the first processing result, obtains a second processing result of the first data. In this way, the first data is processed by the network layer corresponding to the data type, and when the data type of the first data is different, the first processing result corresponding to the different data types can be obtained, so that when the first processing result is processed, the processing results corresponding to the different data types can be obtained, thereby improving the accuracy of model reasoning.

[0221] The data processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device, an augmented reality / virtual reality device, a robot, a wearable device, a super mobile personal computer, a netbook or a personal digital assistant, etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0222] The data processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0223] The data processing device provided in the embodiment of the present application can implement each process implemented by each embodiment of the above-mentioned data processing method. To avoid repetition, it will not be described again here.

[0224] Alternatively, if Fig.10 As shown, an embodiment of the present application also provides an electronic device 1200, including a processor 1201 and a memory 1202, wherein the memory 1202 stores programs or instructions that can be executed on the processor 1201, and when the program or instructions are executed by the processor 1201, the various steps of the above-mentioned data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0225] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0226] Fig.11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0227] The electronic device 1300 includes but is not limited to: a radio frequency unit 1301, a network module 1302, an audio output unit 1303, an input unit 1304, a sensor 1305, a display unit 1306, a user input unit 1307, an interface unit 1308, a memory 1309, and a processor 1310 and other components.

[0228] Those skilled in the art will appreciate that the electronic device 1300 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1310 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0229] The processor 1310 is configured to obtain first data;

[0230] Inputting the first data into a network layer corresponding to a data type of the first data for processing, to obtain a first processing result of the network layer;

[0231] Based on the first processing result, a second processing result of the first data is obtained.

[0232] In some embodiments of the present application, when the data type of the first data is voice data, the network layer includes an attention module, a first normalization module, a first linear transformation module corresponding to the voice data, and a second normalization module;

[0233] The processor 1310 is specifically configured to:

[0234] Inputting the first data into the attention module to obtain an attention distribution result of the first data;

[0235] Inputting the attention distribution result into the first normalization module to obtain a first normalized result of the attention distribution result;

[0236] Inputting the first normalization result into the linear transformation module to obtain a linear transformation result of the first normalization result;

[0237] The linear transformation result is input into the second normalization module to obtain the first processing result.

[0238] In some embodiments of the present application, when the data type of the first data is text data, the network layer includes an attention module, a first normalization module, a second linear transformation module corresponding to the text data, and a second normalization module;

[0239] The processor 1310 is specifically configured to:

[0240] Inputting the first data into the attention module to obtain an attention distribution result of the first data;

[0241] Inputting the attention distribution result into the first normalization module to obtain a first normalized result of the attention distribution result;

[0242] Inputting the first normalization result into the second linear transformation module to obtain a linear transformation result of the first normalization result;

[0243] The linear transformation result is input into the second normalization module to obtain the first processing result.

[0244] In some embodiments of the present application, the processor 1310 is specifically configured to:

[0245] The first processing result is input into a multimodal model for decoding to obtain the second processing result.

[0246] In some embodiments of the present application, the processor 1310 is specifically configured to:

[0247] Obtaining first coded data and second coded data, wherein the first coded data is obtained by encoding data of a first data type in the target data, and the second coded data is obtained by encoding data of a second data type in the target data, and the first data type and the second data type are different;

[0248] Mapping the data type of the second encoded data to the first data type to obtain mapping data;

[0249] The first encoded data and the mapping data are concatenated to obtain the first data.

[0250] In some embodiments of the present application, the target data includes at least one first voice data; the processor 1310 is further configured to:

[0251] Before acquiring the first data, in response to a first input of a user to a sound receiving control on the first interface, the at least one first voice data collected by the microphone is acquired.

[0252] In some embodiments of the present application, the target data includes at least one first text data; the processor 1310 is further configured to:

[0253] Before acquiring the first data, in response to the user's second input to the text control on the first interface, the at least one first text data input by the user on the first interface is acquired.

[0254] In some embodiments of the present application, the processor 1310 is further configured to:

[0255] After obtaining a second processing result of the first data based on the first processing result, in response to a third input from a user, decoding the second processing result using a text encoder to obtain second text data corresponding to the second processing result, wherein the third input is used to trigger displaying the second processing result in text form;

[0256] The display unit 1306 is configured to display the second text data on the first interface.

[0257] In some embodiments of the present application, after obtaining a second processing result of the first data based on the first processing result, the second processing result is mapped to second voice data in response to a fourth input of the user, and the fourth input is used to trigger playing the second processing result in voice mode;

[0258] Decoding the second voice data to obtain third voice data corresponding to the processing result;

[0259] The audio output unit 1303 is used to play the third voice data.

[0260] In the electronic device provided in the present application, the electronic device obtains first data; inputs the first data into a network layer corresponding to the data type of the first data for processing, and obtains a first processing result of the network layer; based on the first processing result, obtains a second processing result of the first data. In this way, the first data is processed by the network layer corresponding to the data type, and when the data type of the first data is different, the first processing result corresponding to the different data types can be obtained, so that when the first processing result is processed, the processing results corresponding to the different data types can be obtained, thereby improving the accuracy of model reasoning.

[0261] It should be understood that in the embodiment of the present application, the input unit 1304 may include a graphics processor (Graphics Processing Unit, GPU) 13041 and a microphone 13042, and the graphics processor 13041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1306 may include a display panel 13061, and the display panel 13061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1307 includes a touch panel 13071 and at least one of other input devices 13072. The touch panel 13071 is also called a touch screen. The touch panel 13071 may include two parts: a touch detection device and a touch controller. Other input devices 13072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0262] The memory 1309 can be used to store software programs and various data. The memory 1309 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1309 may include a volatile memory or a non-volatile memory, or the memory 1309 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1309 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0263] The processor 1310 may include one or more processing units; optionally, the processor 1310 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1310.

[0264] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0265] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0266] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned data processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0267] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0268] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned data processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0269] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0270] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0271] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire first data; Inputting the first data into a network layer corresponding to a data type of the first data for processing, to obtain a first processing result of the network layer; Based on the first processing result, a second processing result of the first data is obtained.

2. The method according to claim 1, characterized in that In the case where the data type of the first data is voice data, the network layer includes an attention module, a first normalization module, a first linear transformation module corresponding to the voice data, and a second normalization module; Inputting the first data into a network layer corresponding to the data type of the first data for processing to obtain a first processing result of the network layer includes: Inputting the first data into the attention module to obtain an attention distribution result of the first data; Inputting the attention distribution result into the first normalization module to obtain a first normalized result of the attention distribution result; Inputting the first normalization result into the linear transformation module to obtain a linear transformation result of the first normalization result; The linear transformation result is input into the second normalization module to obtain the first processing result.

3. The method according to claim 1, characterized in that: When the data type of the first data is text data, the network layer includes an attention module, a first normalization module, a second linear transformation module corresponding to the text data, and a second normalization module; Inputting the first data into a network layer corresponding to the data type of the first data for processing to obtain a first processing result of the network layer includes: Inputting the first data into the attention module to obtain an attention distribution result of the first data; Inputting the attention distribution result into the first normalization module to obtain a first normalized result of the attention distribution result; Inputting the first normalization result into the second linear transformation module to obtain a linear transformation result of the first normalization result; The linear transformation result is input into the second normalization module to obtain the first processing result.

4. The method according to claim 1, characterized in that The obtaining, based on the first processing result, a second processing result of the first data, comprises: The first processing result is input into a multimodal model for decoding to obtain the second processing result.

5. The method according to claim 1, characterized in that The obtaining of the first data includes: Obtaining first coded data and second coded data, wherein the first coded data is obtained by encoding data of a first data type in the target data, and the second coded data is obtained by encoding data of a second data type in the target data, and the first data type and the second data type are different; Mapping the data type of the second encoded data to the first data type to obtain mapping data; The first encoded data and the mapping data are concatenated to obtain the first data.

6. The method according to claim 5, characterized in that The target data includes at least one first voice data; Before obtaining the first data, the method further includes: In response to a first input of a user to a sound receiving control on a first interface, the at least one first voice data collected by a microphone is acquired.

7. The method according to claim 5, characterized in that The target data includes at least one first text data; Before obtaining the first data, the method further includes: In response to a second input of the user to the text control on the first interface, the at least one first text data input by the user on the first interface is obtained.

8. The method according to any one of claims 1 to 7, characterized in that: After obtaining a second processing result of the first data based on the first processing result, the method further includes: In response to a third input from a user, decoding the second processing result using a text encoder to obtain second text data corresponding to the second processing result, wherein the third input is used to trigger displaying the second processing result in text form; The second text data is displayed on the first interface.

9. The method according to any one of claims 1 to 7, characterized in that: After obtaining a second processing result of the first data based on the first processing result, the method further includes: In response to a fourth input from a user, mapping the second processing result to second voice data, the fourth input being used to trigger playing the second processing result in voice mode; Decoding the second voice data to obtain third voice data corresponding to the processing result; Play the third voice data.

10. A data processing device, characterized in that: The device comprises: an acquisition module and a processing module; The acquisition module is used to acquire first data; The processing module is used to input the first data into a network layer corresponding to a data type of the first data for processing, so as to obtain a first processing result of the network layer; The processing module is further used to obtain a second processing result of the first data based on the processing result of the network layer.

11. The device according to claim 10, characterized in that In the case where the data type of the first data is voice data, the network layer includes an attention module, a first normalization module, a first linear transformation module corresponding to the voice data, and a second normalization module; The processing module is specifically used for: Inputting the first data into the attention module to obtain an attention distribution result of the first data; Inputting the attention distribution result into the first normalization module to obtain a first normalized result of the attention distribution result; Inputting the first normalization result into the linear transformation module to obtain a linear transformation result of the first normalization result; The linear transformation result is input into the second normalization module to obtain the first processing result.

12. The device according to claim 10, characterized in that When the data type of the first data is text data, the network layer includes an attention module, a first normalization module, a second linear transformation module corresponding to the text data, and a second normalization module; The processing module is specifically used for: Inputting the first data into the attention module to obtain an attention distribution result of the first data; Inputting the attention distribution result into the first normalization module to obtain a first normalized result of the attention distribution result; Inputting the first normalization result into the second linear transformation module to obtain a linear transformation result of the first normalization result; The linear transformation result is input into the second normalization module to obtain the first processing result.

13. The device according to claim 12, characterized in that The processing module is specifically used for: The first processing result is input into a multimodal model for decoding to obtain the second processing result.

14. The device according to claim 10, characterized in that The acquisition module is specifically used for: Obtaining first coded data and second coded data, wherein the first coded data is obtained by encoding data of a first data type in the target data, and the second coded data is obtained by encoding data of a second data type in the target data, and the first data type and the second data type are different; Mapping the data type of the second encoded data to the first data type to obtain mapping data; The first encoded data and the mapping data are concatenated to obtain the first data.

15. The device according to claim 14, characterized in that The target data includes at least one first voice data; The acquisition module is further used for: Before acquiring the first data, in response to a first input of a user to a sound receiving control on the first interface, the at least one first voice data collected by the microphone is acquired.

16. The device according to claim 14, characterized in that The target data includes at least one first text data; The acquisition module is further used for: Before acquiring the first data, in response to a second input of the user to the text control on the first interface, the at least one first text data input by the user on the first interface is acquired.

17. The device according to any one of claims 10 to 16, characterized in that The device further includes: a decoding module, configured to obtain a second processing result of the first data based on the first processing result, and then, in response to a third input from a user, decode the second processing result using a text encoder to obtain second text data corresponding to the second processing result, wherein the third input is used to trigger the display of the second processing result in text form; The device further includes: a display module, configured to display the second text data on the first interface.

18. The device according to any one of claims 10 to 16, characterized in that The device further includes: a decoding module, configured to obtain a second processing result of the first data based on the first processing result, and map the second processing result to second voice data in response to a fourth input of a user, wherein the fourth input is used to trigger playing the second processing result in voice mode; Decoding the second voice data to obtain third voice data corresponding to the second processing result; The device also includes: a playing module, configured to play the third voice data.

19. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of data processing according to any one of claims 1 to 9 are implemented.

20. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of data processing according to any one of claims 1 to 9 are implemented.

Images