Data processing method and device
By converting multimedia data into binary byte sequences and using multimedia processing models for processing, the shortcomings of deep learning models in processing binary data are solved, the prediction capability and efficiency of the model are improved, and efficient processing of cross-modal tasks is achieved.
Patent Information
- Application Number
- CN202411191435.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-19
AI Technical Summary
The existing deep learning models have significant shortcomings in processing native binary data, and are unable to effectively process and simulate binary data, resulting in the underdeveloped model prediction capabilities and speed.
By converting multimedia data into binary byte sequences and processing using a multimedia processing model including linear projection unit, sequence decoding unit and byte decoding unit, the conversion and prediction of byte sequence vectors are realized, and the goal of the target prediction task is constructed. Task data.
It improves the model's comprehensive understanding and simulation capabilities of digital systems, improves the processing efficiency of multimedia data, the applicability and performance of cross-modal tasks, and simplifies the integration process between different data types.
Smart Images

Figure CN118982002A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this specification relate to the field of machine learning technology, and more particularly to data processing methods and devices. Background Art
[0002] With the development of computer and Internet technology, deep learning technology has been applied in more and more scenarios. Deep learning usually focuses on digital media files that are easy for humans to understand, such as text, audio, and images; existing deep learning models mostly operate in the field of media data. For example, language models perform well in text generation and understanding through token prediction; audio processing models achieve advanced speech recognition and speech-to-speech translation by combining text and speech; image processing models generate images by predicting the next pixel. These models have achieved remarkable results in their respective fields. However, this just ignores the native binary data that is ubiquitous in the digital world. Bytes are the basis of all digital data, devices, and software, from computer processors to operating systems in everyday electronic products, all of which are dependent on bytes for processing. However, the model prediction process in the existing technology ignores the most essential elements of the data, resulting in the model prediction capability and speed not being more fully developed. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention
[0003] In view of this, an embodiment of this specification provides a data processing method. One or more embodiments of this specification also relate to a data processing device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.
[0004] According to a first aspect of an embodiment of this specification, a data processing method is provided, including:
[0005] Acquire multimedia data of a target prediction task, and convert the multimedia data into a binary byte sequence;
[0006] Inputting the binary byte sequence into a multimedia processing model, wherein the multimedia processing model comprises a linear projection unit, a sequence decoding unit and a byte decoding unit;
[0007] The linear projection unit is used to convert the binary byte sequence into a byte sequence vector, and the sequence decoding unit is used to process the byte sequence vector to obtain a sequence feature;
[0008] The sequence features are processed by the byte decoding unit to obtain predicted bytes, and target task data corresponding to the target prediction task is constructed based on the predicted bytes.
[0009] Optionally, converting the multimedia data into a binary byte sequence comprises:
[0010] Determining code point information corresponding to the multimedia data according to a conversion strategy associated with the target prediction task;
[0011] The multimedia data is processed according to the code point information and the encoding rules configured in the conversion strategy to obtain a binary byte sequence corresponding to the multimedia data.
[0012] Optionally, converting the binary byte sequence into a byte sequence vector by the linear projection unit includes:
[0013] Inputting the binary byte sequence into the linear projection unit, and converting the binary bytes contained in the binary byte sequence into byte vectors through the linear projection unit;
[0014] Constructing an initial byte sequence vector corresponding to the binary byte sequence according to the byte vector corresponding to each binary byte;
[0015] The initial byte sequence vector is converted into a one-dimensional vector, and the one-dimensional vector is mapped to a binary byte space to obtain a byte sequence vector.
[0016] Optionally, the using the sequence decoding unit to process the byte sequence vector to obtain a sequence feature includes:
[0017] Inputting the byte sequence vector into the sequence decoding unit, and performing autoregressive prediction processing on the byte sequence vector by the sequence decoding unit to obtain sequence features;
[0018] The step of processing the sequence features by using the byte decoding unit to obtain predicted bytes includes:
[0019] The sequence features are input into the byte decoding unit, and the byte decoding unit performs autoregressive reconstruction processing on the sequence features to obtain predicted bytes.
[0020] Optionally, constructing target task data corresponding to the target prediction task based on the predicted bytes includes:
[0021] Determining an inverse conversion strategy corresponding to the conversion strategy, and parsing the predicted bytes according to the inverse conversion strategy;
[0022] The predicted code point information corresponding to the predicted byte is determined according to the parsing result, and the target task data corresponding to the target prediction task is generated according to the predicted code point information.
[0023] Optionally, before the step of inputting the binary byte sequence into the multimedia processing model is executed, the step further includes:
[0024] Obtaining sample data and sample labels corresponding to the sample data, converting the sample data into a sample binary byte sequence, and converting the sample label into a label byte;
[0025] Inputting the sample binary byte sequence into an initial multimedia processing model for processing to obtain training prediction bytes;
[0026] The initial multimedia processing model is trained based on the training prediction bytes and the label bytes until the multimedia processing model that meets the training stop condition is obtained.
[0027] According to a second aspect of an embodiment of this specification, there is provided a data processing device, including:
[0028] An acquisition module, configured to acquire multimedia data of a target prediction task and convert the multimedia data into a binary byte sequence;
[0029] an input module configured to input the binary byte sequence into a multimedia processing model, wherein the multimedia processing model comprises a linear projection unit, a sequence decoding unit and a byte decoding unit;
[0030] a conversion module, configured to convert the binary byte sequence into a byte sequence vector by the linear projection unit, and process the byte sequence vector by the sequence decoding unit to obtain a sequence feature;
[0031] The construction module is configured to use the byte decoding unit to process the sequence features to obtain predicted bytes, and construct the target task data corresponding to the target prediction task based on the predicted bytes.
[0032] According to a third aspect of an embodiment of this specification, a computing device is provided, including:
[0033] Memory and processor;
[0034] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above data processing method are implemented.
[0035] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned data processing method are implemented.
[0036] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which implement the steps of the above-mentioned data processing method when executed by a processor.
[0037] The data processing method provided in this embodiment, in order to improve the model's comprehensive understanding and simulation of digital systems, thereby improving the processing capability and efficiency of multimedia data, can convert the multimedia data into a binary byte sequence after acquiring the multimedia data of the target prediction task; at this time, the binary byte sequence can be input into the multimedia processing model, and the multimedia processing model includes a linear projection unit, a sequence decoding unit and a byte decoding unit; on this basis, the binary byte sequence can be converted into a byte sequence vector by the linear projection unit, and then the byte sequence vector can be processed by the sequence decoding unit to obtain sequence features; and the structural relationship can be mined in the byte sequence dimension. The byte decoding unit is then used to process the sequence features to obtain predicted bytes, and the byte relationship can be mined in the byte dimension. Finally, the target task data corresponding to the target prediction task can be constructed based on the predicted bytes, and the binary byte dimension can be processed by the model, which can simplify the integration process between different data types and improve the applicability and performance of the model in cross-modal tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 is a schematic diagram of a data processing method provided by an embodiment of this specification;
[0039] Figure 2 is a flow chart of a data processing method provided by an embodiment of this specification;
[0040] Figure 3 is a processing flow chart of a data processing method provided by an embodiment of this specification;
[0041] Figure 4 is a structural schematic diagram of a data processing device provided by an embodiment of this specification;
[0042] Figure 5 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION
[0043] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.
[0044] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0045] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0046] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0047] In this specification, a data processing method is provided. This specification also relates to a data processing device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.
[0048] In practical applications, deep learning models mainly focus on processing digital media data that is easy for humans to understand, such as text, audio, and images. Although these models have achieved remarkable results in their respective fields, they have obvious deficiencies in processing native binary data. In particular: Text processing-language models perform well in text generation and understanding through token prediction, but they rely on the tokenization process of text and cannot directly process binary data. Audio processing-although audio processing models can process audio data, they are also limited to specific data types and cannot effectively process and simulate binary data. Image processing-although image processing models can predict the next pixel in an image, they perform poorly when processing long byte sequences and complex binary data. Therefore, an effective solution is urgently needed to solve the above problems.
[0049] See also Figure 1 As shown in the schematic diagram, the data processing method provided by this embodiment, in order to improve the model's comprehensive understanding and simulation of digital systems, thereby improving the processing capability and efficiency of multimedia data, can convert the multimedia data into a binary byte sequence after acquiring the multimedia data of the target prediction task; at this time, the binary byte sequence can be input into the multimedia processing model, and the multimedia processing model includes a linear projection unit, a sequence decoding unit and a byte decoding unit; on this basis, the binary byte sequence can be converted into a byte sequence vector by the linear projection unit, and then the byte sequence vector is processed by the sequence decoding unit to obtain sequence features; and the structural relationship is mined in the byte sequence dimension. Then the sequence features are processed by the byte decoding unit to obtain predicted bytes, and the byte relationship is mined in the byte dimension. Finally, the target task data corresponding to the target prediction task can be constructed based on the predicted bytes, and the binary byte dimension can be processed by the model, which can simplify the integration process between different data types and improve the applicability and performance of the model in cross-modal tasks.
[0050] See also Figure 2 , Figure 2 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.
[0051] Step S202, obtaining multimedia data of a target prediction task, and converting the multimedia data into a binary byte sequence.
[0052] The data processing method provided in this embodiment can be applied to model applications in any scenario, such as text processing scenarios (abstract extraction, text classification, translation), image processing scenarios (image cropping, image generation, graphic generation), audio processing scenarios (noise elimination, audio generation, audio conversion), etc. This embodiment does not make any limitation here. This embodiment uses the application of the data processing method in a text processing scenario as an example to illustrate the data processing method. For descriptions of other scenarios, please refer to the same or corresponding descriptions in this embodiment, and this embodiment does not make any limitation here. In addition, the data processing method provided in this embodiment can also be applied to cross-modal scenarios.
[0053] Specifically, the target prediction task refers to the task triggered in the multimedia data processing scenario. The task can be a text processing task in the text processing scenario, such as abstract extraction, text classification, translation, etc.; it can also be an image processing task in the image processing scenario, such as image cropping, image generation, image and text generation, etc.; it can also be an audio processing task in the audio processing scenario, such as noise elimination, audio generation, audio conversion, etc. Correspondingly, multimedia data is the data that needs to be input into the model for processing when the target prediction task is triggered, such as text, image, video, audio, etc. Correspondingly, the binary byte sequence specifically refers to the sequence of binary bytes obtained after byte encoding of multimedia data, which is the most basic unit of computer processing. By converting multimedia data into binary bytes for model processing, the prediction efficiency can be effectively improved. At the same time, the model can be developed most effectively, and the model can be used to analyze multimedia data from the most basic unit.
[0054] Based on this, in order to improve the model's comprehensive understanding and simulation of digital systems, thereby improving the processing capabilities and efficiency of multimedia data, after obtaining the multimedia data of the target prediction task, the multimedia data can be first converted into a binary byte sequence; at this time, the binary byte sequence can be input into the multimedia processing model, and the multimedia processing model includes a linear projection unit, a sequence decoding unit, and a byte decoding unit; on this basis, the binary byte sequence can be converted into a byte sequence vector through the linear projection unit, and then the byte sequence vector is processed by the sequence decoding unit to obtain sequence features; and the structural relationship is mined in the byte sequence dimension. Then the byte decoding unit is used to process the sequence features to obtain predicted bytes, and the byte relationship is mined in the byte dimension. Finally, the target task data corresponding to the target prediction task can be constructed based on the predicted bytes.
[0055] Furthermore, when constructing a binary byte sequence of multimedia data, in order to achieve cross-modal and more accurate model prediction, it can be completed according to a preset conversion strategy. In this embodiment, the specific implementation method is as follows:
[0056] Determine the code point information corresponding to the multimedia data according to the conversion strategy associated with the target prediction task; process the multimedia data according to the code point information and the encoding rules configured in the conversion strategy to obtain a binary byte sequence corresponding to the multimedia data.
[0057] Specifically, the conversion strategy refers to a strategy for converting multimedia data of an associated target prediction task into a binary byte sequence according to a specified encoding method. The strategy can determine the code point information of each character in the multimedia data, as well as the encoding rules, wherein the encoding rules are used to encode the multimedia data to obtain binary bytes to form a binary byte sequence.
[0058] Based on this, after obtaining the multimedia data of the target prediction task, the code point information corresponding to the multimedia data can be determined according to the conversion strategy associated with the target prediction task; on this basis, the multimedia data can be processed according to the code point information and the encoding rules configured in the conversion strategy to realize the conversion of the multimedia data into multiple binary bytes, thereby forming a binary byte sequence corresponding to the multimedia data, so as to facilitate subsequent input model prediction processing.
[0059] For example, a user submits the question text "Who is the author of ancient poem A" through the client. After receiving the question text, the corresponding code point of each character unit in the Unicode encoding can be determined first. On this basis, UTF-8 (8-bit Unicode Transformation Format, Unicode) can be used to encode each character unit. In UTF-8 encoding, the code point corresponding to each character unit will be converted into a sequence of bytes of a set length. By calculating the byte sequence of UTF-8 encoding and performing byte allocation, the hexadecimal byte sequence {A\xe5\x8f\xa4\xe8\xaf\x97\xe7\x9a\x84\xe4\xbd\x9c\xe8\80\x85\xe6\x98\xaf\xe8\xb0\x81} corresponding to the question text "Who is the author of ancient poem A" can be obtained. On this basis, it is converted into a binary byte sequence, which can be used for subsequent model processing.
[0060] In summary, by adopting a conversion strategy to convert multimedia data, it is possible to convert multimedia data into a binary byte sequence in a unified format according to set specifications, so that subsequent models can make accurate predictions.
[0061] Step S204, inputting the binary byte sequence into a multimedia processing model, wherein the multimedia processing model includes a linear projection unit, a sequence decoding unit and a byte decoding unit.
[0062] Step S206: convert the binary byte sequence into a byte sequence vector by the linear projection unit, and process the byte sequence vector by the sequence decoding unit to obtain sequence features.
[0063] Step S208: Use the byte decoding unit to process the sequence features to obtain predicted bytes, and construct target task data corresponding to the target prediction task based on the predicted bytes.
[0064] Specifically, after obtaining the binary byte sequence corresponding to the multimedia data, the binary byte sequence can be input into a multimedia processing model including a linear projection unit, a sequence decoding unit, and a byte decoding unit, so that the multimedia processing model can output predicted bytes by performing binary byte-level prediction to construct target task data corresponding to the target prediction task for subsequent use.
[0065] Among them, the linear projection unit specifically refers to a unit for mapping binary bytes contained in a binary byte sequence into vector expressions. Accordingly, the sequence decoding unit specifically refers to a decoder for constructing sequence dimension features, and the byte decoding unit specifically refers to a decoder for constructing byte dimension features, which is used to output predicted bytes. Accordingly, the byte sequence vector specifically refers to a sequence vector obtained after vector mapping of a binary byte sequence. The sequence feature specifically refers to the sequence-level feature extracted from the byte sequence vector, and the predicted byte specifically refers to the model output result obtained after processing the multimedia data corresponding to the target prediction task, and the result is a binary byte, which is used to construct the target task data. Accordingly, the target task data specifically refers to the result obtained after the model is processed on the multimedia data. For example, if the target prediction task is a text generation task, the target task data is the text generation result. For another example, if the target prediction task is an audio noise removal task, the target task data is the audio with noise removed.
[0066] That is to say, when the multimedia processing model processes a binary byte sequence, in order to be able to mine the byte relationship between the binary byte levels, the binary byte sequence can be first converted into a byte sequence vector through a linear projection unit. After obtaining the vector expression corresponding to the binary byte sequence, subsequent prediction processing can be performed. In order to improve the model prediction ability and realize sufficient data relationship analysis of binary bytes, the byte sequence vector can be processed by a sequence decoding unit first to obtain the sequence features of the corresponding sequence dimension; the sequence features can then be processed by a byte decoding unit to complete the prediction processing, thereby obtaining predicted bytes; thereafter, the target task data corresponding to the target prediction task is constructed based on the predicted bytes, which can be convenient for downstream use. For example, the generated text is fed back to the user, or user questions are answered based on the generated text. This embodiment is not limited in any way.
[0067] In practical applications, multimedia processing models can use a hierarchical Transformer architecture to split byte sequences into segments to improve computational efficiency. The model can include three main components, namely, a linear projection unit, a sequence decoding unit, and a byte decoding unit, to complete byte-level predictions.
[0068] Furthermore, after the binary byte sequence is input into the multimedia processing model, in order to achieve prediction in the binary byte dimension, it is necessary to first perform a byte-to-vector conversion. In this embodiment, the specific implementation method is as follows:
[0069] The binary byte sequence is input into the linear projection unit, and the binary bytes contained in the binary byte sequence are converted into byte vectors through the linear projection unit; an initial byte sequence vector corresponding to the binary byte sequence is constructed according to the byte vector corresponding to each binary byte; the initial byte sequence vector is converted into a one-dimensional vector, and the one-dimensional vector is mapped to the binary byte space to obtain a byte sequence vector.
[0070] Specifically, the byte vector refers to the vector expression obtained by converting the binary bytes contained in the binary byte into a vector. Correspondingly, the initial byte sequence vector refers to the vector expression obtained by concatenating the byte vectors corresponding to each binary byte. The binary byte space refers to mapping a one-dimensional vector to a byte space, so that the byte sequence vector mapped out by the byte space reflects the intrinsic correlation of the multimedia data, thereby facilitating the model to make predictions.
[0071] Based on this, after the binary byte sequence is input into the multimedia processing model, the sequence can be first vector-mapped through a linear projection unit. At this time, the binary bytes contained in the binary byte sequence can be converted into byte vectors through the linear projection unit; on this basis, the initial byte sequence vector corresponding to the binary byte sequence can be constructed according to the byte vector corresponding to each binary byte; then the initial byte sequence vector can be converted into a one-dimensional vector, and then the one-dimensional vector can be mapped to the binary byte space to obtain the byte sequence vector corresponding to the multimedia resource for subsequent prediction and use.
[0072] In practical applications, when the linear projection unit maps binary byte sequences, each binary byte sequence can be regarded as a matrix, and each byte can be represented as a 257-dimensional vector through one-hot encoding, including all 256 byte values and a <eop>(end of sequence) marker, then convert the sequence into a one-dimensional vector and map it to the binary byte space to get a byte sequence vector.
[0073] In summary, by embedding a linear projection unit in the model to map the binary byte sequence to the binary byte space, it is possible to obtain a byte sequence vector while still being able to characterize the data relationship in the multimedia data, thereby enabling the model to make accurate predictions.
[0074] Furthermore, after obtaining the byte sequence vector, the byte prediction can be completed by combining the sequence decoding unit and the byte decoding unit in the model. In this embodiment, the specific implementation method is as follows:
[0075] The byte sequence vector is input into the sequence decoding unit, and the byte sequence vector is subjected to autoregressive prediction processing by the sequence decoding unit to obtain sequence features; the sequence features are input into the byte decoding unit, and the sequence features are subjected to autoregressive reconstruction processing by the byte decoding unit to obtain predicted bytes.
[0076] Based on this, after obtaining the byte sequence vector mapped by the linear projection unit, the byte sequence vector can be input into the sequence decoding unit, and the byte sequence vector is processed by autoregression prediction through the sequence decoding unit to achieve structural prediction at the sequence level, thereby obtaining sequence features; thereafter, the sequence features are input into the byte decoding unit, and the sequence features are processed by autoregression reconstruction through the byte decoding unit to achieve binary byte prediction in the byte dimension, thereby obtaining predicted bytes, so that the predicted bytes can be subsequently converted into target task data for downstream task processing.
[0077] In practical applications, the sequence decoding unit can autoregressively predict the features of subsequent sequences and learn the structure of the data, while the byte decoding unit can autoregressively reconstruct the byte sequence in the fragment according to the fragment features, thereby obtaining predicted characters for subsequent use.
[0078] In summary, by using a sequence decoder to predict features from the sequence dimension, and then predicting byte generation from the byte dimension based on the prediction results, it is possible to ensure that the model outputs more accurate prediction results from the binary dimension.
[0079] After the predicted bytes are obtained, considering that the final result needs to be fed back to the user or the service end for use, the predicted bytes need to be converted into target task data. In this embodiment, the specific implementation method is as follows:
[0080] Determine the inverse conversion strategy corresponding to the conversion strategy, and parse the predicted bytes according to the inverse conversion strategy; determine the predicted code point information corresponding to the predicted bytes according to the parsing result, and generate the target task data corresponding to the target prediction task according to the predicted code point information.
[0081] Specifically, the inverse conversion strategy specifically refers to the strategy of converting the predicted bytes into target task data, that is, the predicted bytes can be converted into text, images, or audio through the inverse conversion strategy. Correspondingly, the predicted code point information specifically refers to the code point information corresponding to the predicted bytes.
[0082] Based on this, after the model outputs the predicted bytes, the inverse conversion strategy corresponding to the conversion strategy can be determined first. On this basis, the predicted bytes can be parsed according to the inverse conversion strategy; the predicted code point information corresponding to the predicted bytes can be determined according to the parsing result, and then the target task data corresponding to the target prediction task can be generated according to the predicted code point information for downstream use.
[0083] Continuing with the above example, after obtaining the binary byte sequence corresponding to the question text "Who is the author of ancient poem A", the binary byte sequence can be input into the multimedia processing model; first, the binary byte sequence is mapped through the linear projection unit in the multimedia processing model to obtain the byte sequence vector ZLc; secondly, the byte sequence vector ZLc can be input into the sequence decoder in the multimedia processing model, and the sequence features are predicted autoregressively through the sequence decoder to obtain the sequence vector Lc. Finally, the sequence vector Lc is input into the byte decoder in the multimedia processing model, and the byte sequence in the sequence is reconstructed autoregressively through the byte decoder, so that the predicted bytes can be generated. After obtaining the predicted bytes, the predicted bytes are converted according to the inverse process of the above encoding, and the answer text "A" corresponding to the question text "Who is the author of ancient poem A" can be obtained and fed back to the user.
[0084] In summary, by processing multimedia data through a multimedia processing model including a linear projection unit, a sequence decoding unit, and a byte decoding unit, the model can give a prediction result from the byte dimension, thereby effectively improving the model prediction accuracy and processing efficiency. At the same time, less computing resources can be occupied.
[0085] In addition, in order to enable the multimedia processing model to complete predictions from the byte dimension, the model needs to have this ability during the training phase. In this embodiment, the training of the model includes:
[0086] Obtain sample data and sample labels corresponding to the sample data, convert the sample data into sample binary byte sequences, and convert the sample labels into label bytes; input the sample binary byte sequences into an initial multimedia processing model for processing to obtain training prediction bytes; train the initial multimedia processing model based on the training prediction bytes and the label bytes until the multimedia processing model that meets the training stop condition is obtained.
[0087] Specifically, the sample data and sample labels specifically refer to the samples for training the multimedia processing model. By converting them into binary sequences, the model can learn the prediction ability of binary bytes, thereby improving the model's mining ability for byte-level data. Correspondingly, the label byte is the binary byte corresponding to the sample label. The training prediction byte specifically refers to the prediction byte output by the model. Among them, the training stop condition specifically refers to the condition for stopping the training of the multimedia processing model, including but not limited to the loss value comparison condition, the verification condition or the number of iterations condition, which is not limited in this embodiment.
[0088] Based on this, in the model training stage, we can first obtain sample data and sample labels corresponding to the sample data, and then convert the sample data into sample binary byte sequences, and convert the sample labels into label bytes; on this basis, we can train the model, and by inputting the sample binary byte sequence into the initial multimedia processing model for processing, we can obtain the training prediction bytes; on this basis, we can train the initial multimedia processing model in combination with the training prediction bytes and label bytes until we obtain a multimedia processing model that meets the training stop conditions.
[0089] The data processing method provided in this embodiment, in order to improve the model's comprehensive understanding and simulation of digital systems, thereby improving the processing capability and efficiency of multimedia data, can convert the multimedia data into a binary byte sequence after acquiring the multimedia data of the target prediction task; at this time, the binary byte sequence can be input into the multimedia processing model, and the multimedia processing model includes a linear projection unit, a sequence decoding unit and a byte decoding unit; on this basis, the binary byte sequence can be converted into a byte sequence vector by the linear projection unit, and then the byte sequence vector can be processed by the sequence decoding unit to obtain sequence features; and the structural relationship can be mined in the byte sequence dimension. The byte decoding unit is then used to process the sequence features to obtain predicted bytes, and the byte relationship can be mined in the byte dimension. Finally, the target task data corresponding to the target prediction task can be constructed based on the predicted bytes, and the binary byte dimension can be processed by the model, which can simplify the integration process between different data types and improve the applicability and performance of the model in cross-modal tasks.
[0090] The following combination Figure 3 , taking the application of the data processing method provided in this specification in the image data processing scenario as an example, the data processing method is further described. Figure 3 A processing flow chart of a data processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.
[0091] Step S302: Acquire multimedia data of the target prediction task, and determine code point information corresponding to the multimedia data according to a conversion strategy associated with the target prediction task.
[0092] Step S304: Process the multimedia data according to the code point information and the encoding rules configured in the conversion strategy to obtain a binary byte sequence corresponding to the multimedia data.
[0093] Step S306, inputting the binary byte sequence into a multimedia processing model, wherein the multimedia processing model includes a linear projection unit, a sequence decoding unit and a byte decoding unit.
[0094] Step S308: convert the binary bytes contained in the binary byte sequence into byte vectors through a linear projection unit.
[0095] Step S310, constructing an initial byte sequence vector corresponding to the binary byte sequence according to the byte vector corresponding to each binary byte.
[0096] Step S312, converting the initial byte sequence vector into a one-dimensional vector, and mapping the one-dimensional vector to a binary byte space to obtain a byte sequence vector.
[0097] Step S314, performing autoregressive prediction processing on the byte sequence vector through the sequence decoding unit to obtain sequence features.
[0098] Step S316, inputting the sequence features into the byte decoding unit, and performing autoregressive reconstruction processing on the sequence features through the byte decoding unit to obtain predicted bytes.
[0099] Step S318, determining the inverse conversion strategy corresponding to the conversion strategy, and parsing the predicted bytes according to the inverse conversion strategy.
[0100] Step S320, determining the predicted code point information corresponding to the predicted byte according to the parsing result, and generating target task data corresponding to the target prediction task according to the predicted code point information.
[0101] The data processing method provided in this embodiment, in order to improve the model's comprehensive understanding and simulation of digital systems, thereby improving the processing capability and efficiency of multimedia data, can convert the multimedia data into a binary byte sequence after acquiring the multimedia data of the target prediction task; at this time, the binary byte sequence can be input into the multimedia processing model, and the multimedia processing model includes a linear projection unit, a sequence decoding unit and a byte decoding unit; on this basis, the binary byte sequence can be converted into a byte sequence vector by the linear projection unit, and then the byte sequence vector can be processed by the sequence decoding unit to obtain sequence features; and the structural relationship can be mined in the byte sequence dimension. The byte decoding unit is then used to process the sequence features to obtain predicted bytes, and the byte relationship can be mined in the byte dimension. Finally, the target task data corresponding to the target prediction task can be constructed based on the predicted bytes, and the binary byte dimension can be processed by the model, which can simplify the integration process between different data types and improve the applicability and performance of the model in cross-modal tasks.
[0102] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 4 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 4 As shown, the device comprises:
[0103] An acquisition module 402 is configured to acquire multimedia data of a target prediction task and convert the multimedia data into a binary byte sequence;
[0104] An input module 404 is configured to input the binary byte sequence into a multimedia processing model, wherein the multimedia processing model comprises a linear projection unit, a sequence decoding unit and a byte decoding unit;
[0105] A conversion module 406 is configured to convert the binary byte sequence into a byte sequence vector by the linear projection unit, and process the byte sequence vector by the sequence decoding unit to obtain a sequence feature;
[0106] The construction module 408 is configured to use the byte decoding unit to process the sequence features to obtain predicted bytes, and construct the target task data corresponding to the target prediction task based on the predicted bytes.
[0107] In an optional embodiment, the acquisition module 402 is further configured to:
[0108] Determine the code point information corresponding to the multimedia data according to the conversion strategy associated with the target prediction task; process the multimedia data according to the code point information and the encoding rules configured in the conversion strategy to obtain a binary byte sequence corresponding to the multimedia data.
[0109] In an optional embodiment, the conversion module 406 is further configured to:
[0110] The binary byte sequence is input into the linear projection unit, and the binary bytes contained in the binary byte sequence are converted into byte vectors through the linear projection unit; an initial byte sequence vector corresponding to the binary byte sequence is constructed according to the byte vector corresponding to each binary byte; the initial byte sequence vector is converted into a one-dimensional vector, and the one-dimensional vector is mapped to the binary byte space to obtain a byte sequence vector.
[0111] In an optional embodiment, the conversion module 406 is further configured to:
[0112] Inputting the byte sequence vector into the sequence decoding unit, and performing autoregressive prediction processing on the byte sequence vector by the sequence decoding unit to obtain sequence features;
[0113] The construction module 408 is further configured as follows:
[0114] The sequence features are input into the byte decoding unit, and the byte decoding unit performs autoregressive reconstruction processing on the sequence features to obtain predicted bytes.
[0115] In an optional embodiment, the construction module 408 is further configured to:
[0116] Determine the inverse conversion strategy corresponding to the conversion strategy, and parse the predicted bytes according to the inverse conversion strategy; determine the predicted code point information corresponding to the predicted bytes according to the parsing results, and generate target task data corresponding to the target prediction task according to the predicted code point information.
[0117] In an optional embodiment, the device further comprises:
[0118] The training module is configured to obtain sample data and sample labels corresponding to the sample data, convert the sample data into sample binary byte sequences, and convert the sample labels into label bytes; input the sample binary byte sequences into an initial multimedia processing model for processing to obtain training prediction bytes; and train the initial multimedia processing model based on the training prediction bytes and the label bytes until the multimedia processing model that meets the training stop condition is obtained.
[0119] The data processing device provided in this embodiment can, in order to improve the model's comprehensive understanding and simulation of digital systems, thereby improving the processing capability and efficiency of multimedia data, first convert the multimedia data into a binary byte sequence after acquiring the multimedia data of the target prediction task; at this time, the binary byte sequence can be input into the multimedia processing model, and the multimedia processing model includes a linear projection unit, a sequence decoding unit and a byte decoding unit; on this basis, the binary byte sequence can be converted into a byte sequence vector by the linear projection unit, and then the byte sequence vector can be processed by the sequence decoding unit to obtain sequence features; and the structural relationship can be mined in the byte sequence dimension. The byte decoding unit is then used to process the sequence features to obtain predicted bytes, and the byte relationship can be mined in the byte dimension. Finally, the target task data corresponding to the target prediction task can be constructed based on the predicted bytes, and the binary byte dimension can be processed by the model, which can simplify the integration process between different data types and improve the applicability and performance of the model in cross-modal tasks.
[0120] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.
[0121] Figure 5 The structure block diagram of a computing device 500 provided according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.
[0122] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).
[0123] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0124] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 500 may also be a mobile or stationary server.
[0125] The processor 520 is used to execute the following computer executable instructions, which implement the steps of the above data processing method when executed by the processor.
[0126] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.
[0127] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned data processing method when executed by a processor.
[0128] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.
[0129] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which implement the steps of the above data processing method when executed by a processor.
[0130] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above data processing method.
[0131] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0132] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0133] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.
[0134] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0135] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well.< / eop>
Claims
1. A data processing method, characterized in that: include: Acquire multimedia data of a target prediction task, and convert the multimedia data into a binary byte sequence; Inputting the binary byte sequence into a multimedia processing model, wherein the multimedia processing model comprises a linear projection unit, a sequence decoding unit and a byte decoding unit; The linear projection unit is used to convert the binary byte sequence into a byte sequence vector, and the sequence decoding unit is used to process the byte sequence vector to obtain a sequence feature; The sequence features are processed by the byte decoding unit to obtain predicted bytes, and target task data corresponding to the target prediction task is constructed based on the predicted bytes.
2. The data processing method according to claim 1, characterized in that: The converting the multimedia data into a binary byte sequence comprises: Determining code point information corresponding to the multimedia data according to a conversion strategy associated with the target prediction task; The multimedia data is processed according to the code point information and the encoding rules configured in the conversion strategy to obtain a binary byte sequence corresponding to the multimedia data.
3. The data processing method according to claim 1, characterized in that: The converting the binary byte sequence into a byte sequence vector by the linear projection unit includes: Inputting the binary byte sequence into the linear projection unit, and converting the binary bytes contained in the binary byte sequence into byte vectors through the linear projection unit; Constructing an initial byte sequence vector corresponding to the binary byte sequence according to the byte vector corresponding to each binary byte; The initial byte sequence vector is converted into a one-dimensional vector, and the one-dimensional vector is mapped to a binary byte space to obtain a byte sequence vector.
4. The data processing method according to claim 1, characterized in that: The using the sequence decoding unit to process the byte sequence vector to obtain sequence features includes: Inputting the byte sequence vector into the sequence decoding unit, and performing autoregressive prediction processing on the byte sequence vector by the sequence decoding unit to obtain sequence features; The step of processing the sequence features by using the byte decoding unit to obtain predicted bytes includes: The sequence features are input into the byte decoding unit, and the byte decoding unit performs autoregressive reconstruction processing on the sequence features to obtain predicted bytes.
5. The data processing method according to claim 2, characterized in that: The constructing target task data corresponding to the target prediction task based on the predicted bytes includes: Determining an inverse conversion strategy corresponding to the conversion strategy, and parsing the predicted bytes according to the inverse conversion strategy; The predicted code point information corresponding to the predicted byte is determined according to the parsing result, and the target task data corresponding to the target prediction task is generated according to the predicted code point information.
6. The data processing method according to any one of claims 1 to 5, characterized in that: Before the step of inputting the binary byte sequence into the multimedia processing model is executed, the step further includes: Obtaining sample data and sample labels corresponding to the sample data, converting the sample data into a sample binary byte sequence, and converting the sample label into a label byte; Inputting the sample binary byte sequence into an initial multimedia processing model for processing to obtain training prediction bytes; The initial multimedia processing model is trained based on the training prediction bytes and the label bytes until the multimedia processing model that meets the training stop condition is obtained.
7. A data processing device, characterized in that: include: An acquisition module, configured to acquire multimedia data of a target prediction task and convert the multimedia data into a binary byte sequence; an input module configured to input the binary byte sequence into a multimedia processing model, wherein the multimedia processing model comprises a linear projection unit, a sequence decoding unit and a byte decoding unit; a conversion module, configured to convert the binary byte sequence into a byte sequence vector by the linear projection unit, and process the byte sequence vector by the sequence decoding unit to obtain a sequence feature; The construction module is configured to use the byte decoding unit to process the sequence features to obtain predicted bytes, and construct the target task data corresponding to the target prediction task based on the predicted bytes.
8. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 6.
10. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.
Citation Information
Cited By
Data processing method and apparatus
WO2026045026A1
