Multimedia data processing method, device, equipment and medium

Abstract

The present disclosure provides a multimedia data processing method, device, electronic device, storage medium and program product, which relates to the field of artificial intelligence technology, especially to the technical fields of large models, large language models, Transformers (codecs), conversational models, generative models, etc. The specific implementation scheme is: based on the available storage capacity of the storage unit and the data volume of the data block sequence, determine the eviction strategy; repeatedly perform the following operations until n is equal to N: when it is determined that n is less than N, based on the eviction strategy, evict the n-1th data block feature to obtain the n-1th post-eviction feature; based on the n-1th post-eviction feature and the nth data block, obtain the nth data block feature; store the nth data block feature in the storage unit; when it is determined that n is equal to N, obtain the target processing result based on the nth data block feature.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to large models, large language models, Transformers (codecs), conversational models, generative models, etc., and specifically to multimedia data processing methods, devices, electronic devices, storage media, and program products. Background Art

[0002] Data has penetrated into every technical field and has become an important production factor. In the process of processing data, data storage is involved. For example, in the process of streaming data, a lot of intermediate data is stored in the cache for further processing. Another example is to store the target data in the database for subsequent query. The hardware performance of storage units such as cache or database is an important dimension to measure whether the data processing system is stable and efficient. Summary of the invention

[0003] The present disclosure provides a multimedia data processing method, device, electronic device, storage medium and program product.

[0004] According to one aspect of the present disclosure, there is provided a multimedia data processing method, comprising:

[0005] Determining an eviction strategy based on available storage capacity of the storage unit and the data amount of the data block sequence, wherein the data block sequence is obtained by processing multimedia elements of the multimedia data, the eviction strategy is used to characterize the data amount of the data to be evicted, and the data block sequence includes N data blocks, where N is an integer greater than or equal to 2;

[0006] Repeat the following steps until n equals N:

[0007] When it is determined that n is less than N, based on the eviction strategy, the feature of the n-1th data block is evicted to obtain the feature after the n-1th eviction, wherein the feature of the n-1th data block is obtained from the storage unit, and n is an integer greater than or equal to 2 and less than or equal to N;

[0008] Based on the above n-1th post-eviction feature and the nth data block, obtain the nth data block feature; and

[0009] Storing the nth data block feature in the storage unit; and

[0010] When it is determined that n is equal to N, a target processing result is obtained based on the characteristics of the nth data block.

[0011] According to another aspect of the present disclosure, there is provided a multimedia data processing device, comprising:

[0012] a strategy determination module, configured to determine an eviction strategy based on an available storage capacity of a storage unit and a data volume of a data block sequence, wherein the data block sequence is obtained by processing multimedia elements of the multimedia data, the eviction strategy is used to characterize the data volume of the data to be evicted, and the data block sequence includes N data blocks, where N is an integer greater than or equal to 2;

[0013] The repeating operation module is used to repeatedly perform the following operations until n is equal to N:

[0014] an eviction submodule, configured to, when it is determined that n is less than N, evict the n-1th data block feature based on the eviction strategy to obtain the n-1th post-eviction feature, wherein the n-1th data block feature is obtained from the storage unit, and n is an integer greater than or equal to 2 and less than or equal to N;

[0015] a processing submodule, configured to obtain an nth data block feature based on the n-1th post-eviction feature and the nth data block; and

[0016] A storage submodule, used for storing the nth data block feature in the storage unit; and

[0017] The target determination module is used to obtain a target processing result based on the characteristics of the nth data block when it is determined that n is equal to N.

[0018] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method as disclosed in the present disclosure.

[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method of the present disclosure.

[0020] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method of the present disclosure is implemented.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 An exemplary system architecture to which the multimedia data processing method and device according to an embodiment of the present disclosure can be applied is schematically shown;

[0024] Figure 2 A flowchart of a multimedia data processing method according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 3 A schematic diagram of determining an eviction strategy according to an embodiment of the present disclosure is schematically shown;

[0026] Figure 4 A schematic diagram schematically shows a multimedia data processing method according to an embodiment of the present disclosure;

[0027] Figure 5 A schematic diagram schematically shows expulsion data according to an embodiment of the present disclosure;

[0028] Figure 6 A schematic diagram of determining a mask matrix according to an embodiment of the present disclosure is schematically shown;

[0029] Figure 7 A block diagram schematically shows a multimedia data processing device according to an embodiment of the present disclosure; and

[0030] Figure 8 A block diagram of an electronic device suitable for implementing a multimedia data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] The present disclosure provides a multimedia data processing method, device, electronic device, storage medium and program product.

[0033] According to an embodiment of the present disclosure, a multimedia data processing method is provided, including: determining eviction policies based on available storage capacity of a storage unit and the data volume of a data block sequence, wherein the data block sequence is obtained by processing multimedia elements of the multimedia data, the eviction policies are used to characterize the data volume of the data to be evicted, and the data block sequence includes N data blocks, where N is an integer greater than or equal to 2; repeatedly performing the following operations until n is equal to N: when it is determined that n is less than N, based on the eviction policy, evicting an n-1th data block feature to obtain an n-1th post-eviction feature, wherein the n-1th data block feature is obtained from the storage unit, n is an integer greater than or equal to 2, and n is less than or equal to N; obtaining an nth data block feature based on the n-1th post-eviction feature and the nth data block; and storing the nth data block feature in the storage unit; and when it is determined that n is equal to N, obtaining a target processing result based on the nth data block feature.

[0034] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0035] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0036] Figure 1 An exemplary system architecture to which the multimedia data processing method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0037] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, the exemplary system architecture to which the multimedia data processing method and apparatus can be applied may include a terminal device, but the terminal device may implement the multimedia data processing method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0038] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0039] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0040] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0041] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for the content browsed by the user using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0042] It should be noted that the multimedia data processing method provided in the embodiment of the present disclosure can generally be executed by the terminal device 101, 102, or 103. Accordingly, the multimedia data processing apparatus provided in the embodiment of the present disclosure can also be set in the terminal device 101, 102, or 103.

[0043] Alternatively, the multimedia data processing method provided in the embodiment of the present disclosure may also be generally executed by the server 105. Accordingly, the multimedia data processing device provided in the embodiment of the present disclosure may generally be arranged in the server 105. The multimedia data processing method provided in the embodiment of the present disclosure may also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the multimedia data processing device provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0044] For example, during the process of human-computer interaction between the user and the terminal devices 101, 102, and 103, the terminal devices 101, 102, and 103 can obtain the user's multimedia data including multimedia elements such as voice and video, and then send the multimedia data to the server 105, and the server 105 processes the multimedia elements in the multimedia data to obtain a data block sequence. Based on the available storage capacity of the storage unit and the data volume of the data block sequence, an eviction strategy is determined. When it is determined that n is less than N, based on the eviction strategy, the n-1th data block feature is expelled to obtain the n-1th post-eviction feature. Based on the n-1th post-eviction feature and the nth data block, the nth data block feature is obtained. The nth data block feature is stored in the storage unit. When it is determined that n is equal to N, based on the nth data block feature, a target processing result is obtained. Alternatively, a server or server cluster that can communicate with the terminal devices 101, 102, 103 and / or the server 105 analyzes the multimedia elements in the multimedia data and finally obtains the target processing result. The target processing result is sent to the terminal devices 101, 102, and 103. The target processing results are displayed to the user through the terminal devices 101, 102, and 103 to complete the human-computer interaction.

[0045] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0046] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0047] Figure 2 The flowchart of the multimedia data processing method according to the embodiment of the present disclosure is schematically shown.

[0048] like Figure 2 As shown, the method includes operations S210 to S230.

[0049] In operation S210, an eviction strategy is determined based on an available storage capacity of a storage unit and a data amount of a data block sequence.

[0050] In operation S220 , the data block sequence is processed based on the eviction policy.

[0051] In operation S230, when it is determined that n is equal to N, a target processing result is obtained based on the feature of the nth data block.

[0052] According to the embodiments of the present disclosure, Figure 2The operation S220 shown is to process the data block sequence based on the eviction strategy, which may include operations S221 to S223.

[0053] In operation S221, when it is determined that n is less than N, based on the eviction strategy, the feature of the n-1th data block is evicted to obtain the feature after the n-1th eviction.

[0054] In operation S222, an nth data block feature is obtained based on the (n-1)th post-eviction feature and the nth data block.

[0055] In operation S223, the nth data block feature is stored in a storage unit.

[0056] According to an embodiment of the present disclosure, the data block sequence may include N data blocks, where N is an integer greater than or equal to 2.

[0057] According to an embodiment of the present disclosure, the feature of the (n-1)th data block is obtained from a storage unit.

[0058] According to an embodiment of the present disclosure, n is an integer greater than or equal to 2 and less than or equal to N.

[0059] According to the embodiments of the present disclosure, the type of storage unit is not limited, as long as it can be used to store data in the data processing process. For example, the storage unit may include video memory (Cache, also known as graphics card memory), memory (Memory, also known as internal storage), database (Data-Base), etc.

[0060] According to an embodiment of the present disclosure, the data block sequence may be obtained by processing multimedia elements of multimedia data. Multimedia data may include media data for communication or dissemination that combines two or more multimedia elements. A multimedia element may refer to a type of media data that constitutes multimedia data. For example, text, video, image, audio, etc. are all multimedia elements.

[0061] According to an embodiment of the present disclosure, in the process of processing a data block sequence, the current data, such as the nth data block, will be combined with the above content, such as the features of the n-1th data block that has been processed. The calculated output result, such as the features of the n-1th data block, can be stored in a storage unit. In the case where subsequent processing is required, it can be directly obtained from the storage unit, for example, the features of the n-1th data block are obtained from the storage unit, and an operation based on the features after the n-1th expulsion and the nth data block is performed to obtain the features of the nth data block, thereby avoiding repeated calculations.

[0062] According to an embodiment of the present disclosure, when the amount of data in a data block sequence is too large, a large amount of intermediate data is stored in a storage unit, which will occupy the resources of the storage unit, thereby causing pressure on the storage unit, reducing hardware performance and excessive deployment costs.

[0063] According to an embodiment of the present disclosure, an eviction strategy can be determined based on the available storage capacity of a storage unit and the data volume of a data block sequence. The eviction strategy is used to characterize the data volume of the data to be evicted. The eviction strategy is used to evict the processed intermediate data, such as the n-1th data block feature, to reduce the data volume, thereby reducing the data processing volume and avoiding redundant occupation or waste of hardware resources such as storage units without affecting the accuracy of the target processing result.

[0064] According to the embodiments of the present disclosure, the multimedia data processing method can be applied to long text tasks, such as tasks of generating summaries based on long texts or tasks of generating answers based on long text questions. Based on multimedia elements in multimedia data, such as long texts, a data block sequence is obtained. In the case where the amount of data in the data block sequence is too large, the multimedia data processing method provided by the embodiments of the present disclosure can avoid the problem of limiting the long text processing capability due to the mismatch between the hardware performance of the storage unit and the amount of data in the data block sequence, thereby improving the ability to solve long text tasks.

[0065] According to a related example, a processing method of truncating a long text may be adopted, for example, a data block sequence is split into multiple data blocks, and multiple data blocks are processed separately to obtain multiple data block features. Global feature extraction is performed on multiple data block features to obtain target features.

[0066] Compared with the processing method of truncating long texts, the multimedia data processing method provided by the embodiment of the present disclosure can establish context dependencies between multiple data blocks, thereby avoiding the degradation of processing effects caused by truncation of dependencies.

[0067] According to another related example, a non-truncated processing method can be used for long texts, such as offloading the intermediate data generated during the processing from the video memory to another storage space, such as internal memory, thereby reducing the pressure on the video memory and supporting the processing of longer long texts.

[0068] Compared with the non-truncation processing method for long texts, the multimedia data processing method provided by the embodiment of the present disclosure can process long text tasks while avoiding the processing delay problem caused by data transmission between the video memory and another storage space.

[0069] According to an embodiment of the present disclosure, when executing Figure 2Before the operation S210 shown, the multimedia data processing method may include the operation of extracting multimedia elements from the multimedia data and processing the multimedia elements to obtain a data block sequence.

[0070] According to an embodiment of the present disclosure, extracting multimedia elements from multimedia data may include: decomposing the multimedia data to obtain multiple multimedia elements, such as decomposing multimedia data containing video and audio to obtain video multimedia elements and audio multimedia elements.

[0071] According to an embodiment of the present disclosure, processing the multimedia element to obtain the data block sequence may include: splitting the multimedia element to obtain the data block sequence.

[0072] According to an embodiment of the present disclosure, when it is determined that the data volume of a multimedia element is greater than a predetermined data volume threshold, the multimedia element can be split according to the predetermined data volume threshold to obtain a data block sequence. When it is determined that the data volume of a multimedia element is less than or equal to the predetermined data volume threshold, the multimedia element can be randomly split to obtain a data block sequence.

[0073] According to an embodiment of the present disclosure, the data block sequence may be: for a video multimedia element or for an audio multimedia element, multiple data blocks are arranged in time sequence. For a text multimedia element, multiple data blocks are arranged in description sequence.

[0074] According to an embodiment of the present disclosure, by processing multimedia elements in multimedia data in this way to obtain a data block sequence, the data amount of each data block in the data block sequence can be adapted to the hardware unit used for calculation, thereby improving data processing efficiency.

[0075] According to the embodiments of the present disclosure, Figure 2 Operation S210 shown, determining an eviction strategy based on the available storage capacity of the storage unit and the data amount of the data block sequence, may include the following operations.

[0076] For example, based on the available storage capacity of the storage unit and the amount of data in the data block sequence, the eviction ratio of each data block in the data block sequence is determined. Based on the eviction ratios of the multiple data blocks, the eviction strategy is determined.

[0077] According to an embodiment of the present disclosure, the hardware performance detection result may refer to whether the available storage capacity of the storage unit can be used to store the intermediate data obtained by processing the data block sequence.

[0078] According to an embodiment of the present disclosure, determining a hardware performance test result based on the available storage capacity of a storage unit and the data volume of a data block sequence may include: determining a maximum data volume of intermediate data based on the data volume of the data block sequence. Determining a hardware performance test result based on the maximum data volume of the intermediate data and the available storage capacity of the storage unit.

[0079] According to an embodiment of the present disclosure, when the maximum amount of intermediate data is less than or equal to the difference between the available storage capacity of the storage unit and a predetermined storage capacity threshold, it is possible to determine that the hardware performance test result is used to characterize that the storage unit can support the storage of intermediate data of a data block sequence, and there is no need to evict intermediate data, such as the n-1th data block feature. When the maximum amount of intermediate data is greater than the difference between the available storage capacity of the storage unit and a predetermined storage capacity threshold, it is possible to determine that the hardware performance test result is used to characterize that the storage unit cannot support the storage of intermediate data of a data block sequence, and it is necessary to evict intermediate data, such as the n-1th data block feature.

[0080] According to an embodiment of the present disclosure, when it is determined that the hardware performance detection result is used to characterize that the storage unit cannot support the storage of intermediate data of the data block sequence, the eviction ratio of each data block in the data block sequence can be determined based on the available storage capacity of the storage unit and the data volume of the data block sequence. Based on the eviction ratios of each of the multiple data blocks, an eviction strategy is determined.

[0081] According to an embodiment of the present disclosure, the eviction ratio may be expressed as: the ratio of the amount of data to be evicted in the feature of the n-1th data block to the amount of data in the nth data block.

[0082] According to the embodiment of the present disclosure, the eviction ratio of each of the plurality of data blocks may be used as an eviction strategy, but the present invention is not limited thereto. The eviction ratio and the eviction mode may also be used as an eviction strategy.

[0083] According to an embodiment of the present disclosure, the eviction method may include an eviction method of deleting data, but is not limited thereto, and may also include an eviction method of updating data to 0.

[0084] According to other embodiments of the present disclosure, when the maximum amount of intermediate data is greater than the difference between the available storage capacity of the storage unit and the predetermined storage capacity threshold, the eviction ratio of each data block can be determined based on the predetermined available storage capacity of the storage unit. However, it is not limited to this. The eviction ratio can also be predetermined. When it is determined that the intermediate data, such as the n-1th data block feature, is evicted according to the predetermined eviction ratio, the predetermined eviction ratio is used as the eviction ratio, such as the eviction ratio of the n-1th data block.

[0085] According to the embodiments of the present disclosure, the eviction strategy is generated based on the available storage capacity of the storage unit and the data volume of the data block sequence, so that the eviction strategy can be dynamically changed with the change of the available storage capacity of the storage unit or the data volume of the data block sequence, thereby improving the flexibility of the multimedia data processing method. In addition, the eviction strategy includes the eviction ratio of each data block in the data block sequence, so that the processing granularity of the multimedia data processing method is fine and targeted.

[0086] Figure 3 A schematic diagram of determining an eviction strategy according to an embodiment of the present disclosure is schematically shown.

[0087] like Figure 3 As shown, the data block sequence includes a first data block 310, a second data block 320, a third data block 330, a fourth data block 340, a fifth data block 350, and a sixth data block 360. The data amounts of the first data block 310, the second data block 320, the third data block 330, the fourth data block 340, the fifth data block 350, and the sixth data block 360 are 8K, 4K, 4K, 4K, 4K, and 122, respectively.

[0088] like Figure 3 As shown, based on the available storage capacity of the storage unit and the data amount of the data block sequence, determining the eviction strategy includes: the eviction ratio of the first data block 310 is 1, the eviction ratio of the second data block 320 is 0.5, the eviction ratio of the third data block 330 is 0.5, the eviction ratio of the fourth data block 340 is 0.5, the eviction ratio of the fifth data block 350 is 0.5, and the eviction ratio of the sixth data block 360 is 0.5.

[0089] It should be noted that the effect of the eviction ratio of a data block may include: based on the eviction ratio of the current data block, the intermediate data before the current data block is evicted. For example, the effect of the eviction ratio of the nth data block may include: based on the eviction ratio of the nth data block, the feature of the n-1th data block is evicted.

[0090] like Figure 3 As shown, according to the eviction ratio of the second data block 320, 2K of the first data block feature 311 is evicted. According to the eviction ratio of the third data block 330, 2K of the second data block feature 321 is evicted. According to the eviction ratio of the fourth data block 340, 2K of the third data block feature 331 is evicted. According to the eviction ratio of the fifth data block 350, 2K of the fourth data block feature 341 is evicted. According to the eviction ratio of the sixth data block 360, 2K of the fifth data block feature 351 is evicted.

[0091] Thus, the eviction strategy is utilized to reduce the storage capacity of the storage unit and the data processing capacity of the processing process.

[0092] According to the embodiments of the present disclosure, Figure 2 The sub-operation in operation S220 shown, according to the eviction strategy, evicts the n-1th data block feature to obtain the n-1th post-eviction feature, may include: determining an eviction scheme based on the eviction strategy. Based on the eviction scheme, evicts the n-1th data block feature to obtain the n-1th post-eviction feature.

[0093] According to an embodiment of the present disclosure, an eviction scheme is used to characterize the position of an element to be evicted.

[0094] According to an embodiment of the present disclosure, the element position may refer to the position of a character, but is not limited thereto, and may also refer to the position of a vector element in a feature matrix, as long as it is used to characterize the position of the data to be evicted.

[0095] According to an embodiment of the present disclosure, multiple eviction schemes with different positions of elements to be evicted can be determined based on an eviction strategy. Each eviction scheme corresponds to a data block. However, this is not limited to this. Each eviction scheme can also correspond to a processing layer in multiple processing layers. Each eviction scheme can also correspond to a combination of a data block and a processing layer at the same time.

[0096] According to the embodiments of the present disclosure, an eviction scheme is determined based on an eviction strategy, and the eviction granularity can be refined to the element level. While ensuring that eviction is used to reduce the amount of data processing and storage, the richness of the data is ensured, and the problem of data loss due to eviction, which in turn leads to a decrease in accuracy, is avoided.

[0097] According to the embodiments of the present disclosure, Figure 2 Another sub-operation in the operation S220 shown, obtaining the nth data block feature based on the n-1th post-eviction feature and the nth data block, may include: inputting the n-1th post-eviction feature output by the kth processing layer and the nth data block feature output by the kth processing layer into the k+1th processing layer to obtain the nth data block feature output by the k+1th processing layer.

[0098] According to an embodiment of the present disclosure, K processing layers may be used to process a sequence of data blocks. The kth processing layer is the kth layer among the stacked K processing layers.

[0099] According to an embodiment of the present disclosure, k is an integer greater than or equal to 1 and less than K, and K is an integer greater than 1.

[0100] According to an embodiment of the present disclosure, the processing layer may include an attention mechanism (Attention) module and an FFN (FeedForward Neural Networks) module. However, it is not limited to this. The processing layer may also include an encoder of a codec. The processing layer may also include a codec. As long as the input data of the k+1th processing layer depends on the current nth data block, it also depends on the network structure of the features of the n-1th data block.

[0101] According to the embodiments of the present disclosure, in view of the characteristics of the network structure of this processing layer, the multimedia data processing method provided by the embodiments of the present disclosure can ensure the dependence on forward intermediate data while reducing the data processing volume, ensuring that the operation process is adapted to the hardware performance, and improving processing efficiency and operation stability.

[0102] Figure 4 A schematic diagram of a multimedia data processing method according to an embodiment of the present disclosure is schematically shown.

[0103] like Figure 4 As shown, the eviction strategy corresponding to the second data block is determined based on the second data block, and the data volume of the first data block characteristic is: the eviction ratio of the data volume of the second data block, for example, 50%.

[0104] like Figure 4 As shown, the first data block includes characters T0, T1, T2 and T3, and the second data block includes characters T4, T5, T6 and T7. An eviction plan can be determined based on an eviction strategy.

[0105] like Figure 4 As shown, the eviction scheme of the first processing layer M410 can be used to evict the data of the element positions corresponding to the characters T0 and T1 in the first data block obtained from the storage unit V410, and obtain the 0th eviction data 410. The 0th eviction data and the second data block 420 are input to the first processing layer M410, and the second data block feature 430 output by the first processing layer M410 is obtained.

[0106] like Figure 4 As shown, the eviction scheme of the second processing layer M420 can be used to evict the data of the element positions corresponding to the characters T0 and T2 in the first data block feature output by the first processing layer obtained from the storage unit V410, and obtain the first post-eviction feature 440 output by the first processing layer. The first post-eviction feature 440 output by the first processing layer and the second data block feature 430 output by the first processing layer are input to the second processing layer M420, and the second data block feature 450 output by the second processing layer is obtained.

[0107] like Figure 4As shown, the eviction scheme of the third processing layer M430 can be used to evict the data of the element positions corresponding to the characters T0 and T3 in the first data block feature output by the second processing layer obtained from the storage unit V410, and obtain the first post-eviction feature 460 output by the second processing layer. The first post-eviction feature 460 output by the second processing layer and the second data block feature 450 output by the second processing layer are input to the third processing layer M430, and the second data block feature 470 output by the third processing layer is obtained.

[0108] According to the embodiments of the present disclosure, the eviction scheme provided by the embodiments of the present disclosure can make the character positions evicted by different processing layers different, thereby reducing the amount of data processing and storage, enriching the types of input data and avoiding data missing problems caused by eviction.

[0109] According to another embodiment of the present disclosure, Figure 2 Another sub-operation in the operation S220 shown is to evict the n-1th data block feature according to the eviction strategy to obtain the n-1th post-eviction feature, which may include the following operations.

[0110] For example, based on the eviction strategy, K different eviction patterns are determined. A target eviction pattern matching the kth processing layer is determined from the K eviction patterns in a random manner. Based on the target eviction pattern, the n-1th data block feature is evicted to obtain the n-1th post-eviction feature.

[0111] For another example, based on the eviction strategy, K different eviction modes are determined. Based on the weights of the K processing layers, a target eviction mode matching the kth processing layer is determined from the K eviction modes. Based on the target eviction mode, the n-1th data block feature is evicted to obtain the n-1th post-eviction feature.

[0112] According to an embodiment of the present disclosure, the eviction pattern can be used to characterize the element positions to be evicted corresponding to the multi-head attention module of each processing layer.

[0113] According to an embodiment of the present disclosure, the weight of each processing layer is determined based on the attention score matrix of the multi-head attention module.

[0114] According to an embodiment of the present disclosure, the multi-head attention module may include multiple attention processing branches (AttentionHead). Each attention processing branch may include an attention processing function Q represents the query vector, K represents the key vector, V represents the value vector, and d k Represents the coefficient. The data blocks in the data block sequence can be converted into embedding vectors, and the query vector, key vector and value vector are obtained based on the embedding vectors. Q*K TIt can be represented as an attention score matrix. Based on the attention score matrix, a target eviction mode that matches the attention score matrix can be determined. For example, if the element value in the attention score matrix is ​​large, such as the element value exceeds a predetermined value threshold, the element position of the corresponding input data will not be evicted.

[0115] According to an embodiment of the present disclosure, the weight of the processing layer may also be determined based on the attention score matrix. Based on the weight, a target expulsion mode matching the attention score matrix is ​​determined.

[0116] According to an embodiment of the present disclosure, for the Transformer model architecture, different processing layers have specific substructure characteristics. Different positions of elements to be evicted can be set for multiple attention processing branches of the multi-head attention module, and the weight of each processing layer is considered to determine the target eviction mode that matches the processing layer, and the weight is determined based on the attention score matrix of the multi-head attention module. In this way, full use is made of the different focus points of different attention processing branches on features in the multi-head attention module. By setting independent and focus-appropriate eviction modes for different processing layers and different attention processing branches, the combination diversity of the data that is not evicted is maximized, while the amount of knowledge of the learned data is maximized, thereby ensuring that it is compatible with the hardware performance of the storage unit while ensuring the accuracy of the target processing results.

[0117] Figure 5 A schematic diagram of expelling data according to an embodiment of the present disclosure is schematically shown.

[0118] like Figure 5 As shown, for the first processing layer M510, the first eviction mode can be used to evict the data block, which may include: splitting the data block into three parts, and evicting the first element and the third element of the first sub-data block respectively, to obtain the first 0th eviction data 511. Evicting the second element and the fourth element of the second sub-data block 512 respectively, to obtain the second 0th eviction data 512. Evicting the first element and the second element of the third sub-data block 513 respectively, to obtain the third 0th eviction data 513. Inputting the first 0th eviction data 511 to the attention processing branch Head1-1 in the first processing layer M510. Inputting the second 0th eviction data 512 to the attention processing branch Head1-2. Inputting the third 0th eviction data 513 to the attention processing branch Head1-3.

[0119] like Figure 5As shown, for the second processing layer M520, the second eviction mode can be used to evict the data block, which may include: splitting the first data block feature 520 into three parts, and respectively evict the second element and the fourth element of the first sub-data block feature to obtain the first first eviction feature 521. Evicting the first element and the third element of the second sub-data block feature to obtain the second first eviction feature 522. Evicting the second element and the third element of the third sub-data block feature 523 to obtain the third first eviction feature 523. Inputting the first first eviction feature 521 into the attention processing branch Head2-1 in the second processing layer M520. Inputting the second first eviction feature 522 into the attention processing branch Head2-2. Inputting the third first eviction feature 523 into the attention processing branch Head2-3.

[0120] like Figure 5 As shown, for the third processing layer M530, the third mode data block can be used for expulsion, which may include: splitting the second data block feature into three parts, and expelling the first element and the fourth element positions of the first sub-data block feature respectively, to obtain the first second expelled feature 531. Expelling the second element and the third element positions of the second sub-data block feature respectively, to obtain the second second expelled feature 532. Expelling the first element and the fourth element positions of the third sub-data block feature respectively, to obtain the third second expelled feature 533. Inputting the first second expelled feature 531 to the attention processing branch Head3-1 in the third processing layer M530. Inputting the second second expelled feature 532 to the attention processing branch Head3-2. Inputting the third second expelled feature 533 to the attention processing branch Head3-3.

[0121] By using the above-mentioned multiple different eviction modes, the positions of elements to be evicted in different processing layers and different attention processing branches in the processing layers are different, thereby making full use of the different focus points of different attention processing branches in the multi-head attention module on features. By setting independent and focus-appropriate eviction modes for different processing layers and different attention processing branches, the combination diversity of the data that is not evicted is maximized, and the amount of knowledge of the learned data is maximized, thereby ensuring the adaptation to the hardware performance of the storage unit while ensuring the accuracy of the target processing result.

[0122] According to other embodiments of the present disclosure, the n-1th post-eviction feature may include the n-1th post-eviction key feature and the n-1th post-eviction value feature. The n-th data block feature may include the n-th data block key feature and the n-th data block value feature.

[0123] According to other embodiments of the present disclosure, inputting the n-1th expelled feature and the nth data block output by the kth processing layer into the k+1th processing layer to obtain the nth data block feature output by the k+1th processing layer may include the following operations.

[0124] For example, the n-1th expelled key feature output by the kth processing layer and the nth data block key feature output by the kth processing layer are fused to obtain the nth key feature. The n-1th expelled value feature output by the kth processing layer and the nth data block value feature output by the kth processing layer are fused to obtain the nth value feature. The nth key feature and the nth value feature are input into the attention mechanism module of the k+1th processing layer to obtain the nth data block feature output by the k+1th processing layer.

[0125] According to an embodiment of the present disclosure, fusing the n-1th post-expulsion key feature output by the k-th processing layer and the nth data block key feature output by the k-th processing layer to obtain the n-th key feature may include: channel splicing the n-1th post-expulsion key feature output by the k-th processing layer and the nth data block key feature output by the k-th processing layer to obtain the n-th key feature. However, it is not limited to this, and the n-1th post-expulsion key feature output by the k-th processing layer and the nth data block key feature output by the k-th processing layer may also be dot-producted to obtain the n-th key feature. As long as it is a method that can fuse the two features. The fusion method of fusing the n-1th post-expulsion value feature output by the k-th processing layer and the nth data block value feature output by the k-th processing layer to obtain the n-th value feature is consistent with the above explanation and will not be repeated here.

[0126] According to an embodiment of the present disclosure, the nth key feature and the nth value feature are input into the attention mechanism module of the k+1th processing layer to obtain the nth data block feature output by the k+1th processing layer, which may include: the nth key feature, the nth value feature and the nth data block query feature output by the kth processing layer are input into the attention processing function Get the features of the nth data block output by the k+1th processing layer.

[0127] According to the embodiments of the present disclosure, the data block features obtained by the above processing method can be stored in KV-Cache (Key-Value, key-value pair cache) mode, thereby improving the adaptability of the multimedia data processing method to the existing hardware units, thereby increasing the scope of application.

[0128] According to an embodiment of the present disclosure, when executing Figure 2 After the operation S220 shown, the multimedia data processing method may further include an operation of decoding the target feature and the predetermined indicator to obtain a target processing result.

[0129] According to an embodiment of the present disclosure, the target feature and the predetermined indicator can be input into the decoder (Decoder) of the encoder / decoder (Transformer) to obtain the target processing result. However, it is not limited to this. It can also include a decoder of a large model. The large model or large language model can be a GPT model, and the full name of the GPT model is "Generative Pre-training Transformer", and the Chinese name is "Generative Pre-training Transformer".

[0130] According to an embodiment of the present disclosure, the target processing result may include a summary corresponding to the data block sequence. However, it is not limited thereto. The target processing result may also include an answer corresponding to the data block sequence as a question. As long as it is a target processing result output based on the data block sequence.

[0131] According to an embodiment of the present disclosure, the predetermined indicator can be used to characterize the task type, such as a task of generating a summary or a task of generating an answer. However, it is not limited to this. The predetermined indicator can also include a related index of the reference answer. For example, the relevant content of the processing result corresponding to the data block sequence can be determined based on the predetermined indicator. Based on the relevant content, the target processing result corresponding to the data block sequence is determined.

[0132] Exemplarily, determining the target processing result corresponding to the data block sequence based on the relevant content may include: processing the relevant content to obtain auxiliary features. Obtaining the target processing result based on the auxiliary features and the target features. However, it is not limited to this, and the initial processing result may also be determined based on the target features. Obtaining the target processing result based on the initial processing result and the relevant content.

[0133] According to the embodiments of the present disclosure, the target features are processed by predetermined indicators to obtain target processing results. By changing the data type or processing method of the predetermined indicator, the application scope of the multimedia data processing method provided by the embodiments of the present disclosure can be improved and the user experience can be improved.

[0134] According to an embodiment of the present disclosure, when executing Figure 2 After the operation S220 shown, the multimedia data processing method may further include the operations of: training samples and a predetermined mask matrix, training an initial processing layer, and obtaining a processing layer.

[0135] According to an embodiment of the present disclosure, a training sample may include a sample data block sequence and a sample target processing result.

[0136] According to the embodiments of the present disclosure, the meaning and interpretation of the sample data block sequence can be found in the data block sequence. The meaning and interpretation of the sample target processing result can be found in the target processing result. No further details will be given here.

[0137] According to an embodiment of the present disclosure, using training samples and a predetermined mask matrix to train an initial processing layer to obtain a processing layer may include: processing a sample data block sequence and a predetermined mask matrix to obtain a mask sample data block sequence. Inputting the mask sample data block sequence into the initial processing layer to obtain an output result. Based on the output result and the sample target processing result, adjusting the parameters of the initial processing layer to obtain the processing layer.

[0138] According to the embodiments of the present disclosure, the network structure of the initial processing layer can refer to the processing layer, and only the parameters are different, which will not be repeated here.

[0139] According to an embodiment of the present disclosure, a sample data block sequence and a predetermined mask matrix are processed to obtain a mask sample data block sequence, which can be adopted mask can represent a predetermined mask matrix.

[0140] According to an embodiment of the present disclosure, determining a mask matrix may include: determining, based on a mask pattern, an element position to be masked of sample input data of each initial processing layer in a plurality of initial processing layers. Determining a mask matrix based on the element position to be masked of the sample input data. In order to process the sample input data based on the mask matrix, an output result output by the initial processing layer is an output result with a reduced data amount.

[0141] According to an embodiment of the present disclosure, the initial processing layer is trained together with a mask matrix determined based on a mask pattern and a training sample, so that the obtained processing layer can be trained to have high reasoning performance when part of the data is masked. In addition, the mask matrix is ​​determined based on the mask pattern, which can improve the consistency of the mask principle of the training process and the application process of the processing layer, and improve the processing capacity of the trained processing layer.

[0142] Figure 6 A schematic diagram of determining a mask matrix according to an embodiment of the present disclosure is schematically shown.

[0143] like Figure 6As shown, for the attention processing branch Head1 in the initial processing layer, the sample data block sequence may include character 1, character 2, character 3, ..., character 10 (marked with dots). Masking may not be performed during the processing of the first four characters. For character 5, the element position corresponding to character 2 in the feature of the fourth data block (marked with blank non-cross grid) may be masked. For character 6, the element position corresponding to character 2 and character 5 in the feature of the fifth data block may be masked. For character 10, the element position corresponding to character 1-character 5 and character 9 in the feature of the ninth data block may be masked. The relevant data features of four characters (marked with cross grids) are retained as the mean value of each row. Based on the character block diagram, a mask pattern for each attention processing branch in the initial processing layer may be obtained. Based on the mask pattern, a mask matrix 610 corresponding to each attention processing branch in the initial processing layer is obtained.

[0144] It should be noted that the “characters” involved in the embodiments of the present disclosure are all used to indicate “Tokens” in the language model.

[0145] Figure 7 The block diagram of a multimedia data processing device according to an embodiment of the present disclosure is schematically shown.

[0146] like Figure 7 As shown, the multimedia data processing device 700 may include: a strategy determination module 710 , a repeat operation module 720 and a target determination module 730 .

[0147] The strategy determination module 710 is used to determine the mask strategy based on the available storage capacity of the storage unit and the data amount of the data block sequence. The data block sequence is obtained by processing the multimedia elements of the multimedia data. The mask strategy is used to characterize the data amount of the data to be masked. The data block sequence includes N data blocks, where N is an integer greater than or equal to 2.

[0148] The repeat operation module 720 is used for the repeat operation module, and is used for repeatedly performing the following operations until n is equal to N.

[0149] The target determination module 730 is used to obtain a target processing result based on the characteristics of the nth data block when it is determined that n is equal to N.

[0150] According to an embodiment of the present disclosure, the repetitive operation module 720 may include a mask submodule 721 , a processing submodule 722 , and a storage submodule 723 .

[0151] The mask submodule 721 is used to mask the n-1th data block feature based on the mask strategy to obtain the n-1th post-eviction feature when it is determined that n is less than N. The n-1th data block feature is obtained from the storage unit. n is an integer greater than or equal to 2 and less than or equal to N.

[0152] The processing submodule 722 is used to obtain the feature of the nth data block based on the feature after the (n-1) eviction and the nth data block.

[0153] The storage submodule 723 is used to store the features of the nth data block in a storage unit.

[0154] According to an embodiment of the present disclosure, the strategy determination module may include: a ratio determination submodule and a measurement determination submodule.

[0155] The ratio determination submodule is used to determine the mask ratio of each data block in the data block sequence based on the available storage capacity of the storage unit and the data amount of the data block sequence.

[0156] The strategy determination submodule is used to determine the mask strategy based on the mask ratios of the multiple data blocks.

[0157] According to an embodiment of the present disclosure, the mask submodule may include: a scheme determination unit and a first mask unit.

[0158] The scheme determination unit is used to determine a mask scheme based on the mask strategy. The mask scheme is used to characterize the position of the element to be masked.

[0159] The first masking unit is used to mask the feature of the n-1th data block based on a masking scheme to obtain the n-1th post-eviction feature.

[0160] According to an embodiment of the present disclosure, the processing submodule may include: a processing unit.

[0161] The processing unit is used to input the n-1th expulsion feature output by the kth processing layer and the nth data block feature output by the kth processing layer into the k+1th processing layer to obtain the nth data block feature output by the k+1th processing layer. The kth processing layer is the kth layer of the stacked K processing layers, k is an integer greater than or equal to 1 and less than K, and K is an integer greater than 1.

[0162] According to an embodiment of the present disclosure, the expulsion submodule includes: a mode preliminary determination unit, a mode target determination unit and a second expulsion unit.

[0163] The mode preliminary determination unit is used to determine K different eviction modes based on the eviction strategy. The eviction mode is used to characterize the position of the element to be evicted corresponding to the multi-head attention module of each processing layer.

[0164] A mode target determination unit is used to determine a target eviction mode matching the kth processing layer from the K eviction modes based on the respective weights of the K processing layers. The weight of each processing layer is determined based on the attention score matrix of the multi-head attention module.

[0165] The pattern target determination unit can also be used to determine a target eviction pattern matching the kth processing layer from the K eviction patterns in a random manner. Based on the target eviction pattern, the n-1th data block feature is evicted to obtain the n-1th post-eviction feature.

[0166] The second eviction unit is used to evict the feature of the n-1th data block based on the target eviction mode to obtain the feature after the n-1th eviction.

[0167] According to an embodiment of the present disclosure, the n-1th post-eviction feature includes the n-1th post-eviction key feature and the n-1th post-eviction value feature. The n-th data block feature includes the n-th data block key feature and the n-th data block value feature.

[0168] According to an embodiment of the present disclosure, the processing unit includes: a first fusion subunit, a second fusion subunit and a processing subunit.

[0169] The first fusion subunit is used to fuse the n-1th eviction key feature output by the kth processing layer and the nth data block key feature output by the kth processing layer to obtain the nth key feature.

[0170] The second fusion subunit is used to fuse the n-1th expulsion value feature output by the kth processing layer and the nth data block value feature output by the kth processing layer to obtain the nth value feature.

[0171] The processing subunit is used to input the nth key feature and the nth value feature into the attention mechanism module of the k+1th processing layer to obtain the nth data block feature output by the k+1th processing layer.

[0172] According to an embodiment of the present disclosure, it may further include: a training module.

[0173] The training module is used for training the initial processing layer by using the training sample and the expulsion matrix to obtain the processing layer. The expulsion matrix is ​​determined based on the expulsion mode.

[0174] According to an embodiment of the present disclosure, the target determination module may include: a target determination submodule.

[0175] The target determination submodule is used to decode the characteristics of the nth data block and the predetermined indicator to obtain a target processing result.

[0176] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0177] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method as in the embodiment of the present disclosure.

[0178] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute a method as described in an embodiment of the present disclosure.

[0179] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the method according to the embodiment of the present disclosure is implemented.

[0180] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0181] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0182] A number of components in the device 800 are connected to an input / output (I / O) interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0183] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as multimedia data processing methods. For example, in some embodiments, the multimedia data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the multimedia data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the multimedia data processing method in any other appropriate manner (e.g., by means of firmware).

[0184] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0185] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable multimedia data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0186] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0187] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0188] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0189] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0190] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0191] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A multimedia data processing method, comprising: Determining an eviction strategy based on available storage capacity of a storage unit and a data amount of a data block sequence, wherein the data block sequence is obtained by processing multimedia elements of the multimedia data, the eviction strategy is used to characterize the data amount of the data to be evicted, the data block sequence includes N data blocks, and N is an integer greater than or equal to 2; Repeat the following steps until n equals N: When it is determined that n is less than N, based on the eviction strategy, evict the n-1th data block feature to obtain the n-1th post-eviction feature, wherein the n-1th data block feature is obtained from the storage unit, and n is an integer greater than or equal to 2 and less than or equal to N; Based on the (n-1)th post-eviction feature and the (n)th data block, obtaining the (n)th data block feature; and Storing the nth data block feature in the storage unit; and When it is determined that n is equal to N, a target processing result is obtained based on the characteristics of the nth data block; The step of obtaining the nth data block feature based on the n-1th post-eviction feature and the nth data block includes: The n-1th post-expulsion feature output by the kth processing layer and the nth data block feature output by the kth processing layer are input into the k+1th processing layer to obtain the nth data block feature output by the k+1th processing layer, wherein the kth processing layer is the kth layer of the stacked K processing layers, k is an integer greater than or equal to 1 and less than K, and K is an integer greater than 1.

2. The method according to claim 1, wherein: The step of determining the eviction strategy based on the available storage capacity of the storage unit and the data volume of the data block sequence includes: Determining an eviction ratio of each of the data blocks in the data block sequence based on the available storage capacity of the storage unit and the data amount of the data block sequence; and The eviction strategy is determined based on the eviction ratios of the plurality of data blocks.

3. The method according to claim 1, wherein: The step of evicting the n-1th data block feature based on the evicting strategy to obtain the n-1th post-evicting feature includes: Based on the eviction strategy, determining an eviction scheme, wherein the eviction scheme is used to characterize the position of the element to be evicted; and Based on the eviction scheme, the n-1th data block feature is evicted to obtain the n-1th post-eviction feature.

4. The method according to claim 1, wherein: The step of evicting the n-1th data block feature based on the evicting strategy to obtain the n-1th post-evicting feature includes: Based on the eviction strategy, determining K different eviction modes, wherein the eviction mode is used to characterize the position of the element to be evicted corresponding to the multi-head attention module of each processing layer; Determine, in a random manner, a target eviction pattern matching the kth processing layer from the K eviction patterns; and Based on the target eviction mode, the n-1th data block feature is evicted to obtain the n-1th post-eviction feature.

5. The method according to claim 1, wherein: The n-1th post-eviction feature includes the n-1th post-eviction key feature and the n-1th post-eviction value feature; the nth data block feature includes the nth data block key feature and the nth data block value feature; The step of inputting the n-1th post-expulsion feature output by the kth processing layer and the nth data block feature output by the kth processing layer into the k+1th processing layer to obtain the nth data block feature output by the k+1th processing layer comprises: fusing the n-1th post-expulsion key feature output by the kth processing layer and the nth data block key feature output by the kth processing layer to obtain an nth key feature; fusing the (n-1)th post-expulsion value feature output by the (k)th processing layer and the (n)th data block value feature output by the (k)th processing layer to obtain the (n)th value feature; and The nth key feature and the nth value feature are input into the attention mechanism module of the k+1th processing layer to obtain the nth data block feature output by the k+1th processing layer.

6. The method according to any one of claims 1 to 5, further comprising: The initial processing layer is trained using the training samples and a predetermined mask matrix to obtain the processing layer.

7. The method according to any one of claims 1 to 5, wherein: The obtaining of a target processing result based on the feature of the nth data block includes: The nth data block feature and the predetermined indicator are decoded to obtain the target processing result.

8. A multimedia data processing device, comprising: a strategy determination module, configured to determine an eviction strategy based on an available storage capacity of a storage unit and a data amount of a data block sequence, wherein the data block sequence is obtained by processing multimedia elements of the multimedia data, the eviction strategy is used to characterize the data amount of the data to be evicted, and the data block sequence includes N data blocks, where N is an integer greater than or equal to 2; The repeating operation module is used to repeatedly perform the following operations until n is equal to N: an eviction submodule, configured to, when it is determined that n is less than N, evict the n-1th data block feature based on the eviction strategy to obtain the n-1th post-eviction feature, wherein the n-1th data block feature is obtained from the storage unit, and n is an integer greater than or equal to 2 and less than or equal to N; a processing submodule, configured to obtain an nth data block feature based on the n-1th post-eviction feature and the nth data block; and a storage submodule, configured to store the nth data block feature in the storage unit; and A target determination module, used for obtaining a target processing result based on the characteristics of the nth data block when n is determined to be equal to N; Wherein, the processing submodule includes: A processing unit is used to input the n-1th expulsion feature output by the kth processing layer and the nth data block feature output by the kth processing layer into the k+1th processing layer to obtain the nth data block feature output by the k+1th processing layer, wherein the kth processing layer is the kth layer of the stacked K processing layers, k is an integer greater than or equal to 1 and less than K, and K is an integer greater than 1.

9. The device according to claim 8, wherein: The strategy determination module includes: a ratio determination submodule, configured to determine an eviction ratio of each of the data blocks in the data block sequence based on an available storage capacity of the storage unit and a data amount of the data block sequence; and The strategy determination submodule is used to determine the eviction strategy based on the eviction ratios of the plurality of data blocks.

10. The device according to claim 8, wherein: The expulsion submodule comprises: a scheme determining unit, configured to determine an eviction scheme based on the eviction strategy, wherein the eviction scheme is used to characterize the position of the element to be evicted; and The first eviction unit is used to evict the n-1th data block feature based on the eviction scheme to obtain the n-1th post-eviction feature.

11. The device according to claim 8, wherein: The expulsion submodule includes: A preliminary mode determination unit, used for determining K different eviction modes based on the eviction strategy, wherein the eviction mode is used to characterize the position of the element to be evicted corresponding to the multi-head attention module of each processing layer; a pattern target determination unit, configured to determine, in a random manner, a target eviction pattern matching the kth processing layer from the K eviction patterns; and The second eviction unit is used to evict the n-1th data block feature based on the target eviction mode to obtain the n-1th post-eviction feature.

12. The device according to claim 8, wherein: The n-1th post-eviction feature includes the n-1th post-eviction key feature and the n-1th post-eviction value feature; the nth data block feature includes the nth data block key feature and the nth data block value feature; The processing unit comprises: A first fusion subunit is used to fuse the (n-1)th expulsion key feature output by the kth processing layer and the nth data block key feature output by the kth processing layer to obtain the nth key feature; A second fusion subunit is used to fuse the (n-1)th expulsion value feature output by the kth processing layer and the nth data block value feature output by the kth processing layer to obtain the nth value feature; and The processing subunit is used to input the nth key feature and the nth value feature into the attention mechanism module of the k+1th processing layer to obtain the nth data block feature output by the k+1th processing layer.

13. The device according to any one of claims 8 to 12, further comprising: The training module is used to train the initial processing layer using the training samples and the predetermined mask matrix to obtain the processing layer.

14. The device according to any one of claims 8 to 12, wherein: The target determination module comprises: The target determination submodule is used to decode the nth data block feature and the predetermined indicator to obtain the target processing result.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images