Multimodal data processing method, device, medium and computer program product

Abstract

The present invention relates to the technical field of data processing, and discloses a multimodal data processing method, device, medium and computer program product, the method comprising: extracting features from visual data and text data, writing the extracted visual modal features into the host extended memory, and writing the extracted text modal features into the device memory; segmenting the visual modal features to obtain visual feature fragments and write them into the device memory; fusing the text modal features with the visual feature fragments, and writing the intermediate results of the fused features after segmentation and corresponding calculation into the on-chip static memory; and writing the final target result into the device memory. In this way, the video memory occupation can be reduced, the correctness of feature alignment can be guaranteed, and the loss of attention to multimodal feature text can be avoided.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal data processing method, device, medium and computer program product. Background Art

[0002] Traditional multimodal models are mainly designed for image and text modalities, and the core design links are the computational acceleration of the attention mechanism and the alignment of multimodal features. However, the current attention mechanism has the problem of high video memory usage, especially some feature data in the multimodal model occupy a large amount of device memory, which reduces the running speed of the model. In addition, multimodal feature alignment refers to finding the relationship and correspondence between the subcomponents of an instance from two or more modalities. However, feature alignment often requires the alignment operation to be performed under the condition that multiple modal data are complete, otherwise it will make it difficult to align effective features between some features. Summary of the invention

[0003] The purpose of the present invention is to provide a multimodal data processing method, device, medium and computer program product, which can reduce video memory usage, ensure the correctness of feature alignment, and avoid the loss of attention of multimodal feature text.

[0004] In order to solve the above technical problems, the present invention provides a multimodal data processing method, the method comprising:

[0005] Extract features from the visual data and the text data, write the extracted visual modality features into the host extended memory, and write the extracted text modality features into the device memory;

[0006] Segmenting the visual modality feature to obtain visual feature segments and writing the segments into the device memory;

[0007] Fusing the text modal feature with the visual feature fragment to obtain a fused feature;

[0008] The fused features are processed, and the processed results are written into the device memory.

[0009] In a first aspect, in the above-mentioned multimodal data processing method provided by the present invention, before writing the extracted visual modality features into the host extended memory, it also includes:

[0010] The device memory is expanded using a consistency protocol; the expanded memory is used as the host extended memory;

[0011] The host extended memory and the device memory are uniformly addressed.

[0012] On the other hand, in the multimodal data processing method provided by the present invention, segmenting the visual modality features to obtain visual feature segments and writing them into the device memory includes:

[0013] Processing the visual modality feature in blocks to obtain a plurality of visual feature blocks;

[0014] Each of the visual feature blocks is sliced ​​to obtain a plurality of visual feature segments, and the visual feature segments are written into the device memory using a consistency protocol.

[0015] On the other hand, in the multimodal data processing method provided by the present invention, the visual modality feature is processed in blocks to obtain a plurality of visual feature blocks, including:

[0016] The visual modality feature is processed in blocks according to the number of channels of the computer expansion bus standard interface equipped on the server to obtain a plurality of visual feature blocks;

[0017] Each of the visual feature blocks is sliced ​​to obtain a plurality of visual feature segments, including:

[0018] Each of the visual feature blocks is sliced ​​according to the number of heads in the multi-head attention mechanism to obtain multiple visual feature fragments.

[0019] On the other hand, in the multimodal data processing method provided by the present invention, the text modal feature is fused with the visual feature segment to obtain the fused feature, including:

[0020] Loading weights for calculating the attention mechanism in the device memory; the weights for calculating the attention mechanism include query weights, key weights, and value weights;

[0021] Fusion the text modal feature with the visual feature segment to obtain a multimodal feature input segment;

[0022] The weights used to calculate the attention mechanism and the multimodal feature input fragments are calculated to obtain a query matrix, a key matrix, and a value matrix.

[0023] On the other hand, in the multimodal data processing method provided by the present invention, processing the fused features includes:

[0024] Segment the fused features and write the segmentation results into the on-chip static memory;

[0025] An intermediate result is calculated based on the segmentation result and written into the on-chip static memory.

[0026] On the other hand, in the above-mentioned multimodal data processing method provided by the present invention, the fused features are segmented, the segmentation results are written into the on-chip static memory, and the intermediate results are calculated according to the segmentation results and written into the on-chip static memory, including:

[0027] The query matrix, key matrix and value matrix are split to obtain query matrix fragments, key matrix fragments and value matrix fragments and written into the on-chip static memory; while splitting, the slice position index and slice dimension are maintained;

[0028] Obtaining a position encoding of a query matrix segment using the slice position index, the slice dimension, the scaling factor and the translation factor;

[0029] According to the position code of the query matrix fragment, the query matrix fragment and the key matrix fragment, a first intermediate result is calculated and written into the on-chip static memory.

[0030] On the other hand, in the above-mentioned multimodal data processing method provided by the present invention, according to the position encoding of the query matrix fragment, the query matrix fragment and the key matrix fragment, calculating a first intermediate result and writing it into the on-chip static memory includes:

[0031] Transposing the key matrix fragment to obtain a transposed key matrix fragment;

[0032] A matrix multiplication calculation is performed on the transposed key matrix fragment and the query matrix fragment and a position code of the query matrix fragment is added to obtain a first intermediate result and write the result into the on-chip static memory.

[0033] On the other hand, in the above-mentioned multimodal data processing method provided by the present invention, after obtaining the first intermediate result and writing it into the on-chip static memory, it also includes:

[0034] Record and maintain the row maximum value in the first intermediate result;

[0035] The first intermediate result is converted row by row into a probability distribution vector representing the probability of each category by using the row maximum value in the first intermediate result, and a second intermediate result is obtained and written into the on-chip static memory.

[0036] On the other hand, in the multimodal data processing method provided by the present invention, converting the first intermediate result row by row into a probability distribution vector representing the probability of each category includes:

[0037] Calculate the difference between each row element value of the first intermediate result and the row maximum value;

[0038] The calculated difference between the element value of each row and the row maximum value is used as an exponential power for exponential operation to obtain an exponent and a constant.

[0039] On the other hand, in the above-mentioned multimodal data processing method provided by the present invention, after obtaining the second intermediate result and writing it into the on-chip static memory, it also includes:

[0040] updating and maintaining said indices and constants;

[0041] Perform matrix multiplication on the second intermediate result and the value matrix fragment to obtain a third intermediate result and write it into the on-chip static memory; the third intermediate result is a slice in the attention matrix.

[0042] On the other hand, in the multimodal data processing method provided by the present invention, writing the target result obtained by processing into the device memory includes:

[0043] The third intermediate result is written into the corresponding address of the device memory to write the final target result into the device memory; all slices in the attention matrix are used as the target result.

[0044] In order to solve the above technical problems, the present invention further provides a multimodal data processing device, the device comprising:

[0045] Memory for storing computer programs;

[0046] A processor is used to implement the steps of the above-mentioned multimodal data processing method when executing the computer program.

[0047] In order to solve the above technical problem, the present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above multimodal data processing method are implemented.

[0048] In order to solve the above technical problem, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the above multimodal data processing method when executed by a processor.

[0049] It can be seen from the above technical scheme that the present invention provides a multimodal data processing method, which includes: extracting features from visual data and text data, writing the extracted visual modal features into the host extended memory, and writing the extracted text modal features into the device memory; segmenting the visual modal features to obtain visual feature segments and writing them into the device memory; fusing the text modal features with the visual feature segments to obtain fused features; processing the fused features, and writing the purpose results obtained by the processing into the device memory.

[0050] The beneficial effect of the present invention lies in that the above-mentioned multimodal data processing method provided by the present invention, after feature extraction of visual data and text data, writes the extracted visual modal features into the extended memory, and writes the extracted text modal features into the device memory, so that the visual modal features and text modal features are respectively stored in different memories, and the visual modal features with larger memory occupancy are stored in the extended memory, which can reduce the memory occupancy; the complete text modal features are fused with the visual feature fragments obtained after segmentation, which can ensure the correctness of feature alignment and avoid the loss of multimodal feature text attention; the target result output after processing of the fused features is written into the device memory, ensuring the stability of the output of the data processing results.

[0051] In addition, the present invention also provides corresponding multimodal data processing equipment, non-volatile storage media and computer program products for the multimodal data processing method, which have the same or corresponding technical features as the above-mentioned multimodal data processing method and have the same effects as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A flowchart of a multimodal data processing method provided by an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a framework corresponding to the multimodal data processing method provided in an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of the structure of a multimodal data processing device provided by an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of the structure of a multimodal data processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] Traditional multimodal models are mainly designed for image and text modalities, and the core design links are the computational acceleration of the attention mechanism and the alignment of multimodal features. However, the existing attention mechanism has some problems, including high video memory usage and high read and write times of High Bandwidth Memory (HBM).

[0058] The memory of the graphics processing unit (GPU) consists of multiple memories of different sizes and different read and write speeds. The smaller the memory, the faster the read and write speed. The static random access memory (SRAM) memory is distributed on 108 streaming multiprocessors, each of which is 192K in size, totaling 20M, which is equivalent to a computing block, but with small memory. The HBM size is 40GB. The read and write speed of SRAM is 19TB / s, while the read and write speed of HBM is only 1.5TB / s, less than 1 / 10 of SRAM, which is equivalent to slow computing and large memory. Therefore, the computing speed of modern GPUs has far exceeded the video memory access speed, and the bottleneck of most computing operations in transformer (a deep learning model architecture) is video memory access. For video memory-limited operations, input / output (I / O) perception is very important because video memory reading and writing takes up most of the running time. In large multimodal artificial intelligence (AI) models, feature data of visual modalities such as images and videos occupy a large amount of device memory, reducing the running speed of the model.

[0059] In addition, multimodal feature alignment refers to finding the relationship and correspondence between subcomponents of instances from two or more modalities. Multimodal alignment is divided into two types: implicit alignment and explicit alignment. In explicit alignment, the subcomponents between modes are explicitly adjusted. Implicit alignment is used as an intermediate (usually potential) step for another task. However, feature alignment often requires the alignment operation to be performed under the condition that the multimodal data is complete, otherwise it will make it difficult to align effective features between some features.

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 A flowchart of a multimodal data processing method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0062] S101. Extract features from visual data and text data, write the extracted visual modality features into the host extended memory, and write the extracted text modality features into the device memory.

[0063] During implementation, the present invention can perform feature extraction on visual data and text data to obtain visual modal features and text modal features. The visual modal features and text modal features are stored in different memories, wherein the visual modal features are stored in the host extended memory and the text modal features are stored in the device memory. This is to reduce the device memory usage. In practical applications, the device memory can be a field programmable gate array (FPGA) or a video memory on a GPU device.

[0064] It should be noted that the host extended memory of the present invention is obtained by expanding the device memory. The present invention can support CXL memory expansion. CXL (Compute Express Link) is a high-speed serial protocol that allows fast and reliable data transmission between different components inside a computer system. It aims to solve bottleneck problems in high-performance computing, including memory capacity, memory bandwidth, and I / O latency. CXL can also achieve memory expansion and memory sharing, and can communicate with peripherals such as computing accelerators to provide faster and more flexible data exchange and processing methods. In addition to CXL, the present invention can also be other ways for memory expansion, which is not limited here.

[0065] Figure 2 The schematic diagram of the framework corresponding to the multimodal data processing method provided in the embodiment of the present invention. Figure 2 As shown, before executing step S101 to extract features from visual data and text data, the process may also include: moving the visual data and text data from the host independent memory to the host extended memory; the host extended memory is managed by the accelerator card. Figure 2 In the figure, acceleration card 1, acceleration card 2, ..., acceleration card N are shown. In the present invention, the central processing unit (CPU) can move the visual data and text data from the host independent memory to the host extended memory, and the host extended memory is managed by the acceleration card. The host independent memory can be a double rate synchronous dynamic random access memory (DDR) memory inserted in the server. The acceleration card can specifically read the visual data and text data into the computing module C 1 Calculation of visual features and text features is performed to obtain visual modality features and text modality features, and then the visual modality features are written into the host extended memory and the text modality features are written into the device memory, respectively.

[0066] S102: Segment the visual modality features to obtain visual feature segments and write them into the device memory.

[0067] In implementation, since visual modal features occupy much more device memory than textual modal features, the present invention segments the visual modal features in step S101 to obtain visual feature segments, and then writes the visual feature segments into the device memory.

[0068] S103: Fusing the text modal features with the visual feature fragments to obtain fused features.

[0069] It should be noted that the purpose of fusing complete text features with visual feature fragments is to avoid the loss of attention of multimodal feature text.

[0070] S104: Process the fused features and write the processed results into the device memory.

[0071] In the above-mentioned multimodal data processing method provided by an embodiment of the present invention, after feature extraction of visual data and text data, the extracted visual modal features are written into the extended memory, and the extracted text modal features are written into the device memory. In this way, the visual modal features and text modal features are stored in different memories respectively, and the visual modal features with larger memory occupancy are stored in the extended memory, which can reduce the video memory occupancy; the complete text modal features are fused with the visual feature fragments obtained after segmentation, which can ensure the correctness of feature alignment and avoid the loss of multimodal feature text attention; the target result output after processing the fused features is written into the device memory to ensure the stability of the data processing result output.

[0072] Furthermore, in a specific implementation, in the above-mentioned multimodal data processing method provided in an embodiment of the present invention, before writing the extracted visual modal features into the host extended memory, it may also include: using a consistency protocol to perform expansion processing on the device memory; the expanded memory is used as the host extended memory; wherein the host extended memory and the device memory are uniformly addressed.

[0073] In implementation, when the consistency protocol adopts the CXL protocol, the CXL extended memory and the device memory are uniformly addressed, and CXL-type2 devices can access the host memory through the CXL protocol.

[0074] Furthermore, in a specific implementation, in the above-mentioned multimodal data processing method provided in an embodiment of the present invention, step S102 divides the visual modal features to obtain visual feature fragments and writes them into the device memory, which may specifically include: first, performing block processing on the visual modal features to obtain multiple visual feature blocks; then, performing slicing processing on each visual feature block to obtain multiple visual feature fragments, and using a consistency protocol to write the visual feature fragments into the device memory.

[0075] In the implementation, the visual modality features are first processed into blocks to obtain multiple visual feature blocks; Figure 2 As shown, visual feature block 1, visual feature block 2, visual feature block 3, ..., visual feature block T can be obtained; then each visual feature block is sliced, for example, visual feature fragment 2-1 can be obtained; finally, the accelerator card 1 can write the visual feature fragment into the device memory through a consistency protocol (such as the CXL protocol).

[0076] Furthermore, in the specific implementation, the above steps of dividing the visual modal features into blocks to obtain multiple visual feature blocks may specifically include: dividing the visual modal features into blocks according to the number of channels of the computer expansion bus standard interface equipped by the server to obtain multiple visual feature blocks. In other words, the block criterion is adjusted according to the hardware configuration, wherein the block criterion can be adjusted according to the number of channels of the computer expansion bus standard interface equipped by the server, and the computer expansion bus standard interface can select a high-speed serial computer expansion bus standard (Peripheral Component Interconnect Express, PCIE) interface.

[0077] In the above steps, each visual feature block is sliced ​​to obtain a plurality of visual feature fragments, which may specifically include: each visual feature block is sliced ​​according to the number of heads in the multi-head attention mechanism to obtain a plurality of visual feature fragments.

[0078] In implementation, the sharding criterion is adjusted according to the software algorithm parameters, where the sharding criterion can be adjusted according to the number of heads in the Transformer multi-head attention mechanism.

[0079] It should be noted that the traditional feature segmentation and block division belong to the hyperparameter setting, which is difficult to coordinate with the software and hardware configuration. However, the present invention can adjust the block division criteria according to the hard disk configuration and adjust the fragmentation criteria according to the software algorithm parameters to obtain the visual feature fragments more accurately.

[0080] Furthermore, in a specific implementation, in the above-mentioned multimodal data processing method provided in an embodiment of the present invention, step S103 fuses the text modal features with the visual feature fragments to obtain the fused features, which may specifically include: loading the weights for calculating the attention mechanism into the device memory; the weights for calculating the attention mechanism include the query Q weights, the key K weights and the value V weights; fusing the text modal features with the visual feature fragments to obtain the multimodal feature input fragments; calculating the weights for calculating the attention mechanism and the multimodal feature input fragments to obtain the query Q matrix, the key K matrix and the value V matrix.

[0081] In the implementation, before fusing the text modality features with the visual feature segments, e.g. Figure 2 As shown, the query Q weight, key K weight and value V weight can also be loaded in the device memory in advance, the complete text modal feature and visual feature fragments are concatenated as the multimodal feature input fragment X, and the query Q weight, key K weight and value V weight are calculated with the multimodal feature input fragment X to obtain the query Q matrix, key K matrix and value V matrix.

[0082] Furthermore, in a specific implementation, in the above-mentioned multimodal data processing method provided in an embodiment of the present invention, step S104 processes the fused features, which may specifically include: segmenting the fused features and writing the segmentation results into an on-chip static memory; calculating an intermediate result based on the segmentation result and writing it into the on-chip static memory.

[0083] During implementation, the fused features are segmented, and the intermediate results are output by the computing module and stored in the on-chip static memory, in order to reduce the reading, writing and interactive operations of the device memory.

[0084] Further, in a specific implementation, in the above-mentioned multimodal data processing method provided by an embodiment of the present invention, the fused features are segmented in the above-mentioned steps, the segmentation results are written into an on-chip static memory, and an intermediate result is calculated based on the segmentation results and written into the on-chip static memory. Specifically, it may include: segmenting the query Q matrix, the key K matrix and the value V matrix to obtain query Q matrix fragments, key K matrix fragments and value V matrix fragments and write them into the on-chip static memory; while segmenting, maintaining the slice position index and slice dimension; using the slice position index, slice dimension, scaling coefficient and translation coefficient to obtain the position code of the query matrix fragment; according to the position code of the query matrix fragment, the query matrix fragment and the key matrix fragment, a first intermediate result is calculated and written into the on-chip static memory.

[0085] In implementation, the query Q matrix, key K matrix and value V matrix can be segmented according to the segmentation method of the visual features to obtain query Q matrix fragments, key K matrix fragments and value V matrix fragments and write them into the on-chip static memory. The query Q matrix fragments may include Q 1 , Q 2 ,…,Q n , the key K matrix fragment can include K 1 , K 2 …、K n , the value V matrix fragment can include V 1 、V 2 ,…,V n , where n is the total number of matrix fragments.

[0086] It should be noted that the calculation module of the present invention can maintain two constants, the slice position index pos and the slice dimension d, in the register when calculating the query Q and key K attention to ensure the integrity of the attention. The maintenance here can be understood as the need to always pay attention to and update, so each step needs to be recorded and stored in a specific memory address, that is, each step of the calculation can determine whether there is a change, and record and update it. Then, using the slice position index, slice dimension, scaling factor and translation coefficient, the position code of the query Q matrix fragment is obtained; according to the position code of the query matrix fragment, the query matrix fragment and the key matrix fragment, the first intermediate result is obtained and written into the on-chip static memory.

[0087] In the present invention, the position code of the query Q matrix fragment can be obtained by using the following formula:

[0088] ;

[0089] ;

[0090] in, Represents the position encoding of the slice (Position Embedding), Represents the feature slice position index, represents the feature slice dimension, Indicates the even-numbered code, Indicates the odd-numbered code, Indicates a constant parameter (needs to be set to a larger integer such as 10000, 20000, etc.), Indicates that the scaling factor is set to an integer greater than 1. represents the translation coefficient. The above formula is based on As input variables, , , , As a constant or a hyperparameter, the above formula can be used to accurately obtain the position encoding of the query Q matrix fragment, and then accurately obtain the corresponding intermediate result.

[0091] Further, in a specific implementation, in the above-mentioned multimodal data processing method provided by an embodiment of the present invention, the above-mentioned steps are based on the position coding of the query matrix fragment, the query matrix fragment and the key matrix fragment, and the first intermediate result is calculated and written into the on-chip static memory, which can specifically include: transposing the key matrix fragment to obtain the transposed key matrix fragment; performing matrix multiplication calculation on the transposed key matrix fragment and the query matrix fragment and adding the position coding of the query matrix fragment to obtain the first intermediate result and write it into the on-chip static memory.

[0092] In implementation, the query matrix fragment Q 1 , key matrix fragment K 1 , value matrix fragment V 1 For example, Figure 2 As shown, the present invention can convert the key matrix fragment K 1 Transpose to get the transposed key matrix fragment K 1 T ; Then transpose the key matrix fragment K 1 T With the query matrix fragment Q 1 Commonly passed into the calculation module C 1 In. By the calculation module C 1 The query matrix fragment Q 1 and transpose key matrix fragment K 1 T Perform matrix multiplication and add query matrix fragment Q 1 The position code is used to obtain the first intermediate result S 1 , and the first intermediate result S 1 Write to the on-chip static memory. It should be noted that the purpose of the calculation of the intermediate results in the present invention is to split the large matrix into small matrices for calculation, output them to another memory address, and finally splice them together.

[0093] In the present invention, the first intermediate result S can be obtained by the following formula: 1 :

[0094] S 1 =Q 1 ·K 1 T +PE.

[0095] Furthermore, in a specific implementation, in the above-mentioned multimodal data processing method provided in an embodiment of the present invention, after obtaining the first intermediate result and writing it into an on-chip static memory, it may also include: recording and maintaining the row maximum value in the first intermediate result; using the row maximum value in the first intermediate result, converting the first intermediate result row by row into a probability distribution vector representing the probability of each category, obtaining a second intermediate result and writing it into the on-chip static memory.

[0096] In implementation, Figure 2 As shown, the present invention calculates the first intermediate result S 1 When the first intermediate result S is recorded and maintained in the register 1 The maximum value m of the row in the first intermediate result S1 is written into the register or constant memory space in the calculation module, and the first intermediate result S1 is continuously passed to the calculation module. 1 Convert row by row into a probability distribution vector (such as softmax) representing the probability of each category to obtain the second intermediate result P 1 The present invention can convert the first intermediate result S 1 Perform softmax calculation row by row to get the second intermediate result P 1 , and the second intermediate result P 1 Write to on-chip static memory.

[0097] Furthermore, in specific implementation, in the above-mentioned multimodal data processing method provided in an embodiment of the present invention, the first intermediate result is converted row by row into a probability distribution vector representing the probability of each category, which may specifically include: calculating the difference between the element value of each row of the first intermediate result and the row maximum value; using the calculated difference between the element value of each row and the row maximum value as an exponential power to perform an exponential operation to obtain an exponent and a constant.

[0098] In implementation, the first intermediate result S can be calculated first. 1 The value of each row of elements in the same row is equal to the maximum value m of the row. 1 The difference between 1 -m 1 Then the difference between the calculated value of each row element and the row maximum value is used as the exponential power for exponential operation, that is, exp(S 1 -m 1 ), and obtain the exponent and the constant f.

[0099] Furthermore, in a specific implementation, in the above-mentioned multimodal data processing method provided in an embodiment of the present invention, after obtaining the second intermediate result and writing it into the on-chip static memory, it may also include: updating and maintaining exponents and constants; obtaining a third intermediate result based on the second intermediate result and the value matrix fragment and writing it into the on-chip static memory; the third intermediate result is a slice in the attention matrix.

[0100] In implementation, Figure 2 As shown, the present invention calculates the second intermediate result P 1 When the exponent and constant f are updated and maintained in the register, the second intermediate result P can be continued. 1 Sum matrix fragment V 1 Pass in the calculation module C 1 According to the second intermediate result P 1 Sum matrix fragment V 1 To get the third intermediate result A 1 And written to the on-chip static memory. Here the third intermediate result A 1 is the first slice of the attention matrix.

[0101] Furthermore, in a specific implementation, in the above-mentioned multimodal data processing method provided in an embodiment of the present invention, the above-mentioned steps of obtaining a third intermediate result based on the second intermediate result and the value matrix fragment and writing it into an on-chip static memory may specifically include: performing matrix multiplication calculation on the second intermediate result and the value matrix fragment to obtain a third intermediate result and writing it into an on-chip static memory.

[0102] In implementation, Figure 2 As shown, the second intermediate result P 1 Sum matrix fragment V 1 A matrix multiplication calculation is performed to obtain a third intermediate result A1, and the third intermediate result A1 is written into the on-chip static memory.

[0103] It should be pointed out that the row maximum value m, exponent and constant f are stored in registers because the registers are closer to the computing module. Generally speaking, the first intermediate result and the second intermediate result are data with very small numbers or few vector elements. If they are placed in other locations farther away from the computing module, the transmission delay will be higher. The advantage of maintaining them in registers is that the computing module can obtain the row maximum value m, exponent and constant f faster, thereby reducing the transmission delay in the overall process and speeding up the computing efficiency in scenarios with frequent data interaction.

[0104] Furthermore, in a specific implementation, in the above-mentioned multimodal data processing method provided in an embodiment of the present invention, step S104 writes the target result obtained by processing into the device memory, and may also include: writing the third intermediate result into the corresponding address of the device memory to write the final target result into the device memory; all slices in the attention matrix are used as the target result.

[0105] In the implementation, after all the third intermediate results are written to the corresponding addresses of the device memory, it indicates that all slices of the attention matrix have been stored in the device memory, which means that the target result A has been written into the device memory. The target result A can be transmitted to other distributed devices for subsequent collective communication. Figure 2 As shown, the target result A is transmitted to the acceleration card 2, and the target result finally obtained by the acceleration card 2 may be A+B. The acceleration card 2 may transmit the target result A+B to the next acceleration card.

[0106] In the above embodiments, the multimodal data processing method is described in detail, and the present invention also provides embodiments corresponding to the multimodal data processing device and the multimodal data processing equipment. It should be noted that the present invention describes the embodiments of the device part from two perspectives, one is based on the functional module perspective, and the other is based on the hardware perspective.

[0107] Figure 3 A schematic diagram of the structure of a multimodal data processing device provided by an embodiment of the present invention. Based on the perspective of functional modules, the device of this embodiment includes:

[0108] A feature extraction and writing module 10 is used to extract features from visual data and text data, write the extracted visual modality features into the host extended memory, and write the extracted text modality features into the device memory;

[0109] A feature segmentation and writing module 11 is used to segment the visual modality features, obtain visual feature segments and write them into the device memory;

[0110] A feature fusion processing module 12 is used to fuse the text modality feature with the visual feature fragment to obtain a fused feature;

[0111] The target result writing module 13 is used to process the fused features and write the processed target result into the device memory.

[0112] In the above-mentioned multimodal data processing device provided by the embodiment of the present invention, the visual modal features and the text modal features can be stored in different memories respectively through the feature extraction writing module 10, and the visual modal features with larger memory occupancy can be stored in the extended memory to reduce the video memory occupancy; through the interaction of the feature segmentation writing module 11 and the feature fusion processing module 12, the complete text modal features can be fused with the visual feature fragments obtained after segmentation, which can ensure the correctness of feature alignment and avoid the loss of multimodal feature text attention; and the purpose result writing module 13 writes the purpose result output after the fused features are processed into the device memory to ensure the stability of the data processing result output.

[0113] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part refer to the description of the embodiments of the method part, which will not be described here. And it has the same beneficial effects as the multimodal data processing method mentioned above.

[0114] Furthermore, in a specific implementation, the multimodal data processing device provided in the embodiment of the present invention may further include:

[0115] The memory expansion module is used to expand the device memory using a consistency protocol; the expanded memory is used as the host expansion memory; wherein the host expansion memory and the device memory are uniformly addressed.

[0116] Furthermore, in a specific implementation, in the multimodal data processing device provided in the embodiment of the present invention, the feature segmentation and writing module 11 may specifically include:

[0117] A feature block unit, used for performing block processing on the visual modality feature to obtain a plurality of visual feature blocks;

[0118] The feature slicing unit is used to slice each visual feature block to obtain multiple visual feature fragments, and write the visual feature fragments into the device memory using a consistency protocol.

[0119] The feature block division unit can be used to divide the visual modality features into blocks according to the number of channels of the computer expansion bus standard interface equipped with the server to obtain multiple visual feature blocks. The feature slicing unit can be used to slice each visual feature block according to the number of heads in the multi-head attention mechanism to obtain multiple visual feature fragments.

[0120] Furthermore, in a specific implementation, in the above-mentioned multimodal data processing device provided in an embodiment of the present invention, the feature fusion processing module 12 can be specifically used to load weights for calculating the attention mechanism in the device memory; the weights for calculating the attention mechanism include query weights, key weights and value weights; the text modal features are fused with the visual feature fragments to obtain a multimodal feature input fragment; the weights for calculating the attention mechanism and the multimodal feature input fragments are calculated to obtain a query matrix, a key matrix and a value matrix.

[0121] Furthermore, in a specific implementation, in the above-mentioned multimodal data processing device provided in an embodiment of the present invention, the target result writing module 13 can be used to segment the fused features and write the segmentation results into an on-chip static memory; an intermediate result is calculated based on the segmentation results and written into the on-chip static memory.

[0122] Furthermore, in a specific implementation, in the above-mentioned multimodal data processing device provided in an embodiment of the present invention, the target result writing module 13 includes:

[0123] The first processing unit can be used to slice the query matrix, the key matrix and the value matrix to obtain query matrix fragments, key matrix fragments and value matrix fragments and write them into the on-chip static memory; while slicing, maintain the slice position index and slice dimension; use the slice position index, slice dimension, scaling coefficient and translation coefficient to obtain the position code of the query matrix fragment; according to the position code of the query matrix fragment, query the matrix fragment and the key matrix fragment, obtain the first intermediate result and write it into the on-chip static memory.

[0124] Among them, the first processing unit can transpose the key matrix fragment to obtain a transposed key matrix fragment; perform matrix multiplication calculation on the transposed key matrix fragment and the query matrix fragment and add the position code of the query matrix fragment to obtain a first intermediate result and write it into the on-chip static memory.

[0125] Furthermore, in a specific implementation, in the above-mentioned multimodal data processing device provided in the embodiment of the present invention, the target result writing module 13 may also include:

[0126] The second processing unit can be used to record and maintain the row maximum value in the first intermediate result; using the row maximum value in the first intermediate result, the first intermediate result is converted row by row into a probability distribution vector representing the probability of each category, and the second intermediate result is obtained and written into the on-chip static memory.

[0127] Among them, the second processing unit can first calculate the difference between each row element value of the first intermediate result and the row maximum value; then use the calculated difference between each row element value and the row maximum value as the exponential power to perform exponential operation to obtain the exponent and constant.

[0128] Furthermore, in a specific implementation, in the multimodal data processing device provided in the embodiment of the present invention, the target result writing module 13 may further include:

[0129] The third processing unit can be used to update and maintain the exponents and constants; obtain the third intermediate result according to the second intermediate result and the value matrix fragment and write it into the on-chip static memory; the third intermediate result is a slice in the attention matrix.

[0130] The third processing unit may perform matrix multiplication on the second intermediate result and the value matrix fragment to obtain a third intermediate result and write it into the on-chip static memory.

[0131] Furthermore, in a specific implementation, in the above-mentioned multimodal data processing device provided in the embodiment of the present invention, the target result writing module 13 may also include:

[0132] The result writing unit can be used to write the third intermediate result into the corresponding address of the device memory so as to write the final target result into the device memory; all slices in the attention matrix are used as the target results.

[0133] Figure 4 Schematic diagram of the structure of a multimodal data processing device provided by an embodiment of the present invention. This embodiment is based on the hardware perspective, such as Figure 4 As shown, the multimodal data processing device includes:

[0134] A memory 20, for storing computer programs;

[0135] The processor 21 is used to implement the steps of the multimodal data processing method mentioned in the above embodiment when executing the computer program.

[0136] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU; the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0137] The memory 20 may include one or more non-volatile storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the multimodal data processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the multimodal data processing method mentioned above.

[0138] In some embodiments, the multimodal data processing device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26. Those skilled in the art will appreciate that Figure 4 The structure shown in does not constitute a limitation on the multimodal data processing device, and may include more or fewer components than shown in the figure. The multimodal data processing device provided by the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the following method: a multimodal data processing method, the effect is the same as above.

[0139] Finally, the present invention also provides an embodiment corresponding to a non-volatile storage medium. The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.

[0140] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code. The non-volatile storage medium provided by the present invention can realize the multimodal data processing method mentioned above, and the effect is the same as above.

[0141] Finally, the present invention also provides an embodiment corresponding to a computer program product. The computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps described in the above multimodal data processing method embodiment are implemented. The computer program product provided by the present invention can implement the above-mentioned multimodal data processing method, and the effect is the same as above.

[0142] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0143] The above is a detailed introduction to the multimodal data processing method, device, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can refer to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A multimodal data processing method, characterized in that: The method comprises: Feature extraction is performed on visual data and text data, and a consistency protocol is used to expand the device memory; the expanded memory is used as the host extended memory; wherein the host extended memory and the device memory are uniformly addressed; Writing the extracted visual modality features into the host extended memory, and writing the extracted text modality features into the device memory; The visual modality feature is processed in blocks according to the number of channels of the computer expansion bus standard interface equipped on the server to obtain a plurality of visual feature blocks; Slicing each of the visual feature blocks according to the number of heads in the multi-head attention mechanism to obtain a plurality of visual feature fragments, and writing the visual feature fragments into the device memory using a consistency protocol; Fusing the text modal feature with the visual feature fragment to obtain a fused feature; The fused features are processed, and the processed results are written into the device memory.

2. The multimodal data processing method according to claim 1, characterized in that: The text modality feature is fused with the visual feature fragment to obtain a fused feature, including: Loading weights for calculating the attention mechanism in the device memory; the weights for calculating the attention mechanism include query weights, key weights, and value weights; Fusion the text modal feature with the visual feature segment to obtain a multimodal feature input segment; The weights used to calculate the attention mechanism and the multimodal feature input fragments are calculated to obtain a query matrix, a key matrix, and a value matrix.

3. The multimodal data processing method according to claim 2, characterized in that: Process the fused features, including: Segment the fused features and write the segmentation results into the on-chip static memory; An intermediate result is calculated based on the segmentation result and written into the on-chip static memory.

4. The multimodal data processing method according to claim 3, characterized in that: Segmenting the fused features, writing the segmentation results into an on-chip static memory, and calculating intermediate results based on the segmentation results and writing them into the on-chip static memory, including: The query matrix, key matrix and value matrix are split to obtain query matrix fragments, key matrix fragments and value matrix fragments and written into the on-chip static memory; while splitting, the slice position index and slice dimension are maintained; Obtaining a position encoding of a query matrix segment using the slice position index, the slice dimension, the scaling factor and the translation factor; According to the position code of the query matrix fragment, the query matrix fragment and the key matrix fragment, a first intermediate result is calculated and written into the on-chip static memory.

5. The multimodal data processing method according to claim 4, characterized in that: According to the position code of the query matrix fragment, the query matrix fragment and the key matrix fragment, a first intermediate result is calculated and written into the on-chip static memory, including: Transposing the key matrix fragment to obtain a transposed key matrix fragment; A matrix multiplication calculation is performed on the transposed key matrix fragment and the query matrix fragment and a position code of the query matrix fragment is added to obtain a first intermediate result and write the result into the on-chip static memory.

6. The multimodal data processing method according to claim 5, characterized in that: After obtaining the first intermediate result and writing it into the on-chip static memory, the method further includes: Record and maintain the row maximum value in the first intermediate result; The first intermediate result is converted row by row into a probability distribution vector representing the probability of each category by using the row maximum value in the first intermediate result, and a second intermediate result is obtained and written into the on-chip static memory.

7. The multimodal data processing method according to claim 6, characterized in that: Converting the first intermediate result row by row into a probability distribution vector representing the probability of each category, including: Calculate the difference between each row element value of the first intermediate result and the row maximum value; The calculated difference between the element value of each row and the row maximum value is used as an exponential power for exponential operation to obtain an exponent and a constant.

8. The multimodal data processing method according to claim 7, characterized in that: After obtaining the second intermediate result and writing it into the on-chip static memory, the method further includes: updating and maintaining said indices and constants; Perform matrix multiplication on the second intermediate result and the value matrix fragment to obtain a third intermediate result and write it into the on-chip static memory; the third intermediate result is a slice in the attention matrix.

9. The multimodal data processing method according to claim 8, characterized in that: Writing the processed result into the device memory includes: The third intermediate result is written into the corresponding address of the device memory to write the final target result into the device memory; all slices in the attention matrix are used as the target result.

10. A multimodal data processing device, characterized in that: The device comprises: Memory for storing computer programs; A processor, configured to implement the steps of the multimodal data processing method as claimed in any one of claims 1 to 9 when executing the computer program.

11. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multimodal data processing method according to any one of claims 1 to 9 are implemented.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the multimodal data processing method according to any one of claims 1 to 9 are implemented.

Images