Product style trend analysis method and system driven by multi-modal neural network

Abstract

The invention discloses a multi-modal neural network-driven product style trend analysis method and system, and relates to the technical field of image processing, and the method comprises the steps: receiving product multi-modal data, including text, image and audio data, carrying out the tensor extraction of the product multi-modal data, enabling each modal data to correspond to a single-peak tensor, and carrying out the tensor extraction of the product multi-modal data; carrying out feature fusion on the single-peak tensor to obtain a multi-modal feature tensor, and then carrying out dimension reduction processing; calculating a cross-modal multi-head attention score by using the single-modal tensor based on a language tensor, wherein the cross-modal multi-head attention score is used for weighting single-modal features; fusing weighted features, dimensionality reduction tensors and manually labeled style labels, constructing a supervised training target, and generating single-mode pseudo labels for multi-task model training; cross-modal shared information capture is carried out on the unimodal tensor based on a covariance matrix, cross-modal shared information is constrained through modal alignment loss, and under a multi-task learning framework, a trained multi-task model takes output of a multi-modal main task as a final style label to realize product trend analysis.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a multimodal neural network-driven method and system for analyzing product style trends. Background Technology

[0002] The manufacturing industry is undergoing a fundamental transformation from "automation-driven" to "human-centered intelligent collaboration." Under this trend, the fashion industry is also experiencing a profound shift from "product-centric" to "consumer-centric," with personalized aesthetics and immediate style responses becoming core industry concerns. Simultaneously, clothing design and consumer decisions are increasingly influenced by multi-source information such as social media, film and television entertainment, and cultural sentiment, resulting in diverse, dynamic, and fine-grained evolution of style preferences. In this context, fashion shows, as a crucial indicator of global style trends, showcase designers' latest works and trend interpretations, serving as a leading platform for disseminating style trends. Fashion shows not only provide high-quality image data (such as clothing styles and model movements) but also rich textual information (such as design concepts and media reviews) and audio content (such as music rhythms and gait rhythms), creating a highly complex multimodal data scenario. This data contains important clues about consumer aesthetic preferences and market trends. However, traditional style trend analysis methods rely heavily on human experience and expert judgment, making them ill-suited for handling large-scale, heterogeneous, and dynamically changing multimodal data. They exhibit significant shortcomings, particularly in areas such as missing style attribute annotations, coarse intermodal fusion, and insufficient high-order semantic representation. While existing data-driven methods have made some progress in image recognition and text mining, most have failed to effectively integrate modalities such as audio and lack the ability to model the micro-evolution of trends and fine-grained style representation. Summary of the Invention

[0003] To address the shortcomings mentioned in the background section, the present invention aims to provide a product style trend analysis method and system driven by a multimodal neural network.

[0004] Firstly, the objective of this invention can be achieved through the following technical solution: a product style trend analysis method driven by a multimodal neural network, the method comprising the following steps:

[0005] Receive product multimodal data, extract tensors from the product multimodal data, each modality data corresponds to three unimodal tensors, fuse unimodal features of the unimodal tensors to obtain multimodal feature tensors, and perform data dimensionality reduction on the multimodal feature tensors. The product multimodal data includes text data, image data and audio data.

[0006] The unimodal tensor is used to calculate the cross-modal multi-head attention score based on the language tensor. The weighted unimodal features are calculated based on the cross-modal multi-head attention score. The weighted unimodal features, the dimensionality-reduced multimodal fusion tensor, and the manually annotated multimodal style labels are combined to construct a supervised training objective, generate unimodal pseudo-labels, and use the unimodal pseudo-labels to train a multi-task model.

[0007] Based on the covariance matrix, cross-modal shared information is captured from the extracted unimodal tensor. The cross-modal shared information is constrained by the modal alignment loss. Under the multi-task learning framework, the trained multi-task model uses the output of the multimodal master task as the final style label to realize product trend analysis.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the text data originates from runway descriptions, designer reviews, and fashion reports as semantically rich natural language information; the BEPT model is used to model the text; and after segmenting, positionally encoding, and multi-layer Transformer encoding the input text sequence, the text feature tensor T representing the semantic core is extracted. t ;

[0009] Image data includes images of clothing and scene photographs from the fashion show. ResNet-50 was used as the backbone network, and visual features were extracted through a pre-trained model. After removing the final fully connected classification layer, the high-dimensional visual features T after global average pooling were retained. v , as a unimodal representation of image modes;

[0010] The audio data includes background music, rhythm, and sound effects from the show. The VGGish network model is used for audio representation, converting the audio signal into a Log-Mel spectrogram, which is then input into the pre-trained VGGish network model to extract a 128-dimensional audio embedding vector T. a .

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of fusing single-modal features of the single-peak tensor.

[0012] By inputting unimodal features into a high-dimensional space for fusion, multiple unimodal representations are fused using a triple Cartesian product, and the interactions between bimodal and trimodal representations are captured through multi-layer fusion, as shown below:

[0013]

[0014] Where T t T v T aLet t, v, and a represent text, image, and audio, respectively. The implementation of generating unimodal and bimodal dynamics is as follows:

[0015]

[0016] in Let represent the outer product between vectors, where m, t, v, and a represent multimodal, text, image, and audio features, respectively.

[0017] Tensor fusion is performed using a triple Cartesian product, considering different factors at different fusion stages: In the first stage, a one-dimensional tensor T is used. t T v and T a To learn the internal representation of each modality; in the second stage, using We learn the interaction information between each pair of modes. In the third stage, we integrate the results from the previous step to obtain the final result.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: performing data dimensionality reduction of the multimodal feature tensor via GPU Tucker decomposition, including:

[0019] According to {P n The factor matrix is ​​partitioned, and a set of partitioned data blocks is generated for each factor matrix. δ is calculated using a GPU-based data pipeline. ( n ) This includes: GPUTucker copies the tensor block, core tensor, and partitioned sub-factor matrix as input data to the GPU memory, and calls the GPU kernel function K. delta renew The updated Copying to main memory, GPUTucker updates A using the updated δ(n). (n) The updated A (n) The nth factor matrix A is stored in main memory according to the row-oriented update rule. (n) (i n The row-wise update rule for the i-th row of :) is expressed as:

[0020]

[0021] in: It is of length J n The vector,

[0022]

[0023] It is a J n ×J nThe matrix,

[0024]

[0025] It is of length J n vector

[0026]

[0027] The regularization parameter is denoted as λ, J n ×J n The same matrix is ​​represented as I, tensor The non-zero coordinates are denoted as (i1,...,i N ), in tensor In the diagram, the coordinates of the nth dimension index are denoted as i. n (i n ∈{i1,...,i n ,...,i N}).

[0028] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of calculating a cross-modal multi-head attention score from the dimensionality-reduced multimodal feature tensor based on a language tensor, and calculating a weighted unimodal feature based on the cross-modal multi-head attention score, is performed using LGT:

[0029] Feature extraction is performed based on the Transformer method, selecting text features as the main component and image and audio features as secondary components;

[0030] Transformer's MHA uses different subspace representations of queries, keys, and values ​​to capture data dependencies; features of each modality are divided into query Q, key-value K, and value V vectors, and an MHA score is calculated. LGT uses language Q to achieve multimodal information fusion for all three modalities from the perspective of cross-modal interaction. The MHA layer is calculated as follows:

[0031]

[0032] head i =Attention(Q) t W Q ,K t W K V t W V (11)

[0033] F t =MHA(Q t ,K t V t= Concat(head1,…,head) h W O (12)

[0034] The feature vector T t The similarity weights are calculated by dividing the vectors into three vectors: Q, K, and V. Then, a linear transformation is applied to each vector. The Q and K vectors are used in the dot product function and the softmax function to calculate the similarity weights. Finally, the weighted sum of vector V is obtained based on the calculation results. The attention calculation is performed iteratively in the MHA layer. Each calculation is treated as a separate head. The final result is obtained by concatenating the outputs of multiple heads.

[0035] Residual transformation and layer normalization are performed on the outputs of the MHA and FFN layers:

[0036] LayerNorm(x+Sublayer(x))(13)

[0037] The text modality is used to guide the calculation of the MHA score for the secondary components. The Q vector is derived from the text modality, while the K and V vectors are derived from the image and audio modalities, respectively.

[0038] F v =Guided-MHA(Q) t ,K v V v ) = Concat

[0039] (head1,…,head h W O (14)

[0040] F a =Guided-MHA(Q) t ,K a V a ) = Concat

[0041] (head1,…,head h W O (15).

[0042] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of capturing cross-modal shared information based on the covariance matrix of weighted single-modal features, comprising:

[0043] The matrix representing a single batch feature in an audio modality is represented as:

[0044] M a ={Feature_a i}, i = 1, 2, ..., N a (16)

[0045] The matrix representing a single batch feature in an image modality is represented as:

[0046] M v ={Feature_v i}, i = 1, 2, ..., N v (17)

[0047] Construct the covariance matrices for the audio and image modes:

[0048]

[0049] The gradient is calculated using the chain rule, and backpropagation is used to update the model's parameters and fine-tune the shared information learning module.

[0050]

[0051] The shared information learning module minimizes a specific function:

[0052]

[0053] Prove existence: Let ε + Represents the Moore-Penrose pseudoinverse of ε. and Let C represent the covariance matrix respectively. a and C v The rank of a is defined as C. a A linear transformation, A T C a A will not increase C. a The rank, therefore, The covariance matrix is ​​a symmetric matrix.

[0054] By applying singular value decomposition (SVD) to the covariance matrices of the two modes, the following results were obtained:

[0055]

[0056] when When the optimal solution is C ' a =C v The formula is:

[0057]

[0058] Where ε v{1∶R} U v{1∶R} These are the maximum singular value of v and its corresponding left singular vector, respectively.

[0059] when When the optimal solution is:

[0060]

[0061] In conclusion,

[0062]

[0063] Combining the above equations, we obtain:

[0064]

[0065]

[0066] The final result is:

[0067]

[0068] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the labels of the multimodal task and the three unimodal tasks are generated via SSLGM, which is based on the following two key mechanisms:

[0069] The public semantic style feature space is consistent with the label space: fused feature F m eigenvectors and unimodal features F s Distance between feature vectors The semantic or categorical distance between them The two modalities are directly proportional; the labels of the two modalities correspond to the two modalities in the label space.

[0070] The common semantic style feature space is related to the UCL space: the distance within the feature style space. Feature fusion F in contrastive learning with unsupervised learning m and unimodal characteristic F s relative positions Matching;

[0071] The design philosophy of SSLGM can be summarized as follows:

[0072]

[0073] Where s∈{t,v,a}], the unimodal label Y is determined by measuring the distance from the multimodal feature to each unimodal feature. s Compared to multimodal label Y m The deviation;

[0074] Offset size: Any value greater than the maximum distance Cut off as

[0075] in and They are respectively and The average value, ||·||2 is L2 normalized;

[0076] Consider the following relationship to calculate the offset magnitude from multimodal to unimodal labels:

[0077]

[0078] Under current conditions, single-peak label Y s as follows:

[0079]

[0080] set up In order to establish With Y s Y m The connections between them determine the following two relationships:

[0081]

[0082] Based on the above relationships, the unimodal label Y under this condition is obtained by summing using equal weights. s :

[0083]

[0084] Secondly, in order to achieve the above objectives, this invention discloses a multimodal neural network-driven product style trend analysis system, comprising:

[0085] The feature extraction module is used to receive product multimodal data, extract tensors from the product multimodal data, with each modality data corresponding to a single-peak tensor, fuse the single-peak tensors with single-modal features to obtain a multimodal feature tensor, and perform data dimensionality reduction on the multimodal feature tensor. The product multimodal data includes text data, image data, and audio data.

[0086] The feature processing module is used to calculate the cross-modal multi-head attention score from the unimodal tensor based on the language tensor, calculate the weighted unimodal features based on the cross-modal multi-head attention score, and combine the weighted unimodal features, the dimensionality-reduced multimodal fusion tensor and the manually annotated multimodal style labels to construct the supervised training objective, generate unimodal pseudo-labels, and use the unimodal pseudo-labels to train the multi-task model.

[0087] The trend analysis module is used to capture cross-modal shared information of the single-peak tensor after tensor extraction based on the covariance matrix. It constrains cross-modal shared information through modal alignment loss. Under the multi-task learning framework, the trained multi-task model uses the output of the multi-modal main task as the final style label to realize product trend analysis.

[0088] In another aspect of the present invention, in order to achieve the above-mentioned objective, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor. When the processor loads and executes the computer program, it employs a product style trend analysis method driven by a multimodal neural network as described above.

[0089] In another aspect of the present invention, in order to achieve the above-mentioned objective, a computer-readable storage medium is disclosed, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is loaded and executed by a processor, it employs a product style trend analysis method driven by a multimodal neural network as described above.

[0090] The beneficial effects of this invention are:

[0091] This invention significantly improves the accuracy, efficiency, and deployability of style recognition and trend prediction, providing solid support for personalized services and dynamic trend response in intelligent design platforms, and has significant technological advancements and industrialization prospects. Attached Figure Description

[0092] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0093] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0094] Figure 2 This is a schematic diagram of the overall structure of the present invention;

[0095] Figure 3 This is the update factor matrix A of the present invention. (n) A schematic diagram outlining the GPUTucker workflow;

[0096] Figure 4 This is a schematic diagram illustrating the determination of the offset magnitude in this invention;

[0097] Figure 5 This is a schematic diagram illustrating the determination of the offset direction of the present invention;

[0098] Figure 6 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0099] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0100] Example 1:

[0101] like Figure 1 As shown, a product style trend analysis method driven by a multimodal neural network includes the following steps:

[0102] S101: Receive product multimodal data, extract tensors from the product multimodal data, each modality data corresponds to a single-peak tensor, fuse the single-peak tensors with single-modal features to obtain a multimodal feature tensor, and perform data dimensionality reduction on the multimodal feature tensor. The product multimodal data includes text data, image data and audio data.

[0103] To achieve accurate modeling and deep style trend perception of multimodal data from fashion shows, this invention first designs an efficient multimodal representation learning module to extract features and perform structured encoding on heterogeneous data from text, images, and audio. Specifically, the representation learning module uses the BEPT model to process the text modality, ResNet for the image modality, and VGGish for the audio modality, extracting unimodal tensor representations corresponding to the three modalities as the basic input for subsequent multilayer association mining.

[0104] The text data originates from semantically rich natural language information such as runway descriptions, designer reviews, and fashion reports. This invention employs the BEPT model to model the text, which possesses bidirectional perception of contextual information and is optimized for short text tasks. After segmenting, positionally encoding, and multi-layer Transformer encoding the input text sequence, a text feature tensor T representing the semantic core is extracted. t .

[0105] Image modal data mainly includes clothing images and scene photographs from fashion shows. This invention uses ResNet-50 as the backbone network and extracts visual features with strong style awareness through a pre-trained model. After removing the final fully connected classification layer, the high-dimensional visual features T after global average pooling are retained. v This feature serves as a unimodal representation of the image modality. It not only preserves the detailed structure of the clothing but also possesses good style differentiation capabilities.

[0106] Audio information such as background music, rhythm, and sound effects in a fashion show serves as important signals of mood and style. This invention employs the VGGish model for audio representation. First, the audio signal is converted into a Log-Mel spectrogram, and then input into a pre-trained VGGish network to extract a 128-dimensional audio embedding vector T. a This embedding preserves low-level audio features such as rhythm intensity, stylistic inclination, and timbre emotion, providing crucial support for capturing the auditory dimensions of fashion styles.

[0107] Through the feature extraction processes of the three modalities described above, this invention obtains three sets of structurally unified but semantically heterogeneous unimodal feature tensors: text tensor T. t Image Tensor T v Audio Tensor T a These tensors will serve as fundamental inputs in subsequent Unimodal Feature Fusion Decomposition (UFFD), Language-Based Transformer (LGT), and Deep Modal Shared Information Learning (DMSIL) modules. They will participate in the fusion operation of UFFD, the modality correlation modeling of LGT, and the shared representation learning of DMSIL. Furthermore, a partial deep information sharing mechanism is introduced in the representation learning stage, sharing some low-level parameters within the Transformer encoding structure to achieve joint perception of multimodal input features and co-construction of low-level information, providing structural priors and parameter stability for the overall model's multi-task collaborative training.

[0108] UFFD (Unimodal Feature Fusion Decomposition)

[0109] The single-modal feature fusion decomposition module consists of two parts: tensor fusion and GPUTucker decomposition. It focuses on the processing of multimodal tasks. This invention uses the mathematical model triple Cartesian product to fuse features in order to discover low-level feature correlations between multimodal representations. GPUTucker decomposes the fused feature representations to reduce model parameters and lower the risk of overfitting.

[0110] Tensor Fusion

[0111] This part overcomes the shortcomings of traditional fusion methods, such as multimodal splicing. It employs a method of fusion by inputting single-peak features into a high-dimensional space. This invention uses a triple Cartesian product to fuse multiple single-peak representations and captures the interactions between bimodal and trimodal representations through multi-layer fusion, as shown below:

[0112]

[0113] Where T t Tv T a Let t, v, and a represent three unimodal tensors, respectively, representing text, image, and audio. A constant dimension of 1 is added to generate the unimodal and bimodal dynamics. The specific calculations can be implemented as follows:

[0114]

[0115] in Let represent the outer product between vectors, where m, t, v, and a represent multimodal, text, image, and audio features, respectively. Therefore, the fused features can be obtained by calculating the outer product.

[0116] In summary, tensor fusion using a triple Cartesian product considers different factors at different fusion stages. In the first stage, a one-dimensional tensor T is used. t T v and T a To learn the internal representation of each modality. The second stage uses... The process begins by learning the interaction information between each pair of modalities. In the third stage, the result from the previous step is integrated to obtain the final result. Therefore, this part fully fuses the multimodal data by progressively using the Cartesian product. It first performs primary fusion by combining modalities, and then further fusion by combining three modalities. This step-by-step fusion method, from the perspective of multimodal feature fusion, maximizes the acquisition of interaction information embedded in different modalities.

[0117] GPUTucker breakdown

[0118] This invention introduces GPUTucker, a GPU-based Tucker decomposition method proposed in this invention. First, (1) the overall framework of GPUTucker is outlined; then, (2) the core idea of ​​reducing computational burden and optimizing memory usage through tensor partitioning is proposed; next, (3) how to optimize tensor partitioning between communication cost between the host and GPU device and the execution efficiency of GPU kernel functions is explained; finally, (4) a GPU-based data pipeline design is given, which is specifically built for tensor blocks generated by Tucker decomposition.

[0119] (1) Overview of GPU Tucker

[0120] The purpose of introducing GPUTucker is to alleviate computational burden, optimize memory usage, and reduce the risk of overfitting. Two key technologies are employed: (1) tensor partitioning to enhance scalability; and (2) GPU-based data pipelines to accelerate the decomposition process. By partitioning tensors into smaller blocks, memory usage on the GPU can be reduced, and parallel computation can be performed across multiple GPUs. Furthermore, the computation of tensor products is a crucial step in the Tucker decomposition method, and leveraging the computing power of the GPU can significantly accelerate the computation. In conclusion, GPUTucker provides a scalable and efficient solution for decomposing high-order sparse tensors using modern GPU hardware.

[0121] Figure 3 This demonstrates the GPUTucker update factor matrix A. (n) A process overview. Step 1: Based on {P} n The factor matrix is ​​partitioned, generating a set of partitioned data blocks for each factor matrix. Step 2 implements a GPU-based data pipeline to calculate δ. (n) This step is further subdivided into steps 2-1, 2-2, and 2-3, which are executed iteratively for each tensor block. In step 2-1, GPUTucker copies the tensor block, the core tensor, and the partitioned sub-factor matrix as input data to the GPU memory. Then, in step 2-2, the GPU kernel function K is called. delta renew In steps 2-3, the updated Copy to main memory. In step 3, after all factor matrices have been processed in their partition blocks, GPUTucker uses the updated δ (n) Update A (n) The updated A (n) The nth factor matrix A is stored in main memory according to the row-oriented update rule. (n) (i n The row-wise update rule for the i-th row of :) is expressed as:

[0122]

[0123] in: It is of length J n The vector,

[0124]

[0125] It is a J n ×J n The matrix,

[0126]

[0127] It is of length J n vector

[0128]

[0129] The regularization parameter is denoted as λ, J n ×J n The same matrix is ​​represented as I. Tensor The non-zero coordinates are denoted as (i1,...,i N In tensor In the diagram, the coordinates of the nth dimension index are denoted as i. n (i n ∈{i1,...,i n ,...,i N It is worth noting that, due to the independence of rows in the factor matrix, the row-by-row update rule makes parallel computation of the loss function possible.

[0130] (2) Tensor partitioning technique

[0131] The scalability issues in existing methods are particularly pronounced due to the limited memory capacity of GPU devices compared to main memory. Specifically, since the input tensor is sparse, key considerations include the number of non-zero elements |Ω| and the dimensionality of the input tensor {I}. n The current state-of-the-art GPU-based Tucker decomposition method, GTA, decomposes input tensors... The non-zero elements in the dataset are evenly distributed across multiple GPUs to alleviate the computational burden. However, this method has limitations because GTA only considers intermediate data. Size (which depends on) The number of non-zero elements in the distribution is considered, and the distribution becomes invalid when the number of non-zero elements exceeds the GPU memory capacity. Furthermore, GTA does not consider the factor matrix. Size (which depends on) (Dimension size). Therefore, when dealing with large-scale input tensors... At times, GTA cannot load the factor matrix into the GPU. For example, given a fifth-order input tensor... (All n satisfy I) n =100M and J n =10), the factor matrix will need If a number of billions of values ​​are stored using double-precision floating-point numbers, then the corresponding space would be 40GB.

[0132] This invention proposes a dimension partitioning method for handling N-order tensors with large dimensionality. Dimension partitioning refers to the process of handling N-order tensors given a specific N-order tensor. Dimensional partitioning is achieved through N parameters {P} n|1≤n≤N}, will Divide into non-overlapping subtensors. Each parameter P n This indicates that the input tensor is in a specific I n The number of partitions along the axis must satisfy the condition 1 ≤ n ≤ N. This method involves dividing the input tensor into non-overlapping sub-tensors, which are called tensor blocks. Tensor blocks are non-overlapping sub-tensors obtained through dimension partitioning, where the input tensor... Divided into Each tensor block has a size of 1. And identified by index Where 1≤p n ≤P N Each factor matrix {A} (n)} is divided into several sizes The sub-factor matrix, specifically denoted as... Where 1≤p n ≤P N The partitioning process uses a set of partitioning parameters {P}. n It is performed using the formula |1≤n≤N}.

[0133] (3) Optimization of tensor partitioning

[0134] To optimize the performance of GPU-based large-scale tensor Tucker decomposition, the key lies in observing the optimal set of partitioning parameters {P} that minimizes the communication cost between the host and GPU devices. n Since the PCI-E bus bandwidth between the host and GPU is typically limited to 16 GByte / s, this bandwidth has become a performance bottleneck as GPU computing power increases. Because the total computational load of Tucker decomposition does not change with the dimensionality partitioning result, reducing PCI-E communication costs can directly improve overall performance. To minimize memory requirements and data copying overhead between main memory and GPU memory, this invention employs an exhaustive search method to obtain the optimal parameter set for tensor fractal dimension. Assume that each task can use θ amounts of memory, where a task is defined as a task on the GPU targeting tensor blocks. calculate A series of operations.

[0135] The following formula describes the optimization problem in this case. Where Mem(P1,...,P) N ) indicates the use of parameters (P1,...,P N The memory usage function for each task during the process is Cost(P1,...,P). N ) is the communication overhead function when this parameter is used.

[0136] (7) The following formula describes the function Its composition. Among them... Represents a tensor block. Represents the core tensor. Represents the sub-factor matrix, This represents intermediate data. Considering that the input tensor is a sparse tensor, GPUTucker uses specific tensor blocks... The maximum number of non-zero elements determines the required memory usage.

[0138]

[0139] Where β=(p1,...,p N And p n ∈β.

[0140] The following formula defines the function Cost(). In this formula, |G| and These represent the number of GPU devices and the total number of tensor blocks, respectively. Copying is required for all GPU devices. But only with tensor block indexes corresponding Some data is copied to the GPU device. The term $|\delta^{(n)}|$ represents... A copy needs to be generated for all GPU devices, but only for tensor block indexes. corresponding Some will be copied to the GPU device. Item | δ (n) | represents all intermediate data copied from the GPU device to main memory. The following formula includes the GPU streaming cost.

[0141]

[0142] By solving the optimization problem in the equation, the optimal parameters for dimension partitioning can be obtained. This minimizes the communication overhead between the host and the GPU device.

[0143] (4) GPU-based data pipeline

[0144] This invention describes a GPU-based data pipeline for implementing the tensor partitioning method proposed in (2). It summarizes the overall data processing flow from input tensors to returned results. The first step of GPUTucker is to initialize the core tensors. and all factor matrices A (n) Its elements take the value of a random number between 0 and 1. Then, for each dimension I... n (where 1 ≤ n ≤ N) determines the optimal number of partitions {P1, ..., P}. NThe partition parameters of Tucker decomposition are {P1,...,P}. N The allocation is determined based on memory usage. Next, GPUTucker creates a GPU stream and allocates all available GPU memory. GPUTucker reuses allocated memory by adjusting memory pointers at runtime. Since allocating GPU device memory can be more expensive than main memory, this mechanism of reusing pre-allocated memory effectively reduces runtime overhead.

[0145] This invention describes how to perform Tucker decomposition using a GPU. Core Tensor Copy from host memory to GPU device memory (BufG). Here, β represents the set of valid index tuples corresponding to the independent tensor block (p1,...,p...). N ), where 1≤p n ≤P N Tensor blocks and its corresponding sub-factor matrix Tensors are streamed into the GPU's memory, with each block selected and processed sequentially based on its index. When the amount of input data for the kernel function exceeds the stream buffer capacity, GPUTucker employs a multi-streaming strategy to process the tensor blocks. For example, if If a tensor block contains 20 million non-zero elements, and each stream can process 10 million non-zero elements at a time, GPUTucker will use two streams, each responsible for processing the 10 million non-zero elements in the tensor block. Through this mechanism, GPUTucker fully utilizes multi-stream parallelism and maximizes computational resource utilization, ensuring that all data is processed efficiently by the GPU. After the data copy is complete, GPUTucker calls the GPU kernel function K. delta K delta Function calculates intermediate data To update factor matrix A (n) .when After processing, the result (i.e. The data is streamed back to main memory. Once all tensor blocks have been processed, the GPU synchronizes to terminate kernel function K. delta The execution, and utilizes the δ stored in main memory. (n) Update factor matrix A (n) GPUTucker repeats the above calculation process for the other factor matrices. Once all factor matrices have been updated, the approximate tensor is calculated using the formula. The calculation process will be executed cyclically until the convergence condition is met, and its convergence criterion is consistent with that of Algorithm 1.

[0146] It also describes the GPU kernel function K. delta The function uses tensor blocks Core Tensor and sub-factor matrix As input, calculate intermediate data Each GPU thread is assigned a tensor block to process based on its thread ID (tid). The thread retrieves all sub-factor matrices except the one to be updated. The value of . Then, by each The values ​​are multiplied together to update the variable val. This is used to update the factor matrix A. (n) After calculating the val value, the thread adds the value to the intermediate data buffer. The above process is in tensor blocks. The process is repeated on all non-zero elements. Leveraging the GPU's many-core architecture, multiple non-zero elements can be processed in parallel. Before exiting, the kernel function calls the `synchronizedthreads()` function to ensure thread synchronization, guaranteeing that all threads complete their assigned tasks before continuing with subsequent operations.

[0147] This illustrates the timeline of data transfer between the GPU device and the host, as well as the execution of GPU kernel functions, as shown by GPUTucker. The number of asynchronous streams for each GPU device is denoted as G and M, respectively. By default, this invention assumes G = 1 and M = 4. Assume the factor matrix A... (1) When (i.e., n=1) is updated, the input tensor Press (P1,...,P) N (Divide into blocks.) First, a CPU thread divides the core tensor into blocks. Copy to GPU device memory BufG, and then start multiple GPU streams. Each GPU stream performs the following three operations on each tensor block: (1) Tensor block Where β=(p1,...,p N ), sub-factor matrix (1) Copy to GPU device memory BufX and BufA[k] respectively; (2) Call GPU kernel function K delta (3) Copy intermediate data By dividing data transfer into smaller chunks and processing them in parallel, the use of multi-streaming technology in each GPU reduces the data copying overhead between the device and the host, thereby reducing the overall transfer time. This overlapping operation of data transfer and kernel function computation improves GPU resource utilization. After the M GPU streams have completed execution, the cudaStreamSynchronize() function is called to synchronize all GPU threads, ensuring that the corresponding input tensors are synchronized. The intermediate data is calculated precisely.

[0148] S102: Calculate the cross-modal multi-head attention score based on the language tensor using the single-peak tensor, calculate the weighted single-modal features based on the cross-modal multi-head attention score, combine the weighted single-modal features, the dimensionality-reduced multi-modal fusion tensor and the manually annotated multi-modal style labels to construct a supervised training objective, generate single-modal pseudo-labels, and use the single-modal pseudo-labels to train a multi-task model.

[0149] The process of calculating cross-modal multi-head attention scores from unimodal tensors based on language tensors, and then calculating weighted unimodal features based on these scores, is performed using LGT. This part focuses on the processing of three unimodal tasks. This invention uses a Transformer-based method for feature extraction, which allows for a closer examination and identification of deeper correlations between modal representations. As shown, this invention selects text features as the primary component, and image and audio features as secondary components. Subsequently, the weighted unimodal features, the dimensionality-reduced multimodal fusion tensor, and manually annotated multimodal style labels are combined to construct a supervised training objective, generate unimodal pseudo-labels, and use these pseudo-labels to train a multi-task model.

[0150] LGT (Language-Based Transformer)

[0151] Transformer's MHA uses different subspace representations of queries, keys, and values ​​to capture data dependencies. Features for each modality are divided into query (Q), key-value (K), and value (V) vectors, and an MHA score is calculated. If each modality uses its own Q, K, and V vectors to mine internal data dependencies, intermodal information is ignored. To bridge modalities and discover cross-modal information, LGT uses linguistic Q to achieve multimodal information fusion from the perspective of cross-modal interaction for all three modalities. Traditional Transformer encoders consist of MHA layers and feedforward neural network (FFN) layers; this invention uses FFN layers to process linguistic feature vectors. The MHA layer is calculated as follows:

[0152]

[0153] head i =Attention(Q) t W Q ,K t W K V t W V (11)

[0154] F t =MHA(Q t ,K t V t= Concat(head1,…,head) h W O (12)

[0155] First, the feature vector T t The similarity is divided into three vectors: Q, K, and V. A linear transformation is then applied to each vector. Next, the Q and K vectors are used in a dot product function and a softmax function to calculate the similarity weights. Finally, the weighted sum of vector V is obtained from the results of the previous step. This attention calculation is iterated within the MHA layer, with each calculation treated as a separate "head." The final result is obtained by concatenating the outputs of multiple heads.

[0156] In addition, residual transformation and layer normalization (A and L) are performed on the outputs of each layer (MHA and FFN layers):

[0157] LayerNorm(x+Sublayer(x))(13)

[0158] This invention uses textual modalities to guide the calculation of MHA scores for secondary components. This process is called guided MHA. In guided MHA, the Q vector is derived from the textual modalities, while the K and V vectors are derived from the image and audio modalities, respectively. Textual representations are used to incorporate information from different spatial contexts when learning image and audio representations.

[0159] F v =Guided-MHA(Q) t ,K v V v ) = Concat

[0160] (head1,…,head h W O (14)

[0161] F a =Guided-MHA(Q) t ,K a V a ) = Concat

[0162] (head1,…,head h W O (15)

[0163] SSLGM (Self-Supervised Style Tag Generation)

[0164] The labels for the multimodal task and the three unimodal tasks are generated using SSLGM, which is based on the following two key mechanisms:

[0165] The goal of SSLGM is to generate style-attribute labels for each unimodality by applying multimodal labels and modal representations. This invention extends and optimizes the work of Hwang et al., whose design principle is that the distance between two features in a common semantic feature space is proportional to the distance between corresponding labels in the label space. Based on this concept and combining the characteristics of unsupervised contrastive learning, this invention proposes an unsupervised contrastive learning space (UCL space). In the UCL space, this invention maps data from different modalities to a unified representation space. In this space, if data points have similar attributes, they tend to be close to each other, forming tight clusters, thus reflecting the similarity between data points. Conversely, data points belonging to different categories or with significant differences are mapped to the far end of the space, thus highlighting their differences. In summary, the SSLGM project is based on the following two key assumptions and mechanisms:

[0166] (1) The common semantic style feature space is consistent with the label space: fused feature F m eigenvectors and unimodal features F s Distance between feature vectors The semantic or categorical distance between them The two modal labels are directly proportional, and the two modal labels correspond to the two modalities in the label space.

[0167] (2) The common semantic style feature space is related to the UCL space: the distance within the feature style space Feature fusion F in contrastive learning with unsupervised learning m and unimodal characteristic F s relative positions Matching. In summary, the design philosophy of SSLGM can be summarized as follows:

[0168]

[0169] Where s∈{t,v,a}]. The SSLGM method proposed in this invention determines the unimodal label Y by measuring the distance from the multimodal feature to each unimodal feature. s Compared to multimodal label Y m The deviation amount. In calculating the deviation, this invention focuses on two core elements: amplitude and direction.

[0170] Offset magnitude: To calculate the offset, its specific structure is as follows: Figure 4 This invention posits that the maximum distance within the common semantic style feature space is proportional to the maximum distance within the label space. In the dataset, multimodal labels vary between -3 and +3. This means that the labels are... and The distance between multimodal features must be the largest in the common semantic style feature space. Therefore, any distance greater than the maximum distance... Cut off as

[0171]

[0172] in and They are respectively and The average value, ||·||², is L2 normalized. Based on the concepts and key points mentioned above, this invention can consider the following relationship to calculate the offset magnitude from multimodal to unimodal labels:

[0173]

[0174] Under current conditions, single-peak label Y s The following can be estimated:

[0175]

[0176] For the results of UCL, since their range is broader, a maximum distance consistent with the previous settings needs to be defined. Therefore, this invention sets... In order to establish With Y s Y m Regarding the relationship between them, this invention considers the following two relationships:

[0177]

[0178] Based on the above relationships, the unimodal label Y under this condition is obtained by summing using equal weights. s :

[0179]

[0180] In the formula

[0181] Offset Direction: To determine the offset direction, the spatial position of the unimodal feature relative to the multimodal feature is first analyzed. This process initially involves positive annotation. and negative comments The average value of the multimodal features is obtained as a reference benchmark. Then, referring to this benchmark, the multimodal and unimodal features are located in the feature style space. This is achieved by calculating various types of modal representations (e.g., F...). x∈{m,t,s,v} arrive and The L2 distance can be used to derive and determine the direction of the offset:

[0182]

[0183] in ||·||² represents L2 normalization. Based on the above concepts and key points, this invention can consider the following relationship to calculate the offset from a multi-peaked label to a single-peaked label:

[0184]

[0185] In the formula, α and β are the label space weight coefficient and the UCL space weight coefficient, respectively.

[0186] S103: Based on the covariance matrix, cross-modal shared information is captured from the extracted unimodal tensor. Cross-modal shared information is constrained by modal alignment loss. Under the multi-task learning framework, the trained multi-task model uses the output of the multi-modal main task as the final style label to realize product trend analysis.

[0187] The process of capturing cross-modal shared information based on weighted single-modal features using the covariance matrix, and constraining cross-modal shared information through modal alignment loss, includes:

[0188] Deep modality shared information learning is a valuable component in deep learning systems, facilitating the extraction of shared information from multiple modalities. This module can explore the shared information between any two specified modalities. For image and audio modalities, this invention uses feature representations Feature_v and Feature_a, which, since each training iteration is based on a specific batch of data, correspond to the same batch of data. The number of image modality features (denoted as N) can be calculated. v ) and audio modal features (denoted as N) a Set to the same value (N) v =N a Then, each Feature_v and Feature_a is projected into a low-dimensional space of dimension d.

[0189] This invention represents the matrix representing a single batch feature in an audio modality as follows:

[0190] M a ={Feature_a i}, i = 1, 2, ..., N a (16)

[0191] Similarly, the present invention represents the matrix representing a single batch feature in an image modality as follows:

[0192] M v ={Feature_v i}, i = 1, 2, ..., N v (17)

[0193] Furthermore, this invention constructs covariance matrices for audio and image modalities. To capture and quantify shared information between modalities, as described by Sun et al., a function based on the covariance matrix enables an effective assessment of the degree of shared information between audio and image modalities, promoting a comprehensive understanding of their interactions and correlations.

[0194]

[0195] Constructing a loss function is crucial for enabling multimodal sentiment analysis models to focus on and learn shared information across modalities. By designing an appropriate loss function, this invention guides the model to prioritize and emphasize the extraction and utilization of shared information across different modalities. This allows the model to effectively leverage combined knowledge and insights from multiple modalities, thereby improving performance and sentiment analysis results.

[0196]

[0197] The gradient can be calculated using the chain rule because θ share It is a differentiable function that can be backpropagated through the network. This property enables the present invention to efficiently compute and update gradients during training, allowing the model to learn and optimize cross-modal shared information representations. By utilizing backpropagation, the present invention can efficiently update model parameters and fine-tune the shared information learning module, thereby improving performance and enhancing the integration of multimodal features.

[0198]

[0199] The basic concept of this module is to align multimodal distributions by utilizing second-order statistics between modalities. The main objective of this module is to minimize a specific function. By considering second-order statistics, this module aims to capture the statistical dependencies and relationships between different modalities. This alignment of multimodal distributions facilitates a more coherent and integrated representation of multimodal data, ultimately improving the overall performance of the model on the target task.

[0200]

[0201] Prove existence: Let ε + Represents the Moore-Penrose pseudoinverse of ε. and Let C represent the covariance matrix respectively. a and C v The rank of . This invention defines 'a' as C. a A linear transformation. It is important to note that A T C a A will not increase C. a The rank of . Therefore, the present invention has Moreover, the covariance matrix is ​​a symmetric matrix.

[0202] By applying singular value decomposition (SVD) to the covariance matrices of the two modes, the present invention yields the following results:

[0203]

[0204] when When the optimal solution is C ' a =C v The optimal solution at this point is:

[0205]

[0206] Where ε v{1∶R} U v{1∶R} Let v be the maximum singular value of v and the corresponding left singular vector, respectively.

[0207] when When the optimal solution is:

[0208]

[0209] In conclusion,

[0210]

[0211] This is obtained by combining the above equations.

[0212]

[0213] so:

[0214]

[0215] In summary, based on the preceding analysis and considerations, it is clear that this function has an optimal solution. This proof demonstrates the feasibility of finding an optimal solution consistent with the objectives and requirements of the task at hand. This discovery provides confidence in the effectiveness and reliability of functions that achieve desired optimization objectives within the context of multimodal analysis.

[0216] This study proposes a self-supervised label generation multi-layer association mining framework, aiming to achieve collaborative modeling, deep sharing, and accurate prediction of style semantics across modal information. To this end, a unified optimization objective function is constructed, driven by a multi-modal main task, a unimodal auxiliary task, and a deep modality alignment task. This function guides the model to establish consistent representations across different modalities through multi-task joint training, while simultaneously enhancing the expressive power of each modality at the individual level.

[0217]

[0218] The multimodal main task includes: Single-modal assisted tasks: Modal deep alignment task: Where N is the number of training samples.

[0219] Model performance comparison and effectiveness verification (based on mainstream models)

[0220] To verify the effectiveness of the "multimodal neural network-driven product style trend analysis method" proposed in this invention, this paper selects several general multimodal fusion models for experimental comparison, including:

[0221] Table 1. Experimental Comparison Models and Explanations

[0222]

[0223]

[0224] Experiment setup instructions:

[0225] Dataset used: The FashionShow-Style dataset, which we built ourselves, contains text, image, and audio data about fashion shows, with 52,000 samples (42k for training, 5k for validation, and 5k for testing). Input: Text, image, and audio data about fashion shows. Evaluation metrics: MAE, Corr, and F1-Score. MAE: Mean absolute error between predicted style probability and true label; lower values ​​are better. Corr: Pearson correlation coefficient between predicted style vector and true label; higher values ​​are better. F1-Score (harmonic mean): Harmonic mean of precision and recall; higher values ​​are better.

[0226] Table 2 Quantitative Comparison of Model Performance

[0227] Model Name MAE(↓) Corr(↑) F1 Score(↑) TFN 0.218 0.73 0.67 MulT 0.195 0.78 0.72 Self-MM 0.187 0.81 0.74 CLIP 0.201 0.84 0.70 MCMF 0.172 0.82 0.76 Method of the present invention 0.138 0.93 0.85

[0228] The method of this invention outperforms other models in three metrics: FID, IS, and CLIP. In particular, through UFFD (Unimodal Feature Fusion Decomposition) feature fusion optimization, DMSIL (Deep Modal Shared Information Learning) shared feature enhancement, and SSLGM (Self-Supervised Style Label Generation) label generation upgrade, it achieves a breakthrough improvement in prediction accuracy, trend synchronization, and multi-label classification performance, providing a high-precision, low-latency fashion trend analysis solution for Industry 5.0.

[0229] Example 2: To achieve the above objective, such as Figure 6As shown, based on Embodiment 1, this invention discloses a product style trend analysis method driven by a multimodal neural network, characterized by comprising:

[0230] The feature extraction module 11 receives product multimodal data, extracts tensors from the product multimodal data, with each modality data corresponding to a single-peak tensor, fuses single-modal features of the single-peak tensors to obtain multimodal feature tensors, and performs data dimensionality reduction on the multimodal feature tensors. The product multimodal data includes text data, image data, and audio data.

[0231] Feature processing module 12 is used to calculate cross-modal multi-head attention scores from unimodal tensors based on language tensors, calculate weighted unimodal features based on cross-modal multi-head attention scores, and combine the weighted unimodal features, the dimensionality-reduced multimodal fusion tensor, and manually annotated multimodal style labels to construct supervised training objectives, generate unimodal pseudo-labels, and train multi-task models using unimodal pseudo-labels.

[0232] The trend analysis module 13 is used to capture cross-modal shared information of the single-peak tensor after tensor extraction based on the covariance matrix. By constraining cross-modal shared information through modal alignment loss, under the multi-task learning framework, the trained multi-task model uses the output of the multi-modal main task as the final style label to realize product trend analysis.

[0233] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0234] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0235] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0236] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

Claims

1. A product style trend analysis method driven by a multimodal neural network, characterized in that, The method includes the following steps: The system receives multimodal product data, extracts tensors from the multimodal product data, with each modality data corresponding to a single-peak tensor, fuses single-modal features of the single-peak tensors to obtain multimodal feature tensors, and performs data dimensionality reduction on the multimodal feature tensors. The multimodal product data includes text data, image data, and audio data. The unimodal tensor is used to calculate the cross-modal multi-head attention score based on the language tensor. The weighted unimodal features are calculated based on the cross-modal multi-head attention score. The weighted unimodal features, the dimensionality-reduced multimodal fusion tensor, and the manually annotated multimodal style labels are combined to construct a supervised training objective, generate unimodal pseudo-labels, and use the unimodal pseudo-labels to train a multi-task model. Based on the covariance matrix, cross-modal shared information is captured from the extracted unimodal tensor. The cross-modal shared information is constrained by the modal alignment loss. Under the multi-task learning framework, the trained multi-task model uses the output of the multimodal master task as the final style label to realize product trend analysis.

2. The product style trend analysis method driven by a multimodal neural network according to claim 1, characterized in that, The text data originates from runway descriptions, designer reviews, and fashion reports—semantically rich natural language information. The BEPT model is used to model the text; after segmenting, positional encoding, and multi-layer Transformer encoding of the input text sequence, the text feature tensor T representing the semantic core is extracted. t ; Image data includes images of clothing and scene photographs from the fashion show. ResNet-50 was used as the backbone network, and visual features were extracted through a pre-trained model. After removing the final fully connected classification layer, the high-dimensional visual features T after global average pooling were retained. v , as a unimodal representation of image modes; The audio data includes background music, rhythm, and sound effects from the show. The VGGish network model is used for audio representation, converting the audio signal into a Log-Mel spectrogram, which is then input into the pre-trained VGGish network model to extract a 128-dimensional audio embedding vector T. a .

3. The product style trend analysis method driven by a multimodal neural network according to claim 1, characterized in that, The process of fusing single-modal features from a single-peak tensor: By inputting unimodal features into a high-dimensional space for fusion, multiple unimodal representations are fused using a triple Cartesian product, and the interactions between bimodal and trimodal representations are captured through multi-layer fusion, as shown below: Where T t T v T a Let t, v, and a represent text, image, and audio, respectively. The implementation of generating unimodal and bimodal dynamics is as follows: in Let represent the outer product between vectors, where m, t, v, and a represent multimodal, text, image, and audio features, respectively. Tensor fusion is performed using a triple Cartesian product, considering different factors at different fusion stages: In the first stage, a one-dimensional tensor T is used. t T v and T a To learn the internal representation of each modality; in the second stage, using We learn the interaction information between each pair of modes. In the third stage, we integrate the results from the previous step to obtain the final result.

4. The product style trend analysis method driven by a multimodal neural network according to claim 1, characterized in that, The dimensionality reduction of the multimodal feature tensor is performed using GPU Tucker decomposition, including: According to {P n The factor matrix is ​​partitioned, and a set of partitioned data blocks is generated for each factor matrix. δ is calculated using a GPU-based data pipeline. ( n ) This includes: GPUTucker copies the tensor block, core tensor, and partitioned sub-factor matrix as input data to the GPU memory, and calls the GPU kernel function K. delta renew The updated Copying to main memory, GPUTucker updates A using the updated δ(n). (n) The updated A (n) The nth factor matrix A is stored in main memory according to the row-oriented update rule. (n) (i n The row-wise update rule for the i-th row of :) is expressed as: in: It is of length J n The vector, It is a J n ×J n The matrix, It is of length J n vector The regularization parameter is denoted as λ, J n ×J n The same matrix is ​​represented as I, tensor The non-zero coordinates are denoted as (i1,...,i N ), in tensor In the diagram, the coordinates of the nth dimension index are denoted as i. n (i n ∈{i1,...,i n ,...,i N }).

5. The product style trend analysis method driven by a multimodal neural network according to claim 1, characterized in that, The process of calculating cross-modal multi-head attention scores from the dimensionality-reduced multimodal feature tensors based on language tensors, and then calculating weighted unimodal features based on these cross-modal multi-head attention scores, is performed using LGT. Feature extraction is performed based on the Transformer method, selecting text features as the main component and image and audio features as secondary components; Transformer's MHA uses different subspace representations of queries, keys, and values ​​to capture data dependencies; features of each modality are divided into query Q, key-value K, and value V vectors, and an MHA score is calculated. LGT uses language Q to achieve multimodal information fusion for all three modalities from the perspective of cross-modal interaction. The MHA layer is calculated as follows: head i =Attention(Q t W Q ,K t W K ,V t W V ) (11) F t =MHA(Q t ,K t ,V t )=Concat(head1,…,head h )W O (12) The feature vector T t The similarity weights are calculated by dividing the vectors into three vectors: Q, K, and V. Then, a linear transformation is applied to each vector. The Q and K vectors are used in the dot product function and the softmax function to calculate the similarity weights. Finally, the weighted sum of vector V is obtained based on the calculation results. The attention calculation is performed iteratively in the MHA layer. Each calculation is treated as a separate head. The final result is obtained by concatenating the outputs of multiple heads. Residual transformation and layer normalization are performed on the outputs of the MHA and FFN layers: LayerNorm(x+Sublayer(x))(13) The text modality is used to guide the calculation of the MHA score for the secondary components. The Q vector is derived from the text modality, while the K and V vectors are derived from the image and audio modalities, respectively. F v =Guided-MHA(Q t ,K v ,V v )=Concat (head1,…,head h )W O (14) F a =Guided-MHA(Q t ,K a ,V a )=Concat (head1,…,head h )W O (15)。 6. The product style trend analysis method driven by a multimodal neural network according to claim 1, characterized in that, The process of capturing cross-modal shared information based on weighted single-modal features using the covariance matrix includes: The matrix representing a single batch feature in an audio modality is represented as: M a ={Feature_a i },i=1,2,…,N a (16) The matrix representing a single batch feature in an image modality is represented as: M v ={Feature_v i },i=1,2,…,N v (17) Construct the covariance matrices for the audio and image modes: The gradient is calculated using the chain rule, and backpropagation is used to update the model's parameters and fine-tune the shared information learning module. The shared information learning module minimizes a specific function: Prove existence: Let ε + Represents the Moore-Penrose pseudoinverse of ε. and Let C represent the covariance matrix respectively. a and C v The rank of a is defined as C. a A linear transformation, A T C a A will not increase C. a The rank, therefore, The covariance matrix is ​​a symmetric matrix. By applying singular value decomposition (SVD) to the covariance matrices of the two modes, the following results were obtained: when When the optimal solution is C ' a =C v The formula is: Where ε v{1∶R} U v{1∶R} These are the maximum singular value of v and its corresponding left singular vector, respectively. when When the optimal solution is: In conclusion, Combining the above equations, we obtain: The final result is:

7. The product style trend analysis method driven by a multimodal neural network according to claim 1, characterized in that, The labels for the multimodal task and the three unimodal tasks are generated using SSLGM, which is based on the following two key mechanisms: The public semantic style feature space is consistent with the label space: fused feature F m eigenvectors and unimodal features F s Distance between feature vectors The semantic or categorical distance between them The two modalities are directly proportional; the labels of the two modalities correspond to the two modalities in the label space. The common semantic style feature space is related to the UCL space: the distance within the feature style space. Feature fusion F in contrastive learning with unsupervised learning m and unimodal characteristic F s relative positions Matching; The design philosophy of SSLGM can be summarized as follows: Where s∈{t,v,a}], the unimodal label Y is determined by measuring the distance from the multimodal feature to each unimodal feature. s Compared to multimodal label Y m The deviation; Offset size: Any value greater than the maximum distance Cut off as in and They are respectively and The average value, ||·||2 is L2 normalized; Consider the following relationship to calculate the offset magnitude from multimodal to unimodal labels: Under current conditions, single-peak label Y s as follows: set up In order to establish With Y s Y m The connections between them determine the following two relationships: Based on the above relationships, the unimodal label Y under this condition is obtained by summing using equal weights. s : In the formula 8. A product style trend analysis system driven by a multimodal neural network, employing the product style trend analysis method driven by a multimodal neural network as described in any one of claims 1 to 7, characterized in that, include: The feature extraction module is used to receive product multimodal data, extract tensors from the product multimodal data, with each modality data corresponding to a single-peak tensor, fuse the single-peak tensors with single-modal features to obtain a multimodal feature tensor, and perform data dimensionality reduction on the multimodal feature tensor. The product multimodal data includes text data, image data, and audio data. The feature processing module is used to calculate the cross-modal multi-head attention score from the unimodal tensor based on the language tensor, calculate the weighted unimodal features based on the cross-modal multi-head attention score, and combine the weighted unimodal features, the dimensionality-reduced multimodal fusion tensor and the manually annotated multimodal style labels to construct the supervised training objective, generate unimodal pseudo-labels, and use the unimodal pseudo-labels to train the multi-task model. The trend analysis module is used to capture cross-modal shared information of the single-peak tensor after tensor extraction based on the covariance matrix. It constrains cross-modal shared information through modal alignment loss. Under the multi-task learning framework, the trained multi-task model uses the output of the multi-modal main task as the final style label to realize product trend analysis.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The memory stores a computer program that can run on a processor. When the processor loads and executes the computer program, it employs a product style trend analysis method driven by a multimodal neural network as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is loaded and executed by the processor, it employs a product style trend analysis method driven by a multimodal neural network as described in any one of claims 1 to 7.

Images