A Digital Odor Generation Method and System Based on Multimodal Feature Fusion

Abstract

This invention discloses a digital odor generation method and system based on multimodal feature fusion, belonging to the fields of artificial intelligence and digital odor generation. In this method, multimodal raw data such as images, videos, audio, and text are first encoded, normalized, and feature-aligned, mapping them into multimodal representation vectors in a unified semantic space. Subsequently, this vector is input into a digital odor generation model. Through multimodal fusion, generation, and decoding, a digital odor spectral domain vector is output. Non-negative least squares and multi-objective optimization are then used to invert this vector into a digital odor formula that conforms to actual blending characteristics, driving a digital odor generation device to present the odor. Simultaneously, relevant data is stored in a database for subsequent retrieval and model optimization. This method achieves end-to-end generation from multimodal semantic input to a feasible solution, effectively improving the personalization and practicality of odor generation.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and digital odor generation, and particularly relates to a digital odor generation method and system based on multimodal feature fusion. Background Technology

[0002] With the development of intelligent interaction and Internet of Things (IoT) technologies, human-computer interaction has gradually expanded from being primarily visual and auditory to encompassing multiple senses, including touch and smell. Among these, digital odor technology, as an important component for achieving immersive experiences, has gradually become a research hotspot in academic and industrial circles both domestically and internationally.

[0003] In the automotive industry, in-car fragrance systems have been increasingly used to enhance driving comfort. However, most existing in-car fragrance systems are based on fixed scents or simple manual controls, lacking effective integration with in-car interactive content. Especially in digital scent technology, existing solutions often rely on environmental monitoring or driving status detection to trigger digital scent output, failing to fully utilize user input (such as text, video, voice, and audio) as driving signals. This results in shortcomings in the matching of content with digital scents and the personalization of the experience.

[0004] Most existing digital scent generation methods are based on single-modal input. For example, one type of method utilizes the correlation between chemical molecular structure and human olfactory perception to build a predictive model to infer the digital scent attributes corresponding to the molecules; another type relies on digital scent tags or keywords to retrieve fragrance formulas similar to the user's input from a predefined scent database. While these methods achieve digital scent prediction and generation to some extent, their expressive power remains limited, often failing to meet the user's real needs, especially since relying solely on text tags makes it even more difficult to reflect the user's specific scenario or emotional state. Furthermore, these methods are insufficient in terms of personalization, typically based on fixed mappings or simple database searches, lacking adaptive adjustments to different user preferences and historical feedback. At the same time, the generated results also lack flexibility and scalability. When the input involves multimodal information such as text, video, audio, or images, traditional methods struggle to effectively integrate this data and generate reasonable digital scent formulas.

[0005] On the other hand, some research has attempted to introduce machine learning, reinforcement learning, or rule-based hybrid algorithms for digital odor control and release. For example, some methods use reinforcement learning to optimize the control parameters of nozzle arrays or predict the diffusion behavior of digital odors in space based on fluid dynamics models. These methods have made breakthroughs at the hardware control level, but there are still significant shortcomings in the key link of "from user semantic needs to digital odor formulation generation." Especially in multimodal environments, there is a lack of a unified modeling framework to achieve semantic matching and scheme generation of cross-modal features, resulting in the generated digital odors often being inconsistent with users' subjective perceptions, thus affecting the practicality and scale of promotion of digital odor systems.

[0006] In summary, existing digital scent generation technologies generally lack effective fusion of multimodal input information, making it difficult to achieve truly personalized and context-aware digital scent generation. They also lack systematic solutions regarding the interpretability and compliance of the generated results. Therefore, there is an urgent need for a new digital scent generation method based on multimodal feature fusion, capable of integrating multi-source input information within a unified modeling framework to automatically generate digital scent formulas that meet user semantic needs and perceptual experiences, while also considering personalization, interpretability, and practical feasibility. This would drive the development of digital scent technology in automotive applications such as in-vehicle air fresheners. Summary of the Invention

[0007] The technical problem this invention aims to solve is to overcome the shortcomings of existing technologies in digital odor generation, such as single input modality, difficulty in expressing complex user needs, lack of personalization in generated results, and inability to balance interpretability and compliance. To this end, this invention proposes a digital odor generation method and system based on multimodal feature fusion.

[0008] The core idea of ​​this method is as follows: First, a novel multimodal feature fusion encoding mechanism is used to uniformly map heterogeneous data such as text, images, videos, and audio to a shared semantic space, thereby transforming complex perceptual needs into regularized multi-feature fusion vectors. Second, an advanced framework that deeply integrates multimodal Transformer and feature fusion generation model is adopted to generate and represent digital odor features in the spectral domain. Finally, by combining spectral domain inversion and multi-objective optimization algorithms, the generated results are mapped to achievable spice formula ratios, ensuring a high degree of consistency between digital odors and user subjective semantics while meeting safety and compliance constraints, thus achieving accurate, reliable, and personalized generation of digital odors.

[0009] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0010] In a first aspect, the present invention provides a digital odor generation method based on multimodal feature fusion, which includes the following steps:

[0011] S1. Obtain the user-input multimodal raw data, input the raw data of each modality into the corresponding encoder for feature extraction, then normalize the features of different modalities and map them to the semantic space of the same dimension, and obtain the multimodal representation vector as the input for digital odor generation feature fusion after feature alignment.

[0012] S2. Input the multimodal representation vector into the trained digital odor generation model for digital odor generation. First, the multimodal representation vector is processed by multiple stacked multimodal fusion-generation modules. The digital odor features generated by the last multimodal fusion-generation module are input into the digital odor generation decoder. The digital odor generation decoder maps the digital odor features into a digital odor spectral domain vector and then inputs it into the recipe inversion module. Using the non-negative least squares method and multi-objective optimization function, the digital odor spectral domain vector is mapped into a digital odor recipe that conforms to the actual blending characteristics.

[0013] S3. The digital scent formula is transmitted to the digital scent presentation device, which synthesizes and releases the corresponding digital scent according to the proportion instructions of the fragrance base library, so as to realize the real-time experience of the user. At the same time, the digital scent formula and its corresponding multimodal raw data and the data generated by the digital scent generation model are stored in the digital scent formula database for subsequent calls and continuous training.

[0014] Based on the above scheme, each step can be implemented in the following preferred manner.

[0015] As a preferred embodiment of the first aspect mentioned above, in S1, the data modalities in the multimodal raw data include images, videos, speech, and text; wherein, the raw data of the image modality is extracted by a convolutional neural network or a visual Transformer to obtain image features, the raw data of the video modality is extracted by a temporal convolutional network or a temporal Transformer to obtain video features, the raw data of the speech modality is extracted by a Mel spectrum and an acoustic model to obtain speech features, and the raw data of the text modality is extracted by a pre-trained language model to obtain text features.

[0016] As a preferred embodiment of the first aspect mentioned above, in S2, the specific processing flow within each multimodal fusion-generation module is as follows: First, image features, video features, speech features, and text features are input into the cross-modal interaction unit to obtain multiple interactive feature components representing the semantic correlation of different modalities; then, the interactive feature components of each modality are input into the multi-scale fusion learning unit, and global semantic features and local contextual features are extracted from each interactive feature component in parallel using a dual-branch structure and then fused; then, all fused features are input into the adaptive aggregation unit for two-dimensional convolution and weight redistribution to obtain a unified fusion representation; finally, the unified fusion representation is input into the digital odor feature mapping unit to perform semantic-to-digital odor feature mapping, and the mapped digital odor features are used as the output of the multimodal fusion-generation module.

[0017] As a preferred embodiment of the first aspect mentioned above, the specific process within the cross-modal interaction unit is as follows: using image features, video features, speech features, and text features as input features, firstly, the intermodal correlation score matrix is ​​calculated using a multi-head cross-attention mechanism to obtain the interaction result, and then the interaction result is nonlinearly mapped through a feedforward neural network. After residual connection between the output features and input features of the feedforward neural network, layer normalization is performed to obtain the interaction feature components.

[0018] As a preferred embodiment of the first aspect mentioned above, the multi-scale fusion learning unit includes a global fusion branch and a local fusion branch: In the global fusion branch, the input interactive feature components are first linearly projected, and then the projected features are divided into multiple overlapping semantic segments through a sliding window. The generated semantic segments are then concatenated to form a set of semantic blocks, which are then passed sequentially through a block-based self-attention layer and a feedforward neural network to obtain two-dimensional spatial features. The two-dimensional spatial features are then flattened and linearly mapped back to a one-dimensional sequence representation space to obtain global semantic features. In the local fusion branch, the input interactive feature components are passed in parallel through multiple depthwise separable one-dimensional convolutional layers with different dilation rates. The convolution results are then fused along the channel dimension and superimposed onto the interactive feature components in a residual connection manner to form local contextual features. Finally, the global semantic features and local contextual features are further concatenated and fused to obtain the fused features.

[0019] As a preferred embodiment of the first aspect mentioned above, the specific process within the adaptive aggregation unit is as follows: First, all fused features are grouped into a two-dimensional tensor and input into a two-dimensional convolutional network to extract intermodal coupling features. Then, the two-dimensional convolution results are decomposed according to modality categories to obtain convolutional feature vectors corresponding to each modality. Based on the saliency weights of each modality feature, the convolutional feature vectors are weighted and aggregated to obtain a unified fused representation. The saliency weights are calculated based on variance estimation, information gain, modality confidence estimator, or attention weight distribution.

[0020] As a preferred embodiment of the first aspect mentioned above, the specific process within the digital odor feature mapping unit is as follows: First, the unified fusion representation is input into the feature condition injection module to obtain the features after injection conditions and then processed by the feature fusion diffusion model or the feature fusion autoregressive Transformer. During the processing, the unified fusion representation is continuously injected to generate digital odor features consistent with the input semantics.

[0021] As a preferred embodiment of the first aspect mentioned above, the specific process inside the digital odor generation decoder is as follows: First, the digital odor features are input into the feature fusion diffusion model or the feature fusion autoregressive Transformer, and the digital odor features are injected in each diffusion sampling step to generate a comprehensive feature representation, which is then input into the denoising network to gradually generate a digital odor spectral domain vector.

[0022] As a preferred embodiment of the first aspect mentioned above, the specific process within the formula inversion module is as follows: The digital odor spectral vector is taken as input and inversely normalized. Then, the non-negative least squares method is used to fit the inverse normalization result onto the base component matrix of the fragrance base library, thereby mapping the inverse normalization result to the corresponding sparsed formula ratio vector. The formula ratio vector is then corrected through a multi-objective optimization function to obtain a digital odor formula that conforms to actual blending. The base component matrix is ​​a multi-dimensional matrix composed of the physicochemical characteristics, odor molecule properties, and their response values ​​in the spectral space of each odor molecule in the fragrance base library, used to characterize the numerical mapping relationship of different base materials in the odor molecule spectral domain.

[0023] Secondly, the present invention provides a digital odor generation system based on multimodal feature fusion, comprising:

[0024] The data acquisition module is used to acquire multimodal data input by the user;

[0025] The digital odor generation module is used to process multimodal data according to the digital odor generation method based on multimodal feature fusion as described in the first aspect above, generate digital odor formulas, and synthesize and release the corresponding digital odors by the digital odor presentation device. At the same time, the digital odor formulas and their corresponding multimodal data, together with the data generated by the digital odor generation model, are stored in the digital odor formula database.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] This invention proposes a digital scent generation method based on multimodal feature fusion. First, through an innovative multimodal feature alignment and fusion mechanism, originally scattered and diverse image, video, audio, and text data are successfully mapped into a unified semantic space, forming a regular and computable feature fusion vector. This fundamentally solves the technical problem of existing methods that can only handle single-modal input and cannot fully express the user's real needs, laying a solid foundation for personalized digital scent synthesis using generative models. Second, this method no longer relies solely on manually set labels or limited scent database retrieval results. Instead, it directly outputs the spectral features of the digital scent within a unified representation space through a feature fusion generative model, and further inversely obtains the specific formula ratio. This design achieves an end-to-end mapping from semantic needs to formula generation, capturing the inherent correspondence between user input and digital scent perception, making the generated results closer to the user's subjective experience and practical application needs.

[0028] Furthermore, the digital scent generation model employed in this invention integrates a learnable multimodal fusion module and a Transformer decoder, enabling it to dynamically adjust the interaction weights between modalities based on the semantic characteristics of the input data. This overcomes the limitations of existing methods in static fusion and limited expressive power. Simultaneously, this method combines non-negative least squares with multi-objective optimization to achieve a synergistic balance in terms of compliance, safety, cost, and scent retention stability, ensuring that the generated formulas are not only accurate and diverse but also reliably applicable in actual blending. Attached Figure Description

[0029] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0030] Figure 2 This is a schematic diagram of the structure of the digital odor generation model provided in this embodiment;

[0031] Figure 3 This is a schematic diagram of the structure of the multimodal fusion-generation module provided in this embodiment;

[0032] Figure 4 This is a schematic diagram of the cross-modal interaction unit provided in this embodiment;

[0033] Figure 5 This is a schematic diagram of the structure of the multi-scale fusion learning unit provided in this embodiment;

[0034] Figure 6 This is a schematic diagram of the adaptive aggregation unit provided in this embodiment;

[0035] Figure 7 This is a schematic diagram of the structure of the digital odor feature mapping unit provided in this embodiment;

[0036] Figure 8 This is a schematic diagram of the structure of the digital odor generation decoder provided in this embodiment;

[0037] Figure 9 This is a schematic diagram of the formula inversion module provided in this embodiment;

[0038] Figure 10 This is a schematic diagram illustrating the dynamic change trends of different features over time in this embodiment.

[0039] Figure 11 This is a system block diagram of the present invention. Detailed Implementation

[0040] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0041] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0042] like Figure 1 As shown, this invention provides a digital scent generation method based on multimodal feature fusion. This method transforms the original multimodal user input data into a unified semantic representation sequence and employs an advanced model capable of adaptively fusing and deeply learning its inherent semantic associations to achieve end-to-end generation from user input to the target digital scent formula. Figure 1 As shown, in a preferred embodiment of the present invention, the method includes the following steps:

[0043] S1. Obtain the user-input multimodal raw data, input the raw data of each modality into the corresponding encoder for feature extraction, then normalize the features of different modalities and map them to the semantic space of the same dimension. After feature alignment, obtain the multimodal representation vector as the input for digital odor generation feature fusion.

[0044] It should be noted that in S1 of this invention, the data modalities in the multimodal raw data include images, videos, speech, and text; wherein, the raw data of the image modality is extracted by a convolutional neural network or a visual Transformer to obtain image features, the raw data of the video modality is extracted by a temporal convolutional network or a temporal Transformer to obtain video features, the raw data of the speech modality is extracted by Mel spectrum and acoustic model to obtain speech features, and the raw data of the text modality is extracted by a pre-trained language model to obtain text features.

[0045] It should be noted that in S1 of this invention, the multimodal raw data input by the user may include modalities such as images, videos, speech, and text, and the source of the multimodal raw data is not limited. For example, raw data of image and video modalities can be acquired through a camera, raw data of speech modalities can be acquired through a microphone, and raw data of text modalities can be input through a keyboard or terminal. Since the modal forms, lengths, and qualities of different user inputs may differ, the multimodal raw data is irregular and inconsistent in time and dimension. Directly inputting such data into the model cannot yield accurate generation results. Therefore, it is necessary to reconstruct and align it to generate a structurally regular, unified semantic representation sequence that can be processed by the model, i.e., the aforementioned multimodal representation vector.

[0046] It should be noted that before data reconstruction, the input multimodal raw data can be preprocessed, including masking for missing modalities, keyframe sampling for excessively long video sequences, noise reduction for speech signals, and word segmentation and cleaning for text input. After preprocessing, the four modalities—image, video, speech, and text—are input into their respective encoders for feature extraction. One of the core innovations of this invention lies in this multimodal data reconstruction step. This step first extracts features using dedicated encoders for different modalities, and then unifies the features of each modality into a shared semantic space through feature normalization, dimension mapping, and alignment methods. For example, image features can be spatial semantic embeddings extracted by convolutional neural networks and visual Transformers; video features can be dynamic semantic embeddings extracted by temporal convolutional networks (TCNs) or temporal Transformers; speech features can be semantic embeddings obtained from Mel spectrum and acoustic models; and text features can be contextual embeddings obtained from pre-trained language models. For ease of description, image features, video features, speech features, and text features are denoted as four modal feature components. The semantic representation corresponding to each input multimodal raw data can be regarded as a multimodal representation vector formed by the combination of the four modal feature components. That is, the reconstructed multimodal representation vector can be regarded as a multivariate sequence composed of multiple modal univariate features, namely image features, video features, speech features, and text features, each corresponding to a univariate feature of one modality. In addition, all univariate features need to be normalized and mapped to the same dimension before being input into the model. For example, the maximum and minimum value normalization is used to scale all feature values ​​to the range of [0,1] or [-1,1] to improve the efficiency and stability of subsequent model training.

[0047] In this embodiment, an input image with a resolution of 224×224 is processed by convolutional and attention layers to obtain a 1024-dimensional vector; a 5-second speech signal is processed by Mel spectrum transform and acoustic model to obtain a 128×100 acoustic matrix and mapped to a 768-dimensional semantic vector; a text input with a length of 20 tokens is encoded by a language model to obtain a 20×768 context matrix; finally, the features of the above different modalities are uniformly mapped to a 1024-dimensional multimodal representation vector.

[0048] S2. Input the multimodal representation vector into the trained digital odor generation model for digital odor generation. First, the multimodal representation vector is processed by multiple stacked multimodal fusion-generation modules. The digital odor features generated by the last multimodal fusion-generation module are input into the digital odor generation decoder. The digital odor generation decoder maps the digital odor features into a digital odor spectral domain vector and then inputs it into the formula inversion module. Using the non-negative least squares method and multi-objective optimization function, the digital odor spectral domain vector is mapped into a digital odor formula that conforms to the actual blending characteristics.

[0049] It should be noted that in step S2, a deep learning model for digital odor generation under multimodal feature fusion needs to be pre-built and trained; this model is referred to as the digital odor generation model. For example... Figure 2 As shown, the multimodal representation vector serves as the input to the aforementioned digital odor generation model. The overall architecture of the digital odor generation model adopted in this invention includes multiple stacked multimodal fusion-generation blocks (MFG blocks), and information flow is enhanced through residual connections. The last multimodal fusion-generation block is then connected to the digital odor generation decoder and the recipe inversion module in sequence to output the digital odor recipe.

[0050] It should be noted that, in this invention, as Figure 3 As shown, the specific processing flow inside each multimodal fusion-generation module is as follows: First, image features, video features, speech features, and text features are input into the cross-modal interaction unit to obtain multiple interaction feature components representing the semantic correlation of different modalities; then, the interaction feature components of each modality are input into the multi-scale fusion learning unit, and global semantic features and local contextual features are extracted from each interaction feature component in parallel using a dual-branch structure and then fused; then, all fused features are input into the adaptive aggregation unit for two-dimensional convolution and weight redistribution to obtain a unified fusion representation; finally, the unified fusion representation is input into the digital odor feature mapping unit to perform semantic-to-digital odor feature mapping, and the mapped digital odor features are used as the output of the multimodal fusion-generation module.

[0051] It should be noted that in S2 of the present invention, as Figure 4As shown, the specific process inside the cross-modal interaction unit is as follows: Image features, video features, speech features, and text features are used as input features. First, the intermodal correlation score matrix is ​​calculated using a multi-head cross-attention mechanism to obtain the interaction result. Then, the interaction result is nonlinearly mapped through a feed-forward network (FFN). The output features and input features of the feed-forward network are connected by a residual connection and then subjected to layer normalization to obtain the interaction feature components.

[0052] It should be noted that in the cross-modal interaction unit, the number of heads, hidden dimensions, and context window size of the multi-head cross-attention are all optimizable parameters and can be adjusted according to the actual task scale; these are not limited in this invention. Furthermore, it should be noted that the cross-modal interaction unit designed in this invention can dynamically adjust information flow and weight allocation based on the feature relevance of the input modality, overcoming the shortcomings of existing multimodal fusion methods such as static splicing, fixed weighting, and difficulty in adaptively modeling modal differences.

[0053] It should be noted that in S2 of the present invention, as Figure 5 As shown, the multi-scale fusion learning unit includes a global fusion branch and a local fusion branch:

[0054] In the global fusion branch, the input interactive feature components are first linearly projected, and then the projected features are divided into multiple overlapping semantic segments through a sliding window. The generated semantic segments are then spliced ​​together to form a set of semantic blocks, which are then passed through a block self-attention layer and a feedforward neural network in sequence to obtain two-dimensional spatial features. The two-dimensional spatial features are then flattened and linearly mapped back to a one-dimensional sequence representation space to obtain global semantic features.

[0055] In the local fusion branch, the input interactive feature components are passed in parallel through multiple depthwise separable one-dimensional convolutional layers with different dilation rates. Then, the convolution results are fused in the channel dimension and superimposed on the interactive feature components in the form of residual connections to form local contextual features.

[0056] Finally, the global semantic features and local contextual features are further spliced ​​and fused to obtain the fused features.

[0057] In embodiments of the present invention, the multi-scale fusion learning unit employs a parallel dual-branch structure to deeply extract the semantic patterns of each interactive feature component. Specifically, the multi-scale fusion learning unit includes a global fusion branch and a local fusion branch. The global fusion branch efficiently captures long-range dependencies through overlapping block partitioning and self-attention mechanisms, while the local fusion branch captures fine-grained contextual dynamic patterns under different receptive fields through multi-scale convolution. Specifically, in the global fusion branch, the interactive feature components of different modalities are first linearly projected, and then the projected features are concatenated and input into a windowed self-attention layer and a feedforward neural network (FFN) to capture long-range dependencies between modalities, thereby obtaining global semantic features. In the local fusion branch, the interactive feature components of each modality are respectively input into multiple depthwise separable 1D convolutional layers with different dilation rates to extract local contextual information, and the extracted results are superimposed with the interactive feature components through residual connections to obtain local contextual features. Finally, the global semantic features and local contextual features are further concatenated and linearly mapped into a unified semantic representation, forming a fused feature as the output of the multi-scale fusion learning unit.

[0058] It should be noted that in S2 of the present invention, as Figure 6 As shown, the specific process inside the adaptive aggregation unit is as follows: First, all fused features are combined into a two-dimensional tensor and input into a two-dimensional convolutional network (2D-CNN) to extract intermodal coupling features. Then, the two-dimensional convolution results are decomposed according to modality categories to obtain the convolutional feature vectors corresponding to each modality. Based on the saliency weights of each modality feature, the convolutional feature vectors are weighted and aggregated to obtain a unified fused representation.

[0059] It should be noted that in the adaptive aggregation unit, the significance weights are calculated based on variance estimation, information gain, modality confidence estimator, or attention weight distribution. It should also be noted that the above weighted aggregation can not only be based on the weight calculation methods listed in this invention, but can also be adaptively adjusted by those skilled in the art according to actual needs; therefore, no limitations are imposed in this invention.

[0060] It should be noted that in S2 of the present invention, as Figure 7As shown, the specific process inside the digital odor feature mapping unit is as follows: First, the unified fusion representation is input into the feature condition injection module to obtain the features after injection conditions and then processed by the feature fusion diffusion model (CDM) or the feature fusion autoregressive transformer (C-AT). During the processing, the unified fusion representation is continuously injected to generate digital odor features that are consistent with the input semantics.

[0061] In the embodiments of the present invention, the design of the aforementioned digital odor feature mapping unit draws upon the field of feature fusion generation modeling, aiming to learn the nonlinear mapping relationship between the fused unified representation and the target digital odor spectral domain. Its specific processing flow is as described above and will not be repeated here.

[0062] It should be noted that in S2 of the present invention, as Figure 8 As shown, the specific process inside the digital odor generation decoder is as follows: First, the digital odor features are input into the feature fusion diffusion model or the feature fusion autoregressive Transformer, and digital odor features are injected in each diffusion sampling step to generate a comprehensive feature representation, which is then input into the denoising network to gradually generate the digital odor spectral domain vector.

[0063] It should be noted that in S2 of the present invention, as Figure 9 As shown, the specific process inside the formula inversion module is as follows: The digital odor spectral vector is taken as input and inversely normalized. Then, the non-negative least squares method is used to fit the inverse normalization result on the base component matrix of the fragrance base library, thereby mapping the inverse normalization result to the corresponding sparse formula ratio vector. The formula ratio vector is then corrected through a multi-objective optimization function to obtain a digital odor formula that conforms to actual blending. The base component matrix is ​​a multi-dimensional matrix composed of the physicochemical characteristics, odor molecule properties and their response values ​​in the spectral space of each odor molecule in the fragrance base library, which is used to characterize the numerical mapping relationship of different base materials in the odor molecule spectral domain.

[0064] In an embodiment of the present invention, the digital odor decoder receives the digital odor features output by the last multimodal fusion-generation module and maps them to a spectral vector of the target digital odor (i.e., the aforementioned digital odor spectral vector) using a feature fusion diffusion model or a feature fusion autoregressive Transformer. The formula inversion module then generates a formula proportion vector based on non-negative least squares (NNLS). Matching degree, compliance, cost, and longevity stability are then fused as constraint features. The formula proportion vector is corrected using a multi-objective optimization function (MOOF) to ensure that the generated digital odor is semantically consistent with the user input, thus obtaining a final digital odor formula that satisfies both digital odor matching degree and safety, compliance, and cost constraints. The specific processing flow of the digital odor decoder and the formula inversion module is as described above and will not be repeated here.

[0065] It should also be noted that the digital odor generation model of the present invention needs to be supervised before it is used for actual reasoning. Its loss function preferably includes the mean square error loss of spectral domain prediction, the mean absolute error loss of the formula ratio, or the cross-entropy loss of expert evaluation labels, so as to ensure that the digital odor spectral domain vector and digital odor formula generated by the model have high accuracy and interpretability.

[0066] In the training process of the digital odor generation model in this invention embodiment, its core is a cross-modal matching operator (CMMO) with a learnable attention matrix as a parameter. Unlike traditional multimodal alignment methods that use static splicing or fixed weighting, this invention combines cross-attention with gated feature fusion normalization (FiLM / Gated Layer Normalization) to achieve bidirectional semantic retrieval and feature interaction between image-text, video-speech, and other modalities. Therefore, unlike traditional static modal splicing methods, it can adaptively optimize attention weights through backpropagation during model training, thereby specifically capturing the potential correspondence between the input modality and the target digital odor features.

[0067] S3. Output the obtained digital scent formula to the application terminal, including: on the one hand, transmitting the digital scent formula to the digital scent presentation device, which synthesizes and releases the corresponding digital scent according to the proportion instructions of the fragrance base library, so as to realize the real-time experience of the user terminal; on the other hand, storing the digital scent formula and its corresponding multimodal raw data together with the data generated by the digital scent generation model into the digital scent formula database for subsequent calls and continuous training.

[0068] It should be noted that in S3 of this invention, the digital scent presentation device is a hardware device used to convert the digital scent formula into a perceptible real scent. In this embodiment, the device may include a control unit, a scent storage unit, a blending and mixing module, a release module, a ventilation and cleaning module, and a communication interface. These can also be adjusted according to actual needs and are not limited in this invention. The aforementioned ratio instruction refers to the set of calculated release ratio parameters of each basic fragrance in the mixed scent.

[0069] Furthermore, in this embodiment, feedback evaluation signals from professional perfumers and end users can also be received. After converting the feedback evaluation signals into strategy scores, the synthesis ratio of the digital scent formula can be fine-tuned, thereby achieving continuous optimization and adaptive improvement based on the expert-user dual closed loop. This allows the generated digital scent formula to gradually meet the personalized needs and dynamic preferences of users while ensuring compliance and feasibility.

[0070] The present invention will further demonstrate the implementation process of the digital odor generation method based on multimodal feature fusion shown in steps S1 to S3 above in a practical application scenario through a specific embodiment, so as to demonstrate its specific digital odor generation effect.

[0071] Example

[0072] The digital odor generation method based on multimodal feature fusion used in this embodiment is implemented as described above and will not be repeated here. The key steps in the specific implementation process of the method of the present invention are described below.

[0073] Step 1: Construct a multimodal digital odor dataset

[0074] First, multimodal raw data consisting of images, videos, audio, and text is acquired, with each piece of multimodal raw data corresponding to a known digital scent recipe or sensory description. In this embodiment, publicly available fragrance databases, user multimodal feedback collected in the laboratory, and recipe corpora annotated by perfumers are used as data sources. Second, the multimodal raw data is cleaned, and missing modalities are filled using mean interpolation or semantic imputation methods, while outliers are removed using thresholding or smoothing. Next, by aligning the data timestamps and semantic tags of different modalities, multimodal pairing relationships are established, uniformly binding image / video frame sequences, audio segments, and text descriptions to the same target digital scent instance, thereby constructing a series of multimodal samples. Subsequently, for each sample in this series, a multidimensional feature vector is calculated and generated. The variable dimensions included in this vector are: [image spatial features, video temporal features, audio acoustic features, text semantic features, recipe spectral domain features, and user sensory tags]. In this process, image spatial features are extracted using a convolutional neural network and a visual Transformer; video temporal features are extracted using a 3D convolutional network and a temporal Transformer; speech acoustic features are extracted using Mel-frequency coefficients and an acoustic encoder; text semantic features are generated from context vectors by a pre-trained large language model; formula spectral domain features are derived from the standardized component matrix of flavor base materials; and user sensory labels are converted from questionnaires or expert scores into numerical representations. Finally, the multidimensional feature vectors of all samples are arranged according to the collection order or semantic label order of their corresponding data instances, thereby generating a regularized multimodal training sequence that can be processed by the model. Each variable dimension in the multimodal training sequence corresponds to a feature sequence. Due to the significant differences in the dimensions and ranges of different modal features, a maximum-minimum normalization method is used to preprocess each training sequence to eliminate the influence of dimensions. The processing formula is as follows:

[0075]

[0076] in, To normalize the data, The raw data for different input modalities, These represent the maximum and minimum values ​​of the corresponding original data, respectively.

[0077] To construct the dataset used for model training, conventional training sample construction methods can be followed, which will not be elaborated upon in this invention. In this embodiment, each training sample in the dataset contains a multimodal training sequence within a historical window and a target digital odor representation within a prediction window, wherein the target digital odor representation includes a digital odor spectral domain feature vector and a recipe ratio vector. The dataset is then divided into a training set, a validation set, and a test set in a 7:2:1 ratio for model training and performance evaluation.

[0078] Step 2: Feature Extraction and Unified Fusion Representation Generation

[0079] In this embodiment, the core task of the digital odor generation model is to map multimodal inputs such as images, videos, audio, and text into a unified semantic space, and then perform deep fusion through a cross-modal interaction mechanism to ultimately generate a high-dimensional semantic representation that can drive digital odor synthesis. In this embodiment, feature extraction is first performed on the data from each modality. Specifically, image features... Through Visual Transformer extract:

[0080]

[0081] in, This represents the input image; Indicates the number of image patches. Indicates the initial feature dimension. Video features. Extracted using a temporal Transformer:

[0082]

[0083] in, This refers to the input video; Number of video frame blocks. Speech features. via Mel spectrum The spectrum is obtained by transformation, and then passed through an acoustic model. extract:

[0084]

[0085] in, This represents the input acoustic signal; Indicates the number of frames in the acoustic signal. Text features. This is obtained through a pre-trained language model (such as InstructBLIP):

[0086]

[0087] in, This represents the input word sequence; This represents the number of tokens in a word sequence. This refers to the pre-trained language model mentioned above.

[0088] Furthermore, in order to achieve a unified representation among multiple modalities, this embodiment projects the features of all modalities onto a unified dimension. :

[0089]

[0090] in, These are learnable parameters; Representing modes The corresponding unified features; This represents the real number field. Through this step, all modal features are mapped to the same feature space, thereby eliminating structural differences between modes.

[0091] After modal alignment, this invention achieves information interaction through cross-modal interaction units. For example, when the text modality is the query modality and the image modality is the support modality, the interaction calculation process is as follows:

[0092]

[0093]

[0094] in, These represent the query, key, and value, respectively. For learnable projection matrices, The dimension of the attention head; This represents semantic transfer or cross-modal attention from image modality to text modality, enabling text representations to incorporate visual information; This represents the softmax activation function; This indicates transpose.

[0095] This operation allows the text modality to selectively absorb semantic information from the image modality. Similarly, multiple interaction paths, such as voice-text and video-image, can be designed to achieve bidirectional enhancement across all modalities.

[0096] After multimodal interaction is completed, all modal features are concatenated and input into the multi-scale fusion learning unit and the adaptive aggregation unit. An attention weight allocation mechanism is used to dynamically adjust the contributions of different modalities. The calculation method is as follows:

[0097]

[0098]

[0099] in, Represents modal weights; and Representing modes and modality Average pooling characteristics; It is a learnable vector.

[0100] The final output is a unified fusion representation. This will serve as a global semantic vector, providing input for subsequent digital odor feature generation and decoding.

[0101] Step 3: Model Training and Inference

[0102] During the training phase, this embodiment uses the aforementioned training samples as input, where each training sample contains modalities such as images, videos, speech, and text. The model generates a corresponding digital odor feature vector by fusing these modal features. As a prediction result, under a supervised learning framework, the target digital odor vector, either calibrated using existing formula annotations or those defined by professional perfumers, is used. As a real label.

[0103] To measure the difference between the predicted results and the true labels, this embodiment uses the mean squared error loss function. The calculation formula is as follows:

[0104]

[0105] in, This indicates the number of training samples in the training batch; The number of dimensions representing digital odor characteristics (e.g., different chemical components or sensory features). Indicates the first The first sample One target feature value, This represents the corresponding model prediction value.

[0106] In terms of optimization methods, this embodiment uses the AdamW optimizer to optimize all learnable parameters in the model. The update process is as follows:

[0107]

[0108] in, and They represent the first The second iteration and the first The model can learn parameters in each iteration; The learning rate; For the first The bias correction estimate of the first-order momentum in the next iteration. No. The bias correction estimate of the second-order momentum in the next iteration; For numerically stable terms, This is the weight decay coefficient.

[0109] To prevent overfitting during training, this embodiment employs early stopping. Specifically, it monitors the loss function on the validation set; if the loss function fails to decrease over multiple consecutive training epochs, the training process is automatically terminated, thus avoiding ineffective training iterations.

[0110] During the inference phase, users can input any combination of modalities (e.g., image + text, video + voice, etc.). The method of this invention first extracts and fuses features from each modal encoder to generate a unified fused representation, which is then mapped to a digital odor feature vector by a decoder. Finally, the aforementioned digital odor feature vector can be further mapped to a specific digital odor recipe or sensory label, realizing a complete generation process from multimodal input feature fusion to the target digital odor output.

[0111] It should also be noted that, based on the above steps, this embodiment compares the performance of the digital odor generation model proposed in this invention (hereinafter referred to as the "invention model") with three representative existing models under the same dataset and experimental conditions. The comparison models include: a unimodal text conditional generation model (Text2Scent); an image-based odor description generation model (Image2OdorNet); and a multimodal feature concatenation model (Concat-Transformer). The implementation methods of these comparison models are existing technologies and will not be elaborated further.

[0112] Table 1 shows the performance comparison results of each model on key indicators such as Odor Consistency Score (OCS), Fragrance Feature Match (FFM), Mean Absolute Error (MAE), and Sensory Similarity Index (SSI).

[0113] Table 1. Comparison of Indicators between the Model of this Invention and Existing Models

[0114]

[0115] As shown in Table 1, the model of this invention has achieved significant advantages in all key metrics: On the one hand, thanks to the introduction of cross-modal interaction units and multi-scale fusion learning units, features of different modalities are fully integrated at the semantic level, significantly improving the consistency and accuracy of aroma generation (OCS increased from 0.881 to 0.928); on the other hand, the adaptive aggregation unit and feature decoding mechanism effectively reduce generation errors, reducing MAE by about 48.5% compared to the traditional Transformer model, from 0.161 to 0.084; in addition, although the model is more complex in structure, thanks to modular design and lightweight parameter sharing strategy, its inference time is only 0.83s, which can still meet the needs of real-time aroma interaction scenarios.

[0116] And such Figure 10As shown in the figure, this diagram illustrates the dynamic trends of image modal features, text modal features, speech modal features, and the fused feature output over time. The solid red line represents image modal features, the dashed blue line represents text modal features, the dotted green line represents speech modal features, and the solid black line represents the fused feature output obtained after multimodal fusion. Figure 10 It can be seen that different modal features exhibit their own independent fluctuation patterns over time: image modal features gradually increase in the initial stage, then enter a decay stage; text modal features show a trend of first decreasing and then increasing; speech modal features are relatively stable in the early stage, and decrease in the middle and late stages. In contrast, the fusion feature output curve is smoother overall with smaller fluctuations, indicating that the method of this invention can effectively balance the information differences between different modalities and achieve dynamic complementarity and stable fusion at the feature level. Through analysis of... Figure 10 Analysis of the results shows that the multimodal feature fusion mechanism of the present invention can maintain the consistency and robustness of feature expression during temporal changes, providing higher feature stability and discrimination ability for subsequent multimodal perception and recognition tasks.

[0117] In summary, the model of this invention exhibits higher aroma generation accuracy and stronger generalization ability under multimodal conditions, verifying the feasibility and advancement of this method in the fields of digital odor generation and human-computer perception interaction.

[0118] It should also be noted that the digital odor generation method based on multimodal feature fusion in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a digital odor generation system based on multimodal feature fusion, corresponding to the digital odor generation method based on multimodal feature fusion provided in the above embodiments, such as... Figure 11 As shown, it includes:

[0119] The data acquisition module is used to acquire multimodal data input by the user;

[0120] The digital odor generation module is used to process multimodal data according to the digital odor generation method based on multimodal feature fusion described in the above embodiments, generate digital odor formulas, and synthesize and release the corresponding digital odors by the digital odor presentation device. At the same time, the digital odor formulas and their corresponding multimodal data, along with the data generated by the digital odor generation model, are stored in the digital odor formula database.

[0121] It is understood that the digital odor generation method based on multimodal feature fusion described in S1-S3 above can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the digital odor generation method based on multimodal feature fusion provided in the above embodiments, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, they can implement the digital odor generation method based on multimodal feature fusion as described in the above embodiments.

[0122] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0123] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0124] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A digital smell generation method based on multi-modal feature fusion, characterized in that, The method comprises the following steps: S1, acquiring multi-modal original data input by a user, inputting original data of each modality into a corresponding encoder for feature extraction, uniformly normalizing and mapping features of different modalities to a same dimension semantic space, and obtaining a multi-modal representation vector as a digital smell generation feature fusion input after feature alignment; S2, inputting the multi-modal representation vector into a trained digital smell generation model for digital smell formula generation, first processing the multi-modal representation vector through a plurality of stacked multi-modal fusion-generation modules, inputting a digital smell feature generated by a last multi-modal fusion-generation module into a digital smell generation decoder, inputting a digital smell spectrum domain vector obtained after the digital smell generation decoder maps the digital smell feature into the digital smell spectrum domain into a formula inversion module, and mapping the digital smell spectrum domain vector into a digital smell formula conforming to actual blending characteristics by using a non-negative least square method and a multi-objective optimization function; S3, transmitting the digital smell formula to a digital smell presentation device, synthesizing and releasing a corresponding digital smell according to a proportion instruction of a fragrance base material library, realizing instant experience at a user end, and storing the digital smell formula, the corresponding multi-modal original data and data generated by the digital smell generation model into a digital smell formula database for subsequent calling and continuous training; The specific process inside the digital smell generation decoder is as follows: first, inputting the digital smell feature into a feature fusion diffusion model or a feature fusion autoregressive Transformer, and injecting the digital smell feature at each diffusion sampling step, so as to generate a comprehensive feature representation and input it into a denoising network to gradually generate a digital smell spectrum domain vector; The specific process inside the formula inversion module is as follows: inputting the digital smell spectrum domain vector as an input and performing inverse normalization, then fitting the inverse normalized result on a base material component matrix of the fragrance base material library by using a non-negative least square method, so as to map the inverse normalized result into a corresponding sparse formula proportion vector, and correcting the formula proportion vector by using a multi-objective optimization function to obtain a digital smell formula conforming to actual blending; wherein, the base material component matrix is a multi-dimensional matrix composed of physicochemical characteristics, odor molecule attributes and response values in a spectrum domain space of each odor molecule in the fragrance base material library, for representing numerical mapping relationships of different base materials in the odor molecule spectrum domain. 2.The digital smell generation method based on multi-modal feature fusion of claim 1, wherein, In S1, data modalities in the multi-modal original data include images, videos, voices and texts; wherein, image features are obtained after original data of the image modality is extracted by a convolutional neural network or a visual Transformer, video features are obtained after original data of the video modality is extracted by a time sequence convolutional network or a time sequence Transformer, voice features are obtained after original data of the voice modality is extracted by a Mel spectrum and an acoustic model in sequence, and text features are obtained after original data of the text modality is extracted by a pre-trained language model. 3.The digital odor generation method based on multi-modal feature fusion of claim 2, wherein, In S2, the specific processing procedure inside each multimodal fusion-generation module is as follows: first, the image features, video features, speech features and text features are input into the cross-modal interaction unit to obtain a plurality of interactive feature components representing semantic correlations of different modalities; then the interactive feature components of each modality are input into the multi-scale fusion learning unit, global semantic features and local context features are extracted from each interactive feature component in parallel through a double-branch structure, and then fused, and finally all the fused features are input into the adaptive aggregation unit for two-dimensional convolution and weight redistribution to obtain a unified fusion representation; finally, the unified fusion representation is input into the digital odor feature mapping unit for semantic-to-digital odor feature mapping, and the mapped digital odor features are taken as the output of the multimodal fusion-generation module. 4.The digital odor generation method based on multi-modal feature fusion of claim 3, wherein, The specific procedure inside the cross-modal interaction unit is as follows: the image features, video features, speech features and text features are taken as input features, the inter-modal correlation score matrix is calculated by using the multi-head cross-attention mechanism, the result after interaction is obtained, and then the result after interaction is nonlinearly mapped by a feedforward neural network, the output features of the feedforward neural network are connected in residual with the input features, and then layer normalization is performed to obtain the interactive feature components. 5.The digital odor generation method based on multi-modal feature fusion of claim 4, wherein, The multi-scale fusion learning unit includes a global fusion branch and a local fusion branch: in the global fusion branch, the input interactive feature components are first linearly projected, then the projected features are divided into a plurality of overlapping semantic segments through a sliding window, the generated semantic segments are spliced to form a semantic block set, and then the semantic block set is sequentially input into a block self-attention layer and a feedforward neural network to obtain two-dimensional spatial features, and then the two-dimensional spatial features are flattened and linearly mapped back to a one-dimensional sequence representation space to obtain global semantic features; In the local fusion branch, the input interactive feature components are parallelly input into a plurality of depth separable one-dimensional convolution layers with different expansion rates, then the convolution results are fused in the channel dimension and stacked on the interactive feature components in a residual connection manner to form local context features; finally, the global semantic features and the local context features are further spliced and fused to obtain the fused features. 6.The digital odor generation method based on multi-modal feature fusion of claim 5, wherein, The specific procedure inside the adaptive aggregation unit is as follows: first, all the fused features are combined into a two-dimensional tensor and input into a two-dimensional convolution network to extract inter-modal coupling features, then the two-dimensional convolution results are decomposed according to the modality categories to obtain the convolution feature vectors corresponding to each modality, and the convolution feature vectors are weighted aggregated according to the saliency weights of the features of each modality to obtain a unified fusion representation; wherein the saliency weights are calculated based on variance estimation, information gain, modality confidence estimator or attention weight distribution.

7. The multi-modal feature fusion based digital odor generation method of claim 6, wherein, The specific procedure inside the digital odor feature mapping unit is as follows: first, the unified fusion representation is input into the feature condition injection module to obtain the features after injection of the condition and processed through a feature fusion diffusion model or a feature fusion autoregressive Transformer, and the unified fusion representation is continuously injected during the processing to generate digital odor features consistent with the input semantics.

8. A digital smell generation system based on multi-modal feature fusion, characterized by, The method comprises: A data acquisition module for acquiring multi-modal data input by a user; A digital smell generation module for processing the multi-modal data according to the multi-modal feature fusion-based digital smell generation method as claimed in any one of claims 1-7, generating a digital smell formula and synthesizing and releasing a corresponding digital smell by a digital smell presentation device, and storing the digital smell formula, the corresponding multi-modal data and data generated by the digital smell generation model into a digital smell formula database.

Images