Generative odor real-time synthesis method and system based on cross-modal submerged space mapping

Abstract

The invention discloses a generative odor real-time synthesis method and system based on cross-modal potential space mapping, and belongs to the technical field of artificial intelligence and olfaction calculation. The method comprises the following steps: acquiring a multi-modal input stream of a current scene, and extracting an emotion semantic feature vector by using a deep neural network; mapping the semantic features into target odor chemical feature vectors by using nonlinear projection through a pre-constructed vision-smell joint embedding space; constructing a convex optimization model based on olfactory perception, and calculating a basic liquid optimal mixing proportionality coefficient matrix capable of fitting the target vector; the matrix is converted into a micro-fluidic driving signal, and the target smell is synthesized in situ in the micro-fluidic chip. The invention further discloses a self-adaptive cleaning logic and olfactory fatigue compensation mechanism based on scene mutation detection. The method solves the problems that in the prior art, label matching is dependent, new smell cannot be synthesized, and dynamic transition is lacked, and olfactory replicating and real-time generation of abstract semantic scenes are achieved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence applications, computational olfaction, and microelectromechanical systems (MEMS), specifically to a method and system for olfactory semantic understanding using multimodal deep learning technology and for real-time odor synthesis using microfluidic technology. Background Technology

[0002] Human perception of the physical world primarily relies on sight, hearing, touch, smell, and taste. With the development of virtual reality (VR), metaverse, and immersive entertainment, digital interaction of sight and hearing has matured, but olfactory interaction is still in its early stages.

[0003] Existing scent playback devices suffer from the following significant technical bottlenecks, limiting their application in high-end AI hardware: Limitations of rule-based matching: Existing technologies mainly employ the logic of "object recognition - tag matching". For example, recognizing a "coffee cup" will trigger the playback of "coffee smell". However, real-world scenarios are often abstract and complex. For example, a "cyberpunk-style rainy night" would correspond to a complex smell of "rain + metal + the smell of overheated neon plastic", which existing technologies cannot reproduce using a pre-set tag library.

[0004] Lack of generation capability: Existing devices can only play fixed odors stored in capsules and cannot synthesize any odors that exist in nature through base colors (base liquids) like RGB colors are displayed on a screen.

[0005] Odor Residue and Cross-contamination: Odor molecules have adsorption properties. Existing devices lack an active cleaning mechanism when switching from "rose scent" to "sea scent," resulting in the former odor remaining in the pipes and forming an unpleasant mixed odor.

[0006] Lack of dynamic transitions: The emotions in movies or games are fluid, while the odor release of existing devices is a discrete on / off quantity, which cannot achieve a smooth and gradual change in odor concentration and type.

[0007] For example, Chinese patent CN115761456A discloses an odor release method that matches corresponding odors by identifying entities in an image. However, this method can only match a single preset odor and does not have the function of odor synthesis. Chinese patent CN213582812U discloses a multimodal design element mapping device that includes an odor generating device, but its core objective is to solve the problem of perceptible mapping of user needs in product innovation design, and its adaptability to dynamic odor synthesis is weak.

[0008] Therefore, there is an urgent need for an intelligent odor synthesis technology that can understand abstract scene semantics, generatively synthesize new odors, and achieve dynamic transitions. Summary of the Invention

[0009] This invention aims to solve the aforementioned technical problems by providing a generative real-time odor synthesis method and system based on cross-modal latent space mapping. Instead of relying on a pre-defined tag library, this invention constructs a "visual-olfactory joint embedding space" that "translates" any input visual / auditory signal into a feature vector of chemical molecules. A mathematical optimization algorithm is then used to calculate the formula, thereby enabling the creation of entirely new odors for unknown and abstract scenarios.

[0010] The technical solution adopted in this invention is as follows: A generative real-time odor synthesis method based on cross-modal latent space mapping includes the following steps: S1. Data Acquisition and Encoding Steps: Acquire multimodal data streams in the environment in real time, wherein the multimodal data streams include at least one or more of video image frame sequences, audio waveform data, and text instruction data; use a pre-trained multimodal encoder model to extract features from the multimodal data streams and output a high-dimensional sentiment semantic feature vector V_sem; S2. Cross-modal alignment step: Construct a visual-olfactory joint embedding space, and use the trained nonlinear mapping function f(·) to project the emotional semantic feature vector V_sem onto the joint embedding space to obtain the corresponding target odor chemical perception vector V_chem; S3. Generative formulation calculation steps: Obtain the chemical property vector group {B_1, B_2, ..., B_N} of the N base liquids currently stored in the device; construct an optimization objective function with non-negative constraints, and calculate a set of mixing weight coefficients W = {w_1, w_2, ..., w_N} such that the distance metric between the mixed synthesis vector Σ(w_i·B_i) and the target odor chemical perception vector V_chem is minimized; S4. Dynamic synthesis control steps: The calculated mixing weight coefficient W is converted into the corresponding pulse width modulation (PWM) signal sequence to control the duty cycle and frequency of the microfluidic micropump array, synthesize the target odor in situ in the microfluidic mixing chamber, and release it through the atomization module.

[0011] Furthermore, the visual-olfactory joint embedding space described in step S2 is constructed in the following manner: Collect a dataset of chemical molecules containing olfactory descriptors and a dataset of visual scenes containing emotion annotations; A dual-tower neural network architecture is constructed, serving as a visual encoder and an olfactory encoder, respectively. Joint training is performed using the contrastive loss function to maximize the cosine similarity between paired visual and olfactory samples in the embedding space and minimize the similarity between unpaired samples. The target odor chemical perception vector V_chem includes the volatility characteristics of odor molecules, fragrance category characteristics, and major functional group characteristics.

[0012] Furthermore, the optimization objective function constructed in step S3 is: min_W (||Σ(w_i·B_i) - V_chem||_2^2 + λ_1·Σ(w_i / (R_i(t)+ε)) + λ_2·F(t)·||W||_1) The constraints are: Σw_i = 1, w_i ≥ 0 Where R_i(t) is the remaining capacity of the i-th base liquid at time t, ε is the zero-smoothing term, F(t) is the user's current olfactory fatigue coefficient, λ_1 is the penalty weight for base liquid balance consumption, and λ_2 is the concentration control weight.

[0013] Furthermore, step S4 also includes adaptive cleaning logic during scene switching: Establish a sliding time window of length T, and calculate the rate of change of the Euclidean distance D_rate of the emotional semantic feature vector V_sem within the window in real time; If D_rate exceeds the preset mutation threshold δ, it is determined to be a scene mutation event; Trigger the cleaning interrupt service routine: Pause the current odor synthesis instruction, control the micropump connected to the cleaning fluid to start at the maximum flow rate Q_max for a duration Δt to flush the microfluidic mixing chamber, and then execute the next frame of odor synthesis instruction; The value of Δt is proportional to the average volatility of the odor formula in the previous frame.

[0014] Furthermore, in step S4, when converting the mixed weighting coefficients W into a PWM signal sequence, a random dithering strategy is adopted: While keeping the total duty cycle constant, the activation phases of the micropumps in each channel are randomly staggered to avoid the superposition of fluid pressure pulses caused by multiple micropumps activating simultaneously, thus ensuring the stability of the flow field within the microfluidic chip.

[0015] Furthermore, the method also includes a user feedback fine-tuning step: Receive user voice feedback commands; Analyze the emotional polarity of the feedback command; If the feedback is negative, a correction vector ΔV is generated based on the feedback content, the target odor chemical perception vector V'_chem = V_chem + α·ΔV is updated, where α is the learning rate, and step S3 is re-executed.

[0016] This invention also provides a generative real-time odor synthesis system based on cross-modal latent space mapping, comprising: The multimodal perception module is used to collect environmental video, audio, and user voice data; The AI ​​inference computing module has a built-in memory and processor, which stores a computer program that executes the above-mentioned method steps. The microfluidic actuator module includes a base fluid storage tank, a cleaning fluid tank, a micro-piezoelectric pump array, a microfluidic mixing chip, and an ultrasonic atomizer; The AI ​​inference computing module sends control commands to the microfluidic execution module via the I2C bus. The microfluidic hybrid chip has a multi-inlet single-outlet structure and internally has microstructures for fluid chaotic convection.

[0017] The beneficial effects of this invention are as follows: True generative olfaction: Utilizing convex optimization algorithms and microfluidics, it achieves odor synthesis similar to "RGB color matching," creating an infinite odor experience with a limited base liquid.

[0018] Deep semantic understanding: It can understand the light and shadow, color tone and emotion in the picture, rather than just recognizing objects. It is suitable for artistic scenes such as movies and games.

[0019] Zero-residue switching: Introducing self-cleaning logic based on semantic mutation detection solves the physical problem of hardware cross-contamination from the software algorithm level.

[0020] Hardware protection and balancing: The optimization function introduces a base fluid consumption balancing term to prevent a certain commonly used base fluid from being depleted too early, thus extending the consumable replacement cycle.

[0021] Dynamic Adaptation: By using the olfactory fatigue coefficient F(t) and a random shaking strategy, dynamic adjustment of odor concentration and stable control of the flow field are achieved. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall process of the method of the present invention; Figure 2 A schematic diagram of the training architecture for the visual-olfactory joint embedding space; Figure 3 A schematic diagram of the optimization process for generative recipe calculation; Figure 4 This is a timing diagram of the scene mutation detection and adaptive cleaning logic; Figure 5 This is a hardware architecture block diagram of the system of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0024] Example 1: System Architecture and Core Algorithm The hardware of this system adopts a layered architecture: the host computer is a high-performance AI computing unit equipped with an NPU (such as RK3588 or Jetson series) responsible for algorithm inference; the slave computer is an STM32F4 series MCU, which integrates a 5-channel microfluidic drive circuit and is responsible for the underlying hardware control; and a specially designed microfluidic odor synthesis module.

[0025] The core process of the software algorithm is as follows: Step 1: Multimodal Feature Encoding The system captures video frames at a rate of 1 fps. The image encoder of the finely tuned CLIP (Contrastive Language-Image Pre-training) model is used as the feature extractor.

[0026] Input: A picture of a desert at sunset.

[0027] Output: A 512-dimensional sentiment semantic feature vector V_sem. This vector not only contains the object features of "desert", but also encodes high-level semantics such as "dryness", "heat", and "solitude".

[0028] Step 2: Latent Space Projection The system pre-defines a mapping matrix M_proj, which is obtained by training on a large-scale "image-odor description" dataset.

[0029] Calculate the target chemical vector: V_chem = M_proj × V_sem.

[0030] The resulting V_chem is a vector representing the physicochemical properties of odor, such as [volatility: 0.8, sweetness: 0.1, woodiness: 0.7, spiciness: 0.4].

[0031] Step 3: Generative Recipe Calculation The system contains 5 base fluids, whose attribute vectors are known: B1 (Citrus): [0.9, 0.8, 0.0, 0.1] B2 (Sandalwood): [0.2, 0.1, 0.9, 0.1] B3 (Peppermint): [0.95, 0.0, 0.1, 0.8] B4 (Musk): [0.1, 0.3, 0.7, 0.2] B5 (Sea Salt): [0.6, 0.0, 0.2, 0.5] The system uses convex optimization solvers (such as CVXPY, OSQP, or interior-point solvers) to solve the following non-negative least squares problem: min_W ||Σw_i·B_i - V_chem||^2 + λ_1·Σ(w_i / (R_i(t)+ε)) + λ_2·F(t)·||W||_1 Assume the current remaining capacity of the base liquid is R = [80%, 60%, 90%, 40%, 85%], olfactory fatigue F(t) = 0.3, λ_1 = 0.1, λ_2 = 0.05, ε = 0.01.

[0032] The weights W = [0.1, 0.6, 0.0, 0.3, 0.0] are obtained by using a convex optimization solver (such as CVXPY or a quadratic programming algorithm).

[0033] This means a blend of 10% citrus, 60% sandalwood, and 30% musk is needed. Note that base liquid 3 (mint) and base liquid 5 (sea salt) have a weight of 0, indicating they are not suitable for the current scenario.

[0034] Step 4: Microfluidic Execution The MCU converts the weight W into a PWM signal.

[0035] Period T = 100ms.

[0036] Channel 1 (Citrus) High Level Duration: 10ms.

[0037] Channel 2 (Sandalwood) high level time: 60ms.

[0038] Channel 3 (Mint) high level time: 0ms.

[0039] Channel 4 (Musk) high level time: 30ms.

[0040] Channel 5 (Sea Salt) High Level Time: 0ms.

[0041] To avoid pressure pulse superposition, a random jitter strategy is adopted: Channel 1 is activated within 0-10ms.

[0042] Channel 2 is activated within 15-75ms.

[0043] Channel 4 is activated in 80-110ms.

[0044] Each micro-pump injects micro-level droplets into the mixing chamber in this sequence. Molecular-level mixing is achieved in the spiral channel using the Coanda Effect and chaotic convection, and the mixture is finally ejected by the atomizing plate.

[0045] Example 2: Adaptive Cleaning Logic The system maintains a sliding window with a length of 5 seconds to record V_sem.

[0046] When the video instantly switches from the "desert" to the "ice and snow world", the vector distance ||V_sem(t) - V_sem(t-1)|| increases abruptly, exceeding the threshold δ = 0.5.

[0047] The system detected a "sudden scene change". At this point, the current "ice and snow flavor" synthesis task was immediately suspended.

[0048] Insert a high-priority interrupt: Enable channel 6 (anhydrous ethanol) and flush the mixing chamber with a 100% full-speed pulse for 500ms.

[0049] After cleaning, the "ice and snow flavor" formula is synthesized. This process is almost imperceptible to the user, but it ensures that the first whiff of the "ice and snow flavor" is pure and free of impurities.

[0050] Since the average volatility of the previous frame's "desert scent" was 0.8 (high volatility), the cleaning time Δt = 500ms. If the previous frame had a low volatility "sandalwood scent" (volatility 0.2), then Δt would be extended to 1200ms to ensure thorough cleaning.

[0051] Example 3: User Feedback Fine-tuning While watching a movie, the system synthesized a "forest flavor." The user commented, "It's too sweet; I want something fresher." The system uses speech recognition and sentiment analysis to identify negative feedback and extract the keyword "fresh".

[0052] The system queries the preset semantic-chemical mapping table and finds that the correction vector corresponding to "fresh" is ΔV = [+0.2 volatility, -0.3 sweetness, +0.1 herbal feel, 0 spiciness].

[0053] Update the target vector: V'_chem = V_chem + 0.3·ΔV (learning rate α=0.3).

[0054] Repeat step S3, calculate the new formula W', and synthesize it immediately. The user smells the adjusted aroma and nods in satisfaction.

[0055] Example 4: Olfactory Fatigue Compensation The system incorporates an olfactory fatigue model. When a user continuously smells the same odor for more than 3 minutes, F(t) gradually increases from 0 to 0.8.

[0056] At this point, the L1 regularization term λ_2·F(t)·||W||_1 in the optimization function will increase significantly, causing the overall weight W to decrease, i.e., the odor concentration will automatically decrease.

[0057] This aligns with the physiological characteristics of human olfactory adaptation, avoiding the discomfort caused by prolonged exposure to high concentrations of odor.

[0058] When the scene changes, F(t) is reset to 0, and the new odor is released at the normal concentration.

[0059] Example 5: Multimodal Fusion In the game scene, the system simultaneously receives: Video frame: A dark dungeon, damp stone walls.

[0060] Audio: dripping water sound, low wind sound.

[0061] Text command: Player enters " / smell danger".

[0062] The multimodal encoder extracts features from three modalities respectively and fuses them through an attention mechanism: V_sem = 0.6·V_visual + 0.3·V_audio + 0.1·V_text The merged V_sem more comprehensively expresses the abstract concept of "dangerous dungeon".

[0063] When mapped to the olfactory space, V_chem exhibits a complex characteristic of "mold + metallic rust + faint bloody smell".

[0064] The system synthesizes and releases this scent, significantly enhancing the player's immersion.

[0065] Example 6: Edge Computing Optimization To reduce latency, the system employs model quantization and pruning techniques: The CLIP encoder was quantized from FP32 to INT8, resulting in a 3x increase in inference speed.

[0066] The mapping matrix M_proj was reduced from 512×128 to 256×64, and the memory usage was halved.

[0067] The optimized solver uses the fast gradient projection algorithm, with a single solution time of <10ms.

[0068] The end-to-end latency from "video frame input" to "odor release" is controlled within 200ms, meeting the requirements for real-time interaction.

[0069] Example 7: Security Protection Mechanism The system has multiple built-in security protections: Concentration limit: Even if the optimization algorithm solves for a high-concentration formula, the system will force the total concentration to be limited to a safe threshold (e.g., total flow rate < 5 ml / min).

[0070] Allergen detection: Users can mark allergenic base fluids (such as "pollen") in the APP, and the system will add a hard constraint of w_i = 0 to the optimization constraints.

[0071] VOC Monitoring: The device has a built-in VOC sensor to monitor the concentration of volatile organic compounds in the air in real time. If the concentration exceeds the standard, the release will be stopped immediately and ventilation will be activated.

[0072] Emergency Stop: Users can immediately interrupt all micro-pumps and start rapid cleaning by speaking "Stop Odor" or pressing a physical button.

[0073] The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art, inspired by the technical solutions of the present invention, can make various modifications and improvements, all of which should be included within the scope of protection of the present invention.

Claims

1. A method for generative odor real-time synthesis based on cross-modal latent space mapping, characterized in that, Comprising the following steps: S1. Data acquisition and encoding step: acquiring a multi-modal data stream in real time in an environment, the multi-modal data stream comprising at least one or more of a sequence of video image frames, audio waveform data, and text instruction data; using a pre-trained multi-modal encoder model to extract features from the multi-modal data stream, outputting a high-dimensional emotional semantic feature vector V_sem; S2. Cross-modal space alignment step: constructing a visual-olfactory joint embedding space, using a trained nonlinear mapping function f(·) to project the emotional semantic feature vector V_sem into the joint embedding space, obtaining a corresponding target odor chemical perception vector V_chem; S3. Generative formula calculation step: acquiring a set of chemical attribute vectors {B_1, B_2,..., B_N} of N basic base liquids currently stored in the device; constructing an optimization objective function with non-negative constraints, calculating a set of mixing weight coefficients W = {w_1, w_2,..., w_N} such that the distance measure between the mixed synthetic vector Σ(w_i·B_i) and the target odor chemical perception vector V_chem is minimized; S4. Dynamic synthesis control step: converting the calculated mixing weight coefficients W into corresponding pulse width modulation PWM signal sequences, controlling the opening duty cycle and frequency of the microfluidic micropump array, synthesizing the target odor in situ in the microfluidic mixing chamber, and releasing it through the atomization module.

2. The method of claim 1, wherein, The visual-olfactory joint embedding space in step S2 is constructed by: Collecting a chemical molecule dataset containing olfactory descriptors and a visual scene dataset containing emotional annotations; Building a double-tower neural network architecture as a visual encoder and an olfactory encoder, respectively; Using a contrast loss function for joint training to maximize the cosine similarity of paired visual and olfactory samples in the embedding space and minimize the similarity of unpaired samples; The target odor chemical perception vector V_chem includes volatility characteristics, fragrance category characteristics, and main functional group characteristics of odor molecules.

3. The method of claim 1, wherein, The optimization objective function constructed in step S3 is: min_W (||Σ(w_i·B_i) - V_chem||_2^2 + λ_1·Σ(w_i / (R_i(t)+ε)) + λ_2·F(t)·||W||_1) The constraint condition is: Σw_i = 1, w_i ≥ 0 Where R_i(t) is the remaining capacity of the i-th base liquid at time t, ε is a zero-smoothing term, F(t) is the user's current olfactory fatigue coefficient, λ_1 is the penalty weight for balanced consumption of base liquids, and λ_2 is the concentration control weight.

4. The method of claim 1, wherein, Step S4 also includes adaptive cleaning logic during scene switching: Establish a sliding time window of length T, and calculate the Euclidean distance rate D_rate of the emotional semantic feature vector V_sem in the window in real time; If D_rate exceeds the preset mutation threshold δ, it is determined that a scene mutation event has occurred; Triggering cleaning interrupt service program: pause the current smell synthesis instruction, control the micro pump connected with cleaning liquid to open at maximum flow rate Q_max for duration Δt, flush the microfluidic mixing chamber, and then execute the next frame of smell synthesis instruction; Wherein, the value of Δt is proportional to the average volatility of the last frame of smell formula.

5. The method of claim 1, wherein, In step S4, when converting the mixing weight coefficient W into a PWM signal sequence, a random jitter strategy is adopted: Under the premise of keeping the total duty cycle unchanged, the opening phase of each channel micro pump is randomly staggered to avoid the superposition of fluid pressure pulses caused by the simultaneous opening of multiple micro pumps, ensuring the stability of the flow field in the microfluidic chip.

6. The method of claim 1, wherein, The method further comprises a user feedback fine-tuning step: Receiving user voice feedback instructions; Analyzing the emotional polarity of the feedback instructions; If it is negative feedback, generate a correction vector ΔV according to the feedback content, update the target smell chemical perception vector V'_chem = V_chem + α·ΔV, where α is the learning rate, and re-execute step S3.

7. A cross-modal latent space mapping based generative odor real-time synthesis system, characterized in that, Comprise: A multi-modal perception module for collecting environmental video, audio and user voice data; An AI inference computing module with a built-in memory and processor, storing a computer program for executing the method steps of any one of claims 1 to 6; A lower computer control module including a microprocessor and a drive circuit for receiving the formula instructions from the AI inference computing module and converting them into PWM control signals; A microfluidic execution module including a base liquid storage bin, a cleaning liquid bin, a micro piezoelectric pump array, a microfluidic mixing chip, and an ultrasonic atomizer; Wherein, the AI inference computing module communicates with the lower computer control module through a serial port or Ethernet, the lower computer control module controls the microfluidic execution module through an I2C bus, the microfluidic mixing chip has a multi-entry single-outlet structure and is internally provided with a microstructure for fluid chaotic convection.

8. A computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the method steps of any one of claims 1 to 7.

Images