Method for converting audiovisual data into olfactory experience based on distributed processing

Abstract

The invention provides a method for converting audio-visual data into olfactory experience based on distributed processing and a method for realizing high-speed and low-delay sound classification by using the distributed processing. A system comprises a data center, a cloud server, a data set, media content, a base station, a game computer, an earphone and an odor generator. The method can be widely applied to scenes such as games, movies, media and the like, and corresponding smell data, such as gunshots appearing in movie scenes or pictures emitted by firearms, can be obtained by identifying sounds or images in the scenes through artificial intelligence; therefore, the corresponding aromatic compounds can be mixed according to the gunpowder odor data obtained by artificial intelligence analysis through the odor generation device, and then corresponding odor is generated and released, so that a user can experience olfactory experience except visual sense and auditory sense, and better on-site experience is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a system and method for rapidly converting video or sound data into odor experience through artificial intelligence technology. Background Art

[0002] With the advancement of technology, people's demand for entertainment experience is getting higher and higher. Hearing and vision are the key factors of entertainment. Movies, TV programs, video games and other forms of entertainment require high-quality sound effects to enhance the audience's entertainment experience. With the development of technology, the high resolution of the display improves the visual sense, the surround sound system enhances the hearing, and even the bass system makes people feel vibrations. In addition to the visual and auditory experience, the olfactory experience has also begun to receive attention. However, smell perception is rarely achieved in current entertainment systems.

[0003] However, the application of olfactory entertainment still faces some challenges, such as how to accurately control and deliver the aroma and taste, how to maintain the stability and consistency of the smell, etc. Therefore, olfactory entertainment still needs more research and development to achieve a better user experience.

[0004] Today, entertainment systems with olfactory information are using predefined media data or metadata to trigger the mechanism of generating odors, and the odors generated are relatively fixed. When the entertainment system plays a predetermined scene, the odor generating device will generate the odor of the relevant scene. However, it is difficult to process odor information in real time with sound data because of the high data computing speed required for the system to generate odors based on the use of real-time sound data, analysis and classification.

[0005] Existing sound classification technology usually needs to be run on a local computer. This method has disadvantages such as slow processing speed, high latency, and heavy computing load. Especially when a large amount of data needs to be processed, it will place a heavy burden on the local computer, causing the computer to run slowly.

[0006] In order to solve this problem, cloud computing technology and distributed system technology have been widely used in recent years. Cloud computing technology can concentrate computing and storage resources on the cloud, providing powerful computing and storage capabilities, and can effectively solve the pressure and delay problems of local computers. Distributed system technology can distribute data and computing tasks to multiple computing nodes for processing, and has high fault tolerance and scalability, which can better support large-scale data processing.

[0007] Therefore, distributing sound and image classification calculations to distributed platforms for processing can greatly improve computing efficiency and processing speed, reduce latency, and better manage and store large amounts of media content and data sets. Such technical applications will have a huge impact and application value in the fields of audio processing, audio recognition, and entertainment. Summary of the invention

[0008] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for converting audiovisual data into olfactory experience based on distributed processing, and a method for high-speed, low-latency sound classification using distributed processing. The system includes a data center, a cloud server, a data set, media content, a base station, a gaming computer, headphones, and an odor generator.

[0009] The present invention is achieved by the following technical means: A method for converting audio-visual data into olfactory experience based on distributed processing, comprising a cloud server, a distributed processing base station, an audio-visual media content storage center, an audio-visual playback unit and an odor generating device; the cloud server is deployed with an artificial intelligence model for converting audio-visual data into odor data, and the distributed processing base station is deployed with an intelligent analysis model for an odor feature data set; the audio-visual media playback unit reads video media content from the audio-visual media content storage center for playback, and simultaneously sends the audio-visual media content to the distributed processing base station, the intelligent analysis model in the distributed processing base station analyzes images and / or audio in the audio-visual media content to obtain olfactory data corresponding to the images and / or audio and transmits the olfactory data to the odor generating device to release the olfactory experience; when the odor feature data set of the intelligent analysis model cannot be analyzed to obtain the corresponding olfactory data, the relevant audio-visual media content is forwarded to the cloud server, and the artificial intelligence model of the cloud server extracts the odor feature data set related to the image and / or audio in the audio-visual media content, and updates the relevant odor feature data set to the intelligent analysis model in the distributed processing base station.

[0010] The step of converting the audio signal in the audio-visual media into olfactory data includes: A1. Capture audio signals from audio-visual media; A2. Normalize the audio signal; A3. Perform Fourier transform on the audio signal to obtain a time-frequency domain signal; A4. Input the time-frequency domain signal into the intelligent analysis model or artificial intelligence model to obtain the corresponding sound classification data; A5. Read the olfactory data corresponding to the sound classification data.

[0011] The step of converting the image signal in the audio-visual media into olfactory data includes: B1. Capture image signals from audio-visual media; B2. Preprocessing the image signal to form an image matrix related to the pixel intensity in the image signal; B3. Input the image matrix into the intelligent analysis model or artificial intelligence model to obtain image classification data; B4. Read the olfactory data corresponding to the image classification data.

[0012] Among them, the distributed processing base station, audio-visual playback unit and odor generating device are arranged locally, the cloud server is arranged remotely, and the audio-visual media content storage center is arranged locally or remotely.

[0013] The local devices are connected via wireless data transmission, including Wifi, Bluetooth, mobile network, and radio frequency transmission.

[0014] Beneficial effects of the present invention: The cloud server uses a large number of media contents to learn, process and analyze through artificial intelligence, and extracts the characteristic data set of the relevant odor and stores it in the cloud server. The distributed processing base station will regularly update the simplified index of the characteristic data set of the odor from the cloud server, and can quickly and efficiently extract the specified number of features and store them in the distributed processing base station. When the game computer streams and downloads the video to play or plays the game, the relevant audio and / or image signals will first be transmitted to the local base station for analysis. The base station will use the embedded low-cost processor (GPU) in the base station to process the input audio and / or image signals in a parallel / streaming manner. Since the base station is generally set locally, that is, it is connected to the same domain network as the game computer, headphones, odor generating device and other devices, thereby reducing the waiting time / delay of the trigger signal for analysis. When there is a feature in the audio and / or image signal that does not exist in the base station, the relevant audio and / or image signal is transmitted to the cloud server for joint operation to extract the characteristic data set of the audio and / or image signal, and transmit the relevant characteristic data set to the base station. At the same time, the cloud server will regularly update the characteristic data set obtained by the operation to all base stations. As a result, distributed processing can be used to greatly improve processing speed and efficiency, reduce the pressure and delay of local base station calculations, and achieve high-speed, low-latency sound classification. In addition, by using data centers and cloud data centers, large amounts of media content and data sets can be effectively stored and managed, which has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention is further described below in conjunction with the accompanying drawings: Figure 1 It is a schematic diagram of the structure of the present invention; Figure 2 It is a flow chart of converting audio signals into olfactory data according to the present invention. DETAILED DESCRIPTION

[0016] Reference Figure 1 As shown, a method for converting audio-visual data into olfactory experience based on distributed processing includes a cloud server, a distributed processing base station, an audio-visual media content storage center, an audio-visual playback unit, and an odor generating device. A GPU server with high computing power is deployed in the cloud server, and an odor feature data set corresponding to the image and / or audio in the audio-visual media content is obtained after training and computing the artificial intelligence model for converting audio-visual data into odor data, and the same model and algorithm as the base station are run to train new data, and more complex models are run and calculated, such as multi-layer DL models (CNN, RNN type) for data set training and complex data prediction. The odor feature data set corresponding to the image and / or audio obtained after running or calculating on the cloud server is updated to the distributed processing base station.

[0017] The distributed processing base station, the audio-visual playback unit and the odor generating device are arranged in the local network. The video playback unit reads the content to be played from the audio-visual media content storage center. The relevant content includes images and / or audio data. The images and audio data are transmitted to the video playback unit and the distributed processing base station of the local network at the same time. Since the local network transmission speed is relatively fast, the response of the system can be improved. At the same time, it will not cause network congestion by initiating too many access requests to the cloud server, and reduce the computing pressure of the cloud server. The distributed processing base station is also deployed with an intelligent analysis model of the odor feature data set, which can obtain the corresponding odor data after the image and audio data played by the audio-visual playback unit is calculated by the intelligent analysis model in real time, and then transmit it to the odor generating device to release the olfactory experience, thereby increasing the user's presence experience.

[0018] The following describes in detail how the distributed processing base station and the cloud server convert the audio signal in the audio-visual media into olfactory data. Figure 2 , the specific steps are as follows: A1. The audio-visual media content storage center sends the audio-visual media according to the request of the audio-visual playback unit, and at the same time transmits the audio signal in the audio-visual media to the distributed processing base station or cloud server. Usually, the audio signal is transmitted to the distributed processing base station or cloud server in real time in the form of .wav files or analog signals.

[0019] A2. The distributed processing base station or cloud server performs normalization processing on the audio signal, including sampling the audio signal, intercepting the audio signal of a specified length, etc.

[0020] A3. Perform Fourier transform (short-time Fourier transform) on the audio signal to convert the time domain representation of the audio into a time-frequency domain signal.

[0021] A4. Input the time-frequency domain signal into the intelligent analysis model or artificial intelligence model. The intelligent analysis model in the distributed processing base station and the artificial intelligence model in the cloud server input the time-frequency domain signal into the convolutional neural network (CNN) to obtain feature maps, and then the classifier predicts the category of the feature to obtain the classification data corresponding to the sound and the smell; A5. The olfactory data is read according to the above classification data and then sent to the odor generating device. The odor generating device mixes different odor generating units according to corresponding proportions based on the olfactory data to obtain the odor corresponding to the olfactory data and releases it.

[0022] The following describes in detail how the base station converts the image signal in the audio-visual media into olfactory data. The specific steps are as follows: B1. The audio-visual media content storage center sends the audio-visual media according to the request of the audio-visual playback unit, and at the same time transmits the image signal in the audio-visual media to the distributed processing base station or cloud server; B2. The distributed processing base station or cloud server preprocesses the image signal to form an image matrix associated with the pixel intensity in the image signal; B3. Input the image matrix into an intelligent analysis model or an artificial intelligence model, which includes a convolutional neural network (CNN) operation to obtain feature maps, and then the classifier predicts the category of the features to obtain classification data corresponding to the image and the odor; B4. Read the olfactory data according to the above classification data and send it to the odor generating device. The odor generating device mixes different odor generating units according to corresponding proportions based on the olfactory data to obtain the odor corresponding to the olfactory data and releases it.

[0023] The present invention can be widely used in scenes such as games, movies, and media. The corresponding odor data can be obtained by using artificial intelligence to recognize the sounds or images in the scenes. For example, when there are gunshots or images of gunfire in a movie scene, the odor generating device can be used to mix the corresponding aromatic compounds according to the gunpowder odor data obtained through artificial intelligence analysis to generate and release the corresponding odor, allowing users to experience olfactory experience in addition to visual and auditory experience, thereby achieving a better on-site experience.

[0024] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of protection of the present disclosure. All should belong to the protection scope of the present invention. Within the protection scope of the present invention, its technical scheme and / or implementation method can have various modifications and changes.

Claims

1. A method for converting audio-visual data into olfactory experience based on distributed processing, characterized in that: It includes a cloud server, a distributed processing base station, an audio-visual media content storage center, an audio-visual playback unit and an odor generating device; the cloud server is deployed with an artificial intelligence model for converting audio-visual data into odor data, and the distributed processing base station is deployed with an intelligent analysis model of an odor feature data set; the audio-visual media playback unit reads video media content from the audio-visual media content storage center for playback, and sends the audio-visual media content to the distributed processing base station at the same time, the intelligent analysis model in the distributed processing base station analyzes the image and / or audio in the audio-visual media content to obtain olfactory data corresponding to the image and / or audio and transmits it to the odor generating device to release the olfactory experience; when the odor feature data set of the intelligent analysis model cannot be analyzed to obtain the corresponding olfactory data, the relevant audio-visual media content is forwarded to the cloud server, and the artificial intelligence model of the cloud server extracts the odor feature data set related to the image and / or audio in the audio-visual media content, and updates the relevant image and / or audio feature data set to the intelligent analysis model in the distributed processing base station.

2. The method for converting audio-visual data into olfactory experience based on distributed processing according to claim 1, characterized in that: The steps of converting audio signals in audio-visual media into olfactory data include: A1. Capture audio signals from audio-visual media; A2. Normalize the audio signal; A3. Perform Fourier transform on the audio signal to obtain a time-frequency domain signal; A4. Input the time-frequency domain signal into the intelligent analysis model or artificial intelligence model to obtain the corresponding sound classification data; A5. Read the olfactory data corresponding to the sound classification data.

3. The method for converting audio-visual data into olfactory experience based on distributed processing according to claim 1, characterized in that: The steps of converting the image signal in the audio-visual media into olfactory data include: B1. Capture image signals from audio-visual media; B2. Preprocessing the image signal to form an image matrix related to the pixel intensity in the image signal; B3. Input the image matrix into the intelligent analysis model or artificial intelligence model to obtain image classification data; B4. Read the olfactory data corresponding to the image classification data.

4. The method for converting audio-visual data into olfactory experience based on distributed processing according to claim 1, characterized in that: The distributed processing base station, the audio-visual playback unit and the odor generating device are arranged locally, the cloud server is arranged remotely, and the audio-visual media content storage center is arranged locally or remotely.

5. The method for converting audio-visual data into olfactory experience based on distributed processing according to claim 4, characterized in that: The local devices are connected via wireless data transmission, including Wifi, Bluetooth, mobile network, and radio frequency transmission.

Images