Interspecies communication systems and programs

Abstract

We provide an interspecies communication system and program that enables high-fidelity interspecies communication. [Solution] The interspecies communication system 100 includes an interface device 10 that detects information of multiple different types of modalities output by the organism and converts the detected modality information into input information, and a translation device 30 that inputs the input information into the trained model and generates translation information that the organism can receive based on the output information output from the trained model. The interface device 10 detects information of multiple different types of modalities, which include at least one of either chemical substances or olfactory information, the organism's biological information, auditory information, or visual information, either the chemical substance or the olfactory information, or the organism's biological information.

Description

Technical Field

[0001] The present invention relates to a heterogeneous communication system and a program.

Background Art

[0002] In recent years, attempts have been made to decode the communication of organisms other than humans using artificial intelligence technology, particularly machine learning. For example, a project (e.g., Project CETI) is known in which a large amount of acoustic signals (called "codas") emitted by specific marine organisms (e.g., sperm whales) are collected and their patterns are analyzed using machine learning (Non-Patent Document 1). There are also attempts to analyze the bioacoustics (bioacoustics) data of various organisms using AI (artificial intelligence) and elucidate their linguistic structures.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technology described in Non-Patent Document 1 primarily focuses on the analysis of acoustic data and associated behavioral data emitted by living organisms. However, biological communication involves a complex combination of multiple modalities, including not only sound and vision, but also chemical substances (such as pheromones) and olfaction. For example, even if the acoustic and behavioral information are the same, the true "context" and "meaning" of the communication may be completely different depending on the presence or absence of chemical substances emitted simultaneously. Conventional systems that rely solely on acoustic and visual information, such as those described in Non-Patent Document 1, cannot eliminate this "ambiguity of context," and have limitations in accurately elucidating communication protocols. In other words, it has been difficult to achieve high-fidelity interspecies communication. In this disclosure, "communication" is described as a concept that includes not only the transmission of information, meaning, or emotion in both directions, but also the transmission of information, meaning, or emotion in one direction. When two-way communication is intended, it is described as "two-way communication."

[0005] This invention has been made in view of the above problems, and aims to provide an interspecies communication system and program that realizes high-fidelity interspecies communication. [Means for solving the problem]

[0006] To solve the above problems, the interspecies communication system inputs information of multiple different modalities output by an organism as learning information into a learning model, and then inputs input information from an organism of a different species into a trained model that has been machine-learned using an unsupervised learning method, and outputs translation information that the organism can receive based on the output information output from the trained model. An interface device that detects information of multiple different types of modalities output by the organism and converts the detected modality information into input information, The system includes a translation device that inputs the aforementioned input information into the trained model and generates translation information that can be received by the organism based on the output information output from the trained model, The interface device is, Either chemical information or olfactory information, The biological information of the aforementioned organism, Auditory information and, Visual information and, The system detects information of the multiple different types of modalities, including at least one of the chemical substances, the olfactory information, or the biological information of the organism.

[0007] Furthermore, the program is a program that, by inputting information of multiple different types of modalities output by an organism as learning information into a learning model, and using an unsupervised learning method, input information from a different species of organism into a trained model, and then causes a processor of an interspecies communication system to execute control processing, which outputs translation information that the organism can receive based on the output information output from the trained model, An interface device that detects information of multiple different types of modalities output by the organism and converts the detected modality information into input information, Either chemical information or olfactory information, The biological information of the aforementioned organism, Auditory information and, Visual information and, The input information is acquired from an interface device that detects at least one of the following: the chemical substance, the olfactory information, or the information of the multiple different types of modalities, including the biological information of the organism. The input information is input to the trained model, and based on the output information output from the trained model, translation information that the organism can receive is generated. [Effects of the Invention]

[0008] The above configuration makes it possible to achieve high-fidelity interspecies communication. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows a system 100 according to one embodiment. [Figure 2] Figure 2 shows an interface device 10 according to one embodiment. [Figure 3] Figure 3 shows a translation device 30 according to one embodiment. [Modes for carrying out the invention]

[0010] One embodiment of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to the following embodiments, and design modifications can be made as appropriate within the scope of satisfying the configuration of the present invention. Furthermore, in the following description, the same reference numerals are used in common across different drawings for the same parts or parts having similar functions, and repeated explanations are omitted. Also, the configurations described in the embodiments and modifications may be combined or modified as appropriate. Furthermore, in order to make the explanation easier to understand, the configurations in the drawings referenced below are simplified or schematic, and some components are omitted.

[0011] In this disclosure, "auditory information" means information based on acoustic signals emitted by living organisms and is used synonymously with "acoustic information." Furthermore, "chemical or olfactory information" is a general term for chemical substances (such as pheromones) released by living organisms and information detected olfactorily by them, and is also called "chemical-olfactory information."

[0012] In the present disclosure, the "organism" refers to the organism to which the translated information is presented by the present system, and is not limited to humans, but includes any species of organism. The "alien organism" means an organism belonging to a species different from the said "organism". For example, the organism may be an animal (e.g., a mammalian animal such as a whale), and the alien organism may be a human, or vice versa. Further, the "organism" includes plants such as insectivorous plants, extraterrestrial organisms, and organisms without a scientific name, etc.

[0013] In the present disclosure, the "biological information" is information within an organism, including any one of blood pressure, body temperature, pulse, respiratory rate, and internal secretions, etc.

[0014] In the interspecies communication system according to one aspect of the present disclosure, the interface device may include any one of an artificial olfactory sensor, a gas chromatography analyzer, or a mass spectrometer that detects the chemical substance or the olfactory information (second configuration).

[0015] In the interspecies communication system according to one aspect of the present disclosure, the translation device may have the learned model constructed using any one of a variational autoencoder, a self-organizing map, or an adversarial generation network as the method of the unsupervised learning (third configuration). This learned model autonomously extracts a common representation in its latent space based on the multimodal input information of an organism.

[0016] In the interspecies communication system according to one aspect of the present disclosure, the translation device may be configured to identify the communication protocol by the organism based on the temporal or contextual correlation existing between the auditory information, the visual information, and the chemical substance or the olfactory information by clustering (fourth configuration). In this case, on the latent space of the learned model, a group of data having similar multimodal patterns is autonomously identified as a cluster of protocol units.

[0017] In the cross-species communication system according to one aspect of the present disclosure, the interface device may further include an output unit that generates information receivable by an organism, including at least one of text, voice, image, motion, chemical substance, light, and electromagnetic wave, based on the output information obtained from the translation device (fifth configuration).

[0018] In the cross-species communication system according to one aspect of the present disclosure, the translation device may further include a bidirectional translation unit that converts input information from an organism into output information receivable by the different organism (sixth configuration). Thereby, bidirectional translation from an organism to a different organism and from a different organism to an organism becomes possible.

[0019] [Configuration of Cross-Species Communication System 100] (Fig. 1: Overall Configuration Diagram of the System) As shown in Fig. 1, the cross-species communication system 100 (hereinafter referred to as "system 100") includes an interface device 10, a communication device 20, and a translation device 30. The target organism B1 outputs auditory information A, visual information V, and chemical substance / olfactory information C (e.g., pheromone). The cross-species communication system 100 includes a processor that executes control processing. The cross-species communication system 100 is provided with a memory in which a program is stored. The program causes the processor to execute each control processing described below.

[0020] (Fig. 2: Details of the Interface Device) As shown in Fig. 2, the interface device 10 converts the information output from the target organism B1 into input information (input information). Further, the interface device 10 presents the output information (output information) generated by the translation device as information (translated information) recognizable by the target organism B2.

[0021] The auditory information sensing unit 11 is composed of a broadband microphone array or the like, collects acoustic signals (voices) emitted by an organism, and acquires bioacoustic data (auditory information).

[0022] The visual information sensing unit 12 consists of a 4K / 8K camera, a depth sensor, etc., and acquires behavioral data (visual information) such as the posture and movement of living organisms.

[0023] The chemical and olfactory information sensing unit 13 may consist of, for example, an artificial olfactory sensor that detects trace amounts of volatile organic compounds (VOCs), a gas chromatography-mass spectrometer (GC-MS) that identifies the molecular structure of a specific chemical substance (such as a pheromone), or a biosensor array that reacts to a specific chemical substance.

[0024] The biological information sensing unit 13a includes a blood pressure monitor (pressure sensor, optical sensor, acoustic sensor, or a detector combining these) for detecting blood pressure, a thermometer (temperature sensor, infrared sensor, or a detector combining these), a pulse meter (optical sensor, pressure sensor, electrical biosignal detection sensor, or a detector combining these), a respiratory rate monitor (accelerometer, temperature sensor, acoustic sensor, electrical impedance sensor, or a detector combining these), and a bodily secretion meter (chemical sensor, electrochemical sensor, biosensor, or a detector combining these).

[0025] The input conversion unit 14 converts the analog and digital data obtained from the auditory information sensing unit 11, the visual information sensing unit 12, and the chemical / olfactory information sensing unit 13 into a unified digital input information format that the translation device 30 can process (for example, a multidimensional data vector sequence with timestamps aligned along a time axis), and sends it to the communication device 20.

[0026] The output unit 15 is a device that presents information recognizable by organism B2, based on the digital output information generated by the translation device 30, which is received via a communication device. The output unit 15 may consist of a speaker, a display, a chemical substance (pheromone) diffuser, an LED light source, or a motor-driven feedback device.

[0027] The interface device 10 may also include an operation unit (touch panel, microphone, keyboard, etc.) that accepts input (text, voice, etc.) from a human user, and this input is used in the bidirectional translation function unit.

[0028] (Figure 3: Details of the translation device) The translation device 30 shown in Figure 3 is, for example, a server computer equipped with a GPU, and plays a central role in the AI ​​(artificial intelligence) processing of this disclosure. The server computer may be on-premises or a cloud server.

[0029] The training data 31 is an unlabeled, multimodal dataset collected time-series from the interface device 10. Specifically, it takes the form of time-series data in which a vector [A(t),V(t),C(t),B(t)] consisting of auditory information A(t), visual information V(t), chemical / olfactory information C(t), and biological information B(t) at time t is continuously accumulated over a certain period.

[0030] The translation model (trained model, artificial intelligence model) 32 is pre-trained using the above training data 31 through unsupervised learning. For example, architectures such as variational autoencoders (VAEs), self-organizing maps (SOMs), and generative adversarial networks (GANs) are employed to enable the extraction of common latent representations from multimodal data.

[0031] The protocol elucidation unit 33 processes the training data 31 in the translation model 32 as input and forms a latent space common to each modality (sound, visual, chemical, and bio-information). The protocol elucidation unit 33 performs clustering processing (e.g., k-means clustering, hierarchical clustering, density-based clustering, etc.) on this latent space to identify data sets with similar multimodal patterns as clusters on a protocol basis.

[0032] The output generation unit 34 generates output information to be output from the interface device 10 based on the protocol identified by the protocol elucidation unit 33 (e.g., cluster 1 = "threat"). This output information may be a human-readable translation (e.g., text information such as "threatening") or feedback to the target organism B2 (e.g., a synthesized signal combining at least two of visual information, auditory information, chemical information, and biological information to induce "play"). The interface device 10 may also include a drive device (e.g., a device that transmits information through voice or movement, such as a robot) as an output unit 15. The drive device transmits output information to the target organism B2 by emitting a voice (outputting auditory information) or by driving (operating) its body (hands, arms, legs, feet, head, and body) with a motor (emitting visual information). The drive device is an example of a "feedback device". The robot may have a shape that mimics a living organism (e.g., an animal), but it may also have a shape other than that.

[0033] (Specific example of operation) This section will specifically explain how the translation device 30 of this embodiment produces remarkable results.

[0034] [scenario] Let's consider a scenario where target organism B1 (let's assume it's a primate) emits "Vocal X" (a sharp, high-pitched sound) and performs "Action Y" (showing its teeth).

[0035] [Limitations of systems (acoustic and visual only) based on comparative examples] This combination of "sound X" and "behavior Y" has been observed in past observations in both "playful (pretending to threaten)" and "genuine threat" situations.

[0036] In the comparative example, a system based solely on acoustic and visual information cannot distinguish between these two contexts and is forced to treat the signal as a single ambiguous state. As a result, it either fails to decipher the protocol or outputs an incorrect translation, such as "playing" or "angry," with a 50% probability.

[0037] [Example of the operation of system 100 (including chemical substance or olfactory information) of this embodiment] (Case 1: Serious intimidation) The interface device 10 detects [sound X] with the auditory information sensing unit 11, detects [behavior Y] with the visual information sensing unit 12, and simultaneously detects [chemical substance K (stress pheromone)] with the chemical / olfactory information sensing unit 13.

[0038] The input information generated at this time is, for example, I1(t)=[A(t)=Voice X, V(t)=Behavior Y, C(t)=Chemical K] It is sent to the translation device 30 as a multimodal vector sequence like this.

[0039] (Case 2: Play) In a different scenario, if the target organism B1 exhibits the same [Sound X] and [Behavior Y] in a context interpreted as "playing," the interface device 10 will detect [Sound X] with the auditory information sensing unit 11 and [Behavior Y] with the visual information sensing unit 12, but the chemical / olfactory information sensing unit 13 will detect [Chemical Substance K (Not Detected)].

[0040] The input information generated at this time is, for example, I2(t)=[A(t)=Voice X,V(t)=Action Y,C(t)=0] This can be expressed as follows (where C(t)=0 indicates that no stress pheromones are detected).

[0041] [Protocol elucidation using unsupervised learning] The protocol elucidation unit 33 of the translation device 30 clusters a large number of multimodal data, including the input information I1 and I2, in an unsupervised (unlabeled) manner.

[0042] As a result, the model autonomously discovers two distinct clusters, based on the fact that while they share the common elements of voice X and behavior Y, the data sets where the chemical / olfactory information C(t) component is [chemical substance K] and the data sets where it is [0] are clearly different.

[0043] Cluster 1: Groups where chemical substance K is included in chemical substance / olfactory information C(t) (= "serious intimidation" protocol)

[0044] Cluster 2: The group where the chemical substance / olfactory information C(t) is 0 (= "play" protocol)

[0045] [Conclusion (Achievement of remarkable effects)] Thus, the system 100 of this embodiment succeeds in clearly separating and clarifying the "combination of voice X and behavior Y," which the comparative system could only recognize as a single ambiguous signal, into two different communication protocols ("serious intimidation" and "play") by adding a decisive input in the form of chemical substance / olfactory information C(t).

[0046] This demonstrates a concrete correlation between input data (multimodal data) and output data (the elucidated protocol), yielding remarkable effects that are difficult to predict from the analysis of acoustic and visual information alone in the comparative example. While the above example uses chemical and olfactory information C(t), using biological information B(t) instead of (or in addition to) chemical and olfactory information C(t) can also produce remarkable effects that are difficult to predict from the analysis of acoustic and visual information alone in the comparative example.

[0047] (Output and two-way communication) The output information generated by the output generation unit 34 of the translation device 30 is sent back to the interface device 10 via the communication device 20. The interface device 10 has an output unit 15.

[0048] (Translation for humans) If the output information is translation information intended for humans, the output unit 15 is, for example, a display, a speaker, or a feedback device driven by a motor. The feedback device includes a drive device (for example, a robot) as described above.

[0049] If the protocol analysis unit 33 determines that the input information is a "threat protocol" (Case 1), the output generation unit 34 can generate text alert information such as "Warning: Target organism B1 is experiencing severe stress and is threatening," and display it on the interface device 10's display.

[0050] (Feedback to living organisms) When the output information is feedback information for organisms other than humans, the output unit is, for example, a speaker, a display, a chemical (pheromone) diffuser, an LED light source, or a feedback device driven by a motor. The feedback device includes a drive device (e.g., a robot) as described above.

[0051] For example, if a human inputs the intention of "friendship" into the system, the translation device 30 generates output information corresponding to a learned "friendship protocol" (e.g., a combination of a specific sound Y and chemical substance Z). Based on this information, the output unit 15 of the interface device 10 plays "sound Y" from a speaker and simultaneously sprays "chemical substance Z" from a diffuser. This attempts to transmit intentions from humans to living organisms across species.

[0052] As described above, this embodiment makes it possible to eliminate the "contextual ambiguity" in biological communication, which is difficult to distinguish by analyzing only acoustic and visual information. For example, even if the acoustic and visual information is the same, by simultaneously analyzing chemical and olfactory information (e.g., the presence or absence of alarm pheromones), the AI ​​model can autonomously (unsupervisedly) distinguish whether it is a "threat" protocol or a "play" protocol. This is a remarkable effect that cannot be predicted by analyzing information from each modality individually, and is only achieved by integrating and analyzing different modalities (especially chemical substances).

[0053] This disclosure is not limited to the embodiments described above, and various modifications are possible within the scope of the technical concept of the present invention. For example, the interface device, communication device, and translation device may be housed in a single enclosure, or they may be processed in a distributed manner using cloud computing. Furthermore, the learning method is not limited to unsupervised learning, and the accuracy of protocol elucidation may be further improved by combining semi-supervised learning or reinforcement learning. In addition, it is possible to add or change the target biological species and the type of modality applied (temperature, pressure, electrical signals, etc.). [Explanation of Symbols]

[0054] 10: Interface device, 11: Auditory information sensing unit, 12: Visual information sensing unit, 13: Chemical / Olfactory information sensing unit, 13a: Biological information sensing unit, 14: Input conversion unit, 15: Output unit, 20: Communication device, 30: Translation device, 31: Training data, 32: Translation model, 33: Protocol elucidation unit, 34: Output generation unit, 100: Interspecies communication system

Claims

1. An interspecies communication system that inputs information of multiple different modalities output by an organism as training information into a learning model, and then inputs input information from a different species of organism into a trained model that has been machine-learned using an unsupervised learning method, and outputs translation information that the organism can receive based on the output information output from the trained model, wherein the system outputs translation information that the organism can receive, An interface device that detects information of multiple different types of modalities output by the organism and converts the detected modality information into input information, The system includes a translation device that inputs the aforementioned input information into the trained model and generates translation information that can be received by the organism based on the output information output from the trained model, The interface device is, Either chemical information or olfactory information, The biological information of the aforementioned organism, Auditory information and, Visual information and, The system detects information of multiple different types of modalities, including at least one of the chemical substances or the olfactory information, or the biological information of the organism. The trained model has a latent space that extracts a common latent representation from information of multiple different types of modalities. The aforementioned translation device, A protocol elucidation unit identifies data sets having similar multimodal patterns as protocol-based clusters by performing clustering processing on the latent space using information of multiple different types of modalities as input. An interspecies communication system comprising: an output generation unit that generates output information to be output from the interface device based on the cluster identified by the protocol deciphering unit.

2. The interspecies communication system according to claim 1, wherein the interface device includes an artificial olfactory sensor, a gas chromatography analyzer, or a mass spectrometer for detecting the chemical substance or the olfactory information.

3. The interspecies communication system according to claim 1 or 2, wherein the translation device has the trained model constructed using one of the following methods for unsupervised learning: a variational autoencoder, a self-organizing map, or a generative adversarial network.

4. The interspecies communication system according to claim 1 or 2, wherein the translation device is configured to identify the protocol of communication by the organism by clustering based on the temporal or contextual correlations that exist between the auditory information, the visual information, and the chemical substance or olfactory information.

5. The interspecies communication system according to claim 1 or 2, further comprising an output unit that generates information receivable by the organism, based on the output information, including at least one of text, voice, images, actions, chemical substances, light, and electromagnetic waves.

6. The interspecies communication system according to claim 1 or 2, further comprising a bidirectional translation unit that converts input information from the organism into output information that can be received by the organism of a different species.

7. A program that, by inputting information of multiple different modalities output by an organism as learning information into a learning model, and using an unsupervised learning method, inputs information from a different species of organism into a trained model, and then causes a processor of an interspecies communication system to execute control processing, which outputs translation information that the organism can receive based on the output information output from the trained model, An interface device that detects information of multiple different types of modalities output by the organism and converts the detected modality information into input information, Either chemical information or olfactory information, The biological information of the aforementioned organism, Auditory information and, Visual information and, The input information is obtained from an interface device that detects at least one of the following: the chemical substance, the olfactory information, or the biological information of the organism, and includes information from the plurality of different modalities. The input information is input to the trained model, and based on the output information output from the trained model, translation information that the organism can receive is generated. The trained model has a latent space that extracts a common latent representation from information of multiple different types of modalities. The aforementioned program, By performing clustering processing on the aforementioned latent space using information from multiple different types of modalities as input, data sets having similar multimodal patterns are identified as protocol-based clusters. A program that generates output information to be output from the interface device based on the identified cluster.

Images