Information processing device, method, and program

Abstract

This technology provides a way to prevent the unintentional leakage of personal information due to odors. [Solution] The information processing device 103 of the present disclosure includes: an assumed category determination unit 107 that determines an assumed odor category to which the assumed odor belongs based on an image; a detected odor category determination unit 108 that determines a detected odor category to which the odor detected by the odor sensor 102 belongs; an odor selection unit 109 that selects odor parameters from among the odor parameters indicating the odor detected by the odor sensor 102 that belong to an odor category included in both the assumed odor category and the detected odor category; and a transmission unit 110 that transmits odor selection information indicating the selected odor parameters.

Description

Technical Field

[0001] This disclosure relates to information processing technology.

Background Art

[0002] In Internet content such as VR, in addition to vision and hearing, expressions using smell have attracted attention and are being studied. Also, research on digitization of olfactory information is progressing, and it is becoming possible to acquire smells, convert them into data, and reproduce the smells remotely.

[0003] There is content that transmits a video with smell information attached and reproduces video, audio, and smell on the receiving side for enjoyment. For example, in Patent Document 1, when a smell generation terminal receives smell identification information included in a web page such as a video, it receives, from a server, formulation information indicating a formulation of smell basic element components for generating the smell specified by the smell identification information, formulates the smell basic element components based on the formulation information, and generates a desired smell. A smell generation system is described.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] As the digitization of smell information makes the smell information in the transmitted data approach the real information acquired by the sensor, a lot of personal information will be included in it. Similar to hearing and vision, it is necessary to protect personal information that enters through smell unintentionally.

[0006] In view of the above problems, an object of this disclosure is to provide a technology that can suppress the unintentional leakage of personal information due to smell. [Means for solving the problem]

[0007] To solve the above problems, an information processing device in one embodiment of the present disclosure includes: an assumed category determination unit that determines an assumed odor category to which an assumed odor belongs based on an image; a detected odor category determination unit that determines a detected odor category to which an odor detected by an odor sensor belongs; an odor selection unit that selects odor parameters from among odor parameters indicating an odor detected by the odor sensor that belong to a category included in both the assumed odor category and the detected odor category; and a transmission unit that transmits odor selection information indicating the selected odor parameters and the captured image.

[0008] A method in another aspect of the present disclosure includes determining an assumed odor category, which is the odor category to which the assumed odor belongs, based on an image; determining a detected odor category, which is the odor category to which the odor detected by the odor sensor belongs; selecting odor parameters from among the odor parameters indicating the odor detected by the odor sensor that belong to the odor category included in both the assumed odor category and the detected odor category; and transmitting odor selection information indicating the selected odor parameters.

[0009] A program in yet another aspect of the present disclosure causes a computer to perform the following actions: determine an assumed odor category, which is the odor category to which an assumed odor belongs, based on an image; determine a detected odor category, which is the odor category to which an odor detected by an odor sensor belongs; select odor parameters from among the odor parameters indicating the odor detected by the odor sensor that belong to the odor category which is included in both the assumed odor category and the detected odor category; and transmit odor selection information indicating the selected odor parameters.

[0010] Furthermore, any combination of the above components, as well as conversions of the expressions of this disclosure between methods, apparatus, systems, recording media, computer programs, etc., are also valid as aspects of this disclosure. [Effects of the Invention]

[0011] This disclosure makes it possible to provide technology that can suppress the unintended leakage of personal information due to odors. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic diagram of the odor transmission system according to the embodiment. [Figure 2] This is a functional block diagram of the odor transmission system according to the embodiment. [Figure 3] This is a diagram illustrating reference odor information. [Figure 4] This is a flowchart showing the processing of the information processing device according to the embodiment. [Modes for carrying out the invention]

[0013] The embodiments of this disclosure will be described below with reference to the drawings. The specific numerical values ​​and other details shown in these embodiments are merely examples to facilitate understanding of the invention and do not limit this disclosure unless otherwise specified. Elements not directly related to this disclosure are omitted from the drawings.

[0014] Figure 1 is a schematic diagram of the odor transmission system 1 according to this embodiment. The odor transmission system 1 includes a data acquisition device 100 and a data presentation device 200. The data acquisition device 100 is configured to communicate with the data presentation device 200 via a predetermined communication network NW. The data acquisition device 100 acquires the odor and captured image of, for example, a subject (user U1 in the example of Figure 1) and transmits them to the data presentation device 200. The data presentation device 200 outputs the odor and captured image received from the data acquisition device 100 to, for example, user U2, so as to generate an odor in conjunction with the display of the captured image. Communication between the data acquisition device 100 and the data presentation device 200 does not have to be in real time; for example, the data acquired by the data acquisition device 100 may be temporarily stored and later transmitted to the data presentation device 200. User U1 and user U2 may be the same person.

[0015] Figure 2 is a functional block diagram of the odor transmission system 1 according to this embodiment. Each functional block shown in each figure can be realized in hardware terms by elements such as a computer processor, CPU, and memory, as well as electronic circuits and mechanical devices, and in software terms by computer programs, etc., but here, the functional blocks that are realized through the cooperation of these are depicted. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by combinations of hardware and software. The data acquisition device 100 includes a camera 101, an odor sensor 102, and an information processing device 103.

[0016] Camera 101 generates captured images by photographing the subject. The captured images in this embodiment are video, but may also be still images. The captured images in this embodiment include the time of capture. Alternatively, a configuration that acquires captured images may be used instead of camera 101.

[0017] The odor sensor 102 is installed on the camera 101 and detects odors in the vicinity of the camera 101. The odor sensor 102 has a sensor element such as a semiconductor or lipid film (sensitive film) that adheres to odor components contained in the air, and detects odors by measuring the physical changes of the sensor element caused by the adhesion of odor components. The odor sensor 102 may be a combination of multiple sensor elements with different response characteristics to odor components so that it can detect various odors. The detection data of the odor sensor 102 in this embodiment includes odor parameters, which will be described later, indicating the detected odor, and the time the odor was detected. The odor sensor 102 may be a separate unit from the camera 101.

[0018] The information processing device 103 performs various processes of the data acquisition device 100. The information processing device 103 includes a storage unit 104, an odor estimation unit 105, an odor comparison unit 106, an assumed category determination unit 107, a detected category determination unit 108, an odor selection unit 109, and a transmission unit 110.

[0019] The memory unit 104 stores algorithms and various data for executing the processing of the information processing device 103 in this embodiment. The memory unit 104 in this embodiment stores reference odor information. The reference odor information is a data list that links each object, the odor category of each object, the representative value of the odor of each object, and the odor range of each object.

[0020] Figure 3 illustrates the reference odor information D. In Figure 3, objects (odor categories) such as banana, yogurt, soy sauce, sauce, and tamagoyaki, the representative value of the odor, the odor width, and the relevance are shown. The odor category and the relevance will be described later. As shown in Figure 3, an odor is represented by a combination of values of a plurality of types of odor parameters P1, …, Pn (n is an integer of 2 or more). The plurality of types of odor parameters are called representative values of the odor. It can also be said that an odor parameter is a value indicating the type of odor of an object and the amount of that odor. If the number of types of odors of an object is plural, the storage unit 104 may store a plurality of types of odor parameters for that object in the reference odor information D. That is, the storage unit 104 may store multi-dimensional odor parameters. Each odor parameter may be associated with a plurality of types of odors detected by the odor sensor 102. That is, when the odor sensor 102 can detect a predetermined odor A, the storage unit 104 may store the corresponding odor parameter PA. If the number of types of odors of an object is singular, the storage unit 104 may store one type of odor parameter for that object in the reference odor information D. That is, the storage unit 104 may store one-dimensional odor parameters. The storage unit 104 may store only one odor parameter of a predetermined object, or may store only one representative value of the odor of a predetermined object.

[0021] Also, even for the same object, the odor may be different. For example, even for the same banana, the odor of a fresh banana and a ripe banana is different. The odor width in Figure 3 represents the range of values of odor parameters regarding the odors that such an object can emit, for each type of odor parameter. The storage unit 104 may store the average value of the odor width as the representative value of the odor. The storage unit 104 may not store the representative value of the odor, and the odor estimation unit 105 described later may calculate the average value of the odor width as the representative value of the odor.

[0022] As shown in FIG. 3, since the representative value of the smell and its smell width are different for each object, a smell category is set for each object. For example, the smell category of the object "banana" is "banana". The smell category of an object indicates the category of the smell to which the smell emitted by the object belongs. Hereinafter, for the sake of simplifying the explanation, it will be described that the object and the smell category correspond one-to-one.

[0023] The reference smell information may include a relevance degree indicating the degree of relevance of an object to a specific situation. As shown in FIG. 3, for each object, a relevance degree to a specific situation is set. For example, for the object "banana", as the relevance degree, "+1 when there is yogurt in the photographed image" is set. This is not limited to this example, and a plurality of relevance degrees may be set for one object. This relevance degree is used, for example, when estimating the detected smell category described later.

[0024] The smell estimation unit 105 estimates a smell parameter indicating the smell assumed around the camera 101 based on the photographed image of the camera 101. "The smell assumed around the camera 101" can also be said to be the smell assumed to be emitted by the object in the photographed image of the camera 101. The smell estimation unit 105 in the present embodiment performs image analysis on the photographed image, recognizes the object in the photographed image, and searches the reference smell information D for the smell parameter corresponding to the recognized object, thereby estimating the smell parameter indicating the assumed smell. The smell estimation unit 105 can recognize the object in the photographed image by using a known object recognition technology such as a machine learning model learned to recognize various objects from the photographed image.

[0025] To simplify the explanation, we will assume that odors are represented in one dimension. That is, we will use only odor parameter P1 from the odor parameters P1, ..., Pn shown in Figure 3. Hereafter, the odor parameter P1 that represents the odor expected from the captured image will be referred to as the camera odor parameter Pc, and the parameter P1 that represents the odor detected by the odor sensor 102 will be referred to as the sensor odor parameter Ps. Here, the camera odor parameter Pc is estimated by the odor estimation unit 105 as an odor parameter having a value within a predetermined range. On the other hand, the sensor odor parameter Ps is detected as an odor parameter having a predetermined value.

[0026] The odor estimation unit 105 estimates odor parameters representing the odor estimated from the captured image acquired by the camera 101 as camera odor parameters Pc. The odor estimation unit 105 recognizes objects from the captured image. If multiple objects are recognized from the captured image, the odor estimation unit 105 obtains the odor widths of the recognized objects from the storage unit 104. The odor estimation unit 105 estimates the minimum value of the camera odor parameter for each odor parameter by adding the minimum odor widths of each object. The odor estimation unit 105 estimates the maximum value of the camera odor parameter Pc for each odor parameter by adding the maximum odor widths of each object. The odor estimation unit 105 supplies the estimated camera odor parameters Pc to the odor comparison unit 106. If only one object is recognized, the odor estimation unit 105 may estimate the odor width of that object as the camera odor parameter Pc.

[0027] For example, when the odor estimation unit 105 recognizes the objects "banana" and "yogurt" from the captured image, it searches for the objects "banana" and "yogurt" in the reference odor information D and extracts the odor widths 2≦P1≦4 for the object "banana" and 3≦P1≦6 for the object "yogurt". The odor estimation unit 105 estimates that the minimum value of the camera odor parameter Pc is the sum of the minimum odor width of the object "banana" ("2") and the minimum odor width of the object "yogurt" ("3"), which is "5". The odor estimation unit 105 also estimates that the maximum value of the camera odor parameter Pc is the sum of the maximum odor width of the object "banana" ("4") and the maximum odor width of the object "yogurt" ("6"), which is "10". In other words, the odor estimation unit 105 estimates that for the camera odor parameter Pc, 5≦Pc≦10.

[0028] The odor comparison unit 106 compares the camera odor parameter Pc with the sensor odor parameter Ps and outputs the comparison result. For example, the odor comparison unit 106 determines whether the difference between the sensor odor parameter Ps and the camera odor parameter Pc is greater than or equal to a predetermined threshold. For example, the odor comparison unit 106 determines whether the difference between the sensor odor parameter Ps and the value closest to the sensor odor parameter Ps within the range of the minimum and maximum values ​​of the camera odor parameter Pc is greater than or equal to a predetermined threshold. If the difference between the sensor odor parameter Ps and the camera odor parameter Pc is greater than or equal to a predetermined threshold, the odor comparison unit 106 sends a partial selection instruction to the assumed category determination unit 107, the detection category determination unit 108, and the odor selection unit 109, indicating that some of the odor parameters of the sensor odor parameter Ps will be selected as a comparison result. For example, if the difference between the sensor odor parameter Ps and the camera odor parameter Pc is less than a predetermined threshold, the odor comparison unit 106 sends a full selection instruction to the odor selection unit 109, indicating that all of the sensor odor parameters Ps will be selected as a comparison result. Hereinafter, the difference between the camera odor parameter Pc, specifically the value closest to the sensor odor parameter Ps within its range from minimum to maximum, and the sensor odor parameter Ps is sometimes referred to as the odor deviation.

[0029] For example, let's explain the comparison process in the odor comparison unit 106 using the case where the camera odor parameter Pc is 5 ≤ ​​Pc ≤ 10 and the value of the sensor odor parameter Ps is "18". In this case, the odor deviation degree, which is the difference between the value of the sensor odor parameter Ps "18" and the maximum value of the camera odor parameter Pc, "10", which is the value closest to the sensor odor parameter Ps "18", is "8". If this odor deviation degree "8" is greater than or equal to a preset threshold (for example, 1), the odor comparison unit 106 determines that the sensor odor parameter Ps and the camera odor parameter Pc are different and outputs a partial selection instruction to select some of the odor parameters from the sensor odor parameter Ps. This threshold is set appropriately depending on the application, hardware specifications, and surrounding environment. Setting this threshold higher is advantageous when the ability to recognize objects from images is not high, or when it is acceptable for some odors different from the odor expected from the image to be present.

[0030] In the above explanation, for simplicity, we described the case where odor is represented in one dimension, but in reality, odor is represented in multiple dimensions. The method by which the odor comparison unit 106 compares odors represented in multiple dimensions is explained below. An odor can be considered as a point on a multidimensional space with multiple types of odor parameters P1, ..., Pn as basis vectors. Also, as mentioned above, odors have width, so they can be represented as a solid in the multidimensional space. For example, the odor comparison unit 106 calculates the distance between the point on the solid represented by the camera odor parameter Pc that is closest to the sensor odor parameter Ps, and the point on the multidimensional space representing the sensor odor parameter Ps, and compares it with a threshold. For example, if this calculated distance is greater than or equal to the threshold, the odor comparison unit 106 determines that the sensor odor parameter Ps and the camera odor parameter Pc are different.

[0031] The assumed category determination unit 107 determines the assumed odor category to which the expected odors around the camera 101 belong, based on the image captured by the camera 101. For example, in response to a partial selection instruction from the odor comparison unit 106, the assumed category determination unit 107 determines the odor category corresponding to the object recognized in the captured image as the assumed odor category. In this embodiment, the assumed category determination unit 107 determines the assumed odor category using the object recognition result in the captured image and the reference odor information D. For example, the assumed category determination unit 107 obtains the object recognition result in the captured image from the odor estimation unit 105 and determines the odor category of the extracted object as the assumed odor category by searching for the object recognized in the captured image from the reference odor information D. For example, if the objects "banana" and "yogurt" are recognized in the captured image, the assumed category determination unit 107 searches for the odor categories corresponding to the objects "banana" and "yogurt" from the reference odor information D and determines the assumed odor category of the captured image to be "banana" and "yogurt". The assumed category determination unit 107 supplies the determined assumed odor category to the odor selection unit 109. The assumed category determination unit 107 may also determine the assumed odor category to which the assumed odor belongs based on the image.

[0032] The detection category determination unit 108 determines the detected odor category to which the odor detected by the odor sensor 102 belongs. For example, in response to a partial selection instruction from the odor comparison unit 106, the detection category determination unit 108 determines the detected odor category based at least on the detection data from the odor sensor 102.

[0033] In this embodiment, the detection category determination unit 108 acquires the sensor odor parameter Ps as detection data from the odor sensor 102 and also acquires reference odor information D from the storage unit 104. Based on the reference odor information D, the sensor odor parameter Ps, the camera odor parameter Pc, and the object recognition result in the captured image, it determines the detected odor category. For example, the detection category determination unit 108 estimates the detected odor category using the object recognition result in the captured image and the reference odor information D. The detection category determination unit 108 searches the reference odor information D for objects whose odor deviation, which is the difference between the sensor odor parameter Ps and the camera odor parameter Pc estimated from the captured image within the range of minimum to maximum values, falls within the odor range. The odor deviation may be the result of odor deviation calculation in the odor comparison unit 106. The detection category determination unit 108 determines the detected odor category by adding the odor category of the extracted object to the estimated detected odor category.

[0034] Here, for example, consider a case where the captured image shows the objects "banana" and "yogurt," but in reality, there is an object "fried egg" outside the camera 101's field of view at the shooting location, and the sensor odor parameter Ps value is "18." In this case, the detection category determination unit 108 estimates the detected odor categories to be "banana" and "yogurt" from the object recognition result in the captured image. The detection category determination unit 108 obtains the odor deviation calculation result, which is the difference between the camera odor parameter Pc and the sensor odor parameter Ps calculated by the odor comparison unit 106, from the odor comparison unit 106. As described above, when the captured image shows the objects "banana" and "yogurt," the camera odor parameter Pc is 5 ≤ ​​Pc ≤ 10. Therefore, the odor deviation, which is the difference between the sensor odor parameter Ps value "18" and the maximum value "10" that is closest to the sensor odor parameter Ps value "18" within the range from the minimum to the maximum value of the camera odor parameter Pc, is "8." The detection category determination unit 108 searches for objects whose odor deviation degree "8" is within the odor range from the reference odor information D, and extracts, for example, the object "fried egg" whose odor range is 7 ≤ P1 ≤ 9. The detection category determination unit 108 adds the odor category of the extracted object "fried egg" to the estimated detected odor categories "banana" and "yogurt". As a result, the detection category determination unit 108 determines the detected odor categories to be "banana", "yogurt", and "fried egg".

[0035] The detection category determination unit 108 may estimate the source of odors not visible in the captured image using various methods. For example, the detection category determination unit 108 may use the relevance score set in the reference odor information D to estimate objects with a high relevance score as the source of odors not visible in the captured image. For example, if the captured image shows the object "sashimi," the detection category determination unit 108 may estimate "soy sauce," which has a relevance score of +1 when sashimi is present in the captured image, as the source of odors not visible in the captured image. Also, for example, if the detection category determination unit 108 estimates that an object in the captured image emits a strong odor based on its appearance, it may weight the odor range of that odor. Furthermore, the detection category determination unit 108 may estimate that objects closer to the camera among the objects visible in the captured image emit a strong odor and weight the odor range of those objects. For example, the detection category determination unit 108 may weight the odor range such that the closer the distance between the recognized object in the captured image and the camera, the larger the maximum value of the odor range of that object becomes. If the odor sensor 102 is capable of detecting odors in enough detail to capture characteristic odor molecules specific to a particular object, the detection category determination unit 108 may estimate the source of the odor that is not visible in the captured image by identifying the object that is the source of the odor using the detection data from the odor sensor 102. Therefore, it is not essential for the detection category determination unit 108 to use the captured image or the object recognition results in the captured image when estimating the source of the odor that is not visible in the captured image. The detection category determination unit 108 may estimate a combination of multiple objects (for example, the object "fried egg" and "sauce") rather than just one object as the source of the odor that is not visible in the captured image. Also, for example, if multiple objects are found to be within the odor range in terms of odor deviation, the detection category determination unit 108 may extract the object that is the source of the odor that is not visible in the captured image using the aforementioned methods such as relevance.

[0036] If the detection category determination unit 108 searches for reference odor information D and fails to extract an object whose odor deviation degree falls within the odor range, it may determine the detected odor category by adding, for example, "unknown" to the estimated detected odor category. For example, consider a case where the captured image shows the objects "banana" and "yogurt," but the sensor odor parameter Ps is "30." If the detection category determination unit 108 fails to extract an object whose odor deviation degree falls within the odor range, it cannot estimate the source of the odor not visible in the captured image, and therefore cannot identify the reason why the sensor odor parameter Ps and the camera odor parameter Pc differ. In this case, the detection category determination unit 108 adds, for example, "unknown" to the estimated detected odor categories "banana" and "yogurt," determining the detected odor categories to be "banana," "yogurt," and "unknown."

[0037] The odor selection unit 109 selects odor parameters from the sensor odor parameters Ps that belong to odor categories included in both the assumed odor category and the detected odor category. In this embodiment, the odor selection unit 109 selects all sensor odor parameters Ps in response to a full selection instruction from the odor comparison unit 106. In addition, in this embodiment, the odor selection unit 109 subtracts odor parameters corresponding to odor categories that do not match the assumed odor category from the sensor odor parameters Ps in response to a partial selection instruction from the odor comparison unit 106. For example, based on the reference odor information D, the odor selection unit 109 subtracts representative values ​​of odors from the sensor odor parameters Ps for odor categories that do not match the assumed odor category. By selecting the odor parameters after the subtraction, the odor selection unit 109 selects odor parameters that belong to categories included in both the assumed odor category and the detected odor category. The odor selection unit 109 supplies odor selection information indicating the selected odor parameters to the transmission unit 110. In the odor selection information, the selected odor parameters are represented as vector elements in a multidimensional space. The odor selection information includes information indicating the selected odor parameters and the time the odor was detected.

[0038] Here, consider, for example, the case where the value of the sensor odor parameter Ps is "18", the assumed odor categories are determined to be "banana" and "yogurt", and the detected odor categories are determined to be "banana", "yogurt", and "fried egg". In this case, the odor selection unit 109 subtracts the odor parameter corresponding to the detected odor category "fried egg", which does not match the assumed odor category, from the sensor odor parameter Ps. According to the reference odor information D in Figure 3, the representative value of the odor for the object "fried egg" is "8", so the odor selection unit 109 subtracts this representative value "8" from the value of the sensor odor parameter Ps "18" and selects the remaining odor parameter. The value of this selected remaining odor parameter is "10". Note that this selected value of the odor parameter is different from "8", which is the sum of the representative values ​​of the odors for the objects "banana" and "yogurt". In other words, the odor parameters selected here are not simply a combination of typical odor parameters of objects in the captured image, but rather, odor parameters of objects not present in the captured image are excluded from the odor parameters that represent the real odor detected by the odor sensor 102. Therefore, it becomes possible to reproduce the sensor odor parameters Ps, which represent the odor detected by the odor sensor 102, more realistically.

[0039] Furthermore, for example, if the detected odor category includes the "unknown" odor category, the detected odor category "unknown" will be an odor category that does not match the assumed odor category. In this case, the odor selection unit 109 may subtract the odor deviation from the sensor odor parameter Ps as the odor parameter corresponding to the detected odor category "unknown" that does not match the assumed odor category. Now, consider the case where the value of the sensor odor parameter Ps is "18", the assumed odor categories are determined to be "banana" and "yogurt", and the detected odor categories are determined to be "banana", "yogurt", and "unknown". In this case, the odor selection unit 109 subtracts the odor parameter corresponding to the detected odor category "unknown" that does not match the assumed odor category from the sensor odor parameter Ps. Here, as described above, the assumed camera odor parameter Pc when the odor categories are "banana" and "yogurt" is 5 ≤ ​​Pc ≤ 10, and the odor deviation from the value of the sensor odor parameter Ps "18" is "8". For example, the odor selection unit 109 subtracts the odor deviation degree "8" from the sensor odor parameter Ps value "18" and selects the odor parameter after the subtraction. The value of this selected odor parameter after the subtraction is "10". Alternatively, the sum of the representative odor value of object "banana" ("3") and the representative odor value of object "yogurt" ("5"), which is "8", may be subtracted from the sensor odor parameter Ps value "18". The detection category determination unit 108 does not need to add, for example, "unknown" to the estimated detected odor categories "banana" and "yogurt" if it is unable to extract an object whose odor deviation degree is within the odor range. In this case, the odor selection unit 109 may subtract the odor deviation degree from the sensor odor parameter Ps if it is unable to extract an object whose odor deviation degree is within the odor range.

[0040] The transmitting unit 110 transmits the odor selection information to the data display device 200. The transmitting unit 110 may also transmit the captured image to the data display device 200. The transmitting unit 110 may also transmit the odor selection information and the captured image together to the data display device 200.

[0041] The data presentation device 200 comprises a receiving unit 201, a cartridge 202, an odor generating unit 203, and a display unit 204.

[0042] Cartridge 202 stores multiple types of elemental odors that make up a scent. Elemental odors are the basic odor elements that constitute a given scent. For example, cartridge 202 can consist of a replaceable container for storing elemental odors.

[0043] The odor generating unit 203 mixes the elemental odors stored in the cartridge 202 to reproduce the odor selected by the odor selection information received from the data acquisition device 100 via the receiving unit 201, and generates the odor. The odor generating unit 203 can generate various odors using multiple types of elemental odors stored in the cartridge 202.

[0044] The display unit 204 displays the captured image received from the data acquisition device 100 via the receiving unit 201. The odor generation unit 203 and the display unit 204 can generate odors and display captured images in coordination so that the odor detection time in the odor selection information and the capture time in the captured image are synchronized. This allows odors to be generated in synchronization with the progress of the captured image.

[0045] Figure 4 is a flowchart showing the processing of the information processing device 103 in this embodiment. Figure 4 shows an example of processing when real-time communication occurs between the data acquisition device 100 and the data presentation device 200.

[0046] In step S101, the odor estimation unit 105 estimates odor parameters that represent the expected odor around the camera 101 based on the captured image acquired from the camera 101. The odor estimation unit 105 supplies information indicating the estimated odor parameters to the odor comparison unit 106.

[0047] In step S102, the odor comparison unit 106 determines whether the difference between the odor parameters indicating the odor detected by the odor sensor 102 and the odor parameters estimated by the odor estimation unit 105 is greater than or equal to a predetermined threshold. If the difference is less than the predetermined threshold (No in step S102), the odor comparison unit 106 supplies a full selection instruction to the odor selection unit 109 to select all odor parameters indicating the odor detected by the odor sensor 102, and the process proceeds to step S103. If the difference is greater than or equal to the predetermined threshold (Yes in step S102), the odor comparison unit 106 supplies a partial selection instruction to the assumed category determination unit 107, the detected category determination unit 108, and the odor selection unit 109 to select some of the odor parameters indicating the odor detected by the odor sensor 102, and the process proceeds to step S104.

[0048] In step S103, the odor selection unit 109 selects all odor parameters that represent the odor detected by the odor sensor 102 and supplies odor selection information indicating the selection result to the transmission unit 110. After step S103, the process proceeds to step S107.

[0049] In step S104, the assumed category determination unit 107 determines the assumed odor category based on the captured image.

[0050] In step S105, the detection category determination unit 108 determines the detected odor category.

[0051] In step S106, the odor selection unit 109 subtracts odor parameters corresponding to detected odor categories that do not match the assumed odor category from the odor parameters representing the odor detected by the odor sensor 102, selects the remaining odor parameters, and supplies odor selection information indicating the selection result to the transmission unit 110.

[0052] In step S107, the transmission unit 110 transmits the odor selection information and the captured image to the data presentation device 200. After step S107, the process ends.

[0053] If the data acquisition device 100 and the data presentation device 200 do not communicate in real time, the odor selection unit 109 selects odor parameters from the detected odor parameter data from the start to the completion of shooting at predetermined intervals, and arranges the selected odor parameters in chronological order to generate odor selection information that shows the selected odor parameters over time. Then, the transmission unit 110 can transmit this odor selection information in synchronization with the captured image.

[0054] Thus, according to this embodiment, it is possible to suppress the transmission of odor parameters indicating the odor of an object not visible in the captured image, which are detected by the odor sensor 102, to the data presentation device 200. As a result, it is possible to suppress the unintentional leakage of personal information due to the odor of an object not present in the captured image, thereby enabling appropriate protection of privacy. For example, consider a scenario where the filming location is the vicinity of the home of a famous person, and there is a sea nearby. In this case, according to this embodiment, even though the sea is not visible in the captured image, the transmission of odor parameters indicating the smell of the sea to the data presentation device 200 is suppressed, thus preventing the unintentional leakage of personal information that the person lives near the sea.

[0055] Those skilled in the art will understand that this disclosure is not limited to the embodiments described above, that various design changes are possible, and that various modifications are possible, and that such modifications are also within the scope of this disclosure. Modifications are described below.

[0056] The storage unit 104 may pre-store exceptional odor information, which indicates odor parameters that should be excluded as exceptions from among the odor parameters related to the odor detected by the odor sensor 102. For example, the odor sensor 102 may pre-detect the odor of the photographer and the subject, and using the detection results, the odor parameters indicating the photographer's odor may be stored in the storage unit 104 as exceptional odor information. In this case, the odor selection unit 109 may acquire the exceptional odor information from the storage unit 104 and subtract the odor parameters indicated by the exceptional odor information from the odor parameters indicating the odor detected by the odor sensor 102. This makes it possible to exclude odor parameters indicating odors that are easily included unintentionally during shooting, such as the odor of the photographer and the subject, from the odor parameters indicating the odor detected by the odor sensor 102. Personal odors contain a lot of privacy information (e.g., health status, lifestyle habits, etc.), so excluding these odor parameters makes it possible to appropriately protect personal information. Furthermore, the odor selection unit 109 may exclude the odor parameter indicated by the exceptional odor information from the odor parameters indicating the odor detected by the odor sensor 102 if it can identify the odor parameter indicated by the exceptional odor information from the odor parameters indicating the odor detected by the odor sensor 102. Also, for example, even if the subject being photographed is registered as an odor category in the reference odor information, and both the assumed odor category and the detected odor category include the subject's odor category, the odor selection unit 109 may exclude the subject's odor parameter indicated by the exceptional odor information from the odor parameters indicating the odor detected by the odor sensor 102 if the exceptional odor information indicates the subject's odor parameter.

[0057] The assumed category determination unit 107 may determine the assumed odor category for objects present in the captured image that have been in the image for a predetermined time or longer. For example, the storage unit 104 may store a history of frames in which recognized objects are shown in the captured image, and the assumed category determination unit 107 may determine the assumed odor category for objects in which the number of frames in which the recognized object is shown exceeds a predetermined threshold in the history from the present to a predetermined frame. Here, for example, an object that appears in the captured image for only a moment is likely to be an object that entered the image unintentionally by the photographer. Therefore, transmitting an odor for such a fleeting object could lead to the unintentional leakage of personal information. In contrast, with this configuration, only objects that have been in the captured image for a predetermined time or longer are subject to odor transmission. Therefore, the transmission of odors from objects that appear for only a moment in the captured image can be suppressed, thus preventing the unintentional leakage of personal information. Alternatively, for example, the assumed category determination unit 107 may determine whether or not there is an object that has been recognized for a predetermined time or longer by the odor estimation unit 105, and if there is an object that has been recognized for a predetermined time or longer, it may determine the assumed odor category for that object.

[0058] In the embodiment, odor parameters of detected odor categories that did not match the assumed odor category were excluded, but this is not limited to the application of the present disclosure. The odor selection unit 109 may select odor parameters of odor categories that match the assumed odor category and the detected odor category. The odor selection unit 109 does not have to exclude odor parameters of detected odor categories that do not match the assumed odor category. In other words, the odor selection unit 109 does not have to subtract odor parameters corresponding to odor categories that do not match the assumed odor category from the sensor odor parameter Ps. The odor selection unit 109 may select odor parameters of odor categories that match the assumed odor category and the detected odor category. In step S106, the odor selection unit 109 does not have to subtract odor parameters corresponding to detected odor categories that do not match the assumed odor category from the sensor odor parameter Ps. In step S106, the odor selection unit 109 may select odor parameters of odor categories that match the assumed odor category and the detected odor category.

[0059] If the assumed category determination unit 107 cannot accurately identify an object recognized in the captured image, it may define multiple assumed odor categories for the unidentified object. For example, consider a case where a black object is captured in the image, and as a result of image analysis of the captured image, the assumed category determination unit 107 determines the assumed odor categories for that black object to be "sauce" and "soy sauce". For example, if the detection category determination unit 108 determines "sauce" as the detected odor category, the odor selection unit 109 should select the odor parameters for the odor category "sauce" that are common to both the assumed odor category and the detected odor category. In this way, even if multiple assumed odor categories are estimated for an object in the captured image, the odor parameters of the appropriate odor category can be selected by determining whether the detected odor category and the assumed odor category match.

[0060] In this embodiment, the data presentation device 200 includes a scent generating unit 203 and a display unit 204, and generates a scent along with the playback of the captured image. However, the application of this disclosure is not limited to this. For example, a microphone may be provided in the data acquisition device 100 to acquire audio data, and a speaker may be provided in the data presentation device 200 to output the audio data, thereby outputting audio along with the captured image and scent. This makes it possible to represent the shooting scene more realistically.

[0061] The data acquisition device 100 is equipped with a storage unit 104 that stores reference odor information D, but the application of this disclosure is not limited to this. A storage unit that stores reference odor information D may be provided in a server or other device located outside the data acquisition device 100. In this case, the data acquisition device 100 only needs to read and acquire the reference odor information D from the server when using it.

[0062] In this embodiment, the odor detection data and odor selection information from the odor sensor 102 include the odor detection time, and the captured image includes the capture time; however, this disclosure is not limited to these embodiments. Other known methods may be used to generate odors in synchronization with the playback of the captured image.

[0063] In the embodiment, odor parameters were estimated by the odor estimation unit 105 and assumed odor categories and detected odor categories were determined based on an object recognized by image analysis and reference odor information D, but the application of this disclosure is not limited to this. For example, when a captured image is input, odor parameters indicating the odor around the camera, assumed odor categories, and detected odor categories may be determined using a machine learning model that has been trained to determine odor parameters indicating the odor around the camera, assumed odor categories, and detected odor categories from the entire captured image.

[0064] In this embodiment, the reference odor data D includes the degree of relevance, but it does not necessarily have to include the degree of relevance.

[0065] In the embodiments, for the sake of simplicity, objects and odor categories have been described as having a one-to-one correspondence; however, this disclosure is not limited to this. For example, an odor category may be defined by a combination of objects such as "a fruit platter," or it may be defined by a general term for specific objects such as "flowers" or "vegetables."

[0066] When a function such as background blurring is used to hide only a portion of a captured image, odor parameters related to the smell of objects present in the hidden portion of the image may be excluded. For example, the assumed category determination unit 107 determines an assumed odor category from the objects present in the portion of the captured image that has not been subjected to background blurring. The odor selection unit 109 selects odor parameters from the detected odor categories that belong to the odor category that matches the assumed odor category determined from the objects present in the portion of the image that has not been subjected to background blurring. This excludes odor parameters related to objects present in the portion of the captured image that has been subjected to background blurring, making it possible to select only odor parameters related to objects in the portion of the image that has not been subjected to background blurring. Since the hidden portion of the captured image is an image that one does not want others to see, excluding odor parameters related to objects present in this portion of the image can appropriately suppress the leakage of personal information.

[0067] For example, if camera 101 has an out-camera that photographs the person being photographed and an in-camera that photographs the photographer, if there is an object that is not visible in the image taken by the out-camera but is visible in the image taken by the in-camera, the odor parameters related to the smell of the object visible in the image taken by the in-camera may be excluded. For example, the assumed category determination unit 107 acquires the image taken by the in-camera from camera 101, performs image analysis on the image taken by the in-camera to recognize an object in the image taken by the in-camera, and estimates odor parameters that indicate the expected smell from the image taken by the in-camera by searching for odor parameters corresponding to the recognized object from the reference odor information D. The odor selection unit 109 may select the odor parameters after subtraction by subtracting the odor parameters that indicate the expected smell from the image taken by the in-camera from the odor parameters that indicate the smell detected by the odor sensor 102.

[0068] The source of the odor not visible in the captured image may be estimated using various sensors different from the camera 101 and the odor sensor 102.

[0069] In summary, the information processing device of this disclosure comprises: an assumed category determination unit 107 that determines an assumed odor category to which an assumed odor belongs based on an image; a detected odor category determination unit 108 that determines a detected odor category to which an odor detected by an odor sensor 102 belongs; an odor selection unit 109 that selects odor parameters from among the odor parameters indicating the odor detected by the odor sensor 102 that belong to a category included in both the assumed odor category and the detected odor category; and a transmission unit 110 that transmits odor selection information indicating the selected odor parameters. With this configuration, it is possible to suppress the unintended leakage of personal information due to the odor of an object not present in the captured image, thereby enabling appropriate protection of privacy.

[0070] In this disclosure, the odor selection unit 109 acquires exceptional odor information indicating odor parameters related to odors detected in advance, and subtracts the odor parameters indicated by the exceptional odor information from the odor parameters indicating the odors detected by the odor sensor 102. With this configuration, odor parameters indicating odors that are likely to be unintentionally included during shooting can be excluded.

[0071] In this disclosure, the assumed category determination unit 107 determines the assumed odor category for objects that are present in the image for a predetermined time or longer. This configuration makes it possible to suppress the unintentional leakage of personal information due to the odor of objects that appear momentarily in the captured image.

[0072] In this disclosure, the odor selection unit 109 subtracts odor parameters corresponding to detected odor categories that do not match the assumed odor category from the odor parameters representing the odor detected by the odor sensor 102. This configuration makes it possible to reproduce the odor detected by the odor sensor 102 more realistically.

[0073] Although this disclosure has been described above with reference to the embodiments described above, this disclosure is not limited to the embodiments described above, and any combination or substitution of the configurations shown in the embodiments is also included in this disclosure. [Explanation of symbols]

[0074] 1 Odor transmission system, 100 Data acquisition device, 101 Camera, 102 Odor sensor, 103 Information processing device, 104 Storage unit, 105 Odor estimation unit, 106 Odor comparison unit, 107 Assumed category determination unit, 108 Detection category determination unit, 109 Odor selection unit, 110 Transmission unit, 200 Data presentation device, 201 Receiving unit, 202 Cartridge, 203 Odor generation unit, 204 Display unit.

Claims

1. Based on the image, the assumed odor category determines the assumed odor category to which the assumed odor belongs, and A detection category determination unit that determines the detected odor category to which the odor detected by the odor sensor belongs, A smell selection unit that selects smell parameters belonging to the smell category that is included in both the assumed smell category and the detected smell category from among the smell parameters that represent the smell detected by the smell sensor, A transmission unit that transmits odor selection information indicating the selected odor parameters, An information processing device equipped with the following features.

2. The odor selection unit acquires exceptional odor information indicating odor parameters related to odors previously detected, and subtracts the odor parameters indicated by the exceptional odor information from the odor parameters indicating the odor detected by the odor sensor. The information processing apparatus according to claim 1.

3. The assumed category determination unit determines the assumed odor category for objects that have been present in the image for a predetermined time or longer. The information processing apparatus according to claim 1.

4. The odor selection unit subtracts odor parameters corresponding to detected odor categories that do not match the assumed odor category from odor parameters representing odors detected by the odor sensor. The information processing apparatus according to any one of claims 1 to 3.

5. Based on the image, determine the assumed odor category to which the assumed odor belongs, and Determining the detected odor category, which is the odor category to which the odor detected by the odor sensor belongs, From among the odor parameters indicating the odor detected by the odor sensor, select the odor parameters belonging to the odor category that is included in both the assumed odor category and the detected odor category, To transmit odor selection information indicating the selected odor parameters, A method that includes [a certain feature].

6. On the computer, Based on the image, determine the assumed odor category to which the assumed odor belongs, and Determining the detected odor category, which is the odor category to which the odor detected by the odor sensor belongs, From among the odor parameters indicating the odor detected by the odor sensor, select the odor parameters belonging to the odor category that is included in both the assumed odor category and the detected odor category, To transmit odor selection information indicating the selected odor parameters, A program that executes the command.

Images