Information processing device, information processing method, and information processing program

Abstract

To provide an information processing device capable of estimating deformation of an object as a target using active acoustic sensing by attaching a smart device (or a pair of a speaker and a microphone) to the object.SOLUTION: An information processing device comprises: an instruction unit which issues an instruction to output sound from a speaker attached to a prescribed object deformable by a user; an estimation unit which estimates deformation of the prescribed object by using active acoustic sensing on the basis of sound acquired by a microphone attached to the prescribed object; and a response unit which outputs a response according to an input by the user based on estimation results of the deformation of the prescribed object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] An active acoustic sensing technology has been disclosed that recognizes hand gestures by attaching a speaker and microphone to the arm. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Study on hand gesture recognition method using active acoustic sensing, Information Processing Society of Japan Research Report, August 6, 2016, [Retrieved November 22, 2021], Internet <URL:https: / / ipsj.ixsq.nii.ac.jp / ej / ?action=repository_action_common_download&item_id=171767&item_no=1&attribute_id=1&file_no=1> Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional technology merely recognizes gestures, and there is room for providing more services to users. For example, when a user applies weight or force to a soft object such as a commercially available stuffed toy or cushion, the soft object may bend and deform. If such deformation can be detected, it can be used for various purposes.

[0005] The present application has been made in consideration of the above, and aims to detect deformation of a target object using active acoustic sensing by attaching a smart device (or a pair of a speaker and a microphone) to the target object. [Means for solving the problem]

[0006] The information processing device according to the present application includes: a command unit that issues a command to output sound from a speaker of a smartphone that is detachably attached to a predetermined object that is a stuffed animal, a cushion, or an object equivalent thereto and that can be deformed by a user's weight or force; a learning unit that uses sound acquired by a microphone of a smartphone detachably attached to the predetermined object as data on sound characteristics for each deformation of the predetermined object, learns with data on an unspecified number of users to build a model of all users until a sufficient number of individual user data has been accumulated to build a model of each individual user, and calibrates for each user to build a model of each individual user at the time when a sufficient number of individual user data has been accumulated; Active acoustic sensing is performed based on sounds acquired by a microphone of a smartphone detachably attached to the predetermined object. In this case, before the model of each individual user is constructed, the model of all users is used, and after the model of each individual user is constructed, the model of each individual user is used. an estimation unit that estimates a deformation of the predetermined object using the above-mentioned method and estimates an input of the user to the predetermined object according to the deformation of the predetermined object; Audio In response, in response to the estimated user input: from the predetermined object different audio and a response unit that outputs the response. [Effects of the Invention]

[0007] According to one aspect of the embodiment, a smart device (or a speaker and microphone pair) can be attached to an object of interest to detect deformation of the object using active acoustic sensing. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram showing an overview of an information processing method according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of a terminal device according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of a server device according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of the user information database. [Figure 6] FIG. 6 is a diagram illustrating an example of the history information database. [Figure 7]FIG. 7 is a diagram illustrating an example of the deformation information database. [Figure 8] FIG. 8 is a flowchart showing a processing procedure according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an information processing device, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to these embodiments. Furthermore, the same components in the following embodiments will be denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0010] [1. Overview of information processing method] First, an overview of an information processing method performed by an information processing device according to an embodiment will be described with reference to Fig. 1. Fig. 1 is an explanatory diagram showing an overview of an information processing method according to an embodiment. Note that Fig. 1 explains an example in which a smart device (or a pair of a speaker and a microphone) is attached to a target object and deformation of the object is detected by active acoustic sensing.

[0011] 1, the information processing system 1 includes a terminal device 10 and a server device 100. The terminal device 10 and the server device 100 are connected to each other via a network N (see FIG. 2) in a wired or wireless manner so as to be able to communicate with each other. In this embodiment, the terminal device 10 cooperates with the server device 100.

[0012] The terminal device 10 is a smart device such as a smartphone or tablet terminal used by a user U, and is a portable terminal device capable of communicating with any server device via a wireless communication network such as 4G (Generation (4G)) or LTE (Long Term Evolution) networks. The terminal device 10 has a screen such as a liquid crystal display with a touch panel function, and accepts various operations on displayed data such as content, such as tapping, sliding, and scrolling, performed by the user U with a finger or a stylus. An operation performed on an area of ​​the screen where content is displayed may be considered an operation on the content. The terminal device 10 may be not only a smart device, but also an information processing device such as a desktop PC (Personal Computer) or a notebook PC.

[0013] The server device 100 is an information processing device that works in conjunction with the terminal device 10 of each user U and provides API (Application Programming Interface) services for various applications (hereinafter referred to as apps) and various data to the terminal device 10 of each user U, and is realized by a computer, a cloud system, etc.

[0014] The server device 100 may also be an information processing device that provides some kind of online web service to the terminal device 10 of each user U. For example, the server device 100 may provide the following web services: internet connection, search service, social networking service (SNS), electronic commerce (EC), electronic payment, online games, online banking, online trading, hotel and ticket reservations, video and music distribution, news, maps, route search, route guidance, line information, operation information, and weather forecast. In practice, the server device 100 may cooperate with various servers that provide the above-mentioned web services and act as an intermediary for the web services or may be responsible for processing the web services.

[0015] The server device 100 can acquire user information about the user U. For example, the server device 100 acquires information about the attributes of the user U, such as the gender, age, and residential area of ​​the user U. The server device 100 then stores and manages the information about the attributes of the user U together with identification information (such as a user ID) that identifies the user U.

[0016] The server device 100 also acquires various types of history information (log data) indicating the behavior of the user U from the terminal device 10 of the user U or from various servers based on the user ID, etc. For example, the server device 100 acquires a location history, which is a history of the user U's location and date and time, from the terminal device 10. The server device 100 also acquires a search history, which is a history of search queries entered by the user U, from a search server (search engine). The server device 100 also acquires a browsing history, which is a history of content viewed by the user U, from a content server. The server device 100 also acquires a purchase history (payment history), which is a history of the user U's product purchases and payment processes, from an e-commerce server or a payment processing server. The server device 100 may also acquire a listing history and a sales history, which are a history of the user U's listings on the marketplace, from the e-commerce server or the payment processing server. The server device 100 also acquires a posting history, which is a history of the user U's posts, from a posting server or SNS server that provides a word-of-mouth posting service.

[0017] [1-1. Estimating Object Deformation Using Active Acoustic Sensing] In this embodiment, a smart device such as a smartphone (or a pair of a speaker and a microphone) is attached to a soft object such as a stuffed toy or a cushion, and the deformation of the soft object is estimated using active acoustic sensing technology. In addition, a user's input based on the deformation of the object is estimated, and processing (or control) is performed according to the input. Here, a stuffed toy is used as an example.

[0018] As shown in FIG. 1, the server device 100 issues a command (instruction) via a network N (see FIG. 2) to output sound from a terminal device 10 or a speaker SK attached inside (or on the surface of) a stuffed toy T on which each user U is seated (step S1). The command (instruction) may be data or a signal. Note that if the speaker SK is independent of the terminal device 10, the server device 100 may directly issue a command to the speaker SK to output sound.

[0019] Next, the terminal device 10 of the user U receives a command from the server device 100 and outputs sound from the terminal device 10 or speaker SK attached inside (or on) the stuffed animal T (step S2). For example, the speaker SK is a vibration speaker that can be used for active acoustic sensing. The speaker SK may be externally attached to the surface of the housing of the terminal device 10, or may be built into or mounted on the housing of the terminal device 10.

[0020] Next, the terminal device 10 of the user U acquires the sound using the terminal device 10 or microphone MK attached inside (or on) the stuffed animal T (step S3). For example, the microphone MK is a piezo microphone that can be used for active acoustic sensing. The microphone MK may be externally attached to the surface of the housing of the terminal device 10, or may be built into or mounted on the housing of the terminal device 10.

[0021] Furthermore, the terminal device 10 that outputs sound (the terminal device 10 as a speaker SK) and the terminal device 10 that acquires sound (the terminal device 10 as a microphone MK) may be different terminal devices 10 (separate terminal devices 10). That is, two terminal devices 10, one as a speaker terminal device 10 and the other as a microphone terminal device 10, may be used.

[0022] Next, the terminal device 10 of the user U transmits information about the sound acquired by the microphone MK to the server device 100 via the network N (see FIG. 2) (step S4). If the microphone MK is independent from the terminal device 10, the microphone MK may directly transmit information about the acquired sound to the server device 100.

[0023] Next, the server device 100 receives information about the sound acquired by the microphone MK from the terminal device 10 of each user U, and estimates the deformation of the stuffed toy T using active acoustic sensing (step S5). That is, the server device 100 estimates the deformation of the stuffed toy T using active acoustic sensing from the sound acquired when the stuffed toy T is deformed due to the weight or force applied by the user U. At this time, the server device 100 may use active acoustic sensing to estimate each deformation of the stuffed toy T from the sound acquired for each deformed state of the stuffed toy T. Furthermore, the server device 100 may estimate the deformation pattern of the stuffed toy T as the deformation of the stuffed toy T, or may estimate the shape of the stuffed toy T after deformation. In this way, the server device 100 can detect the occurrence of various deformations of the stuffed toy T using active acoustic sensing.

[0024] Next, the server device 100 estimates the input of the user U based on the estimated deformation of the stuffed toy T (step S6). Here, the deformation of the stuffed toy T is associated with the input of the user U. The deformation of the stuffed toy T and the input of the user U may be associated automatically by the server device 100 or manually by the user U. The server device 100 estimates the input of the user U that corresponds to the estimated deformation of the stuffed toy T.

[0025] Next, the server device 100 performs processing (or control) according to the input of the estimated user U (step S7). For example, the server device 100 outputs a response according to the input of the estimated user U.

[0026] Specifically, the server device 100 outputs a sound corresponding to the input of the estimated user U from the terminal device 10 or microphone MK attached inside (or on the surface of) the stuffed toy T. Alternatively, the server device 100 may associate the user's input (deformation of the stuffed toy T) with remote control operations (button operations) of home appliances, AV equipment (audio visual), etc., and output a control signal for the home appliances, AV equipment, etc. in accordance with the input of the estimated user U to operate / control them.

[0027] In this case, if the server device 100 identifies the user U based on identification information (such as a user ID) indicating the user U, the server device 100 may change the content of the "input" of the user U corresponding to the transformation of the stuffed toy T and the "response" corresponding to that input, depending on the location information, attribute information, or history information of the user U. In other words, for one "transformation," different "inputs" and "responses" may be set depending on the location information, attribute information, or history information of the user U. For example, even if the transformation of the stuffed toy T is the same, the server device 100 may change the content of the "input" and "response" corresponding to the transformation depending on whether the user U is at home or out.

[0028] The server device 100 may estimate the transformation of the stuffed toy T based on the sound characteristics (properties) for each transformation of the stuffed toy T. In this case, the server device 100 constructs (generates and updates) a model that has learned the sound characteristics for each transformation of the stuffed toy T, and uses the model to estimate the transformation of the stuffed toy T. When the stuffed toy T transforms, the acoustic characteristics of the stuffed toy T change.

[0029] For example, the server device 100 builds a model by learning the features of the sound (output sound) output from the speaker SK and the features of the sound (acquired sound) acquired by the microphone MK when the terminal device 10 (or a pair of speaker and microphone) is attached to the stuffed toy T. In this case, the server device 100 builds a model by learning the features of the acquired sound (or both the features of the output sound and the features of the acquired sound). Alternatively, the server device 100 builds a model by learning the difference between the output sound and the acquired sound.

[0030] That is, a smartphone or the like is attached to a stuffed animal or the like, and while the user applies weight or force to the stuffed animal or the like to deform it, the server device 100 outputs and acquires sound using the smartphone or the like attached to the stuffed animal or the like, and constructs a model that learns the characteristics of the acquired sound (or the characteristics of the difference between the output sound and the acquired sound).

[0031] Furthermore, the server device 100 may learn the characteristics of the acquired sound for the shape of the stuffed toy T after it has been deformed, construct a model, and use the model to estimate the deformation of the stuffed toy T. Alternatively, the server device 100 may learn the difference between the characteristics of the acquired sound for the original shape of the stuffed toy T and the characteristics of the acquired sound for the shape of the stuffed toy T after it has been deformed, construct a model, and use the model to estimate the deformation of the stuffed toy T. The acoustic characteristics of the stuffed toy T change between its original shape and its deformed shape. Note that the original shape of the stuffed toy T refers to the shape of the stuffed toy T when the user is not applying weight or force to the stuffed toy T (in an undeformed state).

[0032] Furthermore, the server device 100 may construct a model that learns the above sound characteristics for each installation position (mounting location) of the terminal device 10 (or speaker and microphone pair) on the stuffed toy T, and use the model to estimate the deformation of the stuffed toy T. Specifically, the server device 100 may construct a model that learns pairs of installation positions on the stuffed toy T and sound characteristics for each deformation of the stuffed toy T, and use the model to estimate the deformation of the stuffed toy T. The acoustic characteristics also change depending on which part (place, part, portion) of the stuffed toy T the terminal device 10 (or speaker and microphone pair) is attached to. The acoustic characteristics also change depending on whether the installation position is outside or inside the stuffed toy T.

[0033] Furthermore, the server device 100 may construct a model that learns the above sound characteristics for each type (model, model number, etc.) of the terminal device 10 (or speaker and microphone pair) attached to the stuffed toy T, and use the model to estimate the deformation of the stuffed toy T. Acoustic characteristics also change depending on the device that outputs or acquires the sound.

[0034] Furthermore, the server device 100 may build a model by learning the above sound characteristics for each stuffed animal T. Specifically, the server device 100 may build a model by learning the above sound characteristics for each type of stuffed animal T (model number, material, size, etc.). For example, the server device 100 may build a model that asks the user U to select a target stuffed animal, and outputs a "transformation" when the user inputs the "model number," "installation position," and "sound" of the stuffed animal.

[0035] Furthermore, it is preferable that the server device 100 perform user calibration. Specifically, the server device 100 learns from data of an unspecified number of users up to a certain point (a predetermined stage) to build a model for all users, and when sufficient data for each user has been accumulated, it performs calibration for each user to build a model for each individual user. For example, the server device 100 may build a global model for all users and then build a local model for each user based on the global model.

[0036] At this time, the server device 100 may construct the model using a federated learning technique. For example, the server device 100 may provide a global model to each user, have each user learn based on the global model to construct a local model, and acquire parameter differences and the like.

[0037] Furthermore, for example, the terminal device 10 (or a pair of a speaker and a microphone) may be attached to a cushion, pillow, etc., not limited to the stuffed toy T, and the server device 100 may learn the deformation and sound characteristics of the cushion, pillow, etc., to construct a model, and use the model to estimate the deformation of the cushion, pillow, etc. from the sound.

[0038] That is, a smartphone or the like is attached to a cushion, pillow, or the like, and the server device 100 constructs a model that learns the characteristics of the acquired sound (or the characteristics of the difference between the output sound and the acquired sound) while the user applies weight or force to the cushion, pillow, or the like to deform it.

[0039] The server device 100 may also estimate the degree of wear of the material of a soft object such as a stuffed toy or a cushion. For example, the terminal device 10 (or a pair of a speaker and a microphone) may be attached to the cushion material, and the server device 100 may learn the degree of wear of the cushion material and sound characteristics to construct a model, and estimate the degree of wear of the cushion material from the sound using the model.

[0040] As described above, in this embodiment, the server device 100 estimates the deformation of a stuffed toy or the like using active acoustic sensing technology, estimates the user's input according to the deformation of the stuffed toy or the like, and outputs a response according to the input. That is, the server device 100 estimates the input made by the user to a specific object (a stuffed toy or the like) that can be deformed by the user according to the characteristics of the sound acquired by the microphone when the terminal device is attached to the specific object (a stuffed toy or the like). In other words, the server device 100 acquires the sound transmitted through the object (a stuffed toy or the like), estimates the deformation of the object based on the acquired sound, estimates the input from the estimated deformation, and outputs a response.

[0041] When acquiring sound, the server device 100 may acquire sound of an appropriate wavelength depending on the target (such as a stuffed animal). In addition, in order to acquire reflected sound, the server device 100 may instruct the terminal device 10 or the speaker SK to output multiple sets of different frequencies at intervals.

[0042] Furthermore, the server device 100 learns the characteristics of the output sound and the acquired sound while having a user U perform various inputs to the stuffed animal (such as holding hands, twisting, and stroking) while placing a smartphone inside the stuffed animal, and constructs a model by learning the characteristics of the output sound and the acquired sound, and distributes the model to the terminal device 10 of each user U. The terminal device 10 of the user U estimates the input using the model and performs information processing corresponding to the estimated input. For example, when the user U shakes hands with the stuffed animal (holding hands + waving), the terminal device 10 estimates that the deformation is that of a handshake using the model and detects the handshake. Then, in response to the user U's input of "shaking hands," the terminal device 10 outputs a sound such as "hello" or "thank you" from the stuffed animal. At this time, the terminal device 10 may estimate deformations of individual states such as "holding hands" and "waving," and estimate multiple inputs according to each deformation. Note that learning and information processing using the model may be processed on the cloud side.

[0043] According to this embodiment, by attaching (or incorporating) a smart device (or a pair of a speaker and a microphone) to (or into) a soft object such as a stuffed toy or a cushion and performing active acoustic sensing, deformations (partial compression, overall compression, twisting, bending, etc.) of the object can be detected. In addition, by assigning commands for operating home appliances (turning lights or power on / off, adjusting audio volume, etc.) to each deformation (or the shape after each deformation), it becomes possible to input commands based on these deformations.

[0044] Specifically, a speaker SK capable of outputting sound containing a sound of a predetermined wavelength and a microphone MK capable of acquiring the output sound output by the speaker SK are attached to a predetermined object such as a stuffed toy or cushion. Then, the server device 100 estimates the deformation of the predetermined object according to the characteristics of the sound (acquired sound) acquired by the microphone MK attached to the predetermined object.

[0045] In practice, in the above embodiment, the server device 100 may be replaced by another terminal device 10 located at a remote location or the terminal device 10 of the target user U. In other words, the subject of the process is not limited to the server device 100, but the other terminal device 10 or the terminal device 10 of the target user U may estimate the deformation of the stuffed toy T using active acoustic sensing.

[0046] Also, stuffed toys, cushions, etc. are merely examples. In reality, the objects are not limited to stuffed toys, cushions, etc., but active acoustic sensing may be performed by attaching (or incorporating) a smart device (or a pair of a speaker and a microphone) to a pillow, a futon, a mat, or the seat or back of an office chair or sofa, etc., to estimate deformation of these objects, estimate an input from user U corresponding to the estimated deformation, and perform processing or control according to the estimated input.

[0047] [2. Example of information processing system configuration] Next, a configuration of an information processing system 1 including a server device 100 according to an embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the information processing system 1 according to an embodiment. As shown in Fig. 2, the information processing system 1 according to an embodiment includes a terminal device 10 and a server device 100. These various devices are connected to each other via a network N so as to be able to communicate with each other via a wired or wireless connection. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

[0048] Furthermore, the number of devices included in the information processing system 1 shown in Fig. 2 is not limited to that shown in the figure. For example, in Fig. 2, for the sake of simplicity, only one terminal device 10 is shown, but this is merely an example and is not limiting, and two or more devices may be included.

[0049] The terminal device 10 is an information processing device used by a user U. For example, the terminal device 10 is a smart device such as a smartphone or a tablet terminal, a feature phone, a PC (Personal Computer), a PDA (Personal Digital Assistant), a game console or AV device with a communication function, a car navigation system, a wearable device such as a smart watch or a head-mounted display, smart glasses, etc.

[0050] In addition, the terminal device 10 can connect to the network N via a wireless communication network such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation: 5th generation mobile communication system), or via short-range wireless communication such as Bluetooth (registered trademark) or wireless LAN (Local Area Network), and communicate with the server device 100.

[0051] The server device 100 is, for example, a computer such as a PC or a blade server, or a mainframe or a workstation, etc. The server device 100 may be realized by cloud computing.

[0052] [3. Example of terminal device configuration] Next, the configuration of the terminal device 10 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the configuration of the terminal device 10. As shown in Fig. 3, the terminal device 10 includes a communication unit 11, a display unit 12, an input unit 13, a positioning unit 14, a sensor unit 20, a control unit 30 (controller), and a storage unit 40.

[0053] (Communications Department 11) The communication unit 11 is connected to a network N (see FIG. 2) by wire or wirelessly, and transmits and receives information to and from the server device 100 via the network N. For example, the communication unit 11 is realized by a NIC (Network Interface Card), an antenna, etc.

[0054] (Display section 12) Display unit 12 is a display device that displays various information such as position information. For example, display unit 12 is a liquid crystal display (LCD) or an organic electro-luminescent display (OLED). Display unit 12 is also a touch panel display, but is not limited to this.

[0055] (Input section 13) The input unit 13 is an input device that accepts various operations from the user U. For example, the input unit 13 has buttons for inputting characters, numbers, etc. The input unit 13 may be an input / output port (I / O port), a USB (Universal Serial Bus) port, etc. If the display unit 12 is a touch panel display, a part of the display unit 12 functions as the input unit 13. The input unit 13 may be a microphone that accepts voice input from the user U. The microphone may be wireless.

[0056] (Positioning unit 14) The positioning unit 14 receives signals (radio waves) transmitted from satellites of a GPS (Global Positioning System), and acquires position information (e.g., latitude and longitude) indicating the current position of the terminal device 10, which is the device itself, based on the received signals. That is, the positioning unit 14 positions the position of the terminal device 10. Note that GPS is merely an example of a GNSS (Global Navigation Satellite System).

[0057] The positioning unit 14 can also measure the position using various methods other than GPS. For example, the positioning unit 14 may measure the position by using various communication functions of the terminal device 10 as an auxiliary positioning means for position correction, etc., as described below.

[0058] (Wi-Fi positioning) For example, the positioning unit 14 uses a Wi-Fi (registered trademark) communication function of the terminal device 10 or a communication network provided by each communication company to measure the position of the terminal device 10. Specifically, the positioning unit 14 performs Wi-Fi communication or the like and measures the distance to a nearby base station or access point, thereby measuring the position of the terminal device 10.

[0059] (Beacon positioning) The positioning unit 14 may also measure the position by using a Bluetooth (registered trademark) function of the terminal device 10. For example, the positioning unit 14 measures the position of the terminal device 10 by connecting to a beacon transmitter connected by the Bluetooth (registered trademark) function.

[0060] (geomagnetic positioning) The positioning unit 14 also measures the position of the terminal device 10 based on a geomagnetic pattern of a structure that has been measured in advance and a geomagnetic sensor that the terminal device 10 has.

[0061] (RFID positioning) Furthermore, for example, if the terminal device 10 has a function of an RFID (Radio Frequency Identification) tag equivalent to a contactless IC card used at station ticket gates, in stores, etc., or has a function of reading an RFID tag, the location where the terminal device 10 was used is recorded together with information on the payment or the like made by the terminal device 10. The positioning unit 14 may obtain such information to determine the location of the terminal device 10. Alternatively, the location may be determined by an optical sensor, an infrared sensor, or the like provided in the terminal device 10.

[0062] The positioning unit 14 may measure the position of the terminal device 10 using one or a combination of the above-mentioned positioning means, as needed.

[0063] (Sensor unit 20) The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10. The connection may be wired or wireless. For example, the sensors may be detection devices other than the terminal device 10, such as wearable devices or wireless devices. In the example shown in FIG. 3 , the sensor unit 20 includes an acceleration sensor 21, a gyro sensor 22, a barometric pressure sensor 23, a temperature sensor 24, a sound sensor 25, a light sensor 26, a magnetic sensor 27, and an image sensor (camera) 28.

[0064] The above-described sensors 21 to 28 are merely examples and are not intended to be limiting. That is, the sensor unit 20 may be configured to include some of the sensors 21 to 28, or may include other sensors such as a humidity sensor in addition to or instead of the sensors 21 to 28.

[0065] The acceleration sensor 21 is, for example, a three-axis acceleration sensor, and detects physical movements of the terminal device 10, such as the direction of movement, speed, and acceleration of the terminal device 10. The gyro sensor 22 detects physical movements of the terminal device 10, such as tilt in three axial directions, based on the angular velocity of the terminal device 10. The air pressure sensor 23 detects, for example, the air pressure around the terminal device 10.

[0066] Since the terminal device 10 includes the acceleration sensor 21, the gyro sensor 22, the atmospheric pressure sensor 23, etc., it is possible to measure the position of the terminal device 10 using a technique such as Pedestrian Dead-Reckoning (PDR) that uses these sensors 21 to 23. This makes it possible to obtain indoor position information that is difficult to obtain using a positioning system such as GPS.

[0067] For example, the number of steps, walking speed, and distance walked can be calculated using a pedometer that uses the acceleration sensor 21. In addition, the direction of travel, line of sight, and body tilt of the user U can be determined using the gyro sensor 22. In addition, the altitude and floor on which the terminal device 10 of the user U is located can be determined from the air pressure detected by the air pressure sensor 23.

[0068] The temperature sensor 24 detects, for example, the temperature around the terminal device 10. The sound sensor 25 detects, for example, the sound around the terminal device 10. The light sensor 26 detects the illuminance around the terminal device 10. The magnetic sensor 27 detects, for example, the geomagnetism around the terminal device 10. The image sensor 28 captures an image around the terminal device 10.

[0069] The above-mentioned air pressure sensor 23, temperature sensor 24, sound sensor 25, light sensor 26, and image sensor 28 can detect the air pressure, temperature, sound, and illuminance, respectively, and capture images of the surroundings, thereby detecting the environment and situation around the terminal device 10. Furthermore, the accuracy of the location information of the terminal device 10 can be improved based on the environment and situation around the terminal device 10.

[0070] (control unit 30) The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM, input / output ports, etc., and various other circuits. The control unit 30 may also be configured with hardware such as an integrated circuit, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 30 includes a transmitting unit 31, a receiving unit 32, and a processing unit 33.

[0071] (Transmitter 31) The transmission unit 31 can transmit, for example, various information input by the user U using the input unit 13, various information detected by each sensor 21 to 28 mounted on or connected to the terminal device 10, and location information of the terminal device 10 measured by the positioning unit 14 to the server device 100 via the communication unit 11.

[0072] (Receiving unit 32) The receiving unit 32 can receive various types of information provided by the server device 100 and requests for various types of information from the server device 100 via the communication unit 11.

[0073] (Processing unit 33) The processing unit 33 controls the entire terminal device 10, including the display unit 12. For example, the processing unit 33 can output various information transmitted by the transmitting unit 31 and various information received from the server device 100 by the receiving unit 32 to the display unit 12 for display.

[0074] In this embodiment, the processing unit 33 outputs sound (output sound) from the terminal device 10 or speaker SK attached to a stuffed toy, a cushion, or the like, and acquires sound (acquired sound) from the terminal device 10 or microphone MK attached to the stuffed toy, a cushion, or the like. At this time, the processing unit 33 may output sound from the speaker SK in response to a command to output sound from the speaker SK received by the receiving unit 32 from the server device 100 or another terminal device 10. Furthermore, the processing unit 33 may transmit information regarding the sound acquired by the microphone MK from the transmitting unit 31 to the server device 100 or another terminal device 10.

[0075] (Storage unit 40) The storage unit 40 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), an optical disk, etc. The storage unit 40 stores various programs, various data, etc.

[0076] [4. Server device configuration example] Next, the configuration of the server device 100 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the server device 100 according to the embodiment. As shown in Fig. 4, the server device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0077] (Communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is also connected to a network N (see FIG. 2) by wire or wirelessly.

[0078] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as an HDD, an SSD, an optical disk, etc. As shown in Fig. 4, the storage unit 120 has a user information database 121, a history information database 122, and a deformation information database 123.

[0079] (User Information Database 121) The user information database 121 stores user information about the user U. For example, the user information database 121 stores various information such as the attributes of the user U. FIG. 5 is a diagram showing an example of the user information database 121. In the example shown in FIG. 5, the user information database 121 has items such as "User ID (Identifier)," "Age," "Gender," "Home," "Workplace," and "Interests."

[0080] The "user ID" indicates identification information for identifying the user U. The "user ID" may be the contact information of the user U (telephone number, email address, etc.), or may be identification information for identifying the terminal device 10 of the user U.

[0081] Furthermore, "age" indicates the age of user U identified by the user ID. Note that "age" may be information indicating the specific age of user U (e.g., 35 years old), or may be information indicating the generation of user U (e.g., 30s). Alternatively, "age" may be information indicating the date of birth of user U, or may be information indicating the generation of user U (e.g., born in the 1980s). Furthermore, "gender" indicates the gender of user U identified by the user ID.

[0082] Furthermore, "home" indicates the location information of the home of user U identified by the user ID. In the example shown in FIG. 5, "home" is illustrated as an abstract code such as "LC11," but it may also be latitude and longitude information, etc. Furthermore, for example, "home" may also be the name of an area or an address.

[0083] Furthermore, "workplace" indicates location information of the workplace (school in the case of a student) of user U identified by the user ID. In the example shown in FIG. 5, "workplace" is illustrated as an abstract code such as "LC12," but it may also be latitude and longitude information, etc. Furthermore, for example, "workplace" may also be the name of a region or an address.

[0084] Furthermore, "interests" indicate the interests of user U identified by the user ID. In other words, "interests" indicate subjects in which user U identified by the user ID is highly interested. For example, "interests" may be search queries (keywords) entered by user U into a search engine. In the example shown in FIG. 5, each user U is shown with one "interest," but there may be multiple "interests."

[0085] For example, in the example shown in FIG. 5, the age of user U identified by user ID "U1" is "20s" and the gender is "male." Furthermore, for example, the home address of user U identified by user ID "U1" is "LC11." Furthermore, for example, the workplace of user U identified by user ID "U1" is "LC12." Furthermore, for example, the user U identified by user ID "U1" is interested in "sports."

[0086] 5, abstract values ​​such as "U1", "LC11", and "LC12" are used for illustration, but "U1", "LC11", and "LC12" are assumed to store information such as specific character strings and numerical values. Below, abstract values ​​may also be illustrated in diagrams relating to other information.

[0087] The user information database 121 may store various types of information depending on the purpose, without being limited to the above. For example, the user information database 121 may store various types of information related to the terminal device 10 of the user U. The user information database 121 may also store information related to the user U's attributes, such as demographic attributes, psychographic attributes, geographic attributes, and behavioral attributes. For example, the user information database 121 may store information such as name, family structure, hometown (hometown), occupation, job title, income, qualifications, type of residence (detached house, apartment, etc.), whether or not the user has a car, commuting time, commuting route, commuter pass area (station, line, etc.), frequently used stations (other than the station nearest to home or workplace), extracurricular activities (location, time zone, etc.), hobbies, interests, lifestyle, etc.

[0088] (History Information Database 122) The history information database 122 stores various information related to history information (log data) that indicates the behavior of the user U. Fig. 6 is a diagram showing an example of the history information database 122. In the example shown in Fig. 6, the history information database 122 has items such as "user ID," "location history," "search history," "browsing history," "purchase history," and "posting history."

[0089] "User ID" indicates identification information for identifying user U. "Location history" indicates the location history, which is the history of user U's location and movements. "Search history" indicates the search history, which is the history of search queries entered by user U. "Browsing history" indicates the browsing history, which is the history of content viewed by user U. "Purchase history" indicates the purchase history, which is the history of purchases made by user U. "Posting history" indicates the posting history, which is the history of posts made by user U. "Posting history" may also include questions about user U's possessions.

[0090] For example, in the example shown in Figure 6, user U, identified by user ID "U1," moved as shown in "Location History #1," searched as shown in "Search History #1," viewed content as shown in "Viewing History #1," purchased specific products at specific stores as shown in "Purchase History #1," and posted as shown in "Posting History."

[0091] Here, in the example shown in Figure 6, abstract values ​​such as "U1", "Location History #1", "Search History #1", "Browsing History #1", "Purchase History #1", and "Post History #1" are used for the illustration, but "U1", "Location History #1", "Search History #1", "Browsing History #1", "Purchase History #1", and "Post History #1" are assumed to store specific information such as character strings and numbers.

[0092] The history information database 122 is not limited to the above and may store various types of information depending on the purpose. For example, the history information database 122 may store the user U's usage history of a predetermined service. The history information database 122 may also store the user U's store visit history or facility visit history. The history information database 122 may also store the user U's payment history (electronic payment) using the terminal device 10.

[0093] (Deformation Information Database 123) The deformation information database 123 stores various information related to the estimated results of deformation of an object. Fig. 7 is a diagram showing an example of the deformation information database 123. In the example shown in Fig. 7, the deformation information database 123 has items such as "user ID," "object," "installation position," "output sound," "acquired sound," "deformation," "input," and "response."

[0094] The "user ID" indicates identification information for identifying the user U. The "object" indicates a predetermined object that can be transformed by the user U and to which the terminal device 10 (or a pair of a speaker and a microphone) is attached. For example, the object is a soft object such as a stuffed toy or a cushion.

[0095] Furthermore, the "installation position" indicates the location (place, part, portion) where the terminal device 10 (or a pair of speaker and microphone) is attached to a predetermined object. The installation position may be information indicating whether the installation position is outside or inside the predetermined object. The installation position may also be a coordinate position in a three-dimensional space related to the predetermined object. The installation position may also be information indicating separately the location where the speaker is attached (speaker position) and the location where the microphone is attached (microphone position).

[0096] Furthermore, "output sound" refers to the characteristics of sound (output sound) output from a terminal device 10 or speaker SK attached to a predetermined location (installation position) of a predetermined object. Furthermore, "acquired sound" refers to the characteristics of sound (acquired sound) acquired by a terminal device 10 or microphone NK attached to a predetermined position of a predetermined object.

[0097] Furthermore, "deformation" indicates the deformation of a specified object due to the weight or force applied by the user U. The deformation of the stuffed animal T is linked to the characteristics of the acquired sound (or the characteristics of the difference between the output sound and the acquired sound). Note that "deformation" may be information indicating the deformation pattern of the stuffed animal T, or may be information indicating the shape of the stuffed animal T after deformation.

[0098] Furthermore, "input" refers to an input associated with the deformation of a predetermined object. For example, the input may be a remote control operation (button operation) of a home appliance, AV device, or the like. Furthermore, "response" refers to a response to the input. For example, the response may be a voice output, or the operation / control of a home appliance, AV device, or the like.

[0099] The contents of each of the "transformation," "input," and "response" items can be changed as desired. In other words, the combination of "transformation," "input," and "response" can be freely changed. Furthermore, for one "transformation," different "input" and "response" can be set according to the user U's location information, attribute information, or history information. For example, an "input" and a "response" can be set for each combination of "transformation" and at least one of the user U's "location information," "attribute information," and "history information."

[0100] For example, in the example shown in Figure 7, when a user U identified by user ID "U1" transforms a specific object, a "stuffed toy," a sound indicated as "output sound #1" is output from the terminal device 10 or speaker SK attached to the "installation location #1" of the "stuffed toy," and a sound indicated as "acquired sound #1" is acquired by the terminal device 10 or microphone MK attached to the "installation location #1" of the "stuffed toy." As a result of active acoustic sensing, "deformation #1," the state in which the "stuffed toy" is transformed by user U applying his / her weight or force, is estimated, and "response #1" corresponding to "input #1" based on the estimated "deformation #1" is output.

[0101] Here, in the example shown in Figure 7, abstract values ​​such as "U1," "Installation position #1," "Output sound #1," "Acquired sound #1," "Variation #1," "Input #1," and "Response #1" are used for the illustration, but "U1," "Installation position #1," "Output sound #1," "Acquired sound #1," "Variation #1," "Input #1," and "Response #1" are assumed to store specific information such as character strings and numerical values.

[0102] The deformation information database 123 is not limited to the above, and may store various types of information depending on the purpose. For example, the deformation information database 123 may store a model constructed by learning the characteristics of the acquired sound (or the characteristics of the difference between the output sound and the acquired sound). The deformation information database 123 may also store information about the type of a predetermined object (model number, material, size, etc.). The deformation information database 123 may also store information about the type of the terminal device 10 (or speaker and microphone) attached to the predetermined object.

[0103] (control unit 130) 4, the explanation will be continued. The control unit 130 is a controller, and is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like, executing various programs (corresponding to examples of information processing programs) stored in a storage device inside the server device 100 using a storage area such as a RAM as a working area. In the example shown in FIG. 4, the control unit 130 has an acquisition unit 131, a command unit 132, an estimation unit 133, a learning unit 134, a response unit 135, and a provision unit 136.

[0104] (Acquisition part 131) The acquisition unit 131 acquires a search query input by the user U. For example, when the user U inputs a search query into a search engine or the like to perform a keyword search, the acquisition unit 131 acquires the search query via the communication unit 110. That is, the acquisition unit 131 acquires, via the communication unit 110, the keywords input by the user U into the search box of a search engine, website, or app.

[0105] Furthermore, the acquisition unit 131 acquires user information about the user U via the communication unit 110. For example, the acquisition unit 131 acquires identification information (such as a user ID) indicating the user U, location information of the user U, attribute information of the user U, etc. from the terminal device 10 of the user U. Furthermore, the acquisition unit 131 may acquire the identification information indicating the user U, attribute information of the user U, etc. when registering the user U. Then, the acquisition unit 131 registers the user information in the user information database 121 of the storage unit 120.

[0106] Furthermore, the acquisition unit 131 acquires various types of history information (log data) indicating the behavior of the user U via the communication unit 110. For example, the acquisition unit 131 acquires various types of history information indicating the behavior of the user U from the terminal device 10 of the user U or from various servers based on the user ID or the like. Then, the acquisition unit 131 registers the various types of history information in the history information database 122 of the storage unit 120.

[0107] (Instruction part 132) The command unit 132 issues a command via the communication unit 110 to output sound from a speaker SK attached to a predetermined object that can be transformed by the user U. The predetermined object is a predetermined object that can be transformed. For example, the object is a soft object such as a stuffed toy or a cushion. For example, the command unit 132 issues a command via the communication unit 110 to output sound from the terminal device 10 or speaker SK attached to the stuffed toy, cushion, or the like.

[0108] (Estimation part 133) The estimation unit 133 estimates the deformation of the predetermined object using active acoustic sensing based on the sound acquired by the microphone MK attached to the predetermined object. For example, the estimation unit 133 estimates the deformation of the predetermined object based on the sound characteristics for each deformation of the predetermined object.

[0109] Furthermore, the estimation unit 133 estimates an input made by the user U by deforming the predetermined object based on the estimated deformation of the predetermined object. At this time, the estimation unit 133 may estimate an input made by the user U by deforming the predetermined object based on the estimated deformation of the predetermined object and position information, attribute information, or history information of the user U. In other words, even if the deformation of the predetermined object is the same, the estimation unit 133 may estimate a different input depending on the position information, attribute information, or history information of the user U.

[0110] (Learning Section 134) The learning unit 134 learns the sound characteristics for each deformation of a predetermined object and constructs a model. At this time, the estimation unit 133 estimates the deformation of the predetermined object using the model.

[0111] For example, the learning unit 134 constructs a model by learning the characteristics of the acquired sound acquired by the microphone MK when the user U deforms a specified object, or by learning both the characteristics of the output sound output from the speaker SP and the characteristics of the acquired sound acquired by the microphone MK.

[0112] Alternatively, the learning unit 134 constructs a model by learning the difference between the characteristics of the output sound output from the speaker SP when the user U deforms a specified object and the characteristics of the acquired sound acquired by the microphone MK.

[0113] Furthermore, the learning unit 134 learns the sound characteristics for each deformation of the predetermined object for each location where a speaker SP and a microphone MK are attached to the predetermined object, and constructs a model.

[0114] At this time, the learning unit 134 learns the sound characteristics of each deformation of a predetermined object for each type of predetermined object and constructs a model.

[0115] Furthermore, the learning unit 134 learns the degree of wear of the material of the predetermined object and the characteristics of the sound acquired by the microphone MK to construct a model.

[0116] (Response section 135) The response unit 135 outputs a response in response to an input from the user U based on the estimation result of the deformation of the predetermined object. For example, the response unit 135 outputs a predetermined sound in response to an input from the user U based on the estimation result of the deformation of the predetermined object. Alternatively, the response unit 135 outputs a control signal for home appliances, AV equipment, etc. in response to the input from the user U based on the estimation result of the deformation of the predetermined object, and operates / controls these.

[0117] Furthermore, even if the deformation of a specified object is the same, if a different input is estimated depending on the user's position information, attribute information, or history information, the response unit 135 outputs a response according to the estimated input of user U, thereby outputting a response according to a different input depending on the user's position information, attribute information, or history information, even if the deformation of a specified object is the same.

[0118] (Provider 136) The providing unit 136 provides information about the response output from the responding unit 126 to a home appliance, AV device, other terminal device, server device, or the like that is the target of input from the user U, via the communication unit 110. Alternatively, the providing unit 136 provides information about the response output from the responding unit 126 to the terminal device 10 or microphone MK of the user U, via the communication unit 110. For example, if the information about the response is audio, the providing unit 136 provides audio data, an instruction to output audio, or the like to the terminal device 10 or microphone MK of the user U, via the communication unit 110.

[0119] [5. Processing Procedure] Next, a processing procedure performed by the terminal device 10 and the server device 100 according to the embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the processing procedure according to the embodiment. Note that the processing procedure shown below is repeatedly executed by the control unit 30 of the terminal device 10 and the control unit 130 of the server device 100.

[0120] 8, the command unit 132 of the server device 100 issues a command to the terminal device 10 or the speaker SK attached to the stuffed toy, cushion, etc. via the communication unit 110 to output sound from the terminal device 10 or the speaker SK attached to the stuffed toy, cushion, etc. (step S101). At this time, the terminal device 10 or the speaker SK attached to the stuffed toy, cushion, etc. outputs sound.

[0121] Next, the estimation unit 133 of the server device 100 receives the sound acquired by the terminal device 10 or the microphone MK attached to the stuffed toy, cushion, etc. via the communication unit 110 (step S102). At this time, the terminal device 10 or the microphone MK attached to the stuffed toy, cushion, etc. acquires the sound output from the speaker SK and transmits information about the acquired sound to the server device 100.

[0122] Next, the estimation unit 133 of the server device 100 estimates the deformation of the stuffed toy T using active acoustic sensing based on the sound acquired by the terminal device 10 or the microphone MK attached to the stuffed toy, cushion, etc. (step S103). At this time, the estimation unit 133 of the server device 100 estimates the deformation of the stuffed toy T using a model described later.

[0123] Next, the learning unit 134 of the server device 100 constructs (generates and updates) a model that has learned the sound characteristics for each transformation of the stuffed toy T (step S104).

[0124] Next, the learning unit 134 of the server device 100 constructs a model that learns the sound characteristics for each deformation of the stuffed animal T for each installation position (mounting location) of the terminal device 10 (or speaker and microphone pair) on the stuffed animal T (step S105).

[0125] Next, the learning unit 134 of the server device 100 constructs a model for each stuffed toy T that has learned the sound characteristics for each transformation of the stuffed toy T (step S106).

[0126] Next, the learning unit 134 of the server device 100 performs user calibration (step S107). Specifically, the learning unit 134 learns from data of an unspecified number of users up to a certain point (a predetermined stage) to build a model of all users, and when sufficient data of each user has been accumulated, the learning unit 134 performs calibration for each user to build a model of each individual user. Then, the estimation unit 133 of the server device 100 estimates the deformation of the stuffed toy T using the model.

[0127] Next, response unit 135 of server device 100 outputs a response in response to the input of user U based on the estimated result of the deformation of the predetermined object (step S108). At this time, estimation unit 133 of server device 100 estimates the input made by user U by deforming the predetermined object based on the estimated deformation of the predetermined object. Response unit 135 outputs a response in response to the estimated input of user U.

[0128] Next, the providing unit 136 of the server device 100 provides information about the response output from the responding unit 126 to the home appliance or AV device that is the target of the input from the user U, or to another terminal device, server device, etc., via the communication unit 110 (step S109). Alternatively, the providing unit 136 provides information about the estimated results of the deformation of each object to the terminal device 10 or microphone MK of the user U via the communication unit 110. For example, the providing unit 136 provides information about the response output from the responding unit 135 to the terminal device 10 or microphone MK of the user U via the communication unit 110.

[0129] [6. Modifications] The terminal device 10 and the server device 100 described above may be implemented in various different forms other than the above embodiment. Therefore, modifications of the embodiment will be described below.

[0130] In the above embodiment, some or all of the processing executed by the server device 100 may actually be executed by the terminal device 10. For example, the processing may be completed in a stand-alone manner (by the terminal device 10 alone). In this case, the terminal device 10 is assumed to have the functions of the server device 100 in the above embodiment. Furthermore, in the above embodiment, the terminal device 10 is linked to the server device 100, and therefore, from the perspective of the user U, it appears that the processing of the server device 100 is also being executed by the terminal device 10. In other words, from another perspective, the terminal device 10 can also be said to be equipped with the server device 100.

[0131] In the above embodiment, a predetermined object such as a stuffed animal may have a plurality of speakers SK and a plurality of microphones MK attached thereto. The number of speakers SK and the number of microphones MK do not have to be the same. The speakers SK and the microphones MK may each be a terminal device 10. For example, sound output from a terminal device 10 or a single speaker SK attached to a stuffed animal, cushion, or the like may be acquired by microphones NK attached to multiple locations on the stuffed animal, cushion, or the like. Conversely, sound output from terminal devices 10 or speakers SK attached to multiple locations on the stuffed animal, cushion, or the like may be acquired by a terminal device 10 or a single microphone NK attached to the stuffed animal, cushion, or the like. Note that if the microphone MK cannot properly acquire the sound when multiple speakers SK are output simultaneously, the timing at which the sounds are output from each speaker SK may be shifted. In this case, the server device 100 may estimate the deformation of the stuffed animal T based on which speaker SK output the sound and which microphone MK acquired the sound.

[0132] Furthermore, in the above embodiment, a plurality of pairs (sets) of a speaker SK and a microphone MK may be installed on a predetermined object such as a stuffed toy. In this case, the number of speakers SK and microphones MK is the same. The speakers SK and microphones MK may each be terminal devices 10. For example, a pair of a speaker SK and a microphone MK may be installed on each of the backrest and seat of an office chair. Note that if the microphone MK cannot properly pick up the sound when multiple speakers SK are output simultaneously, the timing at which the sound is output from each speaker SK may be staggered.

[0133] Furthermore, in the above embodiment, the server device 100 may store the transformation history of the stuffed toy T as history information. For example, the server device 100 may track the transformation history of the stuffed toy T, detect a user U who is transforming the stuffed toy T for a certain period of time or in a predetermined time period, and output an unusual response to the user U. In other words, the server device 100 may infer that the continuation of the transformation for a certain period of time or the transformation of the stuffed toy T in a predetermined time period is an unusual input, and output an unusual response in response to that input.

[0134] Furthermore, in the above embodiment, the server device 100 may change the "input" of the user U corresponding to the deformation of the stuffed toy T and the "response" in response to the input, depending on the location information, attribute information, or history information of the user U. For example, even if the deformation of the stuffed toy T is the same, the server device 100 may change the "input" of the user U corresponding to the deformation and the "response" in response to the input, depending on the location, age, gender, etc. of the user U (and further, the user U's usage history and operation history of a predetermined service, etc.). Furthermore, even if the deformation of the stuffed toy T is the same, the server device 100 may change the content of the "input" and "response" corresponding to the deformation depending on whether the user U is at home or out.

[0135] [7. Effects] As described above, the information processing device (terminal device 10 and server device 100) according to the present application includes a command unit 132 that issues a command to output sound from a speaker attached to a predetermined object that can be deformed by a user U, an estimation unit 133 that estimates the deformation of the predetermined object using active acoustic sensing based on the sound acquired by a microphone attached to the predetermined object, and a response unit 135 that outputs a response in response to an input from the user U based on the estimated result of the deformation of the predetermined object.

[0136] The estimation unit 133 further estimates the input of the user U made by the user U by deforming the predetermined object based on the estimated deformation of the predetermined object. The response unit 135 outputs a response according to the estimated input of the user U.

[0137] The estimation unit 133 estimates the deformation of the predetermined object based on the sound characteristics for each deformation of the predetermined object.

[0138] The information processing device according to the present application further includes a learning unit 134 that learns sound features for each deformation of a predetermined object and constructs a model. An estimation unit 133 estimates the deformation of the predetermined object using the model.

[0139] The learning unit 134 constructs a model by learning the characteristics of the acquired sound acquired by the microphone MK when the user U deforms a specified object, or both the characteristics of the output sound output from the speaker SP and the characteristics of the acquired sound acquired by the microphone MK.

[0140] The learning unit 134 learns the difference between the characteristics of the output sound output from the speaker when the user U has transformed a predetermined object and the characteristics of the acquired sound acquired by the microphone, and constructs a model.

[0141] The learning unit 134 learns sound characteristics for each deformation of a predetermined object for each location where a speaker and a microphone are attached to the predetermined object, and constructs a model.

[0142] The learning unit 134 learns the sound characteristics of each deformation of a predetermined object for each type of predetermined object and constructs a model.

[0143] The learning unit 134 learns the degree of wear of the material of a predetermined object and the characteristics of the sound acquired by the microphone MK to construct a model.

[0144] The estimation unit 133 further estimates the input made by the user U by deforming the predetermined object, based on the estimated deformation of the predetermined object and the position information, attribute information, or history information of the user U. The response unit 135 outputs a response according to the estimated input of the user U, and thereby outputs a response according to an input that differs depending on the position information, attribute information, or history information of the user U, even if the deformation of the predetermined object is the same.

[0145] By using any one or a combination of the above-mentioned processes, the information processing device of the present application can attach a smart device (or a pair of speaker and microphone) to a target object and estimate the deformation of the object using active acoustic sensing.

[0146] [8. Hardware Configuration] The terminal device 10 and the server device 100 according to the above-described embodiments are realized by a computer 1000 having a configuration as shown in Fig. 9, for example. The following description will be given taking the server device 100 as an example. Fig. 9 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output I / F (Interface) 1060, an input I / F 1070, and a network I / F 1080 are connected via a bus 1090.

[0147] The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The arithmetic device 1030 is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

[0148] The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is realized by a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. The secondary storage device 1050 may be an internal storage device or an external storage device. The secondary storage device 1050 may also be a removable storage medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) memory card. The secondary storage device 1050 may also be cloud storage (online storage), a NAS (Network Attached Storage), a file server, or the like.

[0149] The output I / F 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a display, a projector, a printer, etc., and is realized by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (High Definition Multimedia Interface), etc. The input I / F 1070 is an interface for receiving information from various input devices 1020, such as a mouse, a keyboard, a keypad, a button, a scanner, etc., and is realized by a USB, etc.

[0150] Furthermore, the output I / F 1060 and the input I / F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

[0151] The output device 1010 and the input device 1020 may be integrated into one device, such as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated into one device as an input / output I / F.

[0152] The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0153] The network I / F 1080 receives data from other devices via the network N and sends it to the arithmetic device 1030, and also transmits data generated by the arithmetic device 1030 to other devices via the network N.

[0154] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output I / F 1060 and the input I / F 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0155] For example, when the computer 1000 functions as the server device 100, the arithmetic unit 1030 of the computer 1000 executes a program loaded onto the primary storage device 1040 to realize the functions of the control unit 130. The arithmetic unit 1030 of the computer 1000 may also load a program acquired from another device via the network I / F 1080 onto the primary storage device 1040 and execute the loaded program. The arithmetic unit 1030 of the computer 1000 may also cooperate with the other device via the network I / F 1080 to call and use the functions and data of a program from another program of the other device.

[0156] [9. Other] Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the scope of so-called equivalents. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments.

[0157] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0158] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0159] For example, the above-mentioned server device 100 may be realized by multiple server computers, and depending on the function, the configuration can be flexibly changed, such as by calling an external platform using an API (Application Programming Interface) or network computing.

[0160] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0161] Furthermore, the above-mentioned "section, module, unit" can be read as "means" or "circuit," etc. For example, an acquisition unit can be read as an acquisition means or an acquisition circuit. [Explanation of symbols]

[0162] 1. Information Processing Systems 10 Terminal Equipment SK Speaker MK Microphone 100 Server device 110 Communications Department 120 Storage section 121 User Information Database 122 Historical Information Database 123 Deformation Information Database 130 Control Unit 131 Acquisition Department 132 Command part 133 Estimation Department 134 Learning Department 135 Response Section 136 Provision Department

Claims

1. a command unit that issues a command to output sound from a speaker of a smartphone that is detachably attached to a predetermined object that is a stuffed toy, a cushion, or an object equivalent thereto and that can be deformed by a user's weight or force; a learning unit that uses sound acquired by a microphone of a smartphone detachably attached to the predetermined object as data on sound characteristics for each deformation of the predetermined object, learns with data on an unspecified number of users to build a model of all users until a sufficient number of individual user data has been accumulated to build a model of each individual user, and calibrates for each user to build a model of each individual user at the time when a sufficient number of individual user data has been accumulated; an estimation unit that, during active acoustic sensing, estimates deformation of the predetermined object based on sound acquired by a microphone of a smartphone detachably attached to the predetermined object, using a model of all users before a model of each user is constructed, and using the model of each individual user after the model of each individual user is constructed, and estimates an input of the user to the predetermined object in accordance with the deformation of the predetermined object; a response unit that outputs a different voice from the predetermined object in response to the estimated input of the user; An information processing device comprising:

2. The estimation unit further estimates an input of the user made by the user by deforming the predetermined object based on the estimated deformation of the predetermined object; The response unit outputs a response according to the estimated input of the user.

2. The information processing apparatus according to claim 1, wherein:

3. The estimation unit estimates the deformation of the predetermined object based on sound characteristics for each deformation of the predetermined object.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. The learning unit constructs a model by learning a difference between a feature of an acquired sound of the original shape of the predetermined object and a feature of an acquired sound of the deformed shape of the predetermined object; The estimation unit estimates a deformation of the predetermined object using the model.

4. The information processing device according to claim 1, wherein the information processing device is a computer.

5. The learning unit learns features of the acquired sound acquired by the microphone in a state in which the user deforms the predetermined object, or learns both features of the output sound output from the speaker and features of the acquired sound acquired by the microphone to construct a model.

5. The information processing apparatus according to claim 4,

6. The learning unit learns a difference between a feature of an output sound output from the speaker and a feature of an acquired sound acquired by the microphone in a state in which the user has deformed the predetermined object, and constructs a model.

6. The information processing apparatus according to claim 4, wherein:

7. The learning unit learns sound characteristics for each deformation of the predetermined object for each location where the speaker and the microphone are attached to the predetermined object, and constructs a model.

7. The information processing device according to claim 4, wherein:

8. The learning unit learns sound characteristics for each deformation of the predetermined object for each type of the predetermined object and constructs a model.

8. The information processing device according to claim 4, wherein:

9. The learning unit learns the degree of wear of the material of a predetermined object and the characteristics of the acquired sound acquired by the microphone to construct a model.

9. The information processing device according to claim 4, wherein:

10. the estimation unit further estimates an input made by the user by deforming the predetermined object based on the estimated deformation of the predetermined object and position information, attribute information, or history information of the user; The response unit outputs a response according to the estimated input of the user, so that even if the deformation of the predetermined object is the same, a response according to an input that differs depending on the position information, attribute information, or history information of the user is output.

10. The information processing device according to claim 1, wherein the information processing device is a computer.

11. An information processing method executed by an information processing device, a command step of issuing a command to output a sound from a speaker of a smartphone detachably attached to a predetermined object, such as a stuffed toy, a cushion, or an object equivalent thereto, which can be deformed by a user's weight or force; a learning process in which sound acquired by a microphone of a smartphone detachably attached to the predetermined object is used as data on sound characteristics for each deformation of the predetermined object, and a model of all users is constructed by learning with data from an unspecified number of users until a sufficient number of individual user data has been accumulated to construct a model of each individual user, and at the point in time when a sufficient number of individual user data has been accumulated to construct a model of each individual user, calibration is performed for each user to construct a model of each individual user; an estimation step of estimating deformation of the predetermined object using a model of all users before constructing a model of each individual user and using the model of each individual user after constructing a model of each individual user during active acoustic sensing based on sound acquired by a microphone of a smartphone detachably attached to the predetermined object, and estimating an input of the user to the predetermined object in accordance with the deformation of the predetermined object; a response step of outputting a different voice from the predetermined object in response to the estimated input of the user; An information processing method comprising:

12. a command procedure for issuing a command to output sound from a speaker of a smartphone detachably attached to a predetermined object that is a stuffed toy, a cushion, or an object equivalent thereto and that can be deformed by a user's weight or force; a learning procedure in which sound acquired by a microphone of a smartphone detachably attached to the predetermined object is used as data on sound characteristics for each deformation of the predetermined object, and a model of all users is constructed by learning with data of an unspecified number of users until a sufficient number of data of individual users has been accumulated to construct a model of each individual user, and at the point in time when a sufficient number of data of individual users has been accumulated to construct a model of each individual user, calibration is performed for each user to construct a model of each individual user; an estimation procedure of estimating deformation of the predetermined object using a model of all users before a model of each individual user is constructed, and using the model of each individual user after the model of each individual user is constructed, based on sound acquired by a microphone of a smartphone detachably attached to the predetermined object, during active acoustic sensing, and estimating an input of the user to the predetermined object in accordance with the deformation of the predetermined object; a response step of outputting a different voice from the predetermined object in response to the estimated input of the user; An information processing program that causes a computer to execute the above.

Images