Program, Information Processing Method, and Information Processing Apparatus

Abstract

To appropriately grasp the state of a called party before making a call.SOLUTION: An information processing device is caused to perform: acquiring media data related to image data and / or sound data transmitted from a first communication device; determining, based on the media data, a state of a user of the first communication device; and transmitting information about the state of the user to a second communication device that is associated with identification information of the first communication device and can communicate with the first communication device.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a program, an information processing method, and an information processing apparatus.

Background Art

[0002] Conventionally, there is a technology for notifying whether the other party is available before starting a voice call. For example, the calling party operates a telephone, connects to the Internet using the email address of the other party obtained from the notification destination, and obtains information on whether the other party is available, or provides a presence sensor, so that a technology for knowing whether the other party is available is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, although it is possible to know whether the call destination is available, it is not known what state the call destination is actually in at present. For example, even if the Internet used by the call destination is connected, there is a possibility that the person is not at the seat. In addition, when grasping whether a person is at the seat by providing a presence sensor, the cost increases due to providing the presence sensor. Therefore, a technology for appropriately grasping the state of the call destination before a call is required.

[0005] Therefore, an object of the present invention is to provide a program, an information processing method, and an information processing apparatus capable of appropriately grasping the state of a call destination before a call.

Means for Solving the Problems

[0006] A program according to an aspect of the present disclosure causes an information processing apparatus to acquire media data related to image data and / or audio data transmitted from a first communication apparatus, determine a state of a user of the first communication apparatus based on the media data, and transmit information regarding the state of the user to a second communication apparatus associated with the identification information of the first communication apparatus and capable of communicating with the first communication apparatus.

Effect of the Invention

[0007] According to a predetermined aspect of the present disclosure, the state of the call destination can be appropriately grasped before the call.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 7

Figure 8

Figure 9

Modes for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0010] <System Outline> FIG. 1 is a diagram for explaining the outline of the system of the present disclosure. In the example shown in FIG. 1, it is assumed that the user UB (for example, a child) wants to start a voice call, a video call, or a TV call (hereinafter, collectively referred to as a "call") with the user UA (for example, a parent). The user UA uses an information processing device 20A (first communication device) such as a mobile terminal, and the user UB uses an information processing device 20B (second communication device) such as a mobile terminal. The server 10 is an information processing device capable of performing call control for IP (Internet Protocol) phones and TV phones. The outline shown in FIG. 1 includes an example in which a call is realized by P2P communication using IP phones, WebRTC (Web Real-time Communication) technology, etc.

[0011] (1) Transmission of Image Data / Audio Data First, the information processing device 20A turns on the camera and microphone, and acquires images and sounds around the information processing device 20A. The information processing device 20A transmits media data related to the acquired image data and / or audio data (hereinafter, also referred to as "image data / audio data") to the server 10. Note that the information processing device 20A turns on at least one of the camera and the microphone constantly or periodically to acquire image data / audio data.

[0012] (2) Transmission of Status Information Server 10 receives media data related to image data / audio data from information processing device 20A, and based on this media data, determines the state of user UA who uses information processing device 20A. Server 10 determines the current state of user UA by performing, for example, image analysis or audio analysis. The current state of user UA includes, for example, at least one of a state regarding whether user UA is present around information processing device 20A based on an image, a state regarding an emotion based on the facial expression of user UA, a state of a location based on sound information around information processing device 20A, a state regarding bustlingness, and a state indicating whether user UA is available for a call.

[0013] Server 10 transmits information regarding the determined state of user UA (hereinafter also referred to as "state information") to information processing device 20B associated with the identification information (ID) of information processing device 20A. Server 10 holds, for example, call correspondence information that associates the identification information of devices that can communicate with each other.

[0014] (3) After confirming the user's state, sending a call request When information processing device 20B used by user UB receives the state information of user UA, it displays a screen on the display screen to inform user UB of the state of user UA. For example, information processing device 20B displays UI component M1 including a video, an animation, or text indicating the current state of user UA. UI component M1 is an icon, a window, a pop-up, or a predetermined area within an application screen. Also, current or past image data including user UA acquired by information processing device 20A may be displayed. At this time, in order to protect privacy, images other than user UA may be mosaicked or given a different background.

[0015] Assume that information processing device 20B receives an operation from user UB to call user UA. At this time, information processing device 20B transmits a call request to server 10.

[0016] (4) Establishing a call session When the server 10 receives a call request from the information processing device 20B, it sends the call request to the information processing device 20A associated with the identification information of the information processing device 20B. When the information processing device 20A receives the call request and accepts an operation to respond to the call from the user UA, it sends a call response to the call request to the server 10. After that, when a session is established between the information processing device 20A and the information processing device 20B, the server 10 starts transmitting and receiving data for the call (hereinafter, also referred to as "call data"). Also, if the server 10 does not receive a call response within a predetermined time, it may send a notice to that effect to the information processing device 20B and discard the call request.

[0017] Thereby, the user UB can appropriately grasp the current state of the called user UA before the call just by looking at the display screen of the information processing device 20B. For example, if a state where the user UA is in the room and relaxed is displayed on the UI component M1, the user UB can determine that it is okay to make a call and instruct the information processing device 20B to start the call.

[0018] Also, the user UA is notified of his / her own state without being particularly aware of it, and the user UB can check the state of the user UA without performing a special operation and can grasp the timing of the call. Also, since image data and sound data are used for the state determination, the appropriate state of the user UA can be determined, and the user UB can start the call after grasping the current state of the user UA.

[0019] Note that similar processing may be performed from the user UB side to the user UA side. That is, the state of the user UB is displayed on the information processing device 20A of the user UA, and the user UA can grasp the current state of the user UB.

[0020] [First Embodiment] Next, a system configuration example for realizing the above-described system will be described. FIG. 2 is a diagram showing an example of the schematic configuration of the information processing system 1 according to the first embodiment. As shown in FIG. 2, the server 10 and user terminals 20A, 20B, 20C, 20D,... used by each user are connected to be mutually communicable via the network N, thereby constituting the information processing system 1. Hereinafter, when it is not necessary to individually distinguish user terminals, the reference numeral 20 is used. Also, for the server 10, there may be a plurality of connections to the network N.

[0021] The server 10 is, for example, an information processing device capable of transmitting and receiving voice data and image data packetized by IP, and is also a device that performs call control such as outgoing calls (transmissions), incoming calls (incomings), responses, and disconnections of calls. Also, for call control, signaling protocols such as H.323, MGCP (Media Gateway Control Protocol), and SIP (Session Initiation Protocol) may be used.

[0022] The user terminal (communication device) 20 is an information processing device capable of accessing the network, and includes, by way of example and not limitation, mobile terminals such as smartphones, computers (by way of example and not limitation, desktops, laptops, tablets, etc.), media computer platforms (by way of example and not limitation, set-top boxes, digital video recorders, etc.), handheld computer devices (by way of example and not limitation, PDAs (Personal Digital Assistants), email clients, etc.), wearable terminals (by way of example and not limitation, glasses-type devices, watch-type devices, etc.), or other types of computers.

[0023] Network N can be configured to include multiple types of communication lines, communication networks, and various network devices. For example, network N may include a base station wirelessly connected to server 10, an access point for a wireless LAN (such as a WiFi router), a mobile communication network connected to the base station, a telephone line connected from the access point via a router or modem, a public line such as a cable TV line or an optical communication line, the Internet connected to user terminal 20, a mobile communication network, and a gateway device connecting the public line and the Internet. Note that the configuration of network N is not limited to the above example.

[0024] In the system configuration shown in FIG. 2, the current state determined based on the media data regarding the image data / audio data transmitted from user terminal 20A of the first user who is the call destination is displayed on user terminal 20B of the second user who is the call source.

[0025] User terminal 20B receives the status information of the first user obtained via server 10 and generates status notification information that can notify this status information. For example, the second user looks at the status notification information displayed on the screen and determines whether the first user is available for a call, and then performs an operation to start a call. At this time, user terminal 20B transmits a call request to user terminal 20A via server 10.

[0026] When user terminal 20A receives the call request and the second user performs an operation to respond, user terminal 20A transmits a call response to server 10. As a result, a call session is established, the call data including the call content such as voice and image is packetized into IP packets, and the IP packetized call data is transmitted and received between user terminal 20A and user terminal 20B, and the call is made.

[0027] <Hardware Configuration> Next, the hardware configuration of each device according to the information processing system 1 of the present disclosure will be described. FIG. 3 is a diagram showing an example of the hardware configuration of the server 10 according to the first embodiment. As shown in FIG. 3, the server 10 includes a control unit 102, a communication interface 104, and a storage unit 106, and each unit is connected via a bus line 112.

[0028] The control unit 102 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. Further, the control unit 102 may be configured to realize a function of performing call control such as outgoing call, incoming call, response, and disconnection of a call in addition to the function as a general web server by executing an application or the like stored in the storage unit 106. Further, the control unit 102 may be enabled to execute a voice recognition function on the acquired sound data and an object recognition function on the acquired image data.

[0029] The communication interface 104 controls communication with the user terminal 20 via the network N.

[0030] The storage unit 106 stores an application and data (not shown) for realizing a server function for performing call control, including, for example, a large-capacity HDD (Hard Disk Drive) or SSD (Solid State Drive). Further, the storage unit 106 stores a control program 108. Further, the storage unit 106 has an information storage unit 110.

[0031] The control program 108 is a program for determining the state of the user based on the image data / sound data, performing call control, or executing an application. Further, the control program 108 is a program for transmitting a request from the calling side to the called side, receiving a response to the request from the called side, and returning the response to the calling side.

[0032] The information storage unit 110 stores information such as that of each user terminal that utilizes the function of determining the user's state. For example, the information storage unit 110 may function as a location server that stores a URI (Uniform Resource Identifier) corresponding to the IP address of the user terminal 20. Also, the information storage unit 110 stores the image data / audio data included in the media data transmitted from the user terminal 20 in association with the identification information of the user terminal 20 and the like.

[0033] Next, the hardware configuration of the user terminal 20 will be described. FIG. 4 is a diagram showing an example of the hardware configuration of the user terminal 20 according to the first embodiment. As shown in FIG. 4, the user terminal 20 includes a control unit 202, a communication interface 206, a storage unit 208, a display unit 214, an input unit 216, a microphone 220, a speaker 222, and an imaging device 224, and each unit is connected via a bus line 218.

[0034] The control unit 202 includes a CPU, a ROM, a RAM 204, etc. By executing applications and the like stored in the storage unit 208, the control unit 202 is configured to realize, in addition to the functions of a general information processing device, for example, an IP call function. Also, the control unit 202 may be capable of executing a speech recognition function on the input audio data and an object recognition function on the input image data.

[0035] Also, the RAM 204 temporarily holds various information and is used as a work area when the CPU executes various processes.

[0036] The communication interface 206 controls communication with the server 10 via the network N.

[0037] The storage unit 208 includes, for example, an HDD or an SSD, etc., and stores applications and data (not shown) for realizing the functions of a general information processing device, in addition to storing the application program 210. Also, the storage unit 208 has an information storage unit 212.

[0038] The application program 210 is a program for executing the transmission function of media data related to the above-described image data / audio data and the user state display function. Further, the application program 210 is a program for transmitting a request to the called side via the server 10, receiving a response to this request from the called side via the server 10, and starting a call.

[0039] The information storage unit 212 stores information (call response information) associating the identification information of a callable device and the identification information of the user. The call response information may further hold the user state information transmitted from the server 10 in association with the identification information of the transmitting device.

[0040] The display unit 214 is a display such as a touch panel or a liquid crystal monitor, and displays information to the user. For example, the display unit 214 displays an execution screen of an application, and specifically, displays a screen indicating the state of the call destination, a screen during an IP call, and the like.

[0041] The input unit 216 receives an input from the user or receives an instruction from the user. Note that the display unit 214 and the input unit 216 may be configured as a touch panel.

[0042] The microphone 220 is a device that collects sounds such as voices, and may have a noise canceling function or the like. The speaker 222 is a device that converts audio data into physical vibrations and outputs sounds such as music and voices. The imaging device 224 is, for example, a camera, and images the situation around the information processing device 20.

[0043] <Functional Configuration> Next, with reference to FIG. 5, the functions of each device of the information processing system 1 according to the first embodiment will be described. In the example shown in FIG. 5, the user terminal 20A is on the side that transmits media data and on the side that receives a call request. The user terminal 20B is on the side that displays the status information of the user who uses the user terminal 20A and on the side that transmits a call request. Note that since each user terminal 20 can be either the calling side or the called side, it may have each function.

[0044] As described above, the server 10 has a function of controlling calls using the Internet. For example, when the server 10 uses communication in SIP, call control is performed by an exchange of a request (SIP method) based on HTTP (Hyper Text Transfer Protocol) and a response (response code).

[0045] In the call function of the present disclosure, before the above-described exchange of the request and the response, transmission of image data / audio data and screen display regarding the status information of the user are performed. In the user terminal 20A shown in FIG. 5, the function on the transmission side of mainly image data / audio data will be described, and in the user terminal 20B, the function on the display side of mainly the status information of the user will be described.

[0046] Also, in order to describe an example in which the user terminal 20A and the user terminal 20B have the same function, when not distinguishing each function, the symbols A or B are omitted. For example, it is possible to express that the user terminal 20 has an image acquisition unit 302. Note that the user terminal 20 may have either a function on the transmission side of media data or a function on the display side of the status information of the user.

[0047] The user terminal 20A includes an image acquisition unit 302A, an audio acquisition unit 304A, a transmission unit 306A, a reception unit 312A, a data acquisition unit 314A, a generation unit 316A, an output control unit 318A, and a call processing unit 320A. The functions of each unit in the user terminal 20A are realized by the control unit 202 shown in FIG. 4 executing the application program 210.

[0048] In addition, when the user terminal 20A is executing an application related to the call function of the present disclosure, the camera function and the microphone function are set to on, and image data and sound data around the user terminal 20A are input.

[0049] (Data transmission function of user terminal 20A) The image acquisition unit 302A acquires the image data captured by the imaging device 224. For example, the image data is image data including the first user or image data including the state of the room. The acquired image data is stored in a buffer, for example, and deleted after a predetermined time has elapsed.

[0050] The sound acquisition unit 304A acquires the sound data collected by the microphone 220. For example, the sound data is sound data including the voice of the first user or sound data including ambient sounds. The acquired sound data is stored in a buffer, for example, and deleted after a predetermined time has elapsed.

[0051] The transmission unit 306A transmits media data including the image data acquired by the image acquisition unit 302A and / or the sound data acquired by the sound acquisition unit 304A to the server 10. The transmission destination is set in the server 10 in advance. Also, when transmitting the data to the server 10, the transmission unit 306A may transmit the identification information of the user terminal 20A or the identification information of the first user together.

[0052] Through the above processing, the first user can transmit image data and sound data to the server 10 including them in the media data without performing a special operation. For example, even for an elderly first user who is not familiar with a computer or the like, since no special operation is required, the transmission process is easily executed.

[0053] (Session establishment function of user terminal 20A) As will be described later, when a call request is sent from the user terminal 20B, the receiving unit 312A receives the call request via the server 10. When the user terminal 20A receives the call request, it notifies the first user that there is a call request from the second user of the user terminal 20B. Here, when the first user executes an operation to respond to the call request, the user terminal 20A transmits a call response to the user terminal 20B via the server 10.

[0054] When the call response is transmitted by the receiving unit 312A and a session is established between the user terminal 20A and the user terminal 20B via the server 10, the call processing unit 320A transmits and receives call packets obtained by packetizing call data.

[0055] When the receiving unit 312A directly or indirectly receives a call termination request from the user terminal 20B, the call processing unit 320A directly or indirectly transmits a response to the termination request to the user terminal 20B. As a result, the session ends and the call ends. Note that the call termination request may be transmitted from the user terminal 20A.

[0056] (Call control function of the server 10) The server 10 includes a receiving unit 402, a call control unit 404, an acquisition unit 406, a determination unit 408, and a transmission unit 410. The functions of each unit in the server 10 are realized by the control unit 102 shown in FIG. 3 executing a control program 108.

[0057] The receiving unit 402 of the server 10 receives image data / audio data transmitted from the transmission unit 306A of the user terminal 20A. For example, the image data / audio data included in the received media data is stored in, for example, the RAM of the control unit 102.

[0058] The acquisition unit 406 acquires image data / audio data stored in the RAM or the like. Note that the acquisition unit 406 may acquire image data / audio data transmitted by the user terminal 20A from the receiving unit 402.

[0059] The determination unit 408 determines the state of the first user of the user terminal 20A based on the acquired image data / audio data. For example, the determination unit 408 analyzes the image data / audio data and determines the state of the first user, such as whether the first user is near the user terminal 20A. The state of the user may be, for example, one state selected from a plurality of states according to the analysis result of the image data / audio data.

[0060] Also, the determination unit 408 determines the state each time the image data / audio data is acquired, but may transmit the state information of the updated state from the transmission unit 410 only when the state changes. Thereby, the load of the transmission process of the server 10 can be reduced.

[0061] Also, the determination unit 408 may use the learned model obtained by learning the state of the first user with the image data / audio data as learning data to determine the state of the first user.

[0062] The transmission unit 410 transmits information (state information) regarding the state of the first user to the user terminal 20B that is associated with the identification information of the user terminal 20A and can communicate with the user terminal 20A. For example, the transmission unit 410 transmits state information including whether the first user is in a state where a call is possible to the user terminal 20B. Also, the transmission unit 410 may transmit the state information to the user terminal 20B only when the state is updated.

[0063] By the above processing, the state of the first user is determined using image data or audio data, and the state information of the first user as the call destination is notified to the user who is the call source, so that the call source can easily grasp the state of the call destination.

[0064] Further, the determination unit 408 may include performing object recognition on the acquired image data and determining the user's state based on the result of the object recognition. For example, the determination unit 408 may detect whether a person is included in the image data by performing object recognition on the image data. The object recognition may be performed using a learning model that has been machine-learned for each object in advance. Also, the object recognition may be performed using a learning model obtained by performing supervised learning with the image of the first user as learning data.

[0065] In this case, the determination unit 408 determines whether a user (for example, the first user) is present around the user terminal 20A based on the object recognition result. The determination unit 408 determines that a call is possible when the user is present, and determines that a call is not possible when the user is not present. Also, if the determination unit 408 detects from the result of the object recognition that the user is sitting on a sofa or the like and has not moved for a predetermined time, the determination unit 408 may determine that the user is in a relaxed state, and if the determination unit 408 detects that the user is moving, the determination unit 408 may determine that the user is in a busy state. The determination unit 408 may specify the state from the recognition result by associating each recognition result with each state.

[0066] Through the above processing, the determination unit 408 can not only determine whether the first user is simply callable, but also determine a more detailed state. For example, the determination unit 408 can determine states such as whether the first user is absent or moving.

[0067] Further, the determination unit 408 may include performing facial expression recognition of the first user included in the image data and determining the state of the first user based on the result of the facial expression recognition. For example, when the determination unit 408 detects that the first user is included in the image data, the determination unit 408 recognizes the facial expression of the first user. Known techniques may be used for the facial expression recognition. The determination unit 408 uses AI (artificial intelligence) to estimate the joy, surprise, anger, sadness, straight face, etc. of the first user.

[0068] Through the above processing, the determination unit 408 can not only determine whether the first user is simply callable, but also estimate a more detailed emotional state. For example, the determination unit 408 can estimate what kind of emotional state the first user is in based on the expression of the first user.

[0069] Further, the determination unit 408 may include performing frequency analysis on the sound data and determining the state of the first user based on the result of this frequency analysis. For example, the determination unit 408 determines states such as whether the voice of the first user is included, whether the user is on a train, whether the user is outside, and whether the user is listening to music based on the result of the frequency analysis.

[0070] Through the above processing, the determination unit 408 can not only determine whether the first user is simply callable, but also estimate a more detailed state related to sound. For example, the determination unit 408 can estimate states such as whether the surroundings of the first user are lively or quiet.

[0071] The call control unit 404 identifies the user terminal 20A based on the call request from the user terminal 20B received by the reception unit 402. For example, the call control unit 404 identifies call destination information (for example, URI) from the call request and acquires the IP address of the user terminal 20A using the identified call destination information. The call control unit 404 may inquire a location server (not shown) to identify the IP address from the call destination information.

[0072] The transmission unit 410 transmits a call request to the user terminal 20A identified by the call control unit 404. For example, the call control unit 404 transmits a call request to the user terminal 20A having the identified IP address.

[0073] Further, the reception unit 402 receives a call response from the user terminal 20A for the transmitted call request. After that, the call control unit 404 establishes a call session and controls to send and receive call data to and from each other.

[0074] (Status notification function of user terminal 20B) The user terminal 20B includes an image acquisition unit 302B, a sound acquisition unit 304B, a transmission unit 306B, a reception unit 312B, a data acquisition unit 314B, a generation unit 316B, an output control unit 318B, and a call processing unit 320B. The functions of each unit in the user terminal 20B are realized by the control unit 202 shown in FIG. 4 executing the application program 210. Also, the functions of the user terminal 20B are the same as those of the user terminal 20A. Hereinafter, the status notification function will be mainly described.

[0075] The reception unit 312B receives the status information of the first user of the user terminal 20A from the server 10. For example, the received status information of the first user is stored in the RAM of the control unit 202 or the like. At this time, the reception unit 312B may receive the identification information of the user terminal 20A that transmitted the status information of the first user or the identification information of the first user together and store it in the RAM.

[0076] The data acquisition unit 314B acquires, for example, the status information of the first user stored in the RAM. Also, the data acquisition unit 314B may acquire the status information of the first user from the reception unit 312B.

[0077] The generation unit 316B generates status notification information capable of notifying the status of the first user based on the status information of the first user. For example, the generation unit 316B generates UI components including character data, image data, etc. representing the status of the first user indicated by the status information.

[0078] The output control unit 318B controls the output of the status notification information by the generation unit 316B. For example, if the status notification information is status display information, the output control unit 318B controls the display on the display screen of the display unit 214, and if the status notification information is status sound information, the output control unit 318B controls the output of sound from the speaker 222. For example, the output control unit 318B controls the status notification information to be displayed as a pop-up.

[0079] Through the above processing, the second user can grasp the current situation of the first user by looking at the screen of the user terminal 20B. As a result, if the first user is in a callable state, the second user can simply call the first user.

[0080] Also, the output control unit 318B may control the display of the status notification information within a predetermined area of the display screen. For example, the output control unit 318B controls to display the status notification information in a window set in advance within the display screen. Also, the output control unit 318B may control to display the status notification information in a predetermined area set within the execution screen of the application that realizes the disclosed call function. Also, the predetermined area may be an area where a widget capable of executing the above-described application is displayed. For example, the status notification information is displayed on this widget, and this status notification information is appropriately updated on the widget.

[0081] Through the above processing, the second user can grasp the current situation of the first user by looking at the predetermined area within the screen.

[0082] Also, when the status notification information is updated, the output control unit 318B may control the display of the updated status notification information within the predetermined area. For example, when the status notification information is updated to indicate the second state from the first state, the output control unit 318B controls to display the status notification information indicating the second state within the predetermined area.

[0083] Through the above processing, instead of updating the display screen every time the status notification information is generated, the display screen is updated every time the status notification information is updated, so that the update frequency of the screen due to the status notification information can be suppressed, and the processing load on the user terminal 20B can be reduced.

[0084] Further, when an icon of an application that executes a call via the server 10 exists on the display screen, the generation unit 316B may include generating an image of the icon as status notification information. For example, using a function that dynamically changes the display of the icon like a calendar icon, the generation unit 316B sets an image or character of the icon corresponding to each state indicated by the status notification information in advance, and specifies an image or character of the icon according to the status information acquired from the server 10. In this case, the output control unit 318B controls to display the image or character specified by the generation unit 316B on the icon.

[0085] Through the above processing, the second user can grasp the state of the first user by looking at the icon of the application that executes the function of the present disclosure. For example, by looking at the icon displayed on the home screen, it becomes possible to determine whether the first user is in a state where a call can be made.

[0086] Further, the generation unit 316B may generate status notification information including a moving image or an animation indicating the state of the first user. For example, the generation unit 316B may use a moving image or an animation representing the state of the first user indicated by the status information as the status notification information. As a specific example, the moving image is a past moving image of the first user taken in a predetermined room. In this case, when the status information indicates that the first user is in a relaxed state (sitting on a sofa, etc.), the generation unit 316B selects a past moving image of the first user sitting on the sofa in the room, and when the status information indicates that the first user is absent, the generation unit 316B selects a past moving image of the first user not being in the room.

[0087] Further, when the generation unit 316B acquires image data captured by the user terminal 20A as status information, an effect (such as mosaic, different background, etc.) may be applied to images other than the first user to protect privacy.

[0088] In addition, the generation unit 316B may express the state of the first user as an animation with the movement of a predetermined object. For example, when the first user is in a moving or exercising state, the generation unit 316B may move the object quickly, or when the first user is absent, the generation unit 316B may make the object invisible, or when the first user is in a relaxed state, the generation unit 316B may move the object slowly.

[0089] Through the above processing, when displaying the state of the first user, privacy can be protected, and the first user can safely permit the execution of the functions of the present disclosure.

[0090] In addition, when the information regarding the state of the first user indicates that a call is possible, the generation unit 316B may include identifying a UI component for starting a call process for the user terminal 20A. In this case, the output control unit 318B may include controlling to display the UI component on the display screen together with the output of the state notification information.

[0091] For example, when the state information indicates that a call is possible, the generation unit 316B identifies or selects a button for calling the first user. Note that the state information may include a plurality of states, and it is only necessary that whether a call is possible for each state is preset in advance. In addition, the output control unit 318B may control to display the UI component in association with the state notification information. For example, control is performed to display the UI component at a position adjacent to the display information.

[0092] Through the above processing, if the first user is in a callable state, the second user can, by operating the UI component, confirm the state of the first user and then smoothly make a call.

[0093] When the UI component for calling the first user is operated by the second user, the transmission unit 306B transmits a call request to the user terminal 20A via the server 10. If an OK call response is returned from the user terminal 20A in response to this call request, the call is started.

[0094] By executing the functions of each device as described above, the user who is the call originator can grasp the current state of the call recipient before starting a call. Also, if the current state includes the emotions and movements of the call recipient, etc., it becomes possible to make a call after grasping that state, so the start of the call can proceed smoothly.

[0095] <Example screen> Next, with reference to FIG. 6, an example screen including the state of the call recipient according to the first embodiment will be described. FIG. 6A is a diagram showing Example Screen 1 including the state of the call recipient according to the first embodiment. In the example shown in FIG. 6A, the status notification information is displayed on the screen as Pop-up A1. Also, since the status information indicates that the user (User A) is sitting on the sofa, it is determined that a call is possible. Along with the determination that a call is possible, the UI component A2 of the "Call" button is displayed within the screen.

[0096] FIG. 6B is a diagram showing Example Screen 2 including the state of the call recipient according to the first embodiment. In the example shown in FIG. 6B, the status notification information is displayed on the icon B1 of the application that executes the call function of the present disclosure. In this case, after the user clicks on this icon B1 and transitions to the execution screen of this application, a call button may be displayed.

[0097] FIG. 6C is a diagram showing Example Screen 3 including the state of the call recipient according to the first embodiment. In the example shown in FIG. 6C, the status notification information is displayed in a predetermined area C1 of the execution screen of the application that executes the call function of the present disclosure. Also, since the status information indicates that the user is sitting on the sofa, it is determined that a call is possible. Along with the determination that a call is possible, the UI component A2 of the "Call" button is displayed within the execution screen.

[0098] FIG. 6D is a diagram showing an example of a screen 4 including the state of the user of the call destination according to the first embodiment. In the example shown in FIG. 6D, there are a plurality of predetermined areas on the execution screen of the application that executes the call function of the present disclosure, and the current states of different users are displayed in the respective predetermined areas C1 and D1. For example, Mr. B is in an absent state, and a black screen or the like that gives an image of a dark room is displayed in the predetermined area D1 where the status notification information is displayed.

[0099] In the example shown in FIG. 6D, the server 10 sets a plurality of users (Mr. A and Mr. B in the example shown in FIG. 6D) in the call correspondence information as call destination candidates for a predetermined user. The server 10 acquires image data / audio data from each of the user terminals 20 of Mr. A and Mr. B and determines their respective states. When the user terminal 20 receives the state information of each user from the server 10, it can display the status notification information of a plurality of users on the screen as shown in FIG. 6D.

[0100] Thereby, the user who is the call source can easily grasp the current states of the plurality of users who are call destination candidates.

[0101] <Operation> Next, each operation of the information processing system 1 will be described. FIG. 7 is a flowchart showing an example of the operation processing of the server 10 according to the first embodiment. Hereinafter, the operation for the example in which the user terminal 20A of the first user who is the call destination transmits image data or the like and the user terminal 20B of the second user who is the call source displays the state of the first user will be described.

[0102] In step S102, the receiving unit 402 of the server 10 receives the image data / audio data included in the media data transmitted from the user terminal 20A of the first user. Next, the acquisition unit 406 of the server 10 acquires the received image data / audio data.

[0103] In step S104, the determination unit 408 of the server 10 determines the current state of the first user based on the acquired image data / audio data.

[0104] In step S106, the determination unit 408 of the server 10 determines whether the determined state has changed from the current state. If the state has changed (step S106 - YES), the process proceeds to step S108. If the state has not changed (step S106 - NO), the process returns to step S102.

[0105] In step S106, the determination unit 408 of the server 10 specifies, as the notification destination, the user terminal 20B that is associated with the identification information of the user terminal 20A that has transmitted the media data including the image data / audio data or the identification information of the first user and is capable of making a call with the user terminal 20A. The determination unit 408 can specify the notification destination, for example, by referring to the call correspondence information described above.

[0106] In step S108, the transmission unit 410 of the server 10 transmits the information (status information) regarding the determined state of the first user to the user terminal 20B specified as the notification destination.

[0107] FIG. 8 is a flowchart showing an example of the operation process on the display side of the state according to the first embodiment. In the example shown in FIG. 8, the process performed by the user terminal 20B will be mainly described. In the example shown in FIG. 8, the status notification information is status display information, and the output control unit 318B is an example of controlling the display of the status display information on the display screen.

[0108] In step S202, the reception unit 312B of the user terminal 20B receives the status information of the first user from the server 10. Further, the data acquisition unit 314B of the user terminal 20B acquires the received status information.

[0109] In step S204, the generation unit 316B of the user terminal 20B generates status display information based on the acquired status information of the first user. For example, the generation unit 316B generates UI components including images, characters, etc. indicating the status of the first user.

[0110] In step S206, the output control unit 318B of the user terminal 20B controls the display of the generated status display information on the display screen.

[0111] In step S208, the user terminal 20B determines whether an operation on the UI component to start a call has been performed by the second user. If there is an operation to start a call (step S208 - YES), the transmission unit 306B transmits a call request to the user terminal 20A via the server 10, and the process proceeds to step S210. If there is no operation to start a call (step S208 - NO), the process returns to step S202.

[0112] In step S210, the reception unit 312B of the user terminal 20B determines whether it has received a call response from the user terminal 20A via the server. If it has received a call response (step S210 - YES), the process proceeds to step S212. If it has not received a call response (step S210 - NO), it is considered that the first user has not responded, and the call process ends, and the process returns to step S202.

[0113] In step S212, the user terminal 20B starts a call session and starts a call by transmitting and receiving call data with the user terminal 20A.

[0114] As described above, according to the information processing system in the first embodiment, before starting a call, the user who is the call origin can grasp the current status of the user who is the call destination.

[0115] Also, the information processing described in the first embodiment above may be realized as a program for causing a computer to execute. By installing this program from a server or the like and causing a computer to execute it, the above-described information processing can be realized.

[0116] In addition, it is also possible to record this program on a recording medium, have a computer read the recording medium on which this program is recorded, and realize the above-described information processing. The recording medium can store the program in a "non-transitory tangible medium".

[0117] Note that as the recording medium, various types of recording media can be used, such as recording media that optically, electrically, or magnetically record information, like CD-ROMs, flexible disks, magneto-optical disks, etc., and semiconductor memories that electrically record information, like ROMs, flash memories, etc.

[0118] [Second Embodiment] FIG. 9 is a diagram for explaining the outline of the system 2 according to the second embodiment of the present disclosure. The system 2 shown in FIG. 2 is a system in which the system described in the first embodiment is applied to a system for managing images, and includes a management device 10A, a server 10B, a user terminal 20, an image output device 30, and a camera 60. Further, the image output device 30 is connected to a display device 40, and the display device 40 is controlled by a remote control device 50. The display device 40 has a camera 60 attached thereto, and the image acquired by the camera 60 is input to the image output device 30. For example, the camera 60 is a webcam and may incorporate a microphone. The server 10B has the functions of the server 10 described in the first embodiment.

[0119] The management device 10A or the server 10B and the user terminal 20 can communicate with each other via a network N1 such as a wireless LAN, a fourth or fifth generation mobile communication system (4G or 5G), or LTE (Long Term Evolution) as an example of a communication technology. Also, the management device 10A and the image output device 30 can communicate with each other via a wireless network N2, which has a lower communication fee but is slower than the network N1, such as a third generation mobile communication system (3G). For the sake of explanation, the network N1 and the wireless network N2 are described separately, but these networks can be connected to each other via the Internet. Also, the network configuration is not limited to the above example.

[0120] First, the management device 10A obtains information such as the name and address of the image viewer or poster from the image poster through an Internet website. Note that information such as the name and address of the viewer or poster is not essential information. At this time, the management device 10A generates an identifier (referred to as a device ID) of the image output device 30 used by the viewer. Then, the management device 10A notifies the poster of the device ID by, for example, email.

[0121] The management device 10A sets the generated device ID in the image output device 30. Thereafter, the administrator of the management device 10A causes the image output device 30 to be shipped to the address of the image viewer. The image output device 30 incorporates, for example, a 3G communication module and is configured to immediately start communicating with the management device 10A using the set device ID when powered on.

[0122] On the other hand, the image poster downloads an application for sharing images that operates on a user terminal 20 such as a smartphone or a tablet. This application accesses the management device 10A using the notified device ID. The management device 10A can associate the user terminal 20 (poster) with the image output device 30 (viewer) using the device ID notified from the application. This information corresponds to the call correspondence information described above.

[0123] Thereafter, the poster can take pictures of various subjects using the application. The application automatically transmits the image data obtained by the photographing to the management device 10A via the network N1. The poster is not required to perform a special operation to transmit the image data to the management device 10A.

[0124] The management device 10A accumulates the image data transmitted from the application of the user terminal 20 and sequentially distributes it to the image output device 30. The image output device 30 displays the image data on the display device 40 in response to an instruction from the viewer. Further, the image output device 30 may have a microphone and a speaker and is capable of making an IP call.

[0125] Here, the display device 40 is, for example, a television that is popular in ordinary households, and the remote control device 50 is a remote controller. The image output device 30 is connected to the display device 40 by, for example, HDMI (High-Definition Multimedia Interface) (registered trademark), and can acquire the control signal issued from the remote control device 50 via HDMI.

[0126] The image output device 30 can acquire the control signal issued from the remote control device 50 and grasp the content of the operation input by the viewer. That is, the viewer can view the distributed image data using the television remote control with which they are familiar in daily life.

[0127] The camera 60 is, for example, a web camera and may incorporate a microphone. The image (still image or moving image) data / audio data captured by the camera 60 is input to the image output device 30. The image output device 30 transmits the media data related to the input image data / audio data to the server 10.

[0128] A function capable of starting the call described in the first embodiment is implemented in the application for performing the above-described image sharing. In this case, the user terminal 20A shown in FIG. 1 etc. corresponds to the image output device 30. For example, the image output device 30 includes a transmission unit 306, a reception unit 312, a data acquisition unit 314, a generation unit 316, an output control unit 318, and a call processing unit 320 of the user terminal 20.

[0129] Specifically, the media data including the image data / audio data acquired by the camera 60 is transmitted to the server 10B via the image output device 30. The server 10B determines the state of the viewer of the image based on the image data / audio data included in the acquired media data. For example, the server 10B analyzes the image data / audio data to determine the state such as the viewer is moving in the room where the display device 40 is located, sitting on a chair, not in the room, bustling, quiet, etc.

[0130] The server 10B refers to the information held in the management device 10A, identifies the user terminal 20 associated with the image output device 30, and transmits the viewer's status information to this user terminal 20. As described above, the user terminal 20 generates status information (for example, status display information) based on the status information and controls to display it on the display screen (for example, FIG. 6).

[0131] When the user terminal 20 detects that the poster has operated the call button, it transmits a call request to the image output device 30 via the server 10B. When the image output device 30 receives the call request, it displays whether to respond to the display device 40. For example, the first button of the remote control device 50 is assigned to response, and the second button is assigned to rejection.

[0132] When the viewer operates the first button, the image output device 30 detects the operation of the first button and transmits a call response to the user terminal 20 via the server 10B. After that, the call is started.

[0133] As described above, according to the second embodiment, the call function disclosed in the first embodiment can be provided to the system for sharing images between the viewer and the poster. Also, the management device 10A and the server 10B may be the same server.

[0134] [Third Embodiment] Next, a third embodiment of the present disclosure will be described. In the third embodiment, the functions of the determination unit 408 described in the first embodiment are provided in the user terminal 20. For example, the determination unit of the user terminal 20 determines the state of the user using the own terminal based on the image data obtained by the image acquisition unit 302 and / or the sound data obtained by the sound acquisition unit 304.

[0135] The method for determining the user's state is as described above. For example, the determination unit of the user terminal 20 performs object recognition on the image data and / or performs frequency analysis on the sound data, and determines the user's state using the respective results.

[0136] The transmission unit 306 of the user terminal 20 transmits information (status information) regarding the state of the user determined by the determination unit to another user terminal before the call. For example, the transmission unit 306 may first transmit the status information to the server 10 in order to transmit the status information to another user terminal. When the server 10 acquires the status information, it identifies the notification destination (another user terminal) associated with the user terminal 20 that transmitted the status information, and transmits the status information to the identified notification destination. When the notified user terminal 20 receives the status information, it executes the same processing as in the first embodiment.

[0137] As described above, according to the third embodiment, the load of the server 10 can be distributed. In addition, since the user terminal 20 processes the image data and sound data on its own terminal and does not transmit these image data and sound data externally, it is also possible to protect privacy.

[0138] [Fourth Embodiment] Next, a fourth embodiment of the present disclosure will be described. In the fourth embodiment, some of the functions of the determination unit 408 described in the first embodiment are provided in the user terminal 20. For example, the preprocessing unit of the user terminal 20 performs object recognition on the image data obtained by the image acquisition unit 302 and / or performs frequency analysis on the sound data obtained by the sound acquisition unit 304, and detects objects and extracts frequency feature amounts.

[0139] The transmission unit 306 of the user terminal 20 transmits media data including the result of object recognition and / or the processing result information including the feature amount (result of frequency analysis) processed by the preprocessing unit to the server 10. When the server 10 acquires the media data, the determination unit 408 determines the user's state based on the processing result information included in the acquired media data. For example, the determination unit 408 can determine the user's state from the result information by holding a table or the like that associates the user's state with each acquirable result information.

[0140] When the server 10 determines the user's state, it specifies the notification destination associated with the user terminal 20 that transmitted the media data, and transmits information (status information) regarding the user's state to the specified notification destination. When the user terminal 20 of the notification destination receives the status information, it executes the same processing as in the first embodiment.

[0141] As described above, according to the fourth embodiment, it is possible to reduce the communication volume compared to the first embodiment, and the calculation resources of the determination unit that imposes a processing load can be shared.

[0142] As described above, each of the above-described embodiments is an exemplification for explaining the present invention, and the present invention is not intended to be limited only to those embodiments. The present invention can be variously modified without departing from the gist thereof.

[0143] [Modification Example] For example, as Modification Example 1, in addition to output control of the status information notification information generated by the generation unit 316, the output control unit 318 of the user terminal 20 may control the operation of the device according to the status information. As a specific example, the output control unit 318 may also function as an operation control unit that controls devices connected via a network, such as the brightness of the room lighting, the volume of the device (own terminal or music player) during music playback, the pause or volume of the device (own terminal or TV, etc.) during TV or video viewing, according to the status information.

[0144] For example, if the state of the call destination is callable, the output control unit 318 may brighten the room lighting, lower the music volume, or pause the currently playing video. This can create an environment in which the user at the call source can easily make a call. Further, the output control unit 318 may hold control commands associated with each state of the user and execute the control commands specified by the state information.

[0145] Also, as a second modification, in each of the above-described embodiments, the generation unit 316 is provided in the user terminal 20, but may be provided in the server 10. For example, the generation unit of the server 10 generates state information according to the state of the user determined by the determination unit 408.

[0146] In this case, the transmission unit 410 of the server 10 transmits the state information as information (state information) regarding the state of the user to the user terminal 20. That is, in the case of the second modification, the state information is included in the information regarding the state of the user transmitted from the server 10. When the output control unit 318 of the user terminal 20 acquires the state information, it controls to display this state information on the screen. This makes it possible to easily add or modify state information and the like on the server 10 side.

[0147] Also, as a third modification, the determination unit 408 may use information obtained from a predetermined sensor (for example, a photosensor such as an illuminance sensor or a human sensor, an ultrasonic sensor, etc.) instead of or in addition to the media data to determine the state of the user. For example, the determination unit 408 can determine that the user of the call destination is present when the detection signal of the predetermined sensor indicates ON. As a result, although the cost of introducing a predetermined sensor may be incurred, it is possible to determine the state of the user of the call destination before the call in the same manner as in the above-described embodiments. Further, by using the media data and the signal obtained by the predetermined sensor, it becomes possible to more appropriately determine the state of the user of the call destination before the call.

Explanation of Reference Numerals

[0148] 1... Information processing system, 2... System, 10... Server, 10A... Management device, 10B... Server, 20... User terminal, 30... Image output device, 40... Display device, 60... Camera, 102... Control unit, 106... Storage unit, 202... Control unit, 204... RAM, 208... Storage unit, 220... Microphone, 222... Speaker, 224... Imaging device, 302... Image acquisition unit, 304... Sound acquisition unit, 306... Transmission unit, 312... Reception unit, 314... Data acquisition unit, 316... Generation unit, 318... Output control unit, 320... Call processing unit, 402... Reception unit, 404... Call control unit, 406... Acquisition unit, 408... Determination unit, 410... Transmission unit

Claims

The information processing method executed by a communication device used by a first user, which comprises: constantly or periodically acquiring image data and / or audio data; each time the image data and / or audio data is acquired, determining the state of the first user using the communication device based on the image data and / or audio data; associating state information regarding the determined state of the first user with making a call to the first user by call response information pre-held in a server, and outputting the state information to the server in association with the identification information of the communication device or the identification information of the first user for the server to transmit to another communication device used by a second user who can make a call; wherein the server is a server that pre-holds call response information associating the identification information of the first user or the identification information of the communication device with the identification information of the second user or the identification information of the other communication device.

2. The determining comprises: performing object recognition on the image data, and determining the state of the first user based at least on the result of the object recognition. The information processing method according to claim 1.

3. The determining comprises: performing frequency analysis on the audio data, and determining the state of the first user based at least on the result of the frequency analysis. The information processing method according to claim 1 or 2.

4. The state information regarding the state of the first user is output to the other communication device regardless of whether the other communication device has sent a call request to the server. The information processing method according to any one of claims 1 to 3.

5. A program for causing a communication device used by a first user to: constantly or periodically acquire image data and / or audio data; each time the image data and / or audio data is acquired, determine the state of the first user using the communication device based on the image data and / or audio data; associate state information regarding the state of the first user with making a call to the first user by call response information pre-held in a server, and output the state information to the server in association with the identification information of the communication device or the identification information of the first user for the server to transmit to another communication device used by a second user who can make a call; A program, wherein the server is a server that pre-holds the call correspondence information associating the identification information of the first user or the identification information of the communication device with the identification information of the second user or the identification information of the other communication device.

6. An acquisition unit that acquires image data and / or audio data constantly or periodically, A determination unit that determines the state of a first user who uses the communication device based on the image data and / or audio data each time the image data and / or audio data is acquired, A transmission unit that outputs the state information related to the determined state of the first user, associated with the identification information of the communication device or the identification information of the first user, to the server for transmission to another communication device used by a second user who can make a call and is associated with making a call with the first user according to the call correspondence information pre-held by the server. A communication device, wherein the server is a server that pre-holds the call correspondence information associating the identification information of the first user or the identification information of the communication device with the identification information of the second user or the identification information of the other communication device.

7. The communication device used by the second user is To acquire state information via the server without the second user operating, wherein the state information is state information related to the state of a first user who uses the other communication device, determined based on media data related to image data and / or audio data acquired by the other communication device, and the server pre-holds the call correspondence information associating the identification information of the communication device or the identification information of the second user with the identification information of the other communication device or the identification information of the first user, the acquiring To generate state notification information capable of notifying the state of the first user based on the state information related to the state of the first user, To perform output control of the state notification information, An information processing method for executing.

8. The performing the output control includes Performing display control of the state notification information within a predetermined area of the display screen, When the state notification information is updated, performing display control of the updated state notification information within the predetermined area, the information processing method according to Claim 7.

9. The generating includes The information processing method according to claim 8, including generating an image of the icon as the status notification information when an icon of an application for executing a call exists on the display screen via the information processing apparatus.

10. The display is a video or an animation The generating includes The information processing method according to any one of claims 7 to 9, including generating status notification information including a moving image or an animation indicating the status of the first user.

11. The generating includes When information regarding the status of the first user indicates that a call is possible, including identifying a UI component for starting call processing for the other communication apparatus, The output controlling includes The information processing method according to any one of claims 7 to 9, including controlling the display of the UI component on the display screen together with the output of the status notification information.

12. For a communication apparatus used by a second user, Obtaining status information via a server without the second user operating, where the status information is status information regarding the status of a first user using the other communication apparatus, determined based on media data regarding image data and / or audio data acquired by the other communication apparatus, and the server pre-holds call correspondence information associating the identification information of the communication apparatus or the identification information of the second user with the identification information of the other communication apparatus or the identification information of the first user, the obtaining Generating status notification information capable of notifying the status of the first user based on the status information regarding the status of the first user, Controlling the output of the status notification information, A program for causing the above to be executed.

13. A communication apparatus used by a second user, An obtaining unit that obtains status information via a server without the second user operating, where the status information is status information regarding the status of a first user using the other communication apparatus, determined based on media data regarding image data and / or audio data acquired by the other communication apparatus, and the server pre-holds call correspondence information associating the identification information of the communication apparatus or the identification information of the second user with the identification information of the other communication apparatus or the identification information of the first user, the obtaining unit A generating unit that generates status notification information capable of notifying the status of the first user based on the status information regarding the status of the first user, An output control unit that performs output control for outputting the status notification information, A communication device including the above.

14. An information processing apparatus, Acquires media data related to image data and / or audio data from a first communication device, and identification information of the first communication device or identification information of a first user who uses the first communication device, Each time the media data is acquired, determines the status of the first user based on the media data, Identifies the second communication device from the acquired identification information of the first communication device or the identification information of the first user based on call correspondence information in which the identification information of the first communication device or the identification information of the first user pre-stored in a storage unit is associated with the identification information of the second communication device, An information processing method for transmitting information regarding the status of the first user to the identified second communication device regardless of the presence or absence of a call request from the second communication device.

15. The determination includes, Determining the status of the first user based at least on the processing result information included in the media data, the processing result information including an object recognition result of the image data and / or a result of frequency analysis of the audio data, as described in claim 14.

16. In an information processing apparatus, Acquires media data related to image data and / or audio data transmitted from a first communication device, and identification information of the first communication device or identification information of a first user who uses the first communication device, Each time the media data is acquired, determines the status of the first user who uses the first communication device based on the media data, Identifies the second communication device from the acquired identification information of the first communication device or the identification information of the first user based on call correspondence information in which the identification information of the first communication device or the identification information of the first user pre-stored in a storage unit is associated with the identification information of the second communication device, A program for causing the identified second communication device to transmit information regarding the status of the first user regardless of the presence or absence of a call request from the second communication device.

17. An acquisition unit that acquires media data related to image data and / or audio data from a first communication device, and identification information of the first communication device or identification information of a first user who uses the first communication device, A determination unit that determines the state of the first user of the first communication device based on the media data each time the media data is acquired; Identifying the second communication device from the acquired identification information of the first communication device or the identification information of the first user based on call correspondence information in which the identification information of the first communication device or the identification information of the first user pre-stored in the storage unit is associated with the identification information of the second communication device; A transmission unit that transmits information regarding the state of the first user to the specified second communication device regardless of the presence or absence of a call request from the second communication device An information processing apparatus comprising.

Images