Program, information processing method, and information processing device

Abstract

To properly understand the state of a call destination before a call.SOLUTION: An information processing device is caused to acquire media data relating to image data and / or sound data transmitted from a first communication device, determine the state of a user of the first communication device based on the media data, and transmit information relating to the user's state to a second communication device that is associated with identification information of the first communication device and is capable of communicating with the first communication device.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a program, an information processing method, and an information processing device. [Background technology]

[0002] Conventionally, there is a technology that notifies the other party whether they are available to talk before starting a voice call. For example, the caller operates the phone and uses the recipient's email address obtained from the notification recipient. Connect to the Internet to obtain information on whether or not a call is possible, or install a presence sensor. There is a known technology that allows you to know whether the other party is available for a call by checking whether the other party is available for a call or not (for example, , see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-159748 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the prior art, it is possible to know whether the other party is available for communication, but You don't know what the other party's current status is. For example, Even if the internet is connected, there is a possibility that the person is not at their desk. In order to determine whether or not a person is in their seat, a seat presence sensor must be installed, which increases costs. Therefore, there is a need for a technology that can properly grasp the status of the other party before making a call.

[0005] Therefore, a program and information processing method that can properly grasp the status of the other party before the call is developed. The present invention aims to provide a method and an information processing device. [Means for solving the problem]

[0006] In one aspect of the present disclosure, a program is transmitted from a first communication device to an information processing device. Acquiring media data relating to image data and / or sound data; determining a state of the user of the first communication device based on the data; A second communication device associated with the identification information and capable of communicating with the first communication device is Sending status information. [Effects of the Invention]

[0007] According to certain aspects of the present disclosure, the status of the other party can be properly grasped before the call. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram for explaining a system overview of the present disclosure. [Figure 2] 1 is a diagram illustrating an example of a schematic configuration of an information processing system 1 according to a first embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server according to the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a user terminal according to the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of the functions of each device in the information processing system according to the first embodiment. [Figure 6A] FIG. 2 is a diagram showing a first example screen including the status of a callee user according to the first embodiment. [Figure 6B] FIG. 10 is a diagram showing a second example screen including the status of a callee user according to the first embodiment. [Figure 6C] FIG. 10 is a diagram showing a third example screen including the status of a callee user according to the first embodiment. [Figure 6D]FIG. 10 is a diagram showing a fourth example screen including the status of a callee user according to the first embodiment. [Figure 7] 6 is a flowchart showing an example of operation processing of the server according to the first embodiment. [Figure 8] 10 is a flowchart showing an example of operation processing on the status display side according to the first embodiment. [Figure 9] FIG. 10 is a diagram illustrating an overview of a system according to a second embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0010] <System Overview> FIG. 1 is a diagram illustrating an overview of a system according to the present disclosure. In the example shown in FIG. A UB (e.g., a child) makes a voice call, video call, or video call with a user UA (e.g., a parent). (hereinafter, collectively referred to as "calls"). The user UB uses an information processing device 20A (first communication device) such as a mobile terminal. Which information processing device 20B (second communication device) is used? It is an information processing device that can control calls made via the IEEE 802.11n Protocol (RNET) telephone and videophone. The overview shown in Figure 1 is based on IP telephone and WebRTC (Web Real-time Communication) technology. This includes examples where calls are made through P2P communication using techniques such as

[0011] (1) Transmission of image data / audio data First, the information processing device 20A turns on the camera and microphone. The information processing device 20A acquires surrounding images and sounds. The media data related to the data (hereinafter also referred to as "image data / sound data") is stored in the server 1. 0. The information processing device 20A always has at least one of the camera and the microphone. It is turned on at certain times or periodically to acquire image data / audio data.

[0012] (2) Sending status information The server 10 receives media data related to image data / audio data from the information processing device 20A. and based on this media data, The server 10 determines the current state of the user UA by, for example, performing image analysis or sound analysis. The current state of the user UA is determined by, for example, the image-based information processing device 20A. The state of whether or not the user UA is nearby, and the emotion based on the user UA's facial expression The information processing device 20A may be configured to detect the state of the location based on the sound information around the information processing device 20A, or the liveliness of the location based on the sound information around the information processing device 20A. The status includes at least one of the following: a status related to the user UA's status, a status indicating whether the user UA is available for communication, or a status indicating whether the user UA is available for communication. .

[0013] The server 10 receives information about the determined state of the user UA (hereinafter also referred to as "state information"). ) to the information processing device 20B associated with the identification information (ID) of the information processing device 20A. The server 10 associates, for example, the identification information of devices that can communicate with each other. Call handling information is maintained.

[0014] (3) After checking the user's status, send a call request When the information processing device 20B used by the user UB receives the status information of the user UA, For example, a screen that notifies the user UB of the status of the user UA is displayed on the display screen. The device 20B displays a video, animation, text, etc., that indicates the current state of the user UA. UI widget M1 includes icons, windows, and popups. or a predetermined area in the application screen. Current or past image data including the user UA may be displayed. To protect the user's privacy, images other than those of the user UA are blurred or displayed with a different background. You may do so.

[0015] When the information processing device 20B receives an operation to make a call from the user UB to the user UA, At this time, the information processing device 20B transmits a call request to the server 10.

[0016] (4) Establishing a call session When the server 10 receives the call request from the information processing device 20B, 0B's identification information. The information processing device 20A receives the call request and receives an operation to answer the call from the user UA. When the call request is received, the server 10 transmits a call response to the call request to the server 10. The server 10 detects that a session has been established between the information processing device 20A and the information processing device 20B. Once the call is established, transmission and reception of data for the call (hereinafter also referred to as "call data") begins. If the server 10 does not receive a call response within a predetermined time, the server 10 notifies the server 10 of this fact. device 20B and discard the call request.

[0017] As a result, the user UB can, before making a call, simply by looking at the display screen of the information processing device 20B. The current status of the user UA at the other end of the call can be properly understood. For example, if the user UA If the UI widget M1 shows a state of being relaxed in a room, the user U B may determine that it is OK to make a call and instruct the information processing device 20B to start the call. can.

[0018] In addition, the user UA can be notified of his / her own status to the user UB without being particularly aware of it. The UB can check the status of the user UA without any special operation and decide the timing of the call. In addition, image data and sound data are used to determine the status. , the appropriate state of the user UA can be determined, and the user UB can determine the current state of the user UA. This allows you to start a call after understanding the call status.

[0019] Note that the same processing may be performed from the user UB side to the user UA side. That is, the status of the user UB is displayed on the information processing device 20A of the user UA, can grasp the current state of the user UB.

[0020] [First embodiment] Next, an example of a system configuration for realizing the above-mentioned system will be described. 2 is a diagram showing an example of a schematic configuration of an information processing system 1 according to a first embodiment. As shown in the figure, the server 10 and the user terminals 20A, 20B, 20C, and 20D used by the users are ...and are connected to each other via a network N so that they can communicate with each other, thereby enabling information processing System 1 is configured. From now on, if there is no need to distinguish between user terminals individually, the symbol 2 is used. 0 is used. Also, regarding the server 10, even if there are multiple servers connected to the network N, good.

[0021] The server 10 is capable of transmitting and receiving IP packetized audio data and image data, for example. It is an information processing device that controls calls such as making (outgoing) calls, receiving (incoming) calls, answering, and disconnecting calls. For call control, it is also a device that uses H.323 and MGCP (Media Gateway Control Protocol). Signaling protocols such as SIP (Session Initiation Protocol) A rule may also be used.

[0022] The user terminal (communication device) 20 is an information processing device that can access a network. , for example, but not limited to, mobile devices such as smartphones, computers (for example, but not limited to, as a desktop, laptop, tablet, etc.), media computer platform Home (for example, but not limited to, set-top boxes, digital video recorders, etc.) handheld computing devices (including, but not limited to, PDAs) Assistant, email clients, etc.), wearable devices (including but not limited to , eyeglasses, watch devices, etc.), or other types of computers.

[0023] The network N is composed of multiple types of communication lines and networks and various network devices. For example, the network N may include a base station wirelessly connected to the server 10, a wireless LAN AN access points (WiFi routers, etc.), mobile communication networks connected to base stations, access Telephone lines, cable television lines, or Public lines such as optical communication lines, the Internet connected to the user terminal 20, and mobile communication networks It may also include a gateway device that connects a public line to the Internet. The configuration of the network N is not limited to the above example.

[0024] In the system configuration shown in FIG. 2, a signal is sent from the user terminal 20A of the first user who is the destination of the call. The current state is determined based on the media data related to the image / sound data being transmitted. is displayed on the user terminal 20B of the second user who is the caller.

[0025] The user terminal 20B receives the status information of the first user acquired via the server 10. The second user generates status notification information that can notify the user of the status information. The first user checks the displayed status notification information, determines whether or not a call is possible, and performs an operation to start a call. At this time, the user terminal 20B sends a call request via the server 10. and transmits it to the user terminal 20A.

[0026] When the user terminal 20A receives the call request, the second user performs an operation to respond. If the call is successful, the server 10 sends a call response to the server 10. This establishes a call session. The call data including the contents of the call such as voice and images is converted into IP packets. The set call data is transmitted and received between the user terminal 20A and the user terminal 20B. The talk takes place.

[0027] <Hardware configuration> Next, the hardware configuration of each device in the information processing system 1 of the present disclosure will be described. FIG. 3 is a diagram showing an example of the hardware configuration of the server 10 according to the first embodiment. As shown in FIG. 3, the server 10 includes a control unit 102, a communication interface 104, and a storage unit 106. 106, and each part is connected via a bus line 112.

[0028] The control unit 102 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), The control unit 102 also stores data in the storage unit 106. By running applications stored in the In addition, it is configured to realize the function of making, receiving, answering, and disconnecting calls. The control unit 102 may also perform a voice recognition function on the acquired sound data, It may be possible to perform object recognition functions on the image data.

[0029] The communication interface 104 controls communication with the user terminal 20 via the network N. do.

[0030] The storage unit 106 is, for example, a large-capacity HDD (Hard Disk Drive) or an SSD (Solid State Drive). Applications and devices for implementing server functions that perform call control, including the TE Drive The storage unit 106 also stores a control program 108. The storage unit 106 also includes an information storage unit 110 .

[0031] The control program 108 determines the state of the user based on the image data / sound data. A control program is a program that controls calls and executes applications. The system 108 transmits a request from the calling side to the called side and responds to the request from the called side. It is a program that receives the response and returns the response to the caller. .

[0032] The information storage unit 110 stores information on each user terminal that uses the function for determining the user's state. For example, the information storage unit 110 stores the U corresponding to the IP address of the user terminal 20. It functions as a location server that stores RI (Uniform Resource Identifier). Furthermore, the information storage unit 110 may store the information included in the media data transmitted from the user terminal 20. The image data / audio data is stored in association with the identification information of the user terminal 20.

[0033] Next, the hardware configuration of the user terminal 20 will be described. 4 is a diagram illustrating an example of a hardware configuration of the user terminal 20. As shown in FIG. The terminal 20 includes a control unit 202, a communication interface 206, a storage unit 208, and a display unit 214, an input unit 216, a microphone 220, a speaker 222, and an imaging device 224. Each unit is connected via a bus line 218 .

[0034] The control unit 202 includes a CPU, a ROM, a RAM 204, etc. By executing applications stored in the device, the device can be used as a general information processing device. In addition to all the functions, the control unit 202 is configured to realize, for example, an IP call function. It has a voice recognition function for input sound data and an object recognition function for input image data. It may also be possible to perform recognition functions.

[0035] The RAM 204 also temporarily stores various types of information and stores the information required for the CPU to execute various processes. It is sometimes used as a work area.

[0036] The communication interface 206 controls communication with the server 10 via the network N. .

[0037] The storage unit 208 includes, for example, an HDD or an SSD, and has the functionality of a general information processing device. In addition to storing applications and data (not shown) to realize the functions, The storage unit 208 stores a preprogram 210. The storage unit 208 also includes an information storage unit 212. do.

[0038] The application program 210 is a program for processing media data relating to the above-mentioned image data / audio data. It is a program for executing the sending function and the user status display function. The program 210 sends a request to the called party via the server 10, and responds to this request. The server 10 receives a response from the called party and starts a process to start the call. It is grams.

[0039] The information storage unit 212 stores information in which identification information of devices that can make calls and identification information of users are associated with each other. The call response information is further stored in the server 10. The user's status information may be stored in association with the identity of the device that sent it.

[0040] The display unit 214 is a display such as a touch panel or a liquid crystal monitor. For example, the display unit 214 displays an application execution screen. Specifically, it displays a screen showing the status of the other party and a screen showing an IP call in progress.

[0041] The input unit 216 receives inputs from the user and receives instructions from the user. The display unit 214 and the input unit 216 may be configured as a touch panel.

[0042] The microphone 220 is a device that collects sounds such as voices and has a noise cancellation function. The speaker 222 converts sound data into physical vibrations to produce sounds such as music and voice. The image capturing device 224 is, for example, a camera, and is a device that outputs an image. Capture images of the surrounding area.

[0043] <Functional configuration> Next, referring to FIG. 5, the functions of each device in the information processing system 1 according to the first embodiment will be described. In the example shown in FIG. 5, the user terminal 20A is the one that transmits the media data. The user terminal 20B is the side that receives the call request. It is the side that displays the status information of the user and the side that sends the call request. The user terminal 20 can be both a caller and a callee, and may therefore have the functions of both.

[0044] As described above, the server 10 has a function of controlling calls made over the Internet. For example, when using communication in SIP, the server 10 uses HTTP (Hyper Text Requests (SIP methods) and responses (Response Methods) based on the Transfer Protocol Call control is performed by interacting with the

[0045] In the call function of the present disclosure, before the above-mentioned exchange of requests and responses, image data is The user's status information is displayed on the screen. The functions of the user terminal 20A on the image data / audio data transmission side will be mainly explained, and the functions of the user terminal 20B on the image data / audio data transmission side will be explained. In Section 0B, we will mainly explain the functions of displaying user status information.

[0046] In addition, an example will be described in which the user terminal 20A and the user terminal 20B have the same functions. Therefore, when there is no need to distinguish between the functions, the symbols A and B are omitted. It can be expressed as follows: the user terminal 20 has an image acquisition unit 302. 0 performs either the function of transmitting media data or the function of displaying user status information. It may have.

[0047] The user terminal 20A includes an image acquisition unit 302A, a sound acquisition unit 304A, a transmission unit 306A, a reception unit 306B, a reception unit 306C, a reception unit 306D, a reception unit 306E, a reception unit 306F, a reception unit 306H, a reception unit 306I ... unit 312A, data acquisition unit 314A, generation unit 316A, output control unit 318A, and call processing unit The functions of the various units in the user terminal 20A are controlled by the control unit 202 shown in FIG. This is realized by executing the application program 210.

[0048] The user terminal 20A is executing an application related to the call function of the present disclosure. In this case, the camera function and microphone function are set to ON, and image data of the surroundings of the user terminal 20A is acquired. and sound data is input.

[0049] (Data transmission function of user terminal 20A) The image acquisition unit 302A acquires image data captured by the imaging device 224. For example, the image data may be image data including the first user or image data including the state of the room. The acquired image data is stored in a buffer, for example, and deleted after a predetermined time has elapsed. do.

[0050] The sound acquisition unit 304A acquires sound data collected by the microphone 220. For example, The data may be sound data including the voice of the first user or sound data including everyday sounds. The generated sound data is stored in a buffer, for example, and deleted after a predetermined time has elapsed.

[0051] The transmitting unit 306A transmits the image data and / or sound acquired by the image acquiring unit 302A. The media data including the sound data acquired by the acquisition unit 304A is transmitted to the server 10. The destination is set in advance to the server 10. The transmitting unit 306A also transmits data to the server 10. When transmitting the information to the server 10, the identification information of the user terminal 20A or the identification information of the first user is also included. It may be transmitted as follows.

[0052] By the above process, the first user can download image data and sound data without performing any special operations. The data can be included in the media data and transmitted to the server 10. For example, Even an elderly first user who is not familiar with the process can easily perform the sending process without any special operations. It will be carried out.

[0053] (Session establishment function of user terminal 20A) As will be described later, when a call request is sent from the user terminal 20B, the receiving unit 312 A receives the call request via the server 10. The user terminal 20A receives the call request. When the message is received, the first user terminal 20B notifies the second user terminal 20B that there is a call request from the second user terminal 20B. When the first user performs an operation to respond to the call request, The user terminal 20A transmits a call response to the user terminal 20B via the server 10. do.

[0054] When the call response is transmitted from the receiving unit 312A, the user When a session is established between the terminal 20A and the user terminal 20B, the call processing unit 320A transmits and receives call packets, which are packetized call data.

[0055] The receiving unit 312A receives a call termination request directly or indirectly from the user terminal 20B. If received, the call processor 320A may respond to the termination request directly or Indirectly, the message is sent to the user terminal 20B. This ends the session and the call. The call end request may be sent from the user terminal 20A.

[0056] (Call control function of server 10) The server 10 includes a receiving unit 402, a call control unit 404, an acquiring unit 406, a determining unit 408, and The server 10 includes a transmission unit 410. The functions of each unit in the server 10 are controlled by the control unit 102 shown in FIG. This is realized by executing the control program 108.

[0057] The receiving unit 402 of the server 10 receives the image transmitted from the transmitting unit 306A of the user terminal 20A. Receive data / audio data, for example, image data included in the received media data The sound data is stored in, for example, the RAM of the control unit 102.

[0058] The acquisition unit 406 acquires image data / audio data stored in a RAM or the like. The acquiring unit 406 receives the image data / audio data transmitted by the user terminal 20A from the receiving unit 402. The data may be acquired.

[0059] The determination unit 408 determines the first image / audio data of the user terminal 20A based on the acquired image / audio data. 1. Determine the state of the user. For example, the determination unit 408 analyzes the image data / audio data. The state of the first user, such as whether the first user is in the vicinity of the user terminal 20A, is determined. The state of the user can be selected from multiple states depending on the analysis results of the image data / sound data, for example. It may be one state in which

[0060] Furthermore, the determination unit 408 determines the state every time image data / audio data is acquired. Only when the state information has changed, the updated state information is transmitted from the transmitting unit 410. This can reduce the load of the transmission process on the server 10.

[0061] Furthermore, the determination unit 408 uses the image data / audio data as learning data to determine the state of the first user. The state of the first user may be determined using a learning model that has learned the above.

[0062] The transmitting unit 410 is associated with the identification information of the user terminal 20A and communicates with the user terminal 20A. Information about the state of the first user (state information) is transmitted to the user terminal 20B that is available to talk. For example, the transmitting unit 410 transmits status information including whether the first user is available to make a call to the user. The transmitting unit 410 transmits the status to the user terminal 20B only when the status is updated. The information may be transmitted to the user terminal 20B.

[0063] Through the above process, the state of the first user is determined using image data and sound data, and the call is By notifying the original user of the status information of the first user who is the destination of the call, the call The caller can easily understand the status of the callee.

[0064] Furthermore, the determination unit 408 performs object recognition on the acquired image data, and determines the result of the object recognition. For example, the determination unit 408 may determine the state of the user based on the image. Object recognition may be performed on the image data to detect whether a person is included. This may be done using a learning model that has been machine-learned for each object in advance. Body recognition is performed using a learning model that has undergone supervised learning using images of the first user as training data. This may be done in the following manner.

[0065] In this case, the determination unit 408 determines whether there are any users around the user terminal 20A based on the object recognition result. If a user is present, the determination unit 408 determines whether or not a user (for example, a first user) is present. If the user is not present, it is determined that the call is not possible. The determination unit 408 determines, as a result of object recognition, that the user is sitting on a sofa or the like and has not moved for a predetermined period of time. If the user is in a relaxed state, the system determines that the user is in a relaxed state. If it is detected that the user is busy, the determining unit 408 may determine that the user is busy. The state may be identified from the recognition result by associating the recognition result with each state.

[0066] Through the above process, the determination unit 408 determines whether the first user is available for communication or not. For example, the determining unit 408 can determine the state in more detail than the first state. 1 It becomes possible to determine whether the user is absent or not, whether the user is moving or not, etc.

[0067] Furthermore, the determination unit 408 recognizes the facial expression of the first user included in the image data, and The method may further include determining a state of the first user based on a result of the recognition. 408, when detecting that the first user is included in the image data, The facial expression recognition may be performed using a known technology. Using artificial intelligence, the system estimates the first user's happiness, surprise, anger, sadness, and neutral expression.

[0068] Through the above process, the determination unit 408 determines whether the first user is available for communication or not. For example, the determination unit 408 can estimate the emotional state in more detail. The system estimates the current emotional state of the first user based on the facial expression of the first user. It becomes possible to do this.

[0069] The determination unit 408 also performs frequency analysis on the sound data, and determines whether the result of this frequency analysis is For example, the determining unit 408 may determine the state of the first user based on the surrounding As a result of the wave number analysis, it is possible to determine whether the voice of the first user is included, whether the first user is on a train, The system determines whether the user is in a room, listening to music, etc.

[0070] Through the above process, the determination unit 408 determines whether the first user is available for communication or not. For example, the determination unit 4 can estimate the state of the sound in more detail. 08 can estimate the state of the first user's surroundings, such as whether it is noisy or quiet. It becomes possible.

[0071] The call control unit 404 receives the call request from the user terminal 20B by the receiving unit 402. For example, the call control unit 404 identifies the user terminal 20A based on the call request. The destination information (e.g., URI) is identified from the destination list, and the user is notified using the identified destination information. The call control unit 404 acquires the IP address of the terminal 20A. Alternatively, the IP address may be identified from the destination information by querying the server.

[0072] The transmission unit 410 sends a call request to the user terminal 20A identified by the call control unit 404. For example, the call control unit 404 may transmit a message to the user terminal having the specified IP address. Send a call request to end 20A.

[0073] In addition, the receiving unit 402 receives a call response from the user in response to the transmitted call request. After that, the call control unit 404 establishes a call session and It controls the mutual sending and receiving of call data.

[0074] (Status notification function of user terminal 20B) The user terminal 20B includes an image acquisition unit 302B, a sound acquisition unit 304B, a transmission unit 306B, a reception unit 306B, and a reception unit 306C. unit 312B, data acquisition unit 314B, generation unit 316B, output control unit 318B, and call processing unit The functions of the various units in the user terminal 20B are controlled by the control unit 202 shown in FIG. This is realized by executing the application program 210. The functions are the same as those of the user terminal 20A. The following mainly describes the status notification function. do.

[0075] The receiving unit 312B receives the status information of the first user of the user terminal 20A from the server 10. For example, the received status information of the first user is stored in the RAM of the control unit 202. At this time, the receiving unit 312B receives the status information of the first user from the user terminal 20A. The identification information of the first user or the identification information of the second user may also be received and stored in the RAM.

[0076] The data acquisition unit 314B acquires the status information of the first user stored in, for example, the RAM. In addition, the data acquisition unit 314B acquires the status information of the first user from the receiving unit 312B. Good too.

[0077] The generation unit 316B generates a notification of the state of the first user based on the state information of the first user. For example, the generation unit 316B generates status notification information based on the status of the first user indicated by the status information. A UI widget is generated that includes character data, image data, etc. that represent the

[0078] The output control unit 318B controls the output of the status notification information by the generation unit 316B. For example, if the status notification information is status display information, the display unit 318B displays the status information on the display screen of the display unit 214. If the status notification information is status sound information, the display is controlled, and if the status notification information is status sound information, the speaker 222 is controlled to output a sound. For example, the output control unit 318B controls the status notification information to be displayed as a pop-up. and control it in this way.

[0079] Through the above process, the second user can see the screen of the user terminal 20B and understand the information of the first user. As a result, if the first user is available to make a call, The second user can then simply call the first user.

[0080] The output control unit 318B may also control the display of status notification information within a predetermined area on the display screen. For example, the output control unit 318B may display the status of a window that is set in advance on the display screen. The output control unit 318B controls the display of the status notification information. The status notification information is displayed in a predetermined area set in the execution screen of the application being displayed. The predetermined area may be controlled to include a widget that can execute the above-mentioned application. For example, the widget may be an area where status notification information is displayed. This status notification information is updated appropriately on the widget.

[0081] By the above process, the second user can see the current state of the first user by looking at a predetermined area on the screen. It is possible to grasp the situation.

[0082] When the status notification information is updated, the output control unit 318B outputs the updated status notification information. For example, the output control unit 318B may control the display of the status notification information in a predetermined area. When the first state is updated to indicate the second state, the state notification information indicating the second state is Display is controlled within a fixed area.

[0083] By the above process, the status notification information is updated instead of being generated every time. By updating the display screen every time the status information is displayed, the frequency of screen updates due to status notification information can be reduced. This reduces the processing load on the user terminal 20B.

[0084] The generation unit 316B also generates an application icon for executing a call via the server 10. When a icon is present on the display screen, an image of the icon is generated as status notification information. For example, a function that dynamically changes the icon display, such as a calendar icon. Using the above, the generating unit 316B generates an image or The image or text of the icon corresponding to the status information acquired from the server 10 is set. In this case, the output control unit 318B generates the image specified by the generation unit 316B. Or, control is performed so that characters are displayed on the icon.

[0085] Through the above process, the second user can access the icon of the application that executes the function of the present disclosure. By viewing the icon, you can understand the status of the first user. For example, By looking at the icon displayed, it is determined whether the first user is available to make a call. This makes it possible.

[0086] Furthermore, the generating unit 316B generates a state image including a moving image or animation showing the state of the first user. For example, the generating unit 316B may generate the state notification information. The status notification information may be a moving image or animation that shows the user's status. In this case, the video is a video of the first user captured in a specific room. The generation unit 316B generates the state information indicating that the first user is in a relaxed state (sitting on a sofa) To show that the first user is sitting on a sofa in a room, the first user selects a past video of the first user sitting on a sofa in a room. If the first user is absent, select a past video in which the first user is not in the room. Or something like that.

[0087] Furthermore, the generation unit 316B generates, as the status information, an image data captured by the user terminal 20A. When acquiring data, effects (mosaic, different background, etc.) may not be applied to images other than the first user. It may be possible to protect privacy by using a security function.

[0088] Furthermore, the generation unit 316B generates an animation of the first user with the movement of a predetermined object. For example, the generation unit 316B may represent the state of the first user while the first user is moving or exercising. In this state, you can move the object quickly, or if the first user is absent, If you hide the object or the first user is in a relaxed state, Move it slowly.

[0089] By the above process, privacy can be protected when displaying the status of the first user. This allows the first user to authorize the execution of the functions of the present disclosure with peace of mind.

[0090] Furthermore, the generation unit 316B generates a message indicating that the information about the first user's status indicates that the first user is available for communication. In this case, the UI widget for starting the call processing for the user terminal 20A may be specified. In this case, the output control unit 318B outputs the status notification information and also displays the UI widget on the display screen. This may include controlling the display on the surface.

[0091] For example, if the status information indicates that a call is possible, the generation unit 316B may generate a notification to the first user. Identify or select the button to make a call. Note that the status information may include multiple statuses. It is sufficient if whether or not a call is possible for each state is set in advance. 318B may control the UI widget to be displayed in association with the status notification information. For example, the UI widget is controlled to be displayed in a position adjacent to the display information.

[0092] By the above process, the second user can use the UI widget if the first user is available to talk. By doing this, you can check the status of the first user and then make a call smoothly. .

[0093] When the UI widget for making a call to the first user is operated by the second user, the sending unit 306B The user terminal 20A transmits a call request to the user terminal 20A via the server 10. If the user terminal 20A returns an OK call response to the EST, the call starts. will be done.

[0094] By executing the functions of each device, the user who is the caller can In addition, the current status of the user who is the call destination can be grasped. If the emotions and movements of the user on the other end of the call can be included, the call can be made after understanding their state. This allows the start of a call to proceed smoothly.

[0095] <Screen example> Next, an example of a screen including the status of a user who is a call destination according to the first embodiment will be described with reference to FIG. 6. FIG. 6A is a diagram showing a screen example 1 including the status of the user at the other end of the call according to the first embodiment. In the example shown in Fig. 6A, the status notification information is displayed on the screen as a pop-up A1. In addition, the status information indicates that the user (Mr. A) is sitting on the sofa, so he can make a call. In addition, since it is determined that a call is possible, the "Call" button is pressed. The UI widget A2 is displayed on the screen.

[0096] FIG. 6B is a diagram showing a second example screen including the status of the user at the other end of the call according to the first embodiment. In the example shown in FIG. 6B, the state of the icon B1 of the application that executes the call function of the present disclosure is In this case, the user clicks on this icon B1 to display the status notification information. After transitioning to the application execution screen, a button for making a call may be displayed.

[0097] FIG. 6C is a diagram showing a third example screen including the status of the user at the other end of the call according to the first embodiment. In the example shown in FIG. 6C, a predetermined area of ​​the execution screen of the application that executes the call function of the present disclosure is The status notification information is displayed in area C1. Since the status indicates that the call is possible, it is determined that the call is possible. Accordingly, the UI widget A2 of the "Call" button is displayed on the execution screen.

[0098] FIG. 6D is a diagram showing a fourth example screen including the status of the user at the other end of the call according to the first embodiment. In the example shown in FIG. 6D, a plurality of screens are displayed on the execution screen of the application that executes the call function of the present disclosure. There are designated areas, and the current status of different users is displayed in each designated area C1 and D1. For example, if B is absent, the predetermined area D where the status notification information is displayed is 1 displays a black screen that evokes the image of a dark room.

[0099] In the example shown in FIG. 6D, the server 10 selects a plurality of destinations as candidates for a predetermined user. The users (Mr. A and Mr. B in the example shown in FIG. 6D) are set in the call handling information. The server 10 acquires image data / audio data from the user terminals 20 of both A and B. The user terminal 20 receives the status information of each user from the server 10. Upon receiving the notification, the status information of multiple users can be displayed on the screen as shown in FIG. 6D. It is possible.

[0100] This allows the user who is making a call to check the current status of multiple users who are potential callees. It can be easily understood.

[0101] <Operation> Next, each operation of the information processing system 1 will be described. 1 is a flowchart showing an example of the operation process of the server 10. The user terminal 20A of the first user transmits image data and the like, and the user terminal 20 of the second user who is the call originator. The operation will be described for an example in which B displays the status of the first user.

[0102] In step S102, the receiving unit 402 of the server 10 receives the message from the user terminal 20A of the first user. Then, the server receives the image data / audio data contained in the media data sent from the server. The acquisition unit 406 acquires the received image data / audio data.

[0103] In step S104, the determination unit 408 of the server 10 determines whether the acquired image data / audio data The current state of the first user is determined based on the

[0104] In step S106, the determining unit 408 of the server 10 determines whether the determined state is different from the current state. If the state has changed (step S106-YES), the process proceeds to step S106. If the state does not change (step S106-NO), the process proceeds to step S108. Return to step S102.

[0105] In step S106, the determination unit 408 of the server 10 determines whether the media containing the image data / audio data is The data is associated with the identification information of the user terminal 20A that transmitted the data or the identification information of the first user. The user terminal 20B, which is available for communication with the user terminal 20A, is identified as the notification destination. The determination unit 408 can identify the notification destination by, for example, referring to the call response information described above. is possible.

[0106] In step S108, the transmitting unit 410 of the server 10 transmits a message regarding the determined state of the first user. The information (status information) is transmitted to the user terminal 20B specified as the notification destination.

[0107] FIG. 8 is a flowchart showing an example of an operation process on the status display side according to the first embodiment. In the example shown in Fig. 8, the processing performed by the user terminal 20B will be mainly described. The status notification information is status display information, and the output control unit 318B outputs the status display information to the display screen. This is an example of controlling the display on a surface.

[0108] In step S202, the receiving unit 312B of the user terminal 20B receives the first user ID from the server 10. The data acquisition unit 314B of the user terminal 20B receives the status information of the user. Obtain the status information.

[0109] In step S204, the generation unit 316B of the user terminal 20B receives the acquired first user's The generation unit 316B generates status display information based on the status information. For example, the generation unit 316B generates the status display information based on the status information of the first user. Generate UI widgets that include images and text that indicate the status of the device.

[0110] In step S206, the output control unit 318B of the user terminal 20B outputs the generated status display Control the display of information on the display screen.

[0111] In step S208, the user terminal 20B receives a UI widget for initiating a call from the second user. If an operation to start a call has been performed (step S208-YES), S), the transmitting unit 306B transmits a call request to the user terminal 20A via the server 10. Then, the process proceeds to step S210, and if there is no operation to start a call (step S2108- If NO, the process returns to step S202.

[0112] In step S210, the receiving unit 312B of the user terminal 20B receives the signal from the user terminal 20A. If a call response is received, (Step S210-YES), the process proceeds to step S212, where the call response is received. If not (step S210-NO), the call is terminated because the first user did not answer. The process ends, and the process returns to step S202.

[0113] In step S212, the user terminal 20B starts a call session. A starts a call by sending and receiving call data.

[0114] As described above, according to the information processing system of the first embodiment, before starting a call, The user who is the caller can know the current status of the user who is the caller.

[0115] The information processing described in the first embodiment is executed by a computer. This program may be installed from a server or the like and executed as a computer program. By causing a computer to execute the program, the above-described information processing can be realized.

[0116] In addition, this program may be recorded on a recording medium, and the recording medium on which this program is recorded may be used for copying. It is also possible to have a computer read the recording medium and realize the above-mentioned information processing. can store the program in a "non-transitory tangible medium."

[0117] The recording medium may be a CD-ROM, a flexible disk, a magneto-optical disk, etc. Recording media that record information optically, electrically or magnetically, such as ROM and flash memory Various types of recording media can be used, such as semiconductor memory that electrically records information. do.

[0118] [Second embodiment] FIG. 9 is a diagram illustrating an overview of a system 2 according to a second embodiment of the present disclosure. The system 2 to be implemented is the same as the system for managing images described in the first embodiment. The system includes a management device 10A, a server 10B, a user terminal 20, and a display The image output device 30 includes a camera 60. The image output device 30 is connected to a display device 40. The display device 40 is connected to a remote control device 50 and controlled by the remote control device 50. The image captured by the camera 60 is input to the image output device 30. For example, the camera 60 may be a web camera with a built-in microphone. B has the functions of the server 10 described in the first embodiment.

[0119] The management device 10A or the server 10B and the user terminal 20 are connected by wireless communication, which is an example of a communication technology. LAN or 4th or 5th generation mobile communication system (4G or 5G) or LTE (Long They can communicate with each other through the network N1, which is based on the Term Evolution (Terminology Evolution) and other technologies. The processing device 10A and the image output device 30 are connected to a network such as a third generation mobile communication system (3G). Wireless network N2 has lower communication charges but is slower than network N1. For the sake of explanation, the network N1 and the wireless network N Although the two networks are described separately, they are interconnected via the Internet. Furthermore, the network configuration is not limited to the above example.

[0120] The management device 10A first receives a request from the image poster via an internet website. , and obtain information such as the name and address of the viewer or poster of the image. Information such as name and address is not essential information. An identifier (called a device ID) for the image output device 30 to be used is generated. The poster of the article will be notified of the device ID, for example by email.

[0121] The management device 10A sets the generated device ID in the image output device 30. The image output device 30 is sent to the address of the image viewer by the administrator of the management device 10A. The image output device 30 has a built-in 3G communication module, for example, and when the power is turned on, When the device ID is set, the device immediately starts communication with the management device 10A using the set device ID. It is composed.

[0122] On the other hand, the poster of the image uses a user terminal 20 such as a smartphone or tablet. Download a working image sharing application. The device accesses the management device 10A using the notified device ID. The user terminal 20 (posting) uses the device ID notified from the application as a key. This information can be used to associate the image output device 30 (viewer) with the image output device 30 (user). This corresponds to the call handling information.

[0123] After that, the poster can use the application to take photos of various subjects. The application sends the image data acquired by the shooting via the network N1. The poster automatically sends the image data to the management device 10A. No special action is required to

[0124] The management device 10A stores image data transmitted from an application of the user terminal 20. The image output device 30 then outputs the images in response to an instruction from the viewer. The image data is then displayed on the display device 40. The image output device 30 is also equipped with a microphone and a speaker. It may also have a speakerphone and is capable of IP calling.

[0125] Here, the display device 40 is, for example, a television set that is widely used in ordinary homes, and a remote control device. The image output device 30 is, for example, an HDMI (High The display device 40 is connected to the display device 40 via the H-Definition Multimedia Interface (registered trademark). Control signals issued from the remote control device 50 can be obtained via the DMI.

[0126] The image output device 30 receives the control signal from the remote control device 50 and inputs the image data. In other words, the viewer can understand the operation that was performed. It becomes possible to view the distributed image data using a television remote control.

[0127] The camera 60 is, for example, a web camera, and may have a built-in microphone. The captured image (still image or moving image) data / audio data is input to the image output device 30. The image output device 30 supports media data related to the input image data / audio data. The data is sent to Server 10.

[0128] The above-mentioned image sharing application starts the call described in the first embodiment. In this case, the user terminal 20A shown in FIG. For example, the image output device 30 corresponds to the transmitting unit 306 of the user terminal 20, the receiving unit 307 of the user terminal 20, and the a receiving unit 312, a data acquiring unit 314, a generating unit 316, an output control unit 318, and a call processing unit 320 It has.

[0129] Specifically, media data including image data / audio data acquired by the camera 60 The acquired message is transmitted to the server 10B via the image output device 30. The server 10B The state of the viewer of the image is determined based on the image data / audio data included in the media data. For example, the server 10B analyzes the image data / audio data to Viewers moving in a room, sitting in a chair, not in the room, busy, quiet The state is determined as follows:

[0130] The server 10B refers to the information held in the management device 10A and acquires the information related to the image output device 30. The user terminal 20 to which the user is connected is identified, and the status information of the viewer is transmitted to the user terminal 20. As described above, the user terminal 20 generates status notification information (e.g., status display information) based on the status information. The system generates the information and controls it to be displayed on the display screen (see, for example, Figure 6).

[0131] When the user terminal 20 detects that the poster has operated the button to make a call, the user terminal 20 The quest is transmitted to the image output device 30 via the server 10B. The image output device 30 When a call request is received, the display device 40 displays whether or not to answer the call. The first button on the control device 50 is assigned to answer and the second button is assigned to reject.

[0132] When the viewer operates the first button, the image output device 30 detects the operation of the first button. Then, the server 10B transmits a call response to the user terminal 20 via the server 10B. It will begin.

[0133] As described above, according to the second embodiment, the system for sharing images between viewers and contributors is provided with the same functions as those of the first embodiment. The management device 10A and the server 10 B may be the same server.

[0134] [Third embodiment] Next, a third embodiment of the present disclosure will be described. In the third embodiment, The function of the determination unit 408 described above is provided in the user terminal 20. For example, The determination unit 300 determines whether the image data acquired by the image acquisition unit 302 and / or the sound acquisition unit 303 Based on the sound data acquired by 04, the state of the user using the terminal is determined.

[0135] The method for determining the user's state is as described above. For example, performs object recognition on image data and / or frequency analysis on sound data. The results of each are used to determine the user's state.

[0136] The transmitting unit 306 of the user terminal 20 transmits the information about the user's state determined by the determining unit. (state information) to the other user terminal before the call. For example, the transmitting unit 306 In order to transmit status information to other user terminals, the status information is first transmitted to the server 10. When the server 10 acquires the status information, it sends the status information to the user terminal 20. Identify the destination (other user device) and send status information to the destination When the user terminal 20 receives the status information, it performs the same process as in the first embodiment. Execute.

[0137] As described above, according to the third embodiment, it is possible to distribute the load of the server 10. The terminal 20 processes the image data and sound data at its own terminal, and transmits the image data and sound data to the terminal 20. Since the information is not transmitted to the outside, privacy can be protected.

[0138] [Fourth embodiment] Next, a fourth embodiment of the present disclosure will be described. In the fourth embodiment, A part of the functions of the determination unit 408 described above is provided in the user terminal 20. For example, The preprocessing unit of the terminal 20 performs object recognition on the image data acquired by the image acquisition unit 302. and / or performs frequency analysis on the sound data acquired by the sound acquisition unit 304. It detects objects and extracts frequency features.

[0139] The transmitting unit 306 of the user terminal 20 receives the object recognition results processed by the pre-processing unit and / or The media data including the processing result information including the feature amount (the result of the frequency analysis) is sent to the server 10. When the server 10 acquires the media data, the determination unit 408 determines whether the acquired media data is The state of the user is determined based on the processing result information included in the media data. 408 holds a table that associates each obtainable result information with the user's status. By doing so, the user's status can be determined from the result information.

[0140] When the server 10 determines the user's state, the server 10 notifies the user terminal 20 that transmitted the media data. and identifying a notification destination associated with the user, and transmitting information about the user's status to the identified notification destination. When the notification destination user terminal 20 receives the status information, it transmits the status information to the first embodiment. The same processing as in the previous example is performed.

[0141] As described above, according to the fourth embodiment, it is possible to reduce the amount of communication traffic more than in the first embodiment. This allows the computational resources of the determination unit, which places a high processing load on the system, to be shared.

[0142] The above-described embodiments are merely examples for explaining the present invention, and the present invention can be implemented by The present invention is not limited to the above embodiment, and various modifications are possible without departing from the spirit of the present invention. Shape is possible.

[0143] [Variations] For example, in a first modification, the output control unit 318 of the user terminal 20 may generate the following signal: In addition to controlling the output of the generated status information, it also controls the operation of the equipment according to the status information. As a specific example, the output control unit 318 may adjust the brightness of the lighting in the room in accordance with the state information. , the volume of the device playing music (your own device or music player), the device watching TV or videos (your own device or music player) Controlling networked devices, such as pausing or adjusting the volume of a device (such as a mobile phone or television) It may also function as an operation control unit.

[0144] For example, if the call destination is available for communication, the output control unit 318 may turn on the lights in the room. You can also turn down the volume of the music or pause a video that is playing. This makes it possible to create an environment where the caller can easily make a call. 8 stores control commands corresponding to each user's state, and executes them according to the state information. The specified control command may be executed.

[0145] As a second modification, in each of the above-described embodiments, the generation unit 316 may However, it may be provided in the server 10. For example, the generating unit of the server 10 may be provided in the determining unit. Status notification information is generated according to the user's status determined in 408.

[0146] In this case, the transmitting unit 410 of the server 10 receives the information about the user's status (status information). In other words, in the case of the second modification, the server 10 The status notification information is included in the information about the user's status transmitted from the user terminal 20. When the output control unit 318 acquires the status notification information, it displays the status notification information on the screen. This allows the server 10 to easily add or modify status notification information. It becomes possible.

[0147] As a third modification, the determining unit 408 may determine a predetermined A specific sensor (for example, an illuminance sensor, a photo sensor such as a human sensor, an ultrasonic sensor, etc.) For example, the determination unit 408 may determine the user's state by using information acquired from When the detection signal of a specific sensor is ON, it can be determined that a user is present at the other end of the call. This may require some cost to install a specific sensor, but it is possible to The state of the previous user can be determined in the same manner as in the above-described embodiment. By using the media data and signals acquired by a predetermined sensor, more appropriate pre-call It becomes possible to determine the status of the user at the other end of the call. [Explanation of symbols]

[0148] 1...information processing system, 2...system, 10...server, 10A...management device, 10B...server 20...user terminal, 30...image output device, 40...display device, 60...camera, 102... Control unit, 106...storage unit, 202...control unit, 204...RAM, 208...storage unit, 220... Microphone, 222... speaker, 224... imaging device, 302... image acquisition unit, 304... sound acquisition unit , 306...transmitting unit, 312...receiving unit, 314...data acquiring unit, 316...generating unit, 318... Output control unit 320...call processing unit 402...receiving unit 404...call control unit 406...acquisition 408: Determination unit; 410: Transmission unit

Claims

1. In the information processing device, Media data relating to image data and / or sound data transmitted from the first communication device To obtain, determining a state of a user of the first communication device based on the media data; a second communication device associated with the identification information of the first communication device and capable of communicating with the first communication device; and causing the device to transmit information about the user's status.

2. The determining step comprises: Object recognition is performed on image data included in the media data, and at least The process of claim 1 further comprising: determining a state of the user based on a result of the body recognition. Grams.

3. The determining step comprises: Recognizing the facial expression of the user included in the image data included in the media data, determining a state of the user based on at least a result of the facial expression recognition, 3. The program according to claim 1 or 2.

4. The determining step comprises: A frequency analysis is performed on the sound data included in the media data, and The method according to any one of claims 1 to 3, further comprising determining the state of the user based on the result of wave number analysis.

2. The program according to claim 1 .

5. The determining step comprises: The processing result information included in the media data is an object recognition result of the image data. and / or based at least on the processing result information including the result of frequency analysis of the sound data. The program according to claim 1 , further comprising determining the state of the user based on the state of the user.

6. The information processing device Acquiring media data relating to image data and / or sound data from the first communication device and, determining a state of a user of the first communication device based on the media data; a second communication device associated with the identification information of the first communication device and capable of communicating with the first communication device; and transmitting information about the user's status to a device.

7. Acquiring media data relating to image data and / or sound data from the first communication device Tokube and a determination unit that determines a state of a user of the first communication device based on the media data; 、 a second communication device associated with the identification information of the first communication device and capable of communicating with the first communication device; a transmitter for transmitting information about the user's status to a device; An information processing device comprising:

8. To the communication device, Media data relating to image data and / or sound data acquired by other communication devices and acquiring information about the status of the user using the other communication device, the information being determined based on the To do so, a status information device capable of notifying the user of the user's status based on the information about the user's status; generating information; output control of the status notification information; A program that executes the following.

9. The output control includes: Controlling the display of the status notification information within a predetermined area of ​​a display screen; When the status notification information is updated, the updated status notification information is displayed in the predetermined area. The program according to claim 8, further comprising:

10. The generating step comprises: An icon of an application for executing a call via the information processing device is present on the display screen. If the icon exists, an image of the icon is generated as the status notification information.

9. The program described in

11. The generating step comprises: generating status notification information including a moving image or animation showing the status of the user; The program according to any one of claims 8 to 10, comprising:

12. The generating step comprises: If the information about the user's status indicates that the user is available to make a call, identifying a UI component for initiating call processing for the The output control includes: and controlling the display of the UI components on a display screen together with outputting the status notification information. The program according to any one of claims 8 to 10.

13. The communication device Media data relating to image data and / or sound data acquired by other communication devices and acquiring information about the status of the user using the other communication device, the information being determined based on the To do so, a status information device capable of notifying the user of the user's status based on the information about the user's status; generating information; output control of the status notification information; An information processing method that performs the above.

14. Media data relating to image data and / or sound data acquired by other communication devices and acquiring information about the status of the user using the other communication device, the information being determined based on the an acquisition unit; a status information device capable of notifying the user of the user's status based on the information about the user's status; a generating unit that generates information; an output control unit that controls output of the status notification information; A communication device comprising:

15. To the communication device, acquiring image data and / or sound data; The state of the user using the communication device is determined based on the image data and / or sound data. To determine the The information about the user's status is transmitted to another communication device associated with the communication device before the call. output to the location, A program that executes the following.

16. The communication device acquiring image data and / or sound data; The state of the user using the communication device is determined based on the image data and / or sound data. To determine the The information about the user's status is transmitted to another communication device associated with the communication device before the call. output to the location, An information processing method that performs the above.

17. an acquisition unit that acquires image data and / or sound data; The state of the user using the communication device is determined based on the image data and / or sound data. A determination unit that determines The information about the user's status is transmitted to another communication device associated with the communication device before the call. a transmitter for outputting the signal to the device; A communication device comprising:

Images