Information processing method, information processing device, and information processing program

Abstract

An information processing device according to the present invention collects verbal information and non-verbal information for a plurality of users that are in a prescribed space, determines the level of cooperation among the plurality of users on the basis of at least one of the verbal information and the non-verbal information collected, and outputs a control signal for controlling the space in which the users are to a state that corresponds to the determined level of cooperation to an adjustment device.

Description

Information processing method, information processing device, and information processing program

[0001] The present disclosure relates to a technique for controlling a space in which multiple users exist.

[0002] Patent Literature 1 discloses a work support device that sends instructions to fixtures in a workspace according to the actual work situation. This work support device collects the activity history of a worker in a given workspace, determines the usage mode of the fixtures arranged in the worker's workspace, and sends instructions to the fixtures at a predetermined timing based on the activity history and the determined usage mode.

[0003] However, in the conventional technology disclosed in Patent Document 1, the degree of collaboration between users is not determined, and therefore it is not possible to make the state of the space appropriate for each user depending on the degree of collaboration.

[0004] JP 2018-197770 A

[0005] In view of these issues, the present disclosure aims to provide a technology that determines the degree of collaboration among users in a space and makes the state of the space appropriate for each user depending on the degree of collaboration.

[0006] An information processing method in one aspect of the present disclosure is an information processing method in a computer that collects verbal information and nonverbal information of multiple users within a specified space, determines a degree of collaboration among the multiple users based on at least one of the collected verbal information and nonverbal information, and outputs a control signal to an adjustment device to control the space in which the multiple users are located to a state corresponding to the determined degree of collaboration.

[0007] With this configuration, the state of the space can be made appropriate for each user depending on the degree of collaboration.

[0008] It is a diagram showing the overall configuration of an information processing system according to the present embodiment. It is a diagram showing an example of a space to which the information processing system is applied. It is a flowchart showing the processing of the information processing system according to the present embodiment. It is a graph showing an example of the temporal transition of the degree of cooperation.

[0009] (Findings underlying the present disclosure) Many workspaces with various functions, such as coworking spaces and shared offices, have been created. In such workspaces, some users work collaboratively, while others work alone. Therefore, if the workspace state is set to an appropriate state depending on the degree of collaboration among these users, the work of these users can be promoted.

[0010] In the prior art shown in Patent Document 1, the user is prompted to input activity details such as concentrated work, working alone, sharing information, working with others, discussion / consultation, and idea generation on an activity details input screen, and the shape of the fixtures is changed according to the activity details input.

[0011] However, in Patent Document 1, the degree of collaboration between users is not determined from verbal information, such as linguistic information such as conversations and letters, or non-verbal information, such as the user's facial expressions, line of sight, gestures, and hand movements, so it is not possible to make the state of the space appropriate for each user depending on the degree of collaboration. Furthermore, in Patent Document 1, the user is required to perform an operation to input activity details, which is cumbersome for the user.

[0012] Therefore, the inventors focused on the verbal and non-verbal information of the users in a space and discovered that by determining the degree of collaboration between the users from this information, the state of the space can be controlled to be appropriate for each user, which led to the present disclosure.

[0013] (1) An information processing method in one aspect of the present disclosure is an information processing method in a computer, which collects verbal information and nonverbal information of multiple users in a specified space, determines a degree of collaboration among the multiple users based on at least one of the collected verbal information and nonverbal information, and outputs a control signal to an adjustment device to control the space in which the multiple users are located to a state corresponding to the determined degree of collaboration.

[0014] According to this configuration, the degree of collaboration for multiple users is determined based on at least one of the collected verbal information and nonverbal information, and a control signal for controlling the space where the multiple users are present is output to the adjustment device according to the degree of collaboration. Therefore, the state of the space can be made appropriate for each user according to the degree of collaboration. As a result, the user's work can be promoted. Because the degree of collaboration is determined based on at least one of the verbal information and nonverbal information, the degree of collaboration can be determined without requiring users to input details of their activities in the space via an input screen, as in conventional technology. This eliminates processes such as displaying an input screen for activity details and accepting instructions on activity details input by users via the input screen, thereby reducing the processing load on the computer.

[0015] (2) In the information processing method described in (1) above, two or more users among the plurality of users that are within a predetermined distance may be detected as candidate collaboration users based on the collected nonverbal information, and determining the degree of collaboration may include determining the degree of collaboration for a group that includes the detected candidate collaboration users based on the collected nonverbal information.

[0016] In this case, based on the distance between users, it is possible to distinguish between users who are clearly not collaborating and users who may be collaborating, and only process users who may be collaborating, thereby reducing the processing burden.

[0017] (3) In the information processing method described in (2) above, the determination of the degree of collaboration may involve detecting the synchronicity of the actions of the candidate collaboration users based on at least one of the collected verbal information and non-verbal information, and determining the degree of collaboration based on the detected synchronicity of the actions.

[0018] Since users who collaborate with each other work together while interacting with each other, their actions are synchronized. With this configuration, the degree of collaboration is determined from the synchronicity of their actions, so the degree of collaboration can be calculated accurately.

[0019] (4) In the information processing method described in (2) or (3) above, the nonverbal information may include at least one of the facial direction and gaze of the candidate collaboration user, and the degree of collaboration may be determined based on at least one of the facial direction and gaze of the candidate collaboration user.

[0020] Since users who are collaborating work face to face or while gazing at the same object, their facial orientation and gaze are facing each other or directed toward the same object. With this configuration, the degree of collaboration is determined from at least one of the facial orientation and gaze, so the degree of collaboration can be accurately calculated.

[0021] (5) In the information processing method described in any one of (2) to (4) above, the verbal information may include the amount of speech of the candidate collaboration user, and the degree of collaboration may be determined based on the amount of speech.

[0022] Since users who collaborate with each other talk to each other while working on a task, the amount of speech increases. With this configuration, the degree of collaboration is determined from the amount of speech, so the degree of collaboration can be calculated accurately.

[0023] (6) In the information processing method described in (5) above, the nonverbal information includes at least one of the facial direction and gaze of the candidate collaboration user, and further, a period in which the candidate collaboration user is talking to himself is identified based on the amount of speech and at least one of the facial direction and gaze, and the amount of speech is corrected so that the period is removed, and the degree of collaboration may be determined based on the corrected amount of speech.

[0024] In this case, the amount of speech is corrected to remove the period during which the person talked to themselves, and the degree of cooperation is determined from the corrected amount of speech, so that the degree of cooperation can be calculated more accurately.

[0025] (7) In the information processing method described in any of (2) to (6) above, the nonverbal information may include schedule information of the candidate collaboration user, and the degree of collaboration may be determined based on the schedule information.

[0026] According to this configuration, the degree of collaboration can be calculated with high accuracy using the schedule information of the collaboration candidate users.

[0027] (8) In the information processing method described in any of (2) to (7) above, the adjustment device may be a lighting device, and the output may include outputting the control signal to the lighting device to illuminate all of the candidate collaboration users as a whole when the determined degree of collaboration exceeds a threshold, and outputting the control signal to the lighting device to illuminate the candidate collaboration users individually when the determined degree of collaboration does not exceed a threshold.

[0028] According to this configuration, if the degree of collaboration exceeds the threshold, the candidate collaboration users are determined to be collaborating, and all of the candidate collaboration users are lit up as a whole. This creates a sense of unity among the candidate collaboration users and promotes collaborative work. On the other hand, if the degree of collaboration does not exceed the threshold, the candidate collaboration users are determined not to be collaborating, and each candidate collaboration user is lit up individually, which increases the concentration of each candidate collaboration user and promotes the work of each candidate collaboration user.

[0029] (9) In the information processing method described in any of (2) to (8) above, the adjustment device may be a speaker device, and the output may include outputting the control signal to the speaker device that outputs sound directed at all of the candidate collaboration users when the degree of collaboration exceeds a threshold, and outputting the control signal to the speaker device that outputs sound directed at individual users of the candidate collaboration users when the degree of collaboration does not exceed a threshold.

[0030] According to this configuration, if the degree of collaboration exceeds the threshold, the collaboration candidate users are determined to be collaborating, and a sound is output to all of the collaboration candidate users. This creates a sense of unity among the collaboration candidate users and promotes collaborative work. On the other hand, if the degree of collaboration does not exceed the threshold, the collaboration candidate users are determined not to be collaborating, and a sound is output directed at each collaboration candidate user, thereby increasing the concentration of each collaboration candidate user and promoting the work of each collaboration candidate user.

[0031] (10) In the information processing method described in (4) above, the degree of collaboration is set to a larger value the longer the face-to-face time during which at least one of the facial direction and the gaze of the first collaboration candidate user is directed toward a second collaboration candidate user different from the first collaboration candidate user, and the face-to-face time may be the time during which a vector indicating the facial direction or the gaze of the first collaboration candidate user intersects with a specified area including the face of the second collaboration candidate user.

[0032] In this case, the degree of cooperation can be calculated more accurately using at least one of the facial direction and the line of sight.

[0033] (11) In the information processing method described in (5) above, the degree of collaboration may be set to a larger value as the amount of speech increases, and the amount of speech may be a speech time of the collaboration candidate user.

[0034] In this case, the degree of collaboration can be calculated more accurately using the amount of speech.

[0035] (12) In the information processing method described in (7) above, when the schedule information of a first collaboration candidate user and the schedule information of a second collaboration candidate user different from the first collaboration candidate user both indicate a collaborative task, the degree of collaboration may be set to a larger value than when at least one of the schedule information of the first collaboration candidate user and the schedule information of the second collaboration candidate user does not indicate the collaborative task.

[0036] In this case, the degree of cooperation can be calculated more accurately using the schedule information.

[0037] (13) In the information processing method described in any one of (1) to (12) above, the verbal information may include speech data of each of the multiple users obtained by analyzing audio data of the multiple users picked up by a microphone, and the non-verbal information may include at least one of facial direction and gaze of each of the multiple users obtained by analyzing image data of the multiple users captured by a photographing device.

[0038] In this case, verbal information can be obtained from the audio data picked up by the microphone, and non-verbal information can be obtained from the image data captured by the imaging device.

[0039] (14) In the information processing method described in any one of (1) to (13) above, if the group includes multiple groups, the degree of collaboration may be determined for each of the multiple groups, and control of the space may be performed for each of the multiple groups.

[0040] According to this configuration, when a plurality of groups are detected, the state of the space can be controlled individually for each group.

[0041] (15) In another aspect of the present disclosure, an information processing device includes a collection unit that collects verbal information and nonverbal information of multiple users within a specified space, a determination unit that determines the degree of collaboration of each of the multiple users based on at least one of the collected verbal information and nonverbal information, and an adjustment unit that outputs a control signal to an adjustment device to control the space in which the multiple users are located to a state corresponding to the determined degree of collaboration.

[0042] According to this configuration, it is possible to provide an information processing device that makes the state of space appropriate for each user depending on the degree of collaboration.

[0043] (16) In yet another aspect of the present disclosure, an information processing program causes a computer to execute a process of collecting verbal information and nonverbal information of multiple users within a specified space, determining a degree of collaboration among each of the multiple users based on at least one of the collected verbal information and nonverbal information, and outputting a control signal to an adjustment device to control the space in which the multiple users are located to a state corresponding to the determined degree of collaboration.

[0044] According to this configuration, it is possible to provide an information processing program that makes the state of the space appropriate for each user depending on the degree of collaboration.

[0045] The present disclosure can also be realized as an information processing system operated by such an information processing program. Needless to say, such a computer program can be distributed on a non-transitory computer-readable recording medium such as a CD-ROM or via a communication network such as the Internet.

[0046] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all of the embodiments, the respective contents can be combined.

[0047] 1 is a diagram showing the overall configuration of an information processing system 1 according to the present embodiment. The information processing system 1 is a system that determines the degree of collaboration of users in a space based on verbal and non-verbal information of the users in the space, and makes the space appropriate for each user according to the degree of collaboration.

[0048] The information processing system 1 includes a server 10, a microphone device 20, a photographing device 30, a speaker device 40, and a lighting device 50. The server 10, the microphone device 20, the photographing device 30, the speaker device 40, and the lighting device 50 are connected to each other so as to be able to communicate with each other via a network. The network may be a wide area communication network including the Internet and a mobile phone communication network. However, this is just an example, and the network may also be a local area network.

[0049] The server 10 includes a processor 11, a communication device 12, and a memory 13. The processor 11 is configured as a central processing unit (CPU). The processor 11 includes a collection unit 111, a detection unit 112, a determination unit 113, and an adjustment unit 114. The collection unit 111, the detection unit 112, the determination unit 113, and the adjustment unit 114 are realized by the CPU executing an information processing program stored in the memory 13. However, this is just an example, and the collection unit 111, the detection unit 112, the determination unit 113, and the adjustment unit 114 may be configured as dedicated integrated circuits or may be distributed across multiple computers. The server 10 may be a cloud server or an edge server.

[0050] The collection unit 111 collects verbal information and non-verbal information of multiple users in a predetermined space. The collection unit 111 collects, as verbal information, voice data of the users acquired from the microphone device 20. The collection unit 111 acquires, as verbal information, linguistic information such as the content and amount of speech obtained by analyzing the voice data.

[0051] The collection unit 111 applies a voice recognition process to the user voice data acquired from the microphone device 20 to acquire text data indicating the speech content of each of the multiple users. The collection unit 111 detects the speech start time and speech end time for each of the multiple users from the text data, and detects the speech time from the speech start time to the speech end time as the speech volume. Hereinafter, data including the text data indicating the speech content of each of the multiple users acquired from the voice data and the speech volume of each of the multiple users will be referred to as speech data.

[0052] The collection unit 111 collects, as nonverbal information, image data acquired from the image capturing device 30. The collection unit 111 collects, as nonverbal information, nonverbal information including at least one of the user's facial direction and line of sight obtained by analyzing the image data.

[0053] The collection unit 111 may use the technique of Japanese Patent No. 4907483 to detect the facial orientation and line of sight of the user contained in the image data from the image data.

[0054] Specifically, the collection unit 111 acquires a plurality of template images, each of which is cut out from the surrounding areas of facial feature points such as the eyes, nose, and mouth, prepared in advance for each facial orientation. Next, the collection unit 111 calculates a correlation value between each of the plurality of template images and the user's facial image captured by the image capture device 30, and calculates the weighted sum of the correlation values ​​as the facial orientation. Next, the collection unit 111 uses a face detection module and a face part detection module to detect the three-dimensional coordinates of the inner corners of the left and right eyes for two images captured by the stereo camera constituting the image capture device 30, and detects the perpendicular bisector of a line segment having the three-dimensional positions of the inner corners of the left and right eyes as the gaze direction reference plane. Next, the collection unit 111 detects the area with the lowest brightness from the eye area, including the outer and inner corners of the eyes, as the iris area. Next, the collection unit 111 sets a iris detection filter on the iris region and searches for the center of a circle that maximizes the variance of pixel brightness between a circular first region defined at the center of the iris detection filter and a donut-shaped second region surrounding the first region. Next, the collection unit 111 uses a stereo method to detect this circle center as the three-dimensional position of the iris center. Next, the collection unit 111 detects the gaze direction using the three-dimensional position of the iris center and the gaze direction reference plane. In this way, the facial orientation and gaze are detected three-dimensionally.

[0055] The collection unit 111 may acquire the user's schedule information stored in the memory 13 as nonverbal information.

[0056] The detection unit 112 detects, based on the nonverbal information collected by the collection unit 111, each of two or more users who are within a predetermined distance from the plurality of users as a candidate collaboration user.

[0057] 2 is a diagram showing an example of a space 200 to which the information processing system 1 is applied. The space 200 is, for example, a room in an office such as a coworking space or a shared office. There are six users U1 to U6 in the space 200. The detection unit 112 detects the users U1 to U6 from image data captured by the image capture device 30. The detection unit 112 may detect the users U1 to U6 by, for example, inputting the image data into a person detector, or may detect the users U1 to U6 by pattern matching.

[0058] The detection unit 112 calculates the distance between the users U1 to U6 detected from the image data acquired by the collection unit 111.

[0059] The detection unit 112 clusters users who are within a predetermined distance DR from the detected users U1 to U6. Users U1, U2, and U3 are clustered into one group G1 because their mutual distance D1 is less than or equal to the predetermined distance DR. Users U4 and U5 are also clustered into one group G2 because their mutual distance D1 is also less than or equal to the predetermined distance DR. The detection unit 112 detects each of the multiple users clustered into one of the groups as a candidate collaborating user.

[0060] On the other hand, the user U6 is not detected as a candidate collaborating user because the distance D2 between the user U6 and each of the users U1 to U5 is greater than the predetermined distance DR.

[0061] Returning to FIG. 1 , the determination unit 113 determines the degree of collaboration among multiple users for a group including a candidate collaboration user detected by the detection unit 112, based on at least one of the verbal information and non-verbal information collected by the detection unit 112. When multiple groups are detected by the detection unit 112, the determination unit 113 may determine the degree of collaboration for each of the multiple groups. The degree of collaboration is a numerical value indicating the degree to which collaborative users in a group are cooperating on a task. For example, a value normalized within a predetermined range (0 to 1) is used as the degree of collaboration.

[0062] The determination unit 113 detects the synchronicity of the actions of the candidate collaboration users based on at least one of the collected verbal information and nonverbal information, and determines the degree of collaboration based on the detected synchronicity of the actions. The synchronicity of the actions refers to the degree of interaction between the candidate collaboration users. Specifically, the determination unit 113 calculates the degree of collaboration based on the synchronicity using one of the first to third synchronization determination methods described below.

[0063] (First Synchronization Determination Method) The first synchronization determination method is a method for determining the degree of collaboration based on at least one of the facial orientation and the line of sight of the collaboration candidate user collected by the collection unit 111 .

[0064] The degree of collaboration is set to a larger value the longer the face-to-face time during which at least one of the first collaboration candidate user's facial direction and gaze is directed toward a second collaboration candidate user different from the first collaboration candidate user. The face-to-face time is the time during which a vector indicating the first collaboration candidate user's facial direction or gaze intersects with a predetermined area including the face of the second collaboration candidate user. The predetermined area is an area with a predetermined margin around the face area. The first collaboration candidate user refers to one collaboration candidate user in a certain group. The second collaboration candidate user refers to a collaboration candidate user in the group who is different from the first collaboration candidate user. The determination unit 113 uses image processing to detect a predetermined area including the face of each collaboration candidate user using a stereo method from two images captured by the stereo camera constituting the image capture device 30. This predetermined area is a three-dimensional area, for example, having a rectangular parallelepiped shape.

[0065] The determination unit 113 aggregates the face-to-face time for each of multiple candidate collaboration users in the same group for each unit time, and calculates the face-to-face time for the group. The determination unit 113 calculates the frequency of face-to-face time by dividing the face-to-face time for the group by the unit time, and calculates this frequency as the degree of collaboration for each group. The unit time can be any appropriate value, such as 1 minute, 3 minutes, 5 minutes, or 10 minutes. The determination unit 113 calculates the degree of collaboration in real time by repeating the process of calculating the frequency of face-to-face time for each group for each unit time.

[0066] For example, if two candidate users are in the same group and the unit time is 10 minutes, the face-to-face time between the two candidate users is calculated. If the face-to-face time is 5 minutes, the frequency of face-to-face time is 5 / 10 = 0.5, and the degree of collaboration is calculated as 0.5.

[0067] In this case, the frequency F1 of the face-to-face time when only one of the two candidate users faces the other and the frequency F2 of the face-to-face time when both users face each other are calculated, and the frequencies F1 and F2 are weighted and averaged so that the weighted value of frequency F2 is greater than that of frequency F1, and the average value of the obtained frequencies is calculated as the degree of collaboration.

[0068] When there are four candidate collaboration users in the same group, if the face-to-face time Ta of a first pair consisting of two candidate collaboration users and the face-to-face time Tb of a second pair consisting of two other candidate collaboration users overlap at least partially, the determination unit 113 need only count the face-to-face time of one of the first pair and the second pair for the overlapping period.

[0069] (Second Synchronization Determination Method) The second synchronization determination method is a method for determining the degree of collaboration based on the amount of speech. The amount of speech refers to the amount of time a collaboration candidate user speaks. The greater the amount of speech, the greater the value set for the degree of collaboration. The determination unit 113 aggregates the speech time of each collaboration candidate user for each unit time from the speech data collected by the collection unit 111, divides the aggregated speech time by the unit time to calculate the frequency of the group's speech time, and calculates the calculated frequency of the speech time as the degree of collaboration. The determination unit 113 calculates the degree of collaboration in real time by repeating the process of calculating the frequency of the speech time for each group for each unit time.

[0070] 2, in a certain unit period, the speech times of users U1, U2, and U3 belonging to group G1 are t1, t2, and t3, respectively, and if the unit period is 10 minutes, the degree of cooperation is calculated as (t1 + t2 + t3) / 10. When the speech times t1, t2, and t3 overlap, the determination unit 113 only needs to count one of the speech times.

[0071] The determination unit 113 may identify a period in which the collaboration candidate user is talking to himself / herself based on the amount of speech and at least one of the facial direction and the line of sight, correct the amount of speech so as to remove the period, and determine the degree of collaboration based on the corrected amount of speech. The determination unit 113 may determine that the first collaboration candidate user is talking to himself / herself if at least one of the facial direction and the line of sight of the first collaboration candidate user is not directed toward any collaboration candidate user in the same group during the time when the first collaboration candidate user is speaking. For example, the determination unit 113 may determine that the first collaboration candidate user is not directed toward any collaboration candidate user if a vector indicating at least one of the facial direction and the line of sight of the first collaboration candidate user does not intersect with a predetermined area including the faces of any collaboration candidate users. Alternatively, the determination unit 113 may determine that the first collaboration candidate user is not facing any of the collaboration candidate users if the facial direction and line of sight of the first collaboration candidate user are directed toward the floor, walls, and ceiling of the space 200.

[0072] (Third Synchronization Determination Method) The third synchronization determination method is a method for determining the degree of collaboration when both the schedule information of a first collaboration candidate user and the schedule information of a second collaboration candidate user in the same group indicate a collaborative task, compared to when at least one of the schedule information of the first collaboration candidate user and the schedule information of the second collaboration candidate user does not indicate a collaborative task. The collaborative task may be, for example, an online meeting, an offline meeting, etc.

[0073] For example, if the schedule information of one collaboration candidate user indicates a collaboration task in a certain unit time, the collaboration score of the first collaboration candidate user is set to 1 point. When two collaboration candidate users are in the same group, the collaboration score combinations include a combination in which the schedule information of both collaboration candidate users indicates a collaboration task, a combination in which only the schedule information of one collaboration candidate user or the other collaboration candidate user indicates a collaboration task, and a combination in which neither collaboration candidate user's schedule information indicates a collaboration task. In this case, the collaboration score of the first combination is expressed as [1, 1], the collaboration score of the second combination is expressed as [1, 0] or [0, 1], and the collaboration score of the third combination is expressed as [0, 0]. If the collaboration score for a unit time is [1, 1], the determination unit 113 calculates the degree of collaboration for that unit time as (1 + 1) / 2 = 1.0. If the collaboration score for a unit time is [1, 0] or [0, 1], the determination unit 113 calculates the degree of collaboration for that unit time as (1 + 0) / 2 = 0.5. If the collaboration score for a unit time is [0, 0], the determination unit 113 calculates the degree of collaboration for that unit time as (0 + 0) / 2 = 0. The determination unit 113 may calculate the degree of collaboration in real time by repeating the process of calculating the frequency of speech time for each group for each unit time. Note that when there are n or more candidate collaboration users (n is an integer greater than or equal to 3), the collaboration score for each combination may be calculated in the same way as when there are two users, and the calculated collaboration score may be divided by n to calculate the degree of collaboration.

[0074] The adjustment unit 114 outputs a control signal to the adjustment device for controlling the space where the plurality of users are present to a state according to the determined degree of cooperation. The adjustment device includes a speaker device 40 and a lighting device 50.

[0075] When the degree of collaboration determined by the determination unit 113 exceeds the threshold, the adjustment unit 114 outputs a control signal to the lighting device 50 to collectively illuminate all of the collaboration candidate users. On the other hand, when the degree of collaboration determined by the determination unit 113 does not exceed the threshold, the adjustment unit 114 outputs a control signal to the lighting device 50 to individually illuminate the collaboration candidate users. For example, 0.5 is used as the threshold. However, this is just an example, and an appropriate value other than 0.5 may be used as the threshold.

[0076] 2 , if the degree of cooperation of group G1 exceeds the threshold, adjustment unit 114 sets a first spot area for collectively illuminating users U1, U2, and U3, and outputs a control signal indicating the set first spot area to lighting device 50. On the other hand, if the degree of cooperation of group G1 does not exceed the threshold, adjustment unit 114 sets three second spot areas for individually illuminating users U1, U2, and U3, and outputs a control signal specifying the three second spot areas to lighting device 50.

[0077] The adjustment unit 114 applies a stereo method to the image data collected by the collection unit 111 to calculate three-dimensional regions of users U1, U2, and U3. The adjustment unit 114 may calculate a first spot area and a second spot area from the three-dimensional regions of users U1, U2, and U3. For example, the adjustment unit 114 may determine the three-dimensional positions of the centers of gravity of users U1, U2, and U3 from the three-dimensional regions of each of users U1, U2, and U3, and calculate as the first spot area a region that circumscribes all of users U1, U2, and U3 around the center of gravity and is parallel to the floor, or a region that includes a margin. For example, the adjustment unit 114 may determine the three-dimensional positions of the centers of gravity of users U1, U2, and U3, and calculate as the second spot area three regions that circumscribe each of users U1, U2, and U3 around the center of gravity and are parallel to the floor, or three regions that include a margin for each of the three regions.

[0078] When the degree of collaboration exceeds the threshold, the adjustment unit 114 may output a control signal to the speaker device 40 to output a sound directed at all of the collaboration candidate users. On the other hand, when the degree of collaboration does not exceed the threshold, the adjustment unit 114 may output a control signal to the speaker device 40 to output a sound directed at each of the collaboration candidate users.

[0079] When the degree of collaboration exceeds a threshold, the adjustment unit 114 calculates a first spot area that includes the entire users U1, U2, and U3 from the three-dimensional regions of each of the users U1, U2, and U3, and outputs a control signal indicating the first spot area to the speaker device 40.

[0080] On the other hand, if the degree of collaboration does not exceed the threshold, the adjustment unit 114 calculates three second spot areas corresponding to users U1, U2, and U3 from the three-dimensional regions of each of users U1, U2, and U3, and outputs a control signal to the speaker device 40 to play sound at a predetermined volume in each of the three second spot areas.

[0081] The communication device 12 outputs the control signal generated by the adjustment unit 114 to the speaker device 40 and the lighting device 50. The communication device 12 receives audio data transmitted from the microphone device 20. The communication device 12 receives image data transmitted from the imaging device 30.

[0082] The memory 13 is configured by a non-volatile rewritable storage device such as a solid state drive or a hard disk drive. The memory 13 stores schedule information for each of multiple users. The memory 13 also stores a threshold value used by the determination unit 113 when determining the degree of collaboration.

[0083] The microphone device 20 includes one or more microphones arranged in the space 200, picks up the voices of users in the space 200, and transmits voice data indicating the picked-up voices to the server 10. The microphone device 20 may transmit the voice data to the server 10 at a predetermined sampling period. The microphone device 20 includes a communication circuit and a processor. The communication circuit transmits the voice data to the server 10. The processor converts the voices picked up by the microphones into voice data.

[0084] The image capturing device 30 is placed in the space, captures images of users in the space 200, and transmits image data including the captured images of the users to the server 10. The image capturing device 30 may transmit image data to the server 10 at a predetermined sampling period. The image capturing device 30 may be configured as a stereo camera made up of multiple cameras. Specifically, the image capturing device 30 includes multiple cameras, a communication circuit, and a processor. The multiple cameras may be installed over the entire ceiling of the space 200 so that the entire space 200 can be captured. Alternatively, the image capturing device 30 may include one or more omnidirectional cameras. The communication circuit transmits image data captured by the multiple cameras to the server 10. The processor is responsible for overall control of the image capturing device 30.

[0085] The speaker device 40 illuminates the space 200 in accordance with a control signal transmitted from the server 10. The speaker device 40 includes an array speaker, a communication circuit, a processor, and a memory. The array speaker includes a plurality of speaker elements installed in an array across the entire area of ​​the space 200, for example, on the ceiling. The communication circuit receives a control signal from the server 10. The processor controls the phase and volume of the sound output from each speaker element constituting the array speaker in accordance with the control signal received by the communication circuit, and outputs sound directed toward the first spot area or the second spot area. The processor controls the phase and volume of the sound output from each speaker element so that the sound is output to the first spot area or the second spot area specified by the control signal received by the communication circuit.

[0086] The memory stores sound data for reproducing sounds to be output to the space 200. The sound data is, for example, background music. However, this is just one example, and the sound data may represent sounds of the natural environment, such as the sound of a river, the sound of a waterfall, or the singing of birds. In accordance with a selection instruction from the user, the processor may output the sound of sound data selected by the user from among these sound data from the array speaker.

[0087] The lighting device 50 illuminates the space 200 in accordance with a control signal transmitted from the server 10. The lighting device 50 includes a plurality of light sources, a communication circuit, and a processor. The plurality of light sources are installed in an array across the entire ceiling of the space 200. Each light source is movable, and the direction, light distribution, and light intensity of the spot light can be adjusted. The communication circuit receives the control signal transmitted from the server 10. The processor controls the direction, light distribution, and light intensity of each light source so that the spot light is irradiated onto the first spot area or the second spot area specified by the control signal received by the communication circuit. For example, the processor determines which light source to irradiate, in what direction, and with what light intensity, depending on the area indicated by the first spot area or the area indicated by the second spot area, and controls each light source in accordance with the determination result.

[0088] 3 is a flowchart showing the processing of the information processing system 1 according to the present embodiment. In step S1, the collection unit 111 acquires verbal information by analyzing the voice data transmitted from the microphone device 20. This verbal information includes voice data and the above-mentioned speech data. The speech data includes text data indicating the amount of speech and the content of the speech.

[0089] Next, in step S2, the collection unit 111 acquires nonverbal information from the image data transmitted from the image capture device 30. This nonverbal information includes the image data and at least one of the user's facial orientation and line of sight. The collection unit 111 may acquire schedule information read from the memory 13 as nonverbal information.

[0090] Next, in step S3, the detection unit 112 calculates the distance between the users included in the image data by analyzing the image data acquired by the collection unit 111. The detection unit 112 calculates the three-dimensional position of each user using a stereo method and calculates the distance between the users using this three-dimensional position. In the example of FIG. 2, there are six users U1 to U6, so the distance is calculated for every combination of selecting two users from these six users. Note that the three-dimensional positions of users U1 to U6 can be, for example, predetermined positions of users U1 to U6 (e.g., the center of the face, the center of the torso, the center of the entire body, etc.).

[0091] Next, in step S4, the detection unit 112 clusters the detected users who are within a predetermined distance DR into one or more groups, and detects each of the multiple users belonging to one of the groups as a candidate collaborative user.

[0092] Next, in step S5, the determination unit 113 calculates the degree of cooperation using any one of the first to third synchronization determination methods. If multiple groups are generated by clustering, the degree of cooperation is calculated for each of the multiple groups.

[0093] Next, in step S7, the determination unit 113 determines for each group whether the degree of cooperation is greater than a threshold. For groups whose degree of cooperation is greater than the threshold (YES in step S7), the process proceeds to step S8. On the other hand, for groups whose degree of cooperation is equal to or less than the threshold (NO in step S7), the process proceeds to step S9.

[0094] Next, in step S8, the adjustment unit 114 applies overall control to groups whose degree of collaboration is greater than a threshold. Overall control is control to irradiate spotlight or output sound to all of the candidate collaboration users belonging to the group. In this case, the adjustment unit 114 calculates a first spot area from the image data and outputs a control signal to the speaker device 40 or the lighting device 50 to output sound or spotlight to the first spot area.

[0095] Next, in step S9, the adjustment unit 114 applies individual control to groups whose degree of collaboration is equal to or less than a threshold. Individual control is control to irradiate spotlights or output sounds to each of multiple collaboration candidate users belonging to the group. In this case, the adjustment unit 114 calculates multiple second spot areas corresponding to the multiple collaboration candidate users from the image data, and outputs a control signal to the speaker device 40 or the lighting device 50 to output sounds or spotlights to each of the multiple second spot areas.

[0096] Fig. 4 is a graph showing an example of the change in the degree of cooperation over time. In Fig. 4, the vertical axis represents the degree of cooperation, and the horizontal axis represents time. Since the degree of cooperation is calculated for each unit time T, the graph in Fig. 4 shows that the degree of cooperation changes stepwise for each unit time T. Fig. 4 shows the degree of cooperation of a certain group.

[0097] From unit time T1 to unit time T3, the degree of collaboration is below the threshold (=0.5), so individual control is applied to this group. This allows each candidate collaboration user to concentrate on their work. In unit time T4, the degree of collaboration is greater than the threshold, so overall control is applied to this group. Thereafter, overall control is applied to this group from unit time T5 to T10. This creates a sense of unity among all candidate collaboration users and promotes the work of multiple candidate collaboration users. In unit time T11, the degree of collaboration is below the threshold, so individual control is again applied to this group. Thereafter, individual control is applied to this group until unit time T12. In this way, the degree of collaboration is calculated for each unit time T, so it is possible to switch between individual control and overall control in real time.

[0098] As described above, according to this embodiment, the degree of collaboration for multiple users is determined based on at least one of the collected verbal information and nonverbal information, and a control signal for controlling the space where the multiple users are present to a state corresponding to the degree of collaboration is output to the adjustment device. Therefore, the state of the space can be made appropriate for each user according to the degree of collaboration. As a result, the user's work can be promoted. Because the degree of collaboration is determined based on at least one of the verbal information and nonverbal information, the degree of collaboration can be determined without requiring users to input details of their activities in the space via an input screen, as in conventional technology. This eliminates processes such as displaying an input screen for activity details and accepting instructions on activity details input by users via the input screen, thereby reducing the processing load on the computer.

[0099] The present disclosure can employ the following modifications.

[0100] (1) The adjustment unit 114 may control both the speaker device 40 and the lighting device 50 simultaneously, or may control only one of them.

[0101] (2) In the above embodiment, any one of the first to third synchronization determination methods is described as being adopted, but any two or all of these methods may be adopted. In this case, the determination unit 113 calculates the average value of the degrees of cooperation calculated by each of the first to third synchronization determination methods as the final degree of cooperation, compares this final degree of cooperation with a threshold, and determines whether to apply overall control or individual control.

[0102] (3) The space 200 may be a room in a general house, such as a living room, a dining room, etc. The space 200 may also be a room in a public facility, such as a library.

[0103] (4) In the first synchronization determination method, the determination unit 113 may calculate the degree of cooperation based on the face-to-face time between the collaboration candidate users and the time the collaboration candidate users are facing the same object. For example, the determination unit 113 may calculate the degree of cooperation by tallying the time the first collaboration candidate user and the second collaboration candidate user are facing the same television device for each unit time and dividing this time by the unit time. The determination unit 113 may calculate the degree of cooperation by counting the time during which at least one of the facial direction and the line of sight of the first collaboration candidate user is facing an object and at least one of the facial direction and the line of sight of the second collaboration candidate user is facing the object.

[0104] (5) The adjustment unit 114 may apply individual control to the user U6 shown in FIG.

[0105] The present disclosure is useful for improving an environment in which multiple users work collaboratively.

Claims

1. An information processing method for a computer, comprising: collecting verbal and non-verbal information of a plurality of users in a specified space; determining a degree of collaboration among the plurality of users based on at least one of the collected verbal and non-verbal information; and outputting a control signal to an adjustment device to control the space in which the plurality of users are located to a state according to the determined degree of collaboration.

2. The information processing method of claim 1, further comprising: detecting two or more users among the plurality of users who are within a predetermined distance as potential collaboration users based on the collected nonverbal information; and determining the degree of collaboration for a group including the detected potential collaboration users based on the collected nonverbal information.

3. The information processing method of claim 2, wherein the determination of the degree of collaboration is performed by detecting the synchronicity of the actions of the potential collaboration users based on at least one of the collected verbal information and non-verbal information, and determining the degree of collaboration based on the detected synchronicity of the actions.

4. An information processing method according to claim 2 or 3, wherein the non-verbal information includes at least one of the facial direction and gaze of the candidate collaboration user, and the degree of collaboration is determined based on at least one of the facial direction and gaze of the candidate collaboration user.

5. The information processing method according to claim 2 or 3, wherein the verbal information includes an amount of speech of the candidate collaboration user, and the degree of collaboration is determined based on the amount of speech.

6. The information processing method of claim 5, wherein the nonverbal information includes at least one of the facial direction and gaze of the candidate collaboration user, and further, a period in which the candidate collaboration user is talking to himself is identified based on the amount of speech and at least one of the facial direction and gaze, and the amount of speech is corrected so as to remove that period, and the degree of collaboration is determined based on the corrected amount of speech.

7. The information processing method according to claim 2 or 3, wherein the nonverbal information includes schedule information of the candidate collaboration user, and the degree of collaboration is determined based on the schedule information.

8. An information processing method as described in claim 2 or 3, wherein the adjustment device is a lighting device, and the output includes: outputting the control signal to the lighting device to illuminate all of the candidate collaboration users as a whole when the determined degree of collaboration exceeds a threshold; and outputting the control signal to the lighting device to illuminate the candidate collaboration users individually when the determined degree of collaboration does not exceed a threshold.

9. An information processing method as described in claim 2 or 3, wherein the adjustment device is a speaker device, and the output includes: outputting to the speaker device the control signal that outputs sound directed at all of the candidate collaboration users when the degree of collaboration exceeds a threshold; and outputting to the speaker device the control signal that outputs sound directed at each of the candidate collaboration users when the degree of collaboration does not exceed a threshold.

10. The information processing method of claim 4, wherein the degree of collaboration is set to a larger value the longer the face-to-face time during which at least one of the facial direction and the gaze of the first collaboration candidate user is directed toward a second collaboration candidate user different from the first collaboration candidate user, and the face-to-face time is the time during which a vector indicating the facial direction or the gaze of the first collaboration candidate user intersects with a specified area including the face of the second collaboration candidate user.

11. The information processing method according to claim 5, wherein the degree of collaboration is set to a larger value as the amount of speech increases, and the amount of speech is the amount of time the candidate user speaks.

12. The information processing method of claim 7, wherein the degree of collaboration is set to a larger value when the schedule information of a first collaboration candidate user and the schedule information of a second collaboration candidate user different from the first collaboration candidate user both indicate a collaborative task compared to when at least one of the schedule information of the first collaboration candidate user and the schedule information of the second collaboration candidate user does not indicate the collaborative task.

13. An information processing method according to claim 1 or 2, wherein the verbal information includes speech data of each of the multiple users obtained by analyzing voice data of the multiple users picked up by a microphone, and the non-verbal information includes at least one of facial direction and gaze of each of the multiple users obtained by analyzing image data of the multiple users captured by a photographing device.

14. The information processing method according to claim 2, wherein, when the group includes a plurality of groups, the degree of cooperation is determined for each of the plurality of groups, and the space is controlled for each of the plurality of groups.

15. An information processing device comprising: a collection unit that collects verbal information and non-verbal information of multiple users within a specified space; a determination unit that determines the degree of collaboration of each of the multiple users based on at least one of the collected verbal information and non-verbal information; and an adjustment unit that outputs a control signal to an adjustment device to control the space in which the multiple users are located to a state corresponding to the determined degree of collaboration.

16. An information processing program that causes a computer to execute the following process: collect verbal information and nonverbal information of multiple users within a specified space; determine the degree of collaboration of each of the multiple users based on at least one of the collected verbal information and nonverbal information; and output a control signal to an adjustment device to control the space in which the multiple users are located to a state corresponding to the determined degree of collaboration.

Images