Focus determination system, communication analysis system, and focus determination method

The focus determination system uses video analysis to assess communication focus by detecting head orientations and positions, effectively determining the extent to which individuals are engaged in group interactions.

JP7595269B2Active Publication Date: 2024-12-06PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2023531788
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-06-28
Filing Date
2022-06-16
Publication Date
2024-12-06
Estimated Expiration
2042-06-16

AI Technical Summary

Technical Problem

Existing technologies lack the ability to effectively determine the extent to which a target person is focusing on communication in a group setting, such as a meeting or collaborative workspace.

Method used

A focus determination system that utilizes video information to detect the position and orientation of individuals' heads, determining periods when a target person is facing other participants, displays, or whiteboards, thereby assessing their focus on communication.

Benefits of technology

The system can accurately determine the extent to which a target person is focusing on communication, enabling analysis of communication quality among multiple individuals.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007595269000004
    Figure 0007595269000004
  • Figure 0007595269000005
    Figure 0007595269000005
  • Figure 0007595269000006
    Figure 0007595269000006
Patent Text Reader

Abstract

A focus determining system (39) comprises: a person detecting unit (32) for detecting first positions, which are the positions of a plurality of persons in a space (70), on the basis of video information; a head orientation detecting unit (33) for detecting the orientation of the head of a subject among the plurality of persons, on the basis of the video information; and a focus determining unit (35) for determining, on the basis of the detected first positions and the detected orientation of the head of the subject, that when a person among the plurality of persons, other than the subject, is speaking, a period in which the detected orientation of the head of the subject is facing any of a person among the plurality of persons, other than the subject, a display (50), or a whiteboard (60), is a period during which the subject is focusing on communication.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an attention determination system, a communication analysis system, and an attention determination method. [Background technology]

[0002] In an organization such as a company, it is important for employees to communicate closely with each other and work on their own tasks. As a technology related to such communication, Patent Document 1 discloses an information providing device that provides information on the psychological state of a subject in order to evaluate intellectual activities in a place such as a meeting or a conference. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2004-112518 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention provides an attention determination system and the like that can determine the degree to which a subject is focusing on communication. [Means for solving the problem]

[0005] An attention determination system according to one embodiment of the present invention includes a person detection unit that acquires video information of an image showing a plurality of people staying in a space in which a display and a whiteboard are installed, and detects a first position which is the position of the plurality of people in the space based on the acquired video information; a head orientation detection unit that acquires the video information and detects the head direction of a target person among the plurality of people based on the acquired video information; and an attention determination unit that determines the attention of the target person to a communication conducted by the plurality of people in the space based on the detected first position and the detected head direction of the target person, and the attention determination unit determines that a period during which a person among the plurality of people other than the target person is speaking is a period during which the detected head direction of the target person is directed toward any of the person among the plurality of people other than the target person, the display, or the whiteboard, is a period during which the target person is focusing on the communication.

[0006] A communication analysis system according to one embodiment of the present invention includes the attention determination system and a communication analysis unit that analyzes the quality of communication conducted by the multiple people in the space based on the determined attention of the subject.

[0007] An attention determination method according to one embodiment of the present invention includes a first detection step of acquiring video information of a video showing a plurality of people staying in a space in which a display and a whiteboard are installed, and detecting a first position which is the position of the plurality of people in the space based on the acquired video information; a second detection step of acquiring the video information and detecting a head direction of a target person among the plurality of people based on the acquired video information; and an attention determination step of determining the focus of the target person on communication conducted by the plurality of people in the space based on the detected first position and the detected head direction of the target person, wherein in the attention determination step, when a person among the plurality of people other than the target person is speaking, a period during which the detected head direction of the target person is facing any of the person among the plurality of people other than the target person, the display, or the whiteboard is determined to be a period during which the person is focusing on the communication.

[0008] A program according to one aspect of the present invention is a program for causing a computer to execute the focus determination method. Effect of the Invention

[0009] The focus determination system etc. of the present invention can determine the degree to which a subject is focusing on communication. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing a functional configuration of a communication analysis system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a space to which the communication analysis system according to the embodiment is applied. [Diagram 3] FIG. 3 is an external view of the sensing device according to the embodiment. [Figure 4] FIG. 4 is a flowchart of the operation of the sound source detection unit included in the communication analysis system according to the embodiment. [Diagram 5] FIG. 5 is a diagram showing a schematic diagram of the detection result of the position of a sound source. [Figure 6] FIG. 6 is a flowchart of the operation of the human detection unit included in the communication analysis system according to the embodiment. [Figure 7] FIG. 7 is a diagram showing a schematic diagram of a detection result of a person's position. [Figure 8] FIG. 8 is a flowchart of the operation of the speech amount estimation unit included in the communication analysis system according to the embodiment. [Figure 9] FIG. 9 is a diagram showing a schematic diagram of a detection result of a speaker's position. [Figure 10] FIG. 10 is a diagram illustrating an example of information indicating the amount of speech. [Figure 11] FIG. 11 is a flowchart of the operation of the head direction detection unit included in the communication analysis system according to the embodiment. [Figure 12] FIG. 12 is a diagram showing a three-dimensional coordinate space for explaining the operation of the head direction detection unit included in the communication analysis system according to the embodiment. [Figure 13] FIG. 13 is a diagram for explaining correction of the angle of the head direction. [Figure 14] FIG. 14 is a flowchart of the operation of the focus determination unit included in the communication analysis system according to the embodiment. [Figure 15] FIG. 15 is a diagram for explaining the target direction of the subject. [Figure 16] FIG. 16 is a diagram illustrating an example of information indicating a focus period. [Figure 17] FIG. 17 is a flowchart of the operation of the communication analysis unit included in the communication analysis system according to the embodiment. [Figure 18] FIG. 18 is a diagram showing an example of a display screen of a score indicating the quality of communication, which is displayed on an information terminal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component arrangement and connection forms, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present invention. In addition, among the components in the following embodiments, components that are not described in the independent claims will be described as optional components.

[0012] In addition, each drawing is a schematic diagram and is not necessarily a precise illustration. In each drawing, the same reference numerals are used for substantially the same configurations, and duplicated explanations may be omitted or simplified.

[0013] (Embodiment) [composition] First, a configuration of a communication analysis system according to an embodiment will be described. Fig. 1 is a block diagram showing a functional configuration of the communication analysis system according to an embodiment. Fig. 2 is a diagram showing an example of a space to which the communication analysis system is applied.

[0014] As shown in FIG. 1 and FIG. 2, the communication analysis system 10 is a system for analyzing the quality of communication between multiple people located in a space 70, such as a conference room, used in an office having the space 70. The space 70 is, for example, a closed space, but may be an open space. In addition to a conference room, an open rest area in an office space (a part of the office space where chairs and tables are placed) is exemplified as the space 70. In addition, the space 70 does not need to be physically separated, and may be a place in the entire space separated by lighting or air currents. For example, a warm color area with a color temperature of 3000K may be provided in a corner of an office space illuminated with daylight color with a color temperature of 5000K, and this area may be the space 70.

[0015] The communication analysis system 10 includes a sensing device 20 and an information processing system 30. First, the sensing device 20 will be described with reference to FIG. 3 in addition to FIG. 1 and FIG. 2. FIG. 3 is an external view of the sensing device 20. FIG. 3(a) is a top view of the sensing device 20, and FIG. 3(b) is a side view of the sensing device 20. Note that the distance measurement sensor 23 is not shown in FIG. 3.

[0016] The sensing device 20 is installed on a desk 40 placed in a space 70, and senses sounds, images, and the like in the space 70. Specifically, the sensing device 20 is installed in the center of the desk 40. The sensing device 20 includes a microphone array 21, a plurality of cameras 22, and a distance measurement sensor 23.

[0017] The microphone array 21 acquires sounds in the space 70 and outputs sound information (a plurality of sound signals) of the acquired sounds. Specifically, the microphone array 21 includes a plurality of microphone elements, each of which acquires sounds in the space 70 and outputs a sound signal of the acquired sound.

[0018] Each of the multiple cameras 22 captures an image (in other words, a moving image) of people staying in the space 70, and outputs video information of the image. The camera 22 is a general camera realized by a CMOS image sensor or the like, but may be a fisheye camera or the like. The sensing device 20 includes four cameras so that it can capture the entire area around the sensing device 20 from above the desk 40, but it is sufficient to include at least one camera that can capture the image of all people staying in the space 70.

[0019] The distance measurement sensor 23 measures the distance from the sensing device 20 (camera 22) to an object, and outputs distance information indicating the measured distance to the object. The object is a person staying in the space 70, for example. The distance measurement sensor 23 is, for example, a Time Of Flight (TOF) type LiDAR (Light Detection and Ranging), but may also be a distance image sensor, etc. The sensing device 20 is required to include at least one distance measurement sensor 23, but may also include multiple distance measurement sensors 23 corresponding to the cameras 22.

[0020] Next, the information processing system 30 will be described. The information processing system 30 performs wired or wireless communication with the sensing device 20, and analyzes the quality of communication based on sensing information (specifically, sound information, video information, distance information, etc.) acquired from the sensing device 20 through the communication. The information processing system 30 is, for example, an edge computer installed in a facility having a space 70, but may also be a cloud computer installed outside the facility. When the information processing system 30 is an edge computer, the sensing device 20 and the information processing system 30 may be realized as one integrated device. Also, some functions of the information processing system 30 may be realized as an edge computer, and other functions may be realized by a cloud computer.

[0021] Specifically, the information processing system 30 includes a sound source detection unit 31, a human detection unit 32, a head direction detection unit 33, a speech amount estimation unit 34, an attention determination unit 35, a communication analysis unit 36, and a memory unit 37.

[0022] The sound source detection unit 31 acquires sound information of a sound captured in the space 70 from the sensing device 20, and detects a second position, which is the position of the sound source in the space 70, based on the acquired sound information.

[0023] The human detection unit 32 acquires image information of an image showing a person staying in the space 70 from the sensing device 20, and detects a first position, which is the position of the person in the space 70, based on the acquired image information.

[0024] The head direction detection unit 33 acquires image information of an image showing a person staying in the space 70 from the sensing device 20, and detects the direction of the person's head (in other words, the direction of the face) based on the acquired image information. The head direction detection unit 33 may detect the direction of the person's line of sight based on the acquired image information.

[0025] The speech volume estimation unit 34 estimates the human speech volume based on the first position detected by the human detection unit 32 and the second position detected by the sound source detection unit 31 .

[0026] The attention determination unit 35 determines a person's attention to communication taking place in space 70 among multiple people including the person, based on the first position detected by the person detection unit 32 and the direction of the person's head detected by the head direction detection unit 33.

[0027] The communication analysis unit 36 ​​analyzes the quality of communication based on at least one of the person's speech volume estimated by the speech volume estimation unit 34 and the person's attention determined by the attention determination unit 35. In addition, the communication analysis unit 36 ​​outputs the analysis result.

[0028] The above-described sound source detection unit 31, human detection unit 32, head direction detection unit 33, speech amount estimation unit 34, attention determination unit 35, and communication analysis unit 36 ​​are each realized by a microcomputer or a processor. The functions of the sound source detection unit 31, human detection unit 32, head direction detection unit 33, speech amount estimation unit 34, attention determination unit 35, and communication analysis unit 36 ​​are realized, for example, by the microcomputer or processor executing a computer program stored in the storage unit 37.

[0029] The storage unit 37 is a storage device that stores the computer programs and information necessary to realize the functions of the components described above. The storage unit 37 is realized by, for example, a HDD (Hard Disk Drive), but may also be realized by a semiconductor memory.

[0030] In addition, the system including the sound source detection unit 31, the human detection unit 32, and the speech amount estimation unit 34 is also described as a speaker diarization system 38. That is, the speaker diarization system 38 includes the sound source detection unit 31, the human detection unit 32, and the speech amount estimation unit 34. The speaker diarization system 38 may further include a head direction detection unit 33 or an attention determination unit 35.

[0031] In addition, a system including the human detection unit 32, the head direction detection unit 33, and the attention determination unit 35 is also described as an attention determination system 39. That is, the attention determination system 39 includes the human detection unit 32, the head direction detection unit 33, and the attention determination unit 35. The attention determination system 39 may further include the sound source detection unit 31 or the speech amount estimation unit 34.

[0032] [Operation of sound source detection unit] Next, a more detailed description will be given of the operation of the sound source detection unit 31. FIG.

[0033] First, the sound source detection unit 31 acquires sound information of a sound captured in the space 70 from the microphone array 21 of the sensing device 20 (S11). Specifically, the sound source detection unit 31 acquires a plurality of sound signals output by a plurality of microphone elements included in the microphone array 21.

[0034] Here, each of the acquired sound signals is a time domain signal. The sound source detection unit 31 converts each of the sound signals from the time domain signal to a frequency domain signal by performing a Fourier transform on the sound signals (S12).

[0035] Next, the sound source detection unit 31 calculates a spatial correlation matrix from an input vector determined based on a plurality of sound signals transformed into the frequency domain (S13). Here, the microphone array 21 includes M microphone elements, and the mth sound signal after the Fourier transform is denoted as X m If (ω,t), the input vector x(ω,t) is expressed by the following formula, where T means transpose.

[0036]

number

[0037] The spatial correlation matrix R is expressed by the following formula: H denotes conjugate transpose. In the following, the frequency index ω is omitted for simplicity of explanation.

[0038]

number

[0039] Next, the sound source detection unit 31 calculates eigenvectors by eigenvalue decomposition of the spatial correlation matrix (S14). Specifically, the sound source detection unit 31 calculates eigenvectors e1, . . . e M and the eigenvalue λ1 λ M It is possible to calculate:

[0040]

number

[0041] Next, the sound source detection unit 31 detects the position of the sound source from the eigenvector (S15). Specifically, the sound source detection unit 31 can identify the loudness of the sound and the direction from which the sound comes from using the eigenvector, and can detect the direction from which a relatively loud sound comes as the direction (position) of the sound source.

[0042] As a result, as shown in FIG. 5, the sound source detection unit 31 can detect in which direction (angle) the sound source is located with respect to the position O of the sensing device 20. FIG. 5 is a diagram (a view of the desk 40 from above) that shows a schematic diagram of the detection result of the position of the sound source. In the example of FIG. 5, two sound sources, S1 and S2, are detected. Note that the sound source detection unit 31 only needs to detect at least the two-dimensional position of the sound source (direction represented by an angle in a top view) as shown in FIG. 5, but may also detect the three-dimensional position of the sound source. In the following embodiments, the position of the sound source detected by the sound source detection unit 31 is also referred to as the second position.

[0043] The sound source detection unit 31 can track the position of the sound source (second position) by repeating the operation of Fig. 4 for each unit time. Specifically, the sound source is a speaker (person) staying in the space 70, but it may also be a device installed in the space 70.

[0044] [Operation of the human detection unit] Next, a more specific description will be given of the operation of the human detection unit 32. Fig. 6 is a flowchart of the operation of the human detection unit 32. Note that the following operation is actually performed for video information acquired from each of the four cameras 22, but for convenience, the following description will be given assuming that video information is acquired from one camera 22.

[0045] First, the human detection unit 32 acquires image information of an image captured in the space 70 from the camera 22 of the sensing device 20 (S21).

[0046] Next, the human detection unit 32 identifies an area in the video where a person is shown based on the acquired video information (S22). The human detection unit 32 can identify an area in the video where a person is shown by a method using pattern matching or a method using a machine learning model.

[0047] Next, the human detection unit 32 assigns identification information to the identified regions (i.e., regions where people are present) (S23). For example, the human detection unit 32 identifies three regions and assigns identification information A, B, and C to the identified three regions. Hereinafter, a person corresponding to region A will also be referred to as person A, a person corresponding to region B as person B, and a person corresponding to region C as person C.

[0048] Next, human detection unit 32 identifies the direction in which the person is present based on the position in the image of the area identified in step S23 (S24). Information indicating the installation position of camera 22 (center of desk 40) and the shooting range (angle of view) of camera 22 is stored in advance in storage unit 37, and human detection unit 32 can identify which direction the position in the image corresponds to.

[0049] Next, the human detection unit 32 estimates the distance from the sensing device 20 (camera 22) to the person based on the size of the area identified in step S23 (S25). In this case, the larger the area identified in step S23, the closer the distance from the sensing device 20 (camera 22) to the person is estimated to be. Note that in step S25, the distance from the sensing device 20 (camera 22) to the person may be identified by distance information (actual measured value of distance) acquired from the distance measurement sensor 23.

[0050] As a result of the above steps S24 and S25, the human detection unit 32 can detect the position of a person in the space 70, as shown in Fig. 7. Fig. 7 is a diagram (a view of the desk 40 from above) that shows a schematic diagram of the detection result of the position of a person. In the example of Fig. 7, the positions of three people, person A, person B, and person C, are detected. Note that the human detection unit 32 detects the three-dimensional position of a person (three-dimensional coordinates of the position of the person), but it is sufficient to detect at least the two-dimensional position of the sound source (direction represented by an angle in a top view) as shown in Fig. 7. In the following embodiment, the position of a person detected by the human detection unit 32 is also referred to as a first position.

[0051] The human detection unit 32 can track the position of a person (first position) by repeating the operation of Fig. 6 for each unit time. In this case, the assignment of the identification information in step S23 needs to be performed only once, at the beginning.

[0052] [Operation of speech volume estimation unit] Next, a more detailed explanation will be given of the operation of the speech amount estimation unit 34. FIG.

[0053] The speech amount estimation unit 34 acquires a first position detected by the human detection unit 32 and a second position detected by the sound source detection unit 31 (S31). The first position and the second position acquired at this time are detected at substantially the same timing. "Substantially the same" means that some deviation may be included.

[0054] Next, the speech amount estimation unit 34 converts the first position represented by the three-dimensional coordinates into two-dimensional coordinates (angle) corresponding to the second position (S32). When the position of the microphone array 21 and the position of the camera 22 are different, the first position after being converted into the two-dimensional coordinates is corrected based on the difference between these positions.

[0055] Next, the speech amount estimation unit 34 detects the speaker (the speaker's position) by comparing the converted first position with the second position (S33). Fig. 9 is a diagram showing a schematic diagram of the detection result of the speaker's position. Fig. 9 is a diagram in which the second position (Fig. 5) and the first position (Fig. 7) are superimposed.

[0056] For example, when an angular difference Δθ1 between the second position of the sound source S1 and the first position of the person A is equal to or smaller than a predetermined value, the speech amount estimation unit 34 detects that the sound source S1 is person A. That is, person A is detected as a speaker. Also, when an angular difference Δθ2 between the second position of the sound source S2 and the first position of person C is equal to or smaller than a predetermined value, the speech amount estimation unit 34 detects that the sound source S2 is person C. That is, person C is detected as a speaker.

[0057] The speech amount estimation unit 34 repeats the operation of FIG. 8 for each unit time, thereby tracking each of the person A, the person B, and the person C, and can estimate the speech amount of each of the person A, the person B, and the person C. Specifically, the speech amount estimation unit 34 can estimate the period during which each of the person A, the person B, and the person C is detected as a speaker as the period during which each of the person A, the person B, and the person C is speaking. That is, the speech amount estimation unit 34 can store information indicating the period during which each of the person A, the person B, and the person C is speaking (information indicating the speech amount) in the storage unit 37. FIG. 10 is a diagram showing an example of the information indicating the speech amount. As shown in FIG. 10, the information indicating the speech amount is information in which the speech amount (speech time) is linked to each of the identification information assigned in the above step S23.

[0058] In this way, the speech amount estimation unit 34 can track each of the multiple people based on the first position detected by the person detection unit 32 and the second position detected by the sound source detection unit 31, and estimate the speech amount of each of the multiple people staying in the space 70 (the speech amount of the target person). Such a method of estimating the speech amount by the speech amount estimation unit 34 is useful for analyzing the quality of communication in a meeting or the like involving the movement of seats of multiple people. The movement of seats of multiple people in this case means, for example, moving to use the whiteboard 60.

[0059] Furthermore, the speech amount estimation unit 34 can estimate the amount of speech of each of a plurality of people while maintaining the anonymity of the people, rather than identifying the individual through voice recognition and image recognition.

[0060] [Operation of head direction estimation unit] Next, the operation of the head direction detection unit 33 will be described in more detail. FIG. 11 is a flowchart of the operation of the head direction detection unit 33. FIG. 12 is a diagram showing a three-dimensional coordinate space for explaining the operation of the head direction detection unit 33. As shown in FIG. 12, in the following explanation of the operation of the head direction detection unit 33, the camera 22 is assumed to be installed at a position C (x0, y0, z0) on the desk 40, not at the center position O (0, 0, 0) of the desk 40. In addition to the three-dimensional coordinates in the space 70 indicated by XYZ, coordinates in the image indicated by UV are also shown in FIG.

[0061] First, the head direction detection unit 33 acquires image information of an image captured in the space 70 from the camera 22 of the sensing device 20 (S41).

[0062] Next, the head direction detection unit 33 identifies the direction of the head of the person appearing in the video based on the acquired video information (S42). The head direction detection unit 33 can detect the direction of the person's head by, for example, identifying an area in the video corresponding to the person's head through face recognition processing and applying a machine learning model to the identified area. As shown in FIG. 12, the vector indicating the head direction at this time forms an angle of A° with respect to the line segment (straight line) connecting the position P of the person and the position C of the camera.

[0063] Next, the head direction detection unit 33 acquires the position of the person (more specifically, the position of the person's head) detected by the person detection unit 32 (S43). As shown in Fig. 12, the position of the person at this time is set to P(x1, y1, z1).

[0064] For example, using z1-z0 determined by the actual size of desk 40, horizontal viewing angle α of camera 22, the position (u, v) of the person in the image, and width w of the image, x1 can be expressed by the following formula. Note that information such as the size of desk 40, horizontal viewing angle α of camera 22, and width w of the image is stored in memory unit 37 in advance.

[0065] x1=x0+((uw / 2) / w)×(z1-z0)×tan(α)

[0066] Next, head direction detection unit 33 corrects (S44) the head direction (angle A) based on the position of camera 22 identified in step S42, based on the person's position acquired in step S43 (specifically, the value of coordinate x1). Fig. 13 is a diagram (a view of desk 40 from above) for explaining such angle correction.

[0067] Head direction detection unit 33 can calculate ∠OPC in FIG. 13 based on the coordinate of x1 acquired in step S43, and can set A+∠OPC as the corrected angle.

[0068] The head direction detection unit 33 can track the direction of the person's head by repeating the operation of FIG. 11 every unit time.

[0069] [Operation of the focus judgment unit] Next, a more detailed explanation will be given of the operation of the attention determination unit 35. FIG.

[0070] The attention determination unit 35 acquires the first position detected by the person detection unit 32 and the head direction of the person detected by the head direction detection unit 33 (S51). More specifically, the first position acquired at this time is the first position of each of the multiple people staying in the space 70, and the head direction of the person is the head direction of each of the multiple people.

[0071] Next, the attention determination unit 35 determines the target direction of one of the plurality of people as a target (S52). Fig. 15 is a diagram for explaining the target direction of the target person (a diagram of a desk 40 viewed from above).

[0072] For example, when a person at position P is the target person, the attention determination unit 35 determines the direction connecting position P to position G where another person is present as the target direction. Note that the target direction does not necessarily need to be determined with respect to a person, and may be determined with respect to the display 50 and the whiteboard 60 as described later.

[0073] Next, the attention determination unit 35 determines an allowable range centered on the target direction (S53). For example, in the example of FIG. 15, the range from ∠GPO-β to ∠GPO+β is determined as the allowable range. β is a coefficient for determining the allowable range. The value of β differs depending on whether the target direction is toward a person, the display 50, or the whiteboard 60.

[0074] Next, the attention determination unit 35 determines the attention of the subject by comparing the head direction of the subject acquired in step S51 with the allowable range determined in step S53 (S54). Attention here means attention to communication performed by multiple people including the subject in the space 70. When the head direction of the subject is within the allowable range, the attention determination unit 35 determines that the subject is focusing on communication since it is considered that the subject is looking at other people. On the other hand, when the head direction of the subject is outside the allowable range, the attention determination unit 35 determines that the subject is not focusing on communication since it is considered that the subject is not looking at other people.

[0075] The attention determination unit 35 can determine the attention of each of the persons A, B, and C by repeating the operation of FIG. 14 for each unit time for each of the persons (person A, person B, and person C). Specifically, the attention determination unit 35 can accumulate the periods during which each of the persons A, B, and C was determined to be concentrating, and store information indicating the periods during which the plurality of persons were concentrating on communication in the storage unit 37. FIG. 16 is a diagram showing an example of information indicating the attention period. As shown in FIG. 16, the information indicating the amount of speech is information in which the attention period is linked to each of the identification information assigned in the above step S23.

[0076] In this way, the attention determination unit 35 can determine the subject's attention to communication conducted by multiple people in the space 70 based on the first position detected by the person detection unit 32 and the head orientation detected by the head orientation detection unit 33.

[0077] The attention determination unit 35 may detect the position of the speaker by further acquiring the second position detected by the sound source detection unit 31, or may acquire the position of the speaker detected by the speech amount estimation unit 34. In either case, the attention determination unit 35 can determine the attention by considering whether each of the multiple people is a speaker or not. For example, it can determine that the subject is concentrating on communication only when the subject's head is facing the speaker, and can determine that the subject is not concentrating on communication when looking in the direction of a person who is not speaking.

[0078] Furthermore, when considering whether each of a plurality of people is a speaker, the attention determination unit 35 may determine the attention as follows.

[0079] For example, when a meeting is being held by multiple people in space 70, a target person (assumed to be person C) is unlikely to turn toward the person A who is positioned directly beside the target person, even if that person A is speaking. Therefore, the target person is likely to look toward another person B. The target person may also turn toward the display 50 (shown in FIG. 2) on which the report materials of the meeting and the like are displayed, and the whiteboard 60 (shown in FIG. 2). In such a case, if the target person is determined to be concentrating on communication only when his / her head is facing the speaker as described above, there is a problem that a target person who is actually concentrating on communication may be determined not to be concentrating on communication.

[0080] In such a case, while person A is speaking, the subject may be determined to be concentrating on communication during any of the following periods: (1) the period during which the subject faces the direction of person A; (2) the period during which the subject faces the direction of the display 50; (3) the period during which the subject faces the direction of a person other than person A (person B); and (4) the period during which the subject faces the direction of the whiteboard 60. In other words, when a person other than the subject among multiple people is speaking, the attention determination unit 35 may determine that the period during which the detected direction of the subject's head faces any of the person other than the subject among the multiple people, the display 50, and the whiteboard 60 is the period during which the person is concentrating on communication. Note that the period during which the subject faces the whiteboard 60 may be determined to be the period during which the subject is concentrating on communication under the condition that person A is speaking near the whiteboard.

[0081] [Communication Analysis Unit Operation] Next, a more detailed explanation will be given of the operation of the communication analysis unit 36. FIG.

[0082] The communication analysis unit 36 ​​reads out from the storage unit 37 the information indicating the amount of speech (FIG. 10) and the information indicating the attention period (FIG. 16) stored in the storage unit 37 (S61).

[0083] Next, the communication analysis unit 36 ​​analyzes the quality of communication in the space 70 based on the acquired information indicating the amount of speech and the acquired information indicating the attention period. For example, the communication analysis unit scores the quality of communication using the following scoring criteria.

[0084] The communication analysis unit 36, for example, calculates the ratio of the amount of speech of persons A to C based on the acquired information indicating the amount of speech, and sets a larger first score related to the amount of speech as the minimum / maximum value of this ratio is closer to 1 (i.e., persons A to C are speaking evenly). Specifically, when the ratio of the amount of speech of persons A to C is 1:1.2:1.5, the communication analysis unit 36 ​​calculates the first score based on the difference between 1 (minimum value) / 1.5 (maximum value) and 1.

[0085] Furthermore, for example, based on the acquired information indicating the attention periods, the communication analysis unit 36 ​​sets the second score regarding attention to a larger value as the average value of the attention periods of the persons A to C increases.

[0086] Then, the communication analysis unit 36 ​​calculates the sum of the first score and the second score as a final score indicating the quality of communication in the space 70. Each of the persons A to C can check the final score indicating the quality of communication (i.e., the analysis result of the quality of communication) by accessing the communication analysis system 10 (information processing system 30) using an information terminal such as a smartphone or a personal computer. Fig. 18 is a diagram showing an example of a display screen of the score indicating the quality of communication displayed on the information terminal.

[0087] In this way, the communication analysis unit 36 ​​can analyze the quality of communication performed by a plurality of people in the space 70 based on the estimation result of the amount of speech and the determination result of the focus. Note that the communication analysis unit 36 ​​may analyze the quality of communication performed by a plurality of people in the space 70 based on at least one of the estimation result of the amount of speech and the determination result of the focus. The above-mentioned score calculation criteria are only an example, and the score may be appropriately determined depending on the content required for communication in the space 70.

[0088] [Variations] In the above embodiment, the microphone array 21, the camera 22, and the distance measurement sensor 23 are installed on the desk 40. However, the microphone array 21, the camera 22, and the distance measurement sensor 23 may be installed on the ceiling. In addition, the microphone array 21, the camera 22, and the distance measurement sensor 23 do not need to be concentrated in one place, and the microphone array 21, the camera 22, and the distance measurement sensor 23 may be installed in different places.

[0089] In addition, in the above embodiment, the sound source detection unit 31 and the human detection unit 32 are used in combination to detect a speaker, but a speaker can also be detected by only the human detection unit 32. For example, the human detection unit 32 can detect whether a person is speaking based on the movement of the person in the video.

[0090] In addition, in the above embodiment, the communication analysis unit 36 ​​may calculate a final score indicating the quality of communication (i.e., the analysis result of the quality of communication) in real time when a meeting or the like is being held in the space 70, and control the environment in the space 70 in real time based on the calculated final score.

[0091] For example, when the communication analysis unit 36 ​​determines that the calculated final score is less than a predetermined value (i.e., communication is not active), it controls the environment of the space 70 by sending a control signal to an environmental control device (not shown) installed in the space 70.

[0092] Specifically, environmental control devices include air conditioners that control the temperature environment of space 70, lighting equipment that controls the light environment of space 70, fragrance generators that control the smell of space 70, and music playback devices that control the sound environment of space 70.

[0093] For example, the communication analysis unit 36 ​​can increase the temperature setting of an air conditioner or make lighting brighter than it is now, thereby stimulating communication in the space 70. The communication analysis unit 36 ​​may also activate an aroma generator installed in the space 70 or have a music playback device installed in the space 70 play music, thereby stimulating communication in the space 70.

[0094] [Effects, etc.] As described above, the attention determination system 39 includes a person detection unit 32 that acquires image information of an image showing a plurality of people staying in the space 70 in which the display 50 and the whiteboard 60 are installed, and detects a first position that is the position of the plurality of people in the space 70 based on the acquired image information, a head direction detection unit 33 that acquires image information and detects the head direction of a target person among the plurality of people based on the acquired image information, and an attention determination unit 35 that determines the attention of the target person to the communication performed by the plurality of people in the space 70 based on the detected first position and the detected head direction of the target person. When a person among the plurality of people other than the target person is speaking, the attention determination unit 35 determines that a period during which the detected head direction of the target person is facing any of the person among the plurality of people other than the target person, the display 50, and the whiteboard 60 is a period during which the target person is focusing on the communication.

[0095] Such an attention determination system 39 can determine the degree to which a subject is paying attention to communication.

[0096] Also, for example, a desk 40 is installed in the space 70 , and the human detection unit 32 acquires image information from a camera 22 installed on the desk 40 .

[0097] Such an attention determination system 39 can acquire image information from a camera 22 installed on a desk 40 .

[0098] Furthermore, for example, a plurality of cameras 22 are installed in the space 70, and the video information acquired by the human detection unit 32 includes video information of the images captured by each of the plurality of cameras 22.

[0099] Such an attention determination system 39 can acquire image information from a plurality of cameras 22 installed on a desk 40 .

[0100] Further, for example, a distance measuring sensor 23 that measures the distance from the camera 22 to a target person is installed in the space 70. The human detection unit 32 detects the first position based on the acquired image information and the detection result of the distance measuring sensor 23.

[0101] Such an attention determination system 39 can detect the first position based on the detection result of the distance measurement sensor 23.

[0102] Also, for example, to detect the first position, the human detection unit 32 estimates a distance from the human detection unit 32 to the target based on the size of the target in the image.

[0103] Such an attention determination system 39 can estimate the distance from the human detection unit 32 to a target person based on the size of the target person in the video.

[0104] The communication analysis system 10 also includes an attention determination system 39 and a communication analysis unit 36 ​​that analyzes the quality of communication performed by a plurality of people in a space 70 based on the determined attention of the subject.

[0105] Such a communication analysis system 10 can analyze the quality of communication based on the focus of the subject.

[0106] For example, the communication analysis system 10 further includes a sound source detection unit 31 that acquires sound information of a sound acquired in the space 70 and detects a second position that is the position of a sound source in the space 70 based on the acquired sound information, and a speech amount estimation unit 34 that tracks a target person and estimates the speech amount of the target person being tracked based on the detected first position and the detected second position. The communication analysis unit 36 ​​analyzes the quality of communication based on the determined focus of the target person and the estimated speech amount of the target person.

[0107] Such a communication analysis system 10 can analyze the quality of communication based on the subject's focus and the subject's speech volume.

[0108] Moreover, the attention determination method executed by a computer such as the attention determination unit 35 includes a first detection step of acquiring image information of an image showing a plurality of people staying in the space 70 in which the display 50 and the whiteboard 60 are installed, and detecting a first position that is the position of the plurality of people in the space 70 based on the acquired image information, a second detection step of acquiring image information and detecting the head direction of a target person among the plurality of people based on the acquired image information, and an attention determination step of determining the target person's attention to communication performed by the plurality of people in the space 70 based on the detected first position and the detected head direction of the target person. In the attention determination step, when a person among the plurality of people other than the target person is speaking, a period during which the detected head direction of the target person is facing any of the person among the plurality of people other than the target person, the display 50, and the whiteboard 60 is determined to be a period during which the person is focusing on communication.

[0109] This kind of focus determination method can determine the degree to which a subject is focusing on communication.

[0110] (Other embodiments) Although the embodiment has been described above, the present invention is not limited to the above embodiment.

[0111] For example, in the above embodiment, the communication analysis system is realized by a plurality of devices, but may be realized as a single device. For example, the communication analysis system may be realized as a single device corresponding to an information processing system, a speaker diarization system, or an attention determination system. When the communication analysis system is realized by a plurality of devices, the functional components of the communication analysis system may be distributed among the plurality of devices in any manner.

[0112] Furthermore, the method of communication between the devices in the above-described embodiment is not particularly limited. In addition, a relay device (not shown) may be involved in the communication between the devices.

[0113] In the above embodiment, the processes executed by a specific processing unit may be executed by another processing unit. The order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0114] In the above embodiment, each component may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0115] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or an integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit, or a dedicated circuit.

[0116] Furthermore, the general or specific aspects of the present invention may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0117] For example, the present invention may be realized as a speech volume estimation method executed by a computer such as a speaker diarization system, or as a program for causing a computer to execute such a speech volume estimation method, or as a computer-readable non-transitory recording medium on which such a program is recorded.

[0118] Furthermore, the present invention may be realized as an attention determination method executed by a computer such as an attention determination system, or as a program for causing a computer to execute such an attention determination method, or as a computer-readable non-transitory recording medium on which such a program is recorded.

[0119] In addition, the present invention also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art may think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the spirit of the present invention. [Explanation of symbols]

[0120] 10. Communication Analysis System 20 Sensing Device 21 Microphone Array 22 Camera 23 Distance sensor (sensor) 30 Information Processing Systems 31 Sound source detection unit 32 Human detection units 33 Head direction detection unit 34 Speech volume estimation unit 35 Focus Judgment Unit 36 Communication Analysis Unit 37 Storage Unit 38 Speaker Diarization System 39 Focus Judgment System 40 desks 50 Display 60 Whiteboard 70 space

Claims

1. a person detection unit that acquires image information of an image showing a plurality of people staying in a space in which a display and a whiteboard are installed, and detects a first position that is a position of the plurality of people in the space based on the acquired image information; a head direction detection unit that acquires the image information and detects a head direction of a target person among the plurality of people based on the acquired image information; an attention determination unit that determines an attention of the subject to a communication performed by the plurality of people in the space based on the detected first position and the detected head direction of the subject; The attention determination unit determines, when a person among the plurality of people other than the target person is speaking, a period during which the detected direction of the head of the target person is directed toward any of the person among the plurality of people other than the target person, the display on which materials related to the communication are being displayed, and the whiteboard, as a period during which the target person is concentrating on the communication. Attention determination system.

2. A desk is provided in the space, The human detection unit acquires the image information from a camera installed on the desk. The focus determination system according to claim 1 .

3. A plurality of cameras are installed in the space, The image information acquired by the human detection unit includes image information of images captured by each of the plurality of cameras. The focus determination system according to claim 2 .

4. a sensor that measures a distance from the camera to the subject is installed in the space; The human detection unit detects the first position based on the acquired image information and a detection result of the sensor. The focus determination system according to claim 2 .

5. The human detection unit estimates a distance from the human detection unit to the subject based on a size of the subject in the image to detect the first position. The focus determination system according to claim 1 .

6. The attention determination unit determines that a period during which a detected direction of the head of the target person is directed toward a non-speaking person among the plurality of people other than the target person when the person among the plurality of people is speaking is also a period during which the target person is concentrating on the communication. The focus determination system according to claim 1 .

7. The focus determination system according to any one of claims 1 to 6, and a communication analysis unit configured to analyze the quality of communication performed by the plurality of people in the space based on the determined focus of the target person. Communication analysis system.

8. a sound source detection unit that acquires sound information of a sound acquired in the space and detects a second position that is a position of a sound source in the space based on the acquired sound information; a speech volume estimation unit for tracking the target person based on the detected first position and the detected second position, and estimating a speech volume of the target person being tracked; The communication analysis unit analyzes a quality of the communication based on the determined attention of the subject and the estimated speech volume of the subject. The communication analysis system according to claim 7.

9. A method for determining attention to a subject, the method comprising: a first detection step of acquiring image information of an image showing a plurality of people staying in a space in which a display and a whiteboard are installed, and detecting a first position that is a position of the plurality of people in the space based on the acquired image information; a second detection step of acquiring the image information and detecting a head orientation of a target person among the plurality of people based on the acquired image information; and determining an attention of the subject to communication performed by the plurality of people in the space based on the detected first position and the detected head orientation of the subject, In the attention determination step, when a person among the plurality of people other than the target person is speaking, a period during which the detected direction of the head of the target person is directed toward any of the person among the plurality of people other than the target person, the display on which materials related to the communication are displayed, and the whiteboard is determined to be a period during which the person is concentrating on the communication. Focus determination method.

10. In the attention determination step, when a person other than the target person among the plurality of people is speaking, a period during which the detected direction of the head of the target person is directed toward a person other than the target person who is not speaking among the plurality of people is also determined to be a period during which the target person is concentrating on the communication. The method of determining focus according to claim 9.

11. A program for causing a computer to execute the focus determination method according to claim 9 or 10.

Citation Information

Patent Citations

  • JP2004‐112518A

  • Conference support system, conference support device, and conference support method

    JP2018036868A

  • Method, system, computing device, and program for providing teleconference participant resource feedback

    JP2018137723A

  • Information output device and program

    JP2019057061A

  • Activity evaluation program, apparatus, and method

    JP2019200475A