Display object extraction device

Abstract

To reduce the burden of a manager when extracting a person as a display object, not limited to persons of fixed motion, who would convey the atmosphere of a real space or a virtual space.SOLUTION: A climax determination unit 12 determines whether or not the whole of a real or a virtual space is charged up, in a real or a virtual space where a plurality of persons exist. A person extraction unit 15 extracts a specific person from among the plurality of persons. A first subject determination unit 16 determines that the person extracted by the person extraction unit 15 is a first subject when it is determined by the climax determination unit 12 that the whole of the real or virtual space is charged up, and the difference between the motion of the person extracted by the person extraction unit 15 and an exemplar rhythm is within a first range and the difference between the feature amount of motion of the person extracted by the person extraction unit 15 and the feature amount of an exemplar motion is out of a second range. A display control unit 17 displays on a display device the first subject determined by the first subject determination unit 16.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a display target extraction device. [Background technology]

[0002] Patent document 1 describes an electronic device that extracts from a moving image an object whose movement is represented by rhythm information where the similarity between the rhythm information for extraction and the rhythm information of the object in the moving image is equal to or greater than a predetermined threshold value. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-28419 A Summary of the Invention [Problem to be solved by the invention]

[0004] The electronic device described in Patent Document 1 extracts objects whose movements are similar to the rhythm information for extraction from a moving image. For this reason, if the electronic device described in Patent Document 1 is applied to extract people to be displayed in order to capture images of people in a live music venue or the like and display the captured images of the people on a screen or the like in the venue, it will only extract people with fixed movements. If it is desired to extract people who can convey the atmosphere of a space such as a live music venue as display targets, rather than limiting the people to those with fixed movements, the administrator must manually extract the people to be displayed, which may impose a burden on the administrator.

[0005] The present invention aims to provide a display target extraction device that can reduce the burden on an administrator when extracting a person who can convey the atmosphere of a space as a display target, without being limited to a person with a fixed movement. [Means for solving the problem]

[0006] The present invention provides a display target extraction device including: an excitement determination unit that determines whether an entire real space or virtual space in which a plurality of people exist is excited; a person extraction unit that extracts a specific person from the plurality of people; a first subject determination unit that, when the excitement determination unit determines that the entire real space or virtual space is excited, determines the person extracted by the person extraction unit as a first subject if a difference between a rhythm of the movement of the person extracted by the person extraction unit and an exemplary rhythm is within a first range and a difference between a feature amount of the movement of the person extracted by the person extraction unit and a feature amount of an exemplary movement is outside a second range; and a display control unit that displays the first subject determined by the first subject determination unit on a display device. Effect of the Invention

[0007] According to the display target extraction device of the present invention, it is possible to reduce the burden on an administrator when extracting a person who can convey the atmosphere of a real space or a virtual space as a display target, rather than being limited to a person with a fixed movement. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a display target extraction device and its peripheral devices according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing the mounting position of the gyro sensor and the coordinates measured by the gyro sensor. [Diagram 3] FIG. 3 is a flowchart illustrating an example of a process in which the display target extraction device according to an embodiment displays a first target person. [Figure 4] FIG. 4 is a flowchart illustrating an example of a first excitement determination process performed by the display target extraction device according to an embodiment. [Diagram 5] FIG. 5 is a flowchart illustrating an example of a second excitement determination process performed by the display target extraction device according to an embodiment. [Figure 6] FIG. 6 is a flowchart illustrating an example of a process in which the display target extraction device according to an embodiment displays a second target person. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the device will be described with reference to the accompanying drawings. In the description of the drawings, the same parts are given the same reference numerals and the description will be omitted.

[0010] (Configuration of the display image extraction device) A configuration example of a display target extraction device 10 according to an embodiment and its peripheral devices will be described with reference to Fig. 1. The display target extraction device 10 according to an embodiment is connected to a microphone 20, a gyro sensor 30, and an image capture device 40 by wire or wirelessly.

[0011] In FIG. 1, one microphone 20 is illustrated, but the number of microphones 20 is not particularly limited. When the display target extraction device 10 is applied to a real space, the microphone 20 is installed in a real space where multiple people exist, such as a live venue in the real space, and detects the volume in the real space. In this case, the volume in the real space includes, for example, the volume of a piece of music being played or reproduced in the real space and the volume of cheers of multiple people existing in the real space. In addition, when the display target extraction device 10 is applied to a virtual space, multiple microphones 20 are installed in a real space such as a room where each person in a virtual space where multiple people exist actually exists, such as a live venue in the virtual space. In this case, the volume in the real space includes the volume of cheers of each person existing in the real space. The microphone 20 outputs the detected volume in the real space to the display target extraction device 10.

[0012] Although one gyro sensor 30 is illustrated in FIG. 1, multiple gyro sensors 30 are used and are attached to various parts of the body of multiple people existing in real space in advance. For example, the gyro sensor 30 is attached to at least two locations, namely, both wrists, of each person. Also, the gyro sensor 30 may be attached to four locations, namely, both wrists and both elbows, of each person, or may be attached to six locations, namely, both wrists, both elbows, and the left and right sides of the waist, of each person. When the gyro sensor 30 is attached to each person, a three-dimensional coordinate system is assigned to the gyro sensor 30, with a predetermined position on the body of each person as the origin. The predetermined position as the origin is, for example, between the shoulders of each person.

[0013] The gyro sensor 30 attached to each person detects the movement of each person, with the position of each person in a predetermined posture such as standing upright as the initial position. The gyro sensor 30 detects the coordinates and angular velocity of each part at each time in a three-dimensional coordinate system based on the direction and magnitude of the detected movement.

[0014] 2, for example, assume that the gyro sensors 31 and 32 are attached to the right and left wrists of each person. In this case, the gyro sensor 31 detects the coordinates and angular velocity of the right wrist at each time, and the gyro sensor 32 detects the coordinates and angular velocity of the left wrist at each time.

[0015] 2, for example, assume that gyro sensors 33 and 34 are attached to the right and left elbows of each person in addition to the gyro sensors 31 and 32. In this case, the gyro sensor 33 detects the coordinates and angular velocity of the right elbow at each time, and the gyro sensor 34 detects the coordinates and angular velocity of the left elbow at each time.

[0016] 2, for example, assume that gyro sensors 35 and 36 are attached to the left and right sides of the waist (pelvis) of each person in addition to the gyro sensors 31 to 34. In this case, the gyro sensor 35 detects the coordinates and angular velocity of the right waist at each time, and the gyro sensor 36 detects the coordinates and angular velocity of the left waist at each time.

[0017] The gyro sensors 31 to 34 output the coordinates and angular velocity of each part detected at each time to the display target extraction device 10. Note that the gyro sensors 31 to 34 are assigned an identification ID corresponding to each person, and it is possible to determine which gyro sensor is attached to which person.

[0018] The display target extraction device 10 is a computer equipped with a CPU (Central Processing Unit), a memory, an input / output unit, etc. (not shown). In the display target extraction device 10, the CPU loads various programs stored in the storage unit 13 or the like into the memory, and executes various processes included in the programs. The memory is a storage medium such as a RAM (Random Access Memory).

[0019] The display target extraction device 10 includes a volume acquisition unit 11, an excitement determination unit 12, a storage unit 13, a coordinate acquisition unit 14, a person extraction unit 15, a first target person determination unit 16, and a display control unit 17.

[0020] The storage unit 13 stores various types of information required to realize the functions of the display target extraction device 10. The storage unit 13 is an arbitrary storage medium such as a semiconductor memory.

[0021] The volume acquisition unit 11 acquires the volume in the real space from the microphone 20. When the display target extraction device 10 is applied to the real space, the volume acquisition unit 11 acquires the volume of the entire real space including the volume of the music being played or reproduced in the real space and the volume of cheers and the like of multiple people present in the real space from the microphone 20. When the display target extraction device 10 is applied to a virtual space, the volume acquisition unit 11 acquires the volume of cheers and the like of each person present in the real space from each of the multiple microphones 20.

[0022] The volume acquisition unit 11 continuously acquires the volume in the real space from the time when the performance or playback of the music starts in the real space or the virtual space, and calculates a statistical value of the volume in the real space from the time when the performance or playback of the music starts in a live venue in the real space or the virtual space to the current time. The statistical value may be, for example, the total value, the average value, or the median value of the volume in the real space from the time when the performance or playback of the music starts in the real space or the virtual space to the current time. When the display target extraction device 10 is applied to a virtual space, the volume acquisition unit 11 calculates a statistical value such as the total value, the average value, or the median value of the volume of the entire real space acquired from each of the multiple microphones 20 from the time when the performance or playback of the music starts in a live venue in the virtual space to the current time.

[0023] The volume acquisition unit 11 stores a history of statistical values ​​of the volume in the real space from the start of playing or reproducing a piece of music in the real space or virtual space to the current time in the storage unit 13. The volume acquisition unit 11 may reset the statistical values ​​every time the piece of music being played or reproduced in the real space or virtual space changes.

[0024] The coordinate acquisition unit 14 acquires the coordinates and angular velocity of each part of each person at each time from the gyro sensor 30 that each person wears on their body in real space. The coordinate acquisition unit 14 stores the coordinates and angular velocity of each part of each person at each time acquired from the gyro sensor 30 in the storage unit 13 for each person.

[0025] The excitement determination unit 12 determines whether or not the entire real space or virtual space in which a plurality of people exist is excited. The excitement determination unit 12 performs, for example, a first excitement determination or a second excitement determination.

[0026] First, the first excitement judgment will be described. When the excitement judgment unit 12 performs the first excitement judgment after the performance or reproduction of the first music piece is started in the real space or virtual space, the excitement judgment unit 12 performs the first excitement judgment. In the first excitement judgment, the excitement judgment unit 12 calculates the ratio of "people performing motion" among a plurality of people existing in the real space or virtual space based on the coordinates of each part of each person at each time stored in the storage unit 13 by the coordinate acquisition unit 14. The "people performing motion" here refers to a person whose current movement rhythm matches the model rhythm. The model rhythm here is, for example, the rhythm of the music piece currently being performed or reproduced in the real space or virtual space.

[0027] For example, the excitement determination unit 12 determines the rhythm of the movement of each person based on the coordinates of each part of each person at each time stored in the storage unit 13 by the coordinate acquisition unit 14. Note that a method for determining the rhythm of the movement of each person based on the coordinates of each part of each person at each time is known, so a detailed description will be omitted. The excitement determination unit 12 calculates the difference between the model rhythm and the current movement rhythm, and determines a person whose calculated difference is within a first range as a person whose current movement rhythm matches the model rhythm, that is, a "person performing a motion," and calculates the ratio of "person performing a motion" to all people existing in the real space or virtual space. Note that the model rhythm at each time is stored in the storage unit 13 in advance.

[0028] In the first excitement determination, the excitement determination unit 12 determines that the entire real space or virtual space is excited when the calculated ratio of "people performing motions" is equal to or greater than a first threshold. The first threshold is, for example, 50%.

[0029] Next, the second excitement judgment will be described. After performing the first excitement judgment once, the excitement judgment unit 12 performs the second excitement judgment when performing the second or subsequent excitement judgment. In the second excitement judgment, the excitement judgment unit 12 calculates the ratio of "people performing motion" among a plurality of people existing in the real space or virtual space in the same manner as the first excitement judgment. Furthermore, the excitement judgment unit 12 acquires a history of the statistical values ​​(such as the total value, the average value, or the median value) of the volume in the real space from the time when the performance or reproduction of the music in the real space or virtual space starts to the current time, which is stored in the storage unit 13 by the volume acquisition unit 11. In the second excitement judgment, the excitement judgment unit 12 judges that the whole real space or virtual space is excited when the calculated ratio of "people performing motion" is equal to or greater than the first threshold value and the acquired statistical value of the volume in the real space up to the current time is equal to or greater than the second threshold value. For example, the second threshold value is a statistical value of the volume in the real space up to the time when the excitement determination was performed previously.

[0030] The person extraction unit 15 extracts a specific person from among a plurality of people existing in a real space or a virtual space. When the display target extraction device 10 is applied to a real space, the person extraction unit 15 randomly extracts a specific person by, for example, randomly specifying position information of the audience seats in the real space captured by the imaging device 40. When the display target extraction device 10 is applied to a virtual space, the person extraction unit 15 randomly extracts a person by, for example, randomly specifying an account of a person existing in the virtual space.

[0031] When excitement determination unit 12 determines that the entire real space or virtual space is excited, first subject determination unit 16 acquires the movement of the person extracted by person extraction unit 15. Specifically, first subject determination unit 16 acquires the coordinates of each body part of the extracted person at each time from the coordinates of each body part of each person at each time acquired by coordinate acquisition unit 14 and stored in storage unit 13.

[0032] The first subject determination unit 16 determines whether the movement of the extracted person is a repetitive movement based on the coordinates of each part of the extracted person at each time. Note that a method for determining repetitive movement based on changes in the coordinates of each part of the extracted person at each time is known, so a detailed description will be omitted. The first subject determination unit 16 only needs to be able to determine the rhythm of the movement of the extracted person, and is not required to determine whether the movement of the extracted person is a repetitive movement.

[0033] The first subject determination unit 16 judges the rhythm of the extracted person's movement based on the coordinates of each part of the extracted person at each time. The first subject determination unit 16 judges whether the rhythm of the extracted person's movement matches the model rhythm. The first subject determination unit 16 calculates the difference between the rhythm of the extracted person's movement and the model rhythm, and when the calculated difference is within a second range, determines that the rhythm of the extracted person's movement matches the model rhythm. When the difference between the rhythm of the extracted person's movement and the model rhythm is outside the second range, the first subject determination unit 16 determines that the rhythm of the extracted person's movement does not match the model rhythm. The model rhythm at each time is stored in the storage unit 13 in advance.

[0034] For example, the first subject determination unit 16 determines that the rhythm of the extracted person's movements matches the model rhythm when the difference between the rhythm of the extracted person's movements and the rhythm of the music currently being played or reproduced in the real space or virtual space is within a second range, and determines that the rhythm of the extracted person's movements does not match the model rhythm when the difference between the rhythm of the extracted person's movements and the rhythm of the music currently being played or reproduced in the real space or virtual space is outside the second range.

[0035] Furthermore, the first subject determination unit 16 determines whether or not the extracted feature amount of the person's movement matches the feature amount of the model movement stored in advance in the storage unit 13 based on the change in the coordinates of each part of the extracted person at each time. The first subject determination unit 16 calculates the difference between the feature amount of the extracted person's movement at each time and the feature amount of the model movement, and determines that the extracted feature amount of the person's movement matches the feature amount of the model movement when the calculated difference is within a third range. The first subject determination unit 16 determines that the extracted feature amount of the person's movement does not match the feature amount of the model movement when the difference between the extracted feature amount of the person's movement and the feature amount of the model movement is outside the third range. A method for determining whether the difference between the extracted feature amount of the person's movement and the feature amount of the model movement is within or outside the third range will be described later.

[0036] If the rhythm of the extracted person's movement matches the model rhythm and the characteristic amount of the extracted person's movement does not match the characteristic amount of the model action, i.e., if the difference between the rhythm of the extracted person's movement and the model rhythm is within a second range and the difference between the characteristic amount of the extracted person's movement and the characteristic amount of the model action is outside a third range, the first subject determination unit 16 determines the extracted person to be a first subject whose movement is not similar to the model action.

[0037] First to third examples of a method for determining whether the difference between the extracted feature amount of a person's movement and the feature amount of the model movement is within or outside the third range will be described. First, as a first example, a case where the gyro sensors 31 and 32 are attached to the right and left wrists of each person will be described. Taking the example of a movement of swinging the right hand from side to side in time with the rhythm, the storage unit 13 stores in advance, as a model movement of the right hand, a time t1 when the right wrist is located at the right end point, which is the start point of the movement, and the coordinates (X1, Y1, Z1) of the right wrist at the time t1. Furthermore, the storage unit 13 stores, as a model movement of the right hand, a time t2 when the right wrist moves after the time t1 and is located at the left end point, which is the end point of the movement, and the coordinates (X2, Y2, Z2) of the right wrist at the time t2. Thereafter, the memory unit 13 stores, as an example movement of the right hand, time t3 when the right wrist moves after time t2 and is located at the right end point which is the start point of the movement, the coordinates (X3, Y3, Z3) of the right wrist at time t3, time t4 when the right wrist moves after time t3 and is located at the left end point which is the end point of the movement, and the coordinates (X4, Y4, Z4) of the right wrist at time t4. Thereafter, similarly, the memory unit 13 repeatedly stores the time when the right wrist is located at the right end point which is the start point of the movement and the coordinates of the right wrist at that time, and the time when the right wrist is located at the left end point which is the end point of the movement and the coordinates of the right wrist at that time.

[0038] The first subject determination unit 16 acquires from the storage unit 13 the coordinates of the gyro sensor 31 of the extracted person at time t1, i.e., the coordinates (x1, y1, z1) of the right wrist of the extracted person at time t1. The first subject determination unit 16 also acquires from the storage unit 13 the coordinates (x2, y2, z2) of the right wrist of the extracted person at time t2.

[0039] The first subject determination unit 16 compares the coordinates (x1, y1, z1), (x2, y2, z2) of the right wrist of the extracted person at times t1 and t2 with the coordinates (X1, Y1, Z1), (X2, Y2, Z2) of the right wrist of the model movement at times t1 and t2, respectively, and calculates the difference (deviation) between the coordinates of the right wrist of the extracted person and the coordinates of the model movement at times t1 and t2. Note that the first subject determination unit 16 may calculate only the deviation of the coordinates at time t1 or only the deviation of the coordinates at time t2.

[0040] The first subject determination unit 16 determines that the difference between the extracted feature amount of the person's right hand movement and the feature amount of the model right hand movement is within a third range when the calculated coordinate shift at times t1 and t2 is equal to or smaller than a predetermined value. The first subject determination unit 16 determines that the difference between the extracted feature amount of the person's right hand movement and the feature amount of the model right hand movement is outside the third range when the calculated coordinate shift at times t1 and t2 is greater than a predetermined value.

[0041] The first subject determination unit 16 may determine that the difference between the extracted feature amount of the person's right hand movement and the feature amount of the model right hand movement is within a third range when either the calculated coordinate shift at time t1 or the calculated coordinate shift at time t2 is equal to or smaller than a predetermined value, and may determine that the difference between the extracted feature amount of the person's right hand movement and the feature amount of the model right hand movement is outside the third range when either the calculated coordinate shift at time t1 or the calculated coordinate shift at time t2 is greater than a predetermined value. The first subject determination unit 16 repeats the same process at times t3, t4, and after times t3 and t4.

[0042] In addition to the coordinates, angular velocity can be added as a parameter for comparing the extracted feature amount of the person's movement with the feature amount of the model movement. For example, in the case of waving the right hand from right to left, the storage unit 13 further stores the angular velocities (ωX1, ωY1, ωZ1) and (ωX2, ωY2, ωZ2) of the right wrist at times t1 and t2 as the model movement of the right hand.

[0043] In this case, the first subject determination unit 16 further acquires from the memory unit 13 the angular velocity of the gyro sensor 31 of the extracted person at times t1 and t2, i.e., the accelerations (ωx1, ωy1, ωz1), (ωx2, ωy2, ωz2) of the right wrist of the extracted person at times t1 and t2.

[0044] The first subject determination unit 16 compares the right wrist accelerations (ωx1, ωy1, ωz1), (ωx2, ωy2, ωz2) of the extracted person at times t1 and t2 with the right wrist angular velocities (ωX1, ωY1, ωZ1), (ωX2, ωY2, ωZ2) of the model movement at times t1 and t2, respectively, and calculates the difference (deviation) between the right wrist angular velocities of the extracted person and the model movement at times t1 and t2. Note that the first subject determination unit 16 may calculate only the deviation in angular velocity at time t1 or only the deviation in angular velocity at time t2.

[0045] The first subject determination unit 16 may determine that the difference between the extracted feature amount of the person's right hand movement and the feature amount of the model right hand movement is within a third range when the calculated coordinate deviation at times t1 and t2 is equal to or smaller than a predetermined value and the calculated angular velocity deviation at times t1 and t2 is equal to or smaller than a predetermined value. The first subject determination unit 16 may determine that the difference between the extracted feature amount of the person's right hand movement and the feature amount of the model right hand movement is outside the third range when the calculated coordinate deviation at times t1 and t2 is greater than a predetermined value and the calculated angular velocity deviation at times t1 and t2 is greater than a predetermined value.

[0046] In addition, the first subject determination unit 16 may determine that the difference between the extracted feature amount of the person's right hand movement and the feature amount of the model right hand movement is within the third range when either the calculated deviation of the coordinates and angular velocity at time t1 or the deviation of the coordinates and angular velocity at time t2 is equal to or less than a predetermined value, and may determine that the difference between the extracted feature amount of the person's right hand movement and the feature amount of the model right hand movement is outside the third range when either the calculated deviation of the coordinates and angular velocity at time t1 or the deviation of the coordinates and angular velocity at time t2 is greater than a predetermined value. The first subject determination unit 16 repeats the same process at time t3, t4, and after time t3 and t4. By adding a parameter for comparing the extracted feature amount of the person's movement and the feature amount of the model movement, the accuracy of determining the agreement of the feature amount of the movement can be improved.

[0047] Next, as a second example, a case will be described in which gyro sensors 32 and 33 are attached to the right and left elbows of each person in addition to gyro sensors 31 and 32. Taking the example of a movement of waving the right hand from side to side in time with the rhythm, in addition to the first example, storage unit 13 stores the coordinates (P1, Q1, R1) of the right elbow at time t1 and the coordinates (P2, Q2, R2) of the right elbow at time t2 as model movements of the right hand.

[0048] As in the first example, the first subject determination unit 16 may compare the coordinates of the right wrist and right elbow of the extracted person at times t1 and t2 with the coordinates of the right wrist and right elbow of the model movement at times t1 and t2, respectively, to determine whether the difference between the characteristic amount of the right hand movement of the extracted person and the characteristic amount of the model right movement is within or outside the third range.

[0049] Furthermore, the first subject determination unit 16 may compare a vector connecting the coordinates of the right wrist and right elbow at times t1 and t2 with a vector connecting the coordinates of the right wrist and right elbow at times t1 and t2 of the model movement, in addition to the coordinates of the right wrist and right elbow at times t1 and t2. In this case, the storage unit 13 further stores, as the model movement of the right hand, Equation (1) indicating a vector connecting the coordinates (P1, Q1, R1) of the right elbow and the coordinates (X1, Y1, Z1) of the right wrist at time t1.

number

[0050] Moreover, the storage unit 13 further stores equation (2) indicating a vector connecting the coordinates (X2, Y2, Z2) of the right wrist and the coordinates (P2, Q2, R2) of the right elbow at time t2.

number

[0051] The first subject determination unit 16 acquires from the memory unit 13 the coordinates (x1, y1, z1) of the right wrist and the coordinates (p1, q1, r1) of the right elbow of the extracted person at time t1, and the coordinates (x2, y2, z2) of the right wrist and the coordinates (p2, q2, r2) of the right elbow at time t2.

[0052] The first subject determination unit 16 calculates equation (3) that indicates a vector connecting the coordinates (x1, y1, z1) of the right wrist and the coordinates (p1, q1, r1) of the right elbow of the extracted person at time t1.

number

[0053] In addition, the first subject determination unit 16 calculates equation (4) that indicates a vector connecting the coordinates (x2, y2, z2) of the right wrist and the coordinates (p2, q2, r2) of the right elbow of the extracted person at time t2.

number

[0054] The first subject determination unit 16 compares the formula (3) and formula (4) showing the vector connecting the coordinates of the right wrist and the coordinates of the right elbow of the extracted person at times t1 and t2 with the formula (1) and formula (2) showing the vector connecting the coordinates of the right wrist and the coordinates of the right wrist of the model movement at times t1 and t2, respectively, and calculates the difference (deviation) between the vector of the extracted person and the vector of the model movement at times t1 and t2. Note that the first subject determination unit 16 may calculate only the vector deviation at time t1 or only the vector deviation at time t2.

[0055] The first subject determination unit 16 may determine that the difference between the extracted feature amount of the person's right hand movement and the feature amount of the model right hand gesture is within a third range when the calculated vector deviation at times t1 and t2 is equal to or smaller than a predetermined value. The first subject determination unit 16 may determine that the difference between the extracted feature amount of the person's right hand movement and the feature amount of the model right hand gesture is outside the third range when the calculated vector deviation at times t1 and t2 is greater than a predetermined value.

[0056] The first subject determination unit 16 may determine that the difference between the extracted feature amount of the person's right hand movement and the feature amount of the model right hand movement is within a third range when either the calculated vector deviation at time t1 or the vector deviation at time t2 is equal to or smaller than a predetermined value, and may determine that the difference between the extracted feature amount of the person's right hand movement and the feature amount of the model right hand movement is outside the third range when either the calculated vector deviation at time t1 or the vector deviation at time t2 is greater than a predetermined value. The first subject determination unit 16 repeats the same process at times t3, t4, and after times t3 and t4.

[0057] In the first and second examples, only the movement of the right hand is illustrated, but if the movement of the left hand is also included at the same time, the first subject determination unit 16 performs similar processing on the gyro sensor 32 attached to the left wrist of the extracted person, and determines whether the difference between the characteristic amount of the extracted person's movement and the characteristic amount of the model movement is within or outside the third range.

[0058] Next, as a third example, a case will be described in which gyro sensors 35 and 36 are attached to the right and left hips of each person in addition to the gyro sensors 31 and 32 or the gyro sensors 31 to 34. In this case, in addition to the first or second example, the first subject determination unit 16 can determine whether or not the extracted lower body movement of the person coincides with the lower body movement of the model action.

[0059] In the third example, the storage unit 13 stores information on the type of motion, such as lifting the leg, jumping, squatting, etc., as model motions, and the time corresponding to each motion. The storage unit 13 also stores the coordinates of the gyro sensor 35, i.e., the coordinates (M, N, O) of the right hip, and the coordinates of the gyro sensor 36, i.e., the coordinates (J, K, L) of the left hip, when calibration is performed in advance with each person in a predetermined posture, such as standing upright.

[0060] The first subject determination unit 16 acquires the coordinates (m1, n1, o1) of the right hip and the coordinates (j1, k1, l1) of the left hip at time t1 from the storage unit 13. For example, when the z coordinate o1 of the right hip at time t1 is greater than the z coordinate O of the right hip at the time of calibration of the extracted person, the first subject determination unit 16 determines that the extracted person has performed an action of lifting the right leg at time t1. Also, when the z coordinate o1 of the right hip at time t1 is greater than the z coordinate O of the right hip at the time of calibration and the z coordinate l1 of the left hip at time t1 is greater than the z coordinate L of the left hip at the time of calibration, the first subject determination unit 16 determines that the extracted person has performed an action of jumping at time t1. Furthermore, if the z coordinate o1 of the right hip at time t1 is smaller than the z coordinate O of the right hip at the time of calibration, and the z coordinate l1 of the left hip at time t1 is smaller than the z coordinate L of the left hip at the time of calibration, it is determined that the extracted person performed a squatting motion at time t1.

[0061] The first subject determination unit 16 may determine that the feature amount of the extracted person's movement matches the feature amount of the model movement when the type of the extracted person's movement at time t1 matches the type of the model movement at time t1. The first subject determination unit 16 repeats the same process at times t3, t4, and after times t3 and t4.

[0062] When the first subject determination unit 16 determines that the extracted person is the first subject, the display control unit 17 displays the determined first subject on a display device (not shown) such as a monitor or screen in real space or virtual space for a predetermined period of time.

[0063] When the display target extraction device 10 is applied to a real space, the display control unit 17 outputs, for example, information indicating the position of the determined first subject to the imaging device 40, and outputs a signal to the imaging device 40 that images the real space, instructing the imaging device 40 to image the determined first subject for a predetermined period of time. When the display target extraction device 10 is applied to a virtual space, the display control unit 17 outputs, for example, a signal to instruct the imaging device 40 to image the first subject for a predetermined period of time, to the imaging device 40 installed in the real space where the determined first subject actually exists. Note that when the display target extraction device 10 is applied to a virtual space, the display control unit 17 may display, for a predetermined period of time, an avatar that reflects the movement of the determined first subject in the real space on a monitor or screen in the virtual space.

[0064] In addition, when the first subject determination unit 16 has not determined the first subject, the display control unit 17 displays the person extracted by the person extraction unit 15 as the second subject on a display device (not shown) in the real space or virtual space for a predetermined period of time.

[0065] When the display target extraction device 10 is applied to a real space, the display control unit 17 outputs, for example, information indicating the position of the second subject randomly extracted by the person extraction unit 15 to the imaging device 40, and outputs a signal to the imaging device 40 that images the real space to instruct the imaging device 40 to image the second subject for a predetermined period of time. When the display target extraction device 10 is applied to a virtual space, the display control unit 17 outputs, for example, a signal to instruct the imaging device 40 to image the second subject for a predetermined period of time to the imaging device 40 installed in the real space where the second subject actually exists. Note that when the display target extraction device 10 is applied to a virtual space, the display control unit 17 may display an avatar reflecting the movement of the second subject in the real space on a monitor or screen in the virtual space for a predetermined period of time.

[0066] In addition, when the first subject determination unit 16 determines the first subject, if a second subject is being displayed on the display device, the display control unit 17 ends the display of the second subject and displays the first subject on the display device for a predetermined period of time.

[0067] The image capturing device 40 is configured to be capable of displaying an image captured in real space on a display device such as a monitor or screen in real space or virtual space in real time. The image capturing device 40 captures an image of the first subject or the second subject based on an instruction from the display control unit 17, and outputs the captured image of the first subject or the second subject to the display device in real time.

[0068] (Display object extraction device processing) Next, an example of a process in which the display target extraction device 10 according to an embodiment displays a first target person will be described with reference to Fig. 3. The display target extraction device 10 starts the process in Fig. 3 after a predetermined time has elapsed since the performance or reproduction of a piece of music has started in a real space or a virtual space, for example.

[0069] In step S11 of Fig. 3, the excitement determination unit 12 determines whether or not the entire real space or virtual space in which a plurality of people exist is excited. When making the first excitement determination after the performance or playback of the first piece of music starts in the real space or virtual space, the excitement determination unit 12 performs a first excitement determination shown in Fig. 4. When making a second or subsequent excitement determination after having performed the first excitement determination once, the excitement determination unit 12 performs a second excitement determination shown in Fig. 5.

[0070] The first excitement determination process will be described with reference to Fig. 4. In step S111 in Fig. 4, the excitement determination unit 12 calculates the ratio of "people performing a motion" among a plurality of people existing in the real space or the virtual space at the current time, based on the coordinates of each body part of each person at each time stored in the storage unit 13 by the coordinate acquisition unit 14.

[0071] The process proceeds to step S112, where the excitement determination unit 12 determines whether or not the calculated ratio of "people performing motion" is equal to or greater than a first threshold. If the ratio of "people performing motion" is equal to or greater than the first threshold (YES in step S112), the excitement determination unit 12 determines that the entire real space or virtual space is excited, and the process proceeds to step S12 in Fig. 3. On the other hand, if the ratio of "people performing motion" is lower than the first threshold (NO in step S112), the process returns to step S111 in Fig. 4.

[0072] Next, the second excitement determination process will be described with reference to Fig. 5. In step S113 in Fig. 5, the excitement determination unit 12 calculates the ratio of "people performing motions" among a plurality of people existing in the real space or the virtual space at the current time, based on the coordinates of each body part of each person at each time stored in the storage unit 13 by the coordinate acquisition unit 14.

[0073] The process proceeds to step S114, where the excitement determination unit 12 determines whether the calculated ratio of "people performing motion" is equal to or greater than a first threshold. If the ratio of "people performing motion" is equal to or greater than the first threshold (YES in step S114), the process proceeds to step S115. On the other hand, if the ratio of "people performing motion" is lower than the threshold (NO in step S114), the process returns to step S113.

[0074] In step S115, the excitement determination unit 12 acquires, for example, a history of statistical values ​​of the volume in the real space from the time when the performance or playback of the music in the real space or virtual space begins to the current time, which is stored in the memory unit 13 by the volume acquisition unit 11.

[0075] The process proceeds to step S116, where the excitement determination unit 12 determines whether the acquired statistical value of the sound volume in the real space up to the current time is equal to or greater than the second threshold. If the statistical value of the sound volume in the real space up to the current time is equal to or greater than the second threshold (YES in step S116), the excitement determination unit 12 determines that the real space or the entire virtual space is excited, and the process proceeds to step S12 in Fig. 3. On the other hand, if the statistical value of the sound volume in the real space up to the current time is lower than the second threshold (NO in step S116), the process returns to step S113 in Fig. 5.

[0076] 3, the person extraction unit 15 extracts a specific person from among a plurality of people existing in a real space or a virtual space. When the display target extraction device 10 is applied to a real space, the person extraction unit 15 randomly extracts a person, for example, by randomly specifying position information of each seat in the real space captured by the imaging device 40. When the display target extraction device 10 is applied to a virtual space, the person extraction unit 15 randomly extracts a person, for example, by randomly specifying an account of a person existing in the virtual space.

[0077] The process proceeds to step S13, where the first subject determination unit 16 acquires the movement of the person extracted by the person extraction unit 15. Specifically, the first subject determination unit 16 acquires the coordinates of each body part of the extracted person at each time from the coordinates of each body part of each person at each time acquired by the coordinate acquisition unit 14 and stored in the storage unit 13.

[0078] The process proceeds to step S14, where the first subject determination unit 16 determines whether the movement of the extracted person is a repetitive movement based on the change in coordinates of each part of the extracted person at each time. If the movement of the extracted person is a repetitive movement (YES in step S14), the process proceeds to step S15. On the other hand, if the movement of the extracted person is not a repetitive movement (NO in step S14), the process returns to step S11.

[0079] In step S15, the first subject determination unit 16 determines whether the rhythm of the extracted person's movement matches the model rhythm based on the change in coordinates of each part of the extracted person at each time. The first subject determination unit 16 calculates the difference between the rhythm of the extracted person's movement and the model rhythm, and determines that the rhythm of the extracted person's movement matches the model rhythm when the calculated difference is within a second range. The first subject determination unit 16 determines that the rhythm of the extracted person's movement does not match the model rhythm when the difference between the rhythm of the extracted person's movement and the model rhythm is outside the second range. The model rhythm at each time is stored in the storage unit 13 in advance. If the rhythm of the extracted person's movement matches the model rhythm (YES in step S15), the process proceeds to step S16. On the other hand, if the rhythm of the extracted person's movement does not match the model rhythm (NO in step S15), the process returns to step S11.

[0080] In step S16, the first subject determination unit 16 determines whether the extracted feature amount of the person's movement matches the feature amount of the model movement stored in advance in the storage unit 13 based on the change in coordinates of each part of the extracted person at each time. The first subject determination unit 16 calculates the difference between the feature amount of the extracted person's movement at each time and the feature amount of the model movement, and determines that the extracted feature amount of the person's movement matches the feature amount of the model movement when the calculated difference is within a third range. The first subject determination unit 16 determines that the extracted feature amount of the person's movement does not match the feature amount of the model movement when the difference between the extracted feature amount of the person's movement and the feature amount of the model movement is outside the third range. If the extracted feature amount of the person's movement matches the feature amount of the model movement (YES in step S16), the process returns to step S11. On the other hand, if the extracted feature amount of the person's movement does not match the feature amount of the model movement (NO in step S16), the process proceeds to step S17.

[0081] In step S17, the first subject determination unit 16 determines the extracted person as a first subject, who is a person whose movement is not similar to the model movement.

[0082] The process proceeds to step S18, where the display control unit 17 displays the determined first target person on a display device (not shown) such as a monitor or screen in the real space or virtual space for a predetermined time.

[0083] The process proceeds to step S19, where the display control unit 17 determines whether or not a predetermined time has elapsed since the display of the first target person started. If the predetermined time has elapsed since the display of the first target person started (YES in step S19), the process proceeds to step S20. On the other hand, if the predetermined time has not elapsed since the display of the first target person started (NO in step S19), the process returns to step S19.

[0084] In step S20, the display control unit 17 outputs a signal to the display device to instruct it to end the display of the first target person, and ends the process in FIG.

[0085] Next, an example of a process in which the display target extraction device 10 according to an embodiment displays a second target person will be described with reference to Fig. 6. The display target extraction device 10 performs, for example, the process in Fig. 3 and the process in Fig. 6 in parallel.

[0086] 6, the person extraction unit 15 extracts a person from among a plurality of people existing in a real space or a virtual space. When the display target extraction device 10 is applied to a real space, the person extraction unit 15 randomly extracts a person, for example, by randomly specifying position information of the audience seats in the space imaged by the imaging device 40. When the display target extraction device 10 is applied to a virtual space, the person extraction unit 15 randomly extracts a person, for example, by randomly specifying an account of a person existing in the virtual space.

[0087] The process proceeds to step S22, where the display control unit 17 displays the extracted person as a second subject on a display device (not shown) in the real space or the virtual space for a predetermined time.

[0088] The process proceeds to step S23, where the display control unit 17 determines whether or not the first target person has been determined by the first target person determination unit 16 in the process of Fig. 3. If the first target person has been determined (YES in step S23), the process proceeds to step S25. On the other hand, if the first target person has not been determined (NO in step S23), the process proceeds to step S24.

[0089] In step S24, the display control unit 17 determines whether or not a predetermined time has elapsed since the display of the second target person started. If the predetermined time has elapsed since the display of the second target person started (YES in step S24), the process proceeds to step S25. On the other hand, if the predetermined time has not elapsed since the display of the second target person started (NO in step S24), the process returns to step S24.

[0090] In step S25, the display control unit 17 outputs a signal to the display device to instruct it to end the display of the second target person, and ends the process in FIG.

[0091] (Action and effect) As described above, the display target extraction device 10 according to an embodiment includes the excitement determination unit 12, the person extraction unit 15, the first target person determination unit 16, and the display control unit 17. The excitement determination unit 12 determines whether or not the entire real space or virtual space in which a plurality of people exist is excited. The person extraction unit 15 extracts a specific person from the plurality of people. When the excitement determination unit 12 determines that the entire real space or virtual space is excited, if the difference between the rhythm of the movement of the person extracted by the person extraction unit 15 and the model rhythm is within a second range and the difference between the feature amount of the movement of the person extracted by the person extraction unit 15 and the feature amount of the model action is outside a third range, the first target person determination unit 16 determines the person extracted by the person extraction unit 15 as the first target person. The display control unit 17 displays the first target person determined by the first target person determination unit 16 on the display device.

[0092] According to the display target extraction device 10 of one embodiment, when the entire real or virtual space where multiple people exist is lively, such as a live concert in the real or virtual space, a person (first target person) whose movements generally match the rhythm of the music currently being played or reproduced in the real or virtual space and are not similar to the model movements can be automatically extracted, and the extracted person can be displayed on a display device. This reduces the burden on the administrator when extracting a person who can convey the atmosphere of the real or virtual space as a display target, without being limited to a person with a fixed movement.

[0093] Furthermore, the display control unit 17 of the display target extraction device 10 according to an embodiment displays the second target on the display device, assuming the person extracted by the person extraction unit 15 as the second target, when the first target determination unit 16 has not determined the first target. After displaying the second target on the display device, the display control unit 17 determines whether the first target has been determined by the first target determination unit 16, and when the first target has been determined by the first target determination unit 16, ends the display of the second target and displays an image of the first target on the display device. On the other hand, when the first target has not been determined by the first target determination unit 16, the display of the second target continues for a predetermined time.

[0094] According to the display target extraction device 10 according to an embodiment, when a person (first target person) whose movement matches the rhythm of the music being played or reproduced in the real space or virtual space and is not similar to the model movement is not extracted, a person (second target person) who is a display target can be extracted and displayed on the display device. This makes it possible to prevent only the first target person from being displayed on the display device, and to maintain the atmosphere of the real space or virtual space. In addition, when the second target person is displayed, if the first target person is determined, the display target is automatically switched from the second target person to the first target person. This makes it possible to automatically switch the display target between the first target person and the second target person, thereby reducing the burden on the administrator.

[0095] Moreover, the display target extraction device 10 according to an embodiment further includes a coordinate acquisition unit 14. The coordinate acquisition unit 14 acquires coordinates of each body part of each of the multiple persons at each time from a sensor (gyro sensor 30) that each of the multiple persons wears on the body in real space. The first subject determination unit 16 determines whether or not a difference between a feature amount of the movement of the person extracted by the person extraction unit 15 and a feature amount of the model action falls outside a third range, based on the coordinates of each body part of the person extracted by the person extraction unit 15 at each time.

[0096] According to the display target extraction device 10 according to an embodiment, a first target person can be automatically extracted based on the coordinates of each body part at each time using a sensor worn by a person in a space.

[0097] Furthermore, the coordinate acquisition unit 14 of the display target extraction device 10 according to an embodiment may further acquire the angular velocity of each part of each of the multiple people at each time from the gyro sensor 30. The first subject determination unit 16 may determine whether or not a difference between the feature amount of the movement of the person extracted by the person extraction unit 15 and the feature amount of the model action is outside a third range, based on the coordinates and angular velocity of each part of the person extracted by the person extraction unit 15 at each time. By adding a parameter for comparing the feature amount of the extracted person's movement with the feature amount of the model action, it is possible to improve the accuracy of determining whether the feature amounts of the movement match.

[0098] Moreover, the excitement determination unit 12 of the display target extraction device 10 according to an embodiment calculates the ratio of people among a plurality of people whose difference between the model rhythm and the current movement rhythm is within a first range. When the calculated ratio is equal to or greater than a threshold, the excitement determination unit 12 determines that the entire real space or virtual space is excited.

[0099] According to an embodiment of the display target extraction device 10, the excitement level of the entire real space or virtual space can be automatically determined based on the percentage of people whose current movement rhythm matches the rhythm of the music being played or reproduced in the real space or virtual space.

[0100] As described above, the embodiment of the present invention has been described, but the description and drawings forming a part of this disclosure should not be understood as limiting this invention. From this disclosure, various alternative embodiments, examples and operating techniques will become apparent to those skilled in the art. [Explanation of symbols]

[0101] 10 Display object extraction device 12 Excitement Judgment Section 15 Person extraction part 16 First Subject Judgment Section 17 Display control section

Claims

1. an excitement determination unit that determines whether or not an entire real space or a virtual space in which a plurality of people exist is excited; a person extraction unit that extracts a specific person from the plurality of people; a first subject determination unit that determines, when the excitement determination unit determines that the entire real space or the virtual space is excited, a difference between the rhythm of the movement of the person extracted by the person extraction unit and an example rhythm is within a first range and a difference between a feature amount of the movement of the person extracted by the person extraction unit and a feature amount of an example action is outside a second range, and determines the person extracted by the person extraction unit as a first subject; a display control unit that displays the first subject determined by the first subject determination unit on a display device; A display object extraction device comprising:

2. The display control unit is When the first target person determination unit has not determined the first target person, a person extracted by the person extraction unit is determined as a second target person, and the second target person is displayed on the display device; After displaying the second target person on the display device, the first target person determination unit determines whether or not the first target person has been determined; When the first target person is determined by the first target person determination unit, the display of the second target person is terminated and the first target person is displayed on the display device; If the first target person has not been determined by the first target person determination unit, the display of the second target person is continued for a predetermined time. The display target extraction device according to claim 1 .

3. a coordinate acquisition unit that acquires coordinates of each part of each of the plurality of persons at each time from a sensor worn on a body of each of the plurality of persons in real space; The first subject determination unit determines whether or not a difference between a feature amount of the movement of the person extracted by the person extraction unit and a feature amount of the model action at each time point is outside the second range, based on the coordinates of each part of the person extracted by the person extraction unit. The display target extraction device according to claim 1 .

4. the coordinate acquisition unit further acquires from the sensor an angular velocity of each part of each of the plurality of persons at each time; The first subject determination unit determines whether or not a difference between a feature amount of the movement of the person extracted by the person extraction unit and a feature amount of the model action is outside the second range based on the coordinates and the angular velocity of each part of the person extracted by the person extraction unit at each time point. The display target extraction device according to claim 3 .

5. The excitement determination unit is calculating a ratio of people among the plurality of people whose difference between the model rhythm and the current movement rhythm is within a third range; When the ratio is equal to or greater than a threshold value, it is determined that the entire real space or virtual space is raised. The display target extraction device according to claim 1 .

Images