Information processing system, method, program, and storage medium

Abstract

To provide an information processing system for associating a text object displayed in an AR (augmented-reality) space with a speaker using gesture operation, a method, a program, and a storage medium.SOLUTION: In an information processing system 1, at least one head-mounted display includes: a sound collection unit which collects voice; a text conversion unit which converts the voice collected by the sound collection unit into a text; a display control unit which displays, on an AR space via the display unit of the head-mounted display, the text converted by the text conversion unit, as a text object; a gesture recognition unit which recognizes the gesture operation of a user on the text object; an imaging unit which images a real space; a human body detection unit which detects one or more human bodies from the image captured by the imaging unit; and an association unit which associates the text object with one human body detected by the human body detection unit, in accordance with the gesture operation of the user recognized by the gesture recognition unit.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an information processing system, method, program, and storage medium.

Background Art

[0002] Conventionally, an AR system that provides a user with an explanation or information for an unknown word from audio data around an AR (augmented reality) device has been known (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, conventionally, nothing has been proposed regarding associating a text object displayed in an AR space with a speaker by a gesture operation.

[0005] The problem to be solved by the present invention is to associate a text object displayed in an AR space with a speaker by a gesture operation.

Means for Solving the Problems

[0006] [1] An information processing system according to one aspect is an information processing system including at least one head-mounted display, a sound collection unit that collects sound, a text conversion unit that converts the sound collected by the sound collection unit into text, a display control unit that displays the text converted by the text conversion unit as a text object on an AR space via a display unit of the head-mounted display, A gesture recognition unit that recognizes a user's gesture operation on the text object, An imaging unit that images the real space, A human body detection unit that detects one or more human bodies from the image captured by the imaging unit, An association unit that associates the text object with one human body detected by the human body detection unit according to the gesture operation of the user recognized by the gesture recognition unit, An information processing system comprising the above.

[0007] [2] The information processing system according to one aspect is the information processing system described in the above [1], wherein The information processing system further includes a speaker identification unit that identifies a speaker who is speaking to the user from the one or more human bodies detected by the human body detection unit, The display control unit highlights and displays the human body identified as the speaker by the speaker identification unit in the AR space.

[0008] [3] The information processing system according to one aspect is the information processing system described in the above [1] or [2], wherein The display control unit displays an operator on the AR space together with the text object, The association unit associates the text object with one human body detected by the human body detection unit according to a gesture operation that connects the operator to the one human body detected by the human body detection unit.

[0009] [4] The information processing system according to one aspect is the information processing system described in the above [3], wherein The operator is a string-shaped object.

[0010] [5] The information processing system according to one aspect is the information processing system described in any one of the above [1] to [4], wherein The display control unit moves and displays the text object in the AR space according to the gesture operation of the user recognized by the gesture recognition unit.

[0011] [6] The information processing system according to one aspect is the information processing system described in any one of [1] to [5] above, The display control unit enlarges or reduces and displays the text object in the AR space according to the gesture operation of the user recognized by the gesture recognition unit.

[0012] [7] The information processing system according to one aspect is the information processing system described in any one of [1] to [6] above, The display control unit superimposes and displays one or more text objects associated with a single human body detected by the human body detection unit in time series in the AR space.

[0013] [8] The information processing system according to one aspect is the information processing system described in any one of [1] to [7] above, It further includes a text extraction unit that extracts specific text from a plurality of texts included in the text object.

[0014] [9] The information processing system according to one aspect is the information processing system described in any one of [1] to [8] above, It further includes a priority setting unit that sets a priority according to the content of the text object.

[0015]

[10] The information processing system according to one aspect is the information processing system described in [9] above, The display control unit displays at least one priority indicator in the AR space, The priority setting unit sets the priority according to recognizing a gesture operation in which the gesture recognition unit selects the priority indicator.

[0016]

[11] The information processing system according to one aspect is the information processing system described in any one of [1] to

[10] above, It further includes a transmission unit that transmits information about the text object to another head-mounted display.

[0017]

[12] A method according to one aspect is a method for controlling an information processing system including at least one head-mounted display, the method including the steps of collecting sound, converting the collected sound into text, displaying the converted text as a text object on an AR space via a display unit of the head-mounted display, recognizing a user gesture operation on the text object, imaging the real space, detecting one or more human bodies from the captured image, associating the text object with the detected one human body according to the user gesture operation, and including.

[0018]

[13] A program according to one aspect is a program for causing a computer to execute a method for controlling an information processing system including at least one head-mounted display, the method including the steps of collecting sound, converting the collected sound into text, displaying the converted text as a text object on an AR space via a display unit of the head-mounted display, recognizing a user gesture operation on the text object, imaging the real space, detecting one or more human bodies from the captured image, associating the text object with the detected one human body according to the user gesture operation, and including.

[0019]

[14] A storage medium according to one aspect is a computer-readable storage medium storing the program described in

[13] above.

Advantages of the Invention

[0020] According to the present invention, a text object displayed on an AR space can be associated with a speaker by a gesture operation.

Brief Description of the Drawings

[0021]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 6

Figure 7

Figure 8

Figure 9

Modes for Carrying Out the Invention

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, components having the same functions are denoted by the same reference numerals, and detailed descriptions of the components denoted by the same reference numerals will not be repeated.

[0023] (Functional Configuration of Information Processing System) FIG. 1 is a diagram showing a schematic configuration of an information processing system 1 according to the present embodiment. As shown in FIG. 1, the information processing system 1 according to the present embodiment includes a head-mounted display 2 (terminal device) used by a user and a server 3.

[0024] The head-mounted display 2 and the server 3 are communicably connected to each other via a network 4 such as the Internet. The network 4 may be either a wired line or a wireless line, and the type and form of the line are not limited. At least a part of the head-mounted display 2 and the server 3 is realized by a computer (information processing device). The head-mounted display 2 is worn on the user's head and provides an AR (augmented reality) space in the user's field of view. It is assumed that there are a number of head-mounted displays 2 according to the number of users who use the information processing system 1. The head-mounted display 2 is used, for example, by a user (store clerk) when serving customers in a store such as a restaurant. In this case, the information obtained by each user (store clerk) can be shared among the users (store clerks) via the network 4.

[0025] (Hardware Configuration) Next, the hardware configuration of the information processing system 1 according to the present embodiment will be described. FIG. 2 is a block diagram showing an example of the hardware configuration of the head-mounted display 2 and the server 3 included in the information processing system 1 according to the present embodiment.

[0026] In the head-mounted display 2, the CPU 201 is a processing device that controls the operation of the entire head-mounted display 2. The ROM 202 is a non-volatile memory that stores control programs and various data executed by the CPU 201. The RAM 203 is a volatile memory used for the load area and work area of the program executed by the CPU 201. The storage device 204 is a storage means for storing various information, and it may be built into the head-mounted display 2 main body or have a detachable storage medium. The gesture sensor 205 is a sensor (e.g., an infrared sensor) that recognizes gesture operations performed by the user of the head-mounted display 2. Here, the gesture operation is defined to widely include not only operations by the user's fingers but also gestures using the user's body. For example, gaze input by blinking or eye movement is also included in the gesture operation. The display device 206 is a display device that superimposes and displays various information (text objects, user interfaces, etc.) in the real space. That is, the head-mounted display 2 provides the AR space to the user via this display device 206. The imaging device 207 is a photoelectric conversion element that captures an image of a subject. The microphone 208 is a sound collection device that collects sounds around the head-mounted display 2. The communication I / F (interface) 209 is an interface for connecting to the network 4. The bus 210 is a bus line that interconnects the above components.

[0027] In the server 3, the CPU 301 is a processing device that controls the operation of the entire server 3. The ROM 302 is a non-volatile memory that stores control programs and various data executed by the CPU 301. The RAM 303 is a volatile memory used for the load area and work area of the program executed by the CPU 301. The storage device 304 is a storage means for storing various information, and it may be built into the server 3 main body or have a detachable storage medium. The communication I / F (interface) 305 is an interface for connecting to the network 4. The bus 306 is a bus line that interconnects the above components.

[0028] (Functional configuration) Next, the functional configuration of the information processing system 1 according to the present embodiment will be described. FIG. 3 is a diagram showing an example of the functional configuration of the information processing system 1 according to the present embodiment.

[0029] First, the functional configuration of the head-mounted display 2 will be described. As shown in FIG. 3, the head-mounted display 2 includes a communication unit 21, a control unit 22 that controls the overall operation of the head-mounted display 2, a sound collection unit 23 that collects sounds around the head-mounted display 2, a display unit 24 that displays an AR object in the user's field of view of the head-mounted display 2, an imaging unit 25 that captures an image of a subject, and a storage unit 26 that stores various types of information.

[0030] The communication unit 21 is a communication interface between the head-mounted display 2 and the network 4. The communication unit 21 transmits and receives information between the head-mounted display 2 and the server 3 or another head-mounted display 2 via the network 4. That is, the communication unit 21 functions as a transmission unit and / or a reception unit. The control unit 22 includes a display control unit 22a, a text conversion unit 22b, a gesture recognition unit 22c, a human body detection unit 22d, an association unit 22e, a speaker identification unit 22f, a text extraction unit 22g, and a priority setting unit 22h.

[0031] The display control unit 22a controls the display unit 24 to display an AR object such as a text object converted by the text conversion unit described later in the AR space.

[0032] The text conversion unit 22b converts the voice collected by the sound collection unit 23 described later into text. For example, the text conversion unit 22b performs acoustic analysis on the voice data of the voice collected by the sound collection unit 23, and performs text conversion by analyzing the voice data after the acoustic analysis using an acoustic model learned in advance. The information converted into text by the text conversion unit 22b is displayed on the AR space via the display unit 24 by the display control unit 22a.

[0033] The gesture recognition unit 22c recognizes a user's gesture operation on the text object displayed on the AR space by the display control unit 22a.

[0034] The human body detection unit 22d detects one or more human bodies from the image captured by the imaging unit 25. For the human body detected by the human body detection unit 22d, the display control unit 22a may display a bounding box on the AR space.

[0035] The association unit 22e associates the text object with one human body detected by the human body detection unit 22d according to the user's gesture operation recognized by the gesture recognition unit 22c. Specifically, the association unit 22e associates the text object with the human body using an operator displayed on the AR space by the display control unit 22a together with the text object. Here, the association of the text object with the human body means associating the content of the text object with the speech content of the human body. For example, an ID may be assigned to each detected human body, and the information of the text object may be associated with this ID and stored in the storage unit 26 of the head-mounted display 2 or the storage unit 33 of the server 3.

[0036] FIG. 5A and FIG. 5B are diagrams for explaining the association of a text object with a human body by the association unit 22e. As shown in FIG. 5A, an operator 51 (string-shaped object) is displayed on the AR space of the user of the head-mounted display 2 together with the text object 50. Then, when the user performs a gesture operation to connect the operator 51 to the human body, as shown in FIG. 5B, the text object 50 is associated with the human body 52.

[0037] Although an example in which the operator 51 is displayed as a string-shaped object has been described as an example, the operator 51 may have other shapes as long as it can be operated by the user's gesture operation.

[0038] Returning to FIG. 3, the speaker identification unit 22f identifies a speaker (person) who is speaking to the user of the head-mounted display 2 from the one or more human bodies detected by the human body detection unit. For example, the speaker identification unit 22f may identify a human body located substantially at the center of the field of view (imaging area imaged by the imaging unit 25) of the user of the head-mounted display 2 as the speaker. Further, an emphasis display indicating that the speaker identification unit 22f has identified a human body as the speaker may be displayed on the AR space. As the emphasis display, for example, an outline or a bounding box may be displayed around the human body identified as the speaker, or an object indicating "speaker" may be displayed near the human body identified as the speaker. Thereby, since the user can easily perceive the speaker, the text object can be more easily associated with the human body.

[0039] The text extraction unit 22g extracts specific text from a plurality of texts included in the text object. For example, the text may be extracted using a learning model that has machine-learned specific text as learning data. In this case, for example, when the user is a restaurant staff, text related to the order content, the number of orders, the customer's requests, etc. may be used as learning data.

[0040] The priority setting unit 22h sets a priority according to the content of the text object. Specifically, the priority setting unit 22h may set the priority using the priority indicator displayed by the display control unit 22a in the AR space.

[0041] FIG. 9 is a diagram for explaining priority setting using the priority indicator by the priority setting unit 22h. As shown in FIG. 9, in the AR space, a priority indicator 90 for the user to set the priority by a gesture operation is displayed. The priority indicator 90 includes a plurality of priority indicators 90a to 90e indicating different priorities. For example, the priority may increase from the priority indicator 90a to the priority indicator 90e. Note that the priority indicator may represent its priority by characters, figures, symbols, etc.

[0042] Alternatively, the priority setting unit 22h may automatically set the priority according to the content of the text object. In this case, for example, a learning model obtained by machine learning the content of the text object and its priority as learning data may be used to estimate the priority, and the priority setting unit 22h may set the priority based on the estimation result. The priority set by the priority setting unit 22h is transmitted to the server 3 and other head-mounted displays 2 via the communication unit 21. Thereby, information about the person with whom the user is dealing can be shared with other users. Note that the content shared with other users may include the content of the text object in addition to the priority.

[0043] The sound collection unit 23 collects the sound around the head-mounted display 2 by the microphone 208. The sound information collected by the sound collection unit 23 is sent to the control unit 22, and various processes are performed by the control unit 22.

[0044] The display unit 24 is a see-through type display for displaying various information (images and sounds) to the user from the head-mounted display 2 in the AR space.

[0045] The imaging unit 25 captures an image of a subject by a camera module including an imaging device of the head-mounted display 2. The image captured by the imaging unit 25 is output to the control unit 22, subjected to various image processes, and displayed on the display unit 24 by the display control unit 22a, or transmitted to the server 3 via the network 4 by the communication unit 21.

[0046] The storage unit 26 is a data storage such as a built-in memory or an external memory (such as an SD memory card), for example. The storage unit 26 stores various data handled by the control unit 22, various information downloaded from the server 3 via the network 4 by the communication unit 21, images captured by the imaging unit 25, and the like. Note that the storage unit 26 does not necessarily have to be provided inside the head-mounted display 2, and part or all of the storage unit 26 may be provided in another device communicably connected to the head-mounted display 2 via the network 4.

[0047] Next, the functional configuration of the server 3 will be described. As shown in FIG. 3, the server 3 includes a communication unit 31, a control unit 32, and a storage unit 33.

[0048] The communication unit 31 is a communication interface between the server 3 and the network 4. The communication unit 31 transmits and receives information between the server 3 and the head-mounted display 2 via the network 4.

[0049] The control unit 32 controls the overall operation of the server 3. Note that the control unit 32 may include at least one function of the control unit 22 of the head-mounted display 2.

[0050] The storage unit 33 is a data storage such as a built-in memory or an external memory (such as an SD memory card), for example. The storage unit 33 stores various data handled by the control unit 32, various information received from the head-mounted display 2 via the network 4 by the communication unit 31, and various databases (DBs), and the like.

[0051] Note that the storage unit 33 does not necessarily have to be provided within the server 3, and part or all of the storage unit 33 may be provided within another device that is communicably connected to the server 3 via the network 4.

[0052] Next, a gesture operation on a text object will be described.

[0053] FIG. 6 is a diagram for explaining an enlargement / reduction operation of a text object. As shown in FIG. 6, a user of the head-mounted display 2 can enlarge / reduce a text object 50 displayed on the AR space by a gesture operation. Further, the user can also move the text object 50 to an arbitrary position on the AR space by a gesture operation.

[0054] FIG. 7 is a diagram for explaining a display example of a group of text objects that have been spoken in the past. As shown in FIG. 7, individual text objects 50 included in a group of text objects 70 that have been spoken in the past are displayed on the AR space so as to overlap each other. For example, the group of text objects 70 that have been spoken in the past may be displayed in chronological order. In this case, for example, the group of text objects 70 may be displayed with the text objects 50 stacked from the front to the back in the order of the most recent / oldest speech date and time of the text objects 50.

[0055] And in FIG. 7, a user of the head-mounted display 2 can select an arbitrary text object 50 from among the group of text objects 70 by a gesture operation.

[0056] FIG. 8 is a diagram for explaining an operation of manually extracting the content of a text object. As shown in FIG. 8, a user of the head-mounted display 2 may be able to extract arbitrary text from the content (text) included in the text object by a gesture operation.

[0057] (An example of an operation) Next, an example of the operation of the information processing system 1 will be described. FIG. 4 is a flowchart showing the operation of the information processing system 1.

[0058] First, the sound collection unit 23 of the head-mounted display 2 collects the sound around the head-mounted display 2 (step S1).

[0059] Next, the text conversion unit 22b converts the sound collected by the sound collection unit 23 into text (step S2).

[0060] Next, the display control unit 22a displays the text converted by the text conversion unit 22b as a text object on the AR space via the display unit 24 of the head-mounted display 2 (step S3).

[0061] Next, the gesture recognition unit 22c recognizes the user's gesture operation on the text object (step S4).

[0062] Next, the human body detection unit 22d detects one or more human bodies from the image captured by the imaging unit 25

[0063] (step S5).

[0064] Then, the association unit 22e associates the text object with the one human body detected by the human body detection unit 22d according to the user's gesture operation recognized by the gesture recognition unit 22c (step S6).

[0065] As described above, according to the present embodiment, the information processing system 1 is an information processing system including at least one head-mounted display, and includes a sound collection unit that collects sound, a text conversion unit that converts the sound collected by the sound collection unit into text, a display control unit that displays the text converted by the text conversion unit as a text object on an AR space via a display unit of the head-mounted display, a gesture recognition unit that recognizes a user's gesture operation on the text object, an imaging unit that images a real space, a human body detection unit that detects one or more human bodies from the image imaged by the imaging unit, and an association unit that associates the text object with one human body detected by the human body detection unit according to the user's gesture operation recognized by the gesture recognition unit. Therefore, the text object displayed on the AR space can be associated with the speaker by a gesture operation.

[0066] Any part or all of the above-described functional units may be realized by a program. The program mentioned in this specification may be non-temporarily recorded on a computer-readable recording medium and distributed, or may be distributed via a communication line (including wireless communication) such as the Internet, or may be distributed in a state installed on an arbitrary terminal.

[0067] Based on the above description, those skilled in the art may be able to conceive of additional effects and various modifications of the present invention. However, the aspects of the present invention are not limited to the above-described individual embodiments. Various additions, changes, and partial deletions are possible without departing from the conceptual ideas and spirit of the present invention derived from the content defined in the claims and their equivalents.

[0068] For example, what is described as one device (or component, the same hereinafter) in this specification (including what is depicted as one device in the drawings) may be implemented by a plurality of devices. Conversely, what is described as a plurality of devices in this specification (including what is depicted as a plurality of devices in the drawings) may be implemented by one device. Alternatively, some or all of the means and functions included in a certain device (e.g., a server) may be included in another device (e.g., a head-mounted display).

[0069] Also, not all of the matters described in this specification are essential requirements. In particular, matters described in this specification but not described in the claims can be regarded as optional additional matters.

[0070] It should be noted that the applicant only knows the publicly known inventions described in the documents in the "Prior Art Documents" section of this specification, and the present invention is not necessarily intended to solve the problems in the publicly known inventions in the same documents. The problems to be solved by the present invention should be determined in consideration of the entire specification. For example, in this specification, if there is a description that a predetermined effect is achieved by a specific configuration, it can also be said that the problem that is the reverse of the predetermined effect is solved. However, it is not necessarily the intention that such a specific configuration is an essential requirement.

Explanation of Reference Numerals

[0071] 1 Information processing system 2 Head-mounted display 21 Communication unit 22 Control unit 22a Display control unit 22b Text conversion unit 22c Gesture recognition unit 22d Human body detection unit 22e Association unit 22f Speaker identification unit 22g Text extraction unit 22h Priority setting unit 23 Sound collection unit 24 Display unit 25 Imaging unit 26 Memory unit 3 Server 31 Communication unit 32 Control unit 33 Memory unit

Claims

1. An information processing system comprising at least one head-mounted display, a sound collection unit that collects sound, a text conversion unit that converts the sound collected by the sound collection unit into text, a display control unit that displays, as a text object, the text converted by the text conversion unit on an AR space via a display unit of the head-mounted display, a gesture recognition unit that recognizes a user's gesture operation on the text object, an imaging unit that images the real space, a human body detection unit that detects one or more human bodies from the image captured by the imaging unit, and an association unit that associates the text object with one human body detected by the human body detection unit according to the gesture operation of the user recognized by the gesture recognition unit. An information processing system comprising the above.

2. The information processing system according to claim 1, further comprising a speaker identification unit that identifies a speaker who is speaking to the user from the one or more human bodies detected by the human body detection unit, wherein the display control unit highlights the human body identified as the speaker by the speaker identification unit on the AR space.

3. The display control unit displays an operator on the AR space together with the text object, and the association unit associates the text object with one human body detected by the human body detection unit according to a gesture operation of connecting the operator to the one human body detected by the human body detection unit. The information processing system according to claim 1.

4. The information processing system according to claim 3, wherein the operator is a string-shaped object.

5. The information processing system according to claim 1, wherein the display control unit moves and displays the text object on the AR space according to the gesture operation of the user recognized by the gesture recognition unit.

6. The information processing system according to claim 1, wherein the display control unit enlarges or reduces and displays the text object on the AR space according to the gesture operation of the user recognized by the gesture recognition unit.

7. The information processing system according to claim 1, wherein the display control unit superimposes and displays one or more text objects associated with one human body detected by the human body detection unit on the AR space in time series.

8. The information processing system according to claim 1, further comprising a text extraction unit that extracts specific text from a plurality of texts included in the text object.

9. The information processing system according to claim 1, further comprising a priority setting unit that sets a priority according to the content of the text object.

10. The display control unit displays at least one priority indicator in the AR space. The information processing system according to claim 9, wherein the priority setting unit sets the priority in response to the gesture recognition unit recognizing a gesture operation for selecting the priority indicator.

11. The information processing system according to claim 1, further comprising a transmission unit that transmits information about the text object to another head-mounted display.

12. A method for controlling an information processing system including at least one head-mounted display, the method comprising: collecting sound; converting the collected sound into text; displaying the converted text as a text object in an AR space via a display unit of the head-mounted display; recognizing a user's gesture operation on the text object; imaging the real space; detecting one or more human bodies from the captured image; associating the text object with the detected one human body according to the user's gesture operation; and including.

13. A program for causing a computer to execute a method for controlling an information processing system including at least one head-mounted display, the method comprising: collecting sound; converting the collected sound into text; displaying the converted text as a text object in an AR space via a display unit of the head-mounted display; recognizing a user's gesture operation on the text object; imaging the real space; detecting one or more human bodies from the captured image; associating the text object with the detected one human body according to the user's gesture operation; and including.

14. A computer-readable storage medium storing the program according to claim 13.

Images