Grouping system, program, and control device for terminal devices

Abstract

To flexibly group a plurality of terminal devices.SOLUTION: In a system 100 configured by connecting a plurality of terminal devices 10 and a control apparatus 20 via a communication line, each of the terminal devices 10 includes a sound collection unit. The terminal devices 10 and the control apparatus 20 analyze sound signals acquired by the sound collection units. The control apparatus 20 groups one or more terminal devices 10 of the multiple terminal devices 10, on the basis of analysis information obtained by a sound analysis unit. The control apparatus 20 acquires instruction information to the terminal devices 10 on the basis of the analysis information of the sound signals, and generates command information to respond to the instruction information for at least one specific terminal device 10 belonging to one group.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a system, program, and control system for grouping terminal devices based on voice data acquired by the terminal devices. [Background technology]

[0002] In recent years, the digital transformation (DX) of on-site work has become an urgent issue in order to address various problems such as the increasing complexity of on-site work, the declining workforce, and the transfer of skills held by skilled workers. As one solution to these problems, a remote support system has been proposed that connects wearable devices worn by on-site workers with computers operated by support staff via the internet, etc., to share voice and visual information (Patent Document 1).

[0003] When using such remote support systems, wearable devices are often distributed (loaned) to workers before work begins on-site. In large-scale projects, workers are often divided into groups, and devices are distributed to each group. In this case, it is necessary to associate multiple devices with groups of workers, but generally, each device is pre-assigned to a group, and the devices assigned to the designated group are distributed to the workers belonging to that group. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7023022 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to assign wearable devices to worker groups, it is generally necessary to assign groups to each device before distributing them to workers. However, if devices are grouped in advance, tasks must be performed when distributing the devices, such as checking which group the device belongs to and whether the device corresponds to the worker's group. For example, if a device is mixed up (for example, a device assigned to a different group is distributed to a worker belonging to a different group), confusion may arise in the chain of command from the support staff to the worker, potentially leading to serious mistakes or accidents on site. Furthermore, to prevent such mix-ups, careful checks must be performed when distributing the devices, which raises concerns about reduced work efficiency on site.

[0006] There may also be situations where a worker wants to select and wear, for example, the device with the most battery remaining from among multiple wearable devices. Since checking the battery remaining level of each device one by one is time-consuming, it would be convenient if the device with the most battery remaining among the multiple devices automatically responded to the worker's query. However, if groups are set for each device in advance as described above, it is possible that the multiple devices in front of the worker may belong to different groups. In such cases, it becomes unclear from which range the device with the most battery remaining should respond, making it difficult to achieve the convenient response described above.

[0007] Therefore, one object of the present invention is to propose a technology that allows multiple terminal devices to be flexibly grouped. Another object of the present invention is to propose a technology that allows one or more terminal devices to be easily identified from multiple grouped terminal devices. The present invention provides a means for achieving at least one of the above objects. [Means for solving the problem]

[0008] The first aspect of the present invention relates primarily to a system for grouping multiple terminal devices. The system according to the present invention includes multiple terminal devices and a control device. These terminal devices and the control device are connected via a communication line such as the Internet. Each of the multiple terminal devices includes a sound collection unit (microphone). The multiple terminal devices or the control device also include a voice analysis unit that analyzes the voice signals acquired by the sound collection unit. If a terminal device includes a voice analysis unit, it can analyze the voice signals acquired by the sound collection unit within its own terminal and transmit the analysis information to the control device. If the control device includes voice analysis, the terminal device transmits the voice signals acquired by the sound collection unit to the control device, and the control device analyzes the voice signals received from the terminal devices. The term "voice analysis" as used herein includes not only speech recognition (processing to read textual information from voice signals), but also speech frequency analysis, speaker emotion recognition, attitude recognition, language recognition, voiceprint analysis (voice characteristic analysis), etc. The control device includes a grouping unit that groups one or more of the multiple terminal devices based on the analysis information from the voice analysis unit. Note that "grouping" refers to grouping one or more terminal devices into the same group. For example, the grouping unit may group multiple terminal devices that are determined to have acquired audio signals within a predetermined similarity range as a result of the audio analysis.

[0009] As shown in the configuration above, by enabling the grouping of multiple terminal devices based on the voice signals acquired by each terminal device, it is possible to omit the prior setting of terminal device groups. For example, by deploying multiple terminal devices that are not yet grouped, and speaking to them, the terminal devices within the range of that voice can be set to the same group. This type of processing can also be performed, for example, after the terminal device (wearable device) has been attached to the worker.

[0010] In the system according to the present invention, the control device preferably further includes a command acquisition unit and a command generation unit. The command acquisition unit acquires command information for terminal devices based on analysis information from the voice analysis unit. The command generation unit generates command information in response to the command information acquired by the command acquisition unit for at least one specific terminal device belonging to a certain group. This makes it easy to identify which terminal device should respond to a command based on a user's voice from among a group of terminal devices assigned to the same group based on that user's voice.

[0011] In the system according to the present invention, the control device preferably further includes a terminal identification unit. The terminal identification unit identifies at least one terminal device that will respond to command information from among a plurality of terminal devices belonging to a certain group. The control device then transmits command information that responds to the command information to the terminal device identified by the terminal identification unit. This makes it possible to automatically cause one or more terminal devices to respond to a voice inquiry from a user.

[0012] In the system according to the present invention, the grouping unit of the control device may group together multiple terminal devices that have acquired audio signals within a predetermined time range and within a predetermined error range using the sound collection unit. By grouping terminal devices in this way based on the time at which the audio signals were acquired and the similarity of those audio signals, the accuracy of the grouping process can be improved.

[0013] In the system according to the present invention, each of the multiple terminal devices may further include a location information acquisition unit that acquires its own location information. In this case, it is preferable that the grouping unit of the control device groups together multiple terminal devices that are within a predetermined geographical range and within a predetermined time range, and that have acquired an audio signal within a predetermined error range using a sound collection unit. By grouping multiple terminals in this way, in addition to geographical information at the time the terminal device acquired the audio signal, the accuracy can be further improved.

[0014] In the system according to the present invention, each of the plurality of terminal devices may further include a short-range wireless communication unit. In this case, it is preferable that the grouping unit of the control device group together the plurality of terminal devices that are in a state of communication via the short-range wireless communication unit and that have acquired the audio within a predetermined error range using the sound collection unit within a predetermined time range. In this way, by further adding information on whether the terminal devices are communicating with each other by short-range wireless communication at the time the terminal device acquired the audio signal, the accuracy of grouping the plurality of terminals can be further improved.

[0015] A second aspect of the present invention relates to a program for causing a personal digital assistant (PDA) equipped with a sound collection unit (microphone) to function as a terminal device in the system according to the first aspect. Such a program may be downloadable to the PDA via the Internet, may be pre-installed in the PDA, or may be stored on a recording medium readable by the PDA, such as a CD-ROM.

[0016] A third aspect of the present invention relates to a control device. The control device according to the present invention is connected to a plurality of terminal devices, each including a sound collection unit, via a communication line. The plurality of terminal devices or the control device includes a sound analysis unit that analyzes sound acquired by the sound collection unit. The control device according to the present invention also includes a grouping unit that groups one or more of the plurality of terminal devices based on analysis information from the sound analysis unit.

[0017] A fourth aspect of the present invention is a program for causing a computer to function as the control device according to the third aspect. Such a program may be downloadable to the computer via the Internet, may be pre-installed on the computer, or may be stored on a computer-readable recording medium such as a CD-ROM. [Effects of the Invention]

[0018] According to the present invention, a plurality of terminal devices can be flexibly grouped, or one or more terminal devices can be easily identified from a plurality of grouped terminal devices. [Brief explanation of the drawings]

[0019] [Figure 1] Figure 1 shows an overall diagram of the system according to the present invention. [Figure 2] FIG. 2 shows an overview of information processing by the system according to the present invention. [Figure 3] FIG. 3 shows a neck-worn wearable device as an example of a terminal device. [Figure 4] FIG. 4 is a block diagram showing an example of the functional configuration of the terminal device and the control device included in the system according to the present invention. [Figure 5] FIG. 5 shows a main flow of the processing executed by a system according to an embodiment of the present invention. [Figure 6] Figure 6 shows a subflow of the grouping process shown in Figure 5. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the embodiments described below, but also includes appropriate modifications of the embodiments below within the scope obvious to those skilled in the art.

[0021] FIG. 1 shows a schematic diagram of the overall configuration of a system 100 according to one embodiment of the present invention. The system 100 according to this embodiment is preferably used in applications where a supporter remotely supports work performed by multiple workers on-site via the Internet. In particular, this system is designed to distribute terminal devices 10 to on-site workers and collectively manage the multiple terminal devices 10 using a control device 20. Furthermore, a supporter in a remote location can exchange voice and image data with the terminal devices 10 worn by the on-site workers via a support device 30.

[0022] As shown in FIG. 1, the system 100 includes multiple terminal devices 10, a control device 20, and an assistance device 30. Examples of the terminal device 10 include wearable devices. Examples of wearable devices include neck-worn devices, eyeglass-type devices, head-mounted devices, and wristwatch-type devices. In this embodiment, a neck-worn wearable device is used as the terminal device 10. Each of the terminal devices 10 is worn by a single worker. The control device 20 is connected to the multiple terminal devices 10 via the Internet and has the functions of, for example, receiving information acquired by the multiple terminal devices 10 and storing the information in a database, and transmitting control signals to the multiple terminal devices 10 to remotely control the terminal devices 10. The control device 20 may be implemented as a single server device or multiple server devices interconnected via a network. The assistance device 30 is connected to the terminal devices 10 via the Internet and transmits and receives audio and images. A general-purpose personal computer (PC) can be used as the assistance device 30. The assistance device 30 is basically operated by an assistant to assist a worker wearing multiple terminal devices 10. In this case, the control device 20 may serve as a relay for transmitting and receiving information between the terminal devices 10 and the assistance device 30. Note that the present invention is characterized mainly by information processing performed by the multiple terminal devices 10 and the control device 20. Therefore, the following will specifically describe an embodiment of the present invention, focusing on the terminal devices 10 and the control device 20.

[0023] Figure 2 schematically illustrates a characteristic process of the system according to the present invention. To illustrate with an example, suppose a user (mainly a field worker) asks multiple terminal devices 10, "Which terminal has the most battery remaining?" Multiple terminal devices 10(a) to (c) that are within range of being able to acquire the voice of this question transmit the voice information to the control device 20. At this time, the multiple terminal devices 10(a) to (c) may transmit the voice signal, which is the user's voice converted into an electrical signal, to the control device 20 almost as is, or they may transmit the analysis information obtained by analyzing the voice signal to the control device 20. The "voice information" shown in Figure 2 is a concept that includes such user voice signals and their analysis information.

[0024] Next, when the control device 20 receives audio information from terminal devices 10(a) to (c), it groups the terminal devices 10(a) to (c) based on that audio information. For example, the control device 20 determines the similarity of the audio signals received from each terminal device 10(a) to (c) and groups together terminal devices 10 that transmitted audio signals that fall within a predetermined similarity range. Alternatively, the control device 20 analyzes the audio signals received from each terminal device 10(a) to (c), determines the similarity of the analysis results, and groups together terminal devices 10 that transmitted audio signals that fall within a predetermined similarity range. In the example shown in Figure 2, terminal devices 10(a) to (c) are set to the same group. On the other hand, terminal device 10(d) is not grouped with terminal devices 10(a) to (c) because it was unable to acquire the user's voice inquiry. Furthermore, even if terminal device 10(d) acquires the user's voice at the same time as terminal devices 10(a) to (c), if the similarity is not within the predetermined range, it will not be considered to belong to the same group as terminal devices 10(a) to (c).

[0025] Next, the control device 20 extracts commands issued by the user to the terminal devices 10(a) to (c) from the voice information received from the terminal devices 10(a) to (c). For example, the control device 20 performs voice recognition on the user's voice signal to convert the voice into text and obtain command information such as "Which terminal among these has the most remaining battery power?" The control device 20 also interprets this command information using natural language processing. Specifically, "among these" refers to the group to which the terminal devices 10(a) to (c) belong. Therefore, the control device 20 identifies the terminal with the most remaining battery power among the terminal devices 10(a) to (c). For example, in the example shown in FIG. 2, it is assumed that the terminal device 10(a) has the most remaining battery power among the terminal devices 10(a) to (c). In this case, the control device 20 transmits a command to the terminal device 10(a) to respond to the user's question. When the terminal device 10(a) receives a response command from the control device 20, it notifies the user that it is the terminal with the most remaining battery power by outputting a sound such as "Hello, it's me" from a microphone. Note that the response of the terminal device 10(a) is not limited to outputting a sound, but can also be made by flashing an LED lamp or vibrating a vibrator.

[0026] In this way, users can easily identify the terminal device 10 they are looking for from among multiple terminal devices 10. Finding the terminal with the highest battery level is just one example; this system can be applied to various uses. For example, suppose workers on site are each wearing a terminal device 10, and a site supervisor asks several workers, "Who among you has been working continuously for more than an hour?" Then, among the terminal devices 10 that have received the voice of the question, the terminal device 10 that has been detecting the worker's movement continuously for more than an hour will respond, "Yes, that's me." This allows the site supervisor to prompt the worker wearing the responding terminal device 10 to take a break. The system can also handle other questions such as, "Who among you has been working for the longest time?" Thus, in the system 100 according to the present invention, multiple terminal devices 10 are automatically grouped based on the user's voice inquiry, and the terminal device 10 corresponding to the inquiry automatically responds.

[0027] Next, the configuration of the system 100 according to one embodiment of the present invention will be described in more detail. Figure 3 is an external perspective view showing an example of the terminal device 10. Figure 4 shows an example of the hardware elements of the terminal device 10. As shown in Figure 3, the terminal device 10 in this embodiment is a neck-worn wearable device. The terminal device 10 comprises a left arm portion, a right arm portion, and a main body portion that connects them at the back of the wearer's neck. When wearing the terminal device 10, the main body portion should be brought into contact with the back of the wearer's neck, and the left and right arm portions should be allowed to hang down from the sides of the wearer's neck towards the chest, so that the entire device can be hooked around the neck. Various electronic components are stored inside the housing of the terminal device 10.

[0028] Multiple sound-collecting units 14 (microphones) are provided on both the left and right arms. The sound-collecting units 14 are primarily positioned to acquire sounds from the wearer's surroundings and the voices of those speaking with the wearer. To widely collect sounds generated around the wearer, it is preferable to use omnidirectional microphones for the sound-collecting units 14. Known microphones such as dynamic microphones, condenser microphones, and MEMS (Micro-Electrical-Mechanical Systems) microphones can be used for the sound-collecting units 14. The sound-collecting units 14 convert sound into electrical signals, amplify these electrical signals using an amplifier circuit, and then convert them into digital information using an A / D conversion circuit before outputting them to the control unit 11. The sound signals acquired by the sound-collecting units 14 are transmitted to the control unit 11 located within the housing. In this embodiment, the sound signals acquired by the sound-collecting units 14 are also transmitted to the control device 20 via the internet through the communication unit 13. Furthermore, the sound signals acquired by the sound-collecting units 14 may also be transmitted to a support device 30 (see Figure 1) operated by a remote supporter. This allows sounds acquired by the on-site worker via the terminal device 10 to be shared with the support device 30 of the remote support worker.

[0029] An imaging unit 15 is further provided on the left arm. Specifically, the imaging unit 15 is provided on the distal end surface of the left arm, and this imaging unit 15 can capture still images and moving images of the front side of the wearer. The images captured by the imaging unit 15 are transmitted to the control unit 11 in the housing and stored as image data. A general digital camera may be used as the imaging unit 15. The imaging unit 15 is composed of, for example, a photographing lens, a mechanical shutter, a shutter driver, a photoelectric conversion element such as a CCD image sensor unit, a digital signal processor (DSP) that reads the charge amount from the photoelectric conversion element and generates image data, and an IC memory. The image data captured by the imaging unit 15 is supplied to the control unit 11 and stored in the memory unit 12. A predetermined image analysis process may also be performed on the image data. Still images and moving images captured by the imaging unit 15 are transmitted to the control device 20 via the communication unit 13 and the Internet. Still images and moving images may also be transmitted to a support device 30 (see FIG. 1 ) operated by a supporter in a remote location. As a result, still images and moving images acquired by the on-site worker on the terminal device 10 are also shared with the support device 30 of the supporter in a remote location.

[0030] A non-contact gesture sensor 16 is further provided on the right arm. The gesture sensor 16 is positioned on the tip of the right arm primarily for detecting the wearer's hand movements on the front side of the terminal device 10. The gesture sensor 16 detects, for example, the movements and shapes of the wearer's fingers. An example of the gesture sensor 16 is an optical sensor, which detects the movement and shape of an object by irradiating light from an infrared-emitting LED towards the object and capturing the change in the reflected light with a photodetector. The detection information from the gesture sensor 16 is transmitted to the control unit 11 and is mainly used to control the imaging unit 15 and the sound emission unit 18. Specifically, the detection information from the gesture sensor 16 is used to control the starting and stopping of the imaging unit 15 and the sound emission unit 18. For example, the gesture sensor 16 may control the imaging unit 15 when it detects that an object such as the wearer's hand has come close to the gesture sensor 16, or it may control the imaging unit 15 when it detects that the wearer has performed a predetermined gesture within the detection range of the gesture sensor 16. In this embodiment, it is also possible to swap the positions of the imaging unit 15 and the gesture sensor 16. Furthermore, the gesture sensor 16 may be replaced with a proximity sensor. The proximity sensor may, for example, detect when the wearer's fingers have come within a predetermined range. Known proximity sensors such as optical, ultrasonic, magnetic, capacitive, or thermal sensors can be used.

[0031] A sound-emitting unit (speaker) 18 is provided on the outside of the main unit located at the back of the wearer's neck (opposite the wearer's side). In this embodiment, the sound-emitting unit 18 is positioned to output sound toward the outside of the main unit. By emitting sound toward the back from the back of the wearer's neck in this way, the sound output from the sound-emitting unit 18 is less likely to reach the interlocutor located directly in front of the wearer. This makes it easier for the interlocutor to distinguish between the voice emitted by the wearer and the sound emitted from the sound-emitting unit 18 of the terminal device 10. The sound-emitting unit 18 is an acoustic device that converts electrical signals into physical vibrations (i.e., sound). An example of the sound-emitting unit 18 is a general speaker that transmits sound to the wearer by air vibration. Alternatively, the sound-emitting unit 18 may be a bone conduction speaker that transmits sound to the wearer by vibrating the wearer's bones. In this case, the sound-emitting unit 18 can be provided on the inside of the main unit (wearer side) and configured so that the bone conduction speaker contacts the bones (cervical vertebrae) at the back of the wearer's neck. Furthermore, in this embodiment, the audio signal input to the support device 30 is transmitted to the terminal device 10 via the internet. The terminal device 10 converts the audio signal received from the support device 30 into sound using the sound emission unit 18 and outputs it. This makes it possible to deliver the voice of the support worker operating the support device 30 to the worker wearing the terminal device 10.

[0032] As shown in Figure 4, the control unit 11 of the terminal device 10 performs calculations to control other elements of the terminal device 10. The control unit 11 can utilize a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The control unit 11 basically reads a program stored in the memory unit 12, loads it into the main memory, and executes predetermined calculations according to this program. The control unit 11 can also write and read calculation results according to the program to the memory unit 12 as appropriate. Furthermore, the control unit 11 of the terminal device 10 can execute a pre-existing application program stored in the memory unit 12 according to the instruction information of the RPA program executed by the control device 20, and control various hardware elements or perform predetermined calculations based on that program.

[0033] The storage unit 12 of the terminal device 10 is an element for storing information used in calculation processing by the control unit 11 and the results of such calculations. The storage function of the storage unit 12 can be realized by non-volatile memory such as HDD and SSD. The storage unit 12 may also have the function of main memory for writing or reading intermediate results of calculation processing by the control unit 11. The memory function of the storage unit 12 can be realized by volatile memory such as RAM and DRAM. The storage unit 12 may also store user-specific ID information of the person who possesses it. The storage unit 12 may also store the IP address, which is network identification information of the terminal device 10.

[0034] The communication unit 13 of the terminal device 10 is an element for wireless communication with the control device 20, the support device 30, or another terminal device 10. To communicate with the control device 20 or the support device 30 via the internet, the communication unit 13 may employ a communication module for wireless communication using known mobile communication standards such as 3G (W-CDMA), 4G (LTE / LTE-Advanced), or 5G, or a wireless LAN method such as Wi-Fi (registered trademark). Alternatively, the communication unit 13 may employ a communication module for proximity wireless communication using methods such as Bluetooth (registered trademark) or NFC to communicate directly with another terminal device 10.

[0035] The sensors 17 of the terminal device 10 include, for example, sensor devices for detecting the operation and usage status of the terminal device 10, or the wearer's biometric information. The sensors 17 can be sensor modules commonly found in portable information terminals or wearable devices. For example, the sensors 17 may include a gyroscope, accelerometer, geomagnetic sensor, and battery sensor. The sensors 17 may also include biosensors for detecting the wearer's biometric information, such as a body temperature sensor, heart rate sensor, blood oxygen saturation sensor, blood pressure sensor, and electrocardiogram sensor.

[0036] The location information acquisition unit 19 of the terminal device 10 is an element for acquiring current location information of the terminal device 10. Specifically, the location information acquisition unit 19 has a function of performing positioning using a global positioning system (GPS). Based on radio wave transmission time information included in radio waves transmitted from multiple GPS satellites, the location information acquisition unit 19 measures the time required to receive each radio wave and transmits time information indicating the time to the control unit 11. Based on the acquired time information, the control unit 11 can calculate information regarding the latitude and longitude of the location of the terminal device 10. Alternatively, the location information acquisition unit 19 may acquire current location information by scanning radio waves or beacon signals transmitted from wireless base stations such as Wi-Fi (registered trademark) access points.

[0037] 3 and 4, in this embodiment, the terminal device 10 does not have a display device such as a monitor or a display. Therefore, although an operator can perform relatively simple operations such as turning on / off each hardware element using the gesture sensor 16 or the like, it is difficult for the operator to perform complex operations such as operating an application program. When using such a terminal device 10 without a display device, it is particularly effective to remotely control multiple terminal devices 10 via the Internet using a control device 20 or the like, as in the system according to the present invention.

[0038] Figure 4 further shows the hardware and software components of the control device 20. In this embodiment, the control device 20 can utilize a known cloud system composed of one or more web servers. The control device 20 basically includes a management control unit 21, a database 22, and a communication unit 23. The management control unit 21 can utilize a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The management control unit 21 reads a predetermined program, loads it into main memory, and executes predetermined arithmetic processing according to this program. The storage function of the database 22 can be realized by non-volatile memory such as an HDD or SSD. Also, a known communication module can be used for the communication unit 23. The control device 20 monitors each terminal device 10 by communicating with multiple terminal devices 10, and stores information acquired by various sensors from various terminal devices 10 in the database 22, while constantly updating that information. The control device 20 also groups the terminal devices 10 and controls specific terminal devices 10 to respond to user inquiries.

[0039] As shown in Figure 4, in this embodiment, the management control unit 21 includes a monitoring unit 21a, a voice analysis unit 21b, a grouping unit 21c, a command acquisition unit 21d, a terminal identification unit 21e, and a command generation unit 21g. Each of these functional blocks 21a to 21g is realized by the management control unit 21 executing a predetermined program.

[0040] The monitoring unit 21a registers information collected from each terminal device 10 in the database 22 and updates that information as needed. Specifically, each terminal device 10 continuously or periodically transmits information detected by sensors 17 to the control device 20, such as motion information of the terminal device 10 detected by the gyro sensor, accelerometer, and geomagnetic sensor, and information on battery level and usage status detected by the battery sensor. In addition, if each terminal device 10 detects the wearer's biological information (body temperature, heart rate, blood oxygen saturation, blood pressure, electrocardiogram, etc.) with the sensors 17, it transmits this information to the control device 20. The monitoring unit 21a of the control device 20 registers this information received from each terminal device 10 in the database 22, associating it with each terminal device 10, and updates it as needed. Furthermore, if one terminal device 10 is used by multiple users, the information detected by the sensors 17 of the terminal device 10 may be registered in the database 22, associating it with each user, and updated as needed. As a result, the control device 20 can refer to the information recorded in the database 22 to check the exercise information or biological information for each terminal device 10 or user.

[0041] The voice analysis unit 21b analyzes the user's voice signal acquired by each terminal device 10. Voice analysis includes voice recognition, frequency analysis, speaker emotion recognition, speaker attitude recognition, speaker language recognition, and speaker voiceprint analysis. Voice recognition converts the user's voice signal into text information. Frequency analysis converts the user's voice signal into a spectrum (intensity distribution) broken down into each frequency component contained therein. Speaker emotion recognition, attitude recognition, and language recognition quantify the emotion, attitude, or language of the user who spoke the voice based on the user's voice signal and its spectrum. Voiceprint analysis quantifies the characteristics of the user's voice based on the user's voice signal and its spectrum. These voice analyses may also use trained models obtained by performing machine learning in advance using a large amount of training data. For example, an emotion recognition model can be obtained by performing machine learning using training data in which the voice signals of unspecified speakers are tagged with various emotions. The same applies to attitude recognition and language recognition. These trained models may also be used to analyze the user's voice signal.

[0042] The grouping unit 21c groups one or more terminal devices 10 out of the multiple terminal devices 10 that have acquired user voice signals, mainly based on the analysis information of the voice analysis unit 21b. Specifically, terminal devices 10 that should belong to the same group should basically have acquired the same voice signal from the same user. Therefore, the grouping unit 21c determines the similarity of the voice signals acquired by each terminal device 10, and groups the terminal devices 10 that have acquired voice signals within a predetermined similarity range. When determining whether to assign terminal devices 10 to the same group, the grouping unit 21c may refer to the status of short-range wireless communication and location information of each terminal device 10 in addition to the similarity of the voice signals. A specific example of the processing by the grouping unit 21c will be described in detail later with reference to FIG. 6.

[0043] The command acquisition unit 21d acquires command information for the computer (particularly the terminal device 10) based on the analysis information from the speech analysis unit 21b. Specifically, the speech analysis unit 21b converts the user's voice signal into text information by performing speech recognition processing. The command acquisition unit 21d then performs natural language processing on the text information converted by the speech analysis unit 21b, including, for example, morphological analysis, syntactic analysis, semantic analysis, and contextual analysis, to acquire command information contained in the user's voice. This process of acquiring command information from voice can employ techniques used in well-known voice assistants.

[0044] The terminal identification unit 21e refers to the information recorded in the database 22 to identify the terminal device 10 corresponding to the command information acquired by the command acquisition unit 21d from among the grouped terminal devices 10. For example, if the command from the user is "Which terminal has the most battery remaining?", the terminal identification unit 21e refers to the information on the current battery level of each terminal device 10 recorded in the database 22 to identify the terminal device 10 with the most battery remaining from the group. Depending on the content of the user's command, it is possible that no terminal device 10 in the group will respond to that command, or that two or more terminal devices 10 in the group will respond to the command.

[0045] The command generation unit 21g generates command information for the terminal device 10 identified by the terminal identification unit 21e to respond to the command information acquired by the command acquisition unit 21d. For example, if the command from the user is "Which terminal has the most battery remaining?", the command generation unit 21g should generate command information to cause the terminal device 10 with the most battery remaining in the group to perform some kind of response, such as outputting a sound saying "Yes, that's me" or illuminating a lamp. The command information generated by the command generation unit 21g is transmitted to the terminal device 10 identified by the terminal identification unit 21e via the communication unit 23. Upon receiving this command information, the terminal device 10 performs processing such as outputting a sound or illuminating a lamp in accordance with the command information.

[0046] Although not shown in FIG. 4, the support device 30 may be a known personal computer (PC), smartphone, tablet terminal, or the like. Specifically, the support device 30 includes a control unit configured with a processor such as a CPU, a storage unit including nonvolatile memory and volatile memory, and a communication unit configured with a communication module for connecting to the Internet. The support device 30 also includes input units such as a mouse, keyboard, and microphone, and output units such as a display and speaker. The support device 30 is a terminal operated by a supporter (such as an operator) who provides support information to on-site workers. This allows the supporter to remotely send work instructions and the like to the terminal device 10 of the on-site worker, receive audio and images from the terminal device 10, and output them from the speaker or display.

[0047] Next, referring to Figures 5 and 6, an example of the processing flow of the terminal devices 10 and control device 20 that is executed when a user (mainly a field worker) issues a voice command to multiple terminal devices 10 is shown. Figure 5 shows the main flow, and Figure 6 shows the sub-flow of the grouping process.

[0048] As shown in FIG. 5, each of the multiple terminal devices 10 acquires a user's voice using the sound collection unit 14 (step S1). The acquired user's voice is converted into an audio signal by the sound collection unit 14 and temporarily stored as a digital signal in the storage unit 12 together with meta information such as date and time information when the voice was acquired. Note that the sound collection unit 14 of each terminal device 10 may be always on, or may be turned on only in a specific situation in which the user makes a voice inquiry. For example, the sound collection unit 14 may be turned on only while the terminal device 10 is charging or while the terminal device 10 detects that it is being worn by the user. The sound collection unit 14 may be switched on / off by the gesture sensor 16. The terminal device 10 that has acquired the user's voice in step S1 uses this as a trigger to start subsequent processing.

[0049] Next, at the time the user's voice is acquired, the terminal device 10 checks whether it is communicating via short-range wireless communication with another terminal device 10 (step S2). If the terminal device 10 is communicating via short-range wireless communication, it acquires the identification information (individual identification number, etc.) of the other terminal device 10 and temporarily stores it in the storage unit 12.

[0050] Furthermore, the terminal device 10 acquires location information at the time the user's voice is acquired using the location information acquisition unit 19 (step S3). The terminal device 10 temporarily stores the location information acquired by the location information acquisition unit 19 in the storage unit 12. Note that steps S2 and S3 may be executed in either order. Also, steps S2 and S3 can be executed simultaneously.

[0051] Next, the terminal device 10 transmits its own identification information (individual identification number, etc.) along with the information acquired in steps S1 to S3 to the control device 20 via the communication unit 13 (step S4). Specifically, the audio signal acquired by the sound collection unit 14 and the information of the date and time of acquisition, the identification information of the other terminal device 10 if short-range wireless communication is being performed, and the location information acquired by the location information acquisition unit 19 are transmitted from the terminal device 10 to the control device 20. The control device 20 receives this information from multiple terminal devices 10 (step S5).

[0052] The control device 20 performs voice analysis processing on the voice signals received from each terminal device 10 using the voice analysis unit 21b of the management control unit 21 (step S6). As shown in Figure 5, the voice analysis processing here has two purposes: to group the multiple terminal devices 10 (step S7) and to obtain user commands for the group of terminal devices 10 from the voice signals (step S8). First, the process of grouping the multiple terminal devices 10 will be explained.

[0053] 6 shows a subflow of the grouping process (step S7) executed by the grouping unit 21c of the management and control unit 21 of the control device 20. The grouping unit 21c determines whether the acquisition date and time of the voice signals received from multiple terminal devices 10 is within a predetermined time range (step S7-1). For example, when the control device 20 acquires voice signals from three terminal devices 10, the grouping unit 21c determines whether the date and time at which each terminal device 10 acquired the voice signal is within, for example, 30 seconds. For terminal devices 10 whose acquisition date and time of the voice signal is within the predetermined time range, the process proceeds to the next step (step S7-2), and for terminal devices 10 that are not within the predetermined time range, the process is determined to belong to a different group (step S7-7). Note that terminal devices 10 determined to belong to a different group may also be determined to belong to a different group.

[0054] Next, the grouping unit 21c determines whether the similarity of the voice signals acquired by each terminal device 10 is within a predetermined range based on the analysis information from the voice analysis unit 21b (step S7-2). The method for determining the similarity is not particularly limited, and for example, it may be determined whether the spectra obtained by frequency analyzing the voice signals acquired by each terminal device 10 are similar within a predetermined range. Alternatively, the voice signals acquired by each terminal device 10 may be converted into text information by voice recognition, and it may be determined whether the text information is similar within a predetermined range. Alternatively, it may be determined whether the similarity is determined based on the results of emotion recognition, attitude recognition, language recognition, and voiceprint analysis performed on the user's voice signal. Terminal devices 10 whose voice signal similarity is within the predetermined range proceed to the next step (step S7-3), and terminal devices 10 whose voice signal similarity is not within the predetermined range are determined to be in a different group (step S7-7).

[0055] Next, the grouping unit 21c determines whether terminal devices 10 whose voice signal similarity is within a predetermined range were performing short-range wireless communication at the time the voice signal was acquired (step S7-3). As mentioned above, if terminal devices 10 are performing short-range wireless communication, the identification information of the other party is transmitted to the control device 20. The grouping unit 21c only needs to determine whether terminal devices 10 were performing short-range wireless communication based on this identification information. Terminal devices 10 whose voice signal similarity is within a predetermined range and who were performing short-range wireless communication at that time are determined to be in the same group by the grouping unit 21c (step S7-6). On the other hand, terminal devices 10 that were not performing short-range wireless communication at that time proceed to the next step (step S7-4).

[0056] Next, the grouping unit 21c determines whether terminal devices 10 whose audio signal similarity is within a predetermined range were within a predetermined position range at the time the audio signal was acquired. As described above, since position information at the time of audio signal acquisition is transmitted from each terminal device 10 to the control device 20, the grouping unit 21c only needs to determine whether each terminal device 10 was within a predetermined position range based on the position information acquired from each terminal device 10. The predetermined position range is not particularly limited, but it is preferable to determine that the terminal devices 10 are within the predetermined position range when the distance between them is 1 to 2 m. Terminal devices 10 whose audio signal similarity is within a predetermined range and which were located within the predetermined position range at that time are determined to be in the same group by the grouping unit 21c (step S7-6). On the other hand, terminal devices 10 that were not within the predetermined range at that time proceed to the next step (step S7-5).

[0057] Next, the grouping unit 21c determines whether the terminal devices 10 whose audio signal similarities are within a predetermined range were outside a predetermined location range at the time the audio signals were acquired (step S7-5). That is, although the terminal devices 10 were actually located close to each other at the time the audio signals were acquired, it is possible that acquisition of location information at that time failed due to a communication failure or other reason. In this case, if all terminal devices 10 that were outside the predetermined location range at the time the audio signals were acquired are uniformly determined to be in different groups in step S7-6, terminal devices 10 that should actually belong to the same group may be determined to belong to different groups due to a communication failure or other reason. Therefore, in step S7-5, the grouping unit 21c determines only the terminal devices 10 that were confirmed to be outside the predetermined location range at the time the audio signals were acquired to be in different groups (step S7-7). On the other hand, even if the grouping unit 21c was unable to confirm in step S7-5 that the terminal devices 10 are outside the predetermined location range, the grouping unit 21c determines that these terminal devices 10 belong to the same group because it was confirmed in step S7-2 that the similarity of the audio signals is within the predetermined range (step 7-6).By performing these steps S7-1 to 7-7, it is determined whether or not the multiple terminal devices 10 belong to the same group.

[0058] Next, returning to Figure 5, we will explain the process of obtaining a command for a group of terminal devices 10 from the user's voice signal and sending that command to a specific terminal device 10. First, the voice analysis unit 21b of the management control unit 21 of the control device 20 performs voice recognition processing on the voice signal received from the terminal device 10. This converts the user's voice into text information. At this time, the control device 20 often receives voice signals from multiple terminal devices 10 that are candidates for grouping. Even if multiple terminal devices 10 receive voice signals from the same user in almost the same location, there may be variations in the text information obtained from the voice signal of each terminal device 10 depending on the microphone performance and orientation of each terminal device 10. In this case, the voice analysis unit 21b may select the most reliable voice signal from among the voice signals received from multiple terminal devices 10, for example, the voice signal with the strongest voice component intensity, and perform voice recognition processing only on that selected voice signal. Alternatively, the voice analysis unit 21b may perform voice recognition processing on all voice signals received from multiple terminal devices 10, compare the corresponding character information, and discard character information that is estimated to have not been correctly recognized based on the comparison results, or select the character information that is estimated to have been recognized most correctly.

[0059] Next, the command acquisition unit 21d of the management control unit 21 acquires command information for a group of terminal devices 10 from the character information obtained by speech recognition by the speech analysis unit 21b (step S8). As described above, the command acquisition unit 21d can acquire command information contained in the user's voice by performing natural language processing on this character information, including, for example, morphological analysis, syntactic analysis, semantic analysis, and contextual analysis. This command information is information at a level that the control device 20, which is a computer, can understand. Specifically, this command information is often a command to identify one or more terminal devices 10 from among the multiple terminal devices 10 that were grouped in the grouping process (step S7). For this reason, this command information may also be a query to search for a specific terminal device 10 from the information about terminal devices 10 recorded in the database 22.

[0060] Next, the terminal identification unit 21e of the management control unit 21 accesses the database 22 and refers to information about the terminal devices 10 and the users wearing them for the multiple terminal devices 10 that were grouped in the grouping process (step S7) described above (step S9). As described above, the database 22 stores and updates information about the terminal devices 10 and the users wearing them, which is acquired by the sensors 17 of each terminal device 10 by the monitoring unit 21a of the management control unit 21. The terminal identification unit 21e refers to the information in this database 22 and identifies one or more terminal devices 10 from among the grouped multiple terminal devices 10 that should respond to the command information acquired by the command acquisition unit 21d.

[0061] Next, the command generation unit 21g of the management control unit 21 generates command information for the terminal device 10 identified by the terminal identification unit 21e to respond to the command information from the user. If the command information from the user is for finding a specific terminal device 10 from among the group, the command information may be, for example, control information for outputting sound or light to that specific terminal device 10. The user's command information may be more complex. For example, if the command information from the user requests a specific function to be performed by a specific terminal device 10 in the group (for example, requesting the terminal device 10 with the highest battery level in the group to take a still image), the command information will correspond to such a command information.

[0062] Next, the control device 20 transmits the command information generated by the command generation unit 21g to the terminal device 10 identified by the terminal identification unit 21e via the communication unit 23 (step S12). This identified terminal device 10 receives the command information from the control device 20 (step S13). The terminal device 10 that has received this command information controls various hardware components in accordance with the command information, such as outputting sound or light, or performing other specific functions (step S14). In this way, the system of the present invention can automatically group multiple terminal devices 10 based on a user's voice command, and can automatically cause a specific terminal within the group to respond to the user's command.

[0063] In this specification, embodiments of the present invention have been described with reference to the drawings in order to express the content of the present invention. However, the present invention is not limited to the above embodiments, and includes modifications and improvements that are obvious to those skilled in the art based on the matters described in this specification.

[0064] For example, in the above-described embodiment, the terminal device 10 simply sends the user's voice signal to the control device 20, and the control device 20 performs analysis processing of the user's voice signal collectively. On the other hand, in order to reduce the load on the control device 20, each terminal device 10 may perform analysis processing of the user's voice signal, and the analysis results may be sent from each terminal device 10 to the control device 20. In this case, the function of the voice analysis unit 21b shown in FIG. 4 is realized by the control unit 11 of the terminal device 10. Similarly, the voice analysis step (step S6) shown in FIG. 5 is executed in multiple terminal devices 10.

[0065] Furthermore, not limited to the audio analysis processing described above, it is also possible to distribute the load of the control device 20 to each terminal device 10 by having each terminal device 10 handle some of the processing that was previously performed by the control device 20. [Explanation of symbols]

[0066] 10... Terminal device 11... Control unit 12...Memory unit 13...Communication unit 14...sound collection unit 15...imaging unit 16...Gesture sensor 17...Sensors 18...Sound emitting section 19...Position information acquisition section 20...Control device 21...Management control unit 21a...Monitoring Unit 21b...Speech Analysis Unit 21c...Grouping unit 21d...Instruction acquisition unit 21e... Terminal identification unit 21g... Command generation unit 22...Database 23...Communications Department 30...Support device 100...System

Claims

1. A system including a plurality of terminal devices and a control device connected to the terminal devices via a communication line, each of the plurality of terminal devices includes a sound collection unit; the plurality of terminal devices or the control device includes a sound analysis unit that analyzes the sound signal acquired by the sound collection unit, The control device a grouping unit that groups one or more terminal devices among the plurality of terminal devices based on analysis information by the voice analysis unit; a command acquisition unit that acquires command information for the terminal device based on the analysis information; a command generating unit that generates command information in response to the command information for at least one specific terminal device that belongs to a certain group; system.

2. The control device a terminal identification unit that identifies at least one terminal device that responds to the command information from among the plurality of terminal devices that belong to a certain group; transmits command information in response to the command information to the terminal device identified by the terminal identification unit; The system of claim 1 .

3. The grouping unit groups together the plurality of terminal devices that have acquired the audio signals within a predetermined error range by the sound collection unit within a predetermined time range. The system of claim 1 .

4. Each of the plurality of terminal devices further includes a location information acquisition unit that acquires its own location information; The grouping unit groups together the plurality of terminal devices that are within a predetermined geographical range and that have acquired the audio signals within a predetermined error range by the sound collection unit within a predetermined time range into the same group. The system of claim 3 .

5. each of the plurality of terminal devices further includes a short-range wireless communication unit; The grouping unit groups together the plurality of terminal devices that are in a communication state via the short-range wireless communication unit and that have acquired the audio signal within a predetermined error range by the sound collection unit within a predetermined time range into the same group. The system of claim 3 .

6. A program for causing a portable information terminal to function as the terminal device in the system according to claim 1.

7. A control device connected to a plurality of terminal devices including a sound collection unit via a communication line, the plurality of terminal devices or the control device includes a sound analysis unit that analyzes the sound signal acquired by the sound collection unit, The control device a grouping unit that groups one or more terminal devices among the plurality of terminal devices based on analysis information by the voice analysis unit; a command acquisition unit that acquires command information for the terminal device based on the analysis information; a command generating unit that generates command information in response to the command information for at least one specific terminal device that belongs to a certain group; Control device.

8. A program for causing a computer to function as the control device according to claim 7.

Images