Information Processing System

Abstract

This invention provides an information processing system suitable for the unified management of multiple terminal devices. The solution of this invention is an information processing system 100, which includes multiple terminal devices 10, client devices 20, and a cloud 30 including one or more server devices. The client device or cloud has: a processing instruction unit that designates a specific terminal device from among the multiple terminal devices and instructs the specific terminal device to perform processing according to a predetermined procedure; an authentication information storage unit that stores authentication information required to execute the predetermined procedure; and an authentication information providing unit that provides authentication information or temporary information replacing the authentication information to the specific terminal device designated by the processing instruction unit, the authentication information being required for performing processing according to the procedure instructed by the processing instruction unit. The specific terminal device uses the authentication information provided from the authentication information providing unit or the temporary information to perform processing based on the instructions from the processing instruction unit.

Description

Technical Field

[0001] The present invention relates to an information processing system. More specifically, the information processing system of the present invention relates to remote control of a terminal device and remote support for workers on site. Background Art

[0002] In the past, a remote work support system is known, which has a wearable device worn by an operator and a management device connected to the device via the Internet (for example, Patent Document 1). The management device is operated by a support person who remotely monitors the operator's work on site or gives instructions. The wearable device is equipped with a camera or a fixed-point device and a microphone. The support person controls the wearable device through the management device, thereby confirming the camera's photographic image or sending instructions to the operator through the fixed-point device.

[0003] In addition, the applicant of this case proposed a small and multifunctional neck-hanging wearable device (e.g., Patent Document 2). This neck-hanging device is equipped with input and output devices such as a camera or a speaker, a microphone or a gesture sensor, and can record or analyze the conversation between the interlocutors, as well as analyze the photographic images of the camera. In addition, in this Patent Document 2, it is also proposed to conduct voice calls between people wearing the neck-hanging device, and as long as this neck-hanging device is used, remote conferences can also be held between operators on site.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2020-150400

[0007] Patent Document 2: Japanese Patent Application Publication No. 2021-082802 Summary of the invention

[0008] Problems to be solved by the invention

[0009] However, as in the previous remote work support systems, in the system where the management device controls the terminal devices such as wearable devices through the Internet, the premise is that the application program dedicated to the function is installed in the terminal device in advance according to the function to be executed by the terminal device. For example, in order to execute the functions of camera control, voice analysis, image analysis, and web conferencing, separate programs are required respectively. If the terminal device does not have these programs, even if the terminal device is remotely controlled by the management device, the terminal device cannot actually execute these functions. Therefore, there is a problem that it is necessary to develop programs separately according to the functions of the terminal device, which consumes development costs. In addition, it is also conceivable that the types of programs installed for each terminal device are different. In this case, it is necessary to control the terminal device separately to execute the program, and there is a problem that it becomes difficult to uniformly control and manage multiple terminal devices.

[0010] On the other hand, if a separate application is not installed in the terminal device in advance, the problem of how to handle the authentication information used to execute the program arises. For example, when a specific program needs to be executed with a terminal device, if the program and authentication information are sent to the terminal device, the security of the authentication process cannot be guaranteed. In addition, it is also considered that only a specific program is handed over to the terminal device, and the user wearing the terminal device enters the authentication information (account name and password) into the terminal device, but the operation becomes complicated and there is also a risk of losing the password.

[0011] Furthermore, in server-client systems represented by IoT, it is necessary to study which of the server device and the terminal device is used to perform the processing for realizing a certain service. To give a specific example, in remote support services and work image recording services using terminal devices, since personal information such as faces or car license plates are sometimes captured in the images photographed by the terminal device, mosaic processing is performed on predetermined areas in the image from the perspective of privacy. At this time, for example, whether the mosaic processing is performed in the terminal device or in the server device becomes a problem. For example, for personal information with higher confidentiality, it is necessary to consider mosaic processing in the terminal device, and the personal information is not stored in the server device. On the other hand, for information with low confidentiality, it is more efficient to uniformly perform mosaic processing in a server device with high processing power. In this way, it is hoped that the place where the processing is performed can be appropriately changed according to the situation of using the terminal device or the workplace, but for this purpose, it is usually necessary to prepare a dedicated application in advance according to the place where the processing is performed. In this case, it is also considered that the type of program installed in the terminal device is changed according to the purpose, so that a program for mosaic processing of photographic images is installed in a certain terminal device, and a program for mosaic processing is installed in another terminal device when the photographic images are uploaded to the server device. However, if the type of program installed in each terminal device is different, there is a problem that it becomes difficult to uniformly control and manage multiple terminal devices as mentioned above.

[0012] The present invention has been developed in view of the above-mentioned problems in the prior art, and its object is to solve at least one of the above-mentioned problems. More specifically, the present invention aims to provide an information processing system suitable for unified management of multiple terminal devices.

[0013] Technical solutions to problems

[0014] The present invention relates to an information processing system. The information processing system of the present invention comprises: a plurality of terminal devices, a client device and a cloud. The cloud is constituted to include one or more server devices connected to the plurality of terminal devices and the client device via a network. In the present invention, an example of a terminal device is a wearable device. In particular, the terminal device is preferably a neck-hanging wearable device. However, the terminal device is not limited to a wearable device, and for example, it may be an edge device having various input and output devices such as a camera or a microphone, other sensors (temperature sensors, optical sensors, etc.), a monitor, a projector, a speaker, an actuator (motor, etc.). An example of an edge device is a surveillance camera.

[0015] In the information processing system of the present invention, the client device and / or the cloud has a processing instruction unit, an authentication information storage unit, and an authentication information providing unit. The processing instruction unit specifies a specific terminal device from a plurality of terminal devices, and instructs the specific terminal device to perform a process in accordance with a predetermined program. The authentication information storage unit stores the authentication information required to execute the predetermined program. The authentication information providing unit provides authentication information or temporary information (token for authorization) instead of the specific terminal device specified by the processing instruction unit, and the authentication information is required to perform the process in accordance with the program indicated by the processing instruction unit. An example of authentication information in this case is authentication information for accessing and logging into an external server device of an operator providing a network conference service. Alternatively, the authentication information providing unit provides authentication information or temporary information (token for authorization) instead of the specific terminal device specified by the processing instruction unit, and the authentication information is authentication information for confirming whether the process in accordance with the program indicated by the processing instruction unit is allowed. An example of authentication information in this case is the following authentication information: used to confirm whether the terminal device is given permission to perform the process in the case of a device that pre-restricts the specific process that can be performed in accordance with the program (for example, mosaic processing of a photographic image). Furthermore, each of the plurality of terminal devices uses the authentication information or temporary information provided from the authentication information providing unit to perform processing based on the instructions from the processing instruction unit. For example, the terminal device accesses an external server device that provides a network conference service, and uses the authentication information provided from the authentication information providing unit to perform processing related to the network conference. Furthermore, for example, in the case of a device that pre-restricts the ability to perform mosaic processing on a photographic image, the terminal device confirms whether permission is granted to its own terminal to perform this processing using the authentication information provided from the authentication information providing unit, and performs this processing after obtaining confirmation of permission. Furthermore, in the present invention, the terminal device itself does not need to execute a program, and the terminal device only needs to perform processing that follows the program that has been executed on the client device or the cloud (that is, "processing based on the instructions from the processing instruction unit"). In this case, the terminal device does not need to install the program in advance, but only needs to have a function that is called in accordance with the command in the program that has been executed on the client device or the cloud.

[0016] As in the above structure, the client device or the cloud specifies the processing to be performed by the terminal device, and the authentication information required for the processing or the temporary information instead is provided to the terminal device, thereby, even if the terminal device itself does not have an application or authentication information, the predetermined processing (function) can be executed on each terminal device. As a result, it is easy to uniformly manage and remotely control multiple terminal devices.

[0017] In the information processing system of the present invention, it is preferred that the client device or the cloud further has a program execution unit that executes the program indicated by the processing instruction unit. In this case, the processing instruction unit of the client device or the cloud instructs a specific terminal device to follow the processing of the program executed by the program execution unit. Moreover, the specific terminal device performs processing based on the instructions from the processing instruction unit, in other words, performs processing in accordance with the program executed by the program execution unit. That is, the program indicated by the processing instruction unit is not executed in each terminal device, but in the client device or the cloud. This program includes processing commands (function call commands) for executing multiple functions in the terminal device. Thus, even if the program is not pre-installed in the terminal device, as long as a variety of functions provided as a well-known software development kit (SDK: Software Development Kit) are saved in the terminal device, the terminal device can also perform arbitrary processing. In addition, as described in detail later, in order to process in real time and at high speed in each terminal device in response to the program executed in the client device or the cloud, it is effective to pre-install middleware in the cloud and the terminal device respectively.

[0018] Furthermore, the program executed by the client device or the cloud includes one or more processing commands (function call commands). Therefore, the processing instruction unit instructs the specific terminal device to perform processing based on the processing commands included in the program. In this case, the specific terminal device performs processing based on the processing commands included in the program.

[0019] In the information processing system of the present invention, the program may also be an application for Web conferencing. Furthermore, the authentication information may also include authentication information for the main account of the Web conference and authentication information for the guest account. In this case, the processing instruction unit preferably specifies one terminal device and another terminal device from a plurality of terminal devices, the one terminal device being assigned the main account of the Web conference, and the other terminal device being assigned the guest account for participating in the Web conference hosted by the one terminal device. Thus, a Web conference can be held between a plurality of terminal devices. That is, in the past, when a plurality of operators wore wearable devices respectively and held a Web conference through the wearable devices, the conference was generally hosted by a server device or a client device that managed these terminal devices. In contrast, according to the present invention, a server device or a client device can allocate a main account to a specific terminal device without hosting the conference, thereby enabling Web conferencing between terminal devices.

[0020] In the information processing system of the present invention, it is preferred that the main account of one terminal device and the guest account of another terminal device can be exchanged. For example, in the case where the battery of the terminal device hosting the Web conference may be exhausted or the network connection is known to be disconnected in advance (the radio wave strength changes to below a certain level), the hosting authority can be transferred to another terminal device. In this way, it is easy to continue the Web conference between terminal devices. The main account and the guest account can be automatically exchanged.

[0021] In the information processing system of the present invention, the processing instruction unit of the client device or the cloud can also automatically designate a terminal device to be assigned a main account based on the remaining battery level or communication quality of the terminal device. And, of course, the processing instruction unit of the client device or the cloud can also automatically designate a terminal device to be assigned a main account based on both the remaining battery level and the communication quality of the terminal device. For example, the processing instruction unit can designate the terminal with the largest remaining battery level among multiple terminal devices for a web conference as the main account, or the terminal with the highest call quality as the main account. Moreover, in the case where the remaining battery level or call quality of the terminal device changes during a web conference, the main account and the guest account can also be automatically swapped. In addition, the so-called "call quality" is obtained by quantifying one or more call characteristics related to the degree to which the user's conversation can be heard clearly in a call between multiple terminal devices. The so-called call characteristics here include, for example, electrical characteristics such as the reception intensity of radio waves, frequency characteristics in the communication transmission system, and / or acoustic characteristics such as noise.

[0022] In the information processing system of the present invention, it is preferred that the plurality of terminal devices include a terminal device that does not have a display unit such as a display. In the case where the terminal device does not have a display unit, it is difficult for the user of the terminal device to perform complex operations such as operating an application program, but it is particularly effective to use the system of the present invention and remotely operate the terminal device through a client device for such a terminal device.

[0023] In the information processing system of the present invention, the program includes a plurality of processing commands (function call commands). The processing instruction unit selects a device that performs processing based on the processing command from a plurality of terminal devices, a plurality of server devices and the client device according to the processing command included in the program. Moreover, the terminal device, server device or client device selected by the processing instruction unit executes the processing (function) based on the processing command indicated by the processing instruction unit. Thus, the place where the processing corresponding to the program is executed can be arbitrarily or automatically specified from a plurality of terminal devices, server devices or client devices. In the past, it was common sense that all the processing corresponding to the program was executed in the device installed with the program, but according to the present invention, regardless of whether the program is installed, its execution place can be arbitrarily selected according to the processing to a certain extent. For example, the processing instruction unit only needs to make a plurality of server devices or client devices perform at least a part of the processing based on the processing command included in the program.

[0024] Effects of the Invention

[0025] According to the present invention, it is possible to provide an information processing system suitable for collectively managing a plurality of terminal devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] [ Figure 1 ] Figure 1 An overall diagram showing the information processing system of the present invention;

[0027] [ Figure 2 ] Figure 2 An example of a terminal device (wearable device);

[0028] [ Figure 3 ] Figure 3 This is a block diagram showing an example of hardware elements for each of the terminal device, the client device, and the cloud that constitute the information processing system;

[0029] [ Figure 4 ] Figure 4 is a block diagram showing an example of the main software elements constituting the information processing system;

[0030] [ Figure 5 ] Figure 5 An example of the flow of data or commands when a web conference is held between terminal devices is shown;

[0031] [ Figure 6 ] Figure 6 is a block diagram showing another example of major software elements included in the information processing system;

[0032] [ Figure 7 ] Figure 7 This is a flowchart showing an example of performing mosaic processing on video data obtained by a terminal device. DETAILED DESCRIPTION

[0033] Hereinafter, a mode for carrying out the present invention will be described using the drawings. The present invention is not limited to the mode described below, and also includes modes appropriately modified from the following mode within a range obvious to those skilled in the art.

[0034] Figure 1 The overall structure of an information processing system 100 according to one embodiment of the present invention is schematically shown. The information processing system 100 according to this embodiment is preferably used mainly for a support person to remotely support work performed by a plurality of workers on site via the Internet.

[0035] like Figure 1 As shown, the information processing system 100 includes a plurality of terminal devices 10, a client device 20 and a cloud 30. An example of the terminal device 10 is a wearable device. As wearable devices, for example, neck-hanging type, glasses type, head-worn type and watch type are cited, but in this embodiment, a neck-hanging type wearable device is used. Such a terminal device 10 is worn by a plurality of operators, each of whom wears one. As the client device 20, a general-purpose personal computer (PC) can be used. The client device 20 is basically operated by a support staff for supporting a plurality of operators wearing the terminal devices 10. The cloud 30 is composed of a plurality of server devices interconnected via the Internet. The cloud 30 is connected to a plurality of terminal devices 10 and the client device 20 via the Internet, and also has the function of a relay device, transmitting control commands or data from the client device 20 to the terminal device 10, or conversely transmitting data obtained through the terminal device 10 to the client device 20.

[0036] Figure 2 1 is a perspective view showing an example of the appearance of the terminal device 10. Figure 3 An example of hardware elements of the terminal device 10 is shown in FIG. Figure 2 As shown, the terminal device 10 in this embodiment is a neck-hanging wearable device. The terminal device 10 includes a left arm, a right arm, and a main body connecting these arms behind the neck of the wearer. When wearing the terminal device 10, just make the main body contact the back of the wearer's neck, and make the left arm and the right arm hang from the wearer's neck side toward the chest side, and hang the entire device at the base of the neck. Various electronic components are stored in the housing of the terminal device 10.

[0037] A plurality of sound collecting parts 14 (microphones) are provided on the left arm and the right arm, respectively. The sound collecting part 14 is mainly configured to obtain sounds around the wearer or the voices of the wearer and the interlocutor. It is preferred to use an omnidirectional (non-directional) microphone as the sound collecting part 14 so as to widely collect sounds generated around the wearer. As the sound collecting part 14, any well-known microphone such as a dynamic microphone, a condenser microphone, or a MEMS (Micro-Electrical-Mechanical Systems) microphone may be used. The sound collecting part 14 converts the sound into an electrical signal, amplifies the electrical signal through an amplifier circuit, converts it into digital information through an A / D conversion circuit, and outputs it to the control part 11. The sound signal obtained by the sound collecting part 14 is transmitted to the control part 11 provided in the housing for predetermined analysis and processing.

[0038] A camera unit 16 is further provided on the left arm. Specifically, a camera unit 16 is provided on the front end surface of the left arm, and a still image or a dynamic image of the front side of the wearer can be photographed by this camera unit 16. The image obtained by the camera unit 16 is transmitted to the control unit 11 in the shell and stored as image data. As the camera unit 16, a general digital camera can be used. The camera unit 16 is composed of, for example, a photographic lens, a mechanical shutter, a shutter driver, a photoelectric conversion element such as a CCD image sensor unit, a digital signal processor (DSP) that reads the charge amount from the photoelectric conversion element to generate image data, and an IC memory. The image data obtained by the camera unit 16 is supplied to the control unit 11 and stored in the storage unit 12, and a predetermined image analysis process is performed, or it is transmitted to the cloud through the Internet by the communication unit 13.

[0039] Furthermore, an operation unit 15 is further provided on the left arm. The operation unit 15 receives inputs operated by the wearer. A known switch circuit or a touch panel can be used as the operation unit 15. The operation unit 15 receives, for example, an operation to instruct to start or stop voice input, an operation to instruct to turn on or off the power of the device, an operation to instruct to increase or decrease the volume of the speaker, or an operation required to realize other functions of the terminal device 10. Information inputted through the operation unit 15 is transmitted to the control unit 11.

[0040] A non-contact type sensor 17 is further provided on the right arm. The sensor 17 is mainly used to detect the hand movement of the wearer on the front side of the terminal device 10, and is arranged on the front end surface of the right arm. Examples of the sensor 17 are proximity sensors or gesture sensors. The proximity sensor, for example, detects when the wearer's finger approaches a predetermined range. Well-known sensors such as optical, ultrasonic, magnetic, electrostatic capacitance or temperature sensing can be used as proximity sensors. The gesture sensor, for example, detects the movement or shape of the wearer's finger. An example of a gesture sensor is an optical sensor, which irradiates light from an infrared light-emitting LED toward a target object, and uses a light-receiving element to capture the change of its reflected light, thereby detecting the movement or shape of the target object. The detection information of the sensor 17 is transmitted to the control unit 11, and is mainly used for the control of the camera unit 16. Specifically, the sensing information of the sensor 17 is mainly used for the control of the camera unit 16, such as the start and stop of the camera unit 16. For example, the sensor unit 17 can detect that an object such as the wearer's hand is close to the sensor unit 17 and control the camera unit 16, or can detect that the wearer performs a predetermined gesture within the detection range of the sensor unit 17 and control the camera unit 16. In addition, in this embodiment, the positions of the camera unit 16 and the sensor unit 17 can also be exchanged.

[0041] A sound emitting unit (speaker) 18 is provided on the outside of the main body (the side opposite to the wearer) behind the neck of the wearer. In the present embodiment, the sound emitting unit 18 is configured to output sound toward the outside of the main body. In this way, by emitting sound from the back of the wearer's neck toward the back, the sound output from the sound emitting unit 18 is difficult to be directly transmitted to the interlocutor located directly in front of the wearer. As a result, the interlocutor can easily distinguish between the voice emitted by the wearer himself and the sound emitted from the sound emitting unit 18 of the terminal device 10. The sound emitting unit 18 is an acoustic device that converts an electrical signal into a physical vibration (i.e., sound). An example of the sound emitting unit 18 is a general speaker that transmits sound to the wearer through air vibration. In addition, as the sound emitting unit 18, a bone conduction speaker that transmits sound to the wearer by vibrating the wearer's bones can also be used. In addition, in this case, it is sufficient to configure the sound emitting unit 18 to be provided on the inside of the main body (the wearer's side) and the bone conduction speaker to be in contact with the bones (cervical vertebrae) behind the wearer's neck.

[0042] A light projecting unit 19 is further provided on the left arm and the right arm, respectively. Each light projecting unit 19 is configured to irradiate light into the photographing range of the camera unit 16. Examples of the light projecting unit 19 are well-known laser light sources or projectors. For example, the light projecting unit 19 can indicate a specific place on the front side of the wearer by means of a laser emitted from a laser light source. Furthermore, it is also possible to present text or graphics by scanning the laser emitted from the laser light source (for example, Lissajous scanning). Furthermore, a well-known micro-projector can be used as the light projecting unit 19 to project image light toward the front side of the wearer. Furthermore, each light projecting unit 19 is configured to independently change the irradiation direction of the light by means of a driving unit (actuator) provided inside the housing.

[0043] The control unit 11 of the terminal device 10 performs calculation processing to control other elements of the terminal device 10. A processor such as a CPU can be used as the control unit 11. The control unit 11 basically reads out the program (including the OS) stored in the storage unit 12 and expands it in the main memory, and performs predetermined calculation processing in accordance with the program. In addition, the control unit 11 can appropriately write the calculation results in accordance with the program into the storage unit 12 or read them from the storage unit 12. In addition, as described in detail later, the control unit 11 of the terminal device 10 can follow the command of the application program executed on the client device 20 or the cloud 30, read the predetermined function from the storage unit 12, and execute the function, thereby controlling various hardware elements or performing predetermined calculations.

[0044] The storage unit 12 of the terminal device 10 is an element for storing information used in the calculation processing of the control unit 11 or its calculation results. The storage function of the storage unit 12 can be realized by a non-volatile memory such as an HDD and an SDD. In addition, the storage unit 12 can also have a function as a main memory, and the main memory is used to write or read the intermediate process of the calculation processing performed by the control unit 11. The memory function of the storage unit 11 can be realized by a volatile memory such as a RAM or a DRAM. In addition, the ID information inherent to the user who owns the terminal device can be stored in the storage unit 11. In addition, the identification information of the terminal device 10 on the network, that is, the IP address, can also be stored in the storage unit 11.

[0045] The communication unit 13 of the terminal device 10 is an element for wireless communication with a server device on the cloud or another terminal device 10. In order to communicate with a server device or another neck-hanging device via the Internet, the communication unit 13 may be a communication module for wireless communication using a known mobile communication standard such as 3G (W-CDMA), 4G (LTE / LTE-Advanced), or 5G, or a wireless LAN such as Wi-Fi (registered trademark). In addition, in order to directly communicate with another terminal device 10, the communication unit 13 may be a communication module for short-range wireless communication such as Bluetooth (registered trademark) or NFC.

[0046] In addition, Figure 3 In addition to the structure shown, the terminal device 10 may also be appropriately equipped with module devices such as a gyro sensor, an acceleration sensor, a geomagnetic sensor, a proximity sensor, or a GPS sensor, which are sensors mounted on a general portable information terminal.

[0047] Depend on Figure 2 by Figure 3 It can be seen that in this embodiment, the terminal device 10 does not have a display device such as a display or a monitor. Therefore, although the operator can perform relatively simple operations such as turning on / off each hardware element, it is difficult to perform complex operations such as operating an application program. Therefore, for such a terminal device 10 without a display device, remote operation through the client device 20 is particularly effective.

[0048] Figure 3 The hardware elements of the client device 20 are further shown. A well-known personal computer (PC) can be used as the client device 20. Specifically, the client device 20 includes: a control unit 21, which is composed of a processor such as a CPU; a storage unit 22, which includes a non-volatile memory and a volatile memory; and a communication unit 23, which is composed of a communication module for Internet connection. In addition, the client device 20 further includes: an input device 24 such as a mouse or a keyboard, and a display device 25 such as a display. The client device 20 is a terminal operated by a support person (operator, etc.) who provides support information to the on-site operator. As a result, the support person can send work instructions, etc. to the on-site operator from a remote location.

[0049] Figure 3It also shows the hardware elements of the cloud 30. A well-known cloud system composed of one or more server devices 31 can be used as the cloud 30 in this embodiment. In the information processing system 100 of the present invention, this cloud 30 is basically used to store various types of applications, perform particularly heavy processing, or store highly confidential information. As a result, it is possible to save the time of pre-installing various applications on each terminal device 10, distribute the processing load on each terminal device 10, or ensure the security of highly confidential information.

[0050] Furthermore, the cloud 30 includes a database 32. Furthermore, this database 32 can be constructed by separately setting up a server device for storing information, or it can be virtually constructed by aggregating the storage capacity of each server device. In the present embodiment, in this database 32, authentication information required for the execution of applications used in this system is stored. For example, in order to execute an application, account authentication is required, and it is sufficient to pre-store a specific account name and its password in the database 32. In addition, various data required for utilizing the application can also be pre-registered in this database 32. Furthermore, information related to the wearer (operator) of the terminal device 10 or information related to the operator (support personnel) of the client device 20 can also be registered in this database 32.

[0051] Figure 4 The main software components constituting the information processing system 100 of the present invention are shown. Figure 4 , the basic architecture of the information processing system 100 is described. Figure 2 and Figure 3 The various hardware shown, in particular, the control unit 11, storage unit 12 and communication unit 13 of the terminal device 10, the server device 31 and database 32 of the cloud 30, and the control unit 21, storage unit 22 and communication unit 23 of the client device 20 are implemented.

[0052] In the information processing system 100 of the present invention, SDKs including various functions are basically introduced to each terminal device 10 in advance, and specific functions are read in each terminal device 10 in accordance with the application program executed in the client device 20 or the cloud 30. Thus, even if the application program is not installed in advance in each terminal device 10, processing in accordance with the application program can be performed in each terminal device 10. In addition, the so-called "SDK" refers to a software development kit, which is a group of information obtained by packaging various functions used when executing a program. As the SDK, an SDK provided to application developers such as Android SDK, iOS SDK, Microsoft Windows SDK, etc. can be appropriately used.

[0053] Specifically, Figure 4 As shown, a first middleware 111 and an SDK 112 are provided in each of the plurality of terminal devices 10. Moreover, a foreground and / or background service 113 may be further provided in each terminal device 10. Furthermore, a second middleware 132 is provided in the cloud 30 to cooperate with the first middleware 111 of each terminal device 10. Furthermore, applications 121 and 131 are provided in the client device 20 and / or the cloud 30, and the applications 121 and 131 are designed to utilize the functions of the SDK 112 provided in the terminal device 10. That is, the codes of these applications 121 and 131 include processing commands, and the processing commands are used to call the functions included in the SDK 112. Furthermore, if the application is executed in the client device 20 or the cloud 30, the processing commands are sent from the second middleware 132 in the cloud 30 to the first middleware 111 of the terminal device 10 in real time and at high speed. The control unit 11 of the terminal device 10 reads and executes a specific function from the SDK 112 according to the processing command received in the first middleware 111, so as to make each hardware element 14 to 19 function, or perform a predetermined operation process (image processing, voice processing, etc.). In addition, the control unit 11 of the terminal device 10 can broadcast the processing command (message) received in the first middleware 111 in the terminal and call an appropriate service. In this way, the functions provided by the SDK 112 in the terminal device 10 can be remotely operated via a communication network such as a public network / closed network.

[0054] As the first middleware 111 and the second middleware 132, for example, a message system for exchanging information between multiple applications or services can be used. An example of the second middleware 132 located in the cloud 30 is a message broker, and an example of the first middleware 111 located in each terminal device 10 is a subscriber. In this case, the client device 20 or the application 121, 131 in the cloud 30 can be regarded as the source of the message, that is, the publisher. That is, the source of the message is called the publisher, the receiving side of the message is called the subscriber, and the function of relaying the message between them is called the message broker. The message published by the publisher is distributed to the subscriber via the message broker. The subscriber (first middleware 111) can also register information called subscription to specify the received messages, instead of receiving all messages distributed from the message broker (second middleware 132). That is, between the first middleware 111 and the second middleware 132, it is possible to specify not only which subscriber (first middleware 111) a message is to be transmitted from the publisher, but also which message is to be received at the subscriber (first middleware 111). Such a message system is also called Pub / Sub type (publish-subscribe type).

[0055] Furthermore, the client device 20, which is the source of message publishing, can call and execute the application 131 of the cloud 30 by, for example, following the code of the protocol (API) called REST. The so-called REST (REpresentational State Transfer) refers to one of the design principles suitable for enabling multiple applications to collaborate in a distributed system, and generally refers to the following API: If parameters are specified and a specific URL is accessed using HTTP, a message described in XML, JSON, etc. is sent. In this way, by utilizing the REST API in the communication protocol between the client device 20 and the cloud 30, the installation language of the application becomes free. In addition, since certain processing can be achieved through a combination of REST APIs, low coding can also be achieved. As a result, it is expected that the number of developers of applications for controlling the terminal device 10 will increase.

[0056] exist Figure 4 There are roughly three communication paths between the first middleware 111 and the second middleware 132 shown. The first is the communication path of the command. The command can be asynchronously transmitted from the second middleware 132 to the first middleware 111 in the form of upload / download using the aforementioned Pub / Sub type message system. The processing result of the asynchronous command can also be notified through other paths such as network hooks. In addition, as long as it is simple authentication information such as an account name and password, it can be transmitted to the first middleware 111 through the communication path of this command. The second is the communication path of discrete time series data. The discrete time series data mentioned here refers to time series data with relatively small throughput, such as GPS longitude and latitude information or battery remaining information, wearing / non-wearing status, posture information as a result of interpreting the measurement results of the acceleration sensor, etc. These discrete time series data can be communicated through the same message system, and paths such as network hooks can also be used. That is, the second and first communication paths are actually the same communication path. The third is the communication path of continuous time series data. Typical examples of continuous time series data are voice data or video data. In addition, continuous time series data includes time series data with relatively large throughput, such as data measured at high frequencies. Since such continuous time series data exceeds the communication capacity suitable for the message queuing system, it is necessary to prepare another communication path. For example, the communication paths are WebRTC, HLS (HttpLive Streaming), Amazon Kinesis Video Stream, etc., and you only need to select an appropriate communication method according to the communication content.

[0057] When looking at this architecture from above, it can be understood as an expansion of the concept of OS (Operating System). In other words, the original meaning of OS can be understood as "the basic software used to operate the computer system". It can be considered that the interface of the basic software used to perform the actions of a single terminal device 10 or the coordinated actions of multiple terminal groups is located in Figure 4 The dashed portion shown is between the client device 20 and the cloud 30. Therefore, the concept of the expanded OS includes the terminal device 10 and the cloud 30 on the Internet. This makes sense for developers (users) by considering that the OS is given and not included in the development object (it is an operation object). That is, in the past, it was necessary to start with the construction of cloud services or the development of Internet communication programs. However, since these are provided to developers as functions provided by the OS, developers only need to develop applications without considering the existence of the Internet or cloud services.

[0058] Next, a practical example of the architecture of the information processing system 100 of the present invention will be described.

[0059] Figure 5 This shows an example of a case where processing in accordance with an application requiring account authentication is performed on the terminal device 10. As a specific example, an application for a web conference is given. Figure 5 In the illustrated embodiment, the web conference application 131 is installed in the cloud 30. As the web conference application 131, for example, Zoom (registered trademark) or Microsoft Teams (registered trademark) is conceivable.

[0060] exist Figure 5In the illustrated embodiment, the client device 20 instructs the cloud 30 to execute an application 131 for a web conference. The client device 20 has an application 121 for its purpose. The application 121 of the client device 20 may be a dedicated program for instructing the cloud 30 to execute, or may be a general-purpose program such as a web browser. The client device 20 may issue a predetermined instruction to the cloud 30 through these applications 121. The execution instruction issued to the cloud 30 includes information specifying a terminal device 10 that becomes a participant of the web conference among a plurality of terminal devices 10. Furthermore, the execution instruction issued to the cloud 30 may include information specifying a terminal device 10 (a) to which a master account of the web conference is assigned, and information specifying another terminal device 10 (b) that participates in the web conference with a guest account. In this way, the client device 20 can instruct any terminal device 10 to start a web conference, and the master account and the guest account can be assigned arbitrarily. In this case, the application 121 of the client device 20 or the control unit 21 for controlling the program functions as a "processing instruction unit" described in the claims.

[0061] Furthermore, the allocation of accounts for web conferencing can also be performed automatically in the cloud 30, for example. For example, if the cloud 30 receives an application 131 for web conferencing from the client device 20, the program can be started, and a main account can be allocated to the terminal device 10 (a) designated as a participant in the web conference, which has the best conditions at that time, such as those related to the network environment and the battery, and a guest account can be allocated to the other terminal devices 10 (b). For example, a reasonable method is to select a terminal device 10 with the best network environment and more remaining battery (or connected to an external power supply) as the host. In this way, it is also possible to automatically instruct each terminal device 10 in the cloud 30 on the processing in the web conference. In this case, the application 131 of the cloud 30 and the server device 31 for controlling the program function as the "processing instruction unit" described in the claims.

[0062] Next, the web conference application 131 is executed in the cloud 30, so that the first terminal device 10 (a) performs processing for the master account and the second terminal device 10 (b) performs processing for the guest account. Authentication information is required to perform processing for the master account. For this purpose, the authentication information for the master account is provided to the first terminal device 10 (a) from the cloud 30. This authentication information is authentication information for accessing an external server device of the operator providing the web conference service and logging in as a master account. In addition, the external server device mentioned here can also be regarded as a server device constituting the cloud 30. In the web conference application 131, the authentication information required for performing processing for the master account is stored in the database 32. An example of the authentication information is a combination of an account name and a password or a key. Therefore, in the present embodiment, the database 32 of the cloud 30 functions as the "authentication information storage unit" described in the claims. However, this authentication information does not necessarily have to be stored in the cloud 30, but can also be stored in the client device 20. In this case, this authentication information only needs to be provided from the client device 20 to the first terminal device 10 (a) via the cloud 30. In this case, the storage unit 22 of the client device 20 functions as an “authentication information storage unit” described in the claims.

[0063] Next, the cloud 30 transmits the processing commands or required data generated after executing the application 131 for web conferencing to each terminal device 10 (a) and 10 (b) through the second middleware 132 and the message system. And, at this time, the cloud 30 transmits the authentication information required for the processing of the main account to the first terminal device 10 (a). In addition, the cloud 30 can also issue a temporary token that can replace the authentication information and transmit it to the first terminal device 10 (a). In addition, generally, authentication information is not required in the case of a guest account, but in the case where authentication information for a guest account is required, the cloud 30 also transmits the authentication information for the guest account or a temporary token instead to the second terminal device 10 (b). Therefore, the application 131 of the cloud 30 or the server device 31 used to control this program plays the role of the "authentication information providing unit" described in the claims.

[0064] The first terminal device 10 (a) receives a processing command (function call command), necessary data and authentication information from the cloud 30 through the first middleware 111. Similarly, the second terminal device 10 (b) receives a processing command (function call command) and necessary data from the cloud 30 through the first middleware 111. In response to the processing command from the cloud 30, the first terminal device 10 (a) reads and executes a specific function from the SDK 112. At this time, if predetermined data or authentication information is required, they are used to execute the function. Alternatively, the first terminal device 10 (a) broadcasts the processing command from the cloud 30 in the terminal and calls the appropriate service 113. For example, the first terminal device 10 (a) activates the sound collection unit 14 and the camera unit 16 to obtain voice data and video data, and transmits them to the second terminal device 10 (b) via the Internet. Similarly, the first terminal device 10 (a) also receives voice data and video data from the second terminal device 10 (b). Furthermore, the first terminal device 10 ( a ) can also access an external server device of a company providing a web conference service and perform authentication processing of the master account.

[0065] Furthermore, in the event that the first terminal device 10(a) is not suitable as the host for the web conference, the authentication information for the master account can be transferred from the first terminal device 10(b) to the second terminal device 10(b). For example, when the remaining battery level of the first terminal device 10(a) falls below a predetermined value or the quality of network communication deviates from a predetermined condition, the cloud 30 automatically transfers the authentication information from the first terminal device 10(b) to the second terminal device 10(b). In this way, the web conference can be continued stably.

[0066] In this way, various functions based on the SDK 112 in each terminal device 10 can be remotely operated via the communication network according to the application executed in the cloud 30. In particular, by executing the application in the cloud 30 and appropriately combining and utilizing the functions of the SDK 112 in the terminal device 10, the terminal device 10 can perform substantially the same actions as when the application is installed in the terminal device 10. In the above example, an application for network conferencing is cited, but the present invention is not limited to this and can be applied to various programs.

[0067] Next, another practical example of the architecture of the information processing system 100 of the present invention will be described. In this example, the location where the processing according to the application is performed can be arbitrarily specified to a certain extent.

[0068] Figure 6 The basic architecture of the embodiment shown is similar to Figure 4The embodiment shown is the same, but similar to the SDK 112 and the service 113 of the terminal device 10, the cloud 30 also includes an SDK 133 and a service 134. In addition, the SDK 122 and the service 123 may also be incorporated into the client device 20.

[0069] exist Figure 6 In the system structure shown, the case of providing a remote support service or a work image recording service using the terminal device 10 is studied. In the former, the video captured by the terminal device 10 is used as an image call with the client device 20 on the remote support center side, and is saved as video data to the server device on the cloud 30. In the latter, after the video data is temporarily accumulated in the terminal device 10, it is forwarded and saved in batches to the server device on the cloud 30 as video data. In this service, privacy issues may arise when using or saving video data. Therefore, it is necessary to perform mosaic processing on predetermined areas in the video. Typically, mosaic processing is performed on the background other than the face of the person or the photographic target object captured in the video.

[0070] Figure 7 A flowchart simply shows the processing of an application for automatically mosaicking the face area in a video. Specifically, the application includes the following steps: obtaining video data through the terminal device 10 (step S1); detecting the face area of ​​a person from the video data (step S2); determining whether the face area is detected (step S3); if the face area is detected, mosaicking the area (step S4); and obtaining the final video data (step S5). For example, in Figure 7 The bottom section shows that this flowchart is presented in a functional programming way.

[0071] In this procedure, "MOVIE_DATA" refers to the video data obtained by the terminal device 10. Also, "GET_MOVIE" refers to the process of starting the camera of the terminal device 10 to obtain the video data (step S1), "DETECT_FACE" refers to the face detection process (step S2), and "CREATE_MOSAIC" refers to the mosaic process of the face area (step S4). Also, "i" is the ID of the terminal device 10, and "j" indicates the place where each process is performed. Figure 6In the system shown, since "GET_MOVIE" is a process that can only be executed in the terminal device 10, "i" is assigned as the place where the process is executed. On the other hand, the place where the other processes of "DETECT_FACE" and "CREATE_MOSAIC" are performed can be arbitrarily specified. Specifically, these processes can be selected from the terminal device 10, the client terminal 20 and the cloud 30. In addition, the server device 31 that executes each process can also be individually specified from the multiple server devices 31 that constitute the cloud 30. For example, a so-called fog server can be designated as the execution place of the process, and the fog server is a server device set at the work site (factory site, etc.) of the operator wearing the terminal device 10, or a server device set in the MEC (Multi-Access Edge Computing) area in the local 5G communication network.

[0072] For the face detection processing (step S2) and the face area mosaic processing (step S4), it is essentially arbitrary where this processing is performed. For example, after uploading the original video data from the terminal device 10 to the cloud 30, the server device 31 on the cloud 30 performs the face detection processing and the face area mosaic processing in a centralized manner, thereby seeking to reduce the processing load of the terminal device 10 and improve the efficiency of these processes. However, in the case where privacy protection is given top priority, from the perspective of privacy protection, the following state can be said to be the most appropriate method: the face detection processing and the face area mosaic processing are performed in the terminal device 10 to complete the video data, and the face area mosaic processing is completed at the time when the video data is output from the terminal device 10 to the outside. In this way, it is possible to arbitrarily decide which processing is performed at which location according to the characteristics or conditions of the data being processed.

[0073] An SDK having various functions including the above-mentioned "GET_MOVIE", "DETECT_FACE" and "CREATE_MOSAIC" functions is prepared in advance not only in the terminal device 10 but also in the cloud 30 (or the server device 31 constituting the cloud) or the client device 20. For example, in the client device 20, the following is published: Figure 7 The program shown is transmitted to the cloud 30, and the second middleware 132 of the cloud 30 uniformly distributes the message to the terminal devices 10. If each terminal device 10 receives the message through the first middleware 111, it reads and executes the function specified in the program from the SDK 112. Similarly, the cloud 30 or the client device 20 itself also reads and executes the function specified in the program from the SDK 133, 122. In this way, the place to execute the program can be simply specified according to the processing content.

[0074] Here, the case of mosaic processing on the face area of ​​video data is taken as an example for explanation. However, this is only an example, and the system architecture of the present invention can arbitrarily specify the execution location of the program to a certain extent according to the processing content for various application programs.

[0075] Furthermore, the mosaic processing described in this example may also be set with permissions relative to the executable location. For example, the devices that can perform mosaic processing on, for example, photographic images may be limited in advance, so that only the devices that are granted execution permissions (or allowed to execute) among the terminal device 10, the client device 20, and the cloud 30 can execute the mosaic processing. This permission relative to the executable location may be pre-stored in the database 32 on the cloud 30 as authentication information. For example, in the case where the execution permission is set for the mosaic processing, when it is necessary to perform the processing in the terminal device 10, the cloud 30 provides the terminal device 10 with the authentication information stored in the database 32. As a result, the terminal device 10 can perform the mosaic processing according to the authentication information.

[0076] In the above, in order to present the content of the present invention, the embodiments of the present invention are described with reference to the accompanying drawings. However, the present invention is not limited to the above-mentioned embodiments, and includes obvious changes or improvements obtained by those skilled in the art based on the matters described in the present invention.

[0077] Description of Reference Numerals

[0078] 10: Terminal device

[0079] 11: Control Department

[0080] 12: Storage

[0081] 13: Ministry of Communications

[0082] 14: Sound Collection

[0083] 15: Operation Department

[0084] 16: Camera Department

[0085] 17: Sensor Department

[0086] 18: Playback Department

[0087] 19: Lighting Department

[0088] 20: Client device

[0089] 21: Control Department

[0090] 22: Storage

[0091] 23: Ministry of Communications

[0092] 24 Input section

[0093] 25: Display unit

[0094] 30: Cloud

[0095] 31: Server device

[0096] 32: Database

[0097] 100: Information Processing Systems

[0098] 111: First middleware

[0099] 112: SDK

[0100] 113: Service

[0101] 121: Applications

[0102] 122: SDK

[0103] 123: Service

[0104] 131: Applications

[0105] 132: Second middleware

[0106] 133: SDK

[0107] 134: Service

Claims

1. An information processing system comprising: a plurality of terminal devices; a client device; and a cloud, which includes one or more server devices connected to the plurality of terminal devices and the client device via a network, characterized in that: The cloud has: a predetermined program including a plurality of processing commands; and a second middleware, which transmits a processing command generated after executing the program to the terminal device, The client device has: a processor that, after specifying the terminal device to the cloud, instructs execution of the program, The terminal device comprises: SDK, which is a group of information in which various functions used when executing the program are packaged; and a first middleware receiving the processing command transmitted from the second middleware, The client device or the cloud has: a processing instruction unit that selects a device that performs processing based on the processing instruction from among the terminal device, the server device, and the client device according to the processing instruction included in the program, The terminal device, the server device, or the client device selected by the process instruction unit executes a process based on the process command instructed by the process instruction unit.

2. The information processing system according to claim 1, characterized in that The processing instruction unit further specifies a specific terminal device from among the plurality of terminal devices, and instructs the specific terminal device to execute processing in accordance with the program. The client device or the cloud further has: an authentication information storage unit that stores authentication information required for executing the program; and an authentication information providing unit that provides authentication information or temporary information that replaces the authentication information to the specific terminal device designated by the processing instruction unit, the authentication information being necessary for performing a process in accordance with a procedure instructed by the processing instruction unit or for confirming whether the process is permitted, The specific terminal device performs a process based on an instruction from the process instruction unit using the authentication information or the temporary information provided from the authentication information providing unit.

3. The information processing system according to claim 2, characterized in that: The client device or the cloud further has a program execution unit for executing the program. The processing instruction unit instructs the specific terminal device to perform processing in accordance with the program executed by the program execution unit, The specific terminal device performs a process based on an instruction from the process instruction unit.

4. The information processing system according to claim 2, characterized in that: The program is an application for web conferencing. The authentication information includes authentication information for the main account of the web conference and authentication information for the guest account. The processing instruction unit specifies one terminal device and another terminal device from among the plurality of terminal devices, the one terminal device being assigned a master account for a web conference and the other terminal device being assigned a guest account for participating in the web conference hosted by the one terminal device.

5. The information processing system according to claim 4, characterized in that: The main account of the one terminal device and the guest account of the other terminal device can be exchanged.

6. The information processing system according to claim 4, characterized in that: The processing instruction unit specifies the one terminal device to which the primary account is allocated based on a remaining battery level or communication quality of the terminal device.

7. The information processing system according to claim 1, characterized in that: The terminal device is a terminal device without a display unit.

8. The information processing system according to claim 1, characterized in that: The processing instruction unit causes the plurality of server devices or the client device to perform at least a part of a process based on a processing command included in the program.

Images