Device control method and device control apparatus

Abstract

To provide a device control method capable of automatically controlling a device based on the relationship between device position information acquired by a camera and user position information.SOLUTION: A device control method acquires the image of a room, acquires device position information and person position information from the image, and controls the operating state of a device based on the relationship between the device position information and the person position information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a device control method in a wireless communication device for controlling peripheral devices using information acquired by a camera.

Background Art

[0002] Conventionally, a device control system is known that grasps various indoor environment information and gives control instructions to connected indoor devices.

[0003] Patent Document 1 discloses a device control apparatus that monitors various indoor environmental conditions (presence or absence of people, number of people, position, activity state, posture, size, position of objects, indoor brightness, security abnormality, fire abnormality, etc.) by combining a camera unit installed on the ceiling or wall or a camera unit and a detection unit that detects temperature and sound, and wirelessly sends the indoor information or information processed from the indoor information into a control command to an indoor device.

[0004] Conventionally, a monitoring system is also known that obtains audio data in the direction in which a camera device is imaging a specific subject by installing a camera device and a microphone array device.

[0005] Patent Document 2 discloses an audio processing system and an audio processing method that use audio collected by a microphone array device to perform emphasis processing on audio in a specified direction, then adjust the volume of the audio, suppress the occurrence of a large difference in volume before and after the emphasis processing, and improve convenience for users.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] The invention of Patent Document 1 controls indoor devices based on indoor information obtained from images. In the configuration of Patent Document 1, the indoor devices to be controlled may be installed anywhere, whether inside or outside the image, but they need to be connected to the device control apparatus in advance. That is, in the configuration of Patent Document 1, images are used to monitor the indoor environmental situation, and it does not determine the indoor devices to be controlled based on the images. Also, in the configuration of Patent Document 1, it is not assumed that the user arbitrarily moves the indoor devices, and the installation positions of the indoor devices cannot be obtained from the images.

[0008] Also, the indoor environment shown in Patent Document 1 does not include an acoustic space. Therefore, there is no description of a method for controlling the voice signal itself output by the indoor device based on the indoor information obtained from the image.

[0009] The invention of Patent Document 2, after receiving an input from the user, uses the voice picked up by the microphone array device, and the signal processing unit emphasizes the voice in the specified direction and adjusts the volume of the voice. In the configuration of Patent Document 2, in order to acquire the voice data in the directional state, the user needs to manually specify an arbitrary specified position on the image displayed on the display device. That is, in the configuration of Patent Document 2, the image is used for the user to input operations himself / herself, and it does not automatically output the optimal voice data for the user by the voice processing system based on the image.

[0010] An object of the present invention is to provide a device control method capable of automatically controlling a device based on the relationship between the position information of the device acquired by a camera and the position information of the user.

Means for Solving the Problems

[0011] The device control method is to acquire an indoor image, to acquire the position information of the device and the position information of the person from the image, Based on the relationship between the position information of the device and the position information of the person, the operating state of the device is controlled.

Effect of the Invention

[0012] According to the device control method of the present invention, there is an effect that the device can be automatically controlled based on the relationship between the position information of the device acquired by the camera and the position information of the user.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Modes for Carrying Out the Invention

[0014] Hereinafter, an information processing system according to an embodiment of the present invention will be described with reference to the drawings. The same reference numerals are given to the same parts in each figure. From the second embodiment onward, descriptions of matters common to the first embodiment will be omitted, and only different points will be described. In particular, the same operations and effects due to the same configuration will not be sequentially mentioned for each embodiment.

[0015] <<First Embodiment>> FIG. 1 is a block diagram showing a basic configuration of an information processing system 1. The information processing system 1 includes a data processing device 10, a wireless device 20, a device 30, and a camera 40.

[0016] Note that the wireless device 20 is, for example, a signal processing device that receives an audio signal from the device 30 and performs signal processing. Alternatively, the wireless device 20 may be a wireless access point that connects the device 30 to a network. Also, the device 30 is a desktop small microphone. The device 30 may be a desktop small speaker, a large speaker, a personal computer incorporating a microphone and speaker, a smartphone, a wearable terminal, or a device capable of recording and playing back sound such as music, etc. Hereinafter, these will be collectively referred to as the device 30.

[0017] Note that in the present invention, unless otherwise specified, the wireless device 20 and the device 30 are in a state where the wireless connection is completed.

[0018] The data processing device 10 is connected to the wireless device 20 and the camera 40. The camera 40 photographs the room R1. The camera 40 transmits the photographed image to the data processing device 10. The data processing device 10 acquires the position information of the device 30 and the position information of the user from the image of the room R1 photographed by the camera 40.

[0019] FIG. 2 is a block diagram showing a basic configuration of the data processing device 10. The data processing device 10 includes a CPU 11, a RAM 12, a memory 13, and a user interface (I / F) 14.

[0020] The memory 13 stores a program for operating the data processing device 10. The CPU 11 reads the program stored in the memory 13 into the RAM 12 and controls the operation of the data processing device 10. However, the program does not necessarily have to be stored in the memory 13. The CPU 11 may download the program from, for example, a server each time.

[0021] Next, with reference to FIGS. 3 and 4, the flow from when the camera 40 captures an image of the room R1 to when the data processing device 10 controls the operating state of the device 30 will be described.

[0022] FIG. 3 is an external perspective view of the room R1 according to the first embodiment. In the following description, the room R1 is assumed to be, for example, a conference room, but the room R1 is not limited thereto. In reality, the room R1 may be a hall or a lecture room.

[0023] In the present embodiment, the camera 40 installed in the room R1 is, for example, a ceiling camera. Note that the camera 40 is not limited to a ceiling camera, and may be a camera installed on a wall or a camera placed on a conference monitor. The camera 40 may be installed at a position where it can capture the device 30.

[0024] In FIG. 3, the viewing angle A1 of the camera 40 is indicated by a dashed line. That is, the camera 40 captures the range of the viewing angle A1 and transmits the captured image to the data processing device 10. In the room R1, speakers S1 and table microphones M1 are installed around the user, and speakers S2 and table microphones M2 are installed at positions away from the user. In the present embodiment, the speakers S1 and S2, and the table microphones M1 and M2 are an example of the device 30 of the present invention.

[0025] FIG. 4 is a flowchart showing the operations executed by the CPU 11 of the first embodiment. The data processing device 10 acquires the image of the room R1 captured by the camera 40 (S001).

[0026] Next, the data processing device 10 acquires the position information of the device 30 and the user based on the image received from the camera 40 (S002). Specifically, the data processing device 10 acquires the device information of the device 30 from the acquired image. The device information is information for identifying the device 30. The data processing device 10 acquires the position information of the device 30 based on the acquired device information. The device information is, for example, information related to the external shape of the product and corresponds to the model information. Note that the device information may be a two-dimensional barcode, a manufacturer logo, a product logo, an LED lighting pattern, or a barcode, etc.

[0027] Also, the position information of the device 30 and the user acquired by the data processing device 10 includes two-dimensional position information. The two-dimensional position information is X, Y coordinates (orthogonal coordinates) with a predetermined position (for example, the lower left) of the image captured by the camera 40 as the origin. Further, the data processing device 10 may further acquire, as position information, the distance between the position of the device 30 and the position of the user calculated based on the two-dimensional position information (hereinafter referred to as the device 30 - user distance).

[0028] Next, the data processing device 10 controls the operating state of the device 30 based on the relationship between the position information of the device 30 and the user acquired in step S002 (S003). The data processing device 10 performs control to, for example, turn on or off the power of the device 30.

[0029] The data processing device 10 may determine whether the device 30 - user distance is less than or equal to a predetermined value, and perform control to turn on or off the power of the device 30 based on the determination result.

[0030] FIG. 5 is an example of a flowchart showing the operations executed by the CPU 11 of the first embodiment. Since the processes of steps S001 and S002 in FIG. 5 are the same as the processes of steps S001 and S002 in FIG. 4, the description thereof is omitted.

[0031] After the data processing device 10 acquires the device 30 - user distance in step S002, it determines whether the device 30 - user distance is less than or equal to a predetermined value (S004).

[0032] When it is determined that the value is below the predetermined value (S004: YES), the data processing device 10 turns on the power of the device 30 (S005). When it is determined that the value is greater than the predetermined value (S004: NO), the data processing device 10 turns off the power of the device 30 (S006).

[0033] For example, in the room R1, when it is determined that the distance between the position of the user and the position of the speaker S2 (hereinafter referred to as the user-speaker S2 distance) and the distance between the position of the user and the position of the desktop microphone M2 (hereinafter referred to as the user-desktop microphone M2 distance) are each greater than the predetermined value, the data processing device 10 issues an instruction to turn off the power of the speaker S2 and the desktop microphone M2 to the wireless device 20.

[0034] As described above, in the present embodiment, the data processing device 10 controls the operating state of the device 30 based on the relationship between the position information of the device 30 and the user. For example, the user can simply place the device 30 that they want to use near them and place the device 30 that they do not use far from them, and only turn on the power of the device 30 that they want to use. Therefore, the user of the information processing system 1 can operate only the necessary devices 30. Therefore, the user can experience a new customer experience of feeling an improvement in the energy-saving effect of turning off the power of the device 30 that is not used without particularly performing any work.

[0035] 《Second Embodiment》 In the first embodiment, the data processing device 10 controls the operating state of the device 30 based on the relationship between the position information of the device 30 and the user. However, when the device 30 is an audio device that outputs sound and there are obstacles in the room that affect the sound, the following processing may be executed.

[0036] FIG. 6 is an external perspective view of room R2 according to the second embodiment. The components common to FIG. 3 are denoted by the same reference numerals, and the description thereof is omitted. Room R2 is different from room R1 in that it includes an obstacle O1. The obstacle O1 is a large piece of furniture, a partition, or the like.

[0037] In this embodiment, the number of cameras 40 installed in room R2 may be one or a plurality.

[0038] FIG. 7 is a flowchart showing the operations executed by CPU 11 according to the second embodiment. Since the processes of steps S001 and S002 in FIG. 7 are the same as the processes of steps S001 and S002 in FIG. 4, the description thereof is omitted.

[0039] After acquiring the position information of device 30 and the user in step S002, data processing device 10 further acquires the position information of obstacle O1 based on the image received from camera 40 (S007). At this time, in addition to the position information of the obstacle, data processing device 10 may acquire the actual size of obstacle O1 based on images acquired from different directions by a plurality of cameras 40.

[0040] Next, data processing device 10 controls the operating state of device 30 based on the relationship among the position information of device 30 acquired in step S002, the position information of the user, and the position information of obstacle O1 acquired in step S007 (S008).

[0041] In step S008, the data processing device 10 determines whether there is an obstacle O1 that reflects or blocks sound between the device 30 and the user's position, and adjusts the volume of the device 30 based on the determination result. Alternatively, for example, the data processing device 10 may make the determination using a training model obtained by training a predetermined model such as a neural network with the relationship between the shape and size of the obstacle. Further, the data processing device 10 may make the determination using a training model obtained by training a predetermined model such as a neural network with the relationship between the size of the obstacle O1, the distance between the position of the obstacle O1 and the position of the device 30 (hereinafter referred to as the obstacle O1-device 30 distance).

[0042] When it is determined that the obstacle O1 reflects or blocks the sound output from the speaker S1, the data processing device 10 issues an instruction to, for example, lower the volume of the speaker S1 and increase the volume of the speaker S2 to the wireless device 20. Thereby, the user of the information processing system 1 can obtain a new customer experience that the influence of the sound by the obstacle O1 is minimized and an optimal acoustic environment in the room R2 can be automatically created.

[0043] 《Third Embodiment》 In the first and second embodiments, the data processing device 10 controls the operating state only for the device 30 that is installed within the viewing angle A1 of the camera 40 and for which the data processing device 10 has acquired the position information. However, when the device 30 to be included in the control target is installed outside the viewing angle of the camera 40 and the position information of the device 30 cannot be acquired from the image, the following processing may be executed.

[0044] FIG. 8 is an external perspective view of the room R3 according to the third embodiment. The same reference numerals are given to the configurations common to FIG. 3, and the description thereof is omitted. The room R3 is different from the room R1 in that it includes ceiling speakers S3 and S4. In the present embodiment, the ceiling speakers S3 and S4 are an example of the device 30 of the present invention.

[0045] The user of the information processing system 1 may want to include the ceiling speakers S3 and S4 installed outside the viewing angle of the camera 40 in the control target of the data processing device 10. At this time, the user of the information processing system 1 can input the device information of the ceiling speakers S3 and S4 in advance to the user I / F 14, so that regardless of the installation location, the data processing device 10 can control the operating states of the ceiling speakers S3 and S4.

[0046] FIG. 9 is a flowchart showing operations executed by the CPU 11 according to the third embodiment. Since the processes in steps S001 and S002 in FIG. 9 are the same as the processes in steps S001 and S002 in FIG. 4, the description thereof is omitted.

[0047] The data processing device 10 receives an input from the user of the information processing system 1 via the user I / F 14. Specifically, the user of the information processing system 1 inputs the device information of the device 30 that the user wants to include in the control target of the data processing device 10. For example, the user determines a unique ID (numerical value) etc. of the device 30 that the user wants to include in the control target, and inputs the unique ID as the device information. Further, the user may further input the position information of the device 30 that the user wants to include in the control target as the device information. The data processing device 10 acquires the device information of the device 30 outside the viewing angle input by the user of the information processing system 1 (S011).

[0048] Next, the data processing device 10 acquires an image of the room R3 from the camera 40 (S001), acquires the position information of the device 30 and the user within the viewing angle based on the image (S002), and then controls the operating states of both the device 30 outside the viewing angle and the device 30 within the viewing angle based on the relationship between the position information of the device 30 outside the viewing angle, the position information of the device 30 within the viewing angle, and the position information of the user (S009).

[0049] As described above, in this embodiment, even for the device 30 not included in the image, the user of the information processing system 1 can input the device information to the data processing device 10 in advance, so that the control state by the data processing device 10 can be maintained.

[0050] "Fourth Embodiment" FIGS. 10, 11, and 12 are plans of the large conference room R4 according to the fourth embodiment. The ranges indicated by the broken lines in FIGS. 10, 11, and 12 are the viewing angles A2 to A5 of the camera 40 in the present embodiment.

[0051] In the present embodiment, the number of cameras 40 installed in the large conference room R4 may be one or a plurality. Also, the viewing angles A2 to A5 can be arbitrarily set by the user of the information processing system 1. Further, a monitor (not shown) for displaying the image captured by the camera 40 in the large conference room R4 may be installed so that the user of the information processing system 1 can visually recognize the viewing angles A2 to A5 of the camera 40.

[0052] As shown in FIG. 10, a plurality of table microphones M3 to M7 are installed near the users in the large conference room R4. In the present embodiment, the table microphones M3 to M7 are an example of the device 30 of the present invention.

[0053] In FIG. 10, the table microphones M3 to M7 are already wirelessly connected to the wireless device 20 and are in a state where their operating states are controlled by the data processing device 10. In such a situation, for example, assume that two new users bring a table microphone M8, which is a device not wirelessly connected to the wireless device 20 (hereinafter referred to as an unconnected device), into the large conference room R4 and change the layout as shown in FIG. 11. In such a case, the data processing device 10 acquires the position information of the table microphone M8 from the image newly received from the camera 40, and starts newly controlling the operating state of the table microphone M8 in addition to the table microphones M3 to M7 from the time when the position information is acquired.

[0054] In this embodiment, the data processing device 10 continuously receives images from the camera 40 in real time. That is, the data processing device 10 monitors the movements of the device 30 and the user in real time and acquires the position information of the device 30 and the user. As a result, the user of the information processing system 1 can obtain a new customer experience of always maintaining the operating state of the device 30 in an optimal state for the user.

[0055] Also, the user of the information processing system 1 may want to return from the layout of FIG. 11 to the layout of FIG. 10. In such a case, the user may move the desktop microphone M8 outside the viewing angle A2 of the camera 40. When the data processing device 10 can no longer recognize the desktop microphone M8 from the image received from the camera 40, it issues an instruction to stop the operation of the desktop microphone M8 to the wireless device 20.

[0056] Also, the user of the information processing system 1 may want to change from the lecture-style layout shown in FIG. 11 to the discussion-style layout shown in FIG. 12. In such a case, the user of the information processing system 1 sets a plurality of regions (viewing angles A3, A4, A5) from the image of the camera 40. When the viewing angles A3, A4, A5 are set, the data processing device 10 switches the operating state of the device 30 to control suitable for each viewing angle.

[0057] In this way, in this embodiment, the user of the information processing system 1 can easily create a space like a small meeting room by setting the viewing angles A3, A4, A5 of a smaller space from the viewing angle A2 of the camera 40 without moving around the room.

[0058] In this embodiment, since the data processing device 10 regards the viewing angles A3, A4, and A5 as independent spaces respectively, the control of the desktop microphones M3 to M5 is different for each viewing angle. For example, assume that the discussion of the group shown in the lower left (viewing angle A3) of FIG. 12 ends and the four users within the viewing angle A3 leave the large conference room R4. In this case, when the state where no user is detected from the viewing angle A3 is maintained for a predetermined time, the data processing device 10 regards the discussion as ended, gives an instruction to turn off the power of the desktop microphone M3 to the wireless device 20, and releases the control of the desktop microphone M3.

[0059] Also, assume that finally the discussions of the groups shown in the upper left (viewing angle A4) and the right (viewing angle A5) of FIG. 12 also end and all users leave the large conference room R4. In this case, as described above, when the state where no user is detected from the viewing angles A4 and A5 is maintained for a predetermined time, the data processing device 10 gives an instruction to turn off the power of the desktop microphones M4 and M5 to the wireless device 20, and releases the control of the desktop microphones M4 and M5.

[0060] Note that when releasing the control of the desktop microphones M3 to M5, the data processing device 10 may store the control states of the respective devices 30 and the data processing device 10 of the viewing angles A3 to A5 immediately before the release in the memory 13. Thereby, when the user enters the large conference room R4 again and resumes the discussion in the layout shown in FIG. 12, the data processing device 10 can resume the control of the desktop microphones M3 to M5 only by reading out the previous connection state from the memory 13.

[0061] Also, instead of releasing the control of the desktop microphones M3 to M5, the data processing device 10 may give an instruction to the wireless device 20 to switch the desktop microphones M3 to M5 to the sleep state.

[0062] In addition, the user of the information processing system 1 may input a preferred standard pattern into the user I / F 14 in advance. Specifically, the user of the information processing system 1 inputs, as a standard pattern, a pattern in which the angle of view of the camera 40 is arbitrarily set, into the user I / F 14. At this time, if there is a device 30 for which it is desired to maintain the control state regardless of the user's position or the angle of view, the user of the information processing system 1 inputs the device information of the device 30 into the user I / F 14. In this way, the data processing device 10 that has received the input of the standard pattern starts the control of the device 30 from the state of the standard pattern every time.

[0063] As a result, the user of the information processing system 1 can obtain a new customer experience in which there is no need to wait for the processing of the data processing device 10 and the meeting can be started immediately.

[0064] Finally, the description of this embodiment should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is shown not by the above-described embodiment but by the scope of the claims. Further, the scope of the present invention is intended to include all modifications within the meaning and scope equivalent to the scope of the claims.

Description of Reference Numerals

[0065] 1: Information processing system, 10: Data processing device, 11: CPU, 12: RAM, 13: Memory, 14: User interface, 20: Wireless device, 30: Device, 40: Camera, A1 to A5: Angle of view, R1 to R3: Room, R4: Large conference room, S1, S2: Speaker, S3, S4: Ceiling speaker, M1 to M8: Tabletop microphone, O1: Obstacle

Claims

1. Capture images of the room, Acquire device location information and person location information from the image; controlling an operation state of the device based on a relationship between location information of the device and location information of the person; Device control method.

2. Obtain more location information of obstacles, Controlling an operation state of the device based on a relationship between position information of the device, position information of the person, and position information of the obstacle. The device control method according to claim 1 .

3. If the device can no longer be recognized from the image, Stopping the operation of the device that is no longer recognized; The device control method according to claim 1 or 2.

4. Displaying the image capture range on a monitor. The device control method according to claim 1 or 2.

5. a control device that is connected to the device and controls a device outside the angle of view that is not included in the image based on the image; The device control method according to claim 1 or 2.

6. Recognizing an unconnected device that is not connected to a control device that controls the device from the image; Connecting the unconnected device to the control device; causing the control device to control the unconnected device; The device control method according to claim 1 or 2.

7. The control of the operating state of the device includes: powering the device on and off; connecting or disconnecting said device to a wireless device; adjusting the volume of the device. The device control method according to claim 1 or 2.

8. A camera for capturing images of the room; a processing unit for acquiring device position information and person position information from the image, controlling an operation state of the device based on a relationship between location information of the device and location information of the person; Equipment control device.

9. The processing unit further acquires position information of an obstacle, Controlling an operation state of the device based on a relationship between position information of the device, position information of the person, and position information of the obstacle. The device control device according to claim 8.

10. When the processing unit is unable to recognize the device from the image, Stopping the operation of the device that is no longer recognized; The device control device according to claim 8 or 9.

11. The equipment control device further comprises a monitor; Displaying the image capture range on the monitor. The device control device according to claim 8 or 9.

12. The processing unit is connected to a control device that controls the device, and controls a device outside the angle of view that is not included in the image based on the image. The device control device according to claim 8 or 9.

13. The processing unit recognizes, from the image, an unconnected device that is not connected to a control device that controls the device, and Connecting the unconnected device to the control device; causing the control device to control the unconnected device; The device control device according to claim 8 or 9.

14. The processing unit includes: powering the device on and off; connecting or disconnecting the device to a control device; Adjust the volume of said device; The operating state of the device is controlled by any one of the methods described above. The device control device according to claim 8 or 9.

Images