Communication systems, communication methods, and programs

Abstract

This system allows operators to easily initiate communication with users, even when there is a mix of users who possess wireless terminals and those who do not. [Solution] The identification unit 26 acquires video footage of a remote space captured by a camera. The identification unit 26 identifies a target object from the video footage. The determination unit 27 determines that if the first communication device is associated with the target object identified by the identification unit 26, the first communication device is the communication device used for communication with the target object. If the first communication device is not associated with the identified target object, the determination unit 27 determines that the second communication device is the communication device used for communication with the target object.

Description

Technical Field

[0004] ,

[0005] , ,

[0001] The present invention relates to a communication system, a communication method, and a program.

Background Art

[0002] As related art, Patent Document 1 discloses a glasses-type communication terminal that executes a predetermined process using a user's line of sight as a trigger. In Patent Document 1, a plurality of users each wear a glasses-type communication terminal. The glasses-type communication terminal detects the line of sight of the user wearing the glasses-type communication terminal. Also, the glasses-type communication terminal detects the line of sight of another user wearing another glasses-type communication terminal. The glasses-type communication terminal performs communication with the glasses-type communication terminal of another user when the line of sight of the user and the line of sight of another user match.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, a plurality of users each wear a glasses-type communication terminal, and communication between the glasses-type communication terminals is performed based on the detection result of the line of sight. However, in an area where a plurality of users such as a work site each perform their own work, not all users necessarily wear a glasses-type communication terminal. Patent Document 1 has a problem that communication can only be performed between those who wear the glasses-type communication terminal.

[0005] In view of the above circumstances, an object of the present invention is to provide a communication system, a communication method, and a program that enable an operator to easily start communication with a user even when users who possess a wireless terminal and users who do not possess a wireless terminal are mixed. [Means for solving the problem]

[0006] A communication system according to a first aspect of this disclosure includes a display unit that displays images of a remote space captured by a camera, an identification unit that identifies an object displayed on the display unit, and a determination unit that determines a communication device used for communication with the object identified by the identification unit. In the communication system, the determination unit determines that if a first communication device is associated with the object identified by the identification unit, the first communication device associated with the object is the communication device used for communication with the object, and if the first communication device is not associated with the object identified by the identification unit, the determination unit determines that a second communication device not associated with the object is the communication device used for communication with the object.

[0007] A communication method according to a second aspect of the present disclosure includes displaying an image of a remote space captured by a camera on a display unit, identifying an object displayed on the display unit, determining that if a first communication device is associated with the identified object, the first communication device associated with the object is the communication device used for communication with the object, and determining that if the first communication device is not associated with the identified object, a second communication device not associated with the object is the communication device used for communication with the object.

[0008] A program according to a third aspect of this disclosure causes a computer to perform the following processes: display images of a remote space captured by a camera on a display unit; identify a target object displayed on the display unit; if a first communication device is associated with the identified target object, determine that the first communication device associated with the target object is a communication device used for communication with the target object; and if the first communication device is not associated with the identified target object, determine that a second communication device not associated with the target object is a communication device used for communication with the target object. [Effects of the Invention]

[0009] According to the present invention, even when there is a mix of users who possess wireless terminals and users who do not possess wireless terminals, the operator can easily initiate communication with the users. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram illustrating a schematic configuration example of a communication system. [Figure 2] This is a block diagram showing an example configuration of a remotely controlled vehicle. [Figure 3] This is a block diagram showing an example of a worker terminal configuration. [Figure 4] This is a block diagram showing an example configuration of a voice call control device. [Figure 5] This is a schematic diagram illustrating the work site. [Figure 6] This is a flowchart showing the operating procedure of the voice call control device. [Figure 7] This flowchart shows the procedure for identifying the location of a target object from video footage. [Figure 8] This is a block diagram showing an example configuration of a voice call control device used in a communication system according to a second embodiment of the present invention. [Modes for carrying out the invention]

[0011] Embodiments of this disclosure will be described below with reference to the drawings. Note that the following description and drawings have been omitted and simplified as appropriate for clarity of explanation. Furthermore, in the following drawings, the same elements and similar elements are denoted by the same reference numerals, and redundant explanations have been omitted where necessary.

[0012] Figure 1 is a block diagram illustrating a schematic configuration example of a communication system. The communication system comprises a control console 11, a voice communication control device 20, a remotely controlled vehicle 40, and a worker terminal 60. In the communication system 10, the voice communication control device 20 can be connected to multiple control consoles 11, multiple remotely controlled vehicles 40, multiple worker terminals 60, and multiple transceivers 80. In this embodiment, the communication system 10 is applied to a system in which a remotely controlled vehicle 40 is remotely controlled in a remote space where multiple workers (users) and remotely controlled vehicles 40 are present.

[0013] The control console 11 is a device used by an operator to remotely monitor and remotely control the remotely controlled vehicle 40. The remotely controlled vehicle 40 is a remotely controlled device that is operated or controlled remotely. The remotely controlled vehicle 40 is, for example, a remotely controlled vehicle such as a forklift. The remotely controlled vehicle 40 may also be a small construction machine. The worker terminal 60 is a communication terminal device carried by a worker at the work site. The worker may include a worker who is riding in a vehicle that is not remotely controlled at the work site. In this embodiment, not all workers present at the work site have a worker terminal 60, and some workers do not have a worker terminal 60.

[0014] The remotely controlled vehicle 40 has a remotely controlled vehicle terminal. The remotely controlled vehicle terminal and the worker terminal 60 of the remotely controlled vehicle 40 have wireless communication units. Multiple transceivers 80 are arranged at the work site, and the remotely controlled vehicle terminal and the worker terminal 60 of the remotely controlled vehicle 40 are wirelessly connected to the transceivers 80 at the work site via their respective wireless communication units. The voice communication control device 20 is configured to enable wireless communication with the remotely controlled vehicle terminal and the worker terminal 60 of the remotely controlled vehicle 40 using wireless communication. Wireless communication includes, for example, communication for calls using professional wireless communication.

[0015] FIG. 2 is a block diagram showing a configuration example of the remotely operated vehicle 40. The remotely operated vehicle 40 includes a remotely operated vehicle terminal 50. The remotely operated vehicle terminal 50 has an audio output control unit 41, an audio acquisition unit 42, a video acquisition unit 43, a sensor 44, a wireless communication unit 45, a speaker 46, a microphone 47, and a camera 48. The remotely operated vehicle terminal 50 can be configured as a device having one or more memories and one or more processors. At least a part of each function of the remotely operated vehicle terminal 50 can be realized by one or more processors executing processing according to a program read from one or more memories. The remotely operated vehicle terminal 50 is also called a wireless terminal.

[0016] The wireless communication unit 45 includes a wireless module that performs short-range wireless communication such as Bluetooth (registered trademark). The wireless communication unit 45 includes an antenna and a transceiver. The speaker 46 and the microphone 47 are speakers and microphones arranged outside the vehicle. The audio output control unit 41 outputs audio from the speaker 46 toward the outside of the vehicle. The audio acquisition unit 42 uses the microphone 47 to acquire the voices of workers present around the remotely operated vehicle 40. The camera 48 is an imaging device that captures images of the surroundings of the vehicle. Further, the camera 48 is a ToF (Time of Flight) type distance image sensor (range sensor) that measures distance information for each pixel and can acquire a high-precision distance image. Specifically, the camera 48 is a camera in which the video sensor and the range sensor are arranged on the same optical axis via a spectroscopic prism. The speaker 46, the microphone 47, and the camera 48 may be installed at four locations, such as the front, rear, left, and right of the vehicle, so as to cover the entire circumference of the vehicle.

[0017] The sensor 44 is a vehicle sensor. The sensor 44 includes, for example, a motion sensor. The wireless communication unit 45 transmits and receives audio data to and from the audio call control device 20. Further, the wireless communication unit 45 transmits the video data captured using the camera 48 and the sensor data acquired using the sensor 44 to the audio call control device 20.

[0018] FIG. 3 is a block diagram showing a configuration example of the worker terminal 60. The worker terminal 60 includes an audio output control unit 61, an audio acquisition unit 62, a wireless communication unit 63, earphones 64, and a microphone 65. The worker terminal 60 is configured as, for example, a headset worn by the worker on the head. The worker terminal 60 may be mounted on a vehicle that is not remotely controlled. The worker terminal 60 can be configured as a device having one or more memories and one or more processors. At least a part of each function of the worker terminal 60 can be realized by one or more processors executing processing according to a program read from one or more memories. The worker terminal 60 is also called a wireless terminal.

[0019] The wireless communication unit 63 includes a wireless module that performs short-range wireless communication such as Bluetooth. The wireless communication unit 63 includes an antenna and a transceiver. The earphones 64 and the microphone 65 are earphones that output audio to the worker and a microphone that acquires the worker's audio. The audio output control unit 61 outputs audio from the earphones 64. The audio acquisition unit 62 acquires the worker's audio using the microphone 65. The wireless communication unit 63 transmits and receives audio data to and from the audio call control device 20.

[0020] FIG. 4 is a block diagram showing a configuration example of the audio call control device 20. The audio call control device 20 includes a video output control unit 21, an audio output control unit 22, an audio acquisition unit 23, a terminal position management unit 24, a communication control unit 25, a specifying unit 26, and a determination unit 27. The audio call control device 20 can be configured as a device having one or more memories and one or more processors. At least a part of each function of the audio call control device 20 can be realized by one or more processors executing processing according to a program read from one or more memories.

[0021] The program described above includes, when loaded into a computer, a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored in a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include RAM (Random Access Memory), ROM (Read Only Memory), flash memory, SSD (Solid State Drive), or other memory technologies, CD (Compact Disc), DVD (Digital Versatile Disc), Blu-ray® disc, or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include, a temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically, or otherwise propagating signals.

[0022] The communication control unit 25 controls communication between the voice communication control device 20 and the remotely controlled vehicle terminal 50 and the worker terminal 60. When the communication control unit 25 receives video from the remotely controlled vehicle terminal 50, it may also receive distance information for each pixel, an identification number that identifies the remotely controlled vehicle 40, and the mounting position of the camera 48 that captured the video. The video output control unit 21 displays the multiple video images received by the communication control unit 25 from the remotely controlled vehicle terminal 50 on multiple monitors 12. When the video output control unit 21 outputs the video received from the remotely controlled vehicle terminal 50 to the monitors 12, it may also output the identification number that identifies the remotely controlled vehicle 40, distance information for each pixel, and the mounting position of the camera 48 that captured the video. The video output control unit 21 may also display on the monitors 12 video images taken at the work site that are different from the video images captured by the remotely controlled vehicle terminal 50. The video output control unit 21 corresponds to the display unit. For example, the operator remotely controls the remotely controlled vehicle 40 while checking the work site conditions on multiple monitors 12 that display images of the area surrounding the remotely controlled vehicle 40 equipped with a remotely controlled vehicle terminal 50.

[0023] The video output control unit 21 may switch to display on one monitor 12 a video selected by the operator from among multiple videos. The captured video may be transmitted by means other than the wireless communication for conversation, depending on the required data rate. Control data for the remotely controlled vehicle 40 may also be transmitted by means other than the wireless communication for conversation. The audio output control unit 22 outputs audio from the speaker 13. The audio acquisition unit 23 acquires the operator's voice using the microphone 14. The monitor 12, speaker 13, and microphone 14 are arranged, for example, on the control console 11.

[0024] The terminal location management unit 24 manages the location of the remotely controlled vehicle 40 and the location of the worker terminal 60 at the work site. In this embodiment, the work site is assumed to be a closed space such as an indoor warehouse, where satellite positioning is unavailable. The terminal location management unit 24 manages the location of the remotely controlled vehicle 40 based on the wireless communication status between a plurality of transceivers 80 installed at the work site and the wireless communication unit 45 of the remotely controlled vehicle terminal 50. The terminal location management unit 24 also manages the location of the worker terminal 60 based on the wireless communication status between a plurality of transceivers 80 installed at the work site and the wireless communication unit 63 of the worker terminal 60. The terminal location management unit 24 may also detect or correct the location of the remotely controlled vehicle 40 using video acquired from the remotely controlled vehicle terminal 50 and sensor data acquired from sensors such as motion sensors.

[0025] The identification unit 26 identifies the location of a target object included in the video footage acquired from the remotely controlled vehicle terminal 50, which is remotely controlled by an operator, and is displayed on the monitor 12. Specifically, the identification unit 26 calculates the shooting position and shooting direction of the video based on the position and direction of the remotely controlled vehicle 40 acquired from the terminal position management unit 24 and the mounting position of the camera that captured the video acquired from the video output control unit 21. Using the distance information for each pixel of the video and the position information for each pixel of the video calculated based on the shooting position and shooting direction of the video, the identification unit 26 identifies the location of the target object detected by image recognition of the video. Image recognition of the video may be performed by known methods such as a convolutional neural network (CNN). The target object includes at least one other vehicle different from the remotely controlled vehicle 40 from which the video was acquired, and an operator. The location of the target object identified by the identification unit 26 may also be identified from position information associated with the pixel corresponding to the position of the centroid obtained by the first moment of area of ​​the moment feature for the range of pixels of the detected target object. Furthermore, the identification unit 26 only needs to identify the position of the target object that is within the field of view of the video acquired from the remotely controlled vehicle terminal 50.

[0026] Figure 5 is a schematic diagram illustrating a work site. Figure 5 will be used to explain how to acquire location information for wireless terminals. Multiple transceivers 80 are positioned at the work site. The terminal location management unit 24 stores an indoor map of the work site (the area where workers and remotely controlled vehicles 40 perform their work), and each point in the work site is mapped to local coordinates. If multiple work sites are to be handled, each point in the work site may be mapped to global coordinates. The indoor map includes the locations of the transceivers 80 and feature points such as pillars 81. Mapping refers to acquiring location information and storing it as coordinate locations on the indoor map.

[0027] The terminal location management unit 24 measures the distance between the transceiver 80 and each wireless terminal based on the signal strength received by the transceiver 80 and the remotely controlled vehicle 40, which includes wireless terminals, and the worker terminals 60. Based on the measured distance, the terminal location management unit 24 detects the location of each wireless terminal within the work site, i.e., the location information of the remotely controlled vehicle 40 and the location information of the worker terminals 60, and maps them as needed. The terminal location management unit 24 also transmits the location and orientation of the designated remotely controlled vehicle 40 and worker terminals 60 to the identification unit 26 in response to a request from the identification unit 26.

[0028] The above example described the use of signal strength-based position detection for wireless terminal location. However, position detection is not limited to this. Existing indoor positioning technologies, such as phase-based angle measurement techniques like AoA (Angle of Arrival) or AoD (Angle of Departure), can be used for position detection.

[0029] The terminal location management unit 24 detects the location of the wireless terminal and maps the locations of the remotely controlled vehicle 40 and the worker terminal 60 onto an indoor map. The terminal location management unit 24 also calculates the location and orientation of the remotely controlled vehicle 40 based on data acquired from the motion sensor mounted on the remotely controlled vehicle terminal 50, and maps the location and orientation of the remotely controlled vehicle 40 onto the indoor map. The motion sensor is a 9-axis sensor combining typical 3-axis acceleration, geomagnetic, and angular velocity sensors. Since sensors are affected by magnetism other than the Earth's magnetic field indoors, the location and orientation are estimated without using a geomagnetic sensor in the following description.

[0030] When estimating the position in a horizontal area assumed to be a work site, the terminal position management unit 24 uses only acceleration sensor values ​​to calculate the amount of movement from a composite vector of the horizontal components X and Y axes in the system's coordinate system. If it is necessary to consider elevation differences at the work site, the terminal position management unit 24 estimates the gravity vector using acceleration sensor values ​​and constructs a two-dimensional vector of vertical and horizontal components in the system's coordinate system. In addition to position estimation using acceleration sensors, the terminal position management unit 24 determines the orientation of the remotely controlled vehicle 40 based on the rotation angle obtained from the angular velocity sensor.

[0031] For example, at a work site, the remotely controlled vehicle 40 is parked in a predetermined parking position 82, facing a predetermined direction, at the start of work. After the start of work, the terminal position management unit 24 calculates the amount of movement and direction based on sensor data acquired from the motion sensor, and detects the position of the remotely controlled vehicle 40 after the start of work, starting from the parking position 82. The terminal position management unit 24 may also map-match the position of the remotely controlled vehicle 40 using feature points such as markings on the floor or walls, or the position of pillars 81, which are included in the video footage acquired from the remotely controlled vehicle 40 before it starts moving.

[0032] After the remotely controlled vehicle 40 moves, the position calculated by the motion sensor accumulates an error. The terminal position management unit 24 performs map matching by reflecting the position calculated by the motion sensor to the position of the remotely controlled vehicle 40 detected as the position of the wireless terminal. For map matching, a probabilistic map matching method can be employed, which matches positions based on the distribution of their positioning accuracy, assuming that the observed values ​​from sensors, etc., have positioning errors.

[0033] In estimating the position and orientation of the remotely controlled vehicle 40, as an alternative to the above, an encoder for measuring the rotational speed of the wheels may be placed on the remotely controlled vehicle terminal 50 of the remotely controlled vehicle 40, and the terminal position management unit 24 may calculate the distance traveled from the output of the encoder for each wheel. If the remotely controlled vehicle 40 turns due to the difference in the rotational speed of the left and right wheels, the terminal position management unit 24 may calculate the turning angle from that difference. If the remotely controlled vehicle 40 turns using the steering wheel, the terminal position management unit 24 may calculate the turning angle from the rotation state of the steering wheel. From these results, the distance traveled and the turning angle at each moment during travel can be obtained. The terminal position management unit 24 can estimate the distance traveled and the angle, i.e., the current position and orientation, by accumulating and adding these values ​​to the position and direction at the starting point. Regarding orientation, the remotely controlled vehicle 40 may be equipped with multiple wireless terminals, and the orientation of the vehicle may be estimated from the positional relationship of the multiple wireless terminals.

[0034] Next, Figure 5 will be used to explain how to detect the position of a worker included in the video. For example, suppose that four monitors 12 display video from the front, back, left, and right directions of the remotely controlled vehicle 40. The dashed line in Figure 5 indicates the shooting area 83 that captures the right side of the remotely controlled vehicle 40. The operator selects and specifies the video displaying the shooting area 83 on the right side of the vehicle, for example, using a keyboard (not shown). The identification unit 26 performs object detection on the video of the shooting area 83 on the right side of the vehicle specified by the operator, for example, and detects the entire body of a worker or a part of the worker, such as the head. If the entire body or part of a worker is detected from the video, the identification unit 26 identifies the position of the detected worker based on the position information associated with the pixels of the detected whole body or part of the worker. The identification unit 26 then identifies the worker terminal 60 associated with the worker included in the video of the shooting area 83 on the right side of the vehicle. The identification unit 26 pre-calculates position information for each pixel so that the position of the target object can be obtained from the position of the pixels in the video, and stores this information associated with each pixel.

[0035] The above describes the case where a worker is detected from the video, but if other vehicles are detected from the video, their positions can also be detected from the video in the same manner. If the remotely controlled vehicle 40 is equipped with a ranging device such as LiDAR (Light Detection And Ranging) or radar, the identification unit 26 may measure the distance and direction from the remotely controlled vehicle 40 to the worker or other vehicles using the ranging device and reflect this in the position information for each pixel of the video.

[0036] The determination unit 27 determines whether a wireless terminal or communication device is used for communication between the target object identified by the identification unit 26 and the operator. The determination unit 27 uses the location of the target object identified by the identification unit 26, the location of the remotely operated vehicle 40 managed by the terminal location management unit 24, and the location of the worker terminal 60 to determine whether a wireless terminal is associated with the target object, and determines whether a wireless terminal or communication device is used for communication between the target object and the operator depending on whether a wireless terminal is associated with the target object. For example, if the distance between the location of the target object identified by the identification unit 26 and the locations of the remotely operated vehicle 40 and the worker terminal 60 managed by the terminal location management unit 24 is less than a first distance, the determination unit 27 determines that a wireless terminal is associated with the target object.

[0037] The determination unit 27 determines, if a wireless terminal is associated with the target object identified by the identification unit 26, that the wireless terminal associated with the target object is the communication device used for communication with the target object. For example, if no wireless terminal is associated with the target object identified by the identification unit 26, the determination unit 27 determines that among the wireless terminals not associated with the target object, those located at a distance greater than or equal to a first distance but less than a second distance from the target object are the communication devices used for communication with the target object. The wireless terminal associated with the target object is also called the first communication device. The wireless terminal not associated with the target object is also called the second communication device. The communication control unit 25 establishes communication between the voice call control device 20 and the first communication device or the second communication device according to the determination of the determination unit 27. The first distance is, for example, 1m, and the second distance is, for example, 5m. If multiple communication devices that can be used for communication with the target object are found based on the distance conditions, the communication device closest to the target object is determined to be the communication device used for communication with the target object.

[0038] For example, operator A remotely controls vehicle A at a remote work site while viewing the screen of monitor 12 which displays images of the area around the vehicle. In the voice communication control device 20, the terminal location management unit 24 manages the location of wireless terminals at the work site, i.e., the location of all remotely controlled vehicles 40, including vehicle A and other remotely controlled vehicles, and the location of the worker terminal 60. The identification unit 26 detects other vehicle B or worker B within the field of view of the image that operator A is focusing on, from among the images in the four directions (front, back, left, and right). The identification unit 26 also uses the location of the wireless terminal managed by the terminal location management unit 24 to determine whether the detected other vehicle B or worker B is associated with the wireless terminal.

[0039] If another vehicle B or worker B is associated with a wireless terminal, the determination unit 27 determines, for example, that the worker terminal 60 held by the occupant of the other vehicle B or worker B is the wireless terminal to communicate with. In that case, the communication control unit 25 establishes a connection between the voice call control device 20 and the worker terminal 60 held by the occupant of the other vehicle B or worker B. In this way, the operator can start a call with the occupant of the other vehicle B or worker B. After the call starts, operator A can converse with the occupant of vehicle B or worker B using the speaker 13 and microphone 14 of the control console 11. The occupant of vehicle B or worker B can converse with operator A using the earphone 64 and microphone 65 of the worker terminal 60.

[0040] If no other vehicle B or worker B is associated with a wireless terminal, the determination unit 27 determines, for example, that the remotely controlled vehicle terminal 50 of the remotely controlled vehicle A, which is remotely controlled by operator A, is the wireless terminal to communicate with. In that case, the communication control unit 25 establishes a connection between the voice call control device 20 and the remotely controlled vehicle terminal 50 of the remotely controlled vehicle A. After the call starts, operator A can converse with the occupant or worker B of vehicle B using the speaker 13 and microphone 14 of the control console 11. The occupant or worker B of vehicle B can converse with operator A using the speaker 46 and microphone 47 of the remotely controlled vehicle A. If the remotely controlled vehicle A has multiple speakers and microphones, speakers and microphones positioned in the direction where vehicle B or worker B is located may be selected.

[0041] If no other vehicle B or worker B is associated with a wireless terminal, the determination unit 27 may determine that the remotely controlled vehicle terminal 50 of a remotely controlled vehicle C, which is different from the remotely controlled vehicle A, is the wireless terminal to communicate with. The remotely controlled vehicle terminal 50 of remotely controlled vehicle A is also called the first remotely controlled device. The remotely controlled vehicle terminal 50 of remotely controlled vehicle C is also called the second remotely controlled device. For example, if remotely controlled vehicle C is located closer to vehicle B or worker B than remotely controlled vehicle A, the determination unit 27 may determine that the remotely controlled vehicle terminal 50 of remotely controlled vehicle C is the wireless terminal to communicate with. Furthermore, if a fixed wireless terminal is located at the work site, the determination unit 27 may determine that the fixed wireless terminal located at a distance greater than or equal to a first distance but less than a second distance from worker B is the wireless terminal to communicate with. In that case, the communication control unit 25 may establish a connection between the voice communication control device 20 and at least one of the fixed wireless terminal and the remotely controlled vehicle terminal 50 of remotely controlled vehicle C.

[0042] If the operator wishes to speak with multiple people rather than a specific worker, the determination unit 27 may determine one or more remotely controlled vehicle terminals 50 and a fixed wireless terminal as the communication devices to communicate with. For example, an icon such as "Talk to everyone" may be displayed on the monitor 12. If the operator selects "Talk to everyone," the communication control unit 25 may establish a connection between the voice call control device 20 and the remotely controlled vehicle terminals 50 and the fixed wireless terminal.

[0043] Next, the operating procedure will be explained. Figure 6 is a flowchart showing the operating procedure of the voice communication control device 20. In Figure 6, the voice communication control device 20 receives video from the remotely controlled vehicle 40. The identification unit 26 acquires the video specified by the operator from among the multiple videos displayed on the monitor 12 from the video output control unit 21 (step S1). Next, the identification unit 26 identifies the position of the target object from the video specified by the operator (step S2).

[0044] Here, the specific processing details of step S2 in Figure 6 will be explained using Figure 7. Figure 7 is a flowchart showing the procedure for identifying the position of a target object from video. In Figure 7, the identification unit 26 obtains the shooting position and shooting direction of the video specified by the operator from the video output control unit 21 (step S21). Specifically, the identification unit 26 obtains the position and direction of the remotely controlled vehicle 40 from the terminal position management unit 24, obtains the mounting position of the camera that captured the video and distance information for each pixel of the video from the video output control unit 21, and calculates the shooting position and shooting direction of the video based on the position and direction of the remotely controlled vehicle 40 and the mounting position of the camera.

[0045] Next, the identification unit 26 obtains distance information for each pixel of the video specified by the operator from the video output control unit 21 (step S22). Next, the identification unit 26 associates position information with each pixel of the video specified by the operator (step S23). Specifically, the identification unit 26 calculates position information for each pixel of the video based on the distance information for each pixel of the video, the shooting position of the video, and the shooting direction, and associates the position information with each pixel of the video.

[0046] Next, the identification unit 26 performs image recognition on the video specified by the operator (step S24). Specifically, the identification unit 26 performs image recognition on the video specified by the operator and detects workers or other vehicles present in the video as target objects. Next, the identification unit 26 identifies the location information of the workers or other vehicles detected as target objects (step S25). Specifically, the identification unit 26 identifies the location of the target object based on the location information associated with the pixels of the detected target object.

[0047] Let's return to the explanation of Figure 6. In Figure 6, the determination unit 27 obtains location information of the wireless terminal at the work site from the terminal location management unit 24 (step S3).

[0048] The determination unit 27 uses the location of the target object identified by the identification unit 26 in step S2 and the location information of the wireless terminal acquired in step S3 to determine whether or not there is a wireless terminal associated with the target object (step S4). Specifically, the determination unit 27 determines that there is a wireless terminal associated with the target object detected in step S2 if the worker terminal 60 is located less than a first distance from the location of the target object.

[0049] If the determination unit 27 determines that there is a wireless terminal associated with the target object detected in step S2 (step S4: YES), it connects the voice call control device 20 and the first communication device through the communication control unit 25 (step S5). The determination unit 27 connects, for example, the voice call control device 20 to a worker terminal 60 located less than a first distance from the location of the target object. In this case, the voice call control device 20 establishes communication for voice communication with the worker terminal 60.

[0050] If the determination unit 27 determines that there is no wireless terminal associated with the target object detected in step S3 (step S4: NO), it connects the voice call control device 20 and the second communication device via the communication control unit 25 (step S6). For example, the determination unit 27 connects the voice call control device 20 to the remotely controlled vehicle terminal 50 of the remotely controlled vehicle 40, which is the source of the video, located at a distance greater than or equal to a first distance but less than a second distance from the location of the target object. In this case, the voice call control device 20 establishes communication for voice communication with the remotely controlled vehicle terminal 50 of the remotely controlled vehicle 40.

[0051] The communication control unit 25 initiates communication between the operator and the first or second communication device with which a connection has been established (step S7). When the operator starts a conversation, the voice output control unit 22 of the voice call control device 20 outputs sound to the operator from the speaker 13. The voice acquisition unit 23 also acquires the operator's voice from the microphone 14. If the voice call control device 20 is connected to the worker terminal 60 in step S5, the voice output control unit 61 of the worker terminal 60 outputs sound to the earphone 64. The voice acquisition unit 62 also acquires the worker's voice from the microphone 65. If the voice call control device 20 is connected to the remotely controlled vehicle 40 in step S6, the voice output control unit 41 of the remotely controlled vehicle 40 outputs sound from the speaker 46. The voice acquisition unit 42 also acquires voice from the microphone 47.

[0052] The determination unit 27 may determine whether all workers possess worker terminals 60 when the identification unit 26 has identified the locations of multiple workers around the remotely controlled vehicle 40. For example, the identification unit 26 performs image recognition on the video to detect the locations and number of workers. The determination unit 27 checks the number of worker terminals 60 located within a first distance from the worker locations detected by the identification unit 26. The determination unit 27 compares the number of detected workers with the number of worker terminals 60 located within a first distance from the detected worker locations. If the number of worker terminals 60 located within a first distance from the detected worker locations is less than the number of detected workers, the identification unit 26 determines that some workers do not possess worker terminals 60. In that case, the determination unit 27 may determine the remotely controlled vehicle terminal 50 of the remotely controlled vehicle 40, which is located at a distance greater than or equal to a first distance but less than a second distance from the workers, as the communication destination. Multiple workers can communicate with the operator using the speaker 46 and microphone 47 of the remotely controlled vehicle terminal 50 of the remotely controlled vehicle 40.

[0053] In this embodiment, the terminal location management unit 24 of the voice call control device 20 manages the location of the worker terminal 60 used for conversations held by the worker, and the location of the remotely controlled vehicle 40. When an operator converses with a worker included in the video displayed on the monitor 12, the determination unit 27 determines whether or not there is a wireless terminal associated with that worker, such as the worker terminal 60. The determination unit 27 switches the communication device to which it is connected depending on whether or not there is a wireless terminal associated with the worker.

[0054] When communication is necessary between an operator and a field worker, noise at the site can drown out the voice, making communication difficult. In such cases, using a wireless terminal equipped with a headset is useful. However, not all workers possess wireless terminals. The operator communicates with a target object in the video while viewing video footage of the remote space captured by a camera. If it is necessary to switch communication devices depending on whether a wireless terminal is associated with the target object in the video, the operation of switching communications can be cumbersome, and it may not be possible to communicate at the desired time.

[0055] In this embodiment, depending on whether the worker is carrying a wireless terminal, the communication method is switched between, for example, using the earphone 64 and microphone 65 of the worker terminal 60, or using the speaker 46 and microphone 47 provided on the remotely controlled vehicle 40. This allows operators to easily initiate conversations with on-site workers in spaces where remotely controlled vehicles such as forklifts and workers are present, such as logistics sites, without having to be aware of whether the person they are talking to is carrying a wireless terminal.

[0056] Next, a second embodiment will be described. Figure 8 is a block diagram showing an example of the configuration of a voice call control device used in a communication system according to the second embodiment of the present invention. In this embodiment, the voice call control device 20a has a gaze detection unit 28 in addition to the configuration of the voice call control device 20 shown in Figure 4.

[0057] The gaze detection unit 28 detects the operator's gaze toward the image displayed on the monitor 12. The operator, for example, looks at the monitor 12 displaying the person they wish to converse with and begins the conversation. Alternatively, the operator directs their gaze toward the area of ​​the image displayed on the monitor 12 where the person they wish to converse with is located and begins the conversation. In this embodiment, the identification unit 26 identifies the location of the target object, i.e., the worker or another vehicle, in the area of ​​the image located beyond the gaze detected by the gaze detection unit 28.

[0058] The determination unit 27 determines whether the worker or other vehicle that the user is looking at is an object that has a wireless terminal or an object that does not have a wireless terminal. The operation of switching the conversation route depending on the presence or absence of a wireless terminal may be the same as the operation described in the first embodiment.

[0059] In this embodiment, the gaze detection unit 28 may detect the gaze of a worker at the work site. For example, the gaze detection unit 28 uses video received from the remotely controlled vehicle 40 to detect whether or not the worker is looking at the remotely controlled vehicle 40. The gaze detection unit 28 determines that the worker is looking at the remotely controlled vehicle 40 if, for example, the worker's eyes or face are directed toward the remotely controlled vehicle 40. The gaze detection unit 28 determines that the worker is not looking at the remotely controlled vehicle 40 if the worker's eyes or face are not directed toward the remotely controlled vehicle 40.

[0060] The determination unit 27 may determine, if the worker is looking at the remotely controlled vehicle 40 and the worker has a wireless terminal, that the wireless terminal held by the worker looking at the remotely controlled vehicle 40 is the communication device to communicate with. In that case, if the worker has a worker terminal 60, the communication control unit 25 may connect the voice call control device 20a to the worker terminal 60. If the worker does not have a worker terminal 60, the communication control unit 25 may connect the voice call control device 20a to the remotely controlled vehicle terminal 50 of the remotely controlled vehicle 40 located at a distance greater than or equal to a first distance but less than a second distance from the worker.

[0061] In this embodiment, the gaze detection unit 28 does not necessarily need to be included in the voice communication control device 20a. Gaze detection may be performed in the worker terminal 60. For example, the worker terminal 60 is configured as an eyeglass-type device. The worker terminal 60 may have a built-in camera and use the image from the built-in camera to detect the worker's gaze. In addition to conversations with the operator, the worker terminal 60 may be configured to enable conversations with other worker terminals 60. For example, the worker terminal 60 may be configured to enable wireless conversations between workers who can see each other at the work site but are at a distance from each other.

[0062] In this embodiment, the gaze detection unit 28 detects the operator's gaze. The identification unit 26 identifies the object located in the operator's line of sight. In this embodiment, by combining gaze detection, the operator can easily specify the person they wish to speak with to the voice call control device 20a.

[0063] In the above embodiment, the voice communication control device 20 does not necessarily have to be configured as a single device. The voice communication control device 20 may be configured as, for example, multiple physically separated devices. In that case, some of the functions of the voice communication control device 20 may be located at or near the work site. Other parts of the functions of the voice communication control device 20 may be located at a remote location or in the cloud where remote operation is performed. For example, the process of detecting an object from captured video and determining the coordinate position of the detected object may be performed by a device located at the work site. Also, the location management of wireless terminals may be performed by a device located at the work site.

[0064] Although embodiments of the present invention have been described in detail above, the present invention is not limited to the embodiments described above, and modifications and changes to the above embodiments without departing from the spirit of the present invention are also included in the present invention. [Explanation of symbols]

[0065] 10: Communication Systems 11:Operation console 12: Monitor 13: Speaker 14: Mike 20: Voice call control device 21: Video Output Control Unit 22: Audio Output Control Unit 23: Voice acquisition unit 24: Terminal Location Management Department 25: Communication Control Unit 26: Specific part 27: Judgment section 28: Eye-tracking unit 40: Remotely controlled vehicles 41: Audio Output Control Unit 42: Voice acquisition unit 43: Video Acquisition Unit 44: Sensor 45: Wireless Communication Department 46: Speaker 47: Mike 48: Camera 50: Remotely controlled vehicle terminal 60: Worker terminal 61: Audio Output Control Unit 62: Voice acquisition unit 63: Wireless Communication Department 64: Earphones 65: Mike 80: Transmitter / Receiver 81: Pillar 82: Parking location 83: Shooting area

Claims

1. It is a communication system, A display unit that shows video footage captured by a camera in a remote space, The display unit includes an identification unit that identifies the target object displayed on the display unit, A determination unit that determines the communication device used for communication with the target object identified by the identification unit, Equipped with, A communication system in which the determination unit determines, if the first communication device is associated with the target object identified by the identification unit, the first communication device associated with the target object is used as a communication device for communication with the target object, and if the first communication device is not associated with the target object identified by the identification unit, the second communication device not associated with the target object is used as a communication device for communication with the target object.

2. The communication system according to claim 1, further comprising a second communication device equipped with a camera and associated with a first remotely controlled device remotely operated by an operator.

3. The communication system according to claim 2, wherein the second communication device includes at least one second remotely controlled device separate from the first remotely controlled device and a communication device located within a predetermined distance from the target object.

4. The system further includes a gaze detection unit that detects the operator's gaze toward the display unit, The communication system according to claim 1, wherein the identifying unit identifies a target object based on the detected line of sight of the operator.

5. The video footage captured by the camera of a remote space is displayed on the display unit. Identify the target object displayed on the aforementioned display unit, A communication method that, if a first communication device is associated with the identified target object, determines that the first communication device associated with the target object is the communication device used for communication with the target object, and if the first communication device is not associated with the identified target object, determines that a second communication device not associated with the target object is the communication device used for communication with the target object.

6. The video footage captured by the camera of a remote space is displayed on the display unit. Identify the target object displayed on the aforementioned display unit, A program that causes a computer to perform the following processes: if a first communication device is associated with the identified target object, it determines that the first communication device associated with the target object is the communication device used for communication with the target object; if the first communication device is not associated with the identified target object, it determines that a second communication device not associated with the target object is the communication device used for communication with the target object.

Images