System, Information Processing Apparatus, and Program

Abstract

To improve the operability of a working machine by showing a video of the working machine photographed from the outside of the working machine to a user who remotely operates the working machine.SOLUTION: An accepting unit 211 accepts a user's operation from an operator 31 of an input / output device 3. Further, the reception unit 211 receives data on each attitude of a display device 32 and a support 33 with respect to the reference coordinate system from an identification unit 36 of the input / output device 3. A selection unit 212 selects a selection image to be combined with the video captured by a first camera 11 according to the content of the received user's operation. A generation unit 213 generates a composite video by combining the selected image with the video captured by the selected first camera 11 so as to be arranged around the user. An instruction unit 214 transmits the composite image generated by the generation unit 213 to the display device 32 and instructs to display it.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to the technology of a system for remotely operating a working machine.

Background Art

[0002] A system has been developed that transfers and displays in real time video captured from a moving body such as a vehicle to a remote location, giving the user the feeling of being aboard the moving body.

[0003] Patent Document 1 describes an image processing system that transmits an image corresponding to the operation of a vehicle to give the user a pseudo-experience of traveling with the driver.

[0004] Patent Document 2 describes a VR functional unit that analyzes a video of the real space captured by a camera when rendering a virtual space, extracts the area of the in-flight part from the video, and fits and synthesizes the video within the extracted area onto the top plate of a desk object in the virtual space.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] When remotely operating a working machine such as a crane, a bulldozer, or an excavator, the user can make various judgments as if they were in the operator's cab of the working machine by, for example, viewing the video taken from the operator's cab of the working machine or a position corresponding to the operator's cab. However, there is a limit to the range that can be photographed from the working machine, and the situation where the working machine is placed and the surrounding dangers may appear in the external appearance image taken of the working machine from the outside.

[0007] One of the objects of the present invention is to improve the operability of a working machine by showing the user operating the working machine a video taken of the working machine from the outside.

Means for Solving the Problem

[0008] The system according to claim 1 of the present invention includes a working machine, a first camera that photographs the surroundings from the working machine, an operator that receives the user's operation, and a processor that synthesizes the video taken by the first camera so that one or more selected images selected according to the user's operation from a plurality of images are arranged in the surroundings to generate a synthesized video, and a display device that is worn on the user's head and displays a video in a range corresponding to the posture of the head among the synthesized video. including an external appearance video obtained by photographing the working machine from around the working machine It is a system having. and, when the external appearance video indicates a danger of the working machine, the processor generates the composite video obtained by compositing the external appearance video It is a system. The system according to claim 2 of the present invention is A working machine, a first camera that photographs the surroundings from the working machine, a second camera different from the first camera, attached to the working machine An operator that receives the user's operation, and a processor that synthesizes the video taken by the first camera so that one or more selected images selected according to the user's operation from a plurality of images are arranged in the surroundings to generate a synthesized video, and a display device that is worn on the user's head and displays a video in a range corresponding to the posture of the head among the synthesized video. including a video obtained by photographing from the second camera A system having is. The system according to claim 3 of the present invention is A working machine, a first camera that photographs the surroundings from the working machine, a microphone that collects sounds around the working machine, a sound emitter that emits sounds corresponding to the posture of the user's head among the sounds collected by the microphone, and theAn operator that receives a user's operation, and a processor that generates a composite video by synthesizing the video captured by the first camera so that one or more selected images selected from a plurality of images according to the user's operation are arranged around the periphery thereof, and the front header A display device that is attached to the head and displays a video within a range corresponding to the posture of the head among the composite videos, and a system is. The system according to claim 4 of the present invention is a plurality of A work machine, and the a plurality of Work machine each of A first camera that captures the surroundings from the work machine, an operator that receives a user's operation, a processor that generates a composite video by synthesizing the video captured by the first camera so that one or more selected images selected from a plurality of images according to the user's operation are arranged around the periphery thereof, and a display device that is attached to the user's head and displays a video within a range corresponding to the posture of the head among the composite videos, and has and, when the operation of the user designates a selection image showing another working machine among the selection images and instructs a transfer to the other working machine, the processor generates a composite video in which one or more of the selection images are composited with a video photographed by a first camera that photographs the surroundings from the other working machine A system is. The system according to claim 5 of the present invention is A work machine, a first camera that captures the surroundings from the work machine, an operator that receives a user's operation, and in the video captured by the first camera, including an external appearance video obtained by photographing the working machine from around the working machine A processor that generates a composite video by synthesizing one or more selected images selected from a plurality of images according to the user's operation so that they are arranged around the periphery thereof, and a display device that is attached to the user's head and displays a video within a range corresponding to the posture of the head among the composite videos, and has and the processor generates the composite video by deforming the external appearance video A system is. The system according to claim 6 of the present invention is A work machine, a first camera that captures the surroundings from the work machine, an operator that receives a user's operation, a processor that generates a composite video by synthesizing the video captured by the first camera so that one or more selected images selected from a plurality of images according to the user's operation are arranged around the periphery thereof, and a display device that is attached to the user's head and displays a video within a range corresponding to the posture of the head among the composite videos, and has and the operator photographs the user's eyeballs, detects the user's line of sight, and accepts the operation by the line of sight A system is. The system according to claim 7 of the present invention isa work machine, a first camera that captures images of the surroundings from the work machine, an operator that accepts user operations, and a processor that generates a composite image by combining an image captured by the first camera with one or more selected images selected from a plurality of images in response to the user's operation, so that the selected images appear to be arranged in the surroundings; a measuring instrument that measures the acceleration received by the first camera, a support that supports the user's body, and a driving machine that drives the support according to the acceleration measured by the measuring instrument The device is worn on the user's head, and the synthesized image includes: with respect to the support A display that displays an image in a range corresponding to the head posture; having, wherein the posture of the head with respect to the support is specified using the difference between the postures of the display device with respect to the reference coordinate system in which the support and the driving machine are installed and the support with respect to the reference coordinate system System is.

[0009] Claims of the invention 8 The system according to claim 1 any one of claims 1 to 7 In the aspect described above, the processor is a system that changes at least one of the position, size, hue, saturation, and brightness of the selected image in the composite image in response to an operation by the user.

[0012] Claims of the invention 9 The system according to claim 1 In the aspect described above, the processor is configured to of a working machine The system generates the composite image in which, when a danger is indicated, the exterior image is emphasized compared to when the danger is not indicated.

[0016] Claims of the invention 10 The system according to claim 1 to up to 9 In any one of the aspects described above, the work machine is a system that operates in response to an operation by the user.

[0017] Claim 1 of the present invention 1 The system according to claim 1 to up to 5, 7 or 9 In any one of the above aspects, the operator is a system that accepts the operation by a user's line of sight.

[0018] Claim 1 of the present invention 2 The system according to claim 1 to up to 6, or 9In the aspect according to any one of the claims, a measuring device that measures the acceleration received by the first camera, a support that supports the user's body, and a driving mechanism that drives the support according to the acceleration measured by the measuring device, and the display device is a system that displays an image within a range corresponding to the posture of the head with respect to the support.

[0019] Claim 1 of the present invention 3 The system according to relates to claims 1 to 1 2 In the aspect according to any one of the claims, a positioning device that measures the position of the work machine, and the plurality of images include a map indicating the position measured by the positioning device.

[0020] Claim 1 of the present invention 4 The information processing apparatus according to has a processor, and the processor synthesizes a composite image in which one or more selected images selected according to the user's operation from a plurality of images are arranged around the image captured by the first camera that captures the surroundings from the work machine, including an external appearance video obtained by photographing the working machine from around the working machine and generates a composite image, and causes a display device worn on the user's head to display an image within a range corresponding to the posture of the head among the composite images. or deforming the external appearance video to generate the composite video It is an information processing apparatus that causes a display device worn on the user's head to display an image within a range corresponding to the posture of the head among the composite images.

[0021] Claim 1 of the present invention 5 The program according to causes a computer having a processor to generate a composite image in which one or more selected images selected according to the user's operation from a plurality of images are arranged around the image captured by the first camera that captures the surroundings from the work machine, including an external appearance video obtained by photographing the working machine from around the working machine and causes a display device worn on the user's head to display an image within a range corresponding to the posture of the head among the composite images. a function of deforming the external appearance video to generate the composite video It is a program for causing a computer having a processor to realize a function of generating a composite image in which one or more selected images selected according to the user's operation from a plurality of images are arranged around the image captured by the first camera that captures the surroundings from the work machine,

Brief Description of the Drawings

[0022] [Fig. 1] A diagram showing an example of the configuration of the system 9 according to an embodiment of the present invention. [Fig. 2]A diagram showing an example of the functional configuration of the information processing apparatus 2. [Fig. 3] A diagram showing an example of a composite video in which an appearance video is combined. [Fig. 4] A diagram showing an example of an appearance video including the appearance of the working machine 1.

Mode for Carrying Out the Invention

[0023] <Embodiment> <Configuration of the System> FIG. 1 is a diagram showing an example of the configuration of a system 9 according to an embodiment of the present invention. The system 9 is a simulator system that installs a first camera 11 on a working machine 1 and reproduces and provides to the user U the feeling of moving by the working machine 1.

[0024] The system 9 includes a working machine 1, an information processing apparatus 2, an input / output apparatus 3, and an external camera 4. The system 9 shown in FIG. 1 includes one each of the working machine 1, the information processing apparatus 2, the input / output apparatus 3, and the external camera 4, but the number of each of these may be two or more.

[0025] The working machine 1 is a working machine used for work such as civil engineering and construction, and is, for example, an excavator. This working machine 1 has, for example, a mechanism that photographs the scenery visible from the driver's seat or the like with a camera and sends the video to the information processing apparatus 2. The working machine 1 shown in FIG. 1 includes a first camera 11, a second camera 12, a communication unit 13, a microphone 14, a measuring instrument 15, a positioning device 16, and an operating unit 17.

[0026] The first camera 11 is a camera attached to, for example, the driver's seat of the working machine 1, and photographs the surroundings from the working machine 1. The first camera 11 shown in FIG. 1 has a plurality of sets of an optical system having a wide-angle lens and an imaging element that generates image data in response to the light condensed by the optical system, and images all directions around itself with these. This first camera 11 is, for example, what is called a so-called omnidirectional camera or the like.

[0027] Note that the first camera 11 is not limited to one, and two or more may be provided. For example, a plurality of first cameras 11 may be provided not only in the operator's cab of the working machine 1 but also outside the cab (such as the roof, chassis, arm, etc.).

[0028] The second camera 12 is a camera different from the first camera 11 attached to the working machine 1 together with the first camera 11. That is, the system 9 having the second camera 12 is an example of a system having a second camera different from the first camera attached to the working machine.

[0029] The second camera 12 shown in FIG. 1 is a camera that photographs the range that becomes a blind spot due to a frame, a seat, etc. from the operator's cab where the first camera 11 is installed. This second camera 12 is attached, for example, to the outer wall behind the operator's cab and faces the rear side in the traveling direction, and photographs the contact point between the moving device such as a wheel and the road.

[0030] Note that the number of the second cameras 12 is not limited to one, and it may not be provided, or two or more may be provided. For example, the second camera 12 may be provided not only behind the working machine 1 but also in front, below, left and right, at a plurality of locations that are blind spots from the first camera 11.

[0031] The microphone 14 is a microphone attached to the working machine 1 and collecting the sounds around the working machine 1. That is, the system 9 having the microphone 14 is an example of a system having a microphone that collects the sounds around the working machine.

[0032] The microphone 14 shown in FIG. 1 is installed, for example, in the operator's cab of the working machine 1 or on the top plate of the operator's cab. The microphone 14 may be built into the first camera 11 or the second camera 12.

[0033] The measuring instrument 15 is a measuring instrument that measures the acceleration received by the first camera 11. The measuring instrument 15 is, for example, a triaxial acceleration sensor based on piezoelectric, piezoresistive, capacitive, etc. This measuring instrument 15 may use MEMS (Micro Electro Mechanical Systems). This measuring instrument 15 generates measurement data indicating the measured acceleration and transmits it to the information processing device 2.

[0034] Note that the physical quantity measured by the measuring instrument 15 is not limited to acceleration. The measuring instrument 15 may measure the position. In this case, the information processing device 2 may, for example, obtain the acceleration received by the first camera 11 by second-order differentiating the position indicated by the acquired measurement data with respect to time. Also, the measuring instrument 15 may measure the speed of the first camera 11. In this case, the information processing device 2 may, for example, obtain the acceleration received by the first camera 11 by differentiating the speed indicated by the acquired measurement data with respect to time.

[0035] The positioning device 16 is a positioning device that measures the position of the machine tool 1. For example, it is a receiving device that extracts a navigation signal from radio waves transmitted by navigation satellites of the Global Navigation Satellite System (GNSS) and performs positioning calculations.

[0036] The communication unit 13 has a communication circuit that wirelessly exchanges control signals, video data, etc. with the information processing device 2. The communication unit 13 shown in FIG. 1 may, for example, have a communication line using LPWA (Low Power Wide Area). Also, if it has a mechanism for paying out or accommodating the communication line as the machine tool 1 moves and this communication line does not interfere with the movement of the machine tool 1, the communication unit 13 may connect the machine tool 1 and the information processing device 2 in a communicable manner by wire.

[0037] The communication unit 13 transmits video data indicating the video captured by the first camera 11 and the second camera 12, and measurement data measured by the measuring instrument 15, to the information processing device 2. Further, the communication unit 13 transmits an audio signal indicating the sound collected by the microphone 14, and position data indicating the position of the working machine 1 measured by the positioning device 16, to the information processing device 2.

[0038] The operation unit 17 is configured to control working machines such as the boom, arm, bucket, and shovel of the working machine 1, and moving devices such as wheels and endless tracks, to operate the working machine 1. Based on the control signal received via the communication unit 13, the processor controls the hydraulic output mechanism of the working machine, the gasoline engine, the power transmission mechanism, etc.

[0039] The information processing device 2 is, for example, a computer. The information processing device 2 shown in FIG. 1 includes a processor 21, a memory 22, and a communication unit 23.

[0040] The processor 21 controls each part of the information processing device 2 by reading and executing a computer program (hereinafter simply referred to as a program) stored in the memory 22. The processor 21 is, for example, a CPU (Central Processing Unit).

[0041] The communication unit 23 is a communication circuit that wirelessly connects the information processing device 2 to the working machine 1 so as to be communicable. As described above, the communication unit 23 may connect the working machine 1 and the information processing device 2 to be communicable by wire.

[0042] The memory 22 is a storage means for storing an operating system, various programs, data, etc. that are read by the processor 21. The memory 22 has a RAM (Random Access Memory) and a ROM (Read Only Memory). Note that the memory 22 may have a solid state drive, a hard disk drive, etc.

[0043] The input / output device 3 is installed in a reproduction room or the like, and is a device that outputs video, audio, etc. to the user and inputs operations from the user. The input / output device 3 shown in FIG. 1 has an operator 31, a display 32, a support 33, a sound player 34, a driver 35, and an identification unit 36. The operator 31, the display 32, the sound player 34, the driver 35, and the identification unit 36 are connected to the processor 21 of the information processing device 2 via, for example, a bus, and are controlled by the processor 21. Note that these components of the input / output device 3 may be communicably connected to the processor 21 via the communication unit 23 of the information processing device 2.

[0044] The operator 31 is composed of, for example, a touch panel, a handle, buttons, etc. that can be touched by the user's hand, finger, etc., and receives the user's operations. The operator 31 may be a device that is inside the display 32, captures the user's eyeballs, detects the user's line of sight, and receives operations based on this line of sight. In this case, the operator 31 is an example of an operator that receives the above operations based on the user's line of sight.

[0045] Further, the operator 31 may be a wearable device that is worn on the user's hand, finger, eyelid, tongue, lip, etc. and identifies gestures expressed by these movements. Also, the operator 31 may be a voice recognition device that collects the user's uttered voice and interprets and identifies the operation indicated by this voice through natural language analysis.

[0046] The display 32 is a display that is worn on the head Uh of the user U and displays the video captured by the first camera 11 described above. The display 32 is, for example, the above-described HMD. The display 32 may be a so-called contact optical system HMD that displays video on a liquid crystal display arranged in front of the user U. Also, the display 32 may be a retinal projection type HMD that projects laser light or the like directly onto the user's retina.

[0047] The support 33 is a structure that supports the body of the user U (i.e., the torso Ut and the head Uh), and for example, it is a chair, a seat, etc. shown in FIG. 1. Note that the support 33 is not limited to a chair, a seat, etc., and may be, for example, a band, a rope, a net, etc. that suspends the user U.

[0048] Note that what the support 33 directly contacts and supports is the torso Ut of the user U, and the head Uh is indirectly supported by the support 33 via the torso Ut. Since the head Uh of the user U is not restrained in the state of sitting on the support 33, the user can change the viewing direction by shaking the neck up, down, left, and right.

[0049] The drive mechanism 35 is a mechanism that drives the support 33 according to the acceleration received by the first camera 11, and for example, it is a six-degree-of-freedom actuator device such as a Stewart platform, or a one-degree-of-freedom or multi-degree-of-freedom actuator device, etc.

[0050] The specifying unit 36 specifies the postures of the display device 32 and the support 33 with respect to a reference coordinate system (hereinafter referred to as the reference coordinate system). Here, the reference coordinate system is an inertial coordinate system in a reproduction room or the like where the support 33 and the drive mechanism 35 are installed.

[0051] The specifying unit 36 specifies the postures of the display device 32 and the support 33, for example, by position tracking called the so-called camera method. In this case, light emitting elements such as LEDs (light emitting diodes) are provided on the display device 32 and the support 33 respectively. The specifying unit 36 captures light such as infrared rays emitted from the light emitting elements provided on the display device 32 and the support 33 with, for example, a plurality of cameras installed in the reproduction room, thereby specifying the positions of the light emitting elements, and specifying the postures of the display device 32 and the support 33 from those positions respectively.

[0052] Note that the specific method by the specific part 36 is not limited to the one described above. For example, one or more light-emitting elements fixed at one or more positions in the reproduction room irradiate light such as infrared light, and the irradiated light may be sensed by a plurality of cameras, optical sensors, etc. provided in the display 32 or the support 33. In this case, the specific part 36 is included in the display 32 or the support 33.

[0053] In addition, since the display 32 is attached to the head Uh of the user U, the user U can cause the specific part 36 to identify the change in the posture of the display 32 by moving the head Uh. Therefore, the specific part 36 may function as the operator 31 that receives the change in the posture of the display 32 as the gesture described above.

[0054] The sound player 34 is a sound player such as a speaker that emits sound under the control of the processor 21. When the processor 21 of the information processing apparatus 2 receives an audio signal indicating the sound collected by the microphone 14 via the communication unit 13 and the communication unit 23, among this sound, it emits the sound corresponding to the posture of the head of the user identified by the specific part 36. That is, the system having this sound player 34 is an example of a system having a sound player that emits the sound corresponding to the posture of the head of the user among the sounds collected by the microphone.

[0055] The communication unit 23 of the information processing apparatus 2 receives video data, audio signals, position data, and measurement data from the working machine 1. The drive unit 35 shown in FIG. 1 drives the support 33 according to the measurement data received by the communication unit 23 under the control of the processor 21 of the information processing apparatus 2.

[0056] The processor 21 of the information processing apparatus 2 corrects the video so that the range corresponding to the posture of the display 32 with respect to the support 33 is visible to the user U, and causes the display 32 to display the corrected video.

[0057] For example, the processor 21 shown in FIG. 1 receives data indicating the postures of the display 32 and the support 33 with respect to the reference coordinate system, respectively, specified by the specifying unit 36. Then, the processor 21 corrects the video captured by the first camera 11 using the difference between the posture of the display 32 specified by the specifying unit 36 and the posture of the support 33.

[0058] The display 32 displays the corrected video described above to the user. That is, the display 32 is an example of a display that displays a video within a range corresponding to the posture of the user's head with respect to the support.

[0059] The external camera 4 is a camera that captures the working machine 1 from around the working machine 1. The external camera 4 is connected to the information processing apparatus 2 by wire or wirelessly via an interface (not shown) of the information processing apparatus 2, for example. The external camera 4 supplies the processor 21 with a video (referred to as an appearance video) of the working machine 1 that it has captured. Note that the external camera 4 may be connected to the processor 21 via the communication unit 23.

[0060] <Functional Configuration of Information Processing Apparatus> FIG. 2 is a diagram showing an example of the functional configuration of the information processing apparatus 2. The processor 21 of the information processing apparatus 2 functions as a reception unit 211, a selection unit 212, a generation unit 213, and an instruction unit 214 by executing a program stored in the memory 22. In FIG. 2, both the communication unit 13 and the communication unit 23 are omitted.

[0061] The reception unit 211 receives a signal corresponding to the user's operation from the operator 31 of the input / output device 3 via the bus, and receives the operation by this user. In addition, the reception unit 211 receives data on the postures of the display 32 and the support 33 with respect to the reference coordinate system from the specifying unit 36 of the input / output device 3.

[0062] Furthermore, the reception unit 211 receives an audio signal from the microphone 14 installed in the working machine 1, measurement data indicating the measured acceleration from the measuring instrument 15, and position data indicating the measured position of the working machine 1 from the positioning device 16, respectively.

[0063] The selection unit 212 selects the first camera 11 according to the content of the user operation received by the reception unit 211. For example, the selection unit 212 selects the work machine 1 according to a gesture by the user's hand or line of sight, etc., and selects the first camera 11 installed in its driver's seat or the like. When a plurality of first cameras 11 are installed on one work machine 1, the selection unit 212 may select a predetermined (so-called default) first camera 11 among those plurality of first cameras 11 installed on the selected work machine 1, and thereby obtain an omnidirectional image captured thereby. Thereafter, when the user turns around and the line of sight moves to a range where only the back seat can be seen from the selected first camera 11, or the posture of the head of the turned-around user is detected, the selection unit 212 may select any one of the plurality of first cameras 11 provided on the work machine 1 according to the line of sight or the direction of the head.

[0064] Then, the selection unit 212 selects an image (hereinafter referred to as a selection image) to be synthesized with the image captured by the first camera 11 according to the content of the user operation received. Candidates for a plurality of sources (that is, output sources) of this selection image include the second camera 12, the external camera 4, and the memory 22. The selection unit 212 may select one selection image or may select a plurality of selection images.

[0065] Note that the selection unit 212 may select, as a selection image, an image of a map including the position indicated by this position data from, for example, a map database stored in the memory 22 according to the position data of the work machine 1 received by the reception unit 211 from the positioning machine 16. In this case, the processor 21 is an example of a processor that generates a composite video in which one or more selection images selected according to the user operation are arranged around the first camera 11 from a plurality of images including a map indicating the position of the work machine measured by the positioning machine.

[0066] The generation unit 213 generates a composite video by synthesizing the selected image provided by the above-described source onto the video captured by the first camera 11 selected by the selection unit 212 so as to be arranged around the user.

[0067] The instruction unit 214 transmits the composite video generated by the generation unit 213 to the display 32 and instructs to display it. As described above, the instruction unit 214 corrects the video using the data indicating the postures of the display 32 and the support 33 with respect to the reference coordinate system specified by the specifying unit 36. Thereby, the user views the composite video generated by the generation unit 213 along the direction corresponding to the posture of the display 32 with respect to the support 33.

[0068] For example, the selection unit 212 may select a menu list enumerating instructions to the display 32 according to gestures such as the user's hand or line of sight. The generation unit 213 synthesizes the image of the menu list selected by the selection unit 212 as a selected image onto the video, and the instruction unit 214 may instruct to display the synthesized composite video on the display 32. Thereby, the user views the menu list and further operates the operator 31.

[0069] Therefore, this processor 21 is an example of a processor that generates a composite video by synthesizing one or more selected images selected according to the user's operation from a plurality of images onto the video captured by the first camera installed in the working machine so as to be arranged around the first camera.

[0070] And the information processing apparatus 2 having this processor 21 has a processor, and this processor generates a composite video by synthesizing one or more selected images selected according to the user's operation from a plurality of images onto the video captured by the first camera that captures the surroundings from the working machine so as to be arranged in the surroundings, and displays, on the display worn on the user's head, a video in a range corresponding to the posture of the head among the composite videos. It is an example of an information processing apparatus.

[0071] Further, the display device 32 is an example of a display device that is worn on the user's head and displays an image within a range corresponding to the head posture among the synthesized images.

[0072] Note that the instruction unit 214 may transmit the audio signal received by the reception unit 211 from the microphone 14 to the speaker 34 and instruct the speaker 34 to output audio corresponding to this audio signal. In this case, the speaker 34 outputs the ambient sound collected by the microphone 14 installed in the working machine 1. When a plurality of microphones 14 are provided in the working machine 1 and a plurality of speakers 34 are provided in the input / output device 3, the instruction unit 214 may synthesize, convert a part or all of the audio signals received from the plurality of microphones 14 according to the postures of the support 33 and the display device 32 specified by the specifying unit 36, and adjust the audio output by each speaker 34.

[0073] At this time, for example, the instruction unit 214 causes the speaker 34 to reproduce the audio-visual image with the direction corresponding to the direction of the display device 32 with respect to the support 33 as the front among all directions imaged by the first camera 11. Further, when the microphones 14 are respectively built in the first camera 11, The selection unit 212 may select the first camera 11 according to the content of the received user operation, and also select the microphone 14 built in the selected first camera. In this case, the instruction unit 214 may instruct the speaker 34 to reproduce the audio collected by the microphone 14 built in the selected first camera 11.

[0074] As a result, the user can hear the sound as if he / she were at the installation site of the first camera 11 or the microphone 14. The user grasps the situation around the working machine 1 not only visually but also auditorily, either consciously or unconsciously. Therefore, by reproducing the above-described audio-visual image, the user can obtain a feeling close to the feeling of actually riding on the working machine 1, and it becomes easier to predict an accident or the like.

[0075] Further, the instruction unit 214 may instruct the drive unit 35 to drive the support 33 according to the measurement data indicating the acceleration measured by the measuring instrument 15 and received by the reception unit 211.

[0076] Further, the instruction unit 214 may instruct the operation unit 17 to operate the working machine 1 according to the user operation received by the reception unit 211 from the operator 31. In this case, the working machine 1 is an example of a working machine that operates according to the user operation.

[0077] When the selection unit 212 selects the external camera 4 as a source, the generation unit 213 synthesizes, as a selection image, an appearance video showing the appearance of the working machine 1 photographed by this external camera 4 into the video photographed by the first camera 11.

[0078] FIG. 3 is a diagram showing an example of a composite video in which an appearance video is synthesized. The video G1 shown in FIG. 3 is a video photographed by the first camera 11 provided on the working machine 1. In this video G1, the external camera 4 is reflected in the region R1.

[0079] FIG. 4 is a diagram showing an example of an appearance video including the appearance of the working machine 1. The appearance video G2 shown in FIG. 4 is photographed by the external camera 4. In this appearance video G2, the working machine 1 is reflected in the region R2.

[0080] When the user performs an operation of selecting the appearance video G2 photographed by the external camera 4 as a selection image, the processor 21 of the information processing apparatus 2 that receives this operation performs an operation of fitting the appearance video G2 into a determined frame. This frame is a virtual frame arranged in the space around the first camera 11 that is photographed by the first camera 11. The processor 21 deforms the appearance video G2, for example, so that the four corners of the appearance video G2 are arranged at the four corner points of this frame, and superimposes the deformed appearance video G2c on the video G1 to generate the composite video shown in FIG. 3.

[0081] In this case, the processor 21 is an example of a processor that generates a composite video by synthesizing one or more selected images, selected according to a user operation, from a plurality of images including an appearance video obtained by photographing the working machine from around the working machine, so as to be arranged around the first camera 11.

[0082] Also, when the selection unit 212 selects the second camera 12 as a source, the generation unit 213 synthesizes the video captured by the second camera 12 as a selected image with the video captured by the first camera 11. In this case, the processor 21 is an example of a processor that generates a composite video by synthesizing one or more selected images, selected according to a user operation, from a plurality of images including a video captured by a second camera different from the first camera attached to the working machine, so as to be arranged around the first camera 11.

[0083] <Modification Example> The above is the description of the embodiment, but the content of this embodiment can be modified as follows. Also, the following modification examples may be combined.

[0084] <1> In the above-described embodiment, the processor 21 of the information processing apparatus 2 included in the system 9 selects one or more selected images to be synthesized with the video captured by the first camera 11 from a plurality of images according to a user operation. However, at least one of the position, size, hue, saturation, and brightness of this selected image may be changed according to the received user operation. In this case, this processor 21 is an example of a processor that changes at least one of the position, size, hue, saturation, and brightness in the composite video of the selected image according to a user operation. Thereby, the user can change at least one of the position, size, hue, saturation, and brightness of the selected image to be synthesized with the video captured by the first camera 11.

[0085] <2> In the above-described embodiment, the processor 21 of the information processing apparatus 2 included in the system 9 selected one or more selected images from a plurality of images according to the user's operation. However, one or more selected images may be selected without depending on the user's operation. For example, the processor 21 may determine whether the external appearance video captured by the external camera 4 of the working machine 1 indicates a danger of the working machine 1, and select a selected image based on this determination. In this case, the processor 21 performs object detection using, for example, deep learning on the external appearance video, and determines whether the working machine 1 is facing a learned dangerous state. Here, the danger refers to an event such as the working machine 1 colliding with an obstacle such as a rock or another working machine, or a pedestrian, if it continues to move.

[0086] When it is determined that the external appearance video indicates a danger of the working machine 1, the processor 21 may select this external appearance video as a selected image, synthesize it with the video captured by the first camera 11, and generate a synthesized video.

[0087] Therefore, the processor 21 is an example of a processor that generates a synthesized video obtained by synthesizing this external appearance video when the external appearance video indicates a danger of the working machine.

[0088] <3> In addition, when the processor 21 of the information processing apparatus 2 included in the system 9 determines that the external appearance video captured by the external camera 4 of the working machine 1 indicates a danger of the working machine 1, the processor 21 may emphasize this external appearance video when synthesizing it with the video captured by the first camera 11.

[0089] Specific modes of emphasizing the external appearance video include, for example, changing the color of the frame drawn on the contour of the external appearance video from colorless to red, thickening the thickness of the frame, or displaying characters, numbers, symbols, illustrations, etc. superimposed on the external appearance video.

[0090] The processor 21 is an example of a processor that generates a composite image in which an appearance video is emphasized when the appearance video indicates a danger of the working machine, as compared to when it does not indicate a danger. As a result, the user can distinguish between the appearance video when the working machine 1 indicates a danger and the appearance video when it does not.

[0091] <4> In the above-described embodiment, the processor 21 of the information processing apparatus 2 included in the system 9 selects the first camera 11 according to the content of the received user operation. At this time, the user operation may be to specify a selection image indicating a working machine different from the working machine 1 (that is, another working machine) among the selection image candidates and instruct a changeover to that working machine. When a changeover is instructed, the processor 21 may change the currently selected first camera 11 of the working machine 1 to the first camera 11 of the instructed "other working machine".

[0092] Here, the selection image indicating another working machine may be an image in which symbols indicating the model number or lot number of the working machine are drawn, or may be an image obtained by photographing the appearance of the working machine. Furthermore, the selection image indicating "other working machine" may be an image obtained by photographing the working machine with the external camera 4. Note that the operation instructing a changeover may be detected by the processor 21 performing pattern matching with a predetermined gesture or the like.

[0093] Also, for example, when the operator 31 is a device that receives an operation by the user's line of sight, when the operator 31 detects that the user has continuously looked at a selection image indicating another working machine for a certain period of time or more, the processor 21 may interpret this operation by the line of sight as an operation instructing a changeover to the other working machine that has been continuously looked at.

[0094] When the first camera 11 is changed according to a user operation, the processor 21 generates a composite video in which one or more selected images are combined with the surrounding video captured by the changed first camera 11 as seen from the working machine 1 equipped with the first camera 11, and causes the display 32 to display the composite video. As a result, the user can perform "switching" to change the working machine to be operated only by performing a switching operation on the selected image.

[0095] In this case, the processor 21 is an example of a processor that generates a composite video in which one or more selected images are combined with the video captured by the first camera that captures the surroundings from that working machine, when the user operation designates a selected image indicating another working machine among the selected images and instructs switching to that working machine.

[0096] <5> The program executed by the processor 21 of the information processing apparatus 2 causes a computer having a processor to generate a composite video in which one or more selected images selected according to a user operation from a plurality of images are combined as if they were arranged around the video captured by the first camera that captures the surroundings from the working machine, and to display, on a display worn on the user's head, a video in a range corresponding to the posture of the head among the composite videos. It can be regarded as a program for realizing the function.

[0097] This program can be provided in a state stored in a computer-readable recording medium such as a magnetic recording medium such as a magnetic tape and a magnetic disk, an optical recording medium such as an optical disk, a magneto-optical recording medium, and a semiconductor memory. Further, this program may be downloaded via a communication line such as the Internet. Note that various devices other than the CPU may be applied as the control means exemplified by the above-described processor 21. For example, a dedicated processor or the like may be used.

[0098] <6> In the above-described embodiment, the input / output device 3 had the support 33 and the drive unit 35, but it is not necessary to have these. In this case, the specifying unit 36 only needs to specify the posture of the display 32 in the reference coordinate system, and it is not necessary to specify the posture of the support 33.

[0099] Also, in this case, the processor 21 may correct the video captured by the first camera 11 based on the information on the posture of the display 32 specified by the specifying unit 36. Further, the processor 21 does not necessarily have to correct the video captured by the first camera 11. In this case, the input / output device 3 does not necessarily have to have the specifying unit 36 that specifies the posture of the display 32.

Explanation of Reference Numerals

[0100] 1... working machine, 11... first camera, 12... second camera, 13... communication unit, 14... microphone, 15... measuring instrument, 16... positioning device, 17... operating unit, 2... information processing device, 21... processor, 211... reception unit, 212... selection unit, 213... generation unit, 214... instruction unit, 22... memory, 23... communication unit, 3... input / output device, 31... operator, 32... display, 33... support, 34... sound player, 35... drive unit, 36... specifying unit, 4... external camera, 9... system.

Claims

1. A working machine, a first camera that captures the surroundings from the working machine, an operator that receives a user's operation, a processor that generates a composite video by synthesizing one or more selected images selected according to the user's operation from a plurality of images including an appearance video of capturing the working machine from around the working machine in the video captured by the first camera so as to be arranged around, a display device that is worn on the user's head and displays a video within a range corresponding to the posture of the head among the composite videos, and has, The processor is a system that generates the composite video in which the appearance video is synthesized when the appearance video indicates a danger of the working machine.

2. A working machine, a first camera that captures the surroundings from the working machine, a second camera different from the first camera attached to the working machine, an operator that receives a user's operation, a processor that generates a composite video by synthesizing one or more selected images selected according to the user's operation from a plurality of images including the video captured by the second camera in the video captured by the first camera so as to be arranged around, a display device that is worn on the user's head and displays a video within a range corresponding to the posture of the head among the composite videos, and has a system.

3. A working machine, a first camera that captures the surroundings from the working machine, a microphone that collects sounds around the working machine, a sound emitter that emits a sound corresponding to the posture of the user's head among the sounds collected by the microphone, an operator that receives a user's operation, a processor that generates a composite video by synthesizing one or more selected images selected according to the user's operation from a plurality of images in the video captured by the first camera so as to be arranged around, a display device that is worn on the head and displays a video within a range corresponding to the posture of the head among the composite videos, and has a system.

4. A plurality of working machines, a first camera that captures the surroundings from each of the plurality of working machines, an operator that receives a user's operation, a processor that generates a composite video by synthesizing one or more selected images selected according to the user's operation from a plurality of images in the video captured by the first camera so as to be arranged around, a display device that is worn on the user's head and displays a video within a range corresponding to the posture of the head among the composite videos, and has, The system wherein, when the operation of the user designates a selection image indicating another working machine among the selection images and instructs a transfer to the other working machine, a composite video is generated by combining one or more of the selection images with the video captured by a first camera that captures the surroundings from the other working machine.

5. A working machine, a first camera that captures the surroundings from the working machine, an operator that receives the operation of the user, a processor that generates a composite video by combining one or more selection images selected according to the operation of the user from a plurality of images including an appearance video that captures the working machine from around the working machine, so as to be arranged around the working machine, in the video captured by the first camera, a display device that is worn on the head of the user and displays a video within a range corresponding to the posture of the head, among the composite video, The system wherein the processor generates the composite video by deforming the appearance video.

6. A working machine, a first camera that captures the surroundings from the working machine, an operator that receives the operation of the user, a processor that generates a composite video by combining one or more selection images selected according to the operation of the user from a plurality of images, so as to be arranged around the working machine, in the video captured by the first camera, a display device that is worn on the head of the user and displays a video within a range corresponding to the posture of the head, among the composite video, The system wherein the operator captures the eyes of the user, detects the line of sight of the user, and receives the operation by the line of sight.

7. A working machine, a first camera that captures the surroundings from the working machine, an operator that receives the operation of the user, a processor that generates a composite video by combining one or more selection images selected according to the operation of the user from a plurality of images, so as to be arranged around the working machine, in the video captured by the first camera, a measuring device that measures the acceleration received by the first camera, a support that supports the body of the user, a driving machine that drives the support according to the acceleration measured by the measuring device, a display device that is worn on the head of the user and displays a video within a range corresponding to the posture of the head with respect to the support, among the composite video. The posture of the head with respect to the support is specified using the difference between each posture of the display device with respect to the reference coordinate system in which the support and the drive machine are installed and the posture of the support with respect to the reference coordinate system. A system characterized by that.

8. The processor, In response to the operation of the user, change at least one of the position, size, hue, saturation, and brightness of the selected image in the composite video. The system according to any one of claims 1 to 7.

9. The processor, When the appearance video indicates a danger of the work machine, generate a composite video in which the appearance video is emphasized compared to when it does not indicate the danger. The system according to claim 1.

10. The work machine operates in response to the operation of the user. The system according to any one of claims 1 to 9.

11. The operator accepts the operation by the user's line of sight. The system according to any one of claims 1 to 5, 7 or 9.

12. A measuring instrument that measures the acceleration received by the first camera, A support that supports the user's body, A drive machine that drives the support according to the acceleration measured by the measuring instrument, Having, The display device displays a video within a range corresponding to the posture of the head with respect to the support. The system according to any one of claims 1 to 6, or 9.

13. A positioning device that measures the position of the work machine, having, The plurality of images include a map showing the position measured by the positioning device. The system according to any one of claims 1 to 12.

14. Having a processor, the processor, In the video taken by the first camera that takes pictures of the surroundings from the work machine, generate a composite video in which one or more selected images selected according to the user's operation from a plurality of images including the appearance video taken of the work machine from around the work machine are arranged around the work machine, or generate the composite video by deforming the appearance video, On the display device worn on the user's head, display a video within a range corresponding to the posture of the head among the composite videos. Information processing device.

15. In a computer having a processor, A function of generating a composite video in which one or more selected images selected according to a user's operation from a plurality of images including an external appearance video of photographing the working machine from around the working machine are arranged around the working machine in the video photographed by the first camera photographing the surroundings of the working machine. A function of generating the composite video by deforming the external appearance video. A function of displaying, on a display device worn on the user's head, a video within a range corresponding to the posture of the head among the composite videos. A program for realizing the above.

Images