Operation system

Abstract

SOLUTION: An operation system 10 which enables operation via a mobile body for operation 14 remotely controlled by a user U includes: an image generation part 60 for generating a virtual space image 200b corresponding to a real space surrounding the mobile body for operation 14; and a head-mounted display 18 for being worn by the user U and for providing the virtual space image 200b generated by the image generation part 60 to the user. The image generation part 60 generates the virtual space image 200b corresponding to location and orientation of the mobile body for operation 14.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a work system. [Background technology]

[0002] Patent Document 1 discloses a virtual space display system. In this system, a passenger in an automatically driven vehicle wears a head-mounted display. The head-mounted display displays a virtual space image that differs from the actual environment around the vehicle to the passenger in the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6232649 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 merely discloses displaying a virtual space image. Recently, there has been a demand for providing a good working system.

[0005] The present invention aims to solve the above-mentioned problems. [Means for solving the problem]

[0006] An aspect of the present disclosure is a work system in which work can be performed by a work vehicle remotely controlled by a user, comprising an image generation unit that generates a virtual space image corresponding to the real space around the work vehicle, and a head-mounted display that is worn by the user and provides the user with the virtual space image generated by the image generation unit, wherein the image generation unit generates the virtual space image according to the position and orientation of the work vehicle. [Effects of the Invention]

[0007] According to the present invention, a good working system can be provided. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic configuration diagram of a work system according to a first embodiment. [Figure 2] FIG. 2 is a functional block diagram of the controller according to the first embodiment. [Figure 3] FIG. 3 is a functional block diagram of the working vehicle according to the first embodiment. [Figure 4] FIG. 4 is a functional block diagram of the management device according to the first embodiment. [Figure 5] FIG. 5 is a functional block diagram of the head-mounted display according to the first embodiment. [Figure 6] FIG. 6 is a sequence diagram relating to the operation of a working vehicle. [Figure 7] FIG. 7 is a sequence diagram relating to the display of a virtual space image. [Figure 8] Fig. 8A is a diagram showing a real space image, and Fig. 8B is a diagram showing a virtual space image. [Figure 9] FIG. 9 is a functional block diagram of a management device according to the second embodiment. [Figure 10] FIG. 10 is a functional block diagram of a management device according to the third embodiment. [Figure 11] FIG. 11 is a functional block diagram of a management device according to the fifth embodiment. [Figure 12] FIG. 12 is a functional block diagram of a management device according to the sixth embodiment. [Figure 13] FIG. 13 is a functional block diagram of a working vehicle according to the seventh embodiment. [Figure 14] FIG. 14 is a schematic configuration diagram of a communication system according to the eighth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Remotely controlled mobile work vehicles for light work (e.g., self-propelled vacuum cleaners, self-propelled lawn mowers, etc.) have been developed. A user performs work by remotely controlling the mobile work vehicle with a controller. By using virtual space images for this work, the user can enjoy the work. Specific embodiments of this technology are described below.

[0010] [1 First Embodiment] [1-1 Configuration of Work System 10] FIG. 1 is a schematic configuration diagram of a work system 10 according to a first embodiment. The work system 10 includes a controller 12, a work vehicle 14, a management device 16, and a head-mounted display 18. The work vehicle 14 performs a predetermined task (cleaning, lawn mowing, etc.) while moving. A user U uses the controller 12 to remotely control the work vehicle 14. The user U may wear the head-mounted display 18 while remotely controlling the work vehicle 14. The head-mounted display 18 provides the user U with a virtual space image corresponding to the position and orientation of the work vehicle 14. This allows the user U to perform work in real space while experiencing the virtual space.

[0011] In the work system 10, the devices can communicate wirelessly with each other. For example, the controller 12 and the work vehicle 14 can communicate wirelessly with each other. The work vehicle 14 and the management device 16 can communicate wirelessly with each other. The management device 16 and the head-mounted display 18 can communicate wirelessly with each other.

[0012] 2 is a functional block diagram of the controller 12 according to the first embodiment. The controller 12 is a remote controller for operating a work vehicle 14. The controller 12 is operated by a user U. The controller 12 includes an operation detection unit 20, a communication unit 22, a calculation unit 24, and a storage unit 26.

[0013] The operation detection unit 20 may be configured, for example, by an operator (a lever, a button, etc.) that can be operated by the user U, and a sensor that detects and outputs the operation of the operator (the amount of operation, the direction of operation, etc.). The operation detection unit 20 outputs information indicating the operation of the operator (referred to as operation information) to the calculation unit 24.

[0014] The communication unit 22 may be configured by, for example, a wireless communication module (integrated circuit module) equipped with an antenna, etc. The communication unit 22 may transmit a signal to the outside of the controller 12.

[0015] The calculation unit 24 may be configured by a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). That is, the calculation unit 24 may be configured by processing circuitry. At least a part of the calculation unit 24 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). At least a part of the calculation unit 24 may be realized by an electronic circuit including discrete devices.

[0016] The calculation unit 24 includes an acquisition unit 28 and a communication control unit 30. The acquisition unit 28 and the communication control unit 30 can be realized by the calculation unit 24 executing a program stored in the storage unit 26. The acquisition unit 28 acquires a signal transmitted from outside the calculation unit 24. The communication control unit 30 performs processing to transmit a signal to outside the controller 12 via the communication unit 22.

[0017] The storage unit 26 is a computer-readable storage medium. The storage unit 26 is composed of a volatile memory (not shown) and a non-volatile memory (not shown). The volatile memory is, for example, a random access memory (RAM). The non-volatile memory is, for example, a read-only memory (ROM), a flash memory, etc. Data, etc. are stored in, for example, the volatile memory. Programs, tables, maps, etc. are stored in, for example, the non-volatile memory. At least a part of the storage unit 26 may be provided in the above-mentioned processor, integrated circuit, etc.

[0018] 3 is a functional block diagram of a work vehicle 14 according to the first embodiment. The work vehicle 14 is a vehicle that performs a predetermined task while self-propelled. For example, the work vehicle 14 may be a self-propelled vacuum cleaner, a self-propelled lawnmower, or the like. The work vehicle 14 includes an imaging unit 32, a behavior detection unit 34, a drive unit 36, a communication unit 38, a calculation unit 40, and a storage unit 42.

[0019] The imaging unit 32 may be configured with a camera. The imaging unit 32 captures images of the surroundings of the working vehicle 14 to acquire images of real space (referred to as real space images). The imaging unit 32 outputs the real space images to the calculation unit 40.

[0020] The behavior detection unit 34 is configured with, for example, an encoder, an acceleration sensor, a gyro sensor, etc. The behavior detection unit 34 detects behavior related to the traveling of the working vehicle 14. The behavior detection unit 34 outputs the detected information (referred to as behavior information) to the calculation unit 40.

[0021] The drive unit 36 ​​may be composed of, for example, a battery, a power supply circuit, an electric motor, a power transmission mechanism, and left and right wheels. The power supply circuit supplies power from the battery to the electric motor. The power transmission mechanism transmits power from the electric motor to the left and right wheels.

[0022] The communication unit 38 may be configured, for example, by a wireless communication module equipped with an antenna, etc. The communication unit 38 may transmit signals to the outside of the working vehicle 14. The communication unit 38 may also receive signals from the outside of the working vehicle 14.

[0023] The calculation unit 24 may be configured by a processor such as a CPU or a GPU. That is, the calculation unit 40 may be configured by a processing circuit. At least a part of the calculation unit 40 may be realized by an integrated circuit such as an ASIC or an FPGA. At least a part of the calculation unit 40 may be realized by an electronic circuit including discrete devices.

[0024] The calculation unit 40 includes an acquisition unit 44, a mobile body control unit 46, and a communication control unit 48. The acquisition unit 44, the mobile body control unit 46, and the communication control unit 48 can be realized by the calculation unit 40 executing a program stored in the storage unit 42. The acquisition unit 44 acquires signals transmitted from outside the calculation unit 40. The mobile body control unit 46 controls operations related to the movement of the work mobile body 14. The communication control unit 48 performs processing to transmit signals to outside the work mobile body 14 via the communication unit 38.

[0025] The storage unit 42 is a computer-readable storage medium. The storage unit 42 is composed of a volatile memory (not shown) and a non-volatile memory (not shown). The volatile memory is, for example, a RAM. The non-volatile memory is, for example, a ROM, a flash memory, etc. Data, etc. are stored in, for example, the volatile memory. Programs, tables, maps, etc. are stored in, for example, the non-volatile memory. At least a portion of the storage unit 42 may be provided in the above-mentioned processor, integrated circuit, etc.

[0026] 4 is a functional block diagram of the management device 16 according to the first embodiment. The management device 16 is a device that generates a virtual space image corresponding to a real space. For example, the management device 16 may be a computer. The management device 16 includes a communication unit 50, a calculation unit 52, and a storage unit 54.

[0027] The communication unit 50 may be configured, for example, by a wireless communication module including an antenna, etc. The communication unit 50 may transmit signals to the outside of the management device 16. The communication unit 50 may also receive signals from the outside of the management device 16.

[0028] The calculation unit 52 may be configured by a processor such as a CPU or a GPU. That is, the calculation unit 52 may be configured by a processing circuit. At least a part of the calculation unit 52 may be realized by an integrated circuit such as an ASIC or an FPGA. At least a part of the calculation unit 52 may be realized by an electronic circuit including discrete devices.

[0029] The calculation unit 52 includes an acquisition unit (data acquisition unit) 56, an image recognition unit 58, an image generation unit 60, a sound generation unit 62, and a communication control unit 64. The acquisition unit 56, the image recognition unit 58, the image generation unit 60, the sound generation unit 62, and the communication control unit 64 can be realized by the calculation unit 52 executing a program stored in the storage unit 54. The acquisition unit 56 acquires a signal transmitted from outside the calculation unit 52. The image recognition unit 58 recognizes the real space by performing image recognition. The image generation unit 60 generates a virtual space image corresponding to the real space and generates an image signal indicating the virtual space image. The sound generation unit 62 generates sound effects and generates a sound signal indicating the sound effects. The communication control unit 64 performs processing to transmit a signal to outside the management device 16 via the communication unit 50.

[0030] The storage unit 54 is a computer-readable storage medium. The storage unit 54 is composed of a volatile memory (not shown) and a non-volatile memory (not shown). The volatile memory is, for example, a RAM. The non-volatile memory is, for example, a ROM, a flash memory, etc. Data and the like are stored in, for example, the volatile memory. Programs, tables, maps, etc. are stored in, for example, the non-volatile memory. At least a portion of the storage unit 54 may be provided in the above-mentioned processor, integrated circuit, etc.

[0031] The storage unit 54 stores in advance image data of the virtual space image generated by the image generation unit 60. The storage unit 54 stores in advance sound data of the sound effects generated by the sound generation unit 62.

[0032] 5 is a functional block diagram of the head mounted display 18 according to the first embodiment. In this specification, the head mounted display 18 is also referred to as an HMD (Head Mounted Display) 18. The HMD 18 is worn on the head of a user U. The HMD 18 includes a communication unit 66, a display unit 68, an audio unit (audio output unit) 70, a calculation unit 72, and a storage unit 74.

[0033] The communication unit 66 may be configured, for example, by a wireless communication module including an antenna, etc. The communication unit 66 may transmit signals to the outside of the HMD 18. The communication unit 66 may also receive signals from the outside of the HMD 18.

[0034] The display unit 68 may be configured, for example, by a display device. The screen or face of the display device is located close to both eyes of the user U. The display unit 68 displays (projects) a virtual space image on a display surface based on an image signal transmitted from the calculation unit 72. The display unit 68 may provide the virtual space image to the user U.

[0035] The sound unit 70 may be configured, for example, by an audio device. The speakers (including headphones and earphones) of the sound unit 70 are placed close to the ears of the user U. The sound unit 70 outputs sound effects (sound) from the speakers based on the sound signals transmitted from the calculation unit 72. The sound unit 70 may provide the sound effects to the user U.

[0036] The calculation unit 72 may be configured by a processor such as a CPU or a GPU. That is, the calculation unit 72 may be configured by a processing circuit. At least a part of the calculation unit 72 may be realized by an integrated circuit such as an ASIC or an FPGA. At least a part of the calculation unit 72 may be realized by an electronic circuit including discrete devices.

[0037] The calculation unit 72 includes an acquisition unit 76, a display control unit 78, and an audio control unit 80. The acquisition unit 76, the display control unit 78, and the audio control unit 80 can be realized by the calculation unit 72 executing a program stored in the storage unit 74. The acquisition unit 76 acquires a signal transmitted from outside the calculation unit 72. The display control unit 78 causes the display unit 68 to display a virtual space image. The audio control unit 80 causes the audio unit 70 to output sound effects.

[0038] The storage unit 74 is a computer-readable storage medium. The storage unit 74 is configured by a volatile memory (not shown) and a non-volatile memory (not shown). The volatile memory is, for example, a RAM. The non-volatile memory is, for example, a ROM, a flash memory, etc. Data, etc. are stored in, for example, the volatile memory. Programs, tables, maps, etc. are stored in, for example, the non-volatile memory. At least a portion of the storage unit 74 may be provided in the above-mentioned processor, integrated circuit, etc.

[0039] [1-2 Functions of Work System 10] In the work system 10, the user U operates the work vehicle 14, and a virtual space image is provided to the user U.

[0040] 6 is a sequence diagram relating to the operation of the working vehicle 14. When the user U operates the operation element of the controller 12, the working vehicle 14 moves in accordance with the user U's operation.

[0041] In step S1, the operation detection unit 20 of the controller 12 detects an operation of a control by the user U. The acquisition unit 28 of the controller 12 acquires operation information from the operation detection unit 20.

[0042] In step S2, the communication control unit 30 of the controller 12 generates an operation signal according to the operation information and performs processing to transmit the generated operation signal. The communication unit 22 of the controller 12 transmits the operation signal to the working vehicle 14 according to the processing performed by the communication control unit 30.

[0043] In step S3, the communication unit 38 of the working vehicle 14 receives the operation signal. The acquisition unit 44 of the working vehicle 14 acquires the operation signal via the communication unit 38.

[0044] In step S4, the mobile unit control unit 46 of the work vehicle 14 controls the operation of the work vehicle 14 in response to the operation signal acquired by the acquisition unit 44. For example, the mobile unit control unit 46 controls the operation of the electric motor provided in the drive unit 36.

[0045] 7 is a sequence diagram relating to the display of a virtual space image. While working with the working vehicle 14, the user U is provided with virtual space images and sound effects by the HMD 18.

[0046] In step S11, the imaging unit 32 of the working vehicle 14 constantly or at regular time intervals captures images of the surroundings of the working vehicle 14. The acquisition unit 44 of the working vehicle 14 acquires a real space image from the imaging unit 32.

[0047] In step S12, the communication control unit 48 of the working vehicle 14 generates an image signal corresponding to the real space image and performs processing to transmit the generated image signal. The communication unit 38 of the working vehicle 14 transmits the image signal to the management device 16 in accordance with the processing performed by the communication control unit 48.

[0048] In step S13, the communication unit 50 of the management device 16 receives the image signal. The acquisition unit 56 of the management device 16 acquires the image signal via the communication unit 50.

[0049] In step S14, the image recognition unit 58 of the management device 16 performs image recognition on the real space image represented by the image signal. Furthermore, the image recognition unit 58 determines whether or not a specific space or a specific object exists around the working vehicle 14. Information for recognizing the specific space and the specific object is preset in the memory unit 54.

[0050] In step S15, the image generation unit 60 of the management device 16 generates a virtual space image corresponding to the real space image. For example, the image generation unit 60 generates the virtual space image by superimposing an image of a predetermined character (animal, dinosaur, etc.) on the real space image. The image generation unit 60 may superimpose the image of the predetermined character on a specific space or specific object recognized by the image recognition unit 58, or may superimpose the image of the predetermined character at any position on the real space image. The image generation unit 60 may superimpose an image other than a character, such as an image of a desert, on the floor of the real space image. Data on the image of the predetermined character is stored in the storage unit 54. The image generation unit 60 may generate a virtual space image corresponding to the real space image using computer graphics, etc. The image generation unit 60 may also generate data for another image using image data. In step S15, the sound generation unit 62 of the management device 16 generates sound effects. The sound effect data is stored in the storage unit 54. The sound generation unit 62 may also generate data for another sound effect using sound effect data.

[0051] In step S16, the communication control unit 64 of the management device 16 generates an image signal corresponding to the virtual space image and an audio signal corresponding to the sound effect, and performs processing to transmit the generated image signal and audio signal. The communication unit 50 of the management device 16 transmits the image signal and audio signal to the HMD 18 in accordance with the processing performed by the communication control unit 64.

[0052] In step S17, the communication unit 66 of the HMD 18 receives the image signal and the sound signal. The acquisition unit 76 of the HMD 18 acquires the image signal and the sound signal via the communication unit 66.

[0053] In step S18, the display control unit 78 of the HMD 18 causes the display unit 68 to display a virtual space image in accordance with the image signal acquired by the acquisition unit 76. In addition, the audio control unit 80 of the HMD 18 causes the audio unit 70 to output sound effects in accordance with the audio signal acquired by the acquisition unit 76.

[0054] FIG. 8A is a diagram showing a real space image. FIG. 8B is a diagram showing a virtual space image. According to this embodiment, the imaging unit 32 of the working vehicle 14 acquires an image 200a (corresponding to a real space image) as shown in FIG. 8A. The management device 16 generates an image 200b (corresponding to a virtual space image) as shown in FIG. 8B based on image 200a. The display unit 68 of the HMD 18 displays image 200b. This allows a user U wearing the HMD 18 to experience the sensation of working in a virtual space where various characters exist.

[0055] [1-3 Variations] An operation signal generated by the controller 12 may be transmitted to the work vehicle 14 via the management device 16. The controller 12, the work vehicle 14, the management device 16, and the head-mounted display 18 may communicate with each other via a communication line such as the Internet.

[0056] When the image generation unit 60 of the management device 16 generates the entire virtual space image based on image data generated by computer graphics or the like, it is not necessary to acquire a real space image. In this case, the image generation unit 60 must constantly grasp the position and orientation of the work vehicle 14. For example, the image generation unit 60 grasps the position and orientation of the work vehicle 14 as follows.

[0057] The memory unit 54 of the management device 16 stores in advance a map of the work area, as well as the positions and shapes of objects present within the work area. The map of the work area includes information on a coordinate system with a reference position as its origin. The reference position is the position from which the work vehicle 14 starts moving (such as the position of a charger), and is determined in advance. The mobile unit control unit 46 can estimate the position and orientation of the work vehicle 14 within the work area based on the behavior signal transmitted from the work vehicle 14 and the map of the work area stored in the memory unit 54. The image generation unit 60 generates a virtual space image using computer graphics or the like, corresponding to the position and orientation of the work vehicle 14 estimated by the mobile unit control unit 46.

[0058] [1-4 Effects of the first embodiment] According to the first embodiment, the user U can perform work in real space while experiencing a virtual space. This allows the user U to enjoy working using the work vehicle 14. As a result, the user U becomes more proactive in working using the work vehicle 14, which promotes the work. In other words, according to this embodiment, a good work system 10 can be provided.

[0059] Various functions can be added to the above-described first embodiment and its modifications. Functions that can be added to the first embodiment and its modifications will be described in the second to eighth embodiments.

[0060] [2 Second Embodiment] The image generating unit 60 of the management device 16 may generate a virtual space image that enhances the visibility of the work area in accordance with the work priority level. A specific example of this will be described below.

[0061] 9 is a functional block diagram of the management device 16 according to the second embodiment. In the third embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

[0062] The calculation unit 52 of the management device 16 includes an image determination unit 81. The image determination unit 81 can be realized by the calculation unit 52 executing a program stored in the storage unit .

[0063] The image determination unit 81 determines whether or not there is an area in a predetermined state in real space based on the recognition result of the real space image by the image recognition unit 58. If the working vehicle 14 is a vacuum cleaner, the area in the predetermined state may be an area where the amount of dust is greater than a predetermined value. If the working vehicle 14 is a lawnmower, the area in the predetermined state may be an area where the length of grass is greater than a predetermined value.

[0064] For example, the image determination unit 81 determines the task priority level of an area in a predetermined state to be high, and determines the task priority level of an area other than the area in the predetermined state to be low. The image generation unit 60 generates a virtual space image based on the result of the determination by the image determination unit 81. The image generation unit 60 may increase the visibility of the area in the predetermined state by changing the display color of the area in the virtual space image to a color that is more eye-catching than the display colors of other areas. Alternatively, the image generation unit 60 may increase the visibility of the area in the predetermined state by superimposing a highly eye-catching display object on the area in the virtual space image.

[0065] The image determining unit 81 may determine that the work priority level of a specific area (such as a corner of the work area) is high, and may determine that the work priority level of areas other than the specific area is low.

[0066] According to the second embodiment, by increasing the visibility of areas with high work priority levels, the user U can be prompted to move the work vehicle 14 to areas with high work priority levels. This allows work in areas with high work priority levels to be performed preferentially.

[0067] [3 Third embodiment] The image generation unit 60 of the management device 16 may change the virtual space image depending on the work status of the work vehicle 14. Similarly, the sound generation unit 62 of the management device 16 may change the sound effects depending on the work status of the work vehicle 14.

[0068] 10 is a functional block diagram of the management device 16 according to the third embodiment. In the third embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

[0069] The calculation unit 52 of the management device 16 includes a first determination unit (determination unit) 82. The first determination unit 82 can be realized by the calculation unit 52 executing a program stored in the storage unit .

[0070] The first determination unit 82 determines the work status of the work mobile unit 14. For example, the first determination unit 82 determines the movement speed of the work mobile unit 14 as the work status of the work mobile unit 14. In this case, the first determination unit 82 determines the movement speed of the work mobile unit 14 based on the behavior signal acquired by the acquisition unit 56. Furthermore, the first determination unit 82 compares the movement speed of the work mobile unit 14 with a speed threshold pre-stored in the storage unit 54. If the movement speed is equal to or greater than the speed threshold, the first determination unit 82 determines that the work mobile unit 14 is moving at a high speed and the work status is good. On the other hand, if the movement speed is less than the speed threshold, the first determination unit 82 determines that the work mobile unit 14 is moving at a low speed and the work status is poor (work is slow).

[0071] When the first determination unit 82 determines that the work status of the work vehicle 14 is poor (the work is slow), the image generation unit 60 generates a virtual space image to encourage the user U to work. For example, the image generation unit 60 generates a virtual space image including an image of a character encouraging the user U to work. The image generation unit 60 may also change the display color of the virtual space image to a different display color from normal. The image generation unit 60 may also generate a virtual space image including a message.

[0072] When the first determination unit 82 determines that the work status of the work vehicle 14 is poor (the work is slow), the sound generation unit 62 generates a sound effect to encourage the user U to work. For example, the sound generation unit 62 sets the tempo of the music as the sound effect to encourage the user U to work.

[0073] According to the third embodiment, the user U can be encouraged to work quickly, thereby increasing the speed at which work is performed using the working vehicle 14.

[0074] [4 Fourth embodiment] The image generation unit 60 of the management device 16 may be capable of generating multiple types of virtual space images with different viewpoints. For example, the image generation unit 60 may be capable of generating a first virtual space image that is a virtual space image corresponding to the case where the viewpoint is located on the work vehicle 14, and a second virtual space image that is a virtual space image corresponding to the case where the viewpoint is located outside the work vehicle 14.

[0075] The image generation unit 60 generates a first virtual space image by the method described in the first embodiment. The image generation unit 60 also generates a second virtual space image, which is a bird's-eye view of the work area from above the work area, using computer graphics or the like. Note that if the work area is equipped with an overhead camera (not shown) that overlooks the work area, the image generation unit 60 may generate a second virtual space image that corresponds to the real space image acquired by the overhead camera.

[0076] Furthermore, when generating the second virtual space image, the image generation unit 60 may set a virtual object corresponding to the work vehicle 14 for the work vehicle 14 in the second virtual space image. For example, the image generation unit 60 may superimpose an animal, a car, or the like on the work vehicle 14 in the second virtual space image.

[0077] Furthermore, the virtual space image displayed on HMD 18 may be switchable in response to an operation by user U. In this case, a first image signal indicating the first virtual space image and a second image signal indicating the second virtual space image are transmitted from management device 16 to HMD 18. When user U operates a switch (not shown) on HMD 18, display control unit 78 causes display unit 68 to selectively display either the first virtual space image or the second virtual space image.

[0078] According to the fourth embodiment, the user U can operate a virtual object while viewing the virtual object in a virtual space. This allows the user U to enjoy working using the working vehicle 14 more.

[0079] [5 Fifth embodiment] The image generation unit 60 of the management device 16 may change the virtual space image in accordance with the work history of the user U. Similarly, the sound generation unit 62 of the management device 16 may change the sound effects in accordance with the work history of the user U.

[0080] 11 is a functional block diagram of the management device 16 according to the fifth embodiment. In the fifth embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

[0081] The calculation unit 52 of the management device 16 includes a history management unit 84 and a second determination unit 86. The history management unit 84 and the second determination unit 86 can be realized by the calculation unit 52 executing a program stored in the storage unit 54.

[0082] The history management unit 84 manages the work history of the user U. The work history may be a total time obtained by adding up the time the user U has operated the work vehicle 14. The work history may be a total distance obtained by adding up the distance the user U has moved the work vehicle 14. The work history may be work characteristics of the user U. The work history may be operation characteristics of the user U. The storage unit 54 stores work history information for each user U.

[0083] When the work history is a total time, the history management unit 84 counts the time the work mobile unit 14 worked. The time the work mobile unit 14 worked (working time of the work mobile unit 14) corresponds to the time the user U operated the work mobile unit 14. When the work of the work mobile unit 14 is completed, the history management unit 84 adds the counted time to the total time stored in the memory unit 54. As a result, the working time of the work mobile unit 14 is accumulated in the total time in the memory unit 54. The total time corresponds to the experience value of the user U. The total time serves as a guide for determining the skill level of the user U.

[0084] When the work history is a total distance, the history management unit 84 counts the distance traveled by the work mobile unit 14 based on the detection result of the behavior detection unit 34 of the work mobile unit 14. The distance traveled by the work mobile unit 14 (travel distance of the work mobile unit 14) corresponds to the distance traveled by the user U with the work mobile unit 14. When the work of the work mobile unit 14 is completed, the history management unit 84 adds the counted distance to the total distance stored in the memory unit 54. As a result, the travel distance of the work mobile unit 14 is accumulated in the total distance stored in the memory unit 54. The total distance corresponds to the experience value of the user U. The total distance serves as a guide for determining the skill level of the user U.

[0085] When the work history is work characteristics of the user U, the history management unit 84 stores in the storage unit 54, for example, the work mode set for the work vehicle 14 each time work is performed. Work modes include a mode for beginners and a mode for advanced workers. The user U sets the work mode of the work vehicle 14 according to their own skill level. The work characteristics serve as a guide for determining the skill level of the user U.

[0086] When the work history is the operation characteristics of the user U, the history management unit 84 monitors the movement trajectory of the work vehicle 14, for example, each time work is performed. Furthermore, the history management unit 84 stores areas that have been moved to more than once in the memory unit 54. That is, the history management unit 84 stores areas where work overlaps (referred to as overlapping areas) in the memory unit 54. A user U with a high technical level has few overlapping areas in a single work. A user U with a low technical level has many overlapping areas in a single work. In other words, the operation characteristics serve as a guide for determining the technical level of the user U.

[0087] The second determination unit 86 determines the skill level of the user U based on the work history. For example, the second determination unit 86 determines that the user U is an advanced player when the total time stored in the memory unit 54 is longer than a predetermined time. For example, the second determination unit 86 determines that the user U is an advanced player when the total distance stored in the memory unit 54 is longer than a predetermined distance. For example, the second determination unit 86 determines that the user U is an advanced player when the latest work mode stored in the memory unit 54 is a mode for advanced players. For example, the second determination unit 86 determines that the user U is an advanced player when the amount of overlapping area stored in the memory unit 54 is smaller than a predetermined amount.

[0088] When the second determination unit 86 determines that the user U is an advanced player, the image generation unit 60 generates a virtual space image for advanced players. On the other hand, when the second determination unit 86 determines that the user U is not an advanced player, the image generation unit 60 generates a normal virtual space image.

[0089] When the second determination unit 86 determines that the user U is an advanced player, the sound generation unit 62 generates sound effects for advanced players. On the other hand, when the second determination unit 86 determines that the user U is not an advanced player, the sound generation unit 62 generates normal sound effects.

[0090] According to the fifth embodiment, the virtual space image and sound effects change depending on the skill level of the user U, which can motivate the user U to improve his / her skills.

[0091] [6 Sixth Embodiment] In order to enhance the gaming aspect of the operation system 10, points may be awarded to the user U.

[0092] 12 is a functional block diagram of a management device 16 according to the sixth embodiment. In the sixth embodiment, the same components as those in the fifth embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

[0093] The calculation unit 52 of the management device 16 includes a history management unit 84 and a point assignment unit 88. The history management unit 84 and the point assignment unit 88 can be realized by the calculation unit 52 executing a program stored in the storage unit 54.

[0094] As described in the fifth embodiment, the history management unit 84 manages the work history of the user U. The work history may be a total time obtained by adding up the time during which the user U operated the work vehicle 14. The work history may be a total distance obtained by adding up the distance over which the user U moved the work vehicle 14. The history management unit 84 stores the total time and the total distance in the memory unit 54.

[0095] The point assigning unit 88 assigns points to the user U according to the total time or total distance stored in the storage unit 54. For example, the point assigning unit 88 assigns a predetermined amount of points each time the total time increases by a certain amount. Alternatively, the point assigning unit 88 assigns a predetermined amount of points each time the total distance increases by a certain amount. The point assigning unit 88 stores the total value of the points in the storage unit 54.

[0096] According to the sixth embodiment, points are awarded to the user U according to the work (total time or total distance), so that the user U can be motivated to perform the work.

[0097] [7 Seventh embodiment] The working vehicle 14 may have a function for avoiding contact with obstacles. For example, the vehicle control unit 46 of the working vehicle 14 may control the working vehicle 14 so as to avoid contact between the working vehicle 14 and obstacles.

[0098] 13 is a functional block diagram of a working vehicle 14 according to a seventh embodiment. In the seventh embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and their description will be omitted. The working vehicle 14 includes an external environment detection unit 90 and an airbag unit 92.

[0099] The external environment detection unit 90 may be configured with an external environment sensor such as a radar or LiDAR. The external environment detection unit 90 detects the distance from the work vehicle 14 to surrounding objects (obstacles, etc.). The external environment detection unit 90 outputs the detected information (referred to as external environment information) to the calculation unit 40.

[0100] The airbag unit 92 may be configured as an airbag module. The airbag module includes an airbag, an inflator that supplies gas to the airbag, and a drive circuit that drives the inflator. For example, one airbag is provided on each of the front, rear, left, and right sides of the working vehicle 14.

[0101] The calculation unit 40 includes a prediction unit 94. The prediction unit 94 can be realized by the calculation unit 40 executing a program stored in the storage unit 42.

[0102] The prediction unit 94 predicts whether the work vehicle 14 will come into contact with an obstacle. For example, the prediction unit 94 calculates the TTC (Time To Collision) until the work vehicle 14 comes into contact with the obstacle based on behavior information and external environment information. The prediction unit 94 predicts that the work vehicle 14 will come into contact with the obstacle if the TTC is below a predetermined first time threshold. Furthermore, the prediction unit 94 determines that contact between the work vehicle 14 and the obstacle is unavoidable if the TTC is below a predetermined second time threshold (<first time threshold).

[0103] When the prediction unit 94 predicts that the work vehicle 14 will come into contact with an obstacle, the mobile unit control unit 46 controls the work vehicle 14 to avoid contact between the work vehicle 14 and the obstacle. For example, the mobile unit control unit 46 may change the traveling direction of the work vehicle 14 or may stop the work vehicle 14.

[0104] When the prediction unit 94 determines that contact between the work vehicle 14 and an obstacle is unavoidable, the mobile unit control unit 46 outputs a drive signal to the airbag unit 92. In response to the drive signal, the airbag unit 92 deploys an airbag.

[0105] According to the seventh embodiment, it is possible to prevent damage to the working vehicle 14. Therefore, the user U can operate the working vehicle 14 with peace of mind.

[0106] [8 Eighth embodiment] The image data and sound data stored in the storage unit 54 of the management device 16 may be acquired through network communication.

[0107] 14 is a schematic configuration diagram of a communication system 100 according to the eighth embodiment. In the eighth embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

[0108] The communication system 100 includes a management device 16 of the work system 10, a server 102, and a communication line 104 such as the Internet. The management device 16 and the server 102 are connected to each other via the communication line 104 so as to enable two-way communication. The server 102 can transmit image data of virtual space images and audio data of sound effects to the management device 16 via the communication line 104.

[0109] In the management device 16, the acquisition unit 56 of the calculation unit 52 can acquire image data and sound data from the server 102 via the communication line 104. The acquisition unit 56 stores the acquired image data and sound data in the storage unit 54. The image data and sound data stored in the storage unit 54 may be updated or newly added.

[0110] According to the eighth embodiment, the user U can experience the latest virtual space and the latest sound effects.

[0111] [9 Other] It is also possible to combine two or more of the second to eighth embodiments.

[0112] In each embodiment, the work system 10 does not have to include the management device 16. In this case, the functions of the calculation unit 52 of the management device 16 are provided in the calculation unit 40 of the work vehicle 14 or the calculation unit 72 of the HMD 18.

[0113] [10 Note] The following additional notes are further disclosed regarding the above embodiment.

[0114] (Appendix 1) The work system (10) of the present disclosure is a work system in which work can be performed by a work vehicle (14) remotely operated by a user (U), and includes an image generation unit (60) that generates a virtual space image (200b) corresponding to the real space around the work vehicle, and a head-mounted display (18) that is worn by the user and provides the user with the virtual space image generated by the image generation unit, and the image generation unit generates the virtual space image according to the position and orientation of the work vehicle.

[0115] According to the configuration of Supplementary Note 1, a user can perform work in real space while experiencing a virtual space. This allows the user to enjoy working with the work vehicle. As a result, the user is encouraged to actively work with the work vehicle, which promotes work. In other words, the configuration of Supplementary Note 1 makes it possible to provide an excellent work system.

[0116] (Appendix 2) In the work system described in Supplementary Note 1, the image generation unit may generate the virtual space image in which the visibility of the work target area is increased according to the work priority level.

[0117] According to the configuration of Supplementary Note 2, by increasing the visibility of areas with high work priority, it is possible to encourage the user to move the work vehicle to areas with high work priority, thereby enabling work in areas with high work priority to be performed preferentially.

[0118] (Appendix 3) The work system described in Appendix 1 may include a determination unit (82) that determines the work status of the work vehicle, and the image generation unit may change the virtual space image according to the work status determined by the determination unit.

[0119] (Appendix 4) The work system described in Appendix 3 may further include a sound generation unit (62) that generates a sound signal, and a sound output unit (70) that provides the user with sound corresponding to the sound signal generated by the sound generation unit, and the sound generation unit may change the sound signal according to the work status determined by the determination unit.

[0120] The configurations of Supplementary Note 3 and Supplementary Note 4 can encourage the user to work quickly, thereby increasing the speed of work using the working vehicle.

[0121] (Appendix 5) In the work system described in Appendix 1, the image generation unit is capable of generating a first virtual space image, which is the virtual space image corresponding to the case where the viewpoint is located on the work mobile body, and a second virtual space image, which is the virtual space image corresponding to the case where the viewpoint is located outside the work mobile body, and when generating the second virtual space image, the image generation unit may set a virtual object corresponding to the work mobile body for the work mobile body in the second virtual space image.

[0122] According to the configuration of Supplementary Note 5, the user can operate a virtual object while viewing the virtual object in a virtual space, which allows the user to enjoy working with the working vehicle more.

[0123] (Appendix 6) In the work system described in Appendix 1, the image generation unit may change the virtual space image according to at least one of a total time obtained by adding up the time the user operates the work vehicle, a total distance obtained by adding up the distance the user moves the work vehicle, work characteristics performed by the user, and operation characteristics performed by the user.

[0124] (Appendix 7) The work system described in Appendix 1 may further include a sound generation unit that generates a sound signal, and a sound output unit that provides the user with sound corresponding to the sound signal generated by the sound generation unit, and the sound generation unit may change the sound signal in accordance with at least one of a total time obtained by adding up the time the user operates the work vehicle, a total distance obtained by adding up the distance the user moves the work vehicle, work characteristics performed by the user, and operation characteristics performed by the user.

[0125] According to the configurations of Supplementary Note 6 and Supplementary Note 7, the virtual space images and sound effects change depending on the skill level of the user, thereby motivating the user to improve their skills.

[0126] (Appendix 8) The work system described in Supplementary Note 1 may further include a point awarding unit (88) that awards points to the user in accordance with the work performed by the work vehicle through the remote control by the user.

[0127] According to the configuration of Supplementary Note 8, points are awarded to the user according to the work, so that the user can be motivated to perform the work.

[0128] (Appendix 9) The work system described in Appendix 1 may further include a prediction unit (94) that predicts whether the work vehicle will come into contact with an obstacle, and a vehicle control unit (46) that controls the work vehicle to avoid contact between the work vehicle and the obstacle when the prediction unit predicts that the work vehicle will come into contact with the obstacle.

[0129] (Appendix 10) In the work system described in Supplementary Note 9, the mobile body control unit may deploy an airbag provided on the work mobile body when contact between the work mobile body and the obstacle cannot be avoided.

[0130] The configurations of Supplementary Note 9 and Supplementary Note 10 can prevent damage to the working vehicle, allowing the user to operate the working vehicle with peace of mind.

[0131] (Appendix 11) The work system described in Appendix 1 may further include a data acquisition unit (56) that acquires image data that can be used when generating the virtual space image via network communication, and the image generation unit may generate the virtual space image using the image data acquired by the data acquisition unit.

[0132] (Appendix 12) The work system described in Appendix 4 or 7 may further include a data acquisition unit that acquires acoustic data that can be used when generating the acoustic signal via network communication, and the acoustic generation unit may generate the acoustic signal using the acoustic data acquired by the data acquisition unit.

[0133] According to the configuration of Supplementary Note 11 and the configuration of Supplementary Note 12, the user can experience the latest virtual space and the latest sound effects.

[0134] Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments. [Explanation of symbols]

[0135] 10...Work system 14...Work vehicle 18...Head-mounted display, HMD 46...Mobile control unit 56...Acquisition unit (data acquisition unit) 60...Image generation unit 62...sound generating unit 70...sound unit (sound output unit) 82...First judgment unit (judgment unit) 88...Point assignment unit 94... Prediction unit 200b... Image (virtual space image)

Claims

1. A work system in which work can be performed by a mobile work object remotely operated by a user, an image generation unit that generates a virtual space image corresponding to the real space around the working vehicle; a head-mounted display that is worn by the user and provides the user with the virtual space image generated by the image generation unit; Equipped with The image generation unit generates the virtual space image according to the position and orientation of the work vehicle.

2. 2. The work system according to claim 1, The image generation unit generates the virtual space image with enhanced visibility of the work area according to the work priority level.

3. 2. The work system according to claim 1, a determination unit for determining a work status of the working vehicle, The image generation unit changes the virtual space image in accordance with the work situation determined by the determination unit.

4. 4. The work system according to claim 3, an audio generation unit that generates an audio signal; a sound output unit that provides the user with sound corresponding to the sound signal generated by the sound generation unit; Further provided with The sound generating unit changes the sound signal depending on the work situation determined by the determining unit.

5. 2. The work system according to claim 1, the image generation unit is capable of generating a first virtual space image, which is the virtual space image corresponding to a case where a viewpoint is located on the work vehicle, and a second virtual space image, which is the virtual space image corresponding to a case where a viewpoint is located outside the work vehicle; A work system in which, when generating the second virtual space image, the image generation unit sets a virtual object corresponding to the work vehicle for the work vehicle in the second virtual space image.

6. 2. The work system according to claim 1, The image generation unit changes the virtual space image according to at least one of the total time that the user operates the work vehicle, the total distance that the user moves the work vehicle, the work characteristics of the user, and the operation characteristics of the user.

7. 2. The work system according to claim 1, an audio generation unit that generates an audio signal; a sound output unit that provides the user with sound corresponding to the sound signal generated by the sound generation unit; Further provided with The sound generation unit changes the sound signal in accordance with at least one of a total time obtained by adding up the times for which the user operates the work vehicle, a total distance obtained by adding up the distances for which the user moves the work vehicle, work characteristics performed by the user, and operation characteristics performed by the user.

8. 2. The work system according to claim 1, The work system further includes a point awarding unit that awards points to the user in accordance with the work performed by the work vehicle through the remote control by the user.

9. 2. The work system according to claim 1, a prediction unit that predicts whether the working vehicle will come into contact with an obstacle; a mobile unit control unit that controls the work vehicle so as to avoid contact between the work vehicle and the obstacle when the prediction unit predicts that the work vehicle will come into contact with the obstacle; The working system further comprises:

10. 10. The work system according to claim 9, The mobile body control unit deploys an airbag provided on the mobile body when contact between the mobile body and the obstacle cannot be avoided.

11. 2. The work system according to claim 1, a data acquisition unit that acquires image data that can be used when generating the virtual space image through network communication; The image generation unit generates the virtual space image using the image data acquired by the data acquisition unit.

12. 8. The work system according to claim 4 or 7, a data acquisition unit that acquires, via network communication, acoustic data that can be used when generating the acoustic signal; The sound generation unit generates the sound signal using the sound data acquired by the data acquisition unit.

Images