Information provision device, information provision method, information provision program, and equipment control system

Abstract

The present invention provides an information provision device, method, and program that provide an operator with information about the equipment being operated without delay. [Solution] In the device control system 1, the information providing device 20 generates a second image, which is predicted to be captured at a second time after the first time, using a model, based on a first image captured at a first time by a camera provided on the control device 30 to be operated, the operation history, and information regarding the communication time between the control device 30 and the terminal 10 that transmits control signals in response to the operation of the operator 2, and provides the second image to the terminal 10.

Description

[Technical Field]

[0001] The present invention relates to an information providing apparatus, an information providing method, an information providing program, and an equipment control system. [Background Art]

[0002] Conventionally, techniques for remotely controlling construction machinery and the like are known. For example, Patent Document 1 describes a technique for reducing delay perceived by an operator by predicting and generating a future control signal in consideration of communication and processing delay time in remote control of equipment. [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] International Publication No. 2023 / 275944 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] However, conventional techniques have a problem that it is difficult to provide an operator with information about the equipment to be operated without delay.

[0005] For example, with the technique described in Patent Document 1, equipment can be operated according to a predicted control signal. On the other hand, a user cannot grasp the latest state of the operating equipment through images or the like.

[0006] The present invention has been made in view of the above, and an object of the present invention is to provide an operator with information about the equipment to be operated without delay. [Means for Solving the Problem]

[0007] To solve the above-mentioned problems and achieve the objective, the information providing device of the present invention is characterized by comprising: a generation unit that generates a second image predicted to be captured at a second time later than the first time, based on a first image captured at a first time by a camera provided on the device to be operated, the operation history of the device, and information regarding the communication time between the device and the terminal, using a model; and a provision unit that provides the second image to the terminal. [Effects of the Invention]

[0008] According to the present invention, information regarding the equipment to be operated can be provided to the operator without delay. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a diagram illustrating the overview of the equipment control system according to the first embodiment. [Figure 2] Figure 2 shows an example of the configuration of a terminal according to the first embodiment. [Figure 3] Figure 3 shows an example of the configuration of an information providing device according to the first embodiment. [Figure 4] Figure 4 shows an example of the configuration of a control device according to the first embodiment. [Figure 5] Figure 5 is a time chart of the equipment control system. [Figure 6] Figure 6 is a sequence diagram showing the processing flow of the equipment control system. [Figure 7] Figure 7 is a flowchart showing the flow of the information provision process of the information provision device. [Figure 8] Figure 8 is a flowchart showing the learning process flow of the information provision device. [Figure 9] Figure 9 shows an example of a computer configuration for running the information provision program. [Modes for carrying out the invention]

[0010] The information provision device, information provision method, information provision program, and equipment control system relating to this application will be described in detail below with reference to the drawings. However, the present invention is not limited to the embodiments described below.

[0011] [First Embodiment] [Configuration and processing of the first embodiment] The configuration of the equipment control system will be explained using Figure 1. Figure 1 is a diagram illustrating the overview of the equipment control system according to the first embodiment.

[0012] As shown in Figure 1, the device control system 1 includes a terminal 10, an information providing device 20, and a control device 30. The terminal 10, the information providing device 20, and the control device 30 are connected to each other by a network.

[0013] Part or all of the network may be an APN (All-Photonics Network). APN is a technology that incorporates optical technology from the network to terminals and even within chips, achieving low power consumption and high-speed processing. For example, with APN, by assigning wavelengths to different functions on a single optical fiber, it becomes possible to provide information communication functions such as the internet, sensing functions, etc., without interfering with each other.

[0014] Terminal 10 transmits a control signal in response to the operator 2's operation. Control device 30 operates the construction machine 40 in response to the control signal. The construction machine 40 is an example of the equipment to be operated. The equipment to be operated is not limited to construction machines, but may include automobiles, machine tools, robots, etc. Construction machines are construction machinery such as excavators and cranes.

[0015] Further, the information providing device 20 provides images relating to the construction machine 40. For example, the information providing device 20 provides an image representing a scene visible when looking at the surroundings from the cockpit of the construction machine 40. The information providing device 20 may provide an image actually captured by a camera, or may provide a generated image. Herein, the case where the information providing device 20 provides a generated image will be particularly described.

[0016] The terminal 10 displays an image provided by the information providing device 20 on a screen. After visually recognizing the image displayed on the terminal 10, the operator 2 can perform the following operations. Accordingly, the operator 2 can perform operations while checking the scene visible from the cockpit of the construction machine 40 even when being located at a position remote from the construction machine 40. That is, the device control system 1 implements remote operation of the construction machine 40.

[0017] The terminal 10 and the construction machine 40 may be located in mutually different regions (e.g., Tokyo and Osaka in Japan) or mutually different countries (e.g., Japan and Brazil). Further, the terminal 10 and the information providing device 20 may be located at a position close to the construction machine 40 (e.g., a building provided at the work site of the construction machine 40). Remote operation not only enables operation of a device from a remote location, but also has the advantage that a device can be operated while avoiding adverse effects on the operator 2 from the work environment (heat, cold, accidents, etc.).

[0018] Here, in remote operation of the construction machine 40, delay occurs due to various factors before information of the construction machine 40 is transmitted to the operator 2. For example, consider a case where an image (a still image or a moving image) captured by a camera provided on the construction machine 40, which has been operationally controlled based on a control signal, is transmitted to the terminal 10.

[0019] In this case, first, a mechanical delay occurs between the time the control signal is received and the construction machine 40 actually starts moving. This mechanical delay is, for example, the time required for the hydraulic pressure necessary to operate a hydraulic arm to reach a sufficient level. Additionally, there is an information processing delay required for image encoding, decoding, etc. Furthermore, there is a transmission line delay for sending image data. In reality, it is extremely difficult to completely eliminate these delays.

[0020] In response, the information providing device 20 predicts and generates images captured by the camera installed on the construction machine 40. That is, the information providing device 20 can generate and provide future images. As a result, the operator 2 can obtain images of feedback to their operations with low latency.

[0021] The configuration and operation of each device in the equipment control system 1 will be described in detail below.

[0022] The configuration of terminal 10 will be explained using Figure 2. Figure 2 is a diagram showing an example of the configuration of a terminal according to the first embodiment. Terminal 10 may be a PC, a smartphone, or the like.

[0023] As shown in Figure 2, the terminal 10 includes a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, and a control unit 15.

[0024] The communication unit 11 is an interface for communicating with other devices. For example, the communication unit 11 is a NIC (Network Interface Card).

[0025] The input unit 12 is an interface for receiving data input. For example, the input unit 12 is connected to an input device such as a keyboard, mouse, or touch panel. Alternatively, the input unit 12 may be connected to an input device that mimics the operating equipment (levers, etc.) of the construction machinery 40. The content of the operation input to the input unit 12 is transmitted as a control signal by the communication unit 11 to the information providing device 20 and the control device 30.

[0026] The output unit 13 is an interface for outputting data. For example, the output unit 13 is connected to output devices such as a display and a speaker.

[0027] The storage unit 14 is a storage device such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or optical disc. Alternatively, the storage unit 14 may be a rewritable semiconductor memory such as RAM (Random Access Memory), flash memory, or NVSRAM (Non-Volatile Static Random Access Memory). The storage unit 14 stores the OS (Operating System) and various programs executed on the terminal 10.

[0028] The control unit 15 controls the entire terminal 10. The control unit 15 is, for example, an electronic circuit such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or GPU, or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0029] The control unit 15 has internal memory for storing programs and control data that define various processing procedures, and executes each process using the internal memory. In addition, the control unit 15 functions as various processing units when various programs are run.

[0030] The control unit 15 includes a display control unit 151. The display control unit 151 displays an image. For example, the display control unit 151 outputs an image obtained by appropriately processing the received image data, such as decoding, to the output unit 13.

[0031] The configuration of the information providing device 20 will be explained using Figure 3. Figure 3 is a diagram showing an example of the configuration of the information providing device according to the first embodiment. The information providing device 20 may be a server.

[0032] As shown in Figure 3, the information providing device 20 includes a communication unit 21, a storage unit 24, and a control unit 25.

[0033] The communication unit 21 is an interface for communicating with other devices. For example, the communication unit 21 is a network interface card (NIC).

[0034] The storage unit 24 is a storage device such as an HDD, SSD, or optical disc. Alternatively, the storage unit 24 may be a rewritable semiconductor memory such as RAM, flash memory, or NVSRAM. The storage unit 24 stores the OS and various programs executed by the information providing device 20. The storage unit 24 also stores image information 241 and model information 242.

[0035] Image information 241 is an image received by the information providing device 20 from the control device 30. That is, the information providing device 20 stores the image received from the control device 30 as image information 241 in the storage unit 24. The information providing device 20 receives images actually captured by the camera from the control device 30.

[0036] Model information 242 consists of parameters for building the model. The model may be a machine learning model such as a neural network. For example, model information 242 includes the weights and biases of the neural network. In the learning process described later, model information 242 is updated.

[0037] The control unit 25 controls the entire information provision device 20. The control unit 25 is, for example, an electronic circuit such as a CPU, MPU, or GPU, or an integrated circuit such as an ASIC or FPGA.

[0038] The control unit 25 has internal memory for storing programs and control data that define various processing procedures, and executes each process using the internal memory. In addition, the control unit 25 functions as various processing units when various programs are run.

[0039] The control unit 25 includes a generation unit 251, a provision unit 252, and an update unit 253. The generation unit 251 generates predicted video using a model constructed based on model information 242 (inference processing). The provision unit 252 provides the generated predicted video to the terminal 10. The update unit 253 updates the model parameters based on the predicted video (learning processing).

[0040] The configuration of the control device 30 will be explained using Figure 4. Figure 4 is a diagram showing an example of the configuration of the control device according to the first embodiment. The control device 30 is a computer installed in the construction machine 40.

[0041] As shown in Figure 4, the control device 30 includes a communication unit 31, a camera 32, a sensor 33, a storage unit 34, and a control unit 35.

[0042] The communication unit 31 is an interface for communicating with other devices. For example, the communication unit 31 is a network interface card (NIC).

[0043] Camera 32 captures images. For example, camera 32 is installed in the cockpit of construction equipment 40 and captures images that simulate the view an operator would see if they were actually sitting in the cockpit. Camera 32 may also capture 180-degree panoramic images or 360-degree images.

[0044] Sensor 33 is a group of sensors including one or more sensors. Sensor 33 may include, for example, a sound sensor (microphone), an acceleration sensor, a gyroscope sensor, a temperature sensor, a humidity sensor, a barometric pressure sensor, etc. Sensor 33 may also include optical sensors other than those in camera 32 (e.g., an infrared sensor, an illuminance sensor).

[0045] The storage unit 34 is a storage device such as an HDD, SSD, or optical disc. Alternatively, the storage unit 34 may be a rewritable semiconductor memory such as RAM, flash memory, or NVSRAM. The storage unit 34 stores the OS and various programs executed by the control device 30.

[0046] The control unit 35 controls the entire control device 30. The control unit 35 is, for example, an electronic circuit such as a CPU, MPU, or GPU, or an integrated circuit such as an ASIC or FPGA.

[0047] The control unit 35 has internal memory for storing programs and control data that define various processing procedures, and executes each process using the internal memory. In addition, the control unit 35 functions as various processing units when various programs are run.

[0048] The control unit 35 includes an operation control unit 36. The operation control unit 36 ​​operates various parts of the construction machine 40 based on the received control signals. For example, the operation control unit 36 ​​transmits a drive signal to an actuator provided in the construction machine 40 based on the control signals.

[0049] Using Figure 5, we will explain the events that occur during the remote operation of equipment realized by the equipment control system 1, following the flow of time. Figure 5 is a time chart of the equipment control system.

[0050] The learning and inference processes shown in Figure 5 are performed by the information providing device 20. Operator 2 operates via terminal 10. Images are displayed on the monitor via terminal 10. The construction machine 40, which is the target of the operation, and the control device 30 equipped with a camera 32 are located at the site.

[0051] Time progresses in the order of time t1, time t2, time t3, and time t4. That is, time t2 is a later time than time t1. In other words, time t2 is a future time than time t1.

[0052] At time t1, the image actually captured by camera 32 (hereinafter referred to as "actual image") is transmitted to information providing device 20. The transmitted image is received by information providing device 20 at time t2 and used for learning and inference processing. Note that time t2-t1 is part of the delay time.

[0053] At time t2, operator 2 performs an operation. A control signal based on this operation is transmitted from terminal 10 to information providing device 20. Here, the delay time caused by communication between terminal 10 and information providing device 20 is ignored.

[0054] In the inference process, the information provider 20 generates a predicted image based on information including the actual image and control signals. The predicted image is then provided to the terminal 10 at time t3. In the learning process, the actual image and control signals are used as training data.

[0055] Here, if the predicted image is a prediction of the actual image at time t3 or a time later than time t3, operator 2 can virtually view the current or future actual image without delay. Then, operator 2 can perform the following operations at time t4.

[0056] Furthermore, during the learning process, the model is updated to make the predicted image closer to the actual image, thus improving the accuracy of the predicted image.

[0057] Figure 6 will be used to explain the processing flow of the equipment control system 1. Figure 6 is a sequence diagram showing the processing flow of the equipment control system. Here, the information providing device 20 generates a predicted image. The camera 32 captures the actual image. An image can be described as a collection of multiple images (frames).

[0058] As shown in Figure 6, operator 2 inputs an operation to terminal 10 (step S101). Terminal 10 transmits a control signal corresponding to the operation to the information providing device 20 and the control device 30 (step S102). The control device 30 controls the operation of the construction machine 40 based on the control signal (step S103).

[0059] Furthermore, the control device 30 transmits the actual video footage captured by the camera 32 and related information to the information providing device 20 (step S104).

[0060] Here, the model includes a multimodal LLM (Large Language Model) and a prediction engine. The multimodal LLM creates input data for generating predicted video based on real video and related information, and requests the prediction engine to generate the predicted video (step S105). The prediction engine generates predicted video based on the input data (step S106) and outputs it (step S107).

[0061] The information providing device 20 transmits the predicted video to the terminal 10 (step S108). The terminal 10 displays the received predicted video (step S109).

[0062] Furthermore, operator 2, after reviewing the predicted video, inputs an operation into terminal 10 (step S110). Terminal 10 transmits control signals corresponding to the operation to information provision device 20 and control device 30 (step S111). Control device 30 controls the operation of construction machine 40 based on the control signals (step S112).

[0063] Furthermore, the control device 30 transmits the actual video captured by the camera 32 to the information providing device 20 (step S113). The information providing device 20 performs a learning process (step S114) and updates the model (step S115).

[0064] Here, Figure 6 shows the processes for operating construction machinery (for example, the set of steps S101, S102, and S103), the processes for providing predicted images (for example, the set of steps S104, S105, S106, S107, S108, and S109), and the processes related to learning (for example, the set of steps S113, S114, and S115). The timing at which these processes are performed is not limited to that shown in Figure 6. These processes may be performed asynchronously with respect to each other. In addition, the information providing device 20 may perform the learning process using the actual image transmitted in, for example, S104.

[0065] The following describes the related information. The related information includes at least the operation history of the construction machine 40 and information regarding the communication time between the construction machine 40 and the terminal 10.

[0066] For example, the operation history of the construction machine 40 is a log of operations entered into the terminal 10 up to a certain time, or a set of control signals based on those operations. For example, the operation history includes the type of operation and the amount of operation. The types of operations include moving the construction machine 40 forward, backward, and rotating, and raising and lowering the arm provided on the construction machine 40.

[0067] The operation history of the construction machine 40 is expected to be useful in predicting the state of the construction machine 40 at a future point in time when generating predictive video.

[0068] Information regarding the communication time between the construction machine 40 and the terminal 10 is, for example, the measured value of the communication between the control device 30 (construction machine 40) and the terminal 10. Information regarding the communication time may also be a value predicted in advance as a standard communication time between the control device 30 and the terminal 10.

[0069] Information regarding the communication time between the construction machine 40 and the terminal 10 is expected to be effective in smoothing the time intervals of the generated predicted video. For example, the model is designed to generate predicted video such that the flow of time perceived by operator 2 when viewing the predicted video is as constant as possible.

[0070] For example, let's assume that the communication time between the construction machine 40 and the terminal 10 is predicted to be approximately 20 ms (milliseconds). Also, let's assume that the information providing device 20 is located near the terminal 10, and the communication time between the information providing device 20 and the terminal 10 is similarly approximately 20 ms. Furthermore, let's assume that the actual measured communication time fluctuates within the range of 18 ms to 22 ms.

[0071] In this case, the information providing device 20 generates a predicted image 50ms after the actual image received from the control device 30. If time other than communication time is ignored, operator 2 will see a predicted image 30ms after the current time (50ms after the actual image capture time - 20ms for communication time).

[0072] On the other hand, suppose a delay occurs in communication, and the measured communication time is 40ms. In this case, the information providing device 20 generates a predicted image 70ms after the actual image received from the control device 30, rather than 50ms after the actual image. As a result, even if a delay occurs, operator 2 can see a predicted image 30ms after the current time (70ms after the actual image capture time - 40ms communication time).

[0073] The generation of such predictive images can be achieved by training a model to generate images in the future for a time period corresponding to the actual image capture time and the measured input communication time. Furthermore, the actual images used as training data necessary for such training are continuously captured by camera 32.

[0074] Furthermore, the related information may include sensor values ​​acquired by sensor 33 and calculated values ​​based on those sensor values. For example, the related information may include the tilt of the construction machine 40 acquired by the gyro sensor. This allows the information providing device 20 to generate a predicted image based on the tilt. For example, if the construction machine 40 is tilted in a specific direction on a slope at the time the actual image is captured, it is predicted that it will slide in that direction at a future time.

[0075] Figure 7 illustrates the flow of the information provision process of the information provision device 20. Figure 7 is a flowchart showing the flow of the information provision process of the information provision device.

[0076] First, the generation unit 251 receives a control signal from the terminal 10 (step S201). The generation unit 251 also receives the actual video and related information from the control device 30 (step S202).

[0077] The generation unit 251 creates input information based on the operation history and related information based on the control signals (step S203). The generation unit 251 also generates predicted video using a prediction engine based on the actual video and the input information (step S204).

[0078] The input information is generated by a multimodal LLM. The input information is data that shows the operation history based on control signals and related information, and is in a format that can be input to the prediction engine.

[0079] The actual image shown here is an example of the first image captured at the first time point. The predicted image is an example of the second image that is predicted to be captured at the second time point, which is later than the first time point.

[0080] The data provider 252 transmits the predicted video to the terminal (step S205).

[0081] Figure 8 will be used to explain the learning process flow of the information providing device 20. Figure 8 is a flowchart showing the learning process flow of the information providing device.

[0082] As shown in Figure 8, first, the update unit 253 receives the actual video (step S301). Next, the update unit 253 calculates the difference between the actual video and the predicted video (step S302). The actual video here is an example of a third image captured by the camera at a second time point.

[0083] Then, the update unit 253 updates the parameters of the prediction engine so that the difference becomes smaller (step S303).

[0084] [Effects of the first embodiment] As explained above, the generation unit 251 generates a second image, which is predicted to be captured at a second time after the first time, based on a first image captured at a first time by a camera installed in the device being operated, the operation history of the device, and information regarding the communication time between the device and the terminal, using a model. The providing unit then provides the second image to the terminal.

[0085] As a result, the information providing device 20 can provide operator 2 with information (second image) about the equipment being operated without delay. In addition, operator 2 can avoid the poor tactile experience caused by delays and move on to the next task quickly without waiting for the actual video.

[0086] The update unit 253 updates the model parameters so that the difference between the third image captured by the camera at the second time point and the second image becomes smaller. The third image, which is the actual video, is always available regardless of whether the second image, which is the predicted video, is generated or not. Therefore, the information providing device 20 can obtain sufficient training data and improve the accuracy of the model through learning.

[0087] The generation unit 251 generates a second image based on the measured communication time between the device and the terminal. This allows the information providing device 20 to generate a predictive image that makes the operator 2 perceive the passage of time as constant as possible.

[0088] The generation unit 251 further generates a second image based on the tilt of the device. This makes it easier to predict the direction of movement of the device, thereby improving the accuracy of the predicted image.

[0089] [System configuration, etc.] Each component of the illustrated device is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed or integrated in any unit according to various loads and usage conditions. Furthermore, each processing function performed by each device can be implemented, all or any part of it, by a CPU and a program that is analyzed and executed by that CPU, or by hardware using wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

[0090] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[0091] [program] In one embodiment, the information providing device 20 can be implemented by installing a program that performs the above processing as packaged software or online software on a desired computer. For example, by having the above program run on an information processing device, the information processing device can function as the information providing device 20. The information processing device referred to here includes desktop or notebook personal computers. In addition, mobile communication terminals such as tablet terminals and smartphones are also included in the category of information processing devices.

[0092] Furthermore, the information providing device 20 may be implemented as a web server, or it may be implemented as a cloud service that provides the above-mentioned processing services through outsourcing.

[0093] Figure 9 shows an example configuration of a computer running an information provision program. Computer 1000 has, for example, memory 1010 and a CPU 1020. Computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0094] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM (Random Access Memory) 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0095] The hard disk drive 1090 stores, for example, the OS 1091, application programs 1092, program modules 1093, and program data 1094. That is, the programs that define each process of the information providing device 20 are implemented as program modules 1093 in which executable code for a computer is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, a program module 1093 for performing processes similar to the functional configuration of the information providing device 20 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0096] Furthermore, the configuration data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes the processing of the above-described embodiment.

[0097] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070. [Explanation of Symbols]

[0098] 1. Equipment control system 2 Operators 10 devices 11, 21, 31 Communications Department 12 Input section 13 Output section 14, 24, 34 storage section 15, 25, 35 Control Unit 20 Information provision device 30 Control device 32 cameras 33 sensors 36 Operation Control Unit 40 Construction Machinery 151 Display Control Unit 241 Image Information 242 Model Information 251 Generation part 252 Provision Department 253 Update Department

Claims

1. A generation unit generates a second image, which is predicted to be captured at a second time after the first time, based on a first image captured at a first time by a camera installed on the device to be operated, the operation history of the device, and information regarding the communication time between the device and the terminal, using a model. A providing unit that provides the second image to the terminal, An information providing device characterized by having the following features.

2. Update unit updates the model parameters so that the difference between the third image captured by the camera at the second time and the second image becomes smaller. The information providing device according to claim 1, further comprising the above.

3. The generation unit generates the second image based on the measured value of the communication time between the device and the terminal. The information providing device according to feature 1.

4. The generation unit further generates the second image based on the tilt of the device. The information providing device according to feature 1.

5. A method of providing information performed by a computer, A generation step of generating a second image, which is predicted to be captured at a second time after the first time, based on a first image captured at a first time by a camera installed on the device to be operated, the operation history of the device, and information regarding the communication time between the device and the terminal, using a model; A provision step of providing the second image to the terminal, A method of providing information characterized by including the following.

6. A generation step of generating a second image, which is predicted to be captured at a second time after the first time, based on a first image captured at a first time by a camera installed on the device to be operated, the operation history of the device, and information regarding the communication time between the device and the terminal, using a model; A provision step of providing the second image to the terminal, An information provision program characterized by causing a computer to execute a command.

7. A device control system comprising a terminal, a control device provided in the device, and an information providing device that provides the terminal with information about the device, The aforementioned terminal is The system transmits control signals based on operations entered by the user to the information providing device and the control device. The image received from the aforementioned information providing device is displayed. The aforementioned information providing device is A model is used to generate a second image that is predicted to be captured at a second time, later than the first time, based on a first image captured at a first time by a camera installed in the device, a history of the operation of the device based on the control signal, and information regarding the communication time between the device and the terminal. The second image is transmitted to the terminal. The control device is The operation of the equipment is controlled based on the aforementioned control signal. A device control system characterized by the following features.

Images