Information processing system, information processing device, and information processing method

Abstract

An information processing system according to an embodiment comprises: an operation status recognition unit that recognizes an operation status from an input operation of an operator in an operation environment; a work status recognition unit that recognizes a work status from an image pertaining to operation target work in a work environment; and a display image generation unit that generates a display image for operation according to the operation status and the work status.

Description

Information processing system, information processing device, and information processing method

[0001] The present disclosure relates to an information processing system, an information processing device, and an information processing method.

[0002] In recent years, information processing systems such as remote operation systems and remote operation support systems have been developed (see, for example, Patent Document 1). In such information processing systems, for example, an operator may operate equipment (such as construction machinery) located at a remote location while viewing a display image.

[0003] International Publication No. 2023 / 074148

[0004] However, when an operator operates a device in a remote location, there may be areas that require confirmation and areas that do not require confirmation depending on the operation situation. In such cases, the operator may select or change, for example, the optimal angle of view or resolution. However, it is difficult to select or change the angle of view or resolution in real time, which reduces work efficiency.

[0005] Therefore, the present disclosure provides an information processing system, an information processing device, and an information processing method that can improve work efficiency.

[0006] The information processing system according to the embodiment includes an operation status recognition unit that recognizes an operation status from an input operation by an operator in an operation environment, a work status recognition unit that recognizes a work status from an image relating to the work of an operation target in a work environment, and a display image generation unit that generates a display image according to the operation status and the work status.

[0007] The information processing device according to the embodiment includes an operation status recognition unit that recognizes an operation status from an input operation by an operator in an operation environment, a receiving unit that receives a work status in a work environment, and a display image generation unit that generates a display image according to the operation status and the work status.

[0008] The information processing method according to the embodiment includes a computer recognizing an operation status from an input operation by an operator in an operation environment, recognizing a work status from an image relating to the work of an operation target in a work environment, and generating a display image according to the operation status and the work status.

[0009] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. FIG. 2 is a first diagram illustrating an example of the processing of image generation according to an embodiment. FIG. 3 is a second diagram illustrating an example of the processing of image generation according to an embodiment. FIG. 4 is a fourth diagram illustrating an example of the processing of image generation according to an embodiment. FIG. 1 is a first diagram illustrating an example of the configuration of setting information according to an embodiment. FIG. 2 is a second diagram illustrating an example of the configuration of setting information according to an embodiment. FIG. 3 is a third diagram illustrating an example of the configuration of setting information according to an embodiment. FIG. 1 is a first diagram illustrating a flowchart of an example of the processing of image generation according to an embodiment. FIG. 2 is a second diagram illustrating a flowchart of an example of the processing of image generation according to an embodiment. FIG. 3 is a diagram illustrating an example of the configuration of hardware according to an embodiment.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The embodiments include examples and modifications. Note that the technology according to the present disclosure is not limited to the embodiments. Furthermore, in the embodiments, essentially identical components are designated by the same reference numerals, and redundant explanations will be omitted.

[0011] The present disclosure will be described in the following order: 1. Embodiment 1-1. Configuration example of information processing system 1-2. Processing example of image generation 1-3. Configuration example of setting information 1-4. Processing example flow of image generation 1-5. Actions and effects 2. Other embodiments 3. Configuration example of hardware 4. Supplementary notes

[0012] <1. Embodiment> <1-1. Configuration Example of Information Processing System> An example of the configuration of an information processing system 1 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing system 1 according to an embodiment. The information processing system 1 is an example of a remote control system.

[0013] (Information Processing System) As shown in Fig. 1, the information processing system 1 includes an operation device 10 on the operation environment side and a work device 20 on the work environment side. The operation device 10 and the work device 20 are configured to be able to communicate with each other via, for example, a network 30. The information processing system 1 is divided into an operation environment in which an operator exists and a work environment in which an operation target exists. The operator operates the operation device 10 to move the work device 20 (for example, construction machine 22a) that is the operation target in a remote location.

[0014] The network 30 is, for example, a communication network such as a local area network (LAN), a wide area network (WAN), a cellular network, a fixed telephone network, a regional Internet Protocol (IP) network, or the Internet. The network 30 may be configured as both a wired network and a wireless network, or one of both. The network 30 may also include a core network or a data network other than a core network. The data network may be a private network such as an in-house network.

[0015] (Operation Device) The operation device 10 includes an input device 11 , an output device 12 , and an information processing device 13 .

[0016] The input device 11 includes a controller 11a. The controller 11a accepts input operations from an operator. The controller 11a may be, for example, a lever, a handle, a switch, a button, a keyboard, or a mouse. The controller 11a may be, for example, a construction machine controller, but may also be a device capable of acquiring three-dimensional position and orientation, such as a virtual reality (VR) controller or motion capture. The controller 11a may also be a device that uses hand tracking (hand gestures).

[0017] The output device 12 includes a display 12 a. The display 12 a displays various images. The display 12 a may be, for example, a liquid crystal display, an organic electroluminescence (EL) display, or a head-mounted display (HMD).

[0018] The information processing device 13 includes an operation status recognition unit 13 a, a UI generation unit 13 b, a construction machine control unit (control unit) 13 c, a data transmission / reception unit 13 d, a storage unit 13 e, a communication status detection unit 13 f, and a display image generation unit 13 g. The UI generation unit 13 b is an example of a generation unit.

[0019] The operation status recognition unit 13a recognizes the operation status (e.g., movement amount, movement direction, etc.) from the input operation of the operator to the controller 11a of the input device 11. For example, the operation status recognition unit 13a detects the operation status based on the input value of the controller 11a, i.e., the controller input value.

[0020] The UI generation unit 13b generates an appropriate UI (interface) depending on the operation situation. The UI includes, for example, display rules. The display rules include, for example, display layout information, zoom information, highlighting information, resolution information, FPS (frame rate) information, etc. The display layout information includes, for example, information such as layout rules that define the arrangement of various images.

[0021] The construction machine control unit 13c controls the output device 22 of the work device 20, i.e., the construction machine 22a, in response to an input operation by the operator to the controller 11a of the input device 11. For example, the construction machine control unit 13c controls the construction machine 22a by sending a control signal to the construction machine 22a via the data transmission / reception unit 13d based on the operation status (e.g., the amount of movement, the direction of movement, etc.).

[0022] The data transmitter / receiver 13d performs data communication between the operation device 10 and the working device 20. For example, the data transmitter / receiver 13d is a communication interface for the operation device 10 to communicate with the working device 20. The data transmitter / receiver 13d may be, for example, a wired interface or a wireless interface.

[0023] The storage unit 13e stores various types of information, and may be, for example, a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, or a hard disk.

[0024] The communication status detection unit 13f detects the communication status between the operation device 10 and the maintenance device 20. For example, the communication status detection unit 13f checks whether data conforming to the UI generated by the UI generation unit 13b has been received.

[0025] The display image generation unit 13g generates a display image to be provided to the operator based on the UI generated by the UI generation unit 13b and the data transmitted from the operation apparatus 20, and transmits the generated display image to the display 12a of the output device 12. The display 12a displays the display image transmitted from the display image generation unit 13g.

[0026] (Working Device) The working device 20 includes an input device 21 , an output device 22 , and an information processing device 23 .

[0027] The input device 21 includes a sensor 21 a and a camera 21 b. The sensor 21 a and the camera 21 b are installed in the work environment so that an operator can check the work environment in a remote location. The camera 21 b is an example of an imaging unit.

[0028] The sensor 21a detects, for example, the weather and people in the work environment, etc. Examples of the sensor 21a include a rain gauge, a hygrometer, a thermometer, an infrared sensor, and a camera.

[0029] The camera 21b captures images of the work environment (for example, images including the construction machine 22a and its surroundings). Multiple cameras 21b may be provided as needed. As the camera 21b, for example, a camera that captures video and / or still images may be used.

[0030] The sensor 21a and the camera 21b are used to detect the work environment, but other detection methods may also be used, such as tilt detection using an IMU (Inertial Measurement Unit), human detection within the work space using LiDAR (Light Detection and Ranging), and human detection and detection of falling earth and sand or equipment using a microphone.

[0031] The output device 22 includes, for example, a construction machine 22a. Examples of the construction machine 22a include a hydraulic excavator (such as a shovel, a backhoe, or a drag shovel), a crane, a bulldozer, and a wheel loader.

[0032] The information processing device 23 includes a work situation recognition unit 23a, a data selection unit 23b, and a data transmission / reception unit 23c.

[0033] The work situation recognition unit 23a recognizes the work situation including the construction machine 22a that is the equipment performing the work, based on information from the input device 21 (for example, the sensor 21a, the camera 21b, etc.). The work situation includes, for example, the state of the construction machine 22a to be operated and the weather (work environment).

[0034] The data selection unit 23b selects appropriate transmission data from various data based on the work situation recognized by the work situation recognition unit 23a and data corresponding to the work situation, etc. Then, the data selection unit 23b transmits the selected transmission data to the operation device 10 via the data transmission / reception unit 23c.

[0035] The data transmitter / receiver 23c performs data communication between the operation device 10 and the working device 20. For example, the data transmitter / receiver 23c is a communication interface that enables the working device 20 to communicate with the operation device 10. The data transmitter / receiver 23c may be, for example, a wired interface or a wireless interface.

[0036] The information processing device 13 or 23 may be configured, for example, by both or either of hardware and software, and the configuration is not particularly limited. The information processing device 13 or 23 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a CPLD (Complex Programmable Logic Device). Furthermore, the information processing device 13 or 23 may be realized by a computer such as a CPU (Central Processing Unit), an MCU (Micro Controller Unit), or an MPU (Micro Processing Unit) executing a program stored in a ROM (Read Only Memory) using a RAM (Random Access Memory) as a work area.

[0037] <1-2. Example of Image Generation Processing> An example of image generation processing according to the embodiment will be described with reference to Fig. 2 to Fig. 5. Each of Fig. 2 to Fig. 5 is a diagram for explaining an example of image generation processing according to the embodiment.

[0038] The example in Figure 2 shows a case where no problems occur in either the work environment or the communication situation (an example of a UI change when no problems occur). In addition to the overall image, a different angle image (high resolution) of the excavation site is transmitted from the work environment to the operation environment. The resolution of the different angle image is higher than that of the overall image. Information combining the overall image and the different angle image is displayed in the operation environment, thereby presenting the information required by the operator.

[0039] As shown in Fig. 2, a pre-altered image 100 is displayed on the display 12a of the operation device 10. The operator visually checks the pre-altered image 100 displayed on the display 12a and operates the controller 11a of the operation device 10 to move the construction machine 22a of the work device 20. In the example of Fig. 2, in operation content 110, the operator moves the lever of the controller 11a in small increments (small operation amount). For example, an overall image (the entire environment visible from the operator's seat of the construction machine 22a) is displayed on the display 12a as the pre-altered image 100. The overall image includes the arm 101 and bucket 102 of the construction machine 22a.

[0040] At this time, the operation situation recognition unit 13a of the operation device 10 estimates and recognizes the operator's work content and necessary information as the operation situation in the operator's operation situation (operation environment) 120 from the operation content of the operator's input operation on the controller 11a. For example, the operation situation recognition unit 13a estimates the operation situation by selecting the operation situation from pre-registered operation candidates using the amount of movement and the direction of movement of the controller 11a. Note that the estimation method is not particularly limited, and for example, the operation situation recognition unit 13a may estimate the operation situation from the flow of past operations.

[0041] 2 , in the operation status (operation environment) 120, the operation status is estimated to be “Work content: adjusting excavation location, required information: detailed image of excavation location.” The detailed image of the excavation location is, for example, the detailed image within frame 121. Note that the UI generation unit 13b of the operation device 10 generates a UI (display rule) based on the operation status, and requests data corresponding to the generated UI from the operation device 20.

[0042] The work situation recognition unit 23a of the work device 20 estimates and recognizes the work situation of the construction machine 22a based on information from the sensor 21a and camera 21b of the input device 21 in the work situation (work environment) 130. For example, the work situation recognition unit 23a estimates the work situation by selecting a work situation from pre-registered work candidates using the detection results of the sensor 21a, the image from the camera 21b, the amount of movement and direction of movement of the controller 11a, etc. Note that the estimation method is not particularly limited, and for example, the work situation recognition unit 23a may estimate the work situation from the flow of past work.

[0043] In the example of Figure 2, in work status (work environment) 130, the work status is estimated as "Work environment: no problem (excavation possible as instructed), communication status: no problem (multiple camera data can be transmitted)." Data selection unit 23b of work device 20 selects data corresponding to the estimated work status and transmits it to operation device 10. Note that if the UI needs to be updated in accordance with the data transmitted from work device 20, UI generation unit 13b of operation device 10 updates and generates the UI.

[0044] The display image generating unit 13g of the operation device 10 generates a display image, i.e., an altered image 140, using data transmitted from the work device 20 based on the UI generated by the UI generating unit 13b. The altered image 140 is displayed on the display 12a of the operation device 10. The altered image 140 includes an overall image and an image (high image quality) of the excavation site from a different angle. The display rules (e.g., display layout, resolution, etc.) of the altered image 140 are defined by the UI.

[0045] The communication status detection unit 13f of the operation device 10 detects whether the data transmitted from the operation device 20 is the requested data (desired data). At this time, the communication status detection unit 13f checks whether the requested data was received based on two items: the work environment and the communication status. The work environment may include, for example, information regarding factors that hinder the intended work in the work environment. The communication status may include, for example, information regarding factors that hinder data transmission in communication.

[0046] 3 to 5 show cases where a problem occurs in either the work environment or the communication situation (examples of UI changes when a problem occurs). On the work environment side, the operation device 20 selects data according to the problem that has occurred and information required by the operator, and transmits the selected data to the operation device 10. On the operation environment side, the operation device 10 detects that a problem has occurred from the received data, and presents information that indicates the occurrence of the problem.

[0047] As shown in FIG. 3 , in work status (1) 130A, the work status is estimated as follows: "Work environment: No problem (excavation possible as instructed), communication status: Large delay (limited data volume that can be transmitted and received)." Therefore, the modified image 140A includes an overall image (low image quality) and an excavation site image 141a (high image quality). For example, blurring can be used to reduce the image quality of the overall image. The display rules (e.g., display layout, resolution, etc.) for the modified image 140A are specified by the UI. With this display, even in poor communication conditions, at least the excavation site image 141a is displayed in high image quality. This allows the operator to view the excavation site image 141a, allowing them to continue working.

[0048] As shown in FIG. 4 , in work status (2) 130B, the work status is estimated as "Work environment: Poor visibility has made it impossible to detect the excavation location; Communication status: No problem." Therefore, the changed image 140B is a full image. In other words, the full image is maintained. The display rules for the changed image 140B are stipulated by the UI. The changed image 140B includes an unclear area 141b due to fog or mist. With this display, even in poor visibility, the full image is maintained, allowing the operator to continue working.

[0049] As shown in FIG. 5 , in work situation (3) 130C, the work situation is estimated as "Work environment: people present, communication situation: no problems." Therefore, post-change image 140C includes an entire image and a person highlighting 141c. For example, a colored frame or a blinking display is used to highlight the person. The display rules for post-change image 140C are defined by the UI. With this display, people are highlighted, allowing the operator to continue or interrupt work while paying attention to the people.

[0050] 1-3. Configuration Examples of Setting Information> Configuration examples of the setting information 200, 210, and 220 according to the embodiment will be described with reference to Fig. 6 to Fig. 8. Fig. 6 and Fig. 7 are diagrams showing configuration examples of the setting information 200 or the setting information 210 according to the embodiment. Fig. 8 is a diagram showing a configuration example of the setting information 220 according to the embodiment.

[0051] As shown in FIG. 6 , the setting information 200 includes information such as "operation content," "detection method," "expected situation (operation situation)," "necessary information," and "presented UI." Each "operation content" is associated with a "detection method," "expected situation (operation situation)," "necessary information," and "presented UI." For example, the setting information 200 is stored in the storage unit 13e and used by the operation situation recognition unit 13a, the UI generation unit 13b, and the like. The "necessary information" in FIG. 6 is an example of first information.

[0052] "Operation content" includes, for example, operation content such as "fast aircraft rotation," "slow aircraft rotation," "right rotation," and "left rotation." "Detection method" includes, for example, detection methods such as "controller." "Expected situation (operation situation)" includes, for example, operation situations such as "rough direction adjustment" and "fine direction adjustment." "Required information" includes, for example, information such as "wide-range information," "high-resolution information, target point information." "Presented UI" includes, for example, information such as "low-resolution wide-range display," "high-resolution target range display, alignment auxiliary line display."

[0053] As shown in FIG. 7 , the setting information 210 includes information such as "operation content," "detection method," "expected situation (operation situation)," "necessary information," and "presented UI," similar to the setting information 200 (see FIG. 6 ). Each "operation content" is associated with a "detection method," "expected situation (operation situation)," "necessary information," and "presented UI." For example, similar to the setting information 200 (see FIG. 6 ), the setting information 200 is stored in the storage unit 13 e and used by the operation situation recognition unit 13 a and the UI generation unit 13 b. The "necessary information" in FIG. 7 is an example of first information.

[0054] "Operation content" includes, for example, operation content such as "moving the arm quickly" or "moving the arm widely and quickly to excavate." "Detection method" includes, for example, detection methods such as "controller" or "collision detection." "Expected situation (operation situation)" includes, for example, operation situations such as "rough alignment" or "excavation work on a wide area." "Necessary information" includes, for example, information such as "wide-area information, detailed information around the arm," or "wide-area information." "Presented UI" includes, for example, information such as "bird's-eye view video display, high-resolution / high FPS (frame rate) display of the area around the arm, or enlarged display from a different angle," or "high-resolution / high FPS bird's-eye view video display."

[0055] The operation device 10 determines a UI (display rule) to be presented to the operator based on the operation content of the input operation on the operation environment side. The operation situation recognition unit 13a of the operation device 10 recognizes the operation content of the operator on the controller 11a using a detection method based on the setting information 210, and estimates an expected situation (operation situation). The UI generation unit 13b determines necessary information according to the expected situation and a UI to be presented to the operator based on the setting information 210. The necessary information is, for example, information required by the operator according to the expected situation.

[0056] As shown in FIG. 8 , the setting information 220 includes information such as a "work environment state," a "detection method," an "expected situation (work state)," "necessary information," and a "UI to be presented." Each "work environment state" is associated with a "detection method," an "expected situation (work state)," "necessary information," and a "UI to be presented." For example, the setting information 220 is stored in the work state recognition unit 23 a and used by the work state recognition unit 23 a. The "necessary information" in FIG. 8 is an example of second information.

[0057] The "work environment status" includes, for example, work environment details such as "the size of the reflected object is increasing" or "the rotation of the lever does not match the rotation of the image." The "detection method" includes, for example, detection methods such as "camera" or "camera image, lever operation." The "expected situation (work situation)" includes, for example, operation situations such as "the construction machine is approaching something unexpected (object, person)" or "the construction machine is moving in an unexpected way due to a collision, etc." The "necessary information" includes, for example, information such as "object position and shape information," "comparison information between operation and actual operation, information on the cause of the problem," etc. The "presented UI" includes, for example, information such as "increase the resolution around the object, emphasize the object," "display the overall image, display the position difference between the input and actual operation, highlight any moving objects," etc.

[0058] The working device 20 selects information to be transmitted to the operating environment side from the data on the working environment side. The working situation recognition unit 23a of the working device 20 recognizes the working environment state using a detection method corresponding to the working environment state based on the setting information 220, and estimates an expected situation (work situation). The working situation recognition unit 23a then determines necessary information and a UI to be presented according to the estimated situation, and transmits them to the operating device 10. The necessary information is, for example, information required by the operator according to the expected situation.

[0059] The controller device 10 updates the presented UI in accordance with the transmitted necessary information and the presented UI. That is, when the controller device 10 receives necessary information according to a work situation in addition to necessary information according to an operation situation, the controller device 10 generates a UI adapted to the necessary information according to the operation situation and the work situation, and generates a display image based on the generated UI.

[0060] In addition, when the amount of data that can be transmitted and received is small, the communication bandwidth may be adjusted not only by the resolution but also by the frame rate. To reduce the amount of processing, the frame rate may be adjusted according to the amount of movement, such as by lowering the frame rate for areas that have not been operated for a certain period of time and then raising it again when an obstacle is detected. Furthermore, people's entrances and exits and areas that should not be destroyed may be displayed at all times regardless of the communication status. The movement of people, obstacles, and the arm 101 may also be predicted and this information may be transferred. Safety can be enhanced by preferentially sending error information, such as when the construction machine 22a is not operating as expected or cannot be stopped, to the operating environment.

[0061] <1-4. Flow of an Example of Image Generation Processing> A flow of an example of image generation processing according to the embodiment will be described with reference to Fig. 9 and Fig. 10. Each of Fig. 9 and Fig. 10 is a diagram showing a flowchart of an example of image generation processing according to the embodiment.

[0062] 9, the controller device 10 determines whether a device operation has been detected (step S11). If it determines that a device operation has been detected (step S11: Yes), it calculates information to be presented to the operator (step S12). For example, the controller device 10 calculates information to be presented to the operator from input operation information such as the amount and direction of movement of the controller 11a. On the other hand, if the controller device 10 determines that a device operation has not been detected (step S11: No), it proceeds to step S15.

[0063] The operation device 10 determines whether a UI change is necessary based on the information to be presented and the current UI (step S13), and if it determines that a UI change is necessary (step S13: Yes), it transmits UI change information to the operation device 20 (step S14). On the other hand, if the operation device 10 determines that a UI change is not necessary (step S13: No), the process proceeds to step S15. The UI change information includes, for example, a request for information required depending on the UI to be changed.

[0064] Thereafter, the controller device 10 determines whether or not it has received remote data transmitted from the work device 20 (step S15). If it determines that it has received remote data (step S15: Yes), it checks whether or not the UI needs to be changed depending on the work situation (step S16). If a change is necessary, it performs a UI change process (step S17). On the other hand, if it determines that it has not received remote data (step S15: No), it executes a communication error process (step S18) and proceeds to step S19. As the communication error process, for example, an error image is generated to notify the user of the communication error.

[0065] The controller device 10 performs communication error processing (e.g., an error image) or updates the display based on the received remote data and the UI (step S19), and determines whether the communication has ended (step S20). If the controller device 10 determines that the communication has ended (step S20: Yes), the process ends. On the other hand, if the controller device 10 determines that the communication has not ended (step S20: No), the process returns to step S11.

[0066] 10 , the maintenance device 20 determines whether or not UI change information has been received (step S21). If it determines that UI change information has been received (step S21: Yes), it updates the maintenance situation detection rule based on the received UI change information (step S22). On the other hand, if it determines that UI change information has not been received (step S21: No), the maintenance device 20 proceeds to step S23.

[0067] The maintenance device 20 detects the work status based on the work status detection rule (step S23), selects necessary data according to the work status, and generates transmission data (step S24). The maintenance device 20 determines whether an exceptional situation (unexpected situation) in the environment or communication has occurred (step S25). If it determines that an exceptional situation in the environment or communication has occurred (step S25: Yes), the maintenance device 20 adds work status information to the transmission data (step S26). For example, if an unexpected situation has occurred in the environment or communication status, the maintenance device 20 adds work status information to the transmission data to inform the operator that an unexpected situation has occurred in the environment or communication status.

[0068] The maintenance device 20 transmits the prepared data to the operation device 10 as the remote data (step S27), determines whether the communication has ended (step S28), and ends the process if it determines that the communication has ended (step S28: Yes). On the other hand, if the maintenance device 20 determines that the communication has not ended (step S28: No), the process returns to step S21.

[0069] In addition, when a UI change is necessary depending on the work, the system may not immediately make the change at its discretion, but may instead present the changed display and allow the operator to select whether or not to change the current display. Furthermore, when there are multiple display candidates, the multiple display candidates may be presented to the operator, and the operator may select a display candidate. For example, when it is difficult to narrow down the excavation target, a rough range of potential candidates may be presented, and the operator may narrow down the range by selecting it. Furthermore, the user may be able to adjust the angle of view and resolution settings. Furthermore, the UI presentation may be adjusted according to the operator's level of proficiency, such as providing a high-resolution display with a narrow angle of view for an expert, and a wide angle of view for a beginner.

[0070] Furthermore, when the UI (display rules) or display is switched, the reason for the change may also be communicated to the operator. This notification can increase convenience. For example, a text message saying "A person has entered" may be displayed, or an alarm or other warning sound may be sounded. When changing the UI, priorities may be assigned to conditions. For example, if a person is present around the construction machine 22a, the top priority may be given to a view angle that allows the person to be present for safety reasons. However, if there is no person present and communication is stable, a camera image showing the desired work space from a different angle may also be displayed. For example, the display may be determined according to the level of danger.

[0071] The information processing system 1 described above can also be applied to remote operation of equipment other than the construction machine 22a. For example, it can be applied to a forklift as a factory application. A forklift is equipment used to load, unload, and transport cargo in factories and warehouses. The information processing system 1 can be applied to remotely operating levers for operating the forks used to load cargo, as well as accelerators and brakes. The information processing system 1 makes it possible to prevent collisions with people or other cargo in remote locations and to stack cargo accurately. In addition, irregular situations such as cargo shifting can be handled by transmitting data from the work environment. The information processing system 1 can also be applied to remote operation of a robot arm.

[0072] Typically, an operator must interrupt device operation to determine and switch the optimal display, resulting in reduced work efficiency. Furthermore, to select data corresponding to the UI display specified by the operator, a large amount of data, such as captured video footage, must be transmitted from a remote location. Depending on the network environment, this can cause delays. Furthermore, unexpected phenomena (e.g., fog or mist) may occur in the remote location. In such cases, it is difficult to check the overall situation and readjust the optimal UI. Unless an appropriate selection is made, the work becomes even more difficult.

[0073] On the other hand, in the embodiment, the UI is changed and the data to be transmitted and received is selected depending on the operation situation, the environment of the remote location, the communication state, etc. The operation device 10 determines the optimal UI based on the operator's controller input, etc., and transmits information on data required for UI display to the remote location. The operation device 20 at the remote location acquires data corresponding to the specified UI. Furthermore, if a phenomenon unexpected by the operator occurs at the remote location, the operation device 20 acquires data related to that phenomenon. The operation device 20 transmits the acquired data to the operation device 10. The operation device 10 updates the UI display based on the received data and presents it to the operator.

[0074] Furthermore, the function of updating the UI according to the operation status and work environment eliminates the need for explicit UI change operations, allowing for optimal UI display without placing a burden on the operator. Furthermore, by transmitting and receiving optimal work environment data according to the task, the operator's work accuracy can be improved. Furthermore, by reducing the number of operations required for UI adjustment, the user can concentrate on operating the device, thereby improving work efficiency.

[0075] <1-5. Actions and Effects> As described above, the information processing system 1 according to the embodiment includes an operation status recognition unit 13a that recognizes an operation status (e.g., operation status 120) from an input operation by an operator in the operation environment, a work status recognition unit 23a that recognizes a work status (e.g., work status 130, each of work statuses 130A-130C) from an image (e.g., before-change image 100) related to a work to be performed in the work environment, and a display image generation unit 13g that generates a display image (e.g., after-change image 140, each of after-change images 140A-140C) according to the operation status and the work status (see, for example, FIGS. 1 to 5 ). This makes it possible to display a display image according to the operation status and the work status and provide it appropriately to the operator, thereby improving work efficiency.

[0076] The operation status may also include the work content corresponding to the input operation (see FIGS. 2 to 5), thereby making it possible to generate a display image corresponding to the work content and the work status.

[0077] The operation status may also include necessary information according to the work content (see FIGS. 2 to 5), which makes it possible to reliably generate a display image according to the work content and work status.

[0078] The work situation may also include the situation of the work environment (see FIGS. 2 to 5), which allows a display image to be generated according to the operation situation and the environmental situation.

[0079] The work situation may also include the communication situation between the work environment and the operating environment (see FIGS. 2 to 5), which allows a display image to be generated according to the operating situation and the communication situation.

[0080] The information processing system 1 may further include a generating unit (e.g., UI generating unit 13b) that generates a display rule (e.g., UI) according to the operation status, and the display image generating unit 13g may generate a display image based on the display rule (see FIGS. 1 to 5). This allows the display image to be generated appropriately.

[0081] The generating unit may generate the display rule based on setting information (e.g., each setting information 200, 210) that sets the display rule (e.g., UI to be provided) for each operation content of the input operation (see FIGS. 6 and 7), thereby enabling the display image to be generated appropriately.

[0082] The generation unit may determine first information necessary for generating the display image based on the display rule, and the display image generation unit 13g may generate the display image based on the display rule and the first information (see FIGS. 6 and 7). This allows the display image to be generated appropriately.

[0083] The generation unit may generate the display rule and determine the first information based on setting information that sets the display rule and the first information for each operation content of the input operation (see FIGS. 6 and 7 ), thereby enabling the display image to be generated appropriately.

[0084] The information processing system 1 may further include a data selection unit 23b that acquires second information necessary for generating a display image based on the work status (see FIGS. 1 to 5). This allows the display image to be generated appropriately.

[0085] The generation unit may generate the display image based on the second information in addition to the display rule and the first information (see FIG. 8), thereby making it possible to appropriately generate the display image.

[0086] The generation unit may also update the display rule based on the display rule corresponding to the second information, and generate the display image based on the updated display rule, the first information, and the second information (see FIG. 8 ). This allows the display image to be generated appropriately.

[0087] The information processing system 1 may further include a communication status detection unit 13f that detects whether the second information is desired information (e.g., information as requested) (see FIGS. 1 to 5). This allows the display image to be generated appropriately.

[0088] Furthermore, the operation status recognizing unit 13a may recognize the operation status based on setting information (e.g., each of the setting information 200 and 210) that sets the operation status for each operation content of the input operation (see FIGS. 6 and 7). This allows the operation status to be appropriately recognized.

[0089] The work status recognizing unit 23a may also recognize the work status based on setting information (e.g., setting information 220) that sets the work status for each state of the work environment (see FIG. 8 ). This allows the work status to be appropriately recognized.

[0090] Furthermore, the information processing system 1 may include an operation device 10 having an operation situation recognition unit 13a and a display image generation unit 13g, and a work device 20 connected to the operation device 10 via a network 30 and having a work situation recognition unit 23a (see FIG. 1, etc.). This configuration also makes it possible to achieve improved work efficiency.

[0091] The information processing system 1 may further include an input device 11 that accepts input operations (see FIG. 1, etc.). This configuration also makes it possible to improve work efficiency.

[0092] The information processing system 1 may further include an output device 12 that displays a display image (see FIG. 1, etc.). This configuration also makes it possible to improve work efficiency.

[0093] 2. Other Embodiments The configurations and processes according to the above-described embodiments (including examples and modified examples) may be implemented in various different forms other than the above-described embodiments. For example, the configurations and processes are not limited to the above-described examples and may be implemented in various forms. Furthermore, for example, the configurations, processing procedures, specific names, or information including various data and parameters shown in the above documents and drawings may be changed arbitrarily unless otherwise specified.

[0094] Furthermore, the components and processes according to the above-described embodiments (including examples and modifications) do not necessarily have to be physically configured as shown in the drawings. In other words, the specific forms of distribution and integration of the components and processes are not limited to those shown in the drawings, and all or part of them may be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc.

[0095] Furthermore, the configurations and processes of the above-described embodiments (including examples and modified examples) may be combined as appropriate. For example, at least a part of an embodiment may be combined as appropriate with at least a part of another embodiment. Furthermore, the effects of the embodiments are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0096] 3. Hardware Configuration Example> A specific hardware configuration example of the information devices (e.g., information processing device 13, information processing device 23, etc.) that constitute the information processing system 1 according to the above-described embodiment will be described. The information devices that constitute the information processing system 1 according to the embodiment may be realized by, for example, a computer 500 configured as shown in Fig. 11. Fig. 11 is a diagram showing a hardware configuration example according to the embodiment.

[0097] 11, the computer 500 includes a CPU 510, a RAM 520, a ROM (Read Only Memory) 530, a HDD (Hard Disk Drive) 540, a communication interface 550, and an input / output interface 560. The components of the computer 500 are connected by a bus 570.

[0098] The CPU 510 operates and controls each unit based on programs stored in the ROM 530 or the HDD 540. For example, the CPU 510 loads the programs stored in the ROM 530 or the HDD 540 into the RAM 520 and executes processing corresponding to the various programs.

[0099] The ROM 530 stores boot programs such as a Basic Input Output System (BIOS) executed by the CPU 510 when the computer 500 is started, and programs that depend on the hardware of the computer 500 .

[0100] The HDD 540 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 510 and data used by such programs. Specifically, the HDD 540 is a recording medium that records program data 541 (for example, an information processing program according to an embodiment).

[0101] The communication interface 550 is an interface for connecting the computer 500 to an external network 580 (for example, the Internet). For example, the CPU 510 receives data from other devices and transmits data generated by the CPU 510 to other devices via the communication interface 550.

[0102] The input / output interface 560 is an interface for connecting the input / output device 590 and the computer 500. For example, the CPU 510 receives data from an input device such as a keyboard or a mouse via the input / output interface 560. The CPU 510 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 560.

[0103] The input / output interface 560 may function as a media interface that reads programs and the like recorded on a predetermined recording medium. Examples of the medium include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, and semiconductor memories.

[0104] For example, when the computer 500 functions as an information device according to an embodiment (e.g., the information processing device 13, the information processing device 23, etc.), the CPU 510 of the computer 500 executes an information processing program loaded onto the RAM 520 to realize all or part of the functions of each unit of the information device. The information processing program and data according to the present disclosure are stored in the HDD 540. The CPU 510 reads and executes program data 541 from the HDD 540, but as another example, the CPU 510 may acquire these programs from another device via an external network 580.

[0105] <4. Supplementary Notes> The present technology can also be configured as follows. (1) An information processing system including: an operation status recognition unit that recognizes an operation status from an input operation by an operator in an operation environment; a work status recognition unit that recognizes a work status from an image related to a work of an operation target in a work environment; and a display image generation unit that generates a display image according to the operation status and the work status. (2) The information processing system described in (1), wherein the operation status includes work content according to the input operation. (3) The information processing system described in (2), wherein the operation status includes necessary information according to the work content. (4) The information processing system described in any one of (1) to (3), wherein the work status includes a status of the work environment. (5) The information processing system described in any one of (1) to (4), wherein the work status includes a communication status between the work environment and the operation environment. (6) The information processing system according to any one of (1) to (5), further comprising a generation unit that generates a display rule according to the operation status, wherein the display image generation unit generates the display image based on the display rule. (7) The information processing system according to (6), wherein the generation unit generates the display rule based on setting information that sets the display rule for each operation content of the input operation. (8) The information processing system according to (6) or (7), wherein the generation unit determines first information required to generate the display image based on the display rule, and the display image generation unit generates the display image based on the display rule and the first information. (9) The information processing system according to (8), wherein the generation unit generates the display rule and determines the first information based on setting information that sets the display rule and the first information for each operation content of the input operation. (10) The information processing system according to (8) or (9), further comprising a data selection unit that acquires second information required to generate the display image based on the work status. (11) The information processing system according to (10), wherein the generation unit generates the display image based on the second information in addition to the display rule and the first information.(12) The information processing system according to (11), wherein the generation unit updates the display rule based on a display rule corresponding to the second information, and generates the display image based on the updated display rule, the first information, and the second information. (13) The information processing system according to any one of (10) to (12), further comprising a communication status detection unit that detects whether the second information is desired information. (14) The information processing system according to any one of (1) to (13), wherein the operation status recognition unit recognizes the operation status based on setting information that sets the operation status for each operation content of the input operation. (15) The information processing system according to any one of (1) to (14), wherein the work status recognition unit recognizes the work status based on setting information that sets the work status for each state of the work environment. (16) The information processing system described in any one of (1) to (15), comprising: an operation device having the operation status recognition unit and the display image generation unit; and a work device connected to the operation device via a network and having the work status recognition unit. (17) The information processing system described in any one of (1) to (16), further comprising an input device that accepts the input operation. (18) The information processing system described in any one of (1) to (17), further comprising an output device that displays the display image. (19) An information processing device comprising: an operation status recognition unit that recognizes an operation status from an input operation by an operator in an operation environment; a receiving unit that receives a work status in a work environment; and a display image generation unit that generates a display image according to the operation status and the work status. (20) An information processing method, including: a computer recognizing an operation status from an input operation by an operator in the operation environment; recognizing the work status from an image related to a work of an operation target in the work environment; and generating a display image according to the operation status and the work status. (21) An information processing device including each component of the information processing system according to any one of (1) to (18). (22) An information processing method for performing information processing using the information processing system according to any one of (1) to (18).

[0106] 1 Information processing system 10 Operation device 11 Input device 11a Controller 12 Output device 12a Display 13 Information processing device 13a Operation status recognition unit 13b UI generation unit 13c Construction machine control unit 13d Data transmission / reception unit 13e Memory unit 13f Communication status detection unit 13g Display image generation unit 20 Work device 21 Input device 21a Sensor 21b Camera 22 Output device 22a Construction machine 23 Information processing device 23a Work status recognition unit 23b Data selection unit 23c Data transmission / reception unit 30 Network 100 Image before change 101 Arm 102 Bucket 110 Operation content 120 Operation status (operation environment) 121 Frame 130 Work status (work environment) 130A Work status (1) 130B Work status (2) 130C Work status (3) 140 Image after change 140A Image after change 140B Image after change 140C Image after change 141a Image 141b Blurred area 141c Person highlighting 200 Setting information 210 Setting information 220 Setting information

Claims

1. An information processing system comprising: an operation status recognition unit that recognizes an operation status from an input operation by an operator in an operation environment; a work status recognition unit that recognizes a work status from an image related to a work to be operated in a work environment; and a display image generation unit that generates a display image according to the operation status and the work status.

2. The information processing system according to claim 1, wherein the operation status includes a task content corresponding to the input operation.

3. The information processing system according to claim 2, wherein the operation status includes necessary information according to the work content.

4. The information processing system according to claim 1, wherein the work status includes the status of the work environment.

5. The information processing system according to claim 1, wherein the work status includes a communication status between the work environment and the operating environment.

6. The information processing system according to claim 1, further comprising a generation unit that generates a display rule according to the operation status, wherein the display image generation unit generates the display image based on the display rule.

7. The information processing system according to claim 6, wherein the generation unit generates the display rule based on setting information that sets the display rule for each operation content of the input operation.

8. The information processing system according to claim 6, wherein the generation unit determines first information required to generate the display image based on the display rule, and the display image generation unit generates the display image based on the display rule and the first information.

9. The information processing system according to claim 8, wherein the generation unit generates the display rule and determines the first information based on setting information that sets the display rule and the first information for each operation content of the input operation.

10. The information processing system according to claim 8, further comprising a data selection unit that acquires second information necessary for generating the display image based on the work status.

11. The information processing system according to claim 10, wherein the generation unit generates the display image based on the second information in addition to the display rule and the first information.

12. The information processing system of claim 11, wherein the generation unit updates the display rule based on a display rule corresponding to the second information, and generates the display image based on the updated display rule, the first information, and the second information.

13. The information processing system according to claim 10, further comprising a communication status detection unit that detects whether the second information is desired information.

14. The information processing system according to claim 1, wherein the operation status recognition unit recognizes the operation status based on setting information that sets the operation status for each operation content of the input operation.

15. The information processing system according to claim 1, wherein the work situation recognition unit recognizes the work situation based on setting information that sets the work situation for each state of the work environment.

16. The information processing system according to claim 1, comprising: an operation device having the operation status recognition unit and the display image generation unit; and a work device connected to the operation device via a network and having the work status recognition unit.

17. The information processing system according to claim 1, further comprising an input device that accepts the input operation.

18. The information processing system according to claim 1, further comprising an output device that displays the display image.

19. An information processing device comprising: an operation status recognition unit that recognizes an operation status from an input operation by an operator in an operation environment; a receiving unit that receives a work status in a work environment; and a display image generation unit that generates a display image according to the operation status and the work status.

20. An information processing method including: a computer recognizing an operation status from an input operation by an operator in an operation environment; recognizing a work status from an image related to the work of an operation target in a work environment; and generating a display image according to the operation status and the work status.

Images