Information processing device, information processing method, and program

Abstract

The present disclosure pertains to an information processing device, an information processing method, and a program that make it possible to perform better remote work assistance. In the present invention, an update video acquisition unit acquires, from among videos obtained by imaging a work object, an update video to be used for updating 3D data, an update location identification unit identifies, by comparing already created 3D data of the work object with the update video, an update location in the 3D data of the work object, and a 3D data update unit updates, by using the update video, a portion of the 3D data of the work object. The present technology can be applied to, for example, a remote work assistance system.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that enable better remote work support to be implemented.

[0002] Conventionally, when equipment trouble occurs at a manufacturing site, a supervisor in a remote location (for example, an expert in an office or at home) would instruct workers at the manufacturing site on how to recover from the equipment trouble, what work to do, etc. In such cases, the supervisor can remotely provide appropriate instructions and support while checking 3D data of the equipment created from video footage taken in advance.

[0003] For example, Patent Document 1 discloses a technique for reconstructing the three-dimensional shape of a subject from images captured by multiple imaging devices, in which the external parameters of multiple imaging devices placed at different positions are calibrated.

[0004] International Publication No. 2019 / 225681

[0005] However, in the past, the 3D data of equipment viewed by instructors was created from pre-recorded video, so it was assumed that the actual equipment condition would be different when the on-site worker requested instructions. For example, the display contents of the equipment's operation panel would change continuously depending on the operator's operation. This made it difficult to effectively implement remote work support.

[0006] The present disclosure has been made in light of such circumstances, and aims to enable better remote work support to be implemented.

[0007] An information processing device according to one aspect of the present disclosure includes an update image acquisition unit that acquires update image from images obtained by photographing a work object to be used for updating 3D data, an update location identification unit that compares the update image with 3D data of the work object that has already been created to identify update locations in the 3D data of the work object, and a 3D data update unit that uses the update image to update a portion of the 3D data of the work object.

[0008] An information processing method or program according to one aspect of the present disclosure includes acquiring updated imagery from images obtained by photographing a work object to be used for updating 3D data, comparing the updated imagery with already created 3D data of the work object to identify areas to be updated in the 3D data of the work object, and using the updated imagery to update a portion of the 3D data of the work object.

[0009] In one aspect of the present disclosure, an update image to be used for updating the 3D data is obtained from images obtained by photographing the work object, and the update image is compared with the 3D data of the work object that has already been created to identify the updated parts of the 3D data of the work object, and a portion of the 3D data of the work object is updated using the update image.

[0010] 1 is a diagram illustrating a configuration example and a usage example of an embodiment of a remote work support system to which the present technology is applied. FIG. 1 is a diagram illustrating the processing process of 3D data reading processing and 3D data update processing. FIG. 2 is a block diagram illustrating a configuration example of a remote work support system that executes 3D data update processing. FIG. 3 is a flowchart illustrating 3D data update processing on the client terminal side. FIG. 4 is a flowchart illustrating update video processing. FIG. 5 is a flowchart illustrating 3D data update processing on the server side. FIG. 6 is a block diagram illustrating a configuration example of a smartphone used as a client terminal. FIG. 7 is a diagram illustrating an overview of pointing UI processing. FIG. 8 is a diagram illustrating display of a pointing UI object. FIG. 9 is a block diagram illustrating a configuration example of a remote work support system that executes pointing UI processing. FIG. 9 is a flowchart illustrating pointing UI processing. FIG. 10 is a diagram illustrating changes in the display of a pointing UI object over time. FIG. 11 is a diagram illustrating examples of changes in color saturation and size of a pointing UI object. FIG. 12 is a diagram illustrating changes in display in accordance with multiple pointing UI objects present within a certain range. FIG. 13 is a diagram illustrating changes such that a pointing UI object is displayed that is sized to cover multiple pointing UI objects. FIG. 14 is a diagram illustrating changes such that a past pointing UI object is displayed in its original, prominent state. FIG. 15 is a block diagram illustrating a configuration example of a computer to which the present technology is applied.

[0011] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0012] <Configuration Example and Usage Example of Remote Work Support System> A configuration example of a remote work support system to which the present technology is applied and a usage example of remote work support using the remote work support system will be described with reference to FIG.

[0013] As shown in FIG. 1, the remote operation support system 11 is configured such that a client terminal 12A and a client terminal 12B are connected to a server 13 via a network, and the server 13 includes a storage unit 14.

[0014] 1, a worker at the site uses the client terminal 12A, and an instructor at a remote location uses the client terminal 12B. For example, when responding to a problem, the worker can take a picture of the current state of the equipment 21, which is the object of the work, using the camera of the client terminal 12A in order to explain the situation at the site to the instructor.

[0015] The client terminal 12A transmits to the server 13, from among the images of the equipment 21 captured by the camera, updated images to be used to update the 3D data in the server 13. For example, if a worker captures an image of a portion of the equipment 21 that is necessary to communicate the site situation to an instructor or to respond to instructions from the instructor, the updated image will show an important portion of the equipment 21. In the example shown in FIG. 1 , the worker captures an image of an operation panel 22 provided on the equipment 21 with the client terminal 12A, and an image of the operation panel 22 is displayed on the screen 31 of the client terminal 12A.

[0016] The storage unit 14 stores 3D data (such as three-dimensional scan data and AR map data) of the equipment 21 that is the target of work by the worker. For example, the 3D data of the equipment 21 is created by photographing the equipment 21 in advance.

[0017] The server 13 compares the updated image of the equipment 21 transmitted from the client terminal 12A with the 3D data of the equipment 21 stored in the storage unit 14, identifies a portion of the 3D data of the equipment 21 to be updated, and updates only that portion using the updated image. For example, feature point matching, as described below, can be used to identify the portion of the 3D data of the equipment 21 to be updated. This allows the server 13 to update the 3D data of the equipment 21 stored in the storage unit 14 to the latest state (e.g., the current display content of the operation panel 22) currently being photographed by the worker. The server 13 then transmits the latest 3D data of the equipment 21 to the client terminal 12B.

[0018] The client terminal 12B generates a virtual object of the equipment 21 based on the 3D data of the equipment 21 in its latest state transmitted from the server 13, places the object in the virtual space, and displays a VR image 33 rendered from the virtual space from a desired viewpoint (for example, a viewpoint from which it is easy to give instructions to a worker) on the screen 32. In the example shown in FIG. 1 , the screen 32 of the client terminal 12B also displays a captured image 34 (the same image as that displayed on the screen 31 of the client terminal 12A) currently being captured by the camera of the client terminal 12A. This allows the instructor to give instructions to the workers on site while checking the VR image 33 and the captured image 34 based on the equipment 21 in its latest state.

[0019] In this way, in the remote work support system 11, a 3D data update process is performed in which updated images sequentially transmitted from the client terminal 12A to the server 13 are used to sequentially update only a portion of the 3D data of the equipment 21 corresponding to the updated images. This allows the remote work support system 11 to ensure that the 3D data of the equipment 21 transmitted to the client terminal 12B is always up to date, thereby enabling better remote work support to be implemented.

[0020] For example, when the 3D data of the equipment 21 is created from a video image taken in advance, it is assumed that the situation will be different from the actual equipment when the on-site worker requests instructions, and it is assumed that the instructor will not be able to give appropriate instructions, but the remote operation support system 11 can avoid such a situation. In particular, even if the display content of the operation panel 22 changes successively due to the operator's operation, the instructor can check the latest display content of the operation panel 22 and give instructions.

[0021] Furthermore, in the remote operation support system 11, by updating only a portion of the 3D data (an important portion for which the worker feels that instructions or support are necessary), it is possible to reduce the amount of calculations, such as matching, performed by the server 13 compared to updating the entire 3D data. This allows the 3D data sent to the client terminal 12B to be updated to the latest state in more real time.

[0022] Furthermore, in the remote work support system 11, when it is detected that a body part such as a worker's hand is reflected in an image being captured by the camera of the client terminal 12A, it is possible to update the image with video sections before and after which the body part is not reflected, or video areas in which the body part is not reflected, etc. This makes it possible in the remote work support system 11 to more reliably update the 3D data of the equipment 21 even when the equipment 21 is partially hidden by the reflection of a body part of the worker.

[0023] The client terminal 12A may be a smartphone as shown in Fig. 1, an augmented reality head mounted display (AR HMD), a tablet, etc. The client terminal 12B may be a personal computer as shown in Fig. 1, or a virtual reality head mounted display (VR HMD).

[0024] Here, the remote work support system 11 can be configured so that the client terminal 12A and the client terminal 12B have the same functions by executing a common remote work support application on both the client terminal 12A and the client terminal 12B. Therefore, hereinafter, the client terminal 12A and the client terminal 12B having the same functions will be simply referred to as the client terminal 12. Of course, the client terminal 12A may be configured to have only the functions that are required by the worker on-site, among the functions provided by executing the remote work support application. Similarly, the client terminal 12B may be configured to have only the functions that are required by the instructor at a remote location, among the functions provided by executing the remote work support application.

[0025] <Processing Example of 3D Data Update Processing> A 3D data update processing for sequentially updating a portion of 3D data in the remote operation support system 11 will be described with reference to FIGS. 2 to 7. FIG.

[0026] FIG. 2 is a diagram illustrating the processing steps of the 3D data reading process and the 3D data updating process.

[0027] First, in the remote work support system 11, the client terminal 12 is started up and a remote work support application is executed, and the server 13 is also started up. Then, the client terminal 12A on the site side uses an AR marker, a VPS (Visual Positioning System / Service), or the like to set, for example, the origin position of a virtual coordinate system that is common to the created 3D data.

[0028] Next, a 3D data reading process is performed in the remote operation support system 11. The client terminal 12 performs an internal processing event to start a receiving process, and communicates with the server 13 to request transmission of 3D data. The server 13 performs an external processing event to execute a 3D data request process in response to the request from the client terminal 12, and communicates to transmit the 3D data in response to the request from the client terminal 12. The client terminal 12 performs an external processing event to generate a virtual object based on the 3D data transmitted from the server 13, and places the virtual object in the virtual space.

[0029] Next, a 3D data update process is performed in the remote operation support system 11. The client terminal 12 performs an internal processing event to start a transmission process, and performs communication to transmit updated video to the server 13. The server 13 performs an external processing event to execute a 3D data update process to partially update the 3D data using the updated video transmitted from the client terminal 12, and performs communication to transmit the latest 3D data to the client terminal 12. The client terminal 12 performs an external processing event to update virtual objects based on the latest 3D data transmitted from the server 13, and partially updates the virtual objects arranged in the virtual space.

[0030] Thus, in the remote operation support system 11, in the 3D data reading process, the client terminal 12 can read 3D data from the server 13 and generate a virtual object. Then, in the 3D data update process, the client terminal 12 can send updated video to the server and partially update the virtual object based on the latest 3D data updated on the server 13 using the updated video.

[0031] FIG. 3 is a block diagram showing an example of the functional configuration of the client terminal 12 and the server 13 that execute the 3D data update process.

[0032] The client terminal 12 is configured to include a data determination unit 41, a virtual object generation unit 42, a flag setting unit 43, a 3D data request unit 44, a trigger recognition unit 45, an image acquisition unit 46, an image analysis unit 47, and an updated image transmission unit 48. The server 13 is configured to include a data determination unit 51, a 3D data transmission unit 52, a feature point matching processing unit 53, and a 3D data update unit 54.

[0033] The data determination unit 41 determines whether data has been received from the server 13 and determines whether the type of data is 3D data or a 3D data update notification. For example, if the data determination unit 41 determines that the type of data transmitted from the server 13 is 3D data, it supplies the 3D data to the virtual object generation unit 42, and if it determines that the type of data transmitted from the server 13 is a 3D data update notification, it supplies the 3D data update notification to the flag setting unit 43.

[0034] The virtual object generation unit 42 generates a virtual object based on the 3D data transmitted from the server 13. Then, the virtual object generation unit 42 places the generated virtual object in the virtual space, or updates a virtual object that has already been placed in the virtual space.

[0035] The flag setting unit 43 sets a 3D data request flag indicating a request for 3D data from the server 13 based on a preset setting in accordance with the 3D data update notification transmitted from the server 13. For example, the preset setting for the 3D data request flag can be changed depending on the platform or user settings of the client terminal 12. For example, in the case of a smartphone used on-site such as the client terminal 12A, the preset setting is made to ignore the 3D data update notification and not set the 3D data request flag. In addition, in the case of a personal computer used in a remote location such as the client terminal 12B, the preset setting is made to always set the 3D data request flag when a 3D data update notification is transmitted.

[0036] If the 3D data request flag is set in the flag setting unit 43, the 3D data request unit 44 requests the server 13 for 3D data corresponding to the 3D data update notification.

[0037] The trigger recognition unit 45 recognizes the occurrence of an update video acquisition trigger, which triggers the acquisition of updated video. For example, the trigger recognition unit 45 can recognize the occurrence of an update video acquisition trigger when a conversation between the instructor and the worker begins, when the worker performs an operation to start shooting, or when the smartphone remains in the same position for a longer period of time than a predetermined period of time. In particular, the trigger recognition unit 45 may recognize the occurrence of an update video acquisition trigger when a demonstrative term such as "this" or "that" is included in the conversation between the instructor and the worker, or when the worker points or designates a specific location or range with his or her hand. Then, when the trigger recognition unit 45 recognizes the occurrence of an update video acquisition trigger, it instructs the video acquisition unit 46 to acquire video.

[0038] In accordance with an instruction from the trigger recognition unit 45, the video acquisition unit 46 acquires video for a certain period before and after the update video acquisition trigger and supplies the video to the video analysis unit 47. The video acquisition unit 46 may also acquire video from several minutes before the update video acquisition trigger is recognized.

[0039] The video analysis unit 47 analyzes the video supplied from the video acquisition unit 46 and acquires updated video based on the analysis results. If the video analysis unit 47 detects, based on the video analysis results, that a body part such as a worker's hand is shown in the video, it can identify video sections before and after the body part, or video areas in which the body part is not shown, as video to be updated. Furthermore, based on the video analysis results, if the video shows the worker's hand in the shape of a pointing finger and there is an object within a certain range where the worker is pointing, the video analysis unit 47 can identify video including the object as video to be updated.

[0040] The updated video transmission unit 48 transmits the updated video identified by the video analysis unit 47 from the video acquired by the video acquisition unit 46 to the server 13 .

[0041] The data determination unit 51 determines whether data has been received and whether the type of data is a 3D data request or updated video for data transmitted from the client terminal 12. For example, if the data determination unit 41 determines that the type of data transmitted from the client terminal 12 is a 3D data request, it supplies the 3D data request to the 3D data transmission unit 52, and if it determines that the type of data transmitted from the client terminal 12 is updated video, it supplies the updated video to the feature point matching processing unit 53.

[0042] The 3D data transmission unit 52 transmits 3D data to the client terminal 12 in accordance with the 3D data request transmitted from the client terminal 12 .

[0043] The feature point matching processing unit 53 performs feature point matching between the updated video transmitted from the client terminal 12 and the 3D data corresponding to the work object at the site, and calculates the degree of match between the feature points. If the degree of match between the feature points is equal to or greater than a set value, the feature point matching processing unit 53 supplies the updated video transmitted from the client terminal 12 to the 3D data update unit 54.

[0044] The 3D data update unit 54 uses the updated image transmitted from the client terminal 12 to update a portion (e.g., texture) of the 3D data corresponding to the work object at the site. The 3D data update unit 54 then issues a 3D data update notification to the client terminal 12 to notify that the portion of the 3D data has been updated.

[0045] The 3D data update process executed in the client terminal 12 will be described with reference to the flowcharts shown in FIGS.

[0046] When the remote operation support application is executed on the client terminal 12, the process starts. In step S11, the data determination unit 41 determines whether data transmitted from the server 13 has been received.

[0047] In step S11, if the data determination unit 41 determines that data transmitted from the server 13 has been received, the process proceeds to step S12.

[0048] In step S12, the data determination unit 41 determines whether the type of data transmitted from the server 13 is 3D data or a 3D data update notification.

[0049] In step S12, if the data determination unit 41 determines that the type of data transmitted from the server 13 is 3D data, the data determination unit 41 supplies the 3D data to the virtual object generation unit 42, and the process proceeds to step S13. In step S13, the virtual object generation unit 42 generates a virtual object based on the 3D data transmitted from the server 13 and places the virtual object in the virtual space.

[0050] On the other hand, if the data determination unit 41 determines in step S12 that the type of data transmitted from the server 13 is a 3D data update notification, the data determination unit 41 supplies the 3D data update notification to the flag setting unit 43, and the process proceeds to step S14. In step S14, the flag setting unit 43 sets a 3D data request flag indicating that 3D data is requested from the server 13, based on the above-described pre-settings.

[0051] After the processing of step S13 or step S14, the processing proceeds to step S19.

[0052] On the other hand, if the data determination unit 41 determines in step S11 that data transmitted from the server 13 has not been received, the process proceeds to step S15.

[0053] In step S15, the 3D data request unit 44 determines whether to request the server 13 to transmit 3D data, in accordance with the flag set in the flag setting unit 43. That is, when the 3D data request flag is set in the flag setting unit 43, the 3D data request unit 44 determines to request the server 13 to transmit 3D data.

[0054] In step S15, if the 3D data request unit 44 determines to request the server 13 to transmit 3D data, the process proceeds to step S16. In step S16, the 3D data request unit 44 requests the server 13 to transmit 3D data corresponding to the 3D data update notification, and then the process proceeds to step S19.

[0055] On the other hand, if the 3D data request unit 44 determines in step S15 that it will not request the server 13 to transmit 3D data, the process proceeds to step S17.

[0056] In step S17, the trigger recognition unit 45 determines whether or not it has recognized the occurrence of an update video acquisition trigger that instructs acquisition of an update video.

[0057] If it is determined in step S17 that the trigger recognition unit 45 has recognized the occurrence of an update video acquisition trigger, the process proceeds to step S18, where update video processing, which will be described later with reference to FIG. 5, is performed.

[0058] On the other hand, if the trigger recognition unit 45 determines in step S17 that the occurrence of the update video acquisition trigger has not been recognized, or after the process of step S18, the process proceeds to step S19.

[0059] In step S19, the client terminal 12 waits for a predetermined waiting time before proceeding to step S20.

[0060] In step S20, the client terminal 12 determines whether or not to end the 3D data update process, for example, in accordance with an operational input by the worker.

[0061] If the client terminal 12 determines in step S20 not to end the 3D data update process, the process returns to step S11, and the same process is repeated thereafter. On the other hand, if the client terminal 12 determines in step S20 to end the 3D data update process, the process ends.

[0062] FIG. 5 is a flowchart illustrating the update image processing in step S18 of FIG.

[0063] In step S31, the video acquisition unit 46 acquires video before and after the update video acquisition trigger recognized by the trigger recognition unit 45, and supplies the video to the video analysis unit 47, which then analyzes the video.

[0064] In step S32, the video analysis unit 47 determines whether or not a body part of the worker is shown in the video based on the analysis result of the video analyzed in step S31. If the video analysis unit 47 determines in step S32 that a body part of the worker is shown in the video, the process proceeds to step S33.

[0065] In step S33, the video analysis unit 47 determines whether the body part shown in the video is the worker's hand based on the analysis result of the video in step S31. If the video analysis unit 47 determines in step S33 that the body part shown in the video is the worker's hand, the process proceeds to step S34.

[0066] In step S34, the video analysis unit 47 determines whether the hand of the worker shown in the video is in a pointing shape. If the video analysis unit 47 determines in step S34 that the hand of the worker shown in the video is in a pointing shape, the process proceeds to step S35.

[0067] In step S35, the video analysis unit 47 determines whether or not an object exists within a certain range of the location where the worker is pointing. If the video analysis unit 47 determines in step S35 that an object exists within the certain range of the location where the worker is pointing, the process proceeds to step S36.

[0068] In step S36, the video analysis unit 47 selects the video including the object pointed at by the worker as the updated video, and supplies the updated video to the updated video transmission unit 48. Thereafter, the process proceeds to step S37.

[0069] Furthermore, if it is determined in step S33 that the body part shown in the image is not the worker's hand, if it is determined in step S34 that the worker's hand shown in the image is not in the shape of a pointing finger, or if it is determined in step S35 that there is no object within a certain range where the worker is pointing, the process proceeds to step S37.

[0070] In step S37, the video analysis unit 47 targets the video acquired in step S31, such as video sections before and after which no body parts are shown, and video areas in which no body parts are shown, as updated video, and supplies the updated video to the updated video transmission unit 48.

[0071] If it is determined in step S32 that no body part of the worker is shown in the video, or after the process of step S37, the process proceeds to step S38.

[0072] In step S38, the updated video transmission unit 48 transmits the updated video to the server 13, and then the updated video processing is terminated.

[0073] The 3D data update process executed by the server 13 will be described with reference to the flowchart shown in FIG.

[0074] In step S41, the data determination unit 51 determines whether or not data transmitted from the client terminal 12 has been received.

[0075] In step S41, if the data determination unit 51 determines that data transmitted from the client terminal 12 has been received, the process proceeds to step S42.

[0076] In step S42, the data determination unit 51 determines whether the type of data transmitted from the client terminal 12 is a 3D data request or an updated video.

[0077] In step S42, if the data determination unit 51 determines that the type of data transmitted from the server 13 is a 3D data request, the data determination unit 51 supplies the 3D data request to the 3D data transmission unit 52, and the process proceeds to step S43. In step S43, the 3D data transmission unit 52 transmits the 3D data to the client terminal 12 in response to the 3D data request from the client terminal 12.

[0078] On the other hand, if the data determination unit 51 determines in step S42 that the type of data transmitted from the server 13 is updated video, the updated video is supplied to the feature point matching processing unit 53, and the process proceeds to step S44. In step S44, the feature point matching processing unit 53 performs feature point matching between the updated video transmitted from the client terminal 12 and the 3D data corresponding to the work target at the site, and calculates the degree of match between the feature points.

[0079] In step S45, the feature point matching processing unit 53 determines whether the degree of coincidence of the feature points obtained in step S44 is equal to or greater than a set value.

[0080] If the feature point matching processing unit 53 determines in step S45 that the degree of match is equal to or greater than the set value, the process proceeds to step S46. In step S46, the 3D data update unit 54 updates the texture of the 3D data of the area corresponding to the updated image transmitted from the client terminal 12, among the 3D data corresponding to the work target at the site, with the updated image.

[0081] In step S47, the 3D data update unit 54 issues a 3D data update notification to the client terminal 12 to notify the client terminal 12 that a portion of the 3D data has been updated.

[0082] By executing the above-described 3D data update process by the client terminal 12 and the server 13, the 3D data of the work target portion for which the worker requires instructions from the instructor can be updated. Furthermore, even if the worker's body part, such as the worker's hand, is captured in the image captured by the client terminal 12, the 3D data can be updated using updated image that does not capture the body part.

[0083] FIG. 7 is a block diagram showing an example of the configuration when a smartphone is used as the client terminal 12.

[0084] As shown in FIG. 7, the client terminal 12 includes a sensor unit 61 , a control unit 62 , a storage unit 63 , a video output unit 64 , an audio output unit 65 , and an external communication unit 66 .

[0085] The sensor unit 61 includes a self-position estimation unit 71 , a sound acquisition unit 72 , and a video acquisition unit 73 .

[0086] The self-position estimation unit 71 acquires sensor data necessary to estimate the self-position of the client terminal 12 using, for example, an acceleration sensor, a gyro sensor, SLAM (Simultaneous Localization and Mapping), a direction sensor, or the like, and supplies the sensor data to the control unit 62. The self-position estimation unit 71 may also use a depth sensor.

[0087] The voice acquisition unit 72 acquires the voice of the user using a microphone or the like, and supplies the voice to the control unit 62. For example, the voice acquired by the voice acquisition unit 72 can be used to recognize the occurrence of an update video acquisition trigger.

[0088] The video acquisition unit 73 acquires video using a camera or the like, and supplies the video to the control unit 62 .

[0089] The control unit 62 has a self-position estimation processing unit 81 , a person / hand posture detection unit 82 , an equipment information processing unit 83 , an application execution unit 84 , a recording processing unit 85 , an output control unit 86 , and a communication control unit 87 .

[0090] The self-position estimation processing unit 81 estimates the self-position of the client terminal 12 based on the sensor data (e.g., acceleration data, angular velocity data, SLAM data, orientation data, etc.) supplied from the self-position estimation unit 71. For example, the self-position of the client terminal 12 estimated by the self-position estimation processing unit 81 can be used to set the origin position of the virtual coordinates of the 3D data.

[0091] The human / hand posture detection unit 82 analyzes the video supplied from the video acquisition unit 73 and detects the posture of the user and the posture of the hand shown in the video. For example, the hand posture detected by the human / hand posture detection unit 82 is used to determine whether the hand of the worker shown in the video is in the shape of a pointing finger (step S34 in FIG. 5 described above).

[0092] The equipment information processing unit 83 processes information about the equipment that is the target of remote work support, and, for example, acquires 3D data of the equipment for which a virtual object is to be generated.

[0093] The application execution unit 84 executes the remote operation support application to provide the functions of each block of the client terminal 12 shown in FIG. 3 described above.

[0094] The recording processing unit 85 performs a process of recording data in the storage unit 63. The output control unit 86 controls the output of video to the video output unit 64 and the output of audio to the audio output unit 65. The communication control unit 87 controls communication with the server 13 via the external communication unit 66.

[0095] The storage unit 63 stores various data required for the processing performed by the control unit 62. The video output unit 64 outputs video using, for example, a display. The audio output unit 65 outputs audio using, for example, a speaker. The external communication unit 66 communicates with the outside world using, for example, a wireless LAN (Local Area Network).

[0096] When a personal computer is used as the client terminal 12, an input device such as a mouse can be used as the self-position estimation unit 71 of the sensor unit 61. When an AR HMD or VR HMD is used as the client terminal 12, the sensor unit 61 can include a hand posture estimation unit (such as a depth sensor or an infrared camera) for estimating the posture of the user's hand.

[0097] <Example of Pointing UI Processing> Pointing UI processing for appropriately presenting a pointing UI (User Interface) object indicating the point indicated by the instructor in the remote work support system 11 will be described with reference to FIGS. 8 to 16 .

[0098] For example, a case will be described in which a worker at a site is photographing a work object using a client terminal 12A, and a remote instructor accessing the VR space at the site instructs the worker on a location on the work object while viewing a streaming screen corresponding to the worker's client terminal 12A located in the VR space. In this case, even if the instructor gives instructions to the streaming screen located in the VR space, it is difficult to properly convey the location of the instructions to the worker because the streaming screen located in the VR space and the real space indicated by the streaming screen are located at different positions.

[0099] For example, when an instructor points to a streaming screen in a VR space to convey instructions to a worker, the instructor appears to the worker as if he or she is pointing through the streaming screen into the three-dimensional space beyond. Meanwhile, the worker wants to show the instructor the work site through the streaming screen of client terminal 12A, but the screen of client terminal 12A does not display what the instructor is pointing at. Furthermore, because the worker receives instructions while holding client terminal 12A in one hand, it is conceivable that the worker may be temporarily unable to capture images using client terminal 12A in order to perform the work in accordance with the instructions. In such cases, the instructor will be unable to view the streaming screen.

[0100] Therefore, the remote operation support system 11 executes a pointing UI process that appropriately presents a pointing UI object indicating the location pointed by the instructor, based on the fingertip position of the instructor's hand in the VR space and the streaming screen.

[0101] 8A and 8B are diagrams illustrating an outline of the pointing UI processing, in which A in Fig. 8A is a diagram illustrating an example of the VR space of the instructor, and B in Fig. 8B is a diagram illustrating an example of the AR space of the worker.

[0102] For example, as shown in A of Figure 8, the instructor's VR space is arranged with the instructor's avatar Av, the instructor's hand avatar Avh, a virtual object Ob corresponding to the work object 23, and a streaming screen St corresponding to the screen of the client terminal 12A.

[0103] When the fingertip position of the instructor's hand avatar Avh is in contact with the streaming screen St, in the instructor's VR space, a pointing UI object P1 is displayed at the point where a straight line L1 passing from the head position of the instructor's avatar Av to the fingertip position of the instructor's hand avatar Avh passes through the streaming screen St, and a pointing UI object P1' is displayed at the intersection where the straight line L1 intersects with the surface of the virtual object Ob.

[0104] 8B, a pointing UI object P1 is displayed in the AR space displayed on the client terminal 12A, and the worker can confirm, through the AR space, a pointing UI object P1' indicating the location of the instructor's instruction on the work target 23. In other words, even if the pointing UI object P1' is not actually displayed on the work target 23, the worker can recognize the location of the instructor's instruction from the pointing UI object P1 displayed in the AR space displayed on the client terminal 12A.

[0105] Furthermore, when the fingertip position of the pointing user's hand avatar Avh moves away from the streaming screen St, the display of the pointing UI object is changed. For example, a pointing UI object P0 represents a previously pointed location, and its color (transparency) and size are changed from those of a pointing UI object P1 representing a current pointed location, so that it is displayed less conspicuously than the pointing UI object P1. In the illustrated example, the previously pointed location is represented by a dashed line.

[0106] Display of the pointing UI object will be described with reference to FIG.

[0107] For example, the head position P_Head of the instructor's avatar Av can be obtained by acquiring the position of the device worn on the instructor's head when the client terminal 12B is an HMD, or by acquiring the position of the virtual camera used by the instructor when the client terminal 12B is a personal computer. The index fingertip position P_IndexFingertip of the instructor's hand avatar Avh can be acquired by hand tracking of the device used by the instructor when the client terminal 12B is an HMD, or by acquiring the fingertip position of the virtual hand calculated from the mouse position when the client terminal 12B is a personal computer.

[0108] The position P_Smartphone in the VR space of the streaming screen St corresponding to the screen of the worker's client terminal 12A can be acquired from the client terminal 12A used by the worker. The virtual foot position P_FootPoint of the instructor's avatar Av can be estimated (P_FootPoint = P_Head(x, 0, y)) from the head position P_Head and the position of the coordinate origin that defines the virtual space.

[0109] The vector component Dir_pointing of the line L1 connecting the head position and fingertip of the instructor can be estimated using vector components consisting of the head position P_Head and the fingertip position P_IndexFingertip (Dir_pointing = P_IndexFingertip - P_Head). The head position P_Head is used as position information for the instructor's eyes. The collision detection surface Sur_colider of the 3D data can be generated based on the mesh of the 3D data of the virtual object Ob. The presentation position P_Pointing UI of the pointing UI object is the coordinates of the point closest to the fingertip position P_IndexFingertip among the points where the collision detection surface Sur_colider of the 3D data intersects with a line extending in the direction of the vector component Dir_pointing of the line L1 starting from the fingertip position P_IndexFingertip.

[0110] In the pointing UI processing, when the instructor's hand avatar Avh is in a predetermined hand shape (for example, pointing) and is near the streaming screen St, a pointing UI object is displayed when the following first to third conditions are met: The first condition is when a trigger word such as "here" is recognized, the second condition is when the instructor's hand avatar Avh comes into contact with the streaming screen St, and the third condition is when the distance between the fingertip of the instructor's hand avatar Avh and the streaming screen St is a certain distance and a certain amount of time has passed.

[0111] When these conditions are met, a pointing UI object is displayed at the point where a line extending in the direction of the vector component Dir_pointing of a line L1 starting from the fingertip position P_IndexFingertip intersects with the contact detection surface Sur_colider of the 3D data, the point being closest to the fingertip position P_IndexFingertip.

[0112] FIG. 10 is a block diagram showing an example of the functional configuration of the client terminal 12A and the client terminal 12B that execute the pointing UI process.

[0113] The client terminal 12A includes a screen position recognition unit 91 and a display processing unit 92. The client terminal 12B includes a fingertip position recognition unit 93, a contact determination unit 94, a pointing UI object control unit 95, and a display processing unit 96.

[0114] The screen position recognition unit 91 measures the position and orientation of the worker's client terminal 12A, and recognizes the position in the VR space of the streaming screen St corresponding to the screen of the worker's client terminal 12A based on the measurement results, and notifies the contact judgment unit 94.

[0115] The display processing unit 92 performs display processing such as displaying a pointing UI object in the AR space of the client terminal 12A and changing the display of the pointing UI object in accordance with the control of the pointing UI object control unit 95 .

[0116] The fingertip position recognition unit 93 recognizes the position of the index finger of the avatar Avh of the instructor's hand, and notifies the contact determination unit 94 of the position.

[0117] The contact determination unit 94 determines whether the fingertip position of the indicator's hand avatar Avh is in contact with the streaming screen St based on the position of the index fingertip of the indicator's hand avatar Avh and the position of the streaming screen St in the VR space. If the contact determination unit 94 determines that the fingertip position of the indicator's hand avatar Avh is in contact with the streaming screen St, it notifies the pointing UI object control unit 95 of the fingertip position of the indicator's hand avatar Avh that is in contact with the streaming screen St. Furthermore, if the fingertip position of the indicator's hand avatar Avh has moved away from the streaming screen St, the contact determination unit 94 notifies the pointing UI object control unit 95 of this fact.

[0118] The pointing UI object control unit 95 performs the control necessary to display a pointing UI object at the location indicated by the instructor. For example, as described above with reference to Figure 9, the pointing UI object control unit 95 identifies the point closest to the fingertip position P_IndexFingertip among the points where a line extending in the direction of the vector component Dir_pointing of a line L1 originating at the fingertip position P_IndexFingertip intersects with the collision detection surface Sur_colider of the 3D data. The pointing UI object control unit 95 also performs control to change the display of the pointing UI object as will be described later with reference to Figures 12 to 16.

[0119] The display processing unit 96 performs display processing such as displaying a pointing UI object in the VR space of the client terminal 12B and changing the display of the pointing UI object in accordance with the control of the pointing UI object control unit 95.

[0120] The pointing UI process executed in the client terminal 12A and the client terminal 12B will be described with reference to the flowchart shown in FIG.

[0121] In step S51, the fingertip position recognition unit 93 recognizes the position of the index fingertip of the instructor's hand avatar Avh and notifies the contact determination unit 94. The screen position recognition unit 91 measures the position and orientation of the worker's client terminal 12A, recognizes the position in the VR space of the streaming screen St that corresponds to the screen of the worker's client terminal 12A based on the measurement results, and notifies the contact determination unit 94.

[0122] In step S52, the contact determination unit 94 determines whether the fingertip position of the instructor's hand avatar Avh is in contact with the streaming screen St based on the position of the index fingertip of the instructor's hand avatar Avh and the position of the streaming screen St in the VR space.

[0123] In step S52, if the contact determination unit 94 determines that the fingertip position of the instructor's hand avatar Avh is not in contact with the streaming screen St, the process returns to step S51, and the same process is repeated thereafter.

[0124] On the other hand, if the contact determination unit 94 determines in step S52 that the fingertip position of the instructor's hand avatar Avh is in contact with the streaming screen St, the process proceeds to step S53.

[0125] In step S53, the contact determination unit 94 notifies the pointing UI object control unit 95 of the fingertip position of the avatar Avh of the instructor's hand that is touching the streaming screen St. The pointing UI object control unit 95 performs the control necessary to display a pointing UI object at the location indicated by the instructor. Then, the display processing unit 92 displays the pointing UI object in the AR space of the client terminal 12A in accordance with the control of the pointing UI object control unit 95. Similarly, the display processing unit 96 displays the pointing UI object in the VR space of the client terminal 12B in accordance with the control of the pointing UI object control unit 95.

[0126] In step S54, similarly to step S51, the fingertip position recognition unit 93 recognizes the position of the index finger of the avatar Avh of the instructor's hand, and the screen position recognition unit 91 recognizes the position of the streaming screen St in the VR space.

[0127] In step S55, the contact determination unit 94 determines whether the position of the index finger of the instructor's hand avatar Avh has moved away from the streaming screen St based on the position of the index finger of the instructor's hand avatar Avh and the position of the streaming screen St in the VR space.

[0128] In step S55, if the contact determination unit 94 determines that the fingertip position of the instructor's hand avatar Avh has not left the streaming screen St, the process returns to step S54, and the pointing UI object continues to be displayed.

[0129] On the other hand, if the contact determination unit 94 determines in step S55 that the fingertip position of the instructor's hand avatar Avh has left the streaming screen St, the process proceeds to step S56.

[0130] In step S56, the contact determination unit 94 notifies the pointing UI object control unit 95 that the fingertip position of the instructor's hand avatar Avh has left the streaming screen St. The pointing UI object control unit 95 performs the control necessary to change the display of the pointing UI object. Then, the display processing unit 92 changes the color and size of the pointing UI object displayed in the AR space of the client terminal 12A in accordance with the control of the pointing UI object control unit 95. Similarly, the display processing unit 96 changes the color and size of the pointing UI object displayed in the VR space of the client terminal 12B in accordance with the control of the pointing UI object control unit 95.

[0131] After the process of step S56, the process returns to step S51, and the same process is repeated thereafter.

[0132] By performing the pointing UI process described above, it is possible to appropriately present a pointing UI object that indicates the pointing location of the instructor.

[0133] Changes in the display of the pointing UI object over time will be described with reference to FIGS. 12 and 13. FIG.

[0134] In the pointing UI processing, the display of the pointing UI object can be changed according to the elapsed time from when the fingertip position of the instructor's hand avatar Avh comes into contact with the streaming screen St or the elapsed time from when the fingertip position of the instructor's hand avatar Avh leaves the streaming screen St.

[0135] For example, as described above with reference to Fig. 8 , a pointing UI object is displayed in response to the fingertip position of the instructor's hand avatar Avh coming into contact with the streaming screen St. Then, as shown in the upper part of Fig. 12 , if the fingertip position of the instructor's hand avatar Avh moves a certain distance (e.g., 5 cm) or more away from the streaming screen St, the display of the pointing UI object does not change immediately thereafter. Then, after a certain time T1 has elapsed since the fingertip position of the instructor's hand avatar Avh moved away from the streaming screen St, the display of the pointing UI object changes (from a solid line to a dashed line in the illustrated example).

[0136] As shown in A of FIG. 13 , the timing when the fingertip position of the avatar Avh of the instructor's hand leaves the streaming screen St may be set to t=0, and the saturation and size of the color (including transmittance) of the pointing UI object may change at a constant rate (Color=const×F(t, T1)) until a certain time T1 has elapsed.

[0137] Alternatively, as shown in FIG. 13B, the fingertip position of the instructor's hand avatar Avh may be moved away from the streaming screen St until a predetermined time T constant The saturation and size of the color (including the transmittance) of the pointing UI object are kept constant until a predetermined time T constant Alternatively, the change rate may be set to change at a constant rate from the time T1 until the time T2 has elapsed.

[0138] In this way, the pointing UI object control unit 95 can perform control such that the display of the pointing UI object changes over time, thereby making past pointing UI objects less noticeable.

[0139] Referring to FIG. 14, a description will be given of a change in display in accordance with a plurality of pointing UI objects present within a certain range.

[0140] In the pointing UI process, the display of the pointing UI objects can be changed in accordance with the presence of multiple pointing UI objects within a certain range.

[0141] For example, a case will be described in which, following the display of a pointing UI object P1, an instruction to newly display a pointing UI object P2 is given, as shown in the upper part of Fig. 14. In this case, if the pointing UI object P1 is present within a certain distance (e.g., within 2.5 cm) from the pointing UI object P2, the display may be changed so that the pointing UI object P2 is not displayed, but a pointing UI object P3 of a size that covers the pointing UI object P1 and the pointing UI object P2 is displayed, as shown in the lower part of Fig. 14.

[0142] Referring to FIG. 15, a description will be given of a modification in which, when there are multiple pointing UI objects within a certain distance range, a pointing UI object of a size that covers these pointing UI objects is displayed.

[0143] In the pointing UI processing, when there are multiple pointing UI objects within a certain distance, the size of the pointing UI object closest to the center of gravity of the multiple pointing UI objects can be changed to a size that covers all of the pointing UI objects. For example, this certain distance and the center of gravity of the multiple pointing UI objects can be calculated from the trajectory of the fingertips of the instructor's hand avatar Avh.

[0144] 15 , when the trajectory of the fingertip of the instructor's hand avatar Avh passes through pointing UI objects P1 and P2 and repeatedly traces the same area in a circular motion, a pointing UI object P3 large enough to cover the trajectory of the fingertip may be displayed. In this case, the measurement start point of the trajectory of the fingertip of the instructor's hand avatar Avh may be the fingertip position when a trigger word such as "from here" is recognized, and the measurement end point may be the fingertip position when a trigger word such as "to here" is recognized.

[0145] A pointing UI process for changing a past pointing UI object so that it is displayed in its original prominent state will be described with reference to Fig. 16. Fig. 16A is a diagram showing an example of the VR space of the instructor, and Fig. 16B is a diagram showing an example of the AR space of the worker.

[0146] In the pointing UI processing, the display of the previous pointing UI object is changed to make it less noticeable, as shown on the left side of Fig. 16. If the instructor then moves his / her hand to the area where the previous pointing UI object is presented and a pointing determination is made indicating that the instructor is pointing again, the color and transparency of the pointing UI object can be changed to return it to its original prominent state, as shown on the right side of Fig. 16.

[0147] As described above, the pointing UI processing makes it possible to display a pointing UI object indicating the location of the instructor in a virtual space that includes 3D data of the site, based on the position of the instructor's hand and the position of the worker's device. This allows an instructor in a remote location to indicate the location of the instruction simply by pointing at the streaming screen of the site in the virtual space, enabling more intuitive and easy-to-understand remote work support.

[0148] Furthermore, the pointing UI processing can change the color and size of the pointing UI object depending on the time elapsed since the pointing UI object was pointed at and the number of pointed locations within a certain range. This allows past pointing UI objects that are no longer needed at present to gradually become less noticeable over time. This reduces the number of pointing UI objects that stand out in the virtual space, making it easier for both the operator and the instructor to understand which instructions they should currently be looking at.

[0149] Furthermore, in the pointing UI processing, when the instructor's hand moves again into the area where a previous pointing UI object is presented and a pointing determination is made, the color, transparency, etc. of the pointing UI object can be changed to make it more noticeable. This allows the pointing UI object to be re-emphasized as needed, enabling the instructor to provide remote work support utilizing records of previous instructions.

[0150] <Application example of remote work support system>

[0151] For example, when the remote work support system 11 is used at a building or construction site, on-site workers can photograph the condition of buildings, equipment, etc., and can remotely check, instruct, and provide support for the same. When the remote work support system 11 is used in agriculture, on-site workers can photograph the condition of crops and fields on the farm, and can remotely check, instruct, and provide support for the same. When the remote work support system 11 is used in medical care, on-site workers can photograph the condition and layout of equipment in operating rooms or locations where medical equipment is located, or doctors and nurses can photograph the condition of patients and the status of surrounding equipment in ICUs or operating rooms, and can remotely check, instruct, and provide support for the same.

[0152] Furthermore, when the remote work support system 11 is used in urban development or landscape construction, it allows on-site personnel to photograph the urban development status, layout, landscape, etc., and to remotely confirm, instruct, and provide support for these. Furthermore, when the remote work support system 11 is used in a commercial facility, it allows on-site personnel to photograph the facility layout, product displays, sales status, etc., within the store, and to remotely confirm, instruct, and provide support for these.

[0153] The remote work support system 11 may be used not only for remote confirmation, instructions, and support, but also for, for example, checking before and after work only on-site.

[0154] <Example of Computer Configuration> Next, the above-described series of processes (information processing method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

[0155] FIG. 17 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0156] The program can be recorded in advance on the hard disk 105 or ROM 103 as a recording medium built into the computer.

[0157] Alternatively, the program can be stored (recorded) on a removable recording medium 111 driven by the drive 109. Such a removable recording medium 111 can be provided as a so-called package software. Here, examples of the removable recording medium 111 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0158] The program can be installed into the computer from the removable recording medium 111 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed on the built-in hard disk 105. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.

[0159] The computer includes a CPU (Central Processing Unit) 102 , to which an input / output interface 110 is connected via a bus 101 .

[0160] When a user inputs a command via the input / output interface 110 by operating the input unit 107, the CPU 102 executes a program stored in a read-only memory (ROM) 103 in accordance with the command. Alternatively, the CPU 102 loads a program stored on a hard disk 105 into a random access memory (RAM) 104 and executes the program.

[0161] As a result, the CPU 102 performs processing according to the flowchart described above or processing performed by the configuration of the block diagram described above. Then, the CPU 102 outputs the processing results from the output unit 106 via the input / output interface 110, transmits them from the communication unit 108, or records them on the hard disk 105, as necessary.

[0162] The input unit 107 is made up of a keyboard, a mouse, a microphone, etc. The output unit 106 is made up of an LCD (Liquid Crystal Display), a speaker, etc.

[0163] In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or object-based processing).

[0164] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0165] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0166] Also, for example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0167] Furthermore, for example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0168] Furthermore, for example, the above-described program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0169] Also, for example, each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0170] In addition, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0171] It should be noted that the present technologies described in this specification can be implemented independently and singly, unless a contradiction arises. Of course, any two or more of the present technologies can also be implemented in combination. For example, part or all of the present technologies described in any embodiment can be implemented in combination with part or all of the present technologies described in other embodiments. Furthermore, part or all of any of the present technologies described above can also be implemented in combination with other technologies not described above.

[0172] <Examples of Combinations of Configurations> The present technology can also be configured as follows. (1) An information processing device comprising: an update image acquisition unit that acquires, from images obtained by photographing a work object, update image to be used for updating 3D data; an update portion identification unit that compares already created 3D data of the work object with the update image to identify an update portion of the 3D data of the work object; and a 3D data update unit that updates a portion of the 3D data of the work object using the update image. (2) The information processing device described in (1) above, further comprising: a trigger recognition unit that recognizes the occurrence of an update image acquisition trigger that triggers the acquisition of the update image, wherein the update image acquisition unit acquires the update image from images of a certain period before and after the occurrence of the update image acquisition trigger is recognized. (3) The information processing device described in (2) above, wherein the trigger recognition unit recognizes the occurrence of the update image acquisition trigger based on a conversation between a worker performing work on the work object and an instructor who gives instructions to the worker from a remote location. (4) The information processing device described in (3) above, wherein, when it is detected based on an analysis result of the video that a body part of the worker is shown in the video, the updated video acquisition unit sets video sections before and after the body part is not shown, or a video area in which the body part is not shown, as the updated video. (5) The information processing device described in (3) above, wherein, based on an analysis result of the video, when the hand of the worker shown in the video is in the shape of a pointing finger and an object is present within a certain range where the worker is pointing, the updated video is set to a video including the object. (6) The information processing device described in any of (1) to (5) above, wherein the updated part identification unit performs feature point matching between already created 3D data of the work target object and the updated video, and identifies the updated part if the degree of match between the feature point matching is equal to or greater than a set value.(7) The information processing device according to any one of (1) to (6) above, further comprising: a determination unit that determines that the instructor has designated a designated location when a specific condition is satisfied when an avatar of the hand of an instructor giving instructions from a remote location to a worker performing work on the work object, with the hand avatar taking a predetermined hand shape, is located near a streaming screen corresponding to a screen of the worker's terminal in the VR space of the instructor, and a pointing UI object control unit that controls display of a pointing UI object indicating the designated location. (8) The information processing device according to (7), wherein the determination unit determines that the instructor has designated the designated location when a fingertip position of the avatar of the instructor's hand is in contact with the streaming screen. (9) The information processing device according to (7) or (8), wherein the pointing UI object control unit displays the pointing UI object as the designated location, at a point where a straight line passing from a head position of the avatar of the instructor in the VR space of the instructor and through the fingertip positions of the avatar of the instructor's hand intersects with a surface of a virtual object of the work object. (10) The information processing device according to any of (7) to (9), wherein the pointing UI object control unit changes the display of the pointing UI object according to the elapsed time from the timing when the pointing user finished pointing at the pointing location. (11) The information processing device according to any of (7) to (10), wherein the pointing UI object control unit displays a pointing UI object of a size that covers all of the pointing UI objects when multiple pointing UI objects exist within a certain range. (12) The information processing device according to any of (1) to (11), wherein the pointing UI object control unit displays a previous pointing UI object in its original state when the previous pointing UI object is pointed at again.(13) An information processing method including: an information processing device acquiring, from images obtained by photographing a work object, update imagery to be used for updating 3D data, comparing already created 3D data of the work object with the update imagery to identify updated portions of the 3D data of the work object, and updating a portion of the 3D data of the work object using the update image. (14) A program for causing a computer of an information processing device to execute information processing including: acquiring, from images obtained by photographing a work object, update imagery to be used for updating 3D data, comparing already created 3D data of the work object with the update imagery to identify updated portions of the 3D data of the work object, and updating a portion of the 3D data of the work object using the update image.

[0173] It should be noted that the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0174] REFERENCE SIGNS LIST 11 Remote work support system, 12 Client terminal, 13 Server, 14 Memory unit, 41 Data determination unit, 42 Virtual object generation unit, 43 Flag setting unit, 44 3D data request unit, 45 Trigger recognition unit, 46 Video acquisition unit, 47 Video analysis unit, 48 Updated video transmission unit, 51 Data determination unit, 52 3D data transmission unit, 53 Feature point matching processing unit, 54 3D data update unit, 61 Sensor unit, 62 Control unit, 63 Memory unit, 64 Video output unit, 65 Audio output unit, 66 External communication unit, 71 Self-position estimation unit, 72 Audio acquisition unit, 73 Video acquisition unit, 81 Self-position estimation processing unit, 82 Human / hand posture detection unit, 83 Equipment information processing unit, 84 Application execution unit, 85 Recording processing unit, 86 Output control unit, 87 Communication control unit, 91 Screen position recognition unit, 92 Display processing unit, 93 Fingertip position recognition unit, 94 Contact determination unit, 95 Pointing UI object control unit, 96 Display processing unit

Claims

1. An information processing device comprising: an update image acquisition unit that acquires update image to be used for updating 3D data from images obtained by photographing a work object; an update location identification unit that compares the update image with already created 3D data of the work object to identify the update location of the 3D data of the work object; and a 3D data update unit that uses the update image to update a portion of the 3D data of the work object.

2. An information processing device as described in claim 1, further comprising a trigger recognition unit that recognizes the occurrence of an update video acquisition trigger that triggers the acquisition of the update video, and the update video acquisition unit acquires the update video from video of a certain period before and after the occurrence of the update video acquisition trigger is recognized.

3. The information processing device according to claim 2, wherein the trigger recognition unit recognizes the occurrence of the update image acquisition trigger based on a conversation between a worker performing work on the work object and an instructor who gives instructions to the worker from a remote location.

4. The information processing device of claim 3, wherein when the updated image acquisition unit detects, based on the analysis results of the image, that a body part of the worker is shown in the image, the updated image is the image section before and after the body part is not shown, or the image area in which the body part is not shown.

5. The information processing device of claim 3, wherein the updated image acquisition unit, based on the analysis results of the image, determines that if the worker's hand shown in the image is in the shape of a pointing finger and an object is present within a certain range of where the worker is pointing, the updated image is an image including the object.

6. The information processing device according to claim 1, wherein the update location identification unit performs feature point matching between the already created 3D data of the work object and the updated image, and identifies the update location when the degree of match between the feature point matching is equal to or greater than a set value.

7. The information processing device of claim 1, further comprising: a judgment unit that judges that the instructor has pointed to a specified location when a hand avatar of an instructor giving instructions from a remote location to a worker performing work on the work object is in a specified hand shape and is in the vicinity of a streaming screen corresponding to the screen of the worker's terminal in the instructor's VR space, and certain conditions are satisfied; and a pointing UI object control unit that controls the display of a pointing UI object indicating the specified location.

8. The information processing device according to claim 7, wherein the determination unit determines that the instructor has designated the designated location when the fingertip position of the instructor's hand avatar is in contact with the streaming screen.

9. The information processing device according to claim 7, wherein the pointing UI object control unit displays the pointing UI object with the point where a straight line passing from the head position of the instructor's avatar in the instructor's VR space to the fingertip positions of the instructor's hand avatar intersects with the surface of the virtual object of the work target.

10. The information processing device according to claim 7, wherein the pointing UI object control unit changes the display of the pointing UI object in accordance with the elapsed time from the timing at which the pointing by the pointing user ended.

11. The information processing device according to claim 7, wherein, when a plurality of the pointing UI objects exist within a certain range, the pointing UI object control unit displays the pointing UI object of a size that covers all of the pointing UI objects.

12. The information processing device according to claim 7, wherein the pointing UI object control unit, when a past pointing UI object is pointed to again, displays the pointing UI object in its original state.

13. An information processing method including: an information processing device acquiring update image to be used for updating 3D data from images obtained by photographing a work object; comparing already created 3D data of the work object with the update image to identify an update location of the 3D data of the work object; and updating a portion of the 3D data of the work object using the update image.

14. A program for causing a computer of an information processing device to execute information processing including: acquiring updated image to be used for updating 3D data from images obtained by photographing a work object; comparing already created 3D data of the work object with the updated image to identify updated portions of the 3D data of the work object; and updating a portion of the 3D data of the work object using the updated image.

Images