Information processing method, information processing device, and information processing program

Abstract

An information processing device acquires position information and attitude information of a worker in a real space for updating the position and attitude of a virtual camera provided in a virtual space in which the position and attitude of the worker simulate the real space. On the basis of the position information, it is determined whether or not the worker is in a steady state, and when it is determined that the worker is not in the steady state, the updating frequency of the position information and the attitude information is reduced compared to when it is determined that the worker is in the steady state.

Description

Technical Field

[0001] The present disclosure relates to a technology for remotely assisting workers on site. Background Art

[0002] Patent document 1 discloses that in a 3D model generation device that generates a 3D model of a subject based on camera images of a close-up camera and a pullback camera by using a visual volume intersection method, the 3D model is generated by assuming that the subject exists outside the field of view of the close-up camera.

[0003] However, Patent Document 1 only generates a virtual camera image enclosed in a virtual space, and does not disclose generating a virtual camera image associated with a real space. Therefore, in Patent Document 1, when the position and posture of the virtual camera are updated based on the position and posture of the operator during a period of large movement of the operator, a problem of generating a virtual camera image with low visibility may occur.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2022-29730 Summary of the invention

[0007] The present disclosure has been made to solve the above-mentioned problems, and an object of the present disclosure is to provide a technology for preventing the generation of a virtual camera image with low visibility that changes drastically.

[0008] An information processing method in one embodiment of the present invention is an information processing method in an information processing device that assists a worker in his or her work, and the information processing method performs the following processing: obtaining position information and posture information of the worker in a real space, the position information and posture information being used to update the position and posture of a virtual camera set in a virtual space that simulates the real space, and judging whether the worker is in a stable state based on the position information; if it is judged that the worker is not in the stable state, the updating frequency of the position information and the posture information is reduced compared to a case where the worker is judged to be in the stable state.

[0009] According to the present disclosure, it is possible to prevent the generation of a virtual camera image with drastic changes and low visibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a block diagram showing an example of the overall configuration of the information processing system in the first embodiment.

[0011] Figure 2This is a diagram showing an overview of processing by the information processing system.

[0012] Figure 3 This is a diagram showing what spatial calibration looks like.

[0013] Figure 4 This is a diagram showing how space calibration looks like.

[0014] Figure 5 This is a flowchart showing an example of processing by the information processing system in the first embodiment.

[0015] Figure 6 This is a flowchart showing an example of processing by the information processing system in the second embodiment.

[0016] Figure 7 This is a flowchart showing an example of processing by the information processing system in the third embodiment.

[0017] Figure 8 This is a flowchart showing an example of processing by the information processing system in the fourth embodiment.

[0018] Fig. 9 This is a flowchart showing an example of processing by the information processing system in the fifth embodiment. DETAILED DESCRIPTION

[0019] (Knowledge serving as the basis of the present disclosure)

[0020] When a worker is working on an object at a work site, the worker confirms the instructions from a remote worker outside the work site and performs the work. In this case, if the remote worker can confirm which part of the object the worker is observing, the remote worker can smoothly instruct the worker. To achieve this goal, for example, a shooting device such as a sports camera or smart glasses is installed on the worker's head, and if the image of the work site captured by the shooting device is sent to a remote terminal owned by the remote worker in real time, the remote worker can confirm which part of the object the worker is observing.

[0021] However, from the perspective of safety, there are work sites where it is prohibited to output images to the outside. In this case, there is a problem that a remote person cannot confirm the part that the worker is observing.

[0022] Therefore, the inventors have studied the following technology: by synchronizing the position and posture of the workers on site with the position and posture of the virtual camera, a virtual camera image is generated by reproducing the field of view of the workers on site in the virtual space, and the generated virtual camera image is displayed on the remote terminal of the remote worker.

[0023] In this technology, if the position and posture of the operator are always synchronized with the position and posture of the virtual camera, the synchronization is implemented even during periods when the operator's movements are large, such as during walking or at the beginning and end of walking. During periods of large movements such as the above, since the position and posture of the operator change frequently, a virtual camera image that changes dramatically, such as a small shake, will be generated. Virtual camera images that change dramatically such as the above are difficult to recognize, and a remote person who recognizes the virtual camera image may also feel dizzy. It is sufficient for the remote person to confirm the movement of the operator who is performing a certain operation on the object, and there is no need to confirm subtle movements such as hand shaking. Furthermore, since the operator remains motionless during the operation, the changes in position and posture are not drastic.

[0024] Therefore, the inventors of the present invention acquired the following knowledge and came up with the present disclosure: when the operator is not in a stable state, if the update frequency of the position and posture of the virtual camera based on the position information and posture information is reduced compared to the case where the operator is in a stable state, the generation of virtual camera images with drastic changes and low visibility can be prevented.

[0025] (1) An information processing method in one aspect of the present disclosure is an information processing method in an information processing device that assists a worker in performing a work, the method obtaining position information and posture information of the worker in a real space, the position information and posture information being used to update the position and posture of a virtual camera set in a virtual space that simulates the real space, and determining whether the worker is in a stable state based on the position information. If it is determined that the worker is not in the stable state, the frequency of updating the position information and posture information is reduced compared to a case where the worker is determined to be in the stable state.

[0026] According to this structure, when the operator is not in a stable state, the frequency of updating the position and posture of the virtual camera based on the position information and posture information is reduced compared to the case of being in a stable state. Therefore, the position and posture of the virtual camera are stable, and the generation of a virtual camera image with low visibility that changes dramatically can be prevented.

[0027] (2) In the information processing method described in (1) above, the stable state may indicate a state in which the worker is not accelerating or decelerating.

[0028] According to this configuration, when the worker accelerates or decelerates, the frequency of updating the position and posture of the virtual camera based on the position information and posture information is reduced, thereby preventing the generation of a virtual camera image with low visibility that changes dramatically.

[0029] (3) In the information processing method described in (2) above, the judgment may also include: calculating acceleration based on the position information, and judging that it is not the stable state when the absolute value of the acceleration exceeds the reference acceleration, and judging that it is the stable state when the absolute value of the acceleration is lower than the reference acceleration.

[0030] According to this configuration, when the absolute value of the acceleration calculated based on the position information exceeds the reference acceleration, it is determined that the worker is in the acceleration state or the deceleration state. Therefore, it is possible to accurately sense that the worker is in the acceleration state or the deceleration state.

[0031] (4) In the information processing method described in (1) above, the stable state may indicate a state in which the operator is stopped.

[0032] According to this configuration, when the worker moves, the frequency of updating the position and posture of the virtual camera based on the position information and posture information is reduced, thereby preventing the generation of a virtual camera image with low visibility that changes drastically.

[0033] (5) In the information processing method described in (4) above, the judgment may also include: calculating the speed based on the position information, and when the speed exceeds the reference speed, judging that the operator is not in the stable state; when the speed is lower than the reference speed, judging that the operator is in the stable state.

[0034] According to this configuration, when the speed calculated based on the position information exceeds the reference speed, it is determined that the worker has moved, so whether the worker has moved can be accurately sensed.

[0035] (6) In the information processing method described in any one of (1) to (5) above, the judgment may also include: when it is judged that the operator is not in the stable state, judging whether the period from the last update of the position information and the posture information to the present exceeds a fixed period, and the reduction of the update frequency includes: executing the update when it is judged that the period exceeds the fixed period.

[0036] According to this configuration, it is possible to prevent the position and posture of the virtual camera from not being updated for a long period of time due to the position and posture of the worker.

[0037] (7) In the information processing method described in any one of (1) to (6) above, a low-pass filter is also applied to the displacement of the position information and the displacement of the posture information to perform correction processing for correcting the position information and the posture information. The correction processing may also include: when it is determined that the operator is not in the stable state, lowering the cutoff frequency of the low-pass filter compared to when it is determined that the operator is in the stable state.

[0038] According to this structure, when the operator is not in a stable state, the cutoff frequency of the low-pass filter is shifted to the low-frequency side compared to the case of a stable state, so that the position information and posture information with drastic changes removed can be obtained. As a result, the position information and posture information are stable, and the generation of a virtual camera image with drastic changes and low visibility can be prevented.

[0039] (8) In the information processing method described in any one of (1) to (7) above, the position and posture of the virtual camera can be further updated based on the position information and the posture information, a virtual camera image of the virtual space photographed by the virtual camera can be further generated, and the virtual camera image can be further displayed on a display.

[0040] According to this configuration, it is possible to display a virtual camera image that reproduces the worker's field of view in the real space in the virtual space.

[0041] (9) In the information processing method described in any one of (1) to (8) above, the reduction in the update frequency may also include: not executing the update when it is determined that the operator is not in the stable state; and executing the update when it is determined that the operator is in the stable state.

[0042] According to this configuration, since the position information and the posture information are updated only when the worker is in a stable state, it is possible to more reliably prevent the generation of a virtual camera image with rapid changes and low visibility.

[0043] (10) In another embodiment of the present invention, an information processing device is an information processing system that includes a processor and assists a worker in performing work, wherein the processor performs the following processing: obtaining position information and posture information of the worker in a real space, wherein the position information and posture information are used to update the position and posture of a virtual camera set in a virtual space that simulates the real space, and determining whether the worker is in a stable state based on the position information; if it is determined that the worker is not in the stable state, reducing the frequency of updating the position information and posture information compared to a case where the worker is determined to be in the stable state.

[0044] According to this configuration, it is possible to provide an information processing device that prevents generation of a virtual camera image that changes drastically and has low visibility.

[0045] (11) An information processing program in another embodiment of the present invention is an information processing program that enables a computer to execute an information processing method for assisting a worker's work, and enables the computer to execute the following processing: obtaining position information and posture information of the worker in a real space, the position information and posture information being used to update the position and posture of a virtual camera set in a virtual space that simulates the real space, judging whether the worker is in a stable state based on the position information, and if it is judged that the worker is not in the stable state, reducing the frequency of updating the position information and posture information compared to a case where it is judged that the worker is in the stable state.

[0046] According to this configuration, it is possible to provide an information processing program that prevents generation of a virtual camera image that changes drastically and has low visibility.

[0047] The present disclosure can also be realized as an information updating system that operates according to the above-mentioned information processing program. Furthermore, it goes without saying that the above-mentioned computer program can be distributed via a computer-readable non-transitory recording medium such as a CD-ROM or a communication network such as the Internet.

[0048] It should be noted that the embodiments described below are all ways of representing a specific example of the present disclosure. The numerical values, shapes, structural elements, steps, and the order of steps shown in the following embodiments are only examples and are not intended to limit the present disclosure. In addition, the structural elements not recorded in the independent claims representing the highest concept among the structural elements in the following embodiments are described as arbitrary structural elements. In addition, the respective contents can also be combined in all embodiments.

[0049] (Implementation method 1)

[0050] Figure 1 1 is a block diagram showing an example of the overall structure of the information processing system 100 in Embodiment 1. The information processing system 100 is a system in which a remote person in a remote place assists a worker who performs work on an object. Specifically, the system shares information between the worker and the remote person by reproducing the image recognized by the worker on the scene in a virtual space so that the remote person can recognize the image in the virtual space, and assists the worker's work from a distance. Virtual space refers to an imaginary three-dimensional space constructed in a computer.

[0051] The information processing system 100 includes a detection device 1, a worker terminal 2, and a remote terminal 3. The detection device 1 and the worker terminal 2 are located at a site where an object is installed. The remote terminal 3 is located at a remote location. The detection device 1 and the worker terminal 2 are connected to each other via a communication path C1 so as to be able to communicate. The communication path C1 may be a wireless communication path such as a wireless LAN or Bluetooth (registered trademark), or a wired communication path. The worker terminal 2 and the remote terminal 3 are connected via a network NT. The network NT may be, for example, a wide area communication network including the Internet.

[0052] The detection device 1 is a sensor that detects the position and posture of a worker at a scene. An example of the detection device 1 may be an optical motion capture system or a self-positioning system. The optical motion capture system is a system that detects the position and posture of a worker by photographing a worker with a marker mounted on his head with an optical tracker. The self-positioning system is a system that uses a camera mounted on the worker to capture an image of the scene, and applies SLAM (Simultaneous Localization and Mapping) technology to the obtained image, thereby detecting the position and posture of the worker. It should be noted that in the self-positioning system, laser sensors such as Lidar may be used instead of cameras to detect the position and posture of the worker.

[0053] The detection device 1 sends the detected position information indicating the position of the operator and the posture information indicating the posture of the operator to the operator terminal 2. The position information may be three-dimensional coordinate data indicating the position of a specific part (e.g., head) of the operator in real space. The posture information may be a three-dimensional vector indicating the direction of the operator's head in real space. As the direction of the head, for example, the frontal orientation of the head may be used.

[0054] The detection device 1 detects the position and posture of the worker at a given sampling rate, and transmits the position information and posture information to the worker terminal 2 at a given sampling rate. Thus, the remote terminal 3 can obtain the position information and posture information in real time.

[0055] The operator terminal 2 is, for example, a portable information terminal carried by the operator. An example of a portable information terminal is a smart phone, a tablet computer, and smart glasses. The operator terminal 2 includes a communication unit 21, a processor 22, and a communication unit 25. The communication unit 21 is composed of a communication circuit capable of communicating with the detection device 1. The communication unit 21 receives position information and posture information sent at a given sampling rate from the detection device 1.

[0056] The processor 22 is composed of a central processing unit (CPU) and includes an acquisition unit 23 and a determination unit 24. The processor 22 executes an information processing program to implement the acquisition unit 23 and the determination unit 24. It should be noted that the acquisition unit 23 and the determination unit 24 may also be composed of dedicated hardware circuits such as ASIC.

[0057] The acquisition unit 23 acquires the position information and posture information transmitted from the detection device 1 at a given sampling rate via the communication unit 21 .

[0058] The determination unit 24 determines whether the worker is in a stable state based on the position information acquired by the acquisition unit 23. The stable state refers to a state in which the worker is not accelerating or decelerating, that is, a state in which the worker is stopped or moving at a constant speed.

[0059] The judgment unit 24 calculates the acceleration of the operator by performing a second-order differential on the position information. When the absolute value of the calculated acceleration exceeds the reference acceleration, the judgment unit 24 judges that the operator is not in a stable state, that is, in an acceleration-deceleration state of positive acceleration or deceleration. On the other hand, when the absolute value of the calculated acceleration is lower than the reference acceleration, the judgment unit 24 judges that the operator is in a stable state, that is, not in an acceleration-low speed state. The acceleration state refers to a state from when the operator starts accelerating to move to when the moving speed becomes fixed. The deceleration state refers to a state from when the operator starts decelerating to stop to when the moving operator stops.

[0060] When the determination unit 24 determines that the operator is in the acceleration / deceleration state, the determination unit 24 does not transmit the position information and posture information acquired by the acquisition unit 23 to the remote terminal 3 using the communication unit 25. On the other hand, when the determination unit 24 determines that the operator is not in the acceleration / deceleration state, the determination unit 24 transmits the position information and posture information acquired by the acquisition unit 23 to the remote terminal 3 using the communication unit 25.

[0061] The communication unit 25 is composed of a communication circuit that connects the operator terminal 2 to the network NT. The communication unit 25 transmits the position information and the posture information to the remote terminal 3 under the control of the determination unit 24 .

[0062] The remote terminal 3 is, for example, a terminal installed at a remote location away from the site. The remote terminal 3 may be a desktop computer or a portable computer such as a smart phone or a tablet computer.

[0063] The remote terminal 3 includes a communication unit 31 , a memory 32 , a processor 33 , and a display 34 . The communication unit 31 is composed of a communication circuit that connects the remote terminal 3 to the network NT, and acquires the position information and posture information transmitted from the operator terminal 2 .

[0064] The memory 32 is composed of a rewritable nonvolatile storage device such as a solid state drive or a hard disk drive, and stores a three-dimensional model, position information, posture information, and calibration information. The three-dimensional model is a three-dimensional model that simulates the shape of an object.

[0065] The position information and posture information are position information and posture information sent from the operator terminal 2. The calibration information is information that establishes correspondence between the real space and the virtual space representing the scene. The calibration information includes a real reference vector and a virtual reference vector. The real reference vector is a three-dimensional vector that starts from the initial position of the operator in the real space and ends at the position of the object. The virtual reference vector is a three-dimensional vector that starts from the initial position of the operator in the virtual space and ends at the position of the three-dimensional model of the object. The calibration information is generated by performing the space calibration described later.

[0066] The processor 33 is composed of, for example, a central processing unit (CPU), and includes a calibration execution unit 331 and an image generation unit 332. The calibration execution unit 331 and the image generation unit 332 are implemented by the processor 33 executing an information processing program. However, this is only an example, and the calibration execution unit 331 and the image generation unit 332 may also be composed of a dedicated hardware circuit such as an ASIC. The calibration execution unit 331 executes a spatial calibration that establishes a correspondence between the real space and the virtual space representing the scene. The spatial calibration is performed as a pre-processing before the operator starts working on the scene.

[0067] The image generation unit 332 updates the position and posture of the operator with the position and posture of the virtual camera set in the virtual space based on the position information and posture information. As a result, the position and posture of the operator are synchronized with the position and posture of the virtual camera. The image generation unit 332 generates a virtual camera image obtained by photographing the virtual space with the synchronized virtual camera. The image generation unit 323 displays the virtual camera image on the display 34.

[0068] The display 34 is composed of a display device such as a liquid crystal display panel, and displays the virtual camera image generated by the image generation unit 323 .

[0069] Figure 22 is a diagram showing an overview of the processing of the information processing system 100. An object 211 is set up on site. An operator 212 performs an arbitrary operation on the object 211 on site. The information processing system 100 reproduces the image within the field of vision of the operator 212 in the virtual space. Specifically, the information processing system 100 constructs a three-dimensional model 222 that simulates the object 211 in the virtual space 221, and shoots the three-dimensional model 222 with a virtual camera 223 configured in the virtual space 221. Thus, a virtual camera image 230 is generated. The generated virtual camera image 230 is displayed on the display 34 of the remote terminal 3 located far away. Here, the position and posture of the virtual camera 223 and the operator 212 are synchronized. As a result, the remote person who views the virtual camera image can confirm which position of the object 211 the operator 212 is looking at. As a result, the remote person can provide appropriate assistance to the operator.

[0070] The scene is, for example, a real space where an object is set. An example of a scene is a factory, a laboratory, a test site, a chemical plant, etc. A factory can be a factory that manufactures electrical products such as televisions and washing machines, or a factory that manufactures motor vehicles, iron, etc. These scenes are just examples, and any place can be any place as long as it is a place where operators perform operations on the object. For example, the scene can also be a place where maintenance of machines or equipment is performed.

[0071] An example of an object is a device to be maintained and a product manufactured in a production line. An example of a product is an electric appliance, a motor vehicle, etc. An example of a device is a production line in a factory, etc.

[0072] Next, an example of spatial calibration is described. Figure 3 , Figure 4 600 is a three-dimensional real space representing the scene, and is defined by mutually orthogonal x, y, and z axes. Virtual space 700 is a three-dimensional virtual space simulating the scene, and is defined by mutually orthogonal x, y, and z axes.

[0073] The calibration execution unit 331 obtains the initial position P1 of the marker M1 set at a position away from the object 211 by a reference distance D1 in the real space 600. Here, the height h1 of the initial position P1 from the ground 601 is the same as the height of the center position O1 of the object 211 from the ground 601, and the posture direction D2 of the marker M1 is parallel to the ground 601. The initial position P1 is detected by the detection device 1 and transmitted to the remote terminal 3 via the operator terminal 2.

[0074] Next, the calibration execution unit 331 sets the virtual marker M2 and the three-dimensional model 222 in the virtual space 700 so as to maintain the relative positional relationship between the initial position P1 and the center position O1 in the real space 600. In detail, the calibration execution unit 331 sets the virtual marker M2 so that the posture direction D2´ is parallel to the virtual ground 701 at a position of a height h1´ corresponding to the height h1 from the virtual ground 701 in the virtual space 700. Next, the calibration execution unit 331 sets the three-dimensional model 222 at the center position O1´ away from the virtual marker M2 by a reference distance D1´ in parallel with the posture direction D2´. Next, the calibration execution unit 331 calculates the real reference vector V0 from the initial position P1 to the center position O1 and the virtual reference vector V0´ from the virtual initial position P1´ to the center position O1´. Next, the calibration execution unit 331 stores the real reference vector V0 and the virtual reference vector V0´ as calibration information in the memory 32. Although the spatial calibration using the marker M1 is exemplified here, the spatial calibration may be performed using the position of the worker's head as the initial position P1 instead of the marker M1 .

[0075] Next, an example of synchronization processing is described. Figure 3 , it is assumed that the marker M1 mounted on the head of the operator during operation is located at position P2. In this case, the image generation unit 332 calculates the vector from position P2 to the center position O1 as the real space vector V1. The real space vector V1 is a vector indicating the relative position of the operator during operation relative to the object 211. Next, the image generation unit 332 calculates the difference vector ΔV between the real space vector V1 and the real reference vector V0.

[0076] Next, refer to Figure 4 , the image generation unit 332 adds the differential vector ΔV and the virtual reference vector V0´ in the virtual space 700 to calculate the virtual space vector V1´. The virtual space vector V1´ is a vector that represents the relative position of the operator at work with respect to the three-dimensional model 222. Next, the image generation unit 332 determines the starting point (position P2´) in the virtual space vector V1´ as the position of the virtual camera. Next, the image generation unit 332 determines the posture of the operator detected by the detection device 1 as the posture of the virtual camera. Here, the operator installs the marker M1 on the head, but the present disclosure is not limited to this, and the operator may not install the marker M1 on the head. In this case, the image generation unit 332 only needs to use the position of the operator's head detected by the detection device 1 as the position P2 to calculate the position P2´.

[0077] Figure 5This is a flowchart showing an example of the processing of the information processing system 100 in Embodiment 1. This processing starts when a worker starts working on site. It is assumed that space calibration is executed before starting this processing.

[0078] In step S1, the detection device 1 detects the position information and posture information of the operator. Next, in step S2, the acquisition unit 23 of the operator terminal 2 acquires the position information and posture information using the communication unit 21. Next, in step S3, the determination unit 24 of the operator terminal 2 performs a second-order differential on the position information acquired in step S2 to calculate the acceleration.

[0079] Next, in step S4 , the determination unit 24 determines whether the worker is in an acceleration or deceleration state using the acceleration calculated in step S3 .

[0080] In detail, the judgment unit 24 judges that it is in the acceleration and deceleration state when the absolute value of the acceleration exceeds the reference acceleration, and judges that it is not in the acceleration and deceleration state when the absolute value of the acceleration is lower than the reference acceleration. When it is judged that the operator is in the acceleration and deceleration state (step S4: yes), the processing returns to step S2. In this case, the position information and posture information are not sent to the remote terminal 3. Therefore, the position and posture of the virtual camera will not be updated based on the position information and posture information. On the other hand, when it is judged that the operator is not in the acceleration and deceleration state (step S4: no), the processing enters step S5.

[0081] Next, in step S5 , the determination unit 24 transmits the position information and posture information acquired in step S2 to the remote terminal 3 using the communication unit 25 .

[0082] Next, in step S6, the image generation unit 332 of the remote terminal 3 synchronizes the position and posture of the worker with the position and posture of the virtual camera using the position information and posture information and calibration information transmitted in step S5. The details of the synchronization process are as described above.

[0083] Next, in step S7, the image generation unit 332 generates a virtual camera image captured by the synchronized virtual camera. In this case, the image generation unit 332 generates the virtual camera image by rendering the virtual space using the virtual camera.

[0084] Next, in step S8 , the image generation unit 332 displays the virtual camera image on the display 34 .

[0085] Next, in step S9, the image generation unit 332 determines whether the remote work assistance is continuing. If the remote work assistance is continuing (step S9: Yes), the process returns to step S2. If the remote work assistance is not continuing (step S9: No), the process ends. The end of the remote work assistance is performed by the operator inputting an instruction to end the remote work assistance to the operator terminal 2. Therefore, the image generation unit 332 only needs to determine that the remote work assistance has ended when the communication unit 31 receives the instruction from the operator terminal 2, and determine that the remote work assistance is continuing when the communication unit 31 does not receive the instruction.

[0086] Thus, according to Embodiment 1, since the position information and posture information are output only when the operator is in a stable state, it is possible to prevent the position and posture of the virtual camera from being updated with the position and posture of the operator during a period of large movement. Therefore, the position and posture of the virtual camera are stable, and the generation of a virtual camera image with low visibility that changes dramatically can be prevented.

[0087] (Implementation method 2)

[0088] Embodiment 2 is a method for determining that the operator is not in a stable state when the operator's speed exceeds the reference speed. In this embodiment, the same reference numerals are given to the same structures as those in Embodiment 1 and the description thereof is omitted. Figure 1 .

[0089] Figure 1 In the example, the determination unit 24 calculates the speed of the operator by performing a first-order differentiation on the position information obtained by the acquisition unit 23. When the calculated speed exceeds the reference speed, the determination unit 24 determines that the operator is not in a stable state, that is, is moving. On the other hand, when the calculated speed is lower than the reference speed, the determination unit 24 determines that the operator is in a stable state, that is, is not moving.

[0090] Figure 6 is a flowchart showing an example of the processing of the information processing system 100 in Embodiment 2. Steps S21, S22, S25 to S29 and Figure 5 Steps S1, S2, and S5 to S9 are the same, so their description is omitted.

[0091] In step S23 subsequent to step S22 , the determination unit 24 performs first-order differentiation on the position information acquired by the acquisition unit 23 to calculate the velocity.

[0092] Next, in step S24, the judgment unit 24 determines whether the operator is moving by comparing the speed calculated in step S23 with the reference speed. In detail, when the speed exceeds the reference speed, the judgment unit 24 determines that the operator is moving (step S24: Yes), and returns the processing to step S22. In this case, the position information and posture information are not transmitted to the remote terminal 3. Therefore, the position and posture of the virtual camera will not be updated based on the position information and posture information. On the other hand, when the speed is lower than the reference speed, the judgment unit 24 determines that the operator is not moving (step S24: No), and the processing enters step S25. In this case, the position information and posture information are transmitted to the remote terminal 3.

[0093] As described above, according to the second embodiment, since it is possible to prevent the position and posture of the virtual camera from being updated to the position and posture of the moving worker, it is possible to prevent the generation of a virtual camera image with low visibility that changes drastically.

[0094] (Implementation method 3)

[0095] Embodiment 3 is a method of controlling the update of the position information and posture information in Embodiment 1 by taking into account the period since the last update. In this embodiment, the same reference numerals are used for the same structures as in Embodiments 1 and 2, and the description thereof is omitted. Figure 1 .

[0096] Figure 1 In the case where the judgment unit 24 judges that the operator is in the acceleration and deceleration state, the judgment unit 24 judges whether the period from the last update to the present exceeds the fixed period for the position information and the posture information. If the period exceeds the fixed period, the judgment unit 24 transmits the position information and the posture information to the remote terminal 3 using the communication unit 25. If the period does not exceed the fixed period, the judgment unit 24 does not transmit the position information and the posture information. The fixed period uses an appropriate value such as 1 minute or 3 minutes.

[0097] Figure 7 is a flowchart showing an example of processing of the information processing system 100 in Embodiment 3. Steps S41 to S43, S45 to S49 and Figure 5 Steps S1 to S3 and S5 to S9 are the same, so their description is omitted.

[0098] In step S44, when it is determined that the operator is in an acceleration or deceleration state (step S44: yes), the judgment unit 24 determines whether the period from the last update to the present exceeds a fixed period for the position information and posture information (step S50). When the period exceeds the fixed period (step S50: yes), the processing proceeds to step S45. In this case, the position information and posture information are transmitted to the remote terminal 3. Therefore, the position and posture of the virtual camera are updated based on the position information and posture information. On the other hand, when the period does not exceed the fixed period (step S50: no), the processing returns to step S42. In this case, the position information and posture information are not transmitted to the remote terminal 3. Therefore, the position and posture of the virtual camera will not be updated based on the position information and posture information.

[0099] Thus, according to Embodiment 3, even when it is determined that the operator is not in the acceleration or deceleration state, the position information and posture information are transmitted to the remote terminal 3 when the period from the last update to the present exceeds the fixed period, and the position and posture of the virtual camera are updated using the transmitted position information and posture information. Therefore, it is possible to prevent the position and posture of the operator and the position and posture of the virtual camera from being out of sync for a long time. Thus, the remote personnel can appropriately manage the operator.

[0100] (Implementation method 4)

[0101] Embodiment 4 is a method of controlling the update of the position information and posture information in Embodiment 2 by taking into account the period since the last update. In this embodiment, the same reference numerals are given to the same structures as in Embodiments 1 to 3, and the description thereof is omitted. Figure 1 .

[0102] Figure 1 In the case where the judgment unit 24 judges that the operator is moving, the judgment unit 24 judges whether the period from the last update to the present exceeds the fixed period for the position information and the posture information. If the period exceeds the fixed period, the judgment unit 24 transmits the position information and the posture information to the remote terminal 3 using the communication unit 25. If the period does not exceed the fixed period, the judgment unit 24 does not transmit the position information and the posture information. The fixed period can be an appropriate value such as 1 minute, 3 minutes, 10 minutes, etc.

[0103] Figure 8 is a flowchart showing an example of processing of the information processing system 100 in Embodiment 4. Steps S61 to S63, S65 to S69 and Figure 6 Steps S21 to S23 and S25 to S29 are the same, so their description is omitted.

[0104] In step S64, when it is determined that the operator is moving (step S64: yes), the judgment unit 24 determines whether the period from the last update to the present exceeds the fixed period for the position information and the posture information (step S70). When the period exceeds the fixed period (step S70: yes), the processing proceeds to step S65. In this case, the position information and the posture information are transmitted to the remote terminal 3. Therefore, the position and posture of the virtual camera are updated based on the position information and the posture information. On the other hand, when the period does not exceed the fixed period (step S70: no), the processing returns to step S62. In this case, the position information and the posture information are not transmitted to the remote terminal 3. Therefore, the position and posture of the virtual camera are not updated based on the position information and the posture information.

[0105] Thus, according to Embodiment 4, even when it is determined that the worker has not moved, if the period from the last update of the position information and posture information to the present exceeds the fixed period, the position information and posture information are transmitted to the remote terminal 3, and the position and posture of the virtual camera are updated using the transmitted position information and posture information. Therefore, it is possible to prevent the position and posture of the virtual camera from not being updated for a long time according to the position and posture of the worker, and the remote personnel can appropriately manage the worker.

[0106] (Implementation method 5)

[0107] Embodiment 5 is a method of performing correction processing on the position information and the posture information in Embodiment 1. In this embodiment, the same reference numerals are given to the same components as in Embodiments 1 to 4, and the description thereof is omitted. Figure 1 .

[0108] Figure 1 In the process, the acquisition unit 23 applies a low-pass filter to the displacement of the position information and the displacement of the posture information of the operator to perform correction processing on the position information and the posture information. The displacement of the position information is, for example, the difference between the position information at the current sampling moment and the position information at the previous sampling moment, that is, the moving speed. The position information is composed of three-dimensional coordinate data, so the difference in position information is represented by the difference in three-dimensional coordinate data. The displacement of the posture information is the difference between the posture information at the current sampling moment and the posture information at the previous sampling moment, that is, the speed at which the posture changes. The posture information is represented by a three-dimensional vector, so the difference in posture information is represented by a three-dimensional vector.

[0109] The correction process includes, when the determination unit 24 determines that the worker is not in the acceleration / deceleration state, lowering the cutoff frequency of the low-pass filter compared to when the determination unit 24 determines that the worker is in the acceleration / deceleration state.

[0110] Fig. 9 is a flowchart showing an example of the processing of the information processing system 100 in Embodiment 5. Figure 5 Steps S1 to S2 and S3 to S9 are the same, so their description is omitted. That is, the difference is that a correction process is added in step S83.

[0111] In step S83, the acquisition unit 23 applies a low-pass filter to the displacement of the position information and the displacement of the posture information of the operator to perform a correction process for correcting the position information and the posture information. For example, it is assumed that the frequency band of the displacement of the position information and the displacement of the posture information is pre-divided into a low-frequency region, a medium-frequency region, and a high-frequency region. In this case, when the acquisition unit 23 determines that the operator is in an acceleration and deceleration state, it only needs to set the cutoff frequency of the low-pass filter to the first frequency at the boundary of the low-frequency region and the medium-frequency region of the cutoff frequency. On the other hand, when it is determined that the operator is not in an acceleration and deceleration state, the acquisition unit 23 only needs to set the cutoff frequency of the low-pass filter to the second frequency (> first frequency) at the boundary of the medium-frequency region and the high-frequency region. As a result, when the operator is in an acceleration and deceleration state, the high-frequency region and the medium-frequency region are cut off, and when the operator is not in an acceleration and deceleration state, the high-frequency region is cut off.

[0112] The details of the correction process are as follows. Set the value of the position information or posture information at a certain sampling time t to z(t). Set the displacement between the value at the sampling time (t-1) and the value at the sampling time t to Δz(t). The acquisition unit 23 applies a low-pass filter set with a first frequency or a second frequency to the displacement Δz(t). Set the displacement after applying the low-pass filter to Δz´(t). The acquisition unit 23 calculates the value z´(t) at the corrected sampling time t by adding the value z(t-1) at the sampling time (t-1) and Δz´(t). The acquisition unit 23 applies the above-mentioned processing to each sampling time, thereby performing the correction process.

[0113] Thus, according to Embodiment 5, when the operator is in an acceleration / deceleration state, the cutoff frequency of the low-pass filter is shifted to the low-frequency side compared to the case where the operator is not in an acceleration / deceleration state, thereby obtaining position information and posture information in which drastic changes have been removed. This prevents the generation of a virtual camera image with low visibility due to drastic changes. Furthermore, since the position information is stable, the accuracy of determining whether the operator is in a stable state is improved.

[0114] The present disclosure can adopt the following modified examples.

[0115] (1) In embodiments 1 to 5, the position and posture of the virtual camera based on the position information and posture information are updated only when the camera is in a stable state, but the present disclosure is not limited to this. When the camera is not in a stable state, the frequency of the update can be reduced compared to when the camera is in a stable state. For example, the judgment unit 24 only needs to transmit the position information and posture information to the remote terminal 3 at a first sampling rate when the operator is in a stable state, and transmit the position information and posture information to the remote terminal 3 at a second sampling rate when the operator is not in a stable state. Here, the second sampling rate may adopt a value lower than the first sampling rate. The first sampling rate may adopt the sampling rate at which the detection device 1 detects the position information and posture information.

[0116] (2) The correction process shown in Embodiment 5 can also be applied to Embodiments 2 to 4. In particular, when the correction process is applied to Embodiments 3 and 4, when it is determined to be yes in step S50 or when it is determined to be yes in step S70, the synchronization process is performed using the position information and posture information whose intensity of motion has been reduced, so that the generation of a virtual camera image with drastic changes can be suppressed.

[0117] (3) In Embodiments 1 to 5, the position information is calculated using an optical motion capture system or SLAM technology, but a value detected by a GPS sensor may also be used.

[0118] (4) Figure 1 In the embodiment, the operator terminal 2 includes the acquisition unit 23 and the determination unit 24 , but the acquisition unit 23 and the determination unit 24 may be provided in the detection device 1 or the processor 33 of the remote terminal 3 .

[0119] (5) The information processing device may be constituted by the operator terminal 2 or may be constituted including the operator terminal 2 and the remote terminal 3 . The display 34 may be removed from the components of the remote terminal 3 .

[0120] Industrial Applicability

[0121] The present disclosure is useful in the technical field of remotely assisting workers on site.

Claims

1. An information processing method, which is an information processing method in an information processing device for assisting an operator in performing work, obtaining position information and posture information of the operator in the real space, wherein the position information and posture information are used to update the position and posture of a virtual camera set in a virtual space simulating the real space, determining whether the operator is in a stable state based on the position information, When it is determined that the worker is not in the stable state, the updating frequency of the position information and the posture information is reduced compared to when it is determined that the worker is in the stable state.

2. The information processing method according to claim 1, wherein: The stable state refers to a state in which the worker does not perform acceleration or deceleration.

3. The information processing method according to claim 2, wherein: The judgment includes: Acceleration is calculated based on the position information, and when an absolute value of the acceleration exceeds a reference acceleration, it is determined that the state is not the stable state, and when the absolute value of the acceleration is lower than the reference acceleration, it is determined that the state is the stable state.

4. The information processing method according to claim 1, wherein: The stable state indicates a state in which the worker is stopped.

5. The information processing method according to claim 4, wherein: The judgment includes: The speed is calculated based on the position information, and when the speed exceeds a reference speed, it is determined that the worker is not in the stable state, and when the speed is lower than the reference speed, it is determined that the worker is in the stable state.

6. The information processing method according to claim 1 or 2, wherein: The judgment includes: When it is determined that the operator is not in the stable state, it is determined whether the period from the last update of the position information and the posture information to the present exceeds a fixed period. The reduction of the update frequency includes: The updating is performed when it is determined that the period exceeds the fixed period.

7. The information processing method according to claim 1 or 2, wherein: Further, a low-pass filter is applied to the displacement of the position information and the displacement of the posture information to perform correction processing for correcting the position information and the posture information, The correction process includes: When it is determined that the worker is not in the stable state, the cutoff frequency of the low-pass filter is lowered compared to a case where it is determined that the worker is in the stable state.

8. The information processing method according to claim 1 or 2, wherein: Further, the position and posture of the virtual camera are updated according to the position information and the posture information, Further, a virtual camera image is generated by photographing the virtual space with the virtual camera. Further, the virtual camera image is displayed on a display.

9. The information processing method according to claim 1 or 2, wherein: The reduction of the update frequency includes: If it is determined that the operator is not in the stable state, not executing the updating; and When it is determined that the operator is in the stable state, the updating is performed.

10. An information processing device comprising a processor and assisting an operator in performing operations. The processor performs the following processing: obtaining position information and posture information of the operator in the real space, wherein the position information and posture information are used to update the position and posture of a virtual camera set in a virtual space simulating the real space, determining whether the operator is in a stable state based on the position information, When it is determined that the worker is not in the stable state, the updating frequency of the position information and the posture information is reduced compared to when it is determined that the worker is in the stable state.

11. An information processing program causing a computer to execute an information processing method for assisting an operator in performing work. The computer is caused to execute the following processing: obtaining position information and posture information of the operator in the real space, wherein the position information and posture information are used to update the position and posture of a virtual camera set in a virtual space simulating the real space, determining whether the operator is in a stable state based on the position information, When it is determined that the worker is not in the stable state, the updating frequency of the position information and the posture information is reduced compared to when it is determined that the worker is in the stable state.

Images