Work assistance system

Abstract

The present invention implements more suitable work assistance. This work assistance system displays, as work information, video information having the position and movement of reference work for a work object in a three-dimensional space including the work object on a screen corresponding to the field of view of a worker in a display that is worn on the head by the worker such that the work information is superimposed on the work object. The work assistance system acquires first information having the position and movement of the work information for the work object in a the three-dimensional space when the worker performs work on the work object, has second information having the position and movement of the display according to the position and movement of the head of the worker in the three-dimensional space, calculates the position and movement of display of the work information for the work object on the screen of the display on the basis of the first information and the second information, and limits the movement speed in the field of view of the work information within an allowable range set according to the reference work when displaying the work information for the work object on the screen of the display.

Description

Work support system

[0001] This disclosure relates to the technology of work support systems.

[0002] Regarding the manufacturing and maintenance work of products such as semiconductor manufacturing equipment and charged particle beam equipment, reduction of man-hours is required. Against the backdrop of a shortage of workers, etc., early training of workers, etc. is required. For this purpose, it is considered effective to apply technologies such as AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality) to work to support work. For example, a display such as a head-mounted display or smart glasses is worn on the worker's head. On the screen of the display, as AR or the like, information / images for supporting work (which may be described as work information, work support information, etc.) are displayed. For example, work information in the form of AR video is information on a virtual model of a standard / reference work and information representing movements of hands, etc. Thereby, the worker can efficiently perform the work or learning while viewing the work information superimposed on the actual object such as the work target through the screen of the display.

[0003] As a prior art example, Japanese Unexamined Patent Application Publication No. 2020-144233 (Patent Document 1) can be cited. In Patent Document 1, as a learning support system, etc., there is a description to the effect that "the arithmetic unit 16 overlays the model video on the field-of-view video captured by the imaging unit 15 and displays it on the display unit 19, and dynamically changes the display content of the model video according to the characteristics of the work operation of the learner 102 included in the field-of-view video."

[0004] Japanese Unexamined Patent Application Publication No. 2020-144233

[0005] Regarding the manufacturing and maintenance work of the above products, when applying a work support technology that visualizes a standard work (in other words, a model work, an expert work, etc.) as work information / virtual model such as AR / VR on a display, there are the following problems.

[0006] Examples of target tasks include wiping (cleaning) the surface of objects such as vacuum equipment. For example, an AR object (such as a handprint) representing the hand position and movement of a skilled worker during a wiping operation could be created and displayed on a screen of a display device corresponding to the user's field of view. The worker would then perform or learn the wiping operation by looking at and following the AR object, such as a handprint.

[0007] However, from the perspective of the user (operator), when viewing work information on the display screen, such as a moving AR object like a handprint, the movement of the AR object may be fast, meaning its speed within the field of view may be high. In such cases, the operator may not be able to visually track the AR object and may miss it. In particular, when the work object is a relatively large device or the display screen is relatively small, the AR image containing the work information may not fit within the corresponding field of view. As a result, the operator may not be able to properly track and imitate the standard / exemplary work, leading to deviations from the standard / exemplary work and a decrease in work quality. Alternatively, learning the standard / exemplary work may not be achieved efficiently.

[0008] The purpose of this disclosure is to provide a technology that enables more suitable work support, particularly when standard work is visualized on a display device using work information / virtual models such as AR, as well as when standard work is visualized, making it less likely for workers to miss work information and easier to track standard work.

[0009] A typical embodiment of this disclosure has the configuration shown below. The embodiment is a work support system that provides work information to support a worker's work on a work object, and displays video information having the position and movement of a reference work on the work object in a three-dimensional space including the work object, superimposed on the work object on a screen corresponding to the worker's field of view on a display device worn on the worker's head, and when the worker performs the work on the work object, it acquires first information having the position and movement of the work information on the work object in the three-dimensional space, and has second information having the position and movement of the display device corresponding to the position and movement of the worker's head in the three-dimensional space, and calculates the position and movement of the display of the work information on the work object on the screen of the display device based on the first information and the second information, and when displaying the work information on the work object on the screen of the display device, it limits the in-field movement speed of the work information to be within an allowable range set according to the reference work.

[0010] According to a representative embodiment of this disclosure, regarding the technology for supporting work such as manufacturing and maintenance of the above-mentioned product, particularly when standard work is visualized on a display using work information / virtual models such as AR, it is possible to realize more suitable work support, such as making it less likely for workers to miss work information and making it easier to track standard work. Other issues, configurations, and effects will be shown in the embodiments for carrying out the invention.

[0011] This shows an overview of the configuration of the work support system in Example 1. Example 1 shows an example of a work model for a standard task (first example of a wipe operation). Example 1 shows an example of a work model for a standard task (second example of a wipe operation). Example 1 shows an example of the configuration of the display and control device of the work support system. Example 1 shows the processing flow of the work support system. Example 1 shows an explanatory diagram (part 1) regarding the movement speed within the field of view. Example 1 shows an explanatory diagram (part 2) regarding the movement speed within the field of view. Example 1 shows an explanatory diagram (part 3) regarding the movement speed within the field of view. Example 1 shows experimental results regarding the ease of performing the wipe operation. Example 1 shows experimental results regarding the left-right balance of the O-ring installation operation. Example 1 shows an example of AR object display. Example 1 shows an explanatory diagram (part 1) regarding the O-ring installation operation. Example 1 shows an explanatory diagram (part 2) regarding the O-ring installation operation. Example 1 shows an example of AR object display during the O-ring installation operation. Example 1 shows an example of the configuration of the tolerance range. Example 1 shows an explanatory diagram regarding the design of the tolerance range. Example 1 shows a detailed processing flow. Example 1 shows an example of the configuration of an AR object of a handprint. Example 1 shows an example of an AR object being displayed at the edge of the screen. Example 1 shows an example of displaying a trajectory as an AR object. Example 1 shows an example of display control of work evaluation information (control example 1). Example 1 shows an example of display control of work evaluation information (control example 2). Example 1 shows an example of display control of work evaluation information (control example 3). Example 1 shows other display examples of work evaluation information. Example 1 shows an example of screen area division configuration in Modification 1 of Example 1. Example 2 shows an example of limiting according to depth distance in Modification 2 of Example 1. Example 2 shows a detailed processing flow example. Example 3 shows an example of the screen GUI. Example 3 shows a detailed processing flow example. Example 3 shows steps S30 and S31 of the detailed processing flow. This section shows an example of the GUI for the screen in Example 4, and an example of setting an acceptable range according to proficiency level. Other configuration examples of AR objects are shown in each example. An explanatory diagram regarding the problems in the comparative example is also provided.

[0012] The embodiments of this disclosure will be described in detail below with reference to the drawings. In the drawings, the same parts are generally denoted by the same reference numerals, and repeated descriptions are omitted. In the drawings, the representation of components may not show their actual location, size, shape, extent, etc., in order to facilitate understanding of the invention.

[0013] In explanations, when describing program-based processing, the focus may sometimes be on the program, functions, or processing units. However, the main hardware component is the processor, or a controller, device, computer, or system composed of such a processor. The computer, using its processor, executes processing according to the program read into memory, utilizing resources such as memory and communication interfaces as appropriate. This realizes the specified functions and processing units. The processor is composed of semiconductor devices such as a CPU / MPU or GPU. Processing is not limited to software program processing; it can also be implemented using dedicated circuits. Applicable dedicated circuits include FPGAs, ASICs, CPLDs, etc.

[0014] The program may be pre-installed as data on the target computer, or it may be distributed as data to the target computer from the program source. The program source may be a program distribution server on a communication network, or a non-transient computer-readable storage medium, such as a memory card or disk. The program may consist of multiple modules. The computer system may consist of multiple devices. The computer system may consist of a client-server system, a cloud computing system, an IoT system, etc. Various types of data and information are composed of structures such as tables and lists, but are not limited to these. Representations such as identification information, identifiers, IDs, names, and numbers are interchangeable.

[0015] [Problems and Solutions] In the past, analysis of the work of skilled and unskilled workers in product manufacturing and maintenance tasks, such as wiping, revealed that there are differences in multiple motion elements, such as the position, direction, speed, and trajectory of hand movements during wiping. Similarly, in tasks such as O-ring installation, it was found that there are differences in multiple motion elements, such as the height position of the left and right hands, the speed of movement, and the degree of balance and consistency between the left and right hands.

[0016] However, for unskilled workers, it is difficult to perform the work while considering all of the above multiple operational elements. As a result, the work may deviate from the standards / exemplary work of skilled workers, sometimes leading to the occurrence of foreign matter or leaks. In other words, the quality and efficiency of the work may decrease.

[0017] Therefore, the work support system of this embodiment creates AR data of work information (reference work information for work support, etc.) by creating a work model / virtual model of standard / exemplary work performed by skilled workers that achieve high-quality work.Then, the work support system of this embodiment provides AR images representing the standard / exemplary work as work information to workers, such as unskilled workers, who are wearing a display device, through the screen of the display device (AR device, e.g., head-mounted display) based on the AR data.In this embodiment, work information such as AR images is displayed superimposed on the actual objects such as the work space and the work object in the worker's field of view (FOV), in other words, on the screen of the display device corresponding to the field of view, in which the work space including the work object and the worker's hands and other body parts are visible.As the work space, in the three-dimensional space where the target work is performed, AR images based on the work model / virtual model of the standard / exemplary work are provided.The AR images, which are work information, are AR objects, etc. that represent hand movements, etc., during work, based on the work model / virtual model that satisfies multiple motion elements in the standard / exemplary work.

[0018] The worker views the AR image on the display screen and performs or learns their own work (sometimes referred to as actual work to distinguish it from the standard work) by following the AR object of the standard work. This function may be used during actual manufacturing / maintenance work, or during work learning and training. Through this AR-based work support, even unskilled workers can perform correct / optimal work by using the work of skilled workers as a standard / model and fulfilling multiple operational elements, thereby simplifying work or improving the efficiency of work learning.

[0019] However, when applying AR-based work support to the manufacturing and maintenance of semiconductor manufacturing equipment and other objects, the following challenges arise: The target equipment is relatively large, and the area where AR images can be superimposed is relatively wide. In contrast, the screen of the display device, which is associated with the user's field of view, i.e., the area where AR images can be displayed, is limited in size. It is necessary to superimpose AR images for work support within this limited field of view and screen. In this case, if the movement speed of AR objects on the screen is too fast from the user's perspective, it may become difficult for the user to track the AR objects, and they may be missed.

[0020] Figure 31 is an explanatory diagram of the problem in the comparative example. Let's consider the case of wiping the work surface 3A (e.g., the circular top surface) of the work object 3 (e.g., a vacuum device). In this case, an AR object 3101, such as a handprint mimicking a hand wiping, is displayed as an AR image representing the standard / exemplary work on the screen 3100 of the display device corresponding to the field of view of the user, who is the operator. For example, the handprint AR object 3101 is displayed to move along the movement line 3102 in the action of wiping the work surface 3A from the left edge to the right edge in one direction. The position of the screen 3100 also changes according to the position of the viewpoint and changes in the line of sight.

[0021] In this case, if the movement speed of the AR object 3101 within the screen 3100 is too fast from the user's perspective, it may become difficult for the user to track the AR object 3101, potentially leading to missed objects. For example, if the AR object 3101 is located outside the screen 3100, such as AR object 3101a or AR object 3101c, it may be missed. This makes it difficult for the worker to follow and perform a suitable task that is close to the standard task.

[0022] Therefore, the work support system of this embodiment provides work support functions using AR, etc., as described below. Note that the term AR may be used collectively to include not only AR but also technologies similar to AR.

[0023] In this embodiment, the work information, such as AR images, is displayed on the screen of the display unit based on the positional relationship between the worker's viewpoint (in other words, the position of their head and the position of the display unit) and posture, and the position and posture of the work information, such as AR images. In this embodiment, the movement speed of the work information, such as AR objects, within the field of view is limited to an acceptable range according to the work model of the target work, based on this positional relationship. For example, if the movement speed of the work information within the field of view is too fast, it is limited to below the upper limit, and if the movement speed of the work information within the field of view is too slow, it is limited to above the lower limit.

[0024] In other words, the work support system of this embodiment controls the movement speed of the AR object, etc., displayed within the field of view to be limited to an acceptable range set based on the work model, according to the positional relationship between the position of the user's (corresponding head and display) viewpoint and the position of the superimposed display of the AR object, etc., on the work object. The work support system of this embodiment displays work information such as AR objects, which change to move appropriately in chronological order as the standard work progresses, on the display screen, with the movement speed within the field of view appropriately limited to a predetermined acceptable range (in other words, the range from the lower limit to the upper limit). As a result, the worker can more easily track the work information within the field of view / screen and is less likely to miss anything.

[0025] <Example 1> The work support system of Example 1 will be described using Figure 1 and subsequent figures.

[0026] [Work Support System] Figure 1 shows the overall configuration of the work support system of Embodiment 1. The work support system of Figure 1 includes a display unit 1 worn on the head by a user U1, who is a worker, and a control device 2 that is communicated with the display unit 1. The display unit 1 and the control device 2 are interconnected by, for example, wireless communication WC, but the details are not limited to this.

[0027] User U1, the operator, is a person who performs or learns a predetermined task on the work object 3, for example, a vacuum device. The predetermined task is, for example, a wiping operation. The work object 3, the vacuum device, has, for example, a cylindrical housing (in other words, a vacuum chamber or chamber). User U1 performs a wiping operation on the work surface 3A (in other words, the target surface; for example, the circular top surface) at a predetermined height position on the work object 3. The wiping operation is the operation of wiping a circular area of ​​the work surface 3A with a cloth 5 (which has alcohol attached) held in the hand.

[0028] In other words, the display device 1 is a user terminal, wearable device, etc. In this embodiment, the display device 1 is a head-mounted display or smart glasses. Based on control from the control device 2, the display device 1 displays images and videos (collectively referred to as AR videos 4) as AR or VR on the screen (in other words, the display surface) of the display device 1.

[0029] In particular, this AR image 4 is work information 4 for work support. This work information 4 is image and video information such as work models / virtual models of standard / exemplary work. This work information 4 is, for example, an AR object such as a handprint representing the hand movements of a standard work. The standard / exemplary work is, for example, a work performed by an expert, or a work model constructed based on statistics of the work histories of multiple workers. User U1, who is the worker, performs or learns the target work, which is the wipe work, by visually tracking the handprint, etc. of the work information 4 that is superimposed on the work space including the work object 3.

[0030] The control device 2 is, in other words, a display control device, controller, computer system, etc. The control device 2 transmits control information / display information 6 to the display unit 1. The control information / display information 6 includes data for configuring and displaying the AR image 4, which is the work information 4. The display unit 1 configures and displays the AR image 4, which is the work information 4, on the screen according to the control information / display information 6.

[0031] Furthermore, the display unit 1 can transmit control information / display information 8 to the control device 2 as needed. The control information / display information 8 is information that represents the state of the display unit 1, etc., necessary for AR display control, i.e., information that represents the position and orientation of the viewpoint. The display unit 1 can also detect the position of the user U1's hands during work that overlap the screen based on sensors, and may transmit the detected hand position etc. as control information / display information 8 to the control device 2. The control device 2 can grasp the state of the display unit 1 based on the control information / display information 8 and create control information / display information 6.

[0032] The control device 2 includes a function to control the display of the AR image 4 on the display unit 1 (an AR display control unit 21 corresponding to that function), a function to create and manage AR data such as a work model that forms the basis of the AR image 4 (an AR creation and management unit 22 corresponding to that function), and a database (DB) 23 that holds the AR data such as the work model. More details are shown in Figure 3.

[0033] In the embodiment shown in Figure 1, the display unit 1 and the control device 2 are configured as separate components. However, the system is not limited to this configuration, and the display unit 1 and the control device 2 may be integrated into a single unit. In other words, the functions of the control device 2 may be implemented by the processor or other components of the display unit 1.

[0034] The work information 4 / work model used as AR images / videos / content is created and set to satisfy one or more motion elements / conditions in the standard / exemplary work, such as hand position, orientation, speed, trajectory, distance, time, force, etc. The AR video / AR object has a three-dimensional and time-series position and movement that is superimposed on the work object 3 in three-dimensional space (corresponding work space).

[0035] The method and details of creating the work information 4 / work model / AR data in Figure 1 are not limited. Various methods of creation are possible. For example, AR data for a reference model may be created based on the work history data of a skilled worker. Motion capture or other technologies may be used in this case. Another example is that AR data for a reference model may be created based on statistics such as the average of hand movements from the work history data of multiple workers. It may also be created as a completely virtual work model separate from the actual work history.

[0036] The display method of display device 1 is not limited; it may be a known transparent type, an opaque type (in other words, a VR type), or any device capable of AR display. The transparent type (in other words, a see-through type) is a type in which AR images are superimposed on the real-world scene that can be seen through the screen. The opaque type is a type in which the screen itself is opaque and AR images are superimposed on the captured video of the real-world scene.

[0037] For explanatory purposes, the following coordinate systems, directions, and axes may be used: For the three-dimensional space that is the workspace containing object 3, the spatial coordinate system, or in other words, the world coordinate system, is (X, Y, Z). For the coordinate system on the display unit 1 screen, or in other words, the display coordinate system, is (x, y, z).

[0038] [Wiping Operation] Figure 2A shows the first example of a work model for a standard operation related to wiping. Figure 2A is an X-Y plan view of the work surface 3A from above. The arrow (movement line) 2A01 is a line / vector representing the direction and trajectory of the wiping operation, and represents the direction, etc., when the worker wipes the work surface 3A with the cloth 5 held in their hand. In the virtual model for AR, for example, the hand is represented by an AR image 4 of a handprint 7. In the standard / exemplary operation, the handprint 7 moves at a predetermined speed along the direction and trajectory of the arrow 2A01. The handprint 7 is another name for a hand gesture, etc.

[0039] In the first example of FIG. 2A, the reference / standard operation is composed of repeating the operation of wiping a circular area, which is the working surface 3A, from one end to the other end, for example, in a straight line direction from left to right (arrow 2A01). In other words, this reference / standard operation is composed of an operation such as line sequential scanning. In order not to drop dust on the working surface 3A, such an operation of wiping in one direction is effective.

[0040] As a specific example, this operation is, for example, when wiping arrows 2A02 and 2A03, the operations shown in (1) to (4) in order. First, a hand (corresponding hand shape 7) is placed at the position shown in (1), that is, the left end of arrow 2A02. This hand (corresponding cloth 5) is in contact with the working surface 3A. The hand shape 7 moves to the right while remaining in contact with the working surface 3A along arrow 2A02 and moves to the right end of arrow 2A02; the hand shape 7 extends outside beyond the outer periphery of the working surface 3A at the right end of arrow 2A02 as shown in (2). At this time, the hand (corresponding cloth 5) is in a non-contact state with the working surface 3A. The hand (corresponding hand shape 7) moves in an arc outside the outer periphery of the working surface 3A while remaining in a non-contact state and comes to the position shown in (3), that is, outside the left end of the next arrow 2A03. The hand (corresponding hand shape 7) contacts the left end of the next arrow 2A03, moves to the right end, and the hand (corresponding hand shape 7) extends outside beyond the outer periphery of the working surface 3A at the right end of arrow 2A03 as shown in (4). The same repetition applies to the flow lines of other rows.

[0041] In the above wiping operation, since the wiping direction is in one direction from left to right and the hand extends outside the working surface 3A and then moves to the next row, it is possible to prevent the dust attached to the cloth 5 from being dropped / left on the working surface 3A.

[0042] Similarly, FIG. 2B shows a second example of the working model of the reference operation of the wiping operation. In the second example of FIG. 2B, the reference operation is composed of repeating the operation of wiping a circular area, which is the working surface 3A, in the circumferential direction as shown in the figure, for example, in the circumferential direction of the right turn (arrow 2B01). In other words, this reference / standard operation is composed of a spiral operation. By moving in one direction on the circumference, it is possible to prevent dust from being dropped / left on the working surface 3A.

[0043] This operation, as a specific example, when wiping the arrows (flow lines) 2B01 and 2B02, for example, is the operation shown in (1) to (4) in order. First, the hand (corresponding hand shape 7) is placed at the position shown in (1), that is, the upper end of the arrow 2B01. The hand (corresponding cloth 5) is in contact with the working surface 3A. The hand (corresponding hand shape 7) moves along the arrow 2B01 in an arc while remaining in contact with the working surface 3A, and moves from the position of the lower end of the arrow 2B01 shown in (2) to the position of the upper end of the arrow 2B01 shown in (3). Next, as shown in (4), the hand (corresponding hand shape 7) moves to the upper end of the next arrow 2B02, and similarly moves on the circumference of the arrow 2B01 as shown in (5) and (6). At these times, the hand (corresponding cloth 5) remains in contact with the working surface 3A. The same repetition applies to other circumferences. Although the case of transitioning from the outer circumference to the inner circumference is shown, the reverse is also possible.

[0044] In the AR video image 4 of FIG. 1, the hand shape 7 as in the above example is displayed while moving. The user U1 performs a wiping operation while visually following the hand shape 7 and using the movement of the hand shape 7 as a reference / model. However, as also shown in FIG. 31 described above, when the movement of the hand shape 7 is too fast or the like, the user U1 may not be able to follow the hand shape , and the hand shape may go out of the screen and become invisible.

[0045] Also, for example, in the wiping operation of FIG. 2A, the working position of the operator during the operation (including positions such as the head) is assumed as a reference position. For example, the working position 2A04 is a position in front of the working surface 3A and a position on the front side in the Y direction. In a certain working model, it is fixed at this one working position 2A04, and from this working position 2A04, the operator moves the hand to wipe the entire working surface 3A. In other working models, the working position may be made changeable.Depending on the size of the working surface, for example, the hand may not reach the entire area from one working position. For example, together with the working position 2A04, a working position 2A05 on the opposite side may be provided, and at these positions, the front and rear areas of the working surface may be shared.

[0046] [Display and Control Device] Figure 3 shows an example configuration of the display device 1 and control device 2 shown in Figure 1. The display device 1 (for example, a head-mounted display: HMD) includes a control unit (including a processor) 101, a storage unit (including memory) 102, a display unit (including a display surface) 103, a communication interface 104, a sensor 105, etc. The control unit 101 executes processing according to the program read by the processor into the memory of the storage unit 102. This enables various functional units. One of the functional units is an AR display function that displays AR images on the screen (display surface) of the display unit 103.

[0047] The communication interface 104 performs communication processing with the control device 2 and receives and acquires control information / display information 6 from the control device 2. In addition, the communication interface 104 can transmit control information / display information 8 from the display 1 to the control device 2 as needed.

[0048] The sensors 105 include a gyro sensor, accelerometer, compass sensor, GPS receiver, camera, and distance sensor, and at each point in time, they detect the state of the display unit 1 (corresponding to the head or viewpoint), such as its position, orientation, and posture, as well as the state of the distance to an object (such as the work object 3 or the user U1's hand). The sensor detection information may also be included as part of the control information / display unit information 8.

[0049] The control unit 101 constructs AR video data based on the control information / display information 6 acquired from the control device 2, stores it in the storage unit 102, and controls the display unit 103 to display the AR video 4 on the screen (display surface) of the display unit 103.

[0050] The display unit 1, although not shown in the diagram, includes other components such as a microphone, speaker, button controls, and remote control controls. In addition to displaying the AR image 4, the display unit 1 can also output audio through the speaker. The display unit 1 can also receive voice input from user U1 via the microphone and receive instructions based on voice recognition. The display unit 1 can also receive operation input from buttons and remote controls (not shown) by user U1.

[0051] The control device 2 comprises a control unit 201, a storage unit 202, a communication interface 204, an input / output unit 205, etc. The control unit 201 includes a processor, etc., and controls the control device 2. The storage unit 202 includes a memory, etc., and stores various data and information used for control. The communication interface 204 performs communication processing with the display unit 1, transmits control information / display information 6 to the display unit 1, and receives and acquires control information / display information 8 from the display unit 1. The input / output unit 205 has input devices and output devices connected to or built in, and accepts operation input from the user (which may be user U1 or an administrator, etc.) and outputs information to the user.

[0052] The control unit 201 executes processing according to the program read into the memory of the storage unit 202. This enables the realization of various functional units. In addition to the AR display control unit 21 and AR creation / management unit 22 shown in Figure 1, the functional units in Figure 3 include a work information acquisition unit 24, a display status acquisition unit 25, a field of view movement amount determination unit 26, a field of view movement speed limiting unit 27, a work process management unit 28, a worker information setting unit 29, a work evaluation unit 30, and the like.

[0053] The AR display control unit 21 controls each unit, including the AR creation and management unit 22. The AR creation and management unit 22 creates and sets AR data 23A such as work models / virtual models, and stores and manages it in the DB 23. An example of AR data 23A is data and information representing the position, trajectory, speed, etc., of the handprint 7, etc., of a work model of a standard operation as shown in Figure 2A. The AR data 23A also includes setting information necessary for control, such as tolerance ranges 23B.

[0054] The work information acquisition unit 24 references and acquires AR data 23A and related tolerance ranges 23B, etc., from DB 23, which are necessary for the configuration and display of the work information 4 of the target work.

[0055] The display status acquisition unit 25 acquires control information / display information 8 from the display 1, which represents the status and specifications of the display 1. The specifications of the display 1 include information such as the screen size.

[0056] The field of view movement amount determination unit 26 calculates and determines the field of view movement amount described later, based on the necessary AR data (i.e., data representing the display content of the bill of exchange 7, etc.) acquired using the work information acquisition unit 24 and data such as the state of the display unit 1 acquired using the display unit state acquisition unit 25, by performing coordinate system transformations, etc. This field of view movement amount determines the basic information when there are no restrictions as described below.

[0057] The field of view movement speed limiting unit 27 performs a process to limit the field of view movement speed, as described below, as necessary, with respect to the field of view movement amount calculated and determined by the field of view movement amount determination unit 26. The field of view movement speed limiting unit 27 determines the limited field of view movement speed according to the allowable range 23B associated with the AR data 23A. In other words, the field of view movement speed limiting unit 27 corrects the field of view movement speed corresponding to the field of view movement amount in accordance with the limit, and determines the corrected field of view movement speed. The AR display control unit 21 creates control information / display information 6 that instructs the display content (AR image 4) to be such that the field of view movement speed determined by the field of view movement speed limiting unit 27 is the field of view movement speed, and transmits the control information / display information 6 to the display unit 1 to display the AR image 4.

[0058] The work process management unit 28, etc., will be used in the embodiment described later. The work process management unit 28 stores and manages work process information related to the target work as shown in Figure 2A in DB 23.

[0059] The worker information setting unit 29 will be used in the embodiment described later. The worker information setting unit 29 sets and manages information about the worker (user U1) who performs / learns the work in DB 23. In particular, the worker information setting unit 29 can set and specify the worker's work proficiency level (i.e., information representing the degree of work proficiency). This proficiency level may be expressed as several stages, levels, classes, etc., representing the degree of proficiency. For example, beginner, intermediate, advanced, etc.

[0060] The work evaluation unit 30 performs an evaluation process on the status and results of the work performed by the worker (user U1), and calculates and stores the evaluation result, which includes an evaluation value for the work, a proficiency level, or work warnings and work guidance (collectively referred to as work evaluation information). The evaluation value / proficiency level is a value that represents the quality of the work and the degree of proficiency, and indicates whether it is close to or far from a standard work. Here, the work evaluation unit 30 calculates an evaluation value for each work (a value that indicates whether it is close to or far from a standard work), distinct from the proficiency level set in the worker information setting unit 29. It is also possible to set the proficiency level calculated as an evaluation value by the work evaluation unit 30 as the proficiency level in the worker information setting unit 29. Furthermore, as will be described later, the work evaluation information can also be displayed as an AR image 4.

[0061] Although not shown in the diagram, the control device 2 also has other functions, such as a work history function. When an operator performs actual work, the control device 2 monitors the status of the work based on sensor detection by the display device 1 and records it in DB 23 as work history data. Based on the work history data, the control device 2 displays an AR video 4 representing a reproduction of the actual work at any time, such as when instructed by user U1. User U1 can view the AR video 4 to confirm hand movements, etc., during the actual work. Furthermore, as will be described later, the control device 2 can calculate the difference between the reference work and the actual work in real time during the actual work, or at any time thereafter, based on the work history data, and display an AR video 4 representing the difference.

[0062] [Processing Flow] Figure 4 shows the basic processing flow of the work support system, corresponding to the configuration in Figure 3. In this embodiment, the display movement speed of the AR object within the user U1's field of view, or in other words, within the screen of the display unit 1, is calculated based on the movement state of the AR object of the reference work model in three-dimensional space and the movement state of the user U1's viewpoint (corresponding display unit 1), and the AR object is displayed at the calculated position on the screen. When displaying, the movement speed of the AR object within the field of view is appropriately limited (in other words corrected) based on the allowable range 23B set in correspondence with the work model.

[0063] In step S1, the work support system (particularly the control device 2) sets a work model for the target work. The AR creation and management unit 22 of the control device 2 in Figure 3 creates a work model (for example, Figure 2A) and sets it in DB 23 as AR data 23A. A user, such as an administrator, may also set the AR data 23A based on their input. Once the settings in step S1 are made, the work support function using AR in this embodiment becomes available.

[0064] In step S2, user U1 prepares by attaching the display unit 1. The work support system receives an instruction from user U1 to start using the work support function. Upon receiving this instruction, the work support system, in particular the AR display control unit 21 of the control device 2, turns the work support function ON (enabled).

[0065] In step S3, the control device 2 acquires AR data 23A and the like related to the work model of the target work from DB 23 using the work information acquisition unit 24. The AR data 23A defines the movement of the AR object (e.g., the handprint 7) in a spatial coordinate system including the work object 3 (e.g., the coordinate system (X, Y, Z) in Figure 2A).

[0066] In step S4, the control device 2, using the display state acquisition unit 25, acquires control information / display information 8 representing the state of the display 1 via communication from the display 1, and grasps the current state of the display 1 (position, orientation, posture, etc.) from this information. The current state of the display 1 corresponds to the state from the user U1's viewpoint, and this state is described in a spatial coordinate system that includes the work object 3.

[0067] In step S5, the control device 2 calculates and determines the amount of movement within the field of view for the AR object to be displayed as work information 4 using the in-field movement amount determination unit 26. The in-field movement amount determination unit 26 calculates the amount of movement within the field of view from the work information (AR data 23A) obtained in step S3 and the display state (in other words, viewpoint state) obtained in step S4. This amount of movement within the field of view, as will be described later, is the amount of movement within the screen of the display unit 1 corresponding to the user U1's field of view, and is described in a coordinate system (display coordinate system) based on that field of view (the screen of the display unit 1). Note that the amount of movement within the field of view in step S5 is still basic information before limitations (in other words, before correction).

[0068] In step S6, the control device 2 uses the field of view movement speed limiting unit 27 to calculate the field of view movement speed (field of view movement speed using the angle θ described later) based on the amount of movement within the field of view obtained in step S5, and performs necessary limiting processing on the field of view movement speed based on the allowable range 23B. As a result, if a limit is applied, the field of view movement speed limiting unit 27 determines the field of view movement speed after the limit (in other words, after correction).

[0069] In step S7, the control device 2 determines the work information (AR image) 4 to be displayed based on the information obtained up to step S6, creates control information / display information 6 for displaying the work information (AR image) 4 on the display device 1, and transmits it to the display device 1. The display device 1 displays the work information (AR image) 4 on the screen based on the received control information / display information 6.

[0070] Step S8 is a confirmation of whether to continue the series of processes described in steps S3 to S7 on the timeline. If it is to continue (YES), it returns to step S3, and the process at the next point in time is repeated in the same way. If it is not to continue (NO), for example, if a termination instruction is received from user U1, it proceeds to step S9, in which step S9 the AR display control unit 21 of the control device 2 turns the work support function OFF (disables) and terminates the work support.

[0071] [In-Field of View Movement Speed ​​(1)] Figure 5 is an explanatory diagram of the concept of in-field of view movement speed. The triangle-shaped FOV (Field of View) 500 is the field of view of the display unit 1 screen, which corresponds to the field of view of user U1, the worker, and is constant according to the specifications of the display unit 1. The work object 3, hand, and AR image 4 will be displayed within the field of view 500. The viewpoint 510 is the viewpoint position of user U1 and corresponds to the reference position of the display unit 1. In Figure 5, the coordinate system (display coordinate system) is such that the horizontal direction (screen horizontal direction) within the screen of the display unit 1 is the x direction, the vertical direction (screen vertical direction) within the screen is the z direction, and the depth direction is the y direction. The case of a handprint 7 is shown as an example of an AR object that becomes the AR image 4.

[0072] The movement amount (movement amount within the field of view) 501 is the movement amount (first movement amount) of the handprint 7 (7A) in the horizontal direction (x direction) within the screen when the handprint 7 (7A) is displayed as an AR image (work information) 4 on the screen of the display unit 1, and is the movement vector from the source (point a1) to the destination (point a2). Movement amount 501 is an example in Case A, where the distance in the depth direction from the viewpoint 510 is D1. Similarly, the movement amount (movement amount within the field of view) 502 is the movement amount (second movement amount) of the handprint 7 (7B) in the horizontal direction (x direction) within the screen, and is the movement vector from the source (point b1) to the destination (point b2). Movement amount 502 is an example in Case B, where the distance in the depth direction from the viewpoint 510 is D2.

[0073] The amount of movement within the field of view per unit of time corresponds to the velocity of movement within the field of view. In this example, movement amounts 501 and 502 are the same amount (distance ΔX). For example, if the movement amounts 501 and 502 shown in the figure are (e.g., [m; meters]) in 1 second, the velocity of movement within the field of view [m / s] is ΔX [m / s]. This amount (distance) can be appropriately converted to the number of pixels on the screen, etc.

[0074] In Case A, the amount of movement 501 at depth distance D1 is the same as the amount of movement 502 at depth distance D2 in Case B, both in terms of in-field movement and in-field movement speed. Even in this case, depending on the difference in depth distance, the distance and time that the handprint 7, which is the AR image 4 (AR object), remains within the field of view 500 is relatively shorter when the positional relationship is at distance D2 compared to when the positional relationship is at distance D1. Thus, when displaying an AR object at the same in-field movement speed regardless of depth distance, the handprint 7 (7B) at distance D2, which is closer to the viewpoint 510, is particularly likely to go outside the field of view 500 from the user U1's perspective. Parts 521 and 522 are examples of parts of the handprint 7B that are outside the field of view.

[0075] Therefore, in cases like Case B, compared to cases like Case A, it may be difficult for User U1 to visually track the AR image 4, which is the handprint 7 (7B), and may cause them to miss it. In other words, it may be difficult for User U1 to track the reference work. On the other hand, even if the movement amount is 501 as in Case A, if the movement speed within the field of view is too slow, it may be difficult for User U1 to grasp the speed of the reference work, which may be frustrating or inefficient.

[0076] Therefore, in this embodiment, the in-field movement speed of the AR image 4 (AR object) is limited to within an acceptable range (acceptable range 23B in Figure 3). The acceptable range defines at least an upper limit (at least one threshold) for the in-field movement speed according to the work model of the target work. The control device 2 calculates the in-field movement amount such as the movement amount 502 (in other words, the basic in-field movement speed before limitation) and limits this in-field movement speed according to the acceptable range 23B. That is, the control device 2 determines the in-field movement speed after limitation and controls the display so that the AR image 4 (handprint 7) in the field of view 500 does not move too fast or too slow in accordance with that in-field movement speed.

[0077] In Figure 5, the angle θ(θ1, θ2) is the angle corresponding to the amount of movement within the field of view (ΔX) (in other words, the field of view angle), and its unit is, for example, degrees (°). Angle θ1 is the angle corresponding to the amount of movement 501 when the depth distance D1 is reached. Angle θ2 is the angle corresponding to the amount of movement 502 when the depth distance D2 is reached. The movement speed within the field of view can be expressed using this angle θ, and in that case the unit is degrees (°) / second (s). The control device 2 limits the display speed within the field of view, which is the display speed of the AR object (handprint 7) within the field of view 500, by the angle θ corresponding to the amount of movement within the field of view.

[0078] In this example, in the x-direction movement of the handprint 7B at a depth distance D2 in Case B, if the amount of movement 502 before restriction (in other words, before correction) is ΔX, the in-field movement speed is restricted compared to the allowable range. After restriction, the in-field movement speed (third movement speed) becomes a value less than or equal to the upper limit of the allowable range. Within the field of view 500, the restricted in-field movement speed (third movement speed) is applied to the area excluding the parts 521 and 522 that are outside the field of view. As a result, user U1 can more easily track the handprint 7B in the AR image 4, making it easier to perform or learn tasks effectively.

[0079] The example in Figure 5 shows the case where the AR image 4 (handprint 7) moves horizontally (in the x-direction) within the field of view 500 (corresponding screen). However, the same limitation on the movement speed within the field of view can be applied when the AR image 4 moves vertically (in the z-direction) within the field of view 500 (corresponding screen), that is, when it moves freely in a two-dimensional direction.

[0080] [In-Field of View Velocity (2)] Figure 6 is a supplementary explanatory diagram regarding the calculation of in-field of view velocity in Figure 5. In Figure 6, Case A and Case B are shown separately. The in-field of view velocity expressed using the angle θ is denoted as V [degrees / second]. Consider a triangle formed by the viewpoint 510 and the amount (distance) of movement of the AR image 4 (the handprint 7, which is the AR object) in the x direction. Here, for simplicity, we assume that the handprint 7, which is the AR object, moves perpendicularly across the position in front of the viewpoint 510 (point p0), particularly in the x direction. In this case, the velocity corresponding to the linear amount of movement ΔX of the handprint 7 in the x direction is denoted as C [m / s]. For example, in Case A, the velocity is 601 corresponding to the amount of movement 501, and in Case B, the velocity is 602 corresponding to the amount of movement 502.

[0081] Regarding the amount of movement ΔX, the distance to the starting point on the left (point a1 or point b1) and the distance to the destination on the right (point a2 or point b2) are the same xa, relative to the front position (point p0). Consider a right triangle formed by the viewpoint 510, the front position (point p0), and, for example, the destination position (for example, point a2 or point b2). From this, the in-field movement velocity can be calculated based on formulas such as tan.

[0082] The in-field movement speed V [degrees / second], which corresponds to the movement speed of an AR object (e.g., a handprint 7) seen within the user U1's field of view 500 (the screen of the corresponding display 1), can be calculated using the angle θ as follows. For example, in case A, using the depth distance D1 and the movement speed C (601), the in-field movement speed V1 is V1 = 2 × tan －１ It can be calculated as (C / 2 × 1 / D1). Similarly, in case B, using the depth distance D2 and the movement speed C(602), the in-field movement speed V2 is V2 = 2 × tan －１ It can be calculated as (C / 2 × 1 / D2). The resulting V1 and V2 are V2 > V1. In other words, from the user U1's perspective, the shorter the depth distance, the faster the in-field movement speed V is perceived to be. A more accurate calculation of the in-field movement speed V using the angle θ is shown in Figure 7.

[0083] [In-Field of View Movement Velocity (3)] Figure 7 is a supplementary explanatory diagram detailing the in-field of view movement velocity in Figures 5 and 6. In state A of Figure 7, the angle θ that constitutes the amount of movement within the field of view and the in-field of view movement velocity is conceptually shown in three dimensions. Here, the angle θ is divided into two components: the x-direction (horizontal direction of the screen: let's call it the h-direction) and the z-direction (vertical direction of the screen: let's call it the v-direction), and we consider angles θh and θv. Plane 710 is an arbitrary plane with the direction of the field of view (direction directly in front) from the viewpoint 510 of the observer, user U1, as its normal vector, and is different from the FOV (field of view 500). The point where the normals intersect is p0(700). The depth distance from the viewpoint 510 to the point p0(700) in the front position is Dy. Note that in Figure 7, the viewpoint 510 is shown as a single, fixed viewpoint. The position of the viewpoint 510 in three-dimensional space is E1(Xe1, Ye1, Ze1). Since the user U1's head and the display unit 1 can move, the viewpoint 510 can also move.

[0084] In this example, we assume that an AR object 7a, such as a feature point, is moving from the position of point p0 (700) in the +x direction (rightward) and the -z direction (downward). For example, consider three time points t0, t1, and t2 and three corresponding positions 700, 701, and 702. Positions 700, 701, and 702 are examples of positions on the movement trajectory of the AR object 7a. Position 700 at time point T0 has position coordinates (X0, Y0, Z0) in three-dimensional space (a spatial coordinate system including the work object 3). Similarly, position 701 at time point T1 has position coordinates (X1, Y1, Z1) in three-dimensional space, and position 702 at time point T2 has position coordinates (X2, Y2, Z2) in three-dimensional space.

[0085] The angle θ of an AR object's movement per unit time is given by angle θh as the component in the h direction (x direction) and angle θv as the component in the v direction (z direction). For example, angles θh1 and θv1 correspond to the amount of movement from position 700 to position 701 (Dx1, Dz1 in state B) in the unit time from time T0 to time T1. Similarly, angles θh2 and θv2 correspond to the amount of movement from position 701 to position 702 (Dx2, Dz2 in state B) in the unit time from time T1 to time T2. In this way, we can consider the angle θ(θh, θv) at each time point (or between time points) (step S15 in Figure 15 described later).

[0086] State B represents movement in the x-z plane corresponding to plane 710. For example, the amount of movement (distance) in the x direction (h direction) from point p0 (position 700) to position 701 is Dx1, and the amount of movement (distance) in the z direction (v direction) is Dz1. Although displacement of the AR object is also possible in the y direction, which is the depth direction, this will not be explained. In this embodiment, the limitations on movement speed within the field of view apply to the h direction (horizontal direction of the screen) and the v direction (vertical direction of the screen), but exclude the y direction (depth direction). However, as a variation, the y direction (depth direction) may also be included in the limitations.

[0087] Based on Figure 7, etc., the general method for calculating the amount of movement within the field of view and the speed of movement within the field of view is as follows. In step S3 of Figure 4 mentioned above, information such as the movement trajectory (in other words, the position at each point in time) of the AR object to be displayed (for example, AR object 7a) is obtained. This information is described in a spatial coordinate system including the work object 3, for example, (X, Y, Z). On the other hand, in step S4, information such as the state of the display unit 1, in other words, the state of the viewpoint 510, such as its position and orientation, is obtained. The state of the viewpoint 510 (display unit 1) is also described in a spatial coordinate system. Both the AR object 7a and the viewpoint 510 (display unit 1) move appropriately in three-dimensional space.

[0088] In order to display such a moving AR object 7a on the screen of the moving display unit 1 (viewpoint 510), it is necessary to convert this information into a representation in a coordinate system (display coordinate system, for example, (x, y, z) in Figure 7) based on the display unit 1 (viewpoint 510).

[0089] From the information in step S3, for example, the world coordinates of the target position of the AR object 7a, such as T1 (X1, Y1, Z1) of position 701 and T2 (X2, Y2, Z2) of position 702 can be obtained. Also, from the information in step S4, the world coordinates of the position of the viewpoint 510 (in other words, the camera), such as E1 (Xe1, Ye1, Ze1), can be obtained. Furthermore, from the information in step S4, the orientation and attitude (angles of each axis) of the viewpoint 510 (camera) can be obtained.

[0090] If information in these world coordinate systems is obtained, the angle θ(θh, θv) in Figure 7 can be calculated as follows. The processor of the control device 2 (especially the in-field movement amount determination unit 26) should perform the following calculation.

[0091] Step 1: The processor calculates a vector from the viewpoint 510 (camera) to the target AR object 7a. For example, the vector to position 701 is c1, and the vector to position 702 is c2.

[0092] Step 2: The processor calculates transformation Q to convert the position of the AR object 7a in the world coordinate system to its position in the coordinate system of the viewpoint 510 (display 1). Transformation Q is a transformation to align the orientation of the viewpoint 510 with the axes (X, Y, Z) of the world coordinate system, and is obtained as the inverse transformation of the orientation of the viewpoint 510. Mathematically, transformation Q can be represented as a matrix.

[0093] Step 3: The processor calculates Q × c1. Q × c1 represents the AR object 7a at position 701 as seen from viewpoint 510. In other words, Q × c1 is the position of the AR object 7a on the screen of display 1.

[0094] Step 4: When Q × c1 = (x, y, z), the angle θ(θh, θv) can be found as follows: θh = tan －１ (x / y), θv=tan －１ (z / y).

[0095] The above calculations provide the position of the AR object 7a on the screen (display coordinate system). For example, T1(X1, Y1, Z1) of position 701 is converted to t1(x1, y1, z1) within the screen.

[0096] [In-Field Movement Speed ​​and Ease of Operation] Figure 8A is a supplementary explanatory diagram regarding the problem, and shows a graph of the results of an experiment conducted by the inventor. In this graph, the horizontal axis (presentation condition) represents the in-field movement speed of the AR object (work information 4) on display (x1 < x2 < x3 < x4), and the vertical axis (Easy to execution) represents the ease of operation. The ease of operation is an evaluation value by the worker (for example, out of 100 points). When the in-field movement speed is too fast, for example, as the movement speed increases to x3, x4, the ease of operation decreases to, for example, 50 points or less. If the in-field movement speed is too slow, the work time increases, and if the in-field movement speed is too fast, the ability to follow the standard model work decreases. It is effective to present the AR object to the worker at an appropriate in-field movement speed.

[0097] Figure 8B shows a graph illustrating an example of the displacement of the height position of the left and right hands holding the O-ring during the O-ring installation process described later (for example, Figures 10 to 12). In this graph, the horizontal axis represents time (frames), and the vertical axis represents the height position of the hands (position in the z direction). Comparing the displacement lines of the right hand 8B01 and the left hand 8B01, the height positions are roughly the same, but the displacement is different. However, for example, at time 8B03, the height position of the right hand 8B01 is higher than that of the left hand 8B02, meaning that the balance and degree of agreement are low. When the balance and degree of agreement between the right and left hands are low, it is difficult to achieve proper installation (Figure 11, described later).

[0098] [Display Modes and Control of AR Objects] Based on the work model shown in Figure 2A above, there are various possible modes for the AR objects to be displayed as work information 4, and examples are given below. First, in the example of Figure 2A, a handprint 7, which is modeled after the worker's hand, is displayed, changing color according to the state of work. The state of work includes a state in which the hand (corresponding cloth 5) is in contact with the work surface 3A of the work object 3 ("contact state") and a state in which it is not in contact ("non-contact state"). There are also states in which alcohol has been injected and applied to the cloth 5 and states in which it has not. Furthermore, there are states in which the amount of alcohol has evaporated and is low, based on the elapsed time since the alcohol was injected. As an example of controlling the change in display mode, in the contact state, the handprint 7 is displayed in blue; in the non-contact state, the handprint 7 is displayed in red (or gray, etc.); and in the alcohol injected state, the handprint 7 is displayed in green, and the alcohol evaporation state is indicated by the intensity of the color.

[0099] Figure 9 shows another example of how AR objects are displayed. Here, for simplicity, it shows a top-down, vertical view of the work surface 3A, as in Figure 2A, but in reality, it may also be possible to view the work surface 3A from an oblique angle, as in Figure 1. Screen 910 is the screen of the display unit 1 corresponding to the user U1's field of view 500.

[0100] Example 1 shows an example of displaying a handprint from a contour model, rather than a handprint from a skeletal model. The AR object of the handprint is displayed to move along the reference work path 900 (for example, the straight line corresponding to Figure 2A; not shown).

[0101] Example 2 is an example of displaying an AR object 902 (for example, roughly rectangular) that mimics a cloth (wipe cloth) 5, which is a tool used by the worker. The cloth object is displayed to move along the standard work route 900. In addition to tools, components that make up the work object 3 may also be displayed as AR objects.

[0102] Example 3 is an example of displaying a trajectory object (e.g., an arrow shape) 903 representing the movement path 900 of the standard work along the standard work path 900. The trajectory object 903 may be a simple line or a cloud of points formed by a sequence of positions at each point in time. The trajectory object 903 may remain displayed at the positions it has passed through, or it may disappear over time. The positions from the current point in time to several past points in time may also be displayed.

[0103] Example 4 is a similar example of displaying the trajectory object 904, where the entire trajectory of the work path 900 is displayed from beginning to end (especially when it matches the reference work path 900).

[0104] Example 5 uses an AR object 905 based on the handprint 7 of a skeletal model, similar to the previous example. The difference is that the size of the handprint 7 is changed, or in other words, enlarged or reduced, according to its position / distance in the depth direction (y-direction in the diagram) from the viewpoint of the field of view 500. An example is shown in the working model in Figure 2B. The handprint 905a shown on the left is the display when the handprint 7 is brought closer to the working surface 3A in a non-contact state, and is displayed at a size corresponding to the depth distance. The handprint 905b shown on the right is the display when the handprint 7 is in contact with the working surface 3A, and is displayed at a smaller size than the handprint 905a on the left, and in a different color.

[0105] In this embodiment, in addition to controlling the limit of movement speed within the field of view, the display content and display manner of the AR image 4, which is work information 4, may be dynamically changed according to the worker's work state, etc. As in the example above, the display manner, such as the color of the AR object, may be changed according to the work state. Particularly important operational elements to consider in wipe operations include the direction of the hand and whether or not it is in contact with the work surface 3A. Figure 2A shows a specific example where the color is changed according to whether or not there is contact. As another example, the direction of the hand in a standard operation may be defined as the direction from left to right as the reference direction, and the color may be changed when the direction of the worker's hand movement deviates from the reference direction.

[0106] [O-ring installation work] Figure 10 is an explanatory diagram of O-ring installation work as an example of other target work. Assume that the work object 3 is a vacuum device, as in Figure 1. The vacuum device has, for example, a flange component between the upper chamber and the lower chamber, and an O-ring is fitted into the groove of the flange component. In Figure 10, an O-ring 1002 is installed in a housing 1004 that constitutes part of the vacuum device. There is a member (flange component) 1003 on the upper part of the housing 1004. A ring-shaped groove 1003A is provided on the upper surface of the member 1003. The O-ring 1002 is fitted into the groove 1003A of this member 1003. The top view is a top view of the state in which the O-ring 1002 is fitted into the groove 1003A. Member 1001 is connected to the upper side of member 1003 in which the O-ring 1002 is fitted. In other words, the O-ring 1002 is a ring-shaped sealing material. The O-ring 1002 is a non-rigid member and is subject to bending and other deformations.

[0107] Figure 11 is an illustrative diagram of the work performed on the O-ring 1002 shown in Figure 10. As in state A, the worker grasps, for example, the two ends (parts) 1101 and 1102 of the O-ring 1002 with both hands (actual hands are shown in black). Since the O-ring 1002 is not rigid and is elastic, it actually flexes depending on the position of the hands, as shown in state B. Point A indicates the gripping position by the left hand corresponding to the left end 1101, and point B indicates the gripping position by the right hand corresponding to the right end 1102. Point C indicates the lower end due to flexing. The position of point C is lower than points A and B. State C is a state in which a part of the O-ring 1002 is inserted into a part of the groove 1003A of member 1003. The worker moves both hands, changing the gripping position as appropriate, and inserts the part below the gripping position into the groove 1003A.

[0108] In this type of O-ring installation work, it is preferable for the worker to install the O-ring 1002 uniformly in the horizontal direction, including the left-right direction (X direction) as shown in the figure. For this reason, it is preferable that the worker's left and right hands move at the same speed during the work. Also, in order to accurately install the O-ring 1002 into the groove 1003A, it is preferable that the height positions of the left and right hands are the same. Furthermore, the position in which the left and right hands grasp the O-ring 1002 and the position of the lower end of the O-ring 1002 relative to the groove 1003 are also important.

[0109] A skilled worker can accurately install the O-ring 1002 into the groove 1003A by considering multiple operational elements in such O-ring installation work (i.e., the height position of the left and right hands, the speed of movement, the gripping position, the lower end position, etc.). As a result, the vacuum device after installation will have a higher degree of sealing by the O-ring 1002, in other words, a higher vacuum level, and the probability of leakage can be reduced. Furthermore, a skilled worker can appropriately change the gripping position of the left and right hands on the circumference of the O-ring 1002 as the work progresses.

[0110] On the other hand, for unskilled workers, it is difficult to perform the task while considering all of these multiple operational elements simultaneously, resulting in a higher probability of leaks occurring. Condition D is an example of an undesirable working condition, where the positions and movement speeds of the left and right hands are different, in other words, there is poor balance between the left and right sides, and the O-ring 1002 is not properly fitted into the groove 1003A.

[0111] In this embodiment, AR can also be used to assist with O-ring installation work. For an unskilled worker, work information 4 based on a standard model of O-ring installation work is displayed (for example, Figure 12). The worker performs or learns the work in accordance with the work information 4. By following the work information 4, it becomes easier to simultaneously satisfy multiple motion elements such as the height position of the left and right hands, the speed of movement, and the gripping position. Furthermore, for example, by performing the work in such a way that the actual O-ring 1002 being gripped and the displayed AR object O-ring do not overlap, the worker can correctly recognize and follow the next work action through the AR object O-ring.

[0112] [O-ring installation work: AR display] Figure 12 shows an example of displaying an AR object (work information 4) with the O-ring installation work shown in Figures 10 and 11 as the work model. In Example 1, an AR object (O-ring object) 1201 that mimics the O-ring 1002 is displayed in the screen 1210 corresponding to the field of view 500. In addition, images such as points (for example, circular points A and B) representing the gripping positions are displayed on the O-ring object 1201 at position coordinates corresponding to the positions where it is preferable to grip with the left and right hands. Furthermore, an image such as a point (for example, triangular point C) representing the position corresponding to the lower end position, which is a suitable position for inserting into the groove 1003A, may be displayed. The worker performs or learns the work while following such an O-ring object 1201.

[0113] Furthermore, the following are examples of display control according to the working state. For example, when the AR object 1201 corresponding to the O-ring 1002 is in contact with the member 1003 (groove 1003A), it is displayed in blue, and when it is not in contact, it is displayed in red (or gray, etc.). It may also be displayed in green when the gripping position is being changed.

[0114] Furthermore, in the standard O-ring installation process, the O-ring can be accurately installed into the groove 1003A by slightly changing the gripping position of the O-ring with both hands. Therefore, in the AR display of O-rings / handprints, etc., this change in gripping position may be represented and displayed in the AR image 4. For example, when the gripping position is constant, points A and B are displayed in blue, and when the gripping position is being changed, points A and B are displayed in yellow. This allows for more appropriate work instruction.

[0115] The worker visually identifies the O-ring object 1201 and performs the task by moving their left and right hands to align them with, for example, gripping points A and B. This makes it easy to match / bring the height and gripping positions of the left and right hands to those of the standard operation (the corresponding skilled worker's movements). Then, while maintaining this state, the worker moves their left and right hands to follow the moving and changing O-ring object 1201. This allows the worker to proceed with fitting the O-ring 1002 into the groove 1003A while keeping the height and movement speed of both hands the same.

[0116] In this way, even inexperienced users can correctly perform optimal work actions that take multiple motion elements into consideration simply by focusing on following the AR object. As a result, the probability of leaks can be reduced, or efficient learning can be achieved. During the above work support, the color of the AR object also changes according to the work status, so inexperienced users can easily recognize the work status and perform optimal work.

[0117] Other examples of how the AR object may be displayed in the above O-ring installation operation are as follows. First, only a part of the O-ring object 1201 (corresponding O-ring 1002) in Example 1 may be displayed. For example, only points such as points A and B at the gripping position and point C at the lower end position may be displayed as an image. Alternatively, only a part of the circumference, for example, only the arc portion formed by points A, C, and B, including point C at the lower end, may be displayed as an image.

[0118] Example 2 in Figure 12 shows an example where AR objects 1202L and 1202R, which are handprints representing the left and right hands, are displayed at two gripping positions corresponding to points A and B on the left and right sides of the O-ring (not shown) in the work model / virtual model. The worker performs the task while following the AR objects 1202L and 1202R of the handprints.

[0119] Alternatively, both the O-ring object 1201 as in Example 1 and the handprint AR objects 1202L and 1202R as in Example 2 may be displayed simultaneously.

[0120] In other examples, the trajectory of the movement of the O-ring 1002 during standard operation, or the trajectory of the movement of the gripping position or lower end position, may be displayed as an AR object (trajectory object) representing the trajectory. In other examples, an image may be displayed in the ring-shaped area of ​​the groove 1003A indicating the location where the O-ring 1002 is to be inserted or where it has already been inserted.

[0121] Example 3 is an example in which, in a screen 1210 showing the groove 1003A from diagonally above, images of points A and B indicating the gripping position, an AR object 1203 indicating the location where the lower end of the groove 1003A is inserted, and an AR object 1204 indicating the location where the O-ring 1002 has been inserted in the groove 1003A are displayed.

[0122] In the example shown in Figure 12, the O-ring object 1201 and handprint objects 1202L and 1202R are displayed on the screen 1210, depending on the relationship between the size of the workpiece 3 (part 1003, etc.) and the size of the screen 1210. However, this is not limited to this example, and depending on the size relationship, only a portion of these may be displayed. The movement speed of these AR objects within the field of view presents challenges, similar to those in the wipe operation. Therefore, by similarly applying the function of limiting the movement speed within the field of view in this embodiment to the O-ring installation work (corresponding O-ring and handprint AR objects), more suitable AR display-based work support can be realized. For example, within the screen 1210 corresponding to the field of view 500, the movement speed of the AR objects for the two gripping positions of the left and right hands is limited. User U1 can adjust their work state, including their viewpoint position and left and right hands, so that the AR objects for the two gripping positions can be seen on the screen 1210, and can perform the work suitably while following the AR objects for the two gripping positions.

[0123] [Permissible range for limiting movement speed within the field of view] The work support system of this embodiment creates and sets AR data 23A of the work model in accordance with the target work as in the example above, and sets a permissible range 23B for limiting movement speed within the field of view.

[0124] Figure 13 shows a table (data table) of an example of setting the tolerance range 23B. The tolerance range 23B can be set for each work model of the target work. For example, as shown in row #1, for work model 1 where the target work is wipe work A (for example, Figure 2A), [TL1, TU1] is set as the tolerance range R1. TL1 is the lower threshold that constitutes the range, and TU1 is the upper threshold that constitutes the range. Similarly, the tolerance range can be set for each work model.

[0125] The lower part of the table shows examples of limitations when, for example, tolerance range R1 is applied. If the in-field movement speed of the AR object is, for example, V1, it is within tolerance range R1, so no limitation (in other words, correction) is applied. If the in-field movement speed is, for example, V2, it is outside tolerance range R1, in particular exceeding the upper threshold TU1. In this case, for example, the in-field movement speed of the AR object is limited to the same value as the upper threshold TU1. If the in-field movement speed is, for example, V3, it is outside tolerance range R1, in particular falling below the lower threshold TL1. In this case, for example, the in-field movement speed of the AR object is limited to the same value as the lower threshold TL1.

[0126] For wipe operations as shown in Figure 2A or Figure 2B, the following are examples of setting the tolerance range (R[TL,TU]). Further details will be provided later. In the horizontal direction of the screen (x-direction): [13.0 degrees / second to 23.5 degrees / second] In the vertical direction of the screen (z-direction): [13.0 degrees / second to 23.5 degrees / second]

[0127] For O-ring installation work as shown in Figures 10 to 12, the following are examples of setting the tolerance range (R[TL,TU]): In the horizontal direction of the screen (x direction), [1.9 degrees / sec to 5.1 degrees / sec] In the vertical direction of the screen (z direction), [3.0 degrees / sec to 8.3 degrees / sec]

[0128] The example settings in Figure 13 show a case where a fixed tolerance range is set for each work model of the target work. In this case, the same tolerance range is applied regardless of differences such as the depth distance from the viewpoint to the AR object. In other words, in this embodiment, the tolerance range is a static, fixed setting value depending on the target work model. The tolerance range is not limited to this. In modified examples, the tolerance range may be changed dynamically. In the modified examples described later, different tolerance ranges are applied depending on differences such as the depth distance from the viewpoint to the AR object.

[0129] [Designing the Tolerance Range for In-Field Movement Speed] Figure 14 is an explanatory diagram regarding the design of the tolerance range for in-field movement speed. The basis for the numerical values ​​of the tolerance range will be explained. The upper threshold of the tolerance range is designed based on the hand movement speed of the standard work. In this example, we will focus on a wipe operation (wipe operation A) as shown in Figure 2A. Based on the work model of wipe operation A (work model 1), the calculations for designing and setting the tolerance range (R1 [TL1, TU1]) as shown in Figure 13 for the in-field movement speed as shown in Figures 5 to 7 will be shown below.

[0130] As an experiment, the inventors had multiple workers perform the same wiping task and measured their hand movements. The results showed that the average hand movement speed in the horizontal direction (h direction, x direction) as seen from the worker was 40 cm / second.

[0131] This section specifically explains the allowable range (R1) of movement speed within the field of view in the horizontal direction (h direction, x direction), but the allowable range of movement speed within the field of view in the vertical direction (v direction, z direction) can be calculated and set in a similar manner.

[0132] State A in Figure 14 is a schematic diagram, in this case a Y-Z plane view, of the model of the work object 3 (vacuum device) and the human body model of the worker (user U1) in relation to work model A in three-dimensional space (workspace). Here, based on a general human body database, for example, the arm length is set to 78 cm and the distance from the shoulder to the eye to 19 cm. The height position of the work surface 3A is set to 0, and the height (Z1) from the work surface 3A upwards to the eye (viewpoint 510) is set to 68 cm. The angle (α1) made by the arm with respect to the work surface 3A, which is a horizontal plane (X-Y plane), is set to approximately 45 degrees. Depending on the height position of the work surface 3A and the worker's height, the worker may work standing or sitting.

[0133] State B is a top-down view of the working model in State A, in this case an X-Y plane view, and in particular a diagram relating to the angle θ from viewpoint 510 to view the AR object, the handprint 7. State B is a diagram for considering the upper limit of movement speed within the field of view. With the position of viewpoint 510 as 0, the depth distance from viewpoint 510 to the AR object is set to Y1 = 96 cm.

[0134] According to the calculation formula shown in Figure 6 above, the amount of movement of the AR object within the field of view [degrees] (let's call it M) based on the angle θ is as follows: M = 2 × tan －１ (40 cm / 2 × 1 / 96 cm) = 23.5 [degrees]. Using this in-field movement amount M, the upper limit threshold (TU1) of the acceptable range for the in-field movement speed V [degrees / second] in work model A can be calculated and defined. That is, TU1 = 23.5 [degrees / second].

[0135] Similarly, a lower threshold (TL1) can be calculated and set. State C is a diagram for considering the lower limit of the movement speed within the field of view. The position of viewpoint 510 is set to 0, and the depth distance from viewpoint 510 to the AR object is set to Y2 = 35 cm. The inventor conducted the following experiment. A worker was asked to follow an AR object displayed on the screen of a display device while seated. The AR object was displayed at several different movement speeds. As a result of the experiment, for example, when the AR object moved at a speed of 8 cm / second, the worker felt it was very slow. Based on this result, a lower threshold (TL1) for the movement speed V within the field of view can be set. For example, 2 × tan －１ (8 cm / 2 × 1 / 35 cm) = 13.0 [degrees / second].

[0136] These settings allow for a tolerance range (R1) in the horizontal direction (h direction) of the screen to be set to [TL1, TU1] = 13.0 to 23.5 [degrees / second]. The tolerance range in the vertical direction of the screen can also be set to the same value as the tolerance range (R1) in the horizontal direction.

[0137] State D is a supplementary diagram relating to the in-field movement speed in the vertical direction (v direction) of the screen. In State D, the vertical direction of the display unit 1 screen is z (v), the horizontal direction is x (h), and the depth direction is y in the display coordinate system. The angle (in other words, the field of view angle) when the handprint 7 moves in the vertical direction of the screen is denoted as φ, and the in-field movement amount M and in-field movement speed V are shown using angle φ. In this case as well, similar to State B, etc., the allowable range of the in-field movement speed V can be considered.

[0138] Furthermore, instead of being limited to the threshold values ​​of the tolerance range calculated above, a margin may be set to set a wider range. For example, if a margin of +20% is set for the upper limit threshold (TU1) of the tolerance range (R1) in the horizontal direction of the screen, it will be 23.5 × 1.2 = 28.2 [degrees / second]. If a margin of -20% is set for the lower limit threshold (TL1), it will be 13.0 × 0.8 = 10.4 [degrees / second].

[0139] For the O-ring installation work model shown in Figures 10 to 12, the tolerance range can be designed and set as follows. Based on the results of experiments conducted by the inventor, the average amount of hand movement in the vertical direction during O-ring installation by multiple workers was 5.1 cm. The distance from the viewpoint 510 to the hand was set to 35 cm. The average amount of hand movement in the horizontal direction was 3.125 cm.

[0140] Regarding the amount of movement within the field of view in the vertical direction of the screen, 2 × tan －１ (5.1 cm / 2 × 1 / 35 cm) = 8.3 [degrees]. The upper limit threshold for the allowable range of in-field movement speed in the vertical direction of the screen can be set to 8.3 [degrees / second]. If a margin of +20% is set, it can be set to 8.3 × 1.2 ≈ 10 [degrees / second]. For the amount of in-field movement in the horizontal direction of the screen, 2 × tan －１ (3.125 cm / 2 × 1 / 35 cm) = 5.1 [degrees]. The upper limit threshold for the allowable range of movement speed within the field of view in the horizontal direction of the screen can be set to 5.1 [degrees / second]. If a margin of +20% is set, it can be set to 5.1 × 1.2 ≈ 6 [degrees / second].

[0141] Similarly, lower threshold values ​​can be calculated and defined. For example, the lower threshold for in-field movement in the vertical direction of the screen is 8.3 × 4.6 / 12.5 ≈ 3 degrees / second. If a -20% margin is set, it becomes 2.4 degrees / second. The lower threshold for in-field movement in the horizontal direction of the screen is 5.1 × 4.6 / 12.5 ≈ 1.9 degrees / second. If a -20% margin is set, it becomes 1.5 degrees / second.

[0142] [Processing Flow (1)] Next, we will explain more detailed processing examples. Figure 15 shows a first configuration example of the detailed processing flow of the work support system of this embodiment. In this first configuration example, an acceptable range (upper threshold and lower threshold) for the in-field movement speed is set according to the work model, and if the in-field movement speed of the AR object (step S5 described above) falls outside the acceptable range, the in-field movement speed is limited so that it falls within the acceptable range. This flow corresponds to the calculation of the in-field movement speed using angles θh and θv as shown in Figure 7, and shows what processing is performed when the AR object (feature point or representative position) moves, for example, from position 701 at the first time point to position 702 at the second time point after a predetermined time.

[0143] In step S11, the control device 2 (particularly the processor) obtains information about AR objects from the AR data 23A of the target work model as source data for constructing work information 4. The information about AR objects includes information such as the position coordinates at each point in time in the spatial coordinate system including the work object 3. Also in step S11, the control device 2 extracts feature points of the AR objects.

[0144] Figure 16 shows an example of the configuration of feature points constituting a handprint 7, which is a type of skeletal model, as an AR object. In this example, the skeletal model of the fingers has feature points 1601 at the tips of each joint and bone, as shown by the black circles. Feature points 1601 are displayed as circular images in a predetermined color. There are also lines 1602 connecting the feature points 1601. Lines 1602 are also a type of feature point. Lines 1602 are displayed as line images in a predetermined color.

[0145] In the example shown in Figure 16, each feature point 1601 of the handprint 7 has a position coordinate. By controlling the position coordinate of each feature point 1601, the movement and shape of the hand in the AR object can be expressed in detail. However, the AR object may be a simpler object. For example, one AR object may have one reference position coordinate / representative position coordinate, such as a representative position 1603 based on the center of gravity.

[0146] In step S12, the control device 2 obtains the state of the viewpoint 510 from the display device 1, in other words, the state of the display device 1 such as its position, orientation, and posture.

[0147] In step S13, the control device 2 calculates the amount of movement (distance, displacement) and the speed of movement between the same feature points (or representative positions) of the AR object in the time series. The amount of movement here is, for example, in Figure 7, the amount of movement (e.g., Dx2, Dz2) between the position 701 (X1, Y1, Z1) of the AR object 7a at a certain first time point and the position 702 (X2, Y2, Z2) of the AR object 7b at the next second time point.

[0148] In step S14, the control device 2 determines whether the amount of movement and the speed of movement obtained in step S13 are equal to or greater than a threshold. The threshold here is set in advance. If the answer in step S14 is YES, proceed to step S15; otherwise, proceed to step S21.

[0149] In step S15, the control device 2 calculates the amount of movement within the field of view using angles θ in the horizontal (h) and vertical (v) directions, respectively, with respect to the amount of movement shown in Figure 7. As shown in Figure 7 above, the horizontal (h) component of the amount of movement within the field of view is θh, and the vertical (v) component is θv. Then, the angles relating to the displacement from the first position at the first time point to the second position at the second time point are θh1 and θv1, and the angles relating to the displacement from the second position at the second time point to the third position at the third time point are θh2 and θv2. In other words, at the first time point, it has (θh1, θv1), and at the second time point, it has (θh2, θv2).

[0150] In step S16, the control device 2 determines whether the amount of movement within the field of view {θh1, θv1, θh2, θv2} calculated in step S15 is within the range of the FOV (field of view 500) corresponding to the user U1's viewpoint 510 and the display 1. For example, in case A of Figure 5, it is within the range, while in case B (521, 522), it is outside the range.

[0151] If the answer in step S16 is YES, the process proceeds to step S17; otherwise, the process proceeds to step S23. In step S23, the control device 2 changes the in-field movement speed of the AR object from its original value to 0, and returns to step S11. In this case, as shown in the example in Figure 17, the AR object is displayed as if it has stopped at the edge of the screen of the display device 1.

[0152] Figure 17 shows a specific example corresponding to step S23. In the example in Figure 17, when the handprint 7 moves from left to right, it would normally go off-screen from the right edge of the screen 1700 corresponding to the field of view 500. However, by setting the field of view movement speed of the handprint 7 to 0, the handprint 7 stops at the right edge of the screen 1700. Therefore, compared to the case where the AR object goes out of the field of view, the operator is less likely to miss the AR object. The operator can recognize that the AR object is moving out of the screen 1700, and by directing their field of view in that direction, they can bring the AR object back into the screen 1700 and follow it. As shown in Figure 17, in this embodiment, the field of view movement speed of the AR object is limited to an acceptable range, and as an additional control example, if the AR object goes out of the field of view / screen, the field of view movement speed is set to 0. This makes the AR object appear to stay at the edge of the screen, preventing the AR object from disappearing from the screen.

[0153] Another example of display control is to change the display mode to distinguish between cases where the AR object is actually located exactly at the edge of the screen and cases where it is actually located outside the screen. For example, in a case like Figure 17, an image such as an arrow or a predetermined mark / icon / character that indicates the direction of movement of the handprint 7 or the direction of the handprint 7's original position may be displayed in addition.

[0154] In step S17, the control device 2 determines whether the distance between the working model and the AR object (distance in the depth direction from the viewpoint) is below a threshold. This threshold is set in advance. If the answer in step S17 is YES, the process proceeds to step S18; otherwise, the process proceeds to step S21. If the answer in step S17 is YES, it corresponds to a case where the depth distance is relatively small, such as in case B in Figure 5. If the answer in step S17 is NO, it corresponds to a case where the AR object is within the FOV but is relatively far away, so there is no need to restrict the upper limit of the movement speed within the field of view.

[0155] In step S18, the control device 2 calculates the horizontal difference (θh1 - θh2) and the vertical difference (θv1 - θv2) for the amount of movement within the field of view {θh1, θv1, θh2, θv2}. The control device 2 compares the horizontal difference (θh1 - θh2) and the vertical difference (θv1 - θv2), and sets the larger difference as difference θL (first difference) and the smaller difference as difference θS (second difference). Differences θL and θS are converted into units of movement speed within the field of view [degrees / second].

[0156] In step S19, the control device 2 determines whether the difference θL is less than or equal to the upper limit (upper limit threshold). If the answer in step S19 is YES, the process proceeds to step S20; if the answer is NO (i.e., it exceeds the upper limit), the process proceeds to step S24.

[0157] In step S24, the control device 2 changes the in-field movement speed of the AR object from its original value to a value of ([Upper threshold] / θL). In other words, compared to the in-field movement speed (Va) before correction (before restriction), the in-field movement speed (Vb) after correction (before restriction) is Vb = Va × (TU / θL), where TU is the upper threshold. After step S24, the process returns to step S1. As a result, the in-field movement speed of the AR object is restricted to be slower than its original value, and in particular to be below the upper threshold.

[0158] In step S20, the control device 2 determines whether the difference θL is less than or equal to the lower limit (lower threshold). If the answer in step S20 is YES, the device proceeds to step S21; otherwise, the device proceeds to step S22.

[0159] In step S21, the control device 2 determines whether the component of movement in the depth direction from the viewpoint is the largest in terms of the amount of movement within the field of view. If the answer in step S21 is YES, the device proceeds to step S22; if the answer is NO (i.e., the components in the horizontal direction (h) or vertical direction (v) are larger), the device proceeds to step S25.

[0160] In step S22, the control device 2 does not change the in-field movement speed of the AR object, and returns to step S1. In the case of proceeding to step S22, it is considered normal playback, and the in-field movement speed is not restricted.

[0161] In step S25, the control device 2 changes the in-field movement speed of the AR object from its original value to a value of ([lower threshold] / θL). In other words, compared to the in-field movement speed (Va) before correction (before restriction), the in-field movement speed (Vb) after correction (before restriction) is Vb = Va × (TL / θL), where TL is the lower threshold. After step S25, the process returns to step S1. As a result, the in-field movement speed of the AR object is restricted to be faster than its original value, and in particular to be above the lower threshold.

[0162] The processing described above is repeated for each point in time or each video frame. While the calculations in the above processing flow may be performed for each video frame, for example, if the computer load is too high, they may be performed at a reduced time interval or frame rate as appropriate. For example, they may be performed every 0.5 to 1 second (10 to 20 fps).

[0163] The above flow includes both an upper limit (upper threshold TU) and a lower limit (lower threshold TL). However, variations are possible, such as limiting only by the upper limit or only by the lower limit.

[0164] [Visualization of work and display of work evaluation] This embodiment may have the following additional functions: - A function to display the worker's work status / results as an AR image. - A function to evaluate the worker's work status / results and display the evaluation results as an AR image.

[0165] The work support system in this embodiment detects and monitors the state of the worker's hands, such as hand position, through the display unit 1 during actual work, and records it as work history data. The work support system may also create and display an augmented reality (AR) image representing the state of the actual work. Based on the work history data, the work support system may display the AR image to reproduce the actual work at any time. Alternatively, both the AR image of the reference work and the AR image of the worker's actual work may be displayed simultaneously. For example, the trajectory of the reference work may be displayed in yellow, and the trajectory of the actual worker may be displayed in light blue. This allows the worker to easily recognize the state / results of the actual work.

[0166] Furthermore, the work support system of this embodiment may evaluate the state / results of the worker's work (actual work), create and record evaluation results (work evaluation result information), and display the evaluation results on the screen. An AR image representing the evaluation results may also be displayed on the screen. The evaluation results may include, for example, evaluation values ​​based on the magnitude of the difference between the standard work and the actual work. The evaluation results may be expressed as a display mode of an AR object such as a handprint, or separate from the AR object, work evaluation result information such as explanations, warnings, and instructions in text may be provided. As part of the evaluation, the work support system may calculate, for example, the difference, distance, etc., between the hand position of the standard work and the hand position of the worker's actual work, and display that difference as an AR image. For example, if the difference is larger than a threshold, the part of the difference may be displayed in red. Through these means, the worker can recognize from the evaluation results the parts where the actual work differs from the standard work, and grasp whether the actual work is more suitable than the standard work.

[0167] As described above, the work support system evaluates the state / results of the worker's actual work compared to a standard work. For example, the work support system may evaluate the work quality as poor if the above difference is greater than a threshold, and as good if it is less than the threshold. However, the evaluation value is not limited to this, and may be a multi-level value or a score. The work support system may display the evaluation result (evaluation value) in real time during work execution, or it may display it at any time after execution, such as when instructed by the user.

[0168] In this embodiment, the above-described work support function allows workers to easily recognize whether their hand movements and other actions during work are correct, that is, whether they are close to standard work actions.

[0169] [Example of displaying evaluation results (1)] Figure 18 shows an example of displaying the evaluation results of a wipe operation. State A shows the case where an image representing the overall trajectory of the hand movements of the reference operation (trajectory image 1801) is displayed as an AR video on the screen 1800 of the display unit 1. The portion of the trajectory image 1801 that fits within the screen 1800 is visible. State B shows the case where, based on the work history data, an image representing the overall trajectory of the hand movements of the worker (trajectory image 1802) is displayed as an AR video on the screen 1800. State C shows the case where the worker's trajectory image 1802 (e.g., dashed line, blue) is superimposed on the trajectory image 1801 (e.g., solid line, black) of the reference operation as in State A, and both are displayed simultaneously.

[0170] Furthermore, state D is a case where, based on state C, the difference (trajectory difference) is calculated between the trajectory image 1801 of the reference work and the trajectory image 1802 of the worker, and an image corresponding to the size of the trajectory difference (trajectory difference image 1804) is displayed. In this example, the trajectory difference image 1804 is based on the lines of the trajectory image 1801, and the line thickness (width) increases in areas where the difference is larger. However, this is not the only option; areas with larger differences may be highlighted by changing the color, etc.

[0171] Furthermore, AR objects such as the handprint 7 may be displayed to move along the trajectory described above. In this case, the color, shape, etc., may be changed depending on the position of the handprint 7 and the magnitude of the difference from the position of the reference work. For example, if the difference from the position of the reference work is small and below a threshold, the handprint 7 may be displayed in blue to indicate that it is a correct / suitable work, and if the difference from the position of the reference work is large and exceeds the threshold, the handprint 7 may be displayed in red to indicate that it is an incorrect / inappropriate work.

[0172] [Example of displaying evaluation results (2)] Figure 19 shows other display control examples when displaying the evaluation results of wipe operations. Figure 19 specifically shows control example 1. In control example 1, if the operation is correct, i.e., if the difference from the standard operation is small, the handprint 7 (feature point or contour) is displayed in blue, for example, and if the operation is incorrect (in other words, a deviation), i.e., if the difference from the standard operation is large, the handprint 7 (feature point or contour) is displayed in red, for example. Specific examples are shown at the bottom of the table. Control example 1 has further examples such as Example 1A and Example 1B.

[0173] In Example 1A, a handprint 7 (Figure 16) with feature points from a skeletal model is displayed as an AR object. Example 1A shows a certain work trajectory (a linear movement from left to right) 1901 on the work surface 3A. Work trajectory 1901 is the trajectory of the hand position in the reference work. Work trajectory 1902 is the trajectory of the hand position during the worker's actual work. In this example, as shown in the figure, the difference was small in the left half at first, but then the difference gradually increased in the right half. In Example 1A, the handprint 7 (feature points) is displayed in blue for the correct work portion, and the handprint 7 (feature points) is displayed in red for the incorrect work portion.

[0174] Example 1B uses a handprint 7B outline as an AR object. The correct work areas are displayed in blue, and the incorrect work areas are displayed in red.

[0175] Example 1C uses the trajectory of a handprint 7 (for example, a type of outline) as another example. On the time axis, the handprint 7 moves along the work trajectories 1901 and 1902, and several handprints 7 are displayed simultaneously as trajectories from the present time to several past time points. In Example 1A, etc., one handprint 7 is displayed per time point, but in Example 1C, several handprints 7 are displayed as trajectories for each time point. Handprint 7C1 is at a certain position on the worker's work trajectory 1902 at a certain point in time, and similar handprints corresponding to positions at, for example, three past time points are displayed superimposed as trajectories behind it. At this point, the difference from the work trajectory 1901 of the reference work is small, so it is judged to be a correct work, and the trajectory of handprint 7C1 is displayed, for example, with the outline in black and the inside of the outline in blue. Handprint 7C2 is at a position at a later time point on the work trajectory 1902, and similarly, handprints corresponding to positions at three past time points are displayed superimposed as trajectories behind it. At this point, the difference from the work trajectory 1901 of the standard work is large, so it is judged to be an incorrect work, and the trajectory of the bill of exchange 7C2 is displayed, for example, with the outline in black and the inside of the outline in red. The same can be applied to bill of exchange 7 of the feature point type.

[0176] Example 1D is a variation of Example 1C, in which the handprint (inside the outline) at the current position is displayed in a color corresponding to the magnitude of the difference, similar to the trajectory.

[0177] Figure 20 shows Control Example 2. Control Example 2 uses the positional relationship between the AR object representing the worker's (e.g., skilled worker's) hands and head in the work model of the standard work, and the positional relationship between the AR object representing the worker's hands and head in the actual work. The control device 2 considers and calculates the difference (sometimes referred to as head distance) between the position of the head (corresponding viewpoint) in the standard work and the position of the head (corresponding viewpoint) in the actual work. Then, the control device 2 changes the display mode of the AR object (e.g., handprint) according to the magnitude of this difference (head distance). For example, if the difference (head distance) is small, the handprint is displayed in blue, and if the difference (head distance) is small, the handprint is displayed in red. In Control Example 2, if the position of the worker's head is close to the position of the head in the standard work, it is considered a suitable work, and if the position of the worker's head is far from the position of the head in the standard work, it is considered an unsuitable work.

[0178] State A shows the positional relationship between the AR object of the handprint 7 and the head / viewpoint in a work model of a standard operation based on a skilled worker. Here, the position of the viewpoint 510, the position of the head 520, and the position of the display unit 1 are set to the same position coordinates (for example, H1(X1, Y1, Z1)). However, the positional relationships between each of these positions may be set as well. Assume that at some point in time, the position of the handprint 7 is (X0, Y0, Z0). In State A, the position of the head 520 is at the same position as the X coordinate of the handprint 7 in the X direction of the spatial coordinate system (X1 = X0). Although not shown in the figure, the position of the head 520 in the Z direction is at a predetermined position relative to the Z coordinate of the handprint 7 (for example, shown in the work model as State A in Figure 14). The position of the head 520 in the Y direction (depth direction) is at a depth distance D1 relative to the Y coordinate of the handprint 7 (similar to Case A in Figure 5).

[0179] On the other hand, state B shows the positional relationship between the AR object of the same handprint 7 and the head / viewpoint in the work history of the worker's actual work. The position of the handprint 7 is the same (X0, Y0, Z0). In state B, the position of the head 520 in the X direction is the same as the X coordinate of the head 520 in the reference work. Although not shown in the figure, the position of the head 520 in the Z direction is the same as the position of the head 520 in the reference work. The position of the head 520 in the Y direction (depth direction) is at a depth distance D2 (D2 < D1) relative to the Y coordinate of the handprint 7 (similar to case B in Figure 5).

[0180] Comparing state A and state B, the position of the head 520 is different, and the difference between them, the head distance 2001, is a distance / vector corresponding to the difference in depth distance (D1-D2). This example is not unique; the position of the head 520 may also be shifted in the X direction or the Z direction relative to the reference operation.

[0181] In control example 2, the display mode of the AR object is changed according to the magnitude of the difference in head distance.

[0182] In Example 2A, the feature points of the handprint 7 are displayed in blue if the head distance is less than or equal to the threshold, and in red if the head distance is greater than the threshold. In Example 2A, the handprint 7 is shown in red when the head distance 2001 is greater than the threshold.

[0183] In Example 2B, the outline of the handprint 7 is shown in blue when the head distance is small, and in red when the head distance is large. In Example 2B, the handprint 7 is shown in red when the head distance 2001 is greater than the threshold.

[0184] In Example 2A, etc., differences in hand position, as in Control Example 1, are not reflected in the display pattern of the handprint 7.

[0185] According to a display like that in control example 2, the operator can see the display pattern of the AR object, recognize the difference or deviation from the position of the head of the reference work, and adjust it to be closer to the position of the head of the reference work.

[0186] Figure 21 shows a control example (referred to as control example 3) that combines and applies control example 1 and control example 2 described above. In this control example 3, the control device 2 determines whether the work is correct based on the difference in hand position between the reference work and the worker's actual work, and also determines whether the work is suitable based on the difference in head position (head distance) between the reference work and the worker's work. Then, the control device 2 changes the display mode of the handprint 7 at each point in time according to the state of the determination result.

[0187] In Example 3A, a handprint 7 with an outline is used. In Example 3A, in the same results as in Control Example 1, the trajectory of handprint 7 is displayed in blue for correct work and in red for incorrect work. Also, in the same results as in Control Example 2, the outline of handprint 7 at the current time is displayed in blue when the head distance is small and in red when the head distance is large. In Example 3A, as a state of position at a certain point in time, the outline of handprint 7D1 is displayed in red when the head distance is large. The trajectory of handprint 7D1 is displayed in blue because the difference from the hand position of the reference work is small. Also, as a state of position at a different point in time, the outline of handprint 7D2 is displayed in blue when the head distance is small. The trajectory of handprint 7D2 is displayed in red because the difference from the hand position of the reference work is large.

[0188] Example 3B is a variation. It uses a type of handprint 7 that has feature points. In Example 3B, the feature points of the handprint 7's trajectory are displayed in blue when the operation is correct, and in red when the operation is incorrect. Also, when the head distance is small, the feature points of the handprint 7 at the current time are displayed in blue, and when the head distance is large, the feature points of the handprint 7 at the current time are displayed in red.

[0189] As shown in the display example 3, the operator can recognize the state from multiple perspectives in comparison with the standard operation based on the display manner of the AR object.

[0190] The control example above shows its application to wiping operations, but it is not limited to this and can be similarly applied to O-ring installation operations, etc. In the case of O-ring installation operations, when determining whether the work is done correctly, for example, the balance and consistency of the height positions of the left and right hands may be judged. The control device 2 calculates the difference between the height positions of the left hand and the right hand at each point in time and evaluates the balance and consistency. If the difference is below a threshold (in other words, if the balance and consistency are high), the control device 2 determines that the work is correct / suitable, and if it exceeds the threshold, it determines that the work is incorrect / inappropriate.

[0191] Furthermore, in the various display control examples described above, the colors to be changed are not limited to binary (two levels), but may also be multi-level (multi-level). For example, blue, yellow, and red could be assigned according to the large, medium, and small differences.

[0192] Furthermore, the meaning of the display modes of AR objects (for example, if the outline of a handprint is red, it means XX, and if it is blue, it means YY) may be explained in advance in the user interface guide or manual, or explanatory information on the meaning may be displayed on the screen of the display unit 1.

[0193] When displaying AR objects as part of the work history and evaluation results as described above, the function of limiting movement speed within the field of view can be applied in the same way as when displaying AR objects for the reference work.

[0194] [Other Display Control Examples] Figure 22 shows other display control examples related to the display of evaluation results.

[0195] Example 1 is an example in which, based on the work history data of an operator's actual work (e.g., O-ring installation work), the measured height positions of the left and right hands, and the evaluation value of balance and consistency based on the difference between them, are displayed on the screen of the display unit 1 as evaluation results. On the screen 2200 (in other words, the display area) of the display unit 1 corresponding to the field of view 500, along with the display of AR objects of work information 4, for example, an AR object 1201 representing an O-ring in a standard operation, the evaluation result (work evaluation information) 2210 regarding the actual height positions of the left and right hands during the work (e.g., the actual gripping position of the O-ring) is displayed in text in a part of the area (e.g., the upper left). The evaluation result 2210 here includes a message 2211 indicating that the balance and consistency of the height positions of the left and right hands is low, the measured values ​​2212 of the height positions of the left and right hands, the difference value 2213 between the height positions of the left and right hands, and the evaluation value (score) 2214 of the work.

[0196] Message 2211 is not limited to this example; it could also be used as advice (instruction) to bring the work closer to the work model of the standard task. For example, "Please raise your left hand a little higher."

[0197] The evaluation value 2214 can be calculated by the control device 2 based on the difference value 2213, for example. The evaluation value 2214 shown in the figure is the evaluation value at a certain point in time, but it is not limited to this; the control device 2 may also calculate the evaluation value for the entire operation (in other words, the overall evaluation value) and display it at the end of the operation.

[0198] This is not the only example; the display of such evaluation results (work evaluation information) 2210 can also be done at a different time than the display of the AR object for the standard work or the display of the work history (AR object such as the trajectory), and only the evaluation results can be displayed.

[0199] In Example 2, an AR object (for example, a handprint 7 representing how to move a component) is displayed superimposed on the work object 3, including components, across most of the screen 2200. The control device 2 displays the evaluation result (work evaluation information) 2220 within the screen 2200. In doing so, the control device 2 selects a position that does not overlap the work object 3, including components, and the AR object (handprint 7) as much as possible, and displays the evaluation result 2220 there. In this example, since the area above the work object 3 and AR object near the center of the screen 2200 is empty, the evaluation result 2220, for example, is displayed in that upper area, near the center.

[0200] In another example of display control, when displaying the work history slip 7, the evaluation result may be displayed in a nearby position that does not overlap with the slip 7 (it may overlap with the work object 3), following the position of the slip 7. In this case, the evaluation result is displayed while moving as appropriate.

[0201] In Example 3, first, the control device 2 displays a handprint 7 of a reference work AR object on the work surface 3A, and the worker performs the work so as to follow the handprint 7. At that time, the control device 2 detects the work and evaluates it in comparison with the reference work at each point in time. The control device 2 displays the evaluation results (work evaluation information) in real time on the screen 2200. At that time, the control device 2 displays the evaluation results (work evaluation information) in a position that does not overlap with the actual position of the worker's hand. In Example 3, hands 2231, 2232, and 2233 are examples of the actual hand positions at each point in time. For example, for the position of hand 2232 at a certain point in time, the handprint 2242 is displayed as the evaluation result. The handprint 2242 is an image processed based on the base handprint image so that it is displayed in a manner corresponding to the evaluation result. For example, the handprint 2242 is displayed in a color corresponding to the difference from the hand position in the reference work, by applying the same evaluation method as in Control Example 1. For example, in the case of a correct operation with a small difference, the outline, interior of the outline, or feature points of the bill 2242 are displayed in blue, and in the case of an incorrect operation with a large difference, the outline, interior of the outline, or feature points of the bill 2242 are displayed in red.

[0202] In Example 3, the handprint 2241 is displayed in blue near the top of hand 2231. The handprint 2242 is displayed in yellow near the top of hand 2232. The handprint 2243 is displayed in red near the top of hand 2233. In this example, the evaluation value for each point in time is also displayed overlaid on the handprint. The difference in distance from the hand position of the reference operation may also be displayed on the handprint.

[0203] In addition, the handprint in Example 3 may be created as a copy based on an image of an actual hand. For example, the handprint 2242 may be created by processing an image of hand 2232.

[0204] In an example of applying Example 3 to O-ring installation work, a handprint representing the evaluation result is displayed near each hand (left and right). For example, regarding the height position of the left hand, if it is close to the height position of the left hand in the standard work, the handprint is displayed in blue; if it is farther away, the handprint is displayed in red. Similarly, regarding the height position of the right hand, if it is close to the height position of the right hand in the standard work, the handprint is displayed in blue; if it is farther away, the handprint is displayed in red.

[0205] [Effects, etc.] According to the above embodiment, since work information 4 that satisfies multiple motion elements in the standard / exemplary work is displayed as AR video 4, workers can easily perform / learn correct / suitable work that satisfies multiple motion elements by following the AR video 4. This can suppress incorrect movements that unconsciously occur in workers such as inexperienced workers.

[0206] By having the worker follow the AR image 4 while performing the task, the focus becomes concentrated on a single point, making it easier for even unskilled workers to perform the task and facilitating the execution of correct / optimal tasks that satisfy multiple motion elements.

[0207] Since the work information 4 representing the standard / executive work is displayed as an AR image 4, the worker can recognize whether the work / action they are performing is correct / suitable and close to the standard work.

[0208] Since the movement speed within the field of view is restricted in a timely manner, even if the work object 3 is large, the worker can more easily track the AR object within the limited field of view / screen size, preventing it from being overlooked. The movement speed of the AR object within the field of view can be restricted according to the positional relationship between the work object 3 and the AR object and the user U1's viewpoint, head, and display 1, making it easier for the worker to track the AR object and preventing it from being overlooked. As a result, work efficiency is improved and workers can be trained more quickly.

[0209] The tasks covered are not limited to the wiping tasks mentioned above. Other tasks include the assembly and disassembly of various products and their component parts. Each product and component part may also require cleaning and wiping.

[0210] Example 1 describes a case where there are upper and lower limits (upper threshold) that constitute the acceptable range. The upper limit restriction modifies the AR object's movement speed within the field of view to be at or above the upper limit if it is too fast. The lower limit restriction modifies the AR object's movement speed within the field of view to be at or above the lower limit if it is too slow. However, the restriction is not limited to this, and any restriction that uses at least one of the upper and lower limits is acceptable.

[0211] In this embodiment, the concept of in-field movement speed / in-field movement amount is expressed using an angle θ for calculation purposes, and therefore the unit is expressed in terms of angle θ, but it is not limited to this. For example, it may also be expressed as distance or speed based on pixels in the horizontal or vertical direction of the screen, and conversion is possible as appropriate.

[0212] [Modification (1)] The following modification of this embodiment is also possible. Figure 23 shows the screen 2300 corresponding to the field of view 500 as seen from viewpoint 510 in the modification (referred to as Modification 1). In the modification, the area of ​​the screen 2300 is divided into at least two regions in both the x direction (h direction) and the z direction (v direction). In Figure 23, the screen 2300 (in other words, the overall FOV) has its center point p0, with size Sx in the x direction and size Sv in the z direction. In this example, the screen 2300 is horizontally elongated (Sx > Sv). In Figure 23, a region 2301 (in other words, the central region FOV) is provided with smaller sizes Sx2 and Sz2 relative to the center point p0. In this example, Sx2 = Sx ÷ 2, and the distance from the center point p0 of region 2301 to the rightmost position in the x direction corresponds to Sx2 / 2. Furthermore, the distance from its right edge to the right edge of screen 2300 is also equivalent to S x 2 / 2. In other words, in this example, the area from the center point p0 to the right edge of screen 2300 is divided into two equal-sized sections. The same division applies to the z-direction. The area of ​​screen 2300 excluding the rectangular area 2301 (also referred to as the first area) is designated as area 2302 (also referred to as the second area).

[0213] In this modified example, the in-field movement speed of the AR object is controlled based on the region division configuration of the screen 2300 described above. The control device 2 determines a limit on the in-field movement speed of the AR object according to the region (region 2301, region 2302) in which the AR object is located. As a specific example of control, in addition to the control that limits the in-field movement speed within an acceptable range as described above, the control device 2 applies control according to the region division shown in Figure 23. There are several possible details for this control, which are illustrated below.

[0214] First, in the case of the aforementioned control (i.e., without region division), a limit using an allowable range is applied uniformly to the entire screen 2300. Regardless of the position of the AR object within the screen 2300, if the in-field movement speed exceeds, for example, an upper threshold, it is modified / corrected so that it becomes below the upper threshold. Next, in the region division control shown in Figure 23, a limit is imposed so that the in-field movement speed of the AR object becomes slower depending on the division area extending from the center point p0 to the periphery of the field of view (screen edge). There are several possible ways to specifically apply this limit.

[0215] For example, in one control example (control example 1-1), if the AR object is in the region 2301 closer to the center point p0 (e.g., position 2321), no restriction using the tolerance range is applied, and if the AR object is in the region 2302 closer to the periphery (e.g., position 2322), a restriction using the tolerance range is applied.

[0216] Another control example (control example 1-2) is that if the AR object is in the region 2301 closer to the center point p0, a first limit range (first lower threshold and first upper threshold) is applied as an allowable range to restrict it. If the AR object is in the region 2302 closer to the periphery, a second limit range (second lower threshold and second upper threshold), which is narrower than the first limit range, is applied as an allowable range to restrict it more strongly.

[0217] With the control example described above, the AR object's movement speed within the field of view slows down as it moves closer to the edge of the screen 2300. Furthermore, if the AR object reaches the edge of the screen 2300 or beyond, it may be stopped at the edge, as described above (step S23 in Figure 15).

[0218] The configuration of the region division is not limited to the example shown in Figure 23. The region of screen 2300 may be divided into three or more regions from the center point to the periphery, and restrictions of three or more levels may be applied.

[0219] In other variations, instead of region division, a parameter such as the distance from the center point p0 to the periphery, or a coefficient corresponding to that distance, is used. The control device 2 may reflect the parameter value in the in-field movement speed by subtraction or multiplication to obtain the corrected in-field movement speed. For example, in the case of position 2321, the coefficient is determined to be, for example, 0.8 depending on its x-coordinate, and in the case of position 2322, the coefficient is determined to be a smaller value such as 0.4 depending on its x-coordinate. If the in-field movement speed before correction is V1 at position 2321, the corrected in-field movement speed will be V1 × 0.8, and if the in-field movement speed before correction is also V1 at position 2322, the corrected in-field movement speed will be V1 × 0.4, which is slower. In other words, a curve or the like to limit the in-field movement speed according to the above distance parameter may be set.

[0220] [Modification (2)] Another modification (referred to as Modification 2) is also possible. In this modification, different tolerances, or in other words, changing tolerances, are set according to the depth distance from the viewpoint to the AR object. Then, the changing tolerance is applied according to the state of the depth distance and other factors detected at each point in time. For example, using the example in Figure 5, in case A, with a depth distance D1, the first tolerance is applied, and in case B, with a depth distance D2, a second tolerance different from the first tolerance is applied.

[0221] Another control method involves setting a control threshold for depth distance (depth distance threshold), and using this threshold or the range defined by this threshold to determine whether or not a restriction is applied. For example, if the detected depth distance is less than or equal to the threshold, a restriction based on the allowable range is applied. These controls allow for a more suitable display that takes depth distance into consideration.

[0222] Figure 24 shows an example of control in this modified version. In this modified version, the applicability of the tolerance limit and the values ​​of the tolerance limit (lower threshold and upper threshold) are variably determined according to the depth distance to the AR object in the depth direction (y direction) from the viewpoint 510.

[0223] In state A of Figure 24, similar to Figure 5, the in-field movement speed of the AR object is shown according to the depth distance relative to the viewpoint 510. First, a control (referred to as control 1) is used that applies a restriction when the depth distance is within a predetermined range, and does not apply the restriction when it is outside the range. In the example of Figure 24, a control threshold (depth distance threshold) is set for a certain depth distance DX. If the depth distance is within the range 2401 from distance 0 to distance DX, the restriction according to the set allowable range is applied, and if it is outside the range 2401, the restriction is not applied. For example, in the case of distance D2 mentioned above, the restriction is applied, and in the case of distance D1, the restriction is not applied. Note that the processing example such as step S17 in Figure 15 corresponds to an example of this control 1.

[0224] Furthermore, as shown in Figure 20, there may be a difference or discrepancy (the aforementioned head distance) between the viewpoint / head position in the standard operation and the viewpoint / head position in the worker's operation. Therefore, the following control 2 may be used. Control 2 is shown in state B. In state B, the position of the viewpoint 510 / head 520 in the standard operation is set to position H0, which is a depth distance of 0. The depth distance to the AR object (e.g., the handprint 7) in the standard operation is DA. The control device 2 captures the position of the viewpoint 510 / head 520 in the actual worker's operation relative to the position of the viewpoint 510 / head 520 in the standard operation and calculates the aforementioned head distance, etc. For example, the control device 2 determines that the operation is suitable if the position of the actual worker's viewpoint 510 / head 520 (corresponding head distance) is within a predetermined range 2402 from +Dy1 to -Dy1, and determines that it is inappropriate if it is outside the range 2402. Furthermore, the control device 2 may apply a restriction when the position of the worker's viewpoint 510 / head 520 is outside the range 2402, for example, when it is within the range 2403.

[0225] Next, as control method 3, the allowable range of movement speed within the field of view may be determined (in other words, dynamically set) to a different value depending on the depth distance. For example, the following control is possible. A standard allowable range (lower threshold and upper threshold) is set in advance according to the work model of the standard work. This is referred to as the standard allowable range R0 (standard lower threshold TL0 and standard upper threshold TU0). In addition, the work model defines a suitable depth distance (in other words, a standard depth distance: referred to as D0), as in the example of depth distance DA in state B. The control device 2 detects and calculates the depth distance D from the position of the worker's viewpoint 510 / head 520 to the position of the AR object of the standard work at each point in time. The control device 2 calculates the applicable allowable range R (lower threshold TL and upper threshold TU) according to that depth distance D.

[0226] The calculation methods include the following: For example, the upper threshold can be made smaller as the depth distance D decreases. A specific example of this calculation is to use (D / D0) or (D-D0) as a coefficient to reflect in the reference threshold.

[0227] <Example 2> Example 2 will now be described. The basic configuration in Example 2 and others is the same as and common to Example 1. Below, we will describe the components in Example 2 and others that differ from those in Example 1.

[0228] In Example 1, a lower and upper threshold value was used to define the acceptable range of movement speed within the field of view. In contrast, in Example 2, the movement speed of the AR object within the field of view is controlled to be constant when the distance between the AR object of the work model of the reference work and the worker's viewpoint is within a predetermined value.

[0229] [Processing Flow (2)] Figure 25 shows the detailed processing flow (second configuration example) in Example 2. The flow in Figure 25 of Example 2 is a variation of the flow in Figure 15 of Example 1. In Figure 25, the flow to step S26 when NO is selected in step S20 is different, but the other steps are the same as the flow in Figure 15. In step S20, if the angle θL is not below the lower limit (i.e., greater than the lower limit), NO is selected and the process proceeds to step S26. In step S26, the control device 2 changes the in-field movement speed of the AR object by a factor of ([Setting Value] / (θL + θS)). Here, [Setting Value] is a value set in advance according to the work model. The in-field movement speed after the change becomes approximately constant.

[0230] As shown in Figure 25, in Example 2, when the distance to the AR object (depth distance) is less than or equal to a specified value (S17-YES), θL is less than or equal to the upper limit (S19-YES), and θL is greater than the lower limit (S20-NO), the in-field movement speed is changed in step S26 to be approximately constant. This makes it easier for the operator to follow the AR object when the depth distance is small.

[0231] <Example 3> Example 3 will now be described. In Example 3, the control device 2 uses the work process management unit 28 shown in Figure 3 to change the allowable range (lower threshold and upper threshold) of the AR object and the movement speed within the field of view according to the work process. The work process management unit 28 sets AR data 23A of work models / virtual models according to the work process in the DB 23 through the AR creation and management unit 22. For example, multiple work models are prepared, such as a work model for wiping work as shown in Figure 2A and a work model for O-ring installation work as shown in Figure 11. The control device 2 controls the operation to switch the applicable work model according to the work process that the operator is performing or learning.

[0232] For example, the operator specifies the target work process based on the operation input to the display unit 1 or the control device 2. For example, a graphical user interface (GUI) is displayed on the screen of the display unit 1 or the screen of the display connected to the control device 2. The operator, user U1, specifies the target work process using the GUI on that screen.

[0233] State A in Figure 26 is an example of displaying the GUI on the screen 2600 of the display unit 1. The screen 2600 displays a selection of target work processes, and user U1 can select and specify a target work process.

[0234] Furthermore, as shown in state B, the GUI may allow the user to specify the type, color, etc., of the AR object to be used. For example, for wipe operation A, several types of AR objects are pre-configured as usable AR objects, associated with the work model. For example, handprint A (skeletal model), handprint B (contour model), trajectory A, trajectory B, etc. User U1 can select and specify the type and color of the AR object to be used.

[0235] The control device 2 receives input information specifying the target work process, etc. The control device 2 (particularly the work process management unit 28) obtains AR data 23A and tolerance range 23B of the work model corresponding to the target work process, etc. from DB 23 (Figure 13). Then, the control device 2 displays AR objects, etc., based on the AR data 23A on the screen of the display unit 1 according to the flow shown in Figures 4 and 15.

[0236] Furthermore, the GUI offers various modes, such as the mode shown in state C, which displays the AR object of the reference task and allows the user to actually perform or learn the task, and the mode that allows the user to reproduce the task and check the evaluation results based on the task history data. The user can select and execute the appropriate mode.

[0237] According to the functionality of Example 3, a single system can provide work support for various work processes.

[0238] [Processing Flow (3)] Figures 27 and 28 show the detailed processing flow (third configuration example) in Example 3. Figure 27 differs from Figure 15 in the following way: If NO is given in step S14, proceed to step S22A. In step S22A, the in-field movement speed is not changed. Also, if NO is given in step S17, proceed to step S22B. In step S22B, the in-field movement speed is not changed. After step S18, proceed to steps S30, S31, and S32. In step S30, the control device 2 calculates the in-field movement speed for the angle θL (θh). In step S31, the control device 2 calculates the in-field movement speed for the angle θS (θv). In step S32, the control device 2 selects the smaller of the in-field movement speeds calculated in steps S30 and S31 and sets it as that in-field movement speed.

[0239] Figure 28 shows the flow of steps S30 and S31. Step S30 describes the case where the horizontal direction of the screen (h) is θL, and step S31 describes the case where the vertical direction of the screen (v) is θS.

[0240] Step S30 includes steps S40 to S45. In step S40, the control device 2 determines whether (θh1 - θh2) / sec is less than or equal to the upper limit. If YES, proceed to step S41; otherwise, proceed to step S43. In step S43, the control device 2 changes the in-field movement speed by a factor of ([h-side upper limit threshold] / (θh1 - θh2)). [h-side upper limit threshold] is the upper limit setting value for the horizontal (h) side of the screen.

[0241] In step S41, the control device 2 determines whether (θh1 - θh2) / sec is below the lower limit. If YES, proceed to step S42; otherwise, proceed to step S22. In step S42, the control device 2 determines whether the movement component in the depth direction is the largest. If YES, proceed to step S45; otherwise, proceed to step S44. In step S44, the control device 2 changes the in-field movement speed by a factor of ([h-side lower limit threshold] / (θh1 - θh2)). [h-side lower limit threshold] is the lower limit setting value on the horizontal (h) side of the screen. In step S45, the control device 2 does not change the in-field movement speed.

[0242] Step S31 includes steps S46 to S48. In step S46, the control device 2 determines whether (θv1 - θv2) / sec is less than or equal to the upper limit. If YES, proceed to step S48; otherwise, proceed to step S47. In step S47, the control device 2 changes the in-field movement speed by a factor of ([v-side upper limit threshold] / (θv1 - θv2)). [v-side upper limit threshold] is the upper limit setting value on the vertical (h) side of the screen.

[0243] <Example 4> Example 4 will now be described. In Example 4, the allowable range (lower threshold and upper threshold) of the AR object and the movement speed within the field of view is changed according to the worker's skill level.

[0244] In Embodiment 4, the worker information setting unit 29 shown in Figure 3 is used. The worker information setting unit 29 provides a GUI to user U1 and inputs and sets the proficiency level of the target worker based on the user U1's input. The worker information setting unit 29 stores and manages the worker information, including the set proficiency level, in the DB 23 of the storage unit 202 or the like. Note that user U1, who is the worker, may input their own proficiency level, or another user U1, such as a manager or expert, may evaluate and input the proficiency level of the worker.

[0245] Figure 29 shows an example of displaying a GUI related to proficiency on the screen 2900 of the display unit 1. In the GUI, proficiency levels can be set for each worker and for each task (corresponding task model). For example, when worker A is selected as the target worker and task A is selected as the target task, a proficiency level can be set by selecting from the proficiency level options (e.g., level 1 to level 3). Proficiency levels may be set as follows: for example, beginners are level 1, intermediate workers are level 2, and advanced (experts) are level 3. However, proficiency levels may also be expressed as a score or other numerical value.

[0246] The control device 2 (particularly the worker information setting unit 29) determines and controls the permissible range of movement speed of the AR object within the field of view related to the target task based on the skill level value of the target worker. For example, the control device 2 changes and sets the upper limit threshold of the permissible range to a lower value as the skill level value decreases. In this case, the AR object of the standard task presented to a worker such as a beginner is restricted to move more slowly within the field of view, making it easier for the worker to follow the AR object even if they are unfamiliar with the target task.

[0247] The following methods may be used to determine and set the level of proficiency: For each worker and each task, information such as work time, number of tasks, work performance (which may include the quality of product inspection results), and years of service is referenced. This information may be referenced from an external database of this work support system, or it may be entered and stored in the database 23 of this system. The control device 2 calculates the level of proficiency based on this information.

[0248] State B is an example of setting tolerance ranges according to proficiency level. This is the case when the target is wipe operation A (work model 1). For example, in the case of level 1, tolerance range R1-1 is set with a lower threshold of TL1-1 and an upper threshold of TU1-1. In the case of level 2, tolerance range R1-2 is set with a lower threshold of TL1-2 and an upper threshold of TU1-2. In the case of level 3, tolerance range R1-3 is set with a lower threshold of TL1-3 and an upper threshold of TU1-3. Each threshold is different.

[0249] Furthermore, the work evaluation unit 30 in Figure 3 may be used to determine and set the worker's proficiency level based on the worker's work history and evaluation results. In other words, proficiency levels are not limited to being evaluated and set by a person; a computer system may also evaluate and set them.

[0250] The work evaluation unit 30 evaluates the work based on work history data obtained by monitoring the worker's actual work. For example, the work evaluation unit 30 evaluates whether the position and movement of the worker's hands are close to the position and movement of the hands (corresponding AR object) in the reference work, and whether the position of the worker's viewpoint / head is close to the position of the viewpoint / head in the reference work. The work evaluation unit 30 assigns an evaluation value to the evaluation and stores it in DB 23 as the work evaluation result. As mentioned above, the work evaluation result can also be displayed on the screen. The control device 2 calculates and sets the worker's proficiency level based on the work evaluation result. For example, the proficiency level (e.g., level) may be determined by dividing the evaluation value into thresholds.

[0251] Regarding the various calculations mentioned above, while they may be performed in real time at each point in time, they are not limited to this. Alternatively, a correspondence can be defined in advance using a lookup table, and the calculation can be omitted by referencing that lookup table.

[0252] Furthermore, even for the same task, different workers may have preferred in-field movement speeds. For example, worker A might prefer a fast in-field movement speed, while worker B might prefer a slower one. Therefore, it may be possible to allow users to adjust the acceptable range according to individual workers. For example, as in state C, the GUI could display the setting information for the acceptable range for each target worker, allowing user U1 to change the threshold.

[0253] [Other Examples of AR Objects] Figure 30 shows examples of other display modes for AR objects used as work information 4. Example 1 is an example in which the magnitude and direction of a force (e.g., pressing force) applied to the work surface 3A of the work object 3 by a hand or tool is represented and displayed as an AR object. The display of the screen border is omitted. In Example 1, in a certain operation, first, the location to be pressed (e.g., the center of the work surface 3A) is displayed as a circular AR object 3001 as the AR object for the reference operation. In addition, an arrow AR object 3002 representing the force and direction of the pressing is displayed along with the AR object 3001. In this example, the direction of the arrow is downward in the z direction, and the size of this arrow represents the magnitude of the pressing force. In addition, characters representing the pressing action (e.g., "press") or a numerical value representing the pressing force may be displayed. By looking at this AR object, the worker can recognize the pressing force in the reference operation. The worker presses downwards on the area indicated by the AR object 3001 using their hand 3010 or a tool.

[0254] Furthermore, the example above showed how the hands could be represented as an AR object (e.g., handprint 7) as part of the worker's body. However, other parts of the body, such as arms and head, can also be represented as AR objects and displayed on the screen in the same way.

[0255] Furthermore, the example above showed how to represent the position of the hand during work and its trajectory over time using AR objects. However, it is not limited to this; the amount of change in position (velocity), the direction of change in position, etc., can also be represented using AR objects (for example, arrows) and displayed on the screen.

[0256] Alternatively, the operator's line of sight may be detected and utilized. Display unit 1 uses eye-tracking functionality to detect the direction of the operator's gaze. Control device 2 displays AR objects on the screen that represent the direction of the gaze and the point of fixation. In Example 3, a handprint is displayed as AR object 3021 for the wipe operation in the standard operation. Control device 2 also displays AR object 3022 that represents the direction of the operator's (e.g., skilled worker's) line of sight in the standard operation, particularly the point of fixation where the lines of sight on the screen intersect. In this example, AR object 3022 is a white "X". The operator performs the operation by moving their hand 3023 while tracking the AR object 3021 of the handprint with their eyes. Control device 2 may also display AR object 3024 on the screen that represents the direction of the operator's line of sight, particularly the point of fixation where the lines of sight on the screen intersect. In this example, AR object 3024 is a black "X". The worker may perform the work while referring to the AR object 3022 of the gaze point for the standard work.

[0257] Example 4 shows a scenario based on Example 3 where the worker aligns their line of sight (point of gaze) and hands with the line of sight (point of gaze) and handprint of the reference work.

[0258] Furthermore, the distance, elapsed time (time required), and speed (speed of movement within the field of view) of an AR object such as a handprint may be displayed on the screen as an AR object. In Example 5, when a handprint 7 from a reference operation moves from one position to another during a wipe operation, and the worker follows it by moving their hand (for example, with each stroke), the distance 3032 and elapsed time 3033 are displayed along with the trajectory 3031 as part of the work evaluation results. The distance 3032 and elapsed time 3033 are displayed, for example, by a callout object near the trajectory 3031.

[0259] Furthermore, when using tools or other objects (for example, cloth 5) in the work, AR objects representing the name, orientation / position, surface shape, and condition (for example, whether or not alcohol is present) of the tools or other objects may be displayed. In Example 6, an AR object 3041 for the cloth 5 held in the worker's hand, a speech bubble object 3042 showing information about the cloth 5, an arrow object 3043 indicating the direction of movement of the cloth 5, etc. are displayed.

[0260] Furthermore, in tasks involving the use of both hands, such as the O-ring installation process described above, in addition to AR objects representing the left and right arms, hands, and their positions, the relative positions of the left and right arms and hands (in other words, the difference between the left and right hands) may also be represented and displayed using AR objects.

[0261] Example 7 shows an example of displaying the relative positions of the left and right hands as AR objects when displaying work evaluation results. AR object 3051 is a line object representing the height position of the left hand (and the corresponding gripping position on the left side of the O-ring). AR object 3052 is a line object representing the height position of the right hand (and the corresponding gripping position on the right side of the O-ring). AR object 3053 is an arrow object representing the difference between the height positions of the left and right hands. These AR objects represent the balance and consistency between the left and right hands. By looking at these AR objects, the worker can recognize the balance and consistency between the left and right hands. For example, the worker can move their left and right hands so that the AR object 3053 representing the difference in height between the left and right hands becomes smaller.

[0262] Although embodiments of this disclosure have been described in detail above, the invention is not limited to the embodiments described above and can be modified in various ways without departing from the gist of the invention. Each embodiment can be modified by adding, deleting, or replacing components, except for essential components. Unless otherwise specified, each component may be singular or plural. Combinations of each embodiment and its variations are also possible.

[0263] 1...Display unit, 2...Control device, 3...Work object, 3A...Work surface, 4...Work information (AR image), 5...Cloth, 6...Control information / Display information, 7...Handprint (AR object), 8...Control information / Display unit information, U1...User (worker), 500...Field of view, 510...Viewpoint.

Claims

1. A work support system that provides work information to assist a worker in performing work on an object, comprising: displaying video information having the position and movement of a reference work on the object in a three-dimensional space including the object, superimposed on the object on a screen of a display device worn on the worker's head corresponding to the worker's field of view; acquiring first information having the position and movement of the work information on the object in the three-dimensional space when the worker performs the work on the object; having second information having the position and movement of the display device corresponding to the position and movement of the worker's head in the three-dimensional space; calculating the position and movement of the display of the work information on the object on the screen of the display device based on the first and second information; and limiting the in-field movement speed of the work information to be within an allowable range set according to the reference work when displaying the work information on the object on the screen of the display device.

2. A work support system according to claim 1, wherein the restriction is applied such that the speed of movement within the field of view slows down as the position where the work information is displayed in the area of ​​the screen of the display device moves from the center to the periphery.

3. A work support system according to claim 1, wherein the allowable range is calculated or set according to the positional relationship in the depth direction between the position of the viewpoint corresponding to the display and the position of the work information.

4. A work support system according to claim 1, wherein when the position for displaying the work information is outside the area of ​​the screen of the display, the in-field movement speed is set to 0 and the system is stopped at the edge of the screen.

5. In the work support system according to claim 1, the work information is at least one of the following video information with respect to the reference work or actual work: a video showing the position of a body including hands or head; a video showing the work area of ​​the work object; a video showing the position of an object including a tool used by the body or a component constituting the work object; a video showing the relative position or balance of the left and right hands; a video showing the trajectory of the body or the object as it moves; a video showing the distance the body or the object as it moves; a video showing the time the body or the object as it moves; a video showing the speed at which the body or the object as it moves; a work support system.

6. A work support system according to claim 1, wherein the work is wiping the work surface of a product or a component of a product, the directions subject to the restriction are the horizontal and vertical directions of the screen, and the allowable range of the in-field movement speed is a lower limit of 13.0 degrees / second and an upper limit of 23.5 degrees / second.

7. A work support system according to claim 1, wherein the work is the work of assembling or disassembling a product or its constituent parts, the directions subject to the restriction are the horizontal and vertical directions of the screen, and the allowable range of the in-field movement speed is a lower limit of 1.9 degrees / second and an upper limit of 5.1 degrees / second in the horizontal direction of the screen, and a lower limit of 3.0 degrees / second and an upper limit of 8.3 degrees / second in the vertical direction of the screen.

8. A work support system according to claim 1, wherein a level of proficiency in the work is set for each worker, and the allowable range is set according to the level of proficiency.

9. A work support system according to claim 1, wherein when the worker performs the work on the work object, the system detects the position and movement of the actual work on the work object on the screen of the display unit as detection of the state of the actual work, and records it as work history data.

10. A work support system according to claim 9, wherein, when reproducing and displaying the actual work on the work object on the screen of the display based on the work history data, the work support system limits the in-field movement speed of the actual work to be within an acceptable range set according to the reference work.

11. A work support system according to claim 9, wherein the system calculates the difference in position and movement between the reference work and the actual work, and displays an image representing the difference on the screen of the display device.

12. A work support system according to claim 11, wherein the image representing the difference is an image whose display mode, including shape or color, is varied according to the magnitude of the difference.

13. A work support system according to claim 11, wherein the image representing the difference is displayed on the screen of the display device at a position that follows the position of the actual work, or at a predetermined position set in correspondence with the work.

14. A work support system according to claim 9, comprising: calculating the difference in position and movement between the reference work and the actual work; evaluating the actual work based on the difference; creating work evaluation information including an evaluation value or work caution or work instruction as an evaluation result; and displaying the work evaluation information on the screen of the display device.

15. A work support system according to claim 14, wherein the difference in position and movement between the reference work and the actual work is calculated as a difference in hand position, work evaluation information is created based on the difference in hand position, and an image is displayed on the screen of the display device with a different display mode according to the work evaluation information.

16. A work support system according to claim 14, wherein the difference in position and movement between the reference work and the actual work is calculated as a difference in head position, work evaluation information is created based on the difference in head position, and an image is displayed on the screen of the display device with a different display mode according to the work evaluation information.

Images