Operator support method and operator support system

Abstract

The operator assistance system uses a wearable device equipped with a camera, voice interface, and display to capture hand movements and ambient conditions, thereby improving the efficiency of users such as field workers by enabling real-time task monitoring and interaction via natural language. [Solution] The system integrates time-series analysis into a skill assessment mechanism that assesses user proficiency by comparing captured task data with pre-stored data. Based on the assessment, the machine learning system adjusts user instructions to perform specific tasks. The system adapts to the individual user's skill level, thereby improving the workflow and reducing confusion.

Description

Technical Field

[0001] This disclosure generally aims at improving productivity, and more specifically, targets an operator assistance method and an operator assistance system for improving productivity in a manufacturing environment.

Background Art

[0002] As manufacturing processes become increasingly complex, the tasks performed by on-site workers have become more intricate. The development of factory digitalization has led to an increased deployment of fixed digital terminals such as kiosks that provide instructions to operators based on work orders including various subtasks and operation procedures. However, in many factory environments, these digital terminals are often installed in separate locations along with parts storage areas, manuals, and assembly work areas. Such spatial separation can lead to inefficiencies, especially when an operator faces situations or uncertainties outside the scope of the specific instructions provided. In such cases, the operator needs to interrupt the task to seek clarification, thereby disrupting the work flow and reducing overall productivity.

Summary of the Invention

Problems to be Solved by the Invention

[0003] To mitigate these problems, there is a growing interest in wearable devices that can be used on-site. These devices have the potential to provide real-time assistance and information without the user having to leave the workstation. The recent development of generative AI has further enhanced the functionality of these wearable devices by enabling interaction through natural language. However, adjusting the responses from the generative AI system to match the skill levels of individual users remains a significant challenge.

[0004] Therefore, there is a need for systems and methods that integrate these technologies into wearable devices to provide operators with skill level adaptive guidance, thereby improving productivity. [Means for solving the problem]

[0005] In some aspects of the present disclosure, a method for assisting an operator using a device such as a wrist-worn device includes: accessing a database to obtain task data associated with time-series patterns that correspond to multiple skill levels and represent a set of actions associated with a task, in response to obtaining task-related features associated with time-series data associated with objects in a first set of images captured by one or more cameras; applying time-series analysis to the task-related features and task data to determine time-series similarity representing the degree of match between the task-related features and task data; estimating the content of the task based on object recognition results from the time-series similarity; setting user skill level information based on the time-series similarity and task data; generating instructions using machine learning based on at least the skill level information and task-related features; and communicating the instructions to a device coupled to one or more cameras.

[0006] In some embodiments, task-related features include the transition of an object in a first set of images, or the time interval between two events in time-series data representing the duration of the task.

[0007] In some embodiments, setting skill level information involves accessing skill evaluation tables in a database to calculate or adjust time-series similarity, and generating instructions involves using a search-enhancement-generating system that incorporates skill level information based on task-related features identified in time-series data and retrieves information from the database. The search-enhancement-generating system may generate instructions using task-related user input.

[0008] In some embodiments, the method may further include monitoring the user's performance during task execution to collect performance data, analyzing the performance data to adjust the skill level, and storing the performance data or at least one of the user input in a knowledge storage system for future reference. The knowledge storage system may categorize the stored data according to the user's skill level to facilitate modifications to at least one of the instructions or manuals.

[0009] In some embodiments, the device may include a first camera, which is a wide-angle camera configured to simultaneously capture images in real time, including hand gestures involving both hands, in response to receiving user instructions in natural language format in a voice interface. The device may further include a second camera, one or more cameras configured to capture and display a second set of images representing the surrounding environment.

[0010] In some embodiments, the techniques described herein relate to a system that uses a device to assist an operator, the system comprising: a device coupled to one or more cameras; a database configured to store task data associated with time-series patterns representing a set of actions corresponding to multiple skill levels and associated with a task; a task estimation unit configured to analyze task-related features associated with time-series data from objects in a first set of images captured by one or more cameras, and to estimate the content of a task based on object recognition results from time-series similarity; and a computing and communication system configured to be coupled to the database and at least one of the device or the task estimation unit, wherein the computing and communication system comprises: a similarity calculation unit that applies time-series analysis to task-related features and task data to obtain time-series similarity based on the degree of match between task-related features and task data; a skill level determination unit configured to set user skill level information based on time-series similarity and task data; and a work instruction generation unit configured to generate instructions using machine learning based on at least the skill level information and task-related features, and to communicate the instructions to a device, for example, a wrist-worn device.

[0011] In some embodiments, the system may include a voice interface configured to take user instructions in natural language, and the device may include a first camera, which is a wide-angle camera configured to simultaneously capture images in real time, including hand gestures involving both hands, in response to the voice interface taking user instructions.

[0012] In some embodiments, the computing and communication system may include a search-enhancement-generating system that generates instructions based on skill level information by retrieving information from a database. The search-enhancement-generating system may generate instructions using user input related to the task.

[0013] In some embodiments, the device may include a second camera configured to capture and display a second set of images representing the surrounding environment.

[0014] In some embodiments, task-related features associated with time-series data include the transition of an object in a first set of images, or the time interval between two events in the time-series data representing the duration of a task.

[0015] In some embodiments, the system may further include a knowledge storage system that categorizes stored data according to user skill levels to facilitate modifications to at least one of instructions or manuals. The knowledge storage system may store performance data or user input for future reference.

[0016] In some embodiments, computing and communication systems may be configured to monitor and analyze performance data during task execution and adjust skill levels accordingly.

[0017] Aspects of the present disclosure may include a system, which may include means for performing steps including: accessing a database to obtain task data associated with time-series patterns that correspond to multiple skill levels and represent a set of actions associated with a task, in response to obtaining task-related features associated with time-series data associated with objects in a first set of images captured by one or more cameras; means for applying time-series analysis to the task-related features and task data to determine time-series similarity representing the degree of match between the task-related features and task data; means for estimating the content of a task based on object recognition results from the time-series similarity; means for setting user skill level information based on the time-series similarity and task data; means for generating instructions using machine learning based on at least the skill level information and task-related features; and means for communicating the instructions to a device coupled to one or more cameras. [Brief explanation of the drawing]

[0018] [Figure 1] This figure shows wearable devices according to various embodiments of the present disclosure. [Figure 2] This is a functional configuration diagram of a skill evaluation system according to various embodiments of the present disclosure. [Figure 3] This is a process flow for determining a user's skill level according to various embodiments of the present disclosure. [Figure 4] This figure shows the details of the process of the steps shown in Figure 3. [Figure 5] This figure shows the details of the work instruction generation process, as shown in Figure 3. [Figure 6] This figure shows alternative wearable device configurations according to various embodiments of the present disclosure. [Figure 7] This figure shows alternative functional configurations according to various embodiments of the present disclosure. [Figure 8]A flowchart for generating a work instruction in response to a user question according to various embodiments of the present disclosure. [Figure 9] A diagram showing details of the process of the steps shown in FIG. 8. [Figure 10] A diagram showing a system configuration for storing operator knowledge according to various embodiments of the present disclosure. [Figure 11] A flowchart showing an exemplary process for assisting an operator using a device according to various embodiments of the present disclosure. [Figure 12] A diagram showing an exemplary computing environment having an example of a computer device according to various embodiments of the present disclosure.

Mode for Carrying Out the Invention

[0019] The following detailed description provides details of the drawings and exemplary implementations of the present application. Reference numerals and descriptions of overlapping elements between the drawings are omitted for clarity. The terms used throughout the description are provided by way of example and are not intended to be limiting. For example, the use of the term "automatically" may include a fully automatic implementation or a semi-automatic implementation with user or administrator control for a particular aspect of the implementation, depending on the desired implementation by those skilled in the art practicing the implementations of the present application. The selection can be made by the user through a user interface or other input means, or can be realized through a desired algorithm. Exemplary implementations as described herein can be used alone or in combination, and the functionality of the exemplary implementations can be realized through any means according to the desired implementation.

[0020] Existing work assistance devices using AR technology generally require the use of handheld devices such as smartphones or tablets, which can disrupt the workflow and pose safety risks in factory settings. While on-site monitoring solutions using fixed cameras are effective for wide-area monitoring, they suffer from blind spots and generally fail to capture detailed hand movements, making it difficult to fully understand the tasks being performed.

[0021] Existing first-person video analysis technologies using head-mounted or wrist-mounted cameras can capture larger object images compared to fixed cameras, potentially improving the worker's task recognition rate. However, head-mounted cameras often capture irrelevant information unrelated to the task, complicating real-time recognition by requiring additional processing to remove irrelevant data. Conversely, existing wrist-mounted cameras capture only the hand and filter out irrelevant information, but often capture only one hand, making it difficult to recognize detailed aspects of the task. Therefore, it would be desirable to have systems and methods that support task execution without interrupting the workflow, adapt to the user's skill level, reduce the workload on the worker, and thereby improve productivity and efficiency.

[0022] Figure 1 shows a wearable device according to various embodiments of the present disclosure. As shown, the wearable device 10 may be implemented as a wrist-worn device that can be worn on the user's left hand (indicated as number 1 in Figure 1) or right hand (indicated as number 2).

[0023] Figure 2 is a functional configuration diagram of a skill evaluation system according to various embodiments of the present disclosure. In one embodiment, the skill evaluation system 150 comprises a wearable device 10, a computing and communication system 100, and a database 110. As shown in Figure 2, the wearable device 10 comprises a processor 11, cameras 12 and 13, an audio interface 14, a display 15, a wireless communication unit 16, a memory 17, and a task estimation unit 18. The computing and communication system 100 comprises a processor 101, a similarity calculation unit 102, a skill level determination unit 103, a work instruction generation unit 104, and a wireless communication unit 105. The database 110 stores time-series task or feature data 111 corresponding to skill levels, a skill evaluation table 112 corresponding to feature similarity, and a work manual 113.

[0024] During operation, the processor 11 of the wearable device 10 may process image data received from one or more cameras 12, 13, as well as audio data from the voice interface 14. The wearable device 10 may display the results on the display 15. In embodiments, camera 12 may be implemented as a wide-angle camera configured to simultaneously capture the movements of both hands in real time, for example, and camera 13 may be positioned opposite camera 12 and used to perceive ambient conditions or to capture images of other devices. The voice interface 14 comprises a microphone and speaker and is used to receive user questions in natural language and to reproduce system responses by the system 100. In embodiments, the voice interface 14 may utilize a wireless communication device such as Bluetooth earphones. The display 15 may be used to present simple work instructions and images captured by camera 13. In addition, the display 15 may be equipped with touchscreen functionality to allow the user to control and decide on subsequent tasks by touching the screen. The wireless communication unit 16 may be used to transmit the results processed by the processor 11, images captured by cameras 12 and 13, and user input via the audio interface 14, as well as to receive responses from the system 100. The memory 17 temporarily stores the captured images and various data transmitted from the computing and communication system 100. The task estimation unit 18 estimates the task content in a time-series format based on the images captured by camera 12.

[0025] In this embodiment, the processor 101 of the computing and communication system 100 may process data transmitted by the wearable device 10, including data from a similarity calculation unit 102, a skill level determination unit 103, and / or a work instruction generation unit 104, for example, to generate instructions, and may also process data from a wireless communication unit 105, for example, to communicate with the wearable device 10.

[0026] The database 110 may store time-series task data 111 corresponding to skill levels, a skill evaluation table 112 corresponding to feature similarity, and a work manual 113. In an embodiment, the similarity calculation unit 102 may calculate time-series similarity by comparing the feature data calculated by the task estimation unit 18 of the wearable device 10 with the feature data 111 stored in the database 110. The skill level determination unit 103 may then determine the skill level by comparing the results calculated by the similarity calculation unit 102 with the skill evaluation table 112. The skill level information determined by the skill level determination unit 103 and the work manual 113 stored in the database may be input to a generating AI system (not shown) to generate instructions tailored to the user's skill level. The instructions are then returned to the wearable device 10 for display and playback.

[0027] Figure 3 shows a process flow for determining a user's skill level according to various embodiments of the present disclosure. In an embodiment, process 300 may begin in step S11 when a work start signal is transmitted from a wearable device (such as wearable device 10 shown in Figure 1) to a computing and communication system (such as system 100 shown in Figure 1).

[0028] In step S101, upon receiving a work start signal, the computing and communication system may access a database to verify the worker's work order and subtasks.

[0029] In step S102, the system may send a task start signal to the wearable device.

[0030] In step S12, when the wearable device receives a task start signal, it may activate the camera to capture the worker's task, for example, at regular intervals.

[0031] From the image captured in S12, the task estimation unit may, in step S13, estimate the task content based on the object recognition result and transmit this information to the computing and communication system. Subsequently, the similarity calculation unit (102) may compare the task content stored in the database (110) with that calculated in step S13 and calculate the fit of the task content in step S103. The data output from the task estimation unit (18) may include time-series data, including the recognition speed for each of the multiple objects, as shown in the table in Figure 4. The data stored in the database (110) may include task content arranged in time-series format.

[0032] In step S104, the task suitability and skill evaluation table 112 calculated in step S103 may serve as input to a skill level determination unit, for example, to determine the user's skill level. In one embodiment, the skill evaluation table 112 shown in Figure 4 allows for the calculation of the user's skill level based on the suitability of the task content. It is understood that the relationship between the suitability of the task content and the user's skill level information may be modified according to each work order or subtask. The skill level information determined in step S104, along with the work manual (113) stored in the database (110), may be input to the work instruction generation unit (104) to generate work instructions that are adjusted to the user's skill level in step S105. Finally, in step S14, the generated work instructions may be sent to a wearable device (10) for display on a display (15) and playback through a voice interface 14.

[0033] Figure 4 shows details of the processes of steps S103 and S104 shown in Figure 3, according to various embodiments of the present disclosure. In embodiments, in step S103, the tasks estimated by the task estimation unit of the wearable device (10) and the tasks (111) stored in the database (110) serve as input to the similarity calculation unit (102) for calculating task fit. Subsequently, the results calculated in step S103 may be compared with user skill level information data to determine the user's skill level.

[0034] Figure 5 shows details of the work instruction generation process shown in Figure 3 according to various embodiments of the present disclosure. In the embodiments, a work instruction generation processing unit (104) retrieves and extracts relevant information from the skill level calculated in step S104 and the work manual (113) stored in the database (110) as input to a search and extension generation system. The search and extraction results may then be provided to a generation AI system (not shown) which generates and outputs appropriate work instructions, which may be communicated to a wearable device (10) to facilitate worker assistance.

[0035] Figure 6 shows alternative wearable device configurations according to various embodiments of the present disclosure, characterized in that cameras on both hands capture tasks performed by each hand. In this configuration, the camera 19 on the left hand 1 captures tasks performed by the right hand 2, and the camera 21 on the right hand 2 captures tasks performed by the left hand 1. This configuration increases the number of cameras but allows for a one-to-one correspondence between the movement of each hand and the camera, which is expected to improve the estimation accuracy of the task estimation unit 18 of the wearable device 10. The wearable device 20 worn on the right hand also has the same wireless functionality as the wearable device 10. Images captured by camera 21 are transmitted to the wearable device 10, and the task estimation unit 18 uses the results from both cameras 19 and 21 to estimate the task.

[0036] Figure 7 shows alternative functional configurations according to various embodiments of the present disclosure. In embodiments, the functional configuration of Figure 2 may be modified as shown in Figure 7. As shown in Figure 7, the task estimation unit 18 is contained within the processor 101 of the computing and communication system 100. Image data captured by the camera 12 of the wearable device 10 is also transmitted to the computing and communication system 100, which may potentially degrade the real-time performance of task estimation. However, since system resources become available, it is possible to perform a wider range of deep learning processes (such as object detection and feature extraction) compared to the wearable device 10. This may, as a result, potentially improve the accuracy of task estimation.

[0037] Figure 8 shows flowcharts for generating work instructions in response to user questions according to various embodiments of the present disclosure. Unlike the embodiment associated with Figure 3, in which the system proactively presents the next instruction without requiring a question from the worker, the flowchart 800 in Figure 8 generates work instructions when the user presents a question. The processing steps up to S104 are the same as in Figure 3. In the embodiment, after determining the worker's skill level in S104, the system determines whether a question has been detected, for example, from a wearable device 10. If there is no question (S15: no), the process returns to S12 and the skill level evaluation is performed again. If there is a question (S15: yes), as shown in Figure 9, in step S108, the content of the question from the wearable device (10), the skill level determined in step S104, and the work manual 113 stored in the database (110) may be input to the generating AI to output work instructions. In step S14, the generated work instructions are then sent to a wearable device (10), displayed on a display (15), and played back through an audio interface (14).

[0038] In this embodiment, by calculating the worker's skill level before a question arises, it becomes possible to respond to the worker's question and generate a prompt response that is adjusted to the skill level.

[0039] Figure 10 shows a system configuration for storing worker knowledge according to various embodiments of the present disclosure. In the embodiments, a knowledge processing unit (106) may categorize the user's actions and the content of the user's questions based on skill level determination using a skill level determination unit 103, and store them in a knowledge storage 114, thereby enabling improved work efficiency and revision of manuals for greater clarity.

[0040] Figure 11 is a flowchart illustrating exemplary processes for assisting an operator using a device according to various embodiments of the present disclosure. In embodiments, the operator-assisting process 1100 may begin in step 1102 when a database is accessed in response to the device obtaining task-related features associated with time-series data associated with objects in a first set of images captured by a camera coupled to the device. The database stores task data associated with time-series patterns, which correspond to multiple skill levels and represent a set of actions associated with a task.

[0041] In step 1104, time series analysis may be applied to task-related features and task data to determine time series similarity, which represents the degree of match between the task-related features and the task data.

[0042] In step 1106, the content of the task may be estimated based on the object recognition results from the first set of images.

[0043] In step 1108, the user's skill level may be determined based on time-series similarity and task data.

[0044] In step 1110, machine learning may be used to generate instructions based on at least skill level and task-related characteristics.

[0045] Finally, in step 1120, the instructions may be communicated to the device.

[0046] Those skilled in the art will recognize that (1) certain steps may be performed at their discretion, (2) the steps are not limited to the specific order described herein, (3) certain steps may be performed in a different order, and (4) certain steps may be performed simultaneously.

[0047] Figure 12 shows an example computing environment having an example of a computer device suitable for use in several implementations according to various embodiments of the present disclosure. The computer device 1205 of the computing environment 1200 may include one or more processing units, cores, or processors 1210, memory 1215 (e.g., RAM, ROM, and / or others), internal storage 1220 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or an I / O interface 1225, all of which may be coupled by a communication mechanism or bus 1230 for communicating information, or embedded in the computer device 1205. The I / O interface 1225 may also be configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation.

[0048] Computer device 1205 can be communicatively coupled to input / user interface 1235 and output device / interface 1240. Either or both of input / user interface 1235 and output device / interface 1240 may be wired or wireless interfaces and may be detachable. Input / user interface 1235 may include any physical or virtual device, component, sensor, or interface that can be used to provide input (e.g., buttons, touchscreen interfaces, keyboards, pointing / cursor controls, microphones, cameras, Braille, motion sensors, optical readers, and / or others). Output device / interface 1240 may include displays, televisions, monitors, printers, speakers, Braille, etc. In some exemplary implementations, input / user interface 1235 and output device / interface 1240 may embed or be physically coupled to computer device 1205. In other exemplary implementations, other computer devices may function as or provide input / user interface 1235 and output device / interface 1240 of computer device 1205.

[0049] Examples of computer devices 1205 may include advanced mobile devices (e.g., smartphones, devices in automobiles and other machines, devices carried by people and animals), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for portability (e.g., desktop computers, other computers, information kiosks, televisions with one or more processors embedded and / or televisions, radios with them combined, etc.).

[0050] Computer device 1205 can be communicatively coupled to external storage 1245 and network 1250 (for example, via I / O interface 1225) to communicate with any number of networked components, devices, and systems, including one or more computer devices of the same or different configurations. Computer device 1205, or any connected computer device, can function as, provide, or be referred to as a server, client, thin server, general-purpose machine, dedicated machine, or any other label.

[0051] The I / O interface 1225 may include wired and / or wireless interfaces that use any communication or I / O protocol or standard (e.g., Ethernet, 802.11x, Universal System Bus, WiMAX, modem, cellular network protocol, etc.) to communicate information with at least all connected components, devices, and networks of the computing environment 1200. The network 1250 may be any network or combination of networks (e.g., the Internet, local area network, wide area network, telephone network, cellular network, satellite network, etc.).

[0052] Computer device 1205 may use computer-usable or computer-readable media, including temporary and non-temporary media, and / or may use them for communication. Temporary media include transmission media (e.g., metal cables, optical fibers), signals, carrier waves, etc. Non-temporary media include magnetic media (e.g., disks and tapes), optical media (e.g., CD-ROMs, digital video discs, Blu-ray discs), solid-state media (e.g., RAM, ROMs, flash memory, solid storage), and other non-volatile storage or memory.

[0053] Computer device 1205 can be used to implement techniques, methods, applications, processes, or computer executable instructions in several computing environment examples. Computer executable instructions can be retrieved from temporary media and can be stored in and retrieved from non-temporary media. Executable instructions can be in one or more of any programming languages, scripting languages, and machine languages ​​(e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0054] The processor 1210 can run under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be deployed, including a logic unit 1260, an application programming interface (API) unit 1265, an input unit 1270, an output unit 1275, and an inter-unit communication mechanism 1295 for different units to communicate with each other, with the OS, and with other applications (not shown). The units and elements described are diversified in design, function, configuration, or implementation, and are not limited to the description provided. The processor 1210 can be in the form of a hardware processor such as a central processing unit (CPU), or a combination of hardware and software units.

[0055] In some exemplary implementations, when information or execution instructions are received by the API unit 1265, they may be communicated to one or more other units (e.g., logic unit 1260, input unit 1270, output unit 1275). In some examples, the logic unit 1260 may control the flow of information between units and be configured to direct the services provided by the API unit 1265, input unit 1270, and output unit 1275 in some exemplary implementations described above. For example, the flow of one or more processes or implementations may be controlled by the logic unit 1260 alone or in combination with the API unit 1265. The input unit 1270 may be configured to receive input for the computations described in the exemplary implementations, and the output unit 1275 may be configured to provide outputs based on the computations described in the exemplary implementations.

[0056] The processor 1210 can be configured to execute a method or computer instruction, which may involve accessing a database to obtain task data associated with time-series patterns that correspond to multiple skill levels and represent a set of actions associated with a task, in response to obtaining task-related features associated with time-series data associated with objects in a first set of images captured by one or more cameras, for example as described with respect to Figures 1 and 2. The processor 1210 can be configured to execute a method or computer instruction, which may involve applying time-series analysis to task-related features and task data to determine time-series similarity representing the degree of match between task-related features and task data, and estimating the content of the task based on object recognition results from the time-series similarity, for example as described with respect to Figures 3 and 4. The processor 1210 can be configured to execute a method or computer instruction, which may include, for example, setting user skill level information based on time-series similarity and task data, generating instructions using machine learning based on at least the skill level information and task-related features, and communicating the instructions to a device coupled to one or more cameras, as described with respect to Figures 2 and 3.

[0057] Some parts of the detailed description are presented with respect to algorithms and symbolic representations of computer operations. These algorithmic descriptions and symbolic representations are means used by those skilled in the field of data processing to communicate the essence of technological innovations to others skilled in the field. An algorithm is a set of defined steps that lead to a desired final state or result. In exemplary implementations, the steps performed require the physical manipulation of tangible quantities to achieve a tangible result.

[0058] Unless otherwise specifically stated, as is evident from the discussion, any discussion using terms such as “processing,” “computing,” “calculating,” “decision,” and “display” throughout the explanation is understood to include the operations and processes of a computer system or other information processing device that manipulate data presented as physical (electronic) quantities in the registers and memory of a computer system and convert it into other data similarly presented as physical quantities in the memory or registers of a computer system or other information storage, transmission, or display devices.

[0059] Exemplary implementations may also relate to apparatus for performing the operations described herein. This apparatus may include one or more general-purpose computers, which may be specifically constructed for a particular purpose or which may be selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored on computer-readable media, such as computer-readable storage media or computer-readable signal media. Computer-readable storage media may include tangible media, such as optical disks, magnetic disks, read-only memory, random-access memory, solid-state devices, drives, or any other type of tangible or non-temporary medium suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Computer programs may include purely software implementations that include instructions for performing the operations of a desired implementation.

[0060] Various general-purpose systems may be used with the programs and modules illustrated herein, or may prove useful in constructing more specialized devices to perform desired method steps. Furthermore, the exemplary implementations are not described with reference to any particular programming language. It will be recognized that various programming languages ​​may be used to implement the techniques of the exemplary implementations described herein. Instructions in a programming language may be executed by one or more processing devices, such as a central processing unit (CPU), processor, or controller.

[0061] As is known in the art, the operations described above can be implemented by hardware, software, or any combination of software and hardware. Various embodiments of the exemplary implementations may be implemented using circuits and logic devices (hardware), while other embodiments may be implemented using instructions stored in a machine-readable medium (software) that, when executed by a processor, would cause the processor to implement the methods for implementing the implementations of this application. Furthermore, some implementations of this application may be implemented solely by hardware, while other exemplary implementations may be implemented solely by software. Moreover, the various functions described may be implemented in a single unit or spread across multiple components in various ways. When implemented by software, the methods may be executed by a processor, such as a general-purpose computer, based on instructions stored in a computer-readable medium. If desired, the instructions may be stored in the medium in a compressed and / or encrypted form.

[0062] Furthermore, other embodiments of this application will become apparent to those skilled in the art by considering this specification and practicing the techniques of this application. The various embodiments and / or components of the exemplary embodiments described herein may be used individually or in any combination. This specification and the exemplary embodiments are to be considered merely examples, and the true scope and intent of this application are indicated by the following claims. [Explanation of symbols]

[0063] 1 Left hand, 2 Right hand, 10, 20 Wearable device, 11 Processor, 12, 13, 19, 21 Camera, 14 Voice interface, 15 Display, 16 Wireless communication unit, 17 Memory, 18 Task estimation unit, 100 Communication system, 101 Processor, 102 Similarity calculation unit, 103 Skill level determination unit, 104 Work instruction generation unit, 105 Wireless communication unit, 110 Database, 111 Time-series task data, 112 Skill evaluation table, 113 Work manual, 114 Knowledge storage, 150 Skill evaluation system, 1200 Computing environment, 1205 Computer device, 1210 Processor, 1215 Memory, 1220 Internal storage, 1225 I / O interface, 1230 Bus, 1235 Input / user interface, 1240 Output device / interface, 1245 External storage, 1250 network, 1260 logic units, 1265 API units, 1270 input units, 1275 output units, 1295 inter-unit communication mechanisms

Claims

1. A method of assisting an operator using a device, In response to obtaining time-series data associated with objects and associated task-related features in a first set of images captured by one or more cameras, access a database to obtain task data associated with time-series patterns that correspond to multiple skill levels and represent a set of actions associated with a task, Applying time series analysis to the task-related characteristics and the task data to determine the time series similarity, which represents the degree of match between the task-related characteristics and the task data, The content of the task is estimated based on the object recognition results from the time-series similarity, Based on the aforementioned time-series similarity and task data, the user's skill level information is set. Using machine learning to generate instructions based on at least the skill level information and the task-related characteristics, An operator assistance method comprising communicating the aforementioned instructions to a device coupled to one or more cameras.

2. The operator assistance method according to claim 1, wherein the task-related features associated with time-series data include the movement of the object in the first image set.

3. The operator assistance method according to claim 1, wherein the task-related features associated with time-series data include a time interval between two events in the time-series data representing the duration of the task.

4. The operator support method according to claim 1, wherein setting the skill level information includes accessing the skill evaluation table in the database and calculating or adjusting the time-series similarity.

5. The operator assistance method according to claim 1, wherein generating the instructions includes using a search-enhancement-generation system that incorporates the skill level information based on task-related features identified in the time-series data and retrieves the information from the database.

6. The operator assistance method according to claim 5, wherein the search extension generation system further generates the instructions using user input related to the task.

7. This involves monitoring the user's performance while they are executing a task and collecting performance data. Analyzing the aforementioned performance data to adjust the skill level, The performance data or at least one of the user inputs is stored in a knowledge storage system for future reference. The operator support method according to claim 6, further comprising:

8. The operator support method according to claim 7, wherein the knowledge storage system categorizes the stored data according to the user's skill level to facilitate the modification of at least one of instructions or manuals.

9. The operator assistance method according to claim 1, wherein the device is a wrist-worn device, and the first camera among the one or more cameras is a wide-angle camera configured to simultaneously capture images in real time, including hand gestures involving both hands, in response to acquiring user instructions in natural language format at an audio interface.

10. The operator assistance method according to claim 9, wherein the device comprises a second camera among the one or more cameras configured to capture and display a second set of images representing the surrounding environment.

11. A system that uses a device to assist an operator, A device coupled to one or more cameras, A database configured to store task data associated with time-series patterns that represent a series of actions associated with a task, while also supporting multiple skill levels, A task estimation unit is configured to analyze task-related features associated with time-series data from objects in a first set of images captured by one or more cameras, and to estimate the content of the task based on the object recognition result from the time-series similarity. The system comprises the database and a computing and communication system configured to be coupled to at least one of the device or the task estimation unit, and the computing and communication system is A similarity calculation unit that applies time series analysis to the task-related features and the task data, and obtains the time series similarity based on the degree of match between the task-related features and the task data, A skill level determination unit configured to set user skill level information based on the aforementioned time-series similarity and task data, An operator assistance system comprising: a work instruction generation unit configured to generate instructions using machine learning based on at least the skill level information and the task-related characteristics, and to communicate the instructions to the device.

12. The operator support system according to claim 11, further comprising a voice interface configured to acquire user instructions in natural language format.

13. The operator assistance system according to claim 12, wherein the device is a wrist-worn device, and the first camera among the one or more cameras is a wide-angle camera configured to simultaneously capture images in real time, including hand gestures involving both hands, in response to the voice interface receiving the user instruction.

14. The operator support system according to claim 11, further comprising a search extension generation system that generates the instructions based on the skill level information by searching for information from the database of the computing and communication system.

15. The operator support system according to claim 14, wherein the search extension generation system further generates the instructions using user input related to the task.

16. The operator assistance system according to claim 11, wherein the device comprises a second camera among the one or more cameras configured to capture and display a second set of images representing the surrounding environment.

17. The operator support system according to claim 11, wherein the task-related features associated with time-series data include at least one of the transition of the object in the first image set or the time interval between two events in the time-series data representing the duration of the task.

18. The operator support system according to claim 15, further comprising a knowledge storage system that categorizes stored data according to user skill levels to facilitate modification of at least one of instructions or manuals.

19. The operator support system according to claim 18, wherein the knowledge storage system stores performance data or at least one of the user inputs for future reference.

20. The operator support system according to claim 19, wherein the computing and communication system is configured to monitor and analyze the performance data during task execution and to adjust the skill level.

Images