Method and system for determining compliance

Abstract

The invention generally relates to a computer system adapted to determine to what extent a manipulation of a first object by a user using a second object adheres to a predefined task to be performed in relation to the first object. This is in line with the invention of determining a state of the first object when the user has manipulated the first object using the second object by applying a machine learning based object processing scheme, and comparing the state to data defining processing steps to be performed for completing the predefined task. The invention also relates to a corresponding computer implemented method and computer program product.

Description

Technical Field

[0001] This invention generally relates to a computer system adapted to determine the extent to which a user's manipulation of a first object using a second object corresponds to a predetermined task to be performed regarding the first object. This is consistent with the present invention, which determines the state of the first object when the user has manipulated it using a machine learning-based object processing scheme, and compares that state with data used to define the processing steps to be performed to complete the predetermined task. The invention also relates to a corresponding computer and computer program product for implementing the method. Background Technology

[0002] Recent advancements in computer and communication technologies have impacted problem-solving, enabling users to quickly find relevant information to solve entirely new tasks. Typically, users utilize electronic devices, such as mobile phones or computers, to query relevant information in databases. The information generated from the query is then displayed on the screen included with the electronic device.

[0003] In some recent implementations, the display screen can be integrated with a head-mounted device, potentially allowing the generated information to be presented within the user's field of vision, and potentially allowing the generated information to enhance the "realistic view." An example of such a head-mounted device is given in US9285589, which introduces an augmented reality system including a perspective display arranged to display augmented reality images in the user's viewing direction.

[0004] Therefore, US9285589 proposes a very interesting method that allows information to be displayed in the user's viewing direction while the user can simultaneously see their surroundings. The augmented reality system disclosed in US9285589 has shown usefulness in many different areas, such as assisting users in unknown environments or providing guidance when handling the aforementioned novel tasks.

[0005] However, even though the solution proposed in US9285589 has a positive impact on assisting ordinary users, it still relies on users formulating correct queries to present the relevant information. Browsing irrelevant information is certainly time-consuming, and if irrelevant information happens to be presented to the user, it can be even more intrusive.

[0006] Therefore, there seems to be room for further improvement in assisting users, for example, when performing scheduled tasks, the overall goal is to ensure that the task is performed in the best possible way. Summary of the Invention

[0007] According to one aspect of the invention, a computer system is thus provided, adapted to determine a compliance level of a predefined task performed by a user in relation to a first object. The computer system includes a processing unit and an object capturing device, wherein the computer system is adapted to receive data from a memory element arranged to communicate with the processing unit, defining processing steps to be performed to complete the predefined task; using the object capturing device, acquiring a first representation of a region to be performed on the predefined task; using the processing unit, applying a machine learning-based object processing scheme to the acquired first representation to identify the first object and at least a second object involved in performing the predefined task, wherein the first object is independent of the user and the second object is operated by the user; using the processing unit, automatically identifying the state of the first object based on manipulation of the first object using the second object when operated by the user; and using the processing unit, determining a compliance level of the predetermined task based on a combination of the state of the first object and data defined as processing steps to be performed to complete the predetermined task.

[0008] The present invention is based on the understanding that allowing a method to automatically determine the extent to which a user performs a predefined task while manipulating a first object would have significant advantages. This is consistent with the present invention, which is achieved by applying a method to automatically identify the first object to be manipulated by a second object operated by the user and associating it with data defining the processing steps for completing the predetermined task.

[0009] The computer system can then use the results of the correlation in many different ways, for example, to determine quality indicators about the first object being manipulated, such as how well a user performs in a predefined task of manipulating the first object. According to the invention, the results of the correlation can also be used to guide the user / operator in the "right direction," ensuring that the predetermined task is completed optimally.

[0010] The computer system according to the present invention can obviously be used in different fields where object manipulation is to be performed, for example, with regard to a manual or semi-manual process in which a user / operator manipulates a first object using a second object, or with regard to an automatic / autonomous manufacturing process in which a user operates a second object using, for example, some form of control system (e.g., defining a second object), i.e., not necessarily in direct contact with the first object.

[0011] Therefore, according to the present invention, the first object should be understood as independent of the user (i.e., not part of the user, and therefore not, for example, the user's hand / arm / leg), and is an object requiring some form of processing. This means that the first object can be a combination of multiple parts together (possibly over time). As the first object is manipulated (e.g., by adding components to the first object), the first object can "grow" over time. Correspondingly, the second object can also be defined as one or more elements together forming the second object.

[0012] For example, in one exemplary embodiment, the second object can be considered as the user's / operator's hand, wherein the elements forming the second object are the fingers of the user's / operator's hand. However, the second object may also include other elements, such as tools in the user's / operator's hand.

[0013] When the second object is defined as the user's / operator's hand, not only can the hand be identified and tracked, but its posture, including finger positions, can also be determined. Correspondingly, the hand's posture relative to the first object and the manner in which the hand intersects with the first object can also be determined. Tracking intersecting objects can also be used to verify sensitive or restricted parts, such as surfaces of the first object that are not touched. Other cases can verify objects that are being manipulated, such as being pushed, pressed, shaken, or moved.

[0014] The method for identifying and tracking a hand can be similar to that for tracking a first object in general, i.e., using a machine learning-based object processing scheme. The first object is identified as a hand. According to the definition of this invention, a machine learning-based object processing scheme is applied to identify the first and second objects. Generally, it is best to ensure that the machine learning-based object processing scheme has been "trained" to quickly identify different object types, possibly based on previously collected images of different object types. However, training does not necessarily have to be performed for each computer system, but can be performed in advance in a general manner when developing the machine learning-based object processing scheme.

[0015] It should be further understood that machine learning-based object processing schemes can also be used by computer systems to identify the state of objects and determine compliance levels. Such capabilities can be formed by training the machine learning-based object processing scheme based on previously collected data of operations performed on a first object using a second object. For example, a vision system may have collected image data related to a user's manual manipulation of a first object. The image data collected by the vision system can then be used as training data for the machine learning-based object processing scheme.

[0016] It should be further understood that object processing schemes based on machine learning can be implemented using a combination of one or more different machine learning algorithms, including neural networks in deep learning, as well as artificial neural networks (ANNs), such as, but not limited to, convolutional neural networks (CNNs), feedforward neural networks (FNNs), etc.

[0017] The concept according to the invention can generally be implemented using a variety of different sensor systems that include object-capturing devices for acquiring a first representation of an area to perform a predetermined task. Examples of such sensor systems that may include object-capturing devices include image-capturing devices (e.g., cameras), lidar, radar, laser scanners, and thermal sensors. Other sensor systems, both now and in the future, are of course possible and are also within the scope of the invention. It is also possible to combine more than one sensor with an object-capturing device, such as an image-capturing device and lidar.

[0018] In one embodiment, the object capturing device includes an image capturing device, and the first representation includes an image acquired using the image capturing device. In such an embodiment, the machine learning-based object processing scheme is represented by a machine learning-based image processing scheme.

[0019] In some implementations, it may be desirable to implement the machine learning-based object processing scheme as a supervised machine learning process. Because the machine learning process is supervised, it is possible for an operator / user to "correct" decisions made by the machine learning-based object processing scheme that the operator / user considers incorrect. In exemplary implementations involving supervised machine learning processes, the processing unit may be further adapted to access a digital storage unit including a previously stored set of training objects similar to the first and second objects, and to compare the set of training objects with the object including the first representation.

[0020] However, it should be understood that, contrary to the above, machine learning-based object processing schemes can also be implemented as unsupervised machine learning processes, allowing the implementation to be completely autonomous in terms of recognition and determination. Of course, according to the content of the invention, a mixture of supervised and unsupervised participation can be permitted, for example, depending on the implementation state of the image processing scheme, such as allowing the machine learning-based object processing scheme to be initially supervised and then transition to unsupervised, or vice versa.

[0021] According to the present invention, a computer system can be further adapted to use a processing unit to compare a determined compliance level with a predetermined threshold, and to generate a feedback signal using the processing unit if the determined compliance level is lower than the predetermined threshold. That is, if it is determined that the manipulation performed by a second object is below the threshold, for example, in the case of a manual or semi-manual process, the user / operator can be notified. Accordingly, in the case of an automatic / autonomous process, the feedback signal generated by the computer system may be used automatically to "fine-tune" the automatic / autonomous process, with the aim of adjusting the process in such a way that the compliance level of subsequent manipulations is higher than the threshold.

[0022] In manual or semi-manual processes, the user may directly manipulate the first object (using, for example, a tool in their hand that defines the second object). Feedback signals can be used to provide directly relevant information to the user / operator to allow for efficient task completion, where task completion meets predetermined metrics. Therefore, it should be understood that the determination of the compliance level (and the formation of feedback signals) can be performed in a sequential manner, meaning that the execution of predetermined tasks related to the first object should be understood as potentially including multiple subsequent task steps. Therefore, ensuring the correct order of execution of predefined tasks related to the first object is possible; that is, the task steps should also be executed in the correct order. Thus, the determination of the compliance level is not merely a single execution (when all manipulations of the first object have been performed), but rather a real-time process during the time the first object is manipulated. Of course, the above discussion also applies to automatic / autonomous processes in a slightly different way.

[0023] Furthermore, according to the present invention, the formation of feedback signals can be controlled such that the feedback signals depend on the difference between a determined compliance level and a predetermined threshold. Therefore, according to the present invention, the user / operator can be notified or the automated / autonomous process can be adjusted based on the conformity of the actual operation to the predefined expected operation according to the task.

[0024] As discussed above, feedback signals can be provided to the user / operator or used to adjust automated / autonomous processes. When providing feedback signals to the user / operator, a single output interface can be used for communication. In the simplest implementation, a light source can be illuminated if the user / operator fails to complete the task (i.e., falls below a threshold). However, according to the present invention, more complex and multifaceted feedback can certainly be generated using one or more combinations of, for example, image or audio generation devices. For example, audio feedback can be provided in conjunction with image or video clips, explaining where (potentially) a problem occurred and how the user / operator should proceed (this time and / or next time) to ensure that the manipulation conforms to the predefined expectations of the task.

[0025] In a preferred embodiment of the invention, the computer system is further adapted to form an image to be provided at an output interface, wherein the image is formed by enhancing the first representation with a feedback signal. Therefore, according to the content of the invention, any form of augmented reality (AR) scheme can be used to enhance instructions / information as feedback to the user / operator. This AR feedback can also be provided in real time while the user / operator is manipulating the first object. Possibly, the image can be provided using an electronic user device, wherein the electronic user device includes or is associated with a computer system.

[0026] The electronic user device could be, for example, a mobile phone held by the user in front of the first object. However, for ease of operation, it may be preferable to provide a computer configuration (including a camera and a display screen) in such a way that the user / operator can collect the first representation (using the camera) and obtain relevant feedback (using, for example, the display screen).

[0027] However, in a preferred embodiment, it may be desirable to include and / or associate a computer system with a head-mounted device worn by the user / operator, which has an embedded camera and display element. In some embodiments, the processing unit of the computer system may be provided separately from the head-mounted device. In some embodiments, the head-mounted device may be defined as at least one of a virtual reality head-mounted device and an augmented reality head-mounted device.

[0028] As suggested, the processing unit can be provided as an embedded component of the head-mounted device. However, the processing power provided by the processing unit can also be provided elsewhere, remotely from the head-mounted device, such as on a server arranged to network with the head-mounted device (or more generally, a computer system). This server can, in turn, be included in a cloud-based computing system, where the server is defined as a so-called cloud server. Thus, the computing power provided by the present invention can be distributed among multiple processing units on a server, and the location of the processing unit / server does not need to be explicitly defined. The advantage of using a cloud-based solution also lies in the inherent redundancy achieved, and the ability to apply more complex machine learning processes compared to using a single embedded processing unit.

[0029] Within the scope of this invention, the formation of the image can be adjusted to be additionally based on data defined as the processing steps to be performed to complete a predetermined task. Therefore, the image displayed to the user / operator, for example in such embodiments, can include direct instructions on how to complete the task. An advantage of such embodiments is that novice users / operators can be trained to perform new and previously unperformed tasks, wherein the user / operator is provided with continuous information on how to successfully perform such a predetermined task.

[0030] To further improve the accuracy of the computer system, according to the present invention, a machine learning-based object processing scheme is preferably arranged to be operable to determine the interrelationships between first and second objects that may relate to the overall coordinate system within a first representation. The state of the first object can then be further identified based on the determined interrelationships between the first and second objects. In some embodiments, it may be desirable to include more than one separate camera, which may allow for the creation of a further improved first representation, including, for example, higher accuracy in the three-dimensional (3D) orientations relative to the first and second objects and their interrelationships. In such an embodiment, the coordinate system in the first representation may be a three-dimensional coordinate system.

[0031] The computer system described above can be useful in many different situations, including general manufacturing, particularly in the assembly process of the first object. An example of such an assembly process may include an electrical connector, for example, used in the automotive industry. In such an example, the first object can be defined as the electrical connector, and the second object is an operable tool suitable for removing the terminal pins or posts included in the electrical connector.

[0032] According to another aspect of the invention, a computer-implemented method is provided for operating a computer system to determine a compliance level of the execution of a predefined task associated with a first object. The computer system includes a processing unit and an object capturing device. The method includes the steps of: receiving data from a memory element arranged to communicate with the processing unit, defining processing steps to be performed to complete the predetermined task; using the object capturing device to obtain a first representation of a region where the predetermined task is to be performed; applying a machine learning-based object processing scheme to the obtained first representation using the processing unit to identify the first object and at least one second object involved in the execution of the predetermined task, wherein the first object is independent of the user and the second object is operated by the user; automatically identifying a state of the first object using the processing unit based on operations performed on the first object using the second object during user operation; and determining a compliance level of the predetermined task using the processing unit based on a combination of the state of the first object and the data defining the processing steps to be performed to complete the predetermined task. This aspect of the invention provides similar advantages to those discussed above with respect to the preceding aspects of the invention.

[0033] According to another aspect of the invention, a computer program product is provided, comprising a non-transitory computer-readable medium thereon storing computer program means for controlling a computer system adapted to determine a compliance level of a predetermined task performed by a user in relation to a first object. The computer system includes a processing unit and an object capturing device. The computer program product includes code for receiving data from a memory element arranged to communicate with the processing unit, defining processing steps to be performed to complete the predetermined task; code for using the object capturing device to obtain a first representation of a region to be performed on the predetermined task; code for using the processing unit to apply a machine learning-based object processing scheme to the obtained first representation to identify the first object and at least one second object involved in performing the predetermined task, wherein the first object is independent of the user and the second object is operated by the user; code for automatically identifying the state of the first object based on operations performed on the first object using the second object during user operation; and code for using the processing unit to determine a compliance level of the predetermined task based on a combination of the state of the first object and the data defining the processing steps to be performed to complete the predetermined task. Furthermore, this aspect of the invention provides similar advantages to those discussed above with respect to the preceding aspects of this disclosure.

[0034] Software executed by the processing unit for operating according to the content of the present invention may be stored on a computer-readable medium, which is any type of storage device, including removable non-volatile random access memory, hard disk drive, floppy disk, CD-ROM, DVD-ROM, USB memory, SD memory card, solid-state drive, other non-volatile flash memory-based storage media, or similar computer-readable media known in the art.

[0035] In summary, this invention generally relates to a novel concept of determining the extent to which a user's manipulation of a first object using a second object conforms to a predefined task related to the first object to be performed. This is consistent with the present invention, which determines the state of the first object when the user has manipulated it using a machine learning-based object processing scheme, and compares that state with data defined as processing steps to be performed to complete the predetermined task.

[0036] Further features and advantages of the invention will become apparent when examined in light of the appended claims and the following description. Those skilled in the art will recognize that different features of the invention can be combined to create embodiments other than those described below without departing from the scope of the invention. Attached Figure Description

[0037] Various aspects of the present invention, including its particular features and advantages, will be readily understood from the following detailed description and accompanying drawings, wherein: Figure 1A and 1B A computer system according to a currently preferred embodiment of the present invention is conceptually illustrated. Figure 2A and 2B The illustrations depict possible implementations of a machine learning-based object processing scheme used in conjunction with the content of this invention. Figures 3A-3C An exemplary description is provided for determining the relationship between a first object and a second object used to manipulate a first object, and... Figure 4 This is a flowchart illustrating the steps of performing a method according to a currently preferred embodiment of the present invention. Detailed Implementation

[0038] The present invention will now be described more fully with reference to the accompanying drawings, which show presently preferred embodiments of the invention. However, the invention can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided for thoroughness and completeness and to fully express the scope of the invention to those skilled in the art. Similar reference characters refer to similar elements throughout. The following examples illustrate the invention and are not intended to limit its scope.

[0039] Now turn to the attached diagram, especially Figure 1A and 1B Conceptually, it illustrates a computer system 100 suitable for determining the compliance level of a predefined task associated with a first object. Figure 1B In the diagram, the first object is represented as electrical connector 102. Computer system 100 includes at least a single server and a database 104, wherein server 104 includes at least a processing unit (…). Figure 1A and 1B (Not explicitly shown in the text).

[0040] The computer system 100 also includes an object capturing device 106. Figure 1A In this embodiment, an object capture device, such as a camera, is shown embedded in an augmented reality (AR) head-mounted device 106, as is known to those skilled in the art. The AR head-mounted device 106 preferably includes image and audio generation devices for providing information to a user / operator wearing the head-mounted device. It should be understood that in some embodiments, a processing unit may be embedded within the AR head-mounted device.

[0041] For reference, a processing unit may be, for example, a general-purpose processor, a graphics processing unit, a dedicated processor, a circuit containing processing components, a set of distributed processing components, a set of distributed computers configured for processing, a field-programmable gate array (FPGA), etc. A processor may be or include any number of hardware components for performing data, signal, and / or image processing or for executing computer code stored in memory. Implementation using a system-on-a-chip (SoC) is also possible and within its scope. Memory may be one or more devices for storing data and / or computer code for performing or facilitating the various methods described herein. Memory may include volatile memory or non-volatile memory. Memory may include database components, object code components, scripting components, or any other type of information structure for supporting the various activities described herein. According to exemplary embodiments, any distributed or local memory device may be used with the systems and methods of this specification. According to exemplary embodiments, memory may be communicatively connected to a processor (e.g., via circuitry or any other wired, wireless, or network connection) and includes computer code for performing one or more processes described herein.

[0042] As described above, in some embodiments, the present invention can utilize a computer system to assist a user / operator in performing tasks concerning a first object 102, wherein tasks concerning the first object are performed by manipulating the first object 102 using a second object. Figure 1B The image shows an operable tool 108 suitable for removing terminal pins or posts included in electrical connector 102. The quality of task performance conforms to the scope of the invention as defined by the compliance level determined by server 104.

[0043] Therefore, further reference Figure 4 During operation, the computer system 100 is adapted to perform multiple functions to determine the extent to which the user / operator performs the task, that is, the extent to which the user / operator complies with the task at hand.

[0044] The steps performed by computer system 100 include, S1, receiving data from a memory element arranged to communicate with processing unit 102 (e.g., a database) defining processing steps to be performed to complete a predetermined task. Server 104 is also adapted to acquire a first representation of the region to be performed on the predefined task, wherein the first representation also includes a first object 102. S3, server 104 further applies the machine learning-based object processing scheme discussed and exemplified above to the acquired first representation to identify the first object 102 and at least one second object 108 (e.g., a user / operator's hand) involved in performing the predetermined task. Once objects 102, 108, and possibly their interrelationships, are identified, S4, server 104 automatically identifies the state of the first object based on manipulation of the first object using the second object. Then, S5, server 104 can use the state of the first object and the data defining the processing steps to be performed to complete the predetermined task and determine the compliance level of the predetermined task.

[0045] This invention allows for the automatic determination of the execution status of predetermined tasks. The compliance level determined by this computer system 100 can be used in many different ways, for example, to determine quality indicators regarding the manipulated first object 102. According to the invention, the results of correlation can also be used to direct the user / operator "in the right direction," ensuring that predefined tasks are completed optimally.

[0046] The inherent functionality of the AR headset allows users to be provided with instructions on how to alter, for example, the manipulation of the electrical connector 102 by using an operable tool 108 suitable for removing terminal pins or posts included in the electrical connector 102.

[0047] Such instructions may be presented, for example, as an audio-based instruction manual for the operable tool 108, and may also include an overlay video sequence instructing the user / operator how (in a pre-recorded manner) to manipulate the connector 102 using the operable tool 108.

[0048] As described above, the selection of audio / video-based instructions for use of the operable tool 108 can be based on the user / operator's performance of the task and, according to the invention, is indicated by the determined level of compliance.

[0049] Turn now Figure 2A and 2BThe invention provides illustrative descriptions of possible implementations of a machine learning-based object processing scheme. Specifically, as described above, the machine learning-based object processing scheme according to the invention is used to identify a first object and at least a second object involved in performing a predetermined task, and may be used to determine the relationship between the first and second objects, and to determine the state of the first object based on manipulation of the first object using the second object. The following discussion will focus on how to combine the determination of their relationship to identify the first and second objects in a first representation.

[0050] Figure 2A A possible method for implementing a deep neural network 200 adapted to identify a first object and a second object is shown, as well as how these objects are arranged in relation to each other.

[0051] The architecture includes an input layer 202 configured to receive input data from the deep neural network. The input data includes a mathematical representation of the first representation and data relating to processing steps to be performed to complete a predetermined task. It may also include information relating to the area where the object exists (e.g., in the user / operator's work area).

[0052] The first representation, data relating to the processing steps to be performed to complete the predetermined task, and data relating to the area where the object exists, may be provided in the form of a data matrix or a graph. The image data preferably includes a time series of the collected (i.e., collected over time) first representation. In some embodiments, the image data includes historical data (possibly past seconds such as 1, 2, 3, 4, 5 seconds). The input layer includes a node 204 associated with each input.

[0053] In block 206, the deep neural network 200 may also include one or more convolutional layers. The deep neural network based on recursive layers takes the current data from input layer 202 as input, in addition to the previously processed data. In other words, the recursive layer is advantageously used to capture the history of the input data.

[0054] Node 204 of input layer 202 communicates with node 208 of layer 206 via connection 210. Connection 210 and the weights of the connection are determined during training, such as supervised or unsupervised training.

[0055] The identified first and second objects and their relationships are provided as output in output layer 212 in the form of mathematical representations. It should be noted that the number of connections and nodes may vary in each layer. Figure 2A Provided as an example only. Therefore, in some deep neural network designs, it is possible to use... Figure 2A The layer shown is more than one layer.

[0056] In an exemplary embodiment of the present invention, a machine learning-based object processing scheme applies a convolutional neural network to at least a portion of object recognition. In the convolutional neural network, as is known to those skilled in the art, convolutions in the input layers are used to compute the output. Local connections are formed such that each part of the input layer is connected to a node in the output. Filters are applied at each layer, thereby learning the parameters of the filters during the training phase of the neural network.

[0057] The deep neural network can be trained based on supervised learning of a general object that is expected to be included in the first representation and is expected to be used to manipulate the first object. Alternatively, the deep neural network can be trained based on unsupervised learning of a predetermined task to be performed and possible objects to be used when manipulating the first object.

[0058] exist Figures 3A-3C The document provides an exemplary description of determining the state of a first object, which may apply the principles outlined above. Figure 2A and 2B An extension of the image processing schemes discussed.

[0059] Based on the above discussion, Figures 3A-3C The illustration shows a user / operator manipulating an electrical connector 102 using an operable tool 108 suitable for removing terminal pins or posts included in the electrical connector 102. The user / operator holds the electrical connector 102 with their left hand and the operable tool 108 with their right. In the illustrated embodiment, the predetermined task to be performed by the user / operator is to remove the cable 306 and the connected pins 312 from the electrical connector 102, for example, by correctly inserting the operable tool 108 into a predetermined position on the electrical connector 102.

[0060] Figure 3A An example of a starting position that can be captured using a camera embedded in an AR headset 106 is shown. The image captured by the camera is then processed by a processing unit for automatic identification of first and second objects, shown here by an electrical connector 102 and an operable tool 108. As shown, individual objects within the image captured by the camera are each guided by global coordinates of the region where a predefined task is to be performed. However, in Figure 3A In the diagram, the relationship between the electrical connector 102 and the operable tool 108 is shown as misaligned, meaning that the current state of the electrical connector 102 does not conform to the task to be performed. Instead, the electrical connector 102 should be rotated, and the operable tool 108 should be aligned with the electrical connector 102.

[0061] To help users / operators better follow the tasks to be performed, image enhancements captured by the camera are generated, presenting reorientation instructions as shown by arrows 302 and 304. "Virtual axis extension" may also be presented in the enhancements provided to the user / operator.

[0062] Once the user has reoriented the electrical connector 102 and the operable tool 108 to align them, as follows: Figure 3B As shown, the initial predefined task can be considered as executed, which means that the compliance level determined by the processing unit can be defined as being above a predefined threshold, meaning that the operation performed by the user / operator is as expected. Once the electrical connector 102 is aligned with the operable tool 108, further enhancements can be presented to the user / operator (as shown by arrow 308), instructing the user / operator to insert the operable tool 108 into the electrical connector 102.

[0063] When the operable tool 108 is inserted into the electrical connector 102, the user / operator can remove the cable from the electrical connector 102, such as... Figure 3C As shown. Therefore, in this embodiment, the first object does not "grow" (as described above), but is shortened compared to the initial state of the first object.

[0064] In any case, once cable 306 has been removed from electrical connector 102, the overall task related to electrical connector 102 is considered complete, and the user / operator has met expectations. Therefore, the compliance level will be determined to be above a predetermined threshold. Further enhancements can then be formulated to instruct the user / operator regarding this determination.

[0065] As can be understood from the above, compliance should not necessarily be viewed as performing a complete task related to the first object. Instead, the task being performed can be considered as part of a larger task. Therefore, a larger task can include multiple (sub)tasks.

[0066] Finally, although the above discussion concerns the manipulation of the electrical connector 102 by a user / operator using an operable tool 108, the concepts according to the invention can be applied to many other types of tasks involving essentially any type of object, whether performed entirely or semi-manually by a user / operator, or automatically by, for example, a robot. For example, the concepts according to the invention can generally be used in manufacturing (e.g., automotive, avionics, etc.), construction, kindergartens, or in any situation where it is desirable to verify the level of compliance (performed by a user and / or robot) of a task, where the level of compliance can also be used to improve the manner in which the task is performed.

[0067] Furthermore, the control functions of the present invention can be implemented using existing computer processors, or by a dedicated computer processor incorporated into a suitable system for this or another purpose, or by a hardwired system. Embodiments within the scope of the invention include program products comprising machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such machine-readable media can be any available medium accessible to a general-purpose or special-purpose computer or other machine having a processor. For example, such machine-readable media may include RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, solid-state drives or other non-volatile flash memory storage devices, or any other medium that can be used to carry or store desired program code in the form of machine-executable instructions or data structures and is accessible to a general-purpose or special-purpose computer or other machine having a processor. When information is transmitted or provided to a machine via a network or another communication connection (hardwired, wireless, or a combination of hardwired and wireless), the machine correctly regards that connection as a machine-readable medium. Therefore, any such connection is properly referred to as a machine-readable medium. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processor to perform a particular function or group of functions.

[0068] Although the accompanying drawings may show a sequence, the order of the steps may differ from that described. Furthermore, two or more steps may be performed simultaneously or partially simultaneously. This variation will depend on the chosen software and hardware system and the designer's choices. All such variations are within the scope of this invention. Similarly, software implementation can be achieved using standard programming techniques with rule-based logic and other logic to implement various connection steps, processing steps, comparison steps, and decision steps. Moreover, although the invention has been described with reference to specific exemplary embodiments, many different changes, modifications, etc., will become apparent to those skilled in the art.

[0069] Furthermore, through a study of the accompanying drawings, the present invention, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments when practicing the claimed invention. Additionally, in the claims, the word "comprising" does not exclude other elements or steps, and singular nouns do not exclude plural forms.

Claims

1. A computer system adapted to determine a compliance level of a predefined task performed by a user in relation to a first object, the computer system comprising a processing unit and an object capturing device, wherein, The computer system is suitable for: Data defining the processing steps to be executed to complete the predefined task is received from a memory element arranged to communicate with the processing unit. The object capturing device is used to obtain a first representation of the region where a predefined task is to be performed. Using the processing unit, a machine learning-based object processing scheme is applied to the first representation to identify the first object and at least one second object involved in performing the predefined task, wherein the first object is independent of the user, and the second object is operated by the user. Using the processing unit and the machine learning-based object processing scheme, when a second object manipulates a first object, the state of the first object is automatically identified, wherein the second object is manipulated by the user, and the state identification of the first object is also based on the determined relationship between the first and second objects. Using the processing unit, the compliance level of the predefined task is determined based on the combination of the state of the first object and the data defined as the processing steps to be performed to complete the predefined task. The first object is an electrical connector, and the second object is an operable tool suitable for removing terminal pins or posts included in the electrical connector.

2. The computer system according to claim 1, characterized in that... The computer system is further adapted to: The processing unit compares the determined compliance level with a predetermined threshold, and If the determined compliance level is lower than a predetermined threshold, a feedback signal is generated using the processing unit.

3. The computer system according to claim 2, characterized in that... The feedback signal depends on the difference between the determined compliance level and a predetermined threshold.

4. The computer system according to claim 2, characterized in that... The computer system is configured to communicate with an output interface, and the output interface is adapted to transmit the feedback signal to the operator.

5. The computer system according to claim 4, characterized in that... It also includes an output interface, wherein the output interface includes at least one of an image or audio generation device.

6. The computer system according to claim 2, characterized in that... The computer system is also adapted to: The processing unit is used to form an image to be provided at the output interface, wherein the image is formed by enhancing the first representation with the feedback signal.

7. The computer system according to claim 6, characterized in that... The formation of the image is further based on the data that defines the processing steps to be performed to complete the predefined task.

8. The computer system according to claim 1, characterized in that... The first representation includes a set of representations.

9. The computer system according to claim 1, characterized in that... The machine learning-based object processing scheme is based on an unsupervised machine learning process.

10. The computer system according to claim 1, characterized in that... The machine learning-based object processing scheme is based on a supervised machine learning process.

11. The computer system according to claim 1, characterized in that... The machine learning-based object processing scheme is based on a combination of unsupervised and supervised machine learning processes.

12. The computer system according to claim 9, characterized in that... The processing unit is further adapted to: Access includes a digital storage unit containing a previously stored set of training objects that can be compared with the first object and the second object, and The set of training objects is compared with the objects included in the first representation.

13. The computer system according to claim 1, characterized in that... The object capture device includes at least one of an image capture device, a lidar, a radar, a laser scanner, or a thermal sensor.

14. The computer system according to claim 1, characterized in that... The first representation includes an image.

15. An electronic user device comprising a computer system according to any one of the preceding claims.

16. The electronic user device according to claim 15, characterized in that... The electronic user device is at least one of a virtual reality headset, an augmented reality headset, and a computer configuration.

17. A computer-implemented method for operating a computer system to determine the degree of compliance of a predefined task execution associated with a first object, the computer system including a processing unit and an object capturing device, wherein the method includes the following steps: Data defining the processing steps to be executed to complete the predefined task is received from a memory element arranged to communicate with the processing unit. The object capturing device is used to obtain a first representation of the region where the predefined task is to be performed. The processing unit applies a machine learning-based object processing scheme to the obtained first representation to identify the first object and at least one second object involved in performing the predefined task, wherein the first object is independent of the user and the second object is operated by the user. When a second object operates on a first object, the processing unit and the machine learning-based object processing scheme automatically identify the state of the first object. The second object is operated by the user, and the state identification of the first object is also based on the determined relationship between the first and second objects. The processing unit uses a combination of data based on the state of the first object and the processing steps to be performed to complete the predefined task to determine the compliance level of the predefined task; The first object is an electrical connector, and the second object is an operable tool suitable for removing terminal pins or posts included in the electrical connector.

18. The method according to claim 17, characterized in that... Further steps include: The processing unit is used to compare the determined compliance level with a predefined threshold, and If the determined compliance level is lower than a predetermined threshold, the processing unit is used to generate a feedback signal.

19. The method according to claim 18, characterized in that... Further steps include: The processing unit is used to form an image to be provided at the output interface, wherein the image is formed by enhancing the first representation with the feedback signal.

20. The method according to claim 19, characterized in that... The steps to form an image are further based on data defined as processing steps to be performed to complete a predetermined task.

Images