Personnel operation behavior evaluation method and device based on human factor intelligence and electronic equipment

Abstract

The embodiment of the invention provides a personnel operation behavior evaluation method and device based on human intelligence and electronic equipment, and the method comprises the steps: in a process that an operator operates a target object, the electronic equipment obtains the operation behavior data of the operator on the target object, and the eye movement data and physiological data of the operator; and determining a first operation path of the operator on the target object according to the operation behavior data, determining a watching condition of the operator on the interest area according to the eye movement data, and determining a perceived load level of the operator in the process of operating the target object according to the physiological data. And finally, the electronic equipment determines that the operation of the operating personnel on the target object meets the predetermined requirement according to the first operation path, the watching condition of the operating personnel on the interest area and the sensing load level, so that a highly real assembly environment can be established by utilizing a VR technology, the operating personnel can skillfully operate a system in advance in a virtual environment, and the operation efficiency is improved. The reality sense and interactivity of assembly training are improved, and then the efficiency and accuracy of actual assembly work are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the fields of electronic digital data processing, bioinformatics technology, human factors intelligence, and human factors and ergonomics technology, and in particular to a method, device, and electronic device for evaluating personnel operation behavior based on human factors intelligence. Background Art

[0002] The manufacturing industry usually involves a lot of assembly work. For example, the manufacturing process of an aircraft involves a lot of assembly work, including the assembly of the fuselage structure, the installation of the engine and / or the connection of the avionics system. For example, in the manufacturing process of an aircraft, components such as the wings, fuselage and / or tail need to be assembled accurately to ensure the structural strength and aerodynamic performance of the aircraft. In addition, the manufacture of spacecraft also involves a lot of assembly work, and the manufacturing and assembly work of spacecraft has higher requirements for precision and reliability. For example, the assembly of satellites and the assembly of rocket engines have high requirements for precision and reliability. This requires assembly operators to work efficiently according to specific processes.

[0003] Traditional assembly training methods usually rely on physical models or paper manuals, which lack realism and interactivity, making it difficult for assembly operators to quickly become familiar with complex assembly processes. Summary of the invention

[0004] The embodiments of the present application provide a method, device and electronic device for evaluating personnel operation behavior based on human factors intelligence. The embodiments of the present application also provide a computer-readable storage medium to realize the application of virtual reality (VR) technology to assembly training and improve the realism and interactivity of assembly training.

[0005] In a first aspect, an embodiment of the present application provides a method for evaluating personnel operation behavior based on human factors intelligence, including: in the process of the operator operating the target object, obtaining the operator's operation behavior data on the target object, the operator's eye movement data and physiological data; wherein the operation behavior data includes the operator's operation trajectory, operation sequence and the number of interactions with the target object; determining a first operation path of the operator on the target object according to the operation behavior data, determining the operator's gaze on the area of ​​interest according to the eye movement data, and determining the operator's perceived load level in the process of operating the target object according to the physiological data; determining whether the operator's operation of the target object meets predetermined requirements based on the first operation path, the operator's gaze on the area of ​​interest and the perceived load level.

[0006] In one possible implementation, before determining, based on the first operation path, the operator's gaze on the area of ​​interest and the perceived load level, whether the operator's operation on the target object meets predetermined requirements, the method further includes: calculating a first similarity between the first operation path and the second operation path; and based on the operator's gaze on the area of ​​interest, obtaining the operator's first gaze area set and the operator's gaze duration on the area of ​​interest, and calculating a second similarity between the first gaze area set and the second gaze area set; wherein the second operation path includes a predetermined optimal operation path, and the second gaze area set includes a predetermined optimal gaze area set.

[0007] In one possible implementation, determining whether the operator's operation on the target object meets predetermined requirements based on the first operation path, the operator's gaze on the area of ​​interest and the perceived load level includes: when the first similarity is greater than or equal to a first similarity threshold, the second similarity is greater than or equal to a second similarity threshold, the gaze duration is within a predetermined gaze duration range, and the perceived load level is less than or equal to a perceived load level threshold, determining that the operator's operation on the target object meets predetermined requirements.

[0008] In one possible implementation, the calculating the first similarity between the first operation path and the second operation path includes: obtaining a set of overlapping steps between the first operation path and the second operation path, the number of steps included in the first operation path, and the number of steps included in the second operation path; calculating the first similarity between the first operation path and the second operation path according to the number of steps included in the first operation path, the number of steps included in the second operation path, the number of elements in the overlapping step set, and the number of predetermined steps; wherein the predetermined steps include steps that are in both the first operation path and the overlapping step set, and whose sequential position in the overlapping step set is the same as that in the second operation path.

[0009] In one possible implementation, calculating the second similarity between the first gaze area set and the second gaze area set includes: obtaining the intersection and union of the first gaze area set and the second gaze area set; and calculating the quotient of the number of elements in the intersection and the number of elements in the union as the second similarity.

[0010] In one possible implementation, the perceived load level being less than or equal to a perceived load level threshold includes: a difference value of physiological data being less than or equal to a predetermined physiological data threshold; wherein the difference value of physiological data includes a difference value between physiological data collected by the operator during the operation of the target object and physiological data collected by the operator when not operating the target object.

[0011] In one possible implementation, the physiological data of the operator includes one or a combination of the following: skin electrodermal temperature data, pulse data, blood pressure data, blood oxygen data, electrocardiogram data, electromyography data, muscle oxygen data, respiratory data, biomechanical data, near-infrared brain imaging data, electroencephalogram data, transcranial stimulation data, heart rate variability data, heart rate data, image / video data, sound data, gesture or motion data.

[0012] In the above-mentioned personnel operation behavior evaluation method based on human factor intelligence, during the process of the operator operating the target object, the electronic device obtains the operator's operation behavior data on the target object, the operator's eye movement data and physiological data, determines the operator's first operation path on the target object according to the above-mentioned operation behavior data, determines the operator's gaze on the interest area according to the above-mentioned eye movement data, and determines the operator's perceptual load level in the process of operating the target object according to the above-mentioned physiological data. Finally, the electronic device determines that the operator's operation of the target object meets the predetermined requirements based on the first operation path, the operator's gaze on the interest area and the above-mentioned perceptual load level, so that the use of VR technology can be used to build a highly realistic assembly environment, so that the operator can pre-proficiently operate the system in the virtual environment, improve the realism and interactivity of assembly training, and thus improve the efficiency and accuracy of actual assembly work. In addition, through the collection of multimodal data, according to the operator's operation behavior, eye movement data, EEG data and / or skin conduction data, etc., the operator's operation path, attention to the interest area and perceptual load level are evaluated, which can provide quantitative indicators for the training effect and improve the training efficiency and quality.

[0013] In a second aspect, an embodiment of the present application further provides an electronic device, comprising: one or more processors; a memory; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to perform the method provided in the first aspect.

[0014] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the first aspect.

[0015] In a fourth aspect, an embodiment of the present application provides a computer program, which, when executed by a computer, is used to execute the method provided in the first aspect.

[0016] In one possible design, the program in the fourth aspect may be stored in whole or in part on a storage medium packaged together with the processor, or may be stored in whole or in part on a memory not packaged together with the processor.

[0017] In a fifth aspect, an embodiment of the present application provides an operation behavior evaluation device based on human factors intelligence. The above-mentioned device can be set in an electronic device, and the above-mentioned device can include: an acquisition module, used to obtain the operation behavior data of the operator on the target object, the eye movement data and the physiological data of the operator during the process of the operator operating the target object; wherein the operation behavior data includes the operation trajectory, operation sequence and number of interactions with the target object of the operator; a determination module, used to determine a first operation path of the operator on the target object according to the operation behavior data, determine the operator's gaze on the area of ​​interest according to the eye movement data, and determine the operator's perceived load level in the process of operating the target object according to the physiological data; according to the first operation path, the operator's gaze on the area of ​​interest and the perceived load level, determine that the operator's operation of the target object meets predetermined requirements.

[0018] In one possible implementation, the device may further include: a calculation module, configured to calculate a first similarity between the first operation path and the second operation path before the determination module determines that the operation of the operator on the target object meets the predetermined requirements based on the first operation path, the operator's gaze on the area of ​​interest and the perceived load level; and, based on the operator's gaze on the area of ​​interest, obtain the operator's first gaze area set and the operator's gaze duration on the area of ​​interest, and calculate a second similarity between the first gaze area set and the second gaze area set; wherein the second operation path includes a predetermined optimal operation path, and the second gaze area set includes a predetermined optimal gaze area set. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a connected client provided for one embodiment of the present application;

[0021] Figure 3 A schematic diagram of a connectable server provided for one embodiment of the present application;

[0022] Figure 4 A flowchart of a method for evaluating personnel operation behavior based on human intelligence provided in one embodiment of the present application;

[0023] Figure 5 A flowchart of an implementation method of a human factor intelligence-based personnel operation behavior assessment method provided in one embodiment of the present application;

[0024] Figure 6 A schematic structural diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0025] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0026] The existing assembly training methods lack realism and interactivity, making it difficult for assembly operators to quickly become familiar with complex assembly processes. In addition, the existing assembly training methods cannot accurately assess whether the operator's operation path is optimal, whether the operator's attention to the area of ​​interest (AOI) is in place, and the level of perceived load, making it difficult to quantify and optimize the training effect; in addition, the existing assembly training process cannot be adaptively adjusted according to the actual situation of the operator, which may lead to low training efficiency or over-training.

[0027] Based on the above problems, the embodiment of the present application provides a method for evaluating personnel operation behavior based on human factors intelligence, which uses VR technology to build a highly realistic assembly environment, so that operators can pre-proficiently operate the system in the virtual environment, improve the realism and interactivity of assembly training, and thus improve the efficiency and accuracy of actual assembly work.

[0028] In addition, through the collection of multimodal data, the operator's operating path, attention to the area of ​​interest, and perceived load level are evaluated based on the operator's operating behavior, eye movement data, EEG data, and / or electrodermal data, providing quantitative indicators for the training effect.

[0029] Furthermore, the method provided in the embodiment of the present application can perform adaptive training adjustments based on the collected physiological data, dynamically adjust the training difficulty and content according to the actual situation of the operator, improve the training efficiency and quality, and ensure that the operator reaches a proficient level in the shortest time.

[0030] The human factor intelligence-based personnel operation behavior assessment method provided in the embodiment of the present application can be applied to electronic devices, wherein the above-mentioned electronic devices can be personal computers (personal computers, PCs), smart phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (augmented reality, AR) / virtual reality technology (virtualreality, VR) devices, laptop computers or ultra-mobile personal computers (ultra-mobile personal computers, UMPCs) and other devices; the embodiment of the present application does not impose any restrictions on the specific types of electronic devices.

[0031] For example, Figure 1 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown as follows: Figure 1 As shown, the electronic device 100 may include one or more processors 110; a memory 120; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory 120, and the one or more computer programs include instructions. When the above instructions are executed by the electronic device 100, the electronic device 100 can execute the human intelligence-based personnel operation behavior assessment method provided in an embodiment of the present application.

[0032] In addition, the electronic device 100 may further include a communication interface 120. The processor 110, the communication interface 120 and the memory 120 may communicate with each other through an internal connection path to transmit control and / or data signals.

[0033] The processor 110 and the memory 120 may be combined into a processing device, or more commonly, they are independent components, and the processor 110 is used to execute program codes stored in the memory 120. In specific implementation, the memory 120 may also be integrated into the processor 110, or may be independent of the processor 110.

[0034] In addition, in order to make the functions of the electronic device 100 more complete, the electronic device 100 may further include one or more of an input unit 160 and a display unit 170 .

[0035] Optionally, the electronic device 100 may further include a power supply 150 for providing power to various devices or circuits in the electronic device 100 .

[0036] It should be understood that Figure 1The processor 110 in the electronic device 100 shown may be a system on chip SOC, and the processor 110 may include a central processing unit (CPU), and may further include other types of processors, such as a graphics processing unit (GPU) and the like.

[0037] The execution of the human factor intelligence-based personnel operation behavior evaluation method provided in the embodiment of the present application relies on the VR scene, which can highly restore the actual assembly operation environment, making the operator feel as if they are in a real work scene. Therefore, in the embodiment of the present application, the corresponding VR scene can be generated in combination with the specific assembly task, making the training more targeted and practical.

[0038] The interactive settings in the VR scene allow operators to interact naturally with objects in the VR scene. For example, operators can grab, rotate and / or install objects in the VR scene, which improves the realism and participation of state training. In addition, multi-person collaborative work can be set up in the VR scene to evaluate the efficiency of multi-person collaboration.

[0039] After the corresponding software of the VR scene has been installed in the electronic device 100, the operator can run the VR scene through the following operations.

[0040] 1. Run the exe program corresponding to the VR scene, and then select the communication method, such as: single player, multiplayer, client or server.

[0041] 2. When the operator selects the server as the communication method, the operator can first click the server button (server status: off -> on -> waiting for client connection), and then the server will display the IP address of the connected client, such as Figure 2 As shown, double-click the IP address to connect to the client. After the connection is successful, the operator clicks the start button to enter the VR scene. Figure 2 A schematic diagram of a connected client provided for one embodiment of the present application.

[0042] When the operator selects the client communication mode, and the server is turned on, click the refresh button, and the list will display the IP address of the connected server, such as Figure 3 As shown, double-click the IP address to connect to the server. After the connection is successful, the connection status will be: disconnected -> connected, waiting for the server to start. Figure 3 A schematic diagram of a connectable server provided for one embodiment of the present application.

[0043] 3. Left-hand handle function: Touch the touchpad to display the position. When the position is movable, pressing the touchpad will instantly move to the displayed position.

[0044] 4. Right handle function: Touch the Touchpad to display the ray, press the Touchpad to select the operator interface (UI) function.

[0045] 5. Two-handle trigger (Trigger): During the disassembly and assembly process, there will be a prompt when the handle touches the corresponding engine model. Pressing the trigger at the same time will pick up the model, and releasing the trigger will put down the model.

[0046] 6. Communication protocol: enumeration SocketMessageType

[0047] OpenServer=0, open the server

[0048] CloseServer=1, close the server

[0049] Connected = 2, connected

[0050] Unconnected=3, disconnected

[0051] NewMember,New Member

[0052] MemberUnconnected, member disconnected

[0053] MemberTransform, member position

[0054] StartTask, start the task (display interface, model, hide communication interface)

[0055] ModelInteraction

[0056] UIInteraction

[0057] The following is an introduction to the personnel operation behavior evaluation method based on human intelligence.

[0058] Figure 4 This is a flowchart of a method for evaluating human operation behavior based on human intelligence provided in one embodiment of the present application. The method for evaluating human operation behavior based on human intelligence provided in this embodiment is executed by the electronic device 100.

[0059] like Figure 4 As shown, the above-mentioned personnel operation behavior evaluation method based on human factor intelligence may include:

[0060] Step 401 , in the process of an operator operating a target object, obtaining the operator's operating behavior data on the target object, the operator's eye movement data and physiological data.

[0061] The above operation behavior data may include the operator's operation trajectory, operation sequence and number of interactions with the target object. In addition, the above operation behavior data may also include: the first interaction time between the operator and the target object, the operator's holding time of the target object and the sequence of the interaction object.

[0062] In this embodiment, the target object is an object in the VR scene, and the operator operates the target object in the VR scene. For example, the operator's operation on the target object may be an assembly operation of the target object. Specifically, the operator's operation on the target object in the VR scene may be completed through a handle. In this embodiment, the operator's operation on the target object may include: grabbing the object, clicking a button, and / or driving the object.

[0063] In specific implementation, the operator's operation behavior data on the target object, the operator's eye movement data and physiological data can be collected by a data collector. The above data collector can include one or a combination of the following: a human body-related data collector, a machine-related data collector, a human-computer interaction-related data collector, and an environment-related data collector; specifically, the above data collector can be a variety of physiological signal acquisition sensors such as EEG, skin electricity and / or heart rate, or an eye tracker, image or video sensor, machine speed sensor, steering wheel pedal mechanical sensor, temperature sensor or humidity sensor, etc. In addition, the above data collector can be a wearable, non-wearable, contact, non-contact sensor or other type of data collector, and this embodiment does not limit the type of the above data collector.

[0064] In this embodiment, the physiological data of the above-mentioned operators may include one or a combination of the following: skin electrodermal temperature data, pulse data, blood pressure data, blood oxygen data, electrocardiogram data, electromyography data, muscle oxygen data, respiratory data, biomechanical data, near-infrared brain imaging data, electroencephalogram data, transcranial stimulation data, heart rate variability data, heart rate data, image / video data, sound data, gesture or motion data.

[0065] Step 402, determining the operator's first operation path on the target object based on the above operation behavior data, determining the operator's gaze on the area of ​​interest based on the above eye movement data, and determining the operator's perceptual load level during the operation of the target object based on the above physiological data.

[0066] The above-mentioned area of ​​interest may be a target object, or a part of the target object, for example, a certain component of the target object.

[0067] Step 403: Determine, based on the first operation path, the operator's gaze on the area of ​​interest, and the above-mentioned perception load level, whether the operator's operation on the target object meets the predetermined requirements.

[0068] In some examples, before step 403, the electronic device 100 may further calculate a first similarity between the first operation path and the second operation path; and according to the operator's gaze on the interest area, obtain a first gaze area set of the operator and the operator's gaze duration on the interest area, and calculate a second similarity between the first gaze area set and the second gaze area set;

[0069] The second operation path includes a predetermined optimal operation path, and the second gaze area set includes a predetermined optimal gaze area set. Specifically, the predetermined optimal operation path may be an operation path confirmed by an assembly expert in the field, and the predetermined optimal gaze area set may be a gaze area set confirmed by an assembly expert in the field.

[0070] Thus, step 403 may be: when the first similarity is greater than or equal to the first similarity threshold, the second similarity is greater than or equal to the second similarity threshold, the above-mentioned gaze duration is within the predetermined gaze duration range, and the above-mentioned perception load level is less than or equal to the perception load level threshold, it is determined that the operator's operation of the above-mentioned target object meets the predetermined requirements. Among them, the above-mentioned first similarity threshold, the second similarity threshold, the predetermined gaze duration range and the perception load level threshold can be set by oneself during specific implementation, and this embodiment does not limit the size of the first similarity threshold, the second similarity threshold, the predetermined gaze duration range and the perception load level threshold.

[0071] In other words, to determine whether the operator's operation of the target object meets the predetermined requirements, it is necessary to comprehensively examine the operator's performance in three dimensions: operation path, gaze at the area of ​​interest, and perceived load level. Only when these three dimensions reach the preset thresholds can it be determined that the operator has mastered the assembly operation of the target object and the training can be ended. In addition, the above-mentioned first similarity threshold, second similarity threshold, predetermined gaze duration range, and perceived load level threshold are all determined by assembly experts in this field in combination with specific operation data, and are scientific and reliable.

[0072] In some examples, calculating the first similarity between the first operation path and the second operation path may include: obtaining a set of overlapping steps between the first operation path and the second operation path, the number of steps included in the first operation path, and the number of steps included in the second operation path; then, according to the number of steps included in the first operation path, the number of steps included in the second operation path, the number of elements in the above-mentioned overlapping step set, and the number of predetermined steps, calculating the first similarity between the first operation path and the second operation path. The above-mentioned predetermined steps may include steps that are both in the first operation path and in the overlapping step set, and whose sequential position in the above-mentioned overlapping step set is the same as that in the second operation path.

[0073] Specifically, the overlapping step set can be recorded as C, the second operation path can be recorded as A, the first operation path can be recorded as B, and the steps included in the second operation path can be recorded as a. i , the step in the first operation path is b i , where i is a positive integer, i>0. In this way, each step in the first operation path B can be traversed. i =a i When b i Add C to obtain a set C of overlapping steps.

[0074] Then, the first similarity between the first operation path and the second operation path may be calculated according to formula (1).

[0075]

[0076] In formula (1), sim(A,B) is the first similarity; n is the number of steps included in the second operation path A; m is the number of steps included in the first operation path B; |C| is the number of elements in the overlapping step set C, that is, the number of steps included in C; k is the number of predetermined steps. The value range of sim(A,B) is [0,1], where 0 means completely dissimilar and 1 means completely identical.

[0077] In some other examples, calculating the second similarity between the first gaze area set and the second gaze area set may be: obtaining the intersection and union of the first gaze area set and the second gaze area set, and then calculating the quotient of the number of elements in the intersection and the number of elements in the union as the second similarity.

[0078] Specifically, the first gaze area set includes the operator's gaze sequence on the area of ​​interest, and the second gaze area set includes the optimal gaze sequence of the area of ​​interest. That is, the predetermined optimal gaze area set includes the optimal gaze sequence of the area of ​​interest.

[0079] In this example, the second similarity can be calculated according to formula (2).

[0080]

[0081] In formula (2), J(A, B) is the second similarity, A is the second fixation area set, B is the first fixation area set, |A∩B| is the number of elements in the intersection of A and B, and |A∪B| is the number of elements in the union of A and B. The value range of J(A, B) is also [0, 1], where 0 means completely dissimilar and 1 means completely identical.

[0082] According to the second similarity, it can be determined whether the operator is looking at the interest area and whether the sequence of looking at the interest area meets the requirements. When the second similarity is greater than or equal to the second similarity threshold, it means that the operator is looking at the interest area and the sequence of looking at the interest area meets the requirements.

[0083] In addition, for the gaze time of the area of ​​interest, if the operator's gaze time on the area of ​​interest is less than or greater than the gaze time range set by the assembly expert in the system, feedback prompts can be given at the end of the training.

[0084] In some other examples, the above-mentioned perceived load level is less than or equal to the perceived load level threshold value, including: the difference value of the physiological data is less than or equal to the predetermined physiological data threshold value; wherein the difference value of the physiological data includes the difference value between the physiological data collected by the operator during the operation of the target object and the physiological data collected by the operator when the operator does not operate the target object. The size of the above-mentioned predetermined physiological data threshold value can be set according to system performance and / or implementation requirements during specific implementation, and this embodiment does not limit the size of the above-mentioned predetermined physiological data threshold value.

[0085] That is to say, in this example, when the difference between the physiological data collected by the operator during the operation of the target object and the physiological data collected when the operator does not operate the above-mentioned target object is less than or equal to a predetermined physiological data threshold, it can be determined that the operator's perceived load level is normal; and when the difference between the physiological data collected by the operator during the operation of the target object and the physiological data collected when the operator does not operate the above-mentioned target object is greater than the predetermined physiological data threshold, it can be determined that the operator's perceived load level is greater than the perceived load level threshold, that is, the operator's perceived load level is too high. At this time, the electronic device 100 can reduce the training difficulty or prompt the operator to take a rest to avoid excessive fatigue of the operator.

[0086] In the above-mentioned personnel operation behavior evaluation method based on human factor intelligence, during the process of the operator operating the target object, the electronic device 100 obtains the operator's operation behavior data on the target object, the operator's eye movement data and physiological data, determines the operator's first operation path on the target object according to the above-mentioned operation behavior data, determines the operator's gaze on the interest area according to the above-mentioned eye movement data, and determines the operator's perceptual load level in the process of operating the target object according to the above-mentioned physiological data. Finally, the electronic device 100 determines that the operator's operation of the target object meets the predetermined requirements based on the first operation path, the operator's gaze on the interest area and the above-mentioned perceptual load level, so that the use of VR technology can be used to build a highly realistic assembly environment, so that the operator can pre-proficiently operate the system in the virtual environment, improve the realism and interactivity of assembly training, and thus improve the efficiency and accuracy of actual assembly work. In addition, through the collection of multimodal data, according to the operator's operation behavior, eye movement data, EEG data and / or skin conduction data, etc., the operator's operation path, attention to the interest area and perceptual load level are evaluated, which can provide quantitative indicators for the training effect and improve the training efficiency and quality.

[0087] As described above, the implementation process of the method for evaluating personnel operation behavior based on human intelligence provided in the embodiment of the present application can be as follows: Figure 5 As shown, Figure 5 A flowchart for implementing a method for evaluating personnel operation behavior based on human intelligence provided in one embodiment of the present application.

[0088] like Figure 5 As shown, it may include:

[0089] 501, build VR scenes and interactive settings.

[0090] 502, when the operator operates the target object, multimodal data is obtained, wherein the multimodal data includes the operator's operation behavior data on the target object, the operator's eye movement data and physiological data.

[0091] 503, after determining that the operator's operation of the target object meets the predetermined requirements based on the first operation path, the operator's gaze on the interest area, and the sensory load level, the training ends.

[0092] It is to be understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in different orders presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

[0093] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of this application.

[0094] This embodiment can divide the electronic device into functional modules according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0095] Figure 6 A schematic diagram of the structure of an electronic device provided in another embodiment of the present application, in which each functional module is divided according to each function, Figure 6 A possible schematic diagram of the composition of the electronic device 600 involved in the above embodiment is shown. Figure 6 As shown, the electronic device 600 may include: an acquisition module 601 and a determination module 602;

[0096] Among them, the acquisition module 601 is used to obtain the operator's operation behavior data on the target object, the operator's eye movement data and physiological data during the operator's operation of the target object; wherein the above-mentioned operation behavior data includes the operator's operation trajectory, operation sequence and the number of interactions with the target object.

[0097] Determination module 602 is used to determine the first operation path of the operator on the target object based on the above-mentioned operation behavior data, determine the operator's gaze on the area of ​​interest based on the above-mentioned eye movement data, and determine the operator's perceptual load level in the process of operating the target object based on the above-mentioned physiological data; based on the first operation path, the operator's gaze on the area of ​​interest and the above-mentioned perceptual load level, determine whether the operator's operation of the target object meets the predetermined requirements.

[0098] In some examples, the electronic device 600 may further include: a computing module 603;

[0099] The calculation module 603 is used to calculate the first similarity between the first operation path and the second operation path before the determination module 602 determines that the operation of the operator on the target object meets the predetermined requirements based on the first operation path, the operator's gaze on the area of ​​interest and the above-mentioned perception load level; and according to the operator's gaze on the area of ​​interest, obtain the operator's first gaze area set and the operator's gaze duration on the above-mentioned area of ​​interest, and calculate the second similarity between the first gaze area set and the second gaze area set; wherein the second operation path includes a predetermined optimal operation path, and the second gaze area set includes a predetermined optimal gaze area set.

[0100] In this embodiment, the determination module 602 is specifically used to determine that the operator's operation of the target object meets the predetermined requirements when the first similarity is greater than or equal to the first similarity threshold, the second similarity is greater than or equal to the second similarity threshold, the above-mentioned gaze time is within the predetermined gaze time range, and the above-mentioned perception load level is less than or equal to the perception load level threshold.

[0101] The calculation module 603 is specifically used to obtain the set of overlapping steps of the first operation path and the second operation path, the number of steps included in the first operation path and the number of steps included in the second operation path; and calculate the first similarity between the first operation path and the second operation path according to the number of steps included in the first operation path, the number of steps included in the second operation path, the number of elements in the above-mentioned overlapping step set, and the number of predetermined steps. The above-mentioned predetermined steps may include steps that are both in the first operation path and in the above-mentioned overlapping step set, and whose sequential position in the above-mentioned overlapping step set is the same as that in the second operation path.

[0102] The calculation module 603 is specifically configured to obtain the intersection and union of the first gaze area set and the second gaze area set; and calculate the quotient of the number of elements in the intersection and the number of elements in the union as the second similarity.

[0103] In this embodiment, the above-mentioned perceived load level less than or equal to the perceived load level threshold can be: the difference value of the physiological data is less than or equal to the predetermined physiological data threshold; wherein the difference value of the above-mentioned physiological data includes the difference value between the physiological data collected by the operator during the operation of the above-mentioned target object and the physiological data collected by the operator when not operating the above-mentioned target object.

[0104] This application Figure 4 All relevant contents of each step involved in the illustrated method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0105] The electronic device 600 provided in this embodiment is used to execute the present application Figure 4 or Figure 5 The human factor intelligence-based personnel operation behavior assessment method provided by the illustrated embodiment can therefore achieve the same effect as the above-mentioned method.

[0106] It should be understood that the electronic device 600 may correspond to Figure 1 The electronic device 100 shown in FIG. 6 is a block diagram of an electronic device 100. The function of the acquisition module 601 can be Figure 1 The processor 110 and the communication interface 120 in the electronic device 100 are implemented; the functions of the determination module 602 and the calculation module 603 can be implemented by Figure 1 The processor 110 in the electronic device 100 is shown implemented.

[0107] In the case of adopting an integrated unit, the electronic device 600 may include a processing module, a storage module, and a communication module.

[0108] The processing module can be used to control and manage the actions of the electronic device 600, for example, it can be used to support the electronic device 600 to execute the steps executed by the acquisition module 601, the determination module 602 and the calculation module 603. The storage module can be used to support the electronic device 600 to store program codes and data, etc. The communication module can be used to support the communication between the electronic device 600 and other devices.

[0109] Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0110] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device 600 involved in this embodiment can be a Figure 1 Device of the structure shown.

[0111] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the present application. Figure 4 or Figure 5 The method provided by the illustrated embodiment.

[0112] The present application also provides a computer program product, which includes a computer program, which, when executed on a computer, enables the computer to execute the present application. Figure 4 or Figure 5 The method provided by the illustrated embodiment.

[0113] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0114] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0115] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0116] In several embodiments provided in the present application, any function can be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as an independent product. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0117] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for evaluating personnel operation behavior based on human intelligence, characterized in that: include: In the process of the operator operating the target object, obtaining the operator's operation behavior data on the target object, the operator's eye movement data and physiological data; wherein the operation behavior data includes the operator's operation trajectory, operation sequence and number of interactions with the target object; determining a first operation path of the operator on the target object according to the operation behavior data, determining a gaze of the operator on an area of ​​interest according to the eye movement data, and determining a perceptual load level of the operator in the process of operating the target object according to the physiological data; It is determined that the operation of the target object by the operator meets predetermined requirements according to the first operation path, the operator's gaze on the area of ​​interest, and the sensory load level.

2. The method according to claim 1, characterized in that: Before determining, based on the first operation path, the operator's gaze on the area of ​​interest, and the sensory load level, that the operator's operation on the target object meets a predetermined requirement, the method further includes: Calculating a first similarity between the first operation path and the second operation path; and obtaining a first gaze area set of the operator and a gaze duration of the operator on the interest area according to the operator's gaze on the interest area, and calculating a second similarity between the first gaze area set and the second gaze area set; The second operation path includes a predetermined optimal operation path, and the second gaze area set includes a predetermined optimal gaze area set.

3. The method according to claim 2, characterized in that Determining, based on the first operation path, the operator's gaze on the area of ​​interest, and the sensory load level, that the operator's operation on the target object meets a predetermined requirement includes: When the first similarity is greater than or equal to a first similarity threshold, the second similarity is greater than or equal to a second similarity threshold, the gaze duration is within a predetermined gaze duration range, and the perceived load level is less than or equal to a perceived load level threshold, it is determined that the operator's operation of the target object meets predetermined requirements.

4. The method according to claim 2, characterized in that: The calculating the first similarity between the first operation path and the second operation path comprises: Acquire a set of overlapping steps of the first operation path and the second operation path, the number of steps included in the first operation path, and the number of steps included in the second operation path; A first similarity between the first operation path and the second operation path is calculated based on the number of steps included in the first operation path, the number of steps included in the second operation path, the number of elements in the overlapping step set, and the number of predetermined steps; wherein the predetermined steps include steps that are both in the first operation path and in the overlapping step set, and whose sequential position in the overlapping step set is the same as that in the second operation path.

5. The method according to claim 2, characterized in that: The calculating the second similarity between the first gaze area set and the second gaze area set comprises: Obtaining the intersection and union of the first gaze area set and the second gaze area set; The quotient of the number of elements in the intersection and the number of elements in the union is calculated as the second similarity.

6. The method according to claim 3, characterized in that The perceived load level being less than or equal to the perceived load level threshold comprises: a difference value of physiological data being less than or equal to a predetermined physiological data threshold; The difference value of the physiological data includes the difference value between the physiological data collected by the operator during the operation of the target object and the physiological data collected by the operator when the operator does not operate the target object.

7. The method according to any one of claims 1 to 6, characterized in that: The physiological data of the operator includes one or a combination of the following: skin electrodermal temperature data, pulse data, blood pressure data, blood oxygen data, electrocardiogram data, electromyography data, muscle oxygen data, respiratory data, biomechanical data, near-infrared brain imaging data, electroencephalogram data, transcranial stimulation data, heart rate variability data, heart rate data, image / video data, sound data, gesture or motion data.

8. An electronic device, characterized in that: include: one or more processors; Memory; Multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 7.

10. An operation behavior evaluation device based on human intelligence, characterized in that: include: An acquisition module, used to acquire the operator's operation behavior data on the target object, the operator's eye movement data and physiological data during the operator's operation of the target object; wherein the operation behavior data includes the operator's operation trajectory, operation sequence and number of interactions with the target object; A determination module is used to determine a first operation path of the operator on the target object based on the operation behavior data, determine the operator's gaze on the area of ​​interest based on the eye movement data, and determine the operator's perceptual load level in the process of operating the target object based on the physiological data; based on the first operation path, the operator's gaze on the area of ​​interest and the perceptual load level, determine whether the operator's operation of the target object meets predetermined requirements.

11. The device according to claim 10, characterized in that Also includes: a calculation module, configured to calculate a first similarity between the first operation path and the second operation path before the determination module determines that the operation of the operator on the target object meets the predetermined requirement according to the first operation path, the operator's gaze on the area of ​​interest, and the perception load level; and to obtain a first gaze area set of the operator and a gaze duration of the operator on the area of ​​interest according to the operator's gaze on the area of ​​interest, and calculate a second similarity between the first gaze area set and the second gaze area set; The second operation path includes a predetermined optimal operation path, and the second gaze area set includes a predetermined optimal gaze area set.

Images