Human-machine collaborative state feedback method, device and electronic equipment based on virtual-reality fusion

Abstract

The present invention discloses a method, device and electronic device for human-machine collaborative state feedback based on virtual-real fusion. First, the human factor data of the operating object in the human-machine collaborative process of the current task is obtained, as well as an operating screen image obtained by shooting the human-machine collaborative process, which at least includes the setting parameters of the operating device and the operating environment. Then, human-machine collaborative state recognition is performed based on the operating screen image to obtain the target collaborative state data corresponding to the current task, and personnel state recognition is performed based on the human factor data to obtain the personnel state data of the operating object, providing important information for the human-machine collaborative process. Finally, based on the target collaborative state data and the personnel state data, state feedback is performed on the human-machine collaborative process to improve the efficiency and safety of the human-machine collaborative process. The human-machine collaborative system makes corresponding adjustments according to different feedback states to achieve a more flexible and adaptable collaborative process.

Description

Technical Field

[0001] The present invention relates to the field of human-machine collaboration technology, and in particular to a human-machine collaboration state feedback method, device and electronic equipment based on virtual-reality fusion. Background Art

[0002] With the development of industrialization, automation and intelligence, artificial intelligence, virtual reality technology and augmented reality technology are widely used in human-machine collaborative systems to promote the interactive collaboration between humans and machines. Therefore, it is necessary to propose a human-machine collaborative state feedback method based on virtual-reality fusion. Summary of the invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent. To this end, the present invention proposes a human-machine collaborative state feedback method, device and electronic device based on virtual-reality fusion.

[0004] The present invention provides a method for human-machine collaboration state feedback based on virtual-reality fusion, the method comprising: obtaining human factor data of an operating object in a human-machine collaboration process of a current work task, and an operation screen image obtained by photographing the human-machine collaboration process, the operation screen image comprising at least setting parameters of an operating device and an operating environment; performing human-machine collaboration state recognition based on the operation screen image to obtain target collaboration state data corresponding to the current work task; performing personnel state recognition based on the human factor data to obtain personnel state data of the operating object; and performing state feedback on the human-machine collaboration process according to the target collaboration state data and the personnel state data.

[0005] In the present invention, first, the human factor data of the operating object in the human-machine collaborative process of the current operation task, and the operation screen image obtained by shooting the human-machine collaborative process, which at least includes the setting parameters of the operating device and the operating environment. Then, the human-machine collaborative state is identified based on the operation screen image to obtain the target collaborative state data corresponding to the current operation task, and the personnel state is identified based on the human factor data to obtain the personnel state data of the operating object, providing important information for the human-machine collaborative process. Finally, based on the target collaborative state data and the personnel state data, the human-machine collaborative process is fed back to the state to improve the efficiency and safety in the human-machine collaborative process. The human-machine collaborative system makes corresponding adjustments according to different feedback states to achieve a more flexible and adaptive collaborative process. In addition, based on the state feedback of the human-machine collaborative process, decisions can also be made for the execution of subsequent tasks. By optimizing the process in the human-machine collaborative process, the working conditions and state of the operating object are improved, and sustainable human-machine collaboration is achieved.

[0006] In one embodiment, the human factors data includes eye movement signal data, and the personnel status data includes task prediction status data; the personnel status identification based on the human factors data to obtain the personnel status data of the operation object includes: identifying the gaze position based on the eye movement signal data to obtain the gaze position change of the operation object; wherein the gaze position change is used to describe the change of the gaze position of the operation object with the execution time of the current work task; based on the gaze position change, it is judged whether the operation object predicts the next work task of the current work task to obtain the task prediction status data.

[0007] In this embodiment, the gaze position is identified based on the eye movement signal data to obtain the change in the gaze position of the operating object. Based on the change in the gaze position, it is determined whether the operating object predicts the next task of the current task, and task prediction status data is obtained to help the system understand the task intention of the operating object, thereby better assisting the operating object to complete the task.

[0008] In one embodiment, the gaze position is identified based on the eye movement signal data to obtain the gaze position change of the operating object, including: determining the gaze point scanning path of the operating object based on the eye movement signal data; determining the gaze position change of the operating object based on the gaze point scanning path.

[0009] In this embodiment, the gaze point scanning path of the operating object is determined based on the eye movement signal data, and the gaze position change of the operating object is determined based on the gaze point scanning path, so as to infer the intention and expected behavior of the operating object.

[0010] In one embodiment, the current work task corresponds to the current task area range, and the next work task corresponds to the next task area range; the judging based on the change in the gaze position whether the operating object predicts the next work task of the current work task to obtain the task prediction status data includes: judging based on the change in the gaze position whether at least part of the gaze resources of the operating object are transferred from the current task area range to the next task area range to obtain the task prediction status data.

[0011] In this embodiment, based on the change in gaze position, it is determined whether at least part of the gaze resources of the operating object are transferred from the current task area range to the next task area range, and task prediction status data is obtained to help the system understand the task intention of the operating object, thereby determining the status of the operating object to better guide the operating object to complete the task.

[0012] In one embodiment, the human factors data includes at least one of EEG signal data, galvanic skin signal data, and heart rate signal data; the method also includes: performing load state identification based on the EEG signal data to obtain load state data of the operating object; performing emotional state identification based on the galvanic skin signal data and the heart rate signal data to obtain emotional state data of the operating object; and issuing an early warning of the working state of the operating object based on the load state data and / or the emotional state data.

[0013] In this embodiment, load status is identified based on EEG signal data to obtain load status data of the operating object, and emotional status is identified based on skin electrodermal signal data and heart rate signal data to obtain emotional status data of the operating object. The working status of the operating object is warned based on the load status data and / or emotional status data, so that fatigue, stress and negative emotions of the operating object can be detected and responded to in time, work efficiency, safety and satisfaction can be improved, and work accidents and errors can be prevented.

[0014] In one embodiment, the human-machine collaborative state recognition is performed based on the work screen image to obtain the target collaborative state data corresponding to the current work task, including: performing human-machine collaborative state recognition according to the work screen image to obtain the current collaborative state data of the current work task as the target collaborative state data; or performing human-machine collaborative state recognition according to the work screen image to obtain the current collaborative state data of the current work task, and determining the target collaborative state data of the adjacent work task of the current work task based on the current collaborative state data.

[0015] In this embodiment, human-machine collaboration status recognition is performed based on the work screen image to obtain current collaboration status data of the current work task as target collaboration status data, or human-machine collaboration status recognition is performed based on the work screen image to obtain current collaboration status data of the current work task, and target collaboration status data of adjacent work tasks of the current work task are determined based on the current collaboration status data, providing important information for the human-machine collaboration process.

[0016] In one embodiment, the human-machine collaborative state is identified based on the work screen image to obtain the target collaborative state data corresponding to the current work task, including: inputting the work screen image into a target state recognition model to identify the human-machine collaborative state and obtain the target collaborative state data.

[0017] In this embodiment, the working screen image is input into the target state recognition model to identify the human-machine collaborative state, and the target collaborative state data is obtained, which provides important information for the human-machine collaborative process.

[0018] In one embodiment, the target state recognition model is trained by: obtaining a sample screen image obtained by photographing a historical operation process; wherein the label of the sample screen image includes a labeling task identifier, labeling action data, and a labeling tool category corresponding to the labeling action data; inputting the sample screen image into the initial state recognition model for prediction to obtain a predicted task identifier, predicted action data, and a predicted tool category corresponding to the predicted action data; updating the initial state recognition model based on the labeling task identifier, the labeling action data, the labeling tool category, the predicted task identifier, the predicted action data, and the predicted tool category to obtain the target state recognition model.

[0019] In this embodiment, a sample screen image obtained by photographing the historical operation process is obtained, and the sample screen image is input into the initial state recognition model for prediction to obtain the predicted task identification, predicted action data, and the predicted tool category corresponding to the predicted action data. The initial state recognition model is updated based on the labeled task identification, labeled action data, labeled tool category, predicted task identification, predicted action data, and predicted tool category to obtain the target state recognition model, so as to facilitate human-machine collaborative state recognition based on the operation screen image, so as to timely know the target collaborative state data corresponding to the current operation task.

[0020] In one embodiment, the label of the sample screen image is determined in the following manner: the historical operation process is decomposed into tasks to obtain a number of sample operation tasks sorted by time; the labeling task identifier is determined based on the task identifier of the operation task; the actions involved in the sample operation task are encoded to obtain the labeling action data; and the tools used to perform the actions involved in the sample operation task are encoded to obtain the labeling tool category.

[0021] In this implementation, the historical work process is decomposed into tasks to obtain a number of sample work tasks sorted in time. The labeling task identifier is determined based on the task identifier of the work task. The actions involved in the sample work task are encoded to obtain labeling action data. The tools used to perform the actions involved in the sample work task are encoded to obtain labeling tool categories. This allows the action, tool and other task information to be effectively represented in a computer device, providing data support for the subsequent training process of the target state recognition model.

[0022] In one embodiment, the status feedback of the human-machine collaborative process is performed based on the target collaborative status data and the personnel status data, including: determining the preparatory status data of the next work task of the current work task based on the target collaborative status data; wherein the preparatory status data is used to characterize the situation where the operating object needs to make advance preparations for the next work task; and status feedback of the human-machine collaborative process is performed based on the preparatory status data and the personnel status data.

[0023] In this embodiment, the preparatory state data of the next work task of the current work task is determined according to the target collaborative state data, and the state feedback of the human-machine collaborative process is performed based on the preparatory state data and the personnel state data, which can make the human-machine collaboration more efficient and reliable.

[0024] In one embodiment, the status feedback of the human-machine collaborative process based on the preparatory status data and the personnel status data includes: determining a target reminder method corresponding to the personnel status data; displaying the preparatory status data in the target reminder method to remind the operating object to make advance preparations for the next work task.

[0025] In this embodiment, a target reminder method corresponding to the personnel status data is determined, and the preparatory status data is displayed in a target reminder manner to remind the operating object to make advance preparations for the next work task, helping the operating object to avoid missing key steps or required resources and reducing errors and mistakes caused by lack of preparation.

[0026] In one embodiment, the method further includes: making decisions and controlling based on status data fed back from the human-machine collaborative process, so as to execute the next work task in an execution mode that matches the next work task of the current work task, wherein the execution mode is any one of a human-machine collaborative mode, a human-dominated mode, and a machine-dominated mode.

[0027] In this embodiment, decision control is performed based on the status data fed back from the human-machine collaborative process, so that executing the next work task in a manner that matches the next work task of the current work task can effectively balance the decision-making problems of humans and machines, thereby improving the efficiency and safety of the collaborative process.

[0028] The present invention provides a human-machine collaborative state feedback device based on virtual-real fusion, the device comprising:

[0029] A task data acquisition module, used to acquire human factor data of the operation object in the human-machine collaboration process of the current operation task, and an operation screen image obtained by photographing the human-machine collaboration process, wherein the operation screen image at least includes setting parameters of the operation device and the operation environment;

[0030] A human-machine collaborative state recognition module is used to perform human-machine collaborative state recognition based on the operation screen image to obtain target collaborative state data corresponding to the current operation task;

[0031] A personnel status identification module, used to identify the personnel status based on the human factor data to obtain the personnel status data of the operation object;

[0032] A status feedback module is used to provide status feedback on the human-machine collaboration process according to the target collaboration status data and the personnel status data.

[0033] The present invention provides an electronic device, comprising: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, and the instructions are executed by the one or more processors so that the one or more processors implement the steps of the method described in any one of the above-mentioned embodiments.

[0034] The present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

[0035] The present invention provides a computer program product, wherein the computer program product includes instructions, and when the instructions are executed by a processor of a computer device, the computer device is enabled to execute the steps of the method described in any one of the above embodiments.

[0036] In the present invention, first, the human factor data of the operating object in the human-machine collaborative process of the current operation task, and the operation screen image obtained by shooting the human-machine collaborative process, which at least includes the setting parameters of the operating device and the operating environment. Then, the human-machine collaborative state is identified based on the operation screen image to obtain the target collaborative state data corresponding to the current operation task, and the personnel state is identified based on the human factor data to obtain the personnel state data of the operating object, providing important information for the human-machine collaborative process. Finally, based on the target collaborative state data and the personnel state data, the human-machine collaborative process is fed back to the state to improve the efficiency and safety in the human-machine collaborative process. The human-machine collaborative system makes corresponding adjustments according to different feedback states to achieve a more flexible and adaptive collaborative process. In addition, based on the state feedback of the human-machine collaborative process, decisions can also be made for the execution of subsequent tasks. By optimizing the process in the human-machine collaborative process, the working conditions and state of the operating object are improved, and sustainable human-machine collaboration is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1a A schematic diagram of a scenario of a human-machine collaborative state feedback method based on virtual-reality fusion provided in an embodiment of this specification;

[0038] Figure 1b A flowchart of a human-machine collaborative state feedback method based on virtual-reality fusion provided in an embodiment of this specification;

[0039] Figure 2 A schematic diagram of a process for obtaining task prediction status data provided in an implementation manner of this specification;

[0040] Figure 3a A schematic diagram of a flow chart for determining changes in the gaze position of an operation object provided in an embodiment of this specification;

[0041] Figure 3b A schematic diagram of a gaze point scanning path provided in an embodiment of this specification;

[0042] Figure 4 A schematic diagram of the changes of eye movement fixation resources over time provided in an embodiment of this specification;

[0043] Figure 5 A schematic diagram of a process for providing an early warning of the working status of an operation object provided in an implementation manner of this specification;

[0044] Figure 6 A schematic diagram of a process for obtaining a target state recognition model provided in an embodiment of this specification;

[0045] Figure 7a A schematic diagram of a process for determining a label of a sample screen image provided in an embodiment of this specification;

[0046] Figure 7b A schematic diagram of the decomposition of the operation process provided for the implementation method of this specification;

[0047] Figure 8 A schematic diagram of a process for providing status feedback to a human-machine collaborative process provided in an implementation manner of this specification;

[0048] Fig. 9 A flowchart of reminding an operation object to prepare in advance for a next operation task provided in an implementation manner of this specification;

[0049] Fig.10a A schematic diagram of providing early warning feedback for the current task provided in the implementation mode of this specification;

[0050] Fig.10b A flowchart of a human-machine collaborative state feedback method based on virtual-reality fusion provided in an embodiment of this specification;

[0051] Fig.11 A schematic diagram of a human-machine collaborative state feedback device based on virtual-reality fusion provided in an embodiment of this specification;

[0052] Fig.12 This is a diagram of the internal structure of an electronic device provided in accordance with an embodiment of this specification. DETAILED DESCRIPTION

[0053] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0054] With the development of industrialization, automation, and intelligence, artificial intelligence, virtual reality technology, and augmented reality technology are widely used in human-machine collaborative systems to promote interactive collaboration between humans and machines. The implementation of this specification provides a human-machine collaborative state feedback method based on virtual-real fusion. The human-machine collaborative state feedback method based on virtual-real fusion can identify the status of personnel and the status of human-machine collaboration in real time during the human-machine collaborative process by combining augmented reality technology, and provide feedback.

[0055] Specifically, first, the human factor data of the operating object in the human-machine collaboration process of the current task is obtained, and the operating screen image including at least the setting parameters of the operating device and the operating environment is obtained by shooting the human-machine collaboration process. Then, the human-machine collaboration state is identified based on the operation screen image to obtain the target collaboration state data corresponding to the current task, and the personnel state is identified based on the human factor data to obtain the personnel state data of the operating object, providing important information for the human-machine collaboration process. Finally, according to the target collaboration state data and the personnel state data, the human-machine collaboration process is fed back to the state to improve the efficiency and safety of the human-machine collaboration process. The human-machine collaboration system makes corresponding adjustments according to different feedback states to achieve a more flexible and adaptive collaboration process. In addition, based on the state feedback of the human-machine collaboration process, decisions can also be made for the execution of subsequent tasks. Sustainable human-machine collaboration can be achieved by optimizing the process in the human-machine collaboration process, improving the working conditions and state of the operating object.

[0056] The human-machine collaborative state feedback method based on virtual-reality fusion provided in the embodiments of this specification can be applied to augmented reality devices. Figure 1a, the human-machine collaborative state feedback method based on virtual-real fusion may include an augmented reality module, a state recognition module, and a real-time feedback module. The augmented reality module may be implemented by an augmented reality device. The state recognition module and the real-time feedback module may be deployed on a processing device, such as a computer. Before executing task A, the processing device may perform model training using multiple subtasks decomposed from task A, the actions required to execute each subtask, and the tools used when executing the actions involved in each subtask to obtain a target state recognition model. On the one hand, the augmented reality device provides scene presentation during the execution of task A, and on the other hand, the human-machine collaborative screen during the execution of task A is collected in real time by a camera device integrated in the augmented reality device to obtain an operating screen image including at least the set parameters of the operating device and the operating environment. At the same time, the eye movement, physiological, and electroencephalogram signal data of the operating object are collected by the sensor integrated in the augmented reality device to obtain the human factor data of the operating object. The augmented reality device uploads the collected human factor data and the operating screen image to the processing device.

[0057] The processing device uses the target state recognition model to identify the human-machine collaborative state of the collected work screen image, obtain the target collaborative state data corresponding to the current subtask, and then determine the preparatory state data of the next subtask of the current subtask based on the target collaborative state data. The processing device also performs personnel state recognition on the collected human factor data to obtain the personnel state data of the operation object. Based on the preparatory state data and personnel state data, the processing device displays the early warning and real-time picture of the human-machine collaborative process on the AR interface of the augmented reality device to realize feedback early warning. The processing device can also provide status feedback on the human-machine collaborative process based on the preparatory state data and personnel state data.

[0058] In order to achieve human-machine decision-making, the execution mode that matches the next subtask of the current subtask is determined according to the state data fed back from the human-machine collaborative process, including human-machine collaborative mode, human-dominated mode and machine-dominated mode. Then, the next subtask is executed according to the determined execution mode to achieve command control.

[0059] This specification provides a human-machine collaborative state feedback method based on virtual-real fusion. Figure 1b The human-machine collaborative state feedback method based on virtual-reality fusion may include the following steps:

[0060] S110, obtaining human factor data of the operating object in the human-machine collaborative process of the current operation task, and an operation screen image obtained by photographing the human-machine collaborative process.

[0061] Among them, the operation task can be a complete workflow or a specific subtask or operation step. For example, the operation task can be a certain operation step on a factory production line or a certain link in the cargo loading and unloading process at an airport. The human-machine collaboration process can be a process in which the operation object and the operation device cooperate to complete a specific task. In the human-machine collaboration process, the operation object and the operation device cooperate with each other to achieve a common goal through interaction, information exchange and mutual support. The operation object can be a human worker who performs the current operation task. Human factor data can be physiological data and / or behavioral data related to the operation object, such as biological signals such as brain waves, heart rate, and breathing, as well as behavioral information such as action and reaction time. The operation screen image can be an image of the human-machine collaboration process obtained by a shooting device (such as a camera) during the human-machine collaboration process. The operation screen image at least includes the setting parameters of the operation device and the operation environment. The operation device can be a device or machine used by a person in the human-machine collaboration process to complete a specific operation task. The setting parameters of the operation environment can be parameters set and adjusted in the human-machine collaboration process to ensure the correct performance of the work.

[0062] Specifically, because the operating device needs to complete some specific tasks together with the operating object during the human-machine collaboration process, in the human-machine collaboration process of the current task, data is collected and recorded through sensors (such as electroencephalogram (EEG), heart rate monitor) or other technical equipment to obtain the human factor data of the operating object, so as to help the operating device better understand and adapt to the state and needs of the operating object. In addition, in the human-machine collaboration process, the human-machine collaboration process is photographed by an image acquisition device to obtain an operating screen image that at least includes the setting parameters of the operating device and the operating environment, so that the operating device can perceive and understand the task background, thereby better assisting the operating object to complete the task, wherein the image acquisition device can be any one of a camera, a video camera, a fisheye camera, and a surveillance camera.

[0063] In some embodiments, by adding sensors for collecting human factors data such as eye movement, physiology, and EEG signals to augmented reality devices (such as AR glasses), the collection of human factors data such as eye movement data, physiological data, and EEG signal data is realized, and the human factors data of the operating object in the human-machine collaboration process of the current task is obtained. At the same time, the real-time collection of the human-machine operation operation screen is realized through the camera device integrated in the augmented reality device, and the operation screen image including at least the setting parameters of the operating device and the operating environment is obtained.

[0064] In some cases, data preprocessing (such as noise removal, standardization of data format, and smoothing) can be performed on the data of the operating object obtained through sensors (such as electroencephalogram (EEG), heart rate monitor) or other technical equipment and the screen images obtained by image acquisition equipment to obtain human factors data and work screen images, so as to improve the accuracy and reliability of subsequent analysis.

[0065] S120: Perform human-machine collaborative status recognition based on the operation screen image to obtain target collaborative status data corresponding to the current operation task.

[0066] Among them, the human-machine collaborative state can be the relationship and cooperation state between the operating object and the operating device when the operating object and the operating device perform collaborative tasks. For example, the human-machine collaborative state can describe the interaction mode between the operating object and the operating device, the state data of the operating device, etc. in a specific task. The target collaborative state data can be used to describe the cooperation state and behavior between the entities participating in the collaboration (which may be operating objects, robots or other intelligent devices). For example, the target collaborative state data may include at least one of tools, actions and interaction information.

[0067] Specifically, features are extracted from the work screen image to obtain image features that can characterize the collaborative state, so that the extracted image features can be predicted to obtain target collaborative state data, thereby realizing the recognition of the human-machine collaborative state.

[0068] Exemplarily, the target detection algorithm is used to identify the human-machine collaborative state of the work screen image to obtain the target collaborative state data corresponding to the current work task. For example, the posture estimation algorithm can be used to classify and identify the actions in the work screen image. The tool recognition algorithm can be used to analyze the tools used in the work screen image to perform the actions involved in the current work task.

[0069] S130: Perform personnel status identification based on human factor data to obtain personnel status data of the operation object.

[0070] The personnel status data may be obtained by observing and analyzing the physiological data and behavioral data of the person, thereby obtaining information about the current status of the person.

[0071] Specifically, the human factor data is feature extracted to obtain features that can characterize the state of the person, so that the extracted features can be predicted to obtain the person state data, thereby realizing the recognition of the person state. Exemplarily, for different human factor data, the corresponding detection method is used to perform person state recognition on the human factor data to obtain the person state data of the operation object. For example, recognition is performed based on the eye movement data of the operation object to obtain the person state data that can characterize the visual attention distribution and cognitive state of the operation object.

[0072] S140. Provide status feedback on the human-machine collaboration process based on the target collaboration status data and personnel status data.

[0073] Among them, status feedback can be the feedback of the comprehensive analysis results of the target collaborative status data and personnel status data, so as to provide corresponding information in a timely manner and make corresponding adjustments and decisions.

[0074] Specifically, according to the characteristics of different tasks and the needs of human-computer interaction, design corresponding status feedback means, such as interface display, voice prompts, vibration prompts, etc. Perform data analysis on the target collaborative status data and personnel status data to obtain data analysis results. Based on the key indicators and constraints in the data analysis results, determine the status of the human-computer collaborative process, such as the actual situation of the current task execution. According to the status of the human-computer collaborative process, match the corresponding feedback means. Through feedback means, feedback information can be delivered to the operation object and operation device in a timely and accurate manner in the human-computer collaborative system.

[0075] In some embodiments, the analysis results of the target collaborative state data and the personnel state data can be superimposed on the actual field of view of the operating object by using AR technology, which can provide more intuitive and real-time information display and auxiliary decision-making. The operating object can observe the relevant information superimposed on the real scene in real time through a head-mounted display device (such as AR glasses), so as to better understand the human-machine collaborative state and take corresponding actions.

[0076] In other embodiments, the operating object enters the virtual environment through a VR device (such as a VR helmet) and can observe and interact with objects in the virtual environment. The analysis results of the target collaborative state data and the personnel state data are presented in the virtual environment in the form of graphics, animation or other forms. The operating object can understand the current state by observing information such as the virtual interface or object state and take corresponding actions.

[0077] In the above implementation, first, the human factor data of the operating object in the human-machine collaborative process of the current operation task is obtained, and the operating screen image including at least the setting parameters of the operating device and the operating environment is obtained by shooting the human-machine collaborative process. Then, the human-machine collaborative state is identified based on the operating screen image to obtain the target collaborative state data corresponding to the current operation task, and the personnel state is identified based on the human factor data to obtain the personnel state data of the operating object, providing important information for the human-machine collaborative process. Finally, based on the target collaborative state data and the personnel state data, the human-machine collaborative process is fed back to the state to improve the efficiency and safety of the human-machine collaborative process. The human-machine collaborative system makes corresponding adjustments according to different feedback states to achieve a more flexible and adaptive collaborative process. In addition, based on the state feedback of the human-machine collaborative process, decisions can also be made for the execution of subsequent tasks. Sustainable human-machine collaboration can be achieved by optimizing the process in the human-machine collaborative process and improving the working conditions and state of the operating object.

[0078] In some embodiments, see Figure 2 The human factor data includes eye movement signal data, and the personnel status data includes task prediction status data; performing personnel status recognition based on the human factor data to obtain the personnel status data of the operation object may include the following steps:

[0079] S210: Identify the gaze position based on the eye movement signal data to obtain changes in the gaze position of the operating object.

[0080] S220: Based on the change in gaze position, determine whether the operating object predicts the next task of the current task, and obtain task prediction status data.

[0081] Among them, the eye movement signal data can be data reflecting the movement and gaze position of the human eye obtained through eye tracking technology. The gaze position can be a specific area or target position pointed to by the eyes of the operating subject. The change in gaze position is used to describe the change in the gaze position of the operating subject with the execution time of the current task. The next task can be the next task in the current task that needs to be completed by the operating subject in a certain order and executed immediately after the current task. The task prediction status data can be a representation based on the change in gaze position, which is used to describe whether the operating subject has predicted the next task when executing the current task.

[0082] Specifically, an eye tracking device, such as an eye tracker, is used to record the eye movement signal data of the operating object. The eye movement signal data records the information such as the gaze point coordinates and the gaze duration of the operating object in the form of a time series. Then, the eye movement signal data is processed and analyzed using an analysis algorithm such as a neural network or machine learning to detect the gaze point coordinates, thereby obtaining the gaze position of the operating object. According to the recorded gaze position combined with the collaborative state information of the current task and the next task, the change of the gaze position over time is analyzed, so as to understand the distribution of the focus and attention of the operating object, and to determine whether the operating object has predicted the next task. According to the judgment result, the task prediction state data is generated. Exemplarily, the task prediction state data can be expressed as a binary indicator variable, such as "predicted" and "not predicted". When the operating object has predicted the next task of the current task, the task prediction state is "predicted", otherwise it is "not predicted".

[0083] In some embodiments, the system for determining task prediction status data may include an input layer, an identification layer, and a judgment layer. In the input layer, the system pre-inputs the order of each task, the completion duration, the coordinate range of the task area, and the gaze position input in real time. The identification layer is responsible for determining whether the gaze position changes over time and falls within the area range of the current task or the next task. The judgment layer combines the completion time of the task and the information of the identification layer to determine whether the operation object has performed task prediction between the completion of the current task and the next task, thereby obtaining task prediction status data.

[0084] In other embodiments, the predictability of the task can be evaluated based on the eye gaze and the scanning path. In this process, considering the needs of the human-computer interaction task, it can be determined whether the information of the next task has been noticed, and when to start paying attention to the next task during the current task to determine the connection between the tasks. Then, it can be determined whether the next task of the current task has been predicted based on the duration of the current task and the connection between the tasks, thereby obtaining task prediction status data.

[0085] In the above implementation, the gaze position is identified based on the eye movement signal data to obtain the change in the gaze position of the operating object. Based on the change in the gaze position, it is determined whether the operating object predicts the next work task of the current work task, and task prediction status data is obtained to help the system understand the task intention of the operating object, thereby better assisting the operating object to complete the task.

[0086] In some embodiments, see Figure 3a, identifying the gaze position based on the eye movement signal data to obtain the gaze position change of the operation object may include the following steps:

[0087] S310, determining a gaze point scanning path of the operation object based on the eye movement signal data.

[0088] S320: Determine a change in the gaze position of the operation object based on the gaze point scanning path.

[0089] The gaze point scanning path may be a track of a series of gaze points that the eyes pass through when observing a specific task or scene within a period of time. The gaze point scanning path records the order and path of the eyes moving from one gaze point to another.

[0090] Specifically, the eye movement signal data is processed and analyzed using analysis algorithms such as neural networks or machine learning to detect the coordinates of the gaze point, thereby obtaining the gaze position of the operating object. The transfer relationship and duration between gaze positions are analyzed, and a gaze point scanning path is constructed to reveal the distribution and changes of the operating object's attention. Based on the gaze point scanning path, the changes in the gaze position are analyzed, such as counting the gaze positions in different areas and the time when they occur, to quantify the distribution of the operating object's attention and infer the distribution of the operating object's attention to different targets.

[0091] For example, see Figure 3b , Figure 3b Schematic diagram of the fixation scanning path. Figure 3b There are 12 gaze positions in the task n, and the 12 gaze positions are numbered according to the time sequence. The line between each two gaze positions is the gaze point scanning path. According to the gaze point scanning path, it can be determined that gaze positions 1 to 5 are located in the area of ​​task n, gaze positions 5 to 10 switch back and forth between task n and task n+1, and gaze positions 10 to 12 are located in the area of ​​task n+1. Based on the above analysis, the change of gaze positions can be determined.

[0092] In the above implementation, the gaze point scanning path of the operating object is determined based on the eye movement signal data, and the gaze position change of the operating object is determined based on the gaze point scanning path, so that the intention and expected behavior of the operating object can be inferred.

[0093] In some implementations, the current task corresponds to the current task area range, and the next task corresponds to the next task area range. Based on the change in gaze position, judging whether the operation object predicts the next task of the current task, and obtaining task prediction state data, includes: judging whether at least part of the gaze resource of the operation object is transferred from the current task area range to the next task area range based on the change in gaze position, and obtaining task prediction state data.

[0094] Among them, the gaze resource can be the distribution of the operating subject's attention on a specific area or target. The task area range can be the spatial range or target area of ​​the specific task that the operating subject is performing, and the area can be a certain location or object in the physical environment. The current task area range can be the spatial range or area corresponding to the current task when the current task is performed. The next task area range can be the spatial range or area corresponding to the next task when the next task of the current task is performed.

[0095] Specifically, determine the current task area range corresponding to the current task being executed and the next task area range corresponding to the next task. Compare the gaze position with the task area to determine whether the gaze position is within the current task area range or whether it is within the next task area range. Then, by observing the change of the gaze position, determine whether there is a gaze resource transferred from the current task area range to the next task area range, and obtain a judgment result of the gaze position change. Based on the judgment result of the gaze position change, obtain task prediction state data. Exemplarily, the task prediction state data can be expressed as a binary indicator variable, such as "predicted" and "not predicted". When the judgment result of the gaze position change is that at least part of the gaze resources of the operation object are transferred from the current task area range to the next task area range, the task prediction state is "predicted", otherwise it is "not predicted".

[0096] In some embodiments, see Figure 3b , Figure 3b The area 302 in FIG. 1 is the current task area range corresponding to the current job task n being executed. Figure 3b The area 304 in the figure is the next task area range corresponding to the next operation task n+1. The 5th to 10th gaze positions switch back and forth between task n and task n+1, and it can be determined that at least part of the gaze resources of the operation object are transferred from the current task area range to the next task area range.

[0097] In some other implementations, the current task is recorded as task n, and the next task of the current task is recorded as task n+1. Figure 4 , Figure 4 The curve 402 in is the change of the eye movement fixation resource of task n over time. Figure 4Curve 404 in FIG. 4 is the change of eye movement attention resources of task n+1 over time. It is assumed that task n takes 7 seconds to complete, and task n+1 is immediately after task n. If task n is currently being executed, in the first 5 seconds, eye movement attention resources are concentrated on task n. Then, starting from the 5th second, part of the attention resources are transferred to the next task n+1. As time goes by, attention resources gradually tend to be allocated to task n+1. When task n is finished, due to the continuity of the tasks, a small part of the attention resources will still be allocated to task n.

[0098] In the above implementation, based on the change in gaze position, it is determined whether at least part of the gaze resources of the operating object are transferred from the current task area to the next task area, and task prediction status data is obtained to help the system understand the task intention of the operating object, thereby determining the status of the operating object to better guide the operating object to complete the task.

[0099] In some embodiments, see Figure 5 The human factor data includes at least one of brain electroencephalogram signal data, skin electroencephalogram signal data, and heart rate signal data. The method may also include:

[0100] S510: Perform load state identification based on the EEG signal data to obtain load state data of the operation object.

[0101] Among them, EEG signal data can be electrical signals of brain activity of the operating subject detected and recorded by placing electrodes on the scalp surface. Load state can be the degree of psychological and physiological stress that the operating subject endures when performing a task. Load state data can be a record or representation of the load state of the operating subject obtained by a load state identification method based on EEG signal data.

[0102] Specifically, the load state is divided into different categories according to the pre-set judgment criteria. This indicates that there is a clearly defined standard to evaluate and distinguish different load states. Based on the judgment criteria and different categories of load states, a corresponding load state model is established. The EEG signal data of the operating object is collected using an EEG signal acquisition device (such as an electroencephalogram (EEG) device). Then, the load state of the EEG signal data is identified through the load state model, and the load state of the operator when performing the task can be obtained. Among them, support vector machines and machine learning algorithms are used to realize the identification of load states, which can effectively distinguish different categories of load states. According to the real-time evaluation of the load model, the working state of the operator can be accurately monitored and adjusted to improve production efficiency and work quality.

[0103] For example, according to the α / β value of EEG as the judgment standard, the load state is divided into three situations: high, medium and low, and a corresponding load state model is established. By monitoring the frontal EEG signal of the operator, the frontal EEG signal data is obtained. By identifying the frontal EEG signal data through the load state model, the load state of the operator when performing the task can be obtained.

[0104] S520: Emotional state recognition is performed based on the electrodermal signal data and the heart rate signal data to obtain emotional state data of the operation object.

[0105] Among them, the skin electrode signal data can be the electrophysiological response of the skin surface. The skin electrode signal data is usually related to emotional states, such as anxiety, excitement, and stress. The heart rate signal data can measure the changes in the heart beat of the operating object. The heart rate signal data can reflect the rhythmic activity of the heart. The emotional state can be the subjective emotion experienced by the operating object at a specific moment. The emotional state data can be information used to describe the current emotional state of the operating object obtained by analyzing the skin electrode signal data and the heart rate signal data.

[0106] In some cases, the skin electrical signal data and the heart rate signal data can reflect the activity of the sympathetic nerves, and the activity of the sympathetic nerves is related to the emotional state. Therefore, by monitoring the skin electrical signal data and the heart rate signal data of the operation object, information reflecting the emotional state of the operation object can be obtained.

[0107] Specifically, the skin electrode signal data of the operating object is collected using a skin electrode sensor, and the heart rate monitoring device is used to collect the heart rate signal data of the operating object. Machine learning algorithms such as support vector machine (SVM), decision tree, neural network, etc. can be used, and deep learning methods can also be used to establish an emotional state model. In the process of emotional state recognition, the skin electrode signal data and the heart rate signal data are input into the emotional state model. According to the state classification of the emotional state model, the skin electrode signal data and the heart rate signal data are identified to obtain the emotional state data of the operating object. Based on the emotional state data of the operating object, the abnormal emotional state of the operating object can be identified. Abnormal emotional state can also be identified based on the skin electrode signal data and the heart rate signal data to obtain the emotional state data of the operating object. It should be noted that the heart rate signal data can also be an LF indicator (low frequency component) of heart rate variability.

[0108] In some embodiments, the emotional baseline state of each operating object is measured and recorded, and a baseline state database corresponding to the operating object is established. In this process, physiological signals such as skin electrical signal data and heart rate signal data can be used to measure the emotional baseline state of each operating object. The baseline state X value of each operator is obtained from the baseline state database as a reference value for judging abnormal emotions. The actual state Y value of the operating object is compared with the baseline state X value. If Y>aX, it is determined to be abnormal emotion. Among them, the threshold a can be customized according to actual needs, and the threshold a can be determined by experiments or experience.

[0109] S530: Issue an early warning on the working status of the operating object according to the load status data and / or the emotional status data.

[0110] Specifically, early warning indicators and thresholds corresponding to the load status data and emotional status data are set according to actual needs to determine whether there is any abnormality in the working status of the operating object. The load status data and / or emotional status data of the operating object are monitored in real time, and a judgment is made based on the set early warning indicators or thresholds. When the load status data and / or emotional status data of the operating object meets the early warning indicators or exceeds the threshold, the early warning mechanism is triggered to warn the working status of the operating object. Furthermore, the early warning information can be sent to relevant personnel, such as the operating object, supervisors, etc. At the same time, different early warning methods can be adopted, such as sound, vibration, flash, etc., to attract attention in time.

[0111] In the above implementation, load status is identified based on EEG signal data to obtain load status data of the operating object, and emotional status is identified based on skin electrodermal signal data and heart rate signal data to obtain emotional status data of the operating object. The working status of the operating object is warned based on the load status data and / or emotional status data, so that fatigue, stress and negative emotions of the operating object can be detected and responded to in time, thereby improving work efficiency, safety and satisfaction and reducing the chances of work accidents and errors.

[0112] It should be noted that the attention status of the operating object and the performance of the operating object can be evaluated based on the task prediction state data. For example, when the operating object is under high load, mental fatigue or abnormal emotions, the task prediction state data of the operating object can be determined based on the preparatory state data. By analyzing the task prediction state data, it can be determined whether the operating object allocates attention resources to the next operating task. If the operating object can effectively allocate attention resources to the next operating task, it can be considered that the operating object is suitable for working in a high-intensity operating environment. If the operating object does not allocate attention resources to the next operating task, it can be considered that the operating object is not suitable for high intensity.

[0113] In addition, when the operating subject waits for a long time, he may be distracted. Based on the task prediction status data, it can be judged whether the operating subject allocates attention resources to the next task. If the operating subject can allocate attention resources to the next task, it can be considered that the operating subject is suitable for working in a working environment that requires high concentration of attention.

[0114] When an emergency occurs, the required prejudgment state data can be obtained. Using the prejudgment state data required in the emergency, the preparatory state data of the operating object can be determined. Based on the preparatory state data, it is judged whether the operating object allocates attention resources to the emergency. If the operating object can effectively allocate attention resources to deal with the emergency, it can be considered that the operating object has the ability to deal with the emergency. If the operating object fails to allocate attention resources to deal with the emergency, and the operating object is not in a high load or fatigue state, it can be considered that the operating object lacks the ability to handle complex tasks.

[0115] In some embodiments, human-machine collaborative status recognition is performed based on the work screen image to obtain target collaborative status data corresponding to the current work task, which may include: human-machine collaborative status recognition is performed based on the work screen image to obtain current collaborative status data of the current work task as target collaborative status data.

[0116] Specifically, feature extraction is performed on the work screen image to obtain image features that can characterize the collaborative state, so that the extracted image features can be predicted to obtain the current collaborative state data of the current work task, such as the state of the tools, the relative positions between the tools, the interactive actions between the operating object and the operating device, etc. In order to monitor the current collaborative state data in real time and provide corresponding feedback to reduce the occurrence of errors and delays, the current collaborative state data can be used as the target collaborative state data.

[0117] In the above implementation, the human-machine collaborative state is identified based on the work screen image to obtain the current collaborative state data of the current work task as the target collaborative state data, providing important information for the human-machine collaborative process.

[0118] In some embodiments, human-machine collaboration status recognition is performed based on the work screen image to obtain target collaboration status data corresponding to the current work task, which may include: human-machine collaboration status recognition is performed based on the work screen image to obtain current collaboration status data of the current work task, and target collaboration status data of adjacent work tasks of the current work task is determined based on the current collaboration status data.

[0119] Among them, the operation process is task-decomposed to obtain task requirement data, which includes multiple operation tasks sorted by time. The task requirement data includes the current operation task, which corresponds to the adjacent operation task. For example, the adjacent operation task can be the operation task that is arranged before and adjacent to the current operation task in the task requirement data. The adjacent operation task can be the operation task that is arranged after and adjacent to the current operation task in the task requirement data. It should be noted that the task execution order in the pre-determined task requirement data may be different from the task execution order in the actual operation process.

[0120] Specifically, feature extraction is performed on the work screen image to obtain image features that can characterize the collaborative state, so that the extracted image features can be predicted to obtain the current collaborative state data of the current work task. Then, according to the task sorting in the task requirement data, the current collaborative state data is used to find the adjacent work tasks of the current work task. After determining the adjacent work tasks, the collaborative state data corresponding to the adjacent work tasks can be searched from the storage unit and used as the target collaborative state data. The target detection algorithm can also be used to perform collaborative state recognition on the real-time work screen image to obtain the collaborative state data of the adjacent work tasks of the current work task, and use it as the target collaborative state data.

[0121] Exemplarily, the task requirement data may include task 1, task 2, and task 3 arranged in the order of execution. However, during the actual execution process, if the currently executed task is task 2, since task 1 has not been executed, the adjacent task 1 can be used as the next task. The collaborative state data of task 1 can be used as the target collaborative state data. If the currently executed task is task 2, since task 1 has been executed, the adjacent task 3 can be used as the next task. The collaborative state data of task 3 can be used as the target collaborative state data.

[0122] In the above implementation, the human-machine collaboration state is identified based on the work screen image to obtain the current collaboration state data of the current work task, and the target collaboration state data of the adjacent work tasks of the current work task are determined based on the current collaboration state data, providing important information for the human-machine collaboration process.

[0123] In some embodiments, human-machine collaborative state recognition is performed based on the work screen image to obtain target collaborative state data corresponding to the current work task, which may include: inputting the work screen image into a target state recognition model to recognize the human-machine collaborative state and obtain the target collaborative state data.

[0124] Specifically, an initial state recognition model can be built in advance, and the initial state recognition model can be trained to obtain a target state recognition model. The target state recognition model is deployed in a processing device. The processing device is usually connected to an image acquisition device. The image acquisition device is used to shoot the operation screen to obtain an operation screen image. The image acquisition device sends the operation screen image to the processing device, and the processing device uses the operation screen image as the input of the target state recognition model. The target state recognition model extracts features from the operation screen image to obtain image features that can characterize the collaborative state, so that predictions can be made based on the extracted image features. The target state recognition model outputs the target collaborative state data to realize the recognition of the human-machine collaborative state. It can be understood that in some cases, the target state recognition model can also be deployed in the image acquisition device, and the image acquisition device is used to shoot the operation screen to obtain the operation screen image, and the operation screen image is input into the target state recognition model to recognize the human-machine collaborative state, and obtain the target collaborative state data.

[0125] In the above implementation, the work screen image is input into the target state recognition model to identify the human-machine collaborative state, and the target collaborative state data is obtained, which provides important information for the human-machine collaborative process.

[0126] In some embodiments, see Figure 6 , the target state recognition model is trained by the following method:

[0127] S610: Acquire a sample screen image obtained by photographing the historical operation process.

[0128] S620: Input the sample screen image into the initial state recognition model for prediction, and obtain the prediction task identifier, prediction action data, and the prediction tool category corresponding to the prediction action data.

[0129] S630: Update the initial state recognition model based on the labeling task identifier, the labeling action data, the labeling tool category, the prediction task identifier, the prediction action data, and the prediction tool category to obtain a target state recognition model.

[0130] The label of the sample screen image includes a labeling task identifier, a labeling action data, and a labeling tool category corresponding to the labeling action data. The task identifier can be used to identify a specific task or operation type. The action data can be used to characterize the action required to perform the task. The tool category can be used to characterize the tool used to perform the task.

[0131] Specifically, a sample screen image is obtained by photographing the historical operation process using an image acquisition device. The sample screen image is annotated by using a professional annotation tool, and each frame of the sample screen image is assigned a corresponding annotation task identifier, annotation action data, and annotation tool category, thereby obtaining a label for the sample screen image. The sample screen image is input into the initial state recognition model for prediction, and a prediction task identifier, prediction action data, and a prediction tool category corresponding to the prediction action data are obtained. Then, a first loss value is determined based on the annotation task identifier and the prediction task identifier. A second loss value is determined based on the annotation action data and the prediction action data. A third loss value is determined based on the annotation tool category and the prediction tool category. The initial state recognition model is updated based on the first loss value, the second loss value, and the third loss value. By analogy, the updated initial state recognition model is continued to be trained, and when the model training stop condition is reached, the target state recognition model can be obtained. Among them, the model training stop condition can be that the model loss value tends to converge, or that the training rounds reach a preset number of rounds.

[0132] In some embodiments, the initial state recognition model can adopt the YOLO V5 model. The YOLO V5 model is a neural network model algorithm for object detection tasks, and its main goal is to identify objects present in an image and determine the position of the identified objects on the image. The model divides the image into an S×S grid, and each grid cell is responsible for detecting the object. By predicting each grid cell, the position of the bounding box, the confidence score of the box, and the category probability of the grid cell containing the object can be obtained. Then, based on the position of the bounding box, the confidence score of the box, and the category probability of the grid cell containing the object, data analysis can be performed to determine the output result of the model, thereby achieving the object detection task.

[0133] In the above implementation, a sample screen image is obtained by photographing the historical operation process, and the sample screen image is input into the initial state recognition model for prediction to obtain the predicted task identifier, predicted action data, and the predicted tool category corresponding to the predicted action data. The initial state recognition model is updated based on the labeled task identifier, labeled action data, labeled tool category, predicted task identifier, predicted action data, and predicted tool category to obtain the target state recognition model, so as to facilitate human-machine collaborative state recognition based on the operation screen image, so as to timely know the target collaborative state data corresponding to the current operation task.

[0134] In some embodiments, see Figure 7a , determine the label of the sample screen image in the following way:

[0135] S710: Decompose the historical operation process into tasks to obtain a number of sample operation tasks sorted by time.

[0136] S720: Determine a labeling task identifier based on the task identifier of the operation task.

[0137] S730: Encode the actions involved in the sample task to obtain labeled action data.

[0138] S740: Encode the tools used when executing the actions involved in the sample job task to obtain a labeling tool category.

[0139] Specifically, according to the nature and complexity of the task and the key nodes in the execution process, a suitable task decomposition method is selected. According to the determined task decomposition method, the historical operation process is decomposed to obtain a number of sample tasks sorted by time. Each sample task corresponds to its own action, the tool used when executing the action involved in the sample task, and other information. The sample tasks are independent of each other, that is, there is a sequence of execution between the sample tasks, but there is no interlaced part in space. It can also be understood that the actions and tools for completing each sample task are independent of each other. A unique task identifier is assigned to each sample task, such as a number or a name. According to the task identifier of the sample task, a unique labeling task identifier is assigned to each sample task. The task identifier of the sample task can be directly used as the labeling task identifier, or the task identifier of the sample task can be encoded to obtain the labeling task identifier. The actions involved in the sample task are encoded, and the actions involved in each sample task are represented as a unique labeling action data, so that each action can be accurately identified and recorded in subsequent processing and analysis. The tools used to perform the actions involved in the sample work tasks are encoded, and each tool used to perform the actions involved in the sample work tasks is represented as a unique labeled tool category, so that each tool can be accurately identified and recorded in subsequent processing and analysis.

[0140] For example, based on the task of the operating device and the type of task that the operating object needs to perform, the tasks can be differentiated according to the requirements of the time series, and the work process that needs to be completed can be decomposed into a series of tasks. First, for the work process that needs to be completed, the components of the task can be analyzed according to the process of human-machine collaboration, including the actions involved and the tools required to complete these actions. Then, the work process can be decomposed into a series of tasks, and the actions required to perform each task and the tools used to perform the actions involved in each task can be determined. During the task execution process, the tools used to perform the actions involved in each task are used to complete the required actions at the defined location to achieve the execution of the task. Please refer to Figure 7b, the operation process can be decomposed into task 1, task 2...task n, and the components of each task include corresponding actions and tools. For example, the components of task 1 include action 1 required to perform task 1 and tool 1 used when performing action 1 involved in task 1; the components of task 2 include action 2 required to perform task 2 and tool 2 used when performing action 2 involved in task 2; the components of task n include action n required to perform task n and tool n used when performing action n involved in task n.

[0141] In the above implementation, the historical work process is decomposed into tasks to obtain a number of sample work tasks sorted in time, the labeling task identifier is determined based on the task identifier of the work task, the actions involved in the sample work task are encoded to obtain the labeled action data, and the tools used when executing the actions involved in the sample work task are encoded to obtain the labeling tool category, so that the action, tool and other task information can be effectively represented in the computer device, providing data support for the subsequent training process of the target state recognition model.

[0142] In some embodiments, see Figure 8 , according to the target collaborative state data and the personnel state data, the state feedback of the human-machine collaborative process can include the following steps:

[0143] S810. Determine the preparation state data of the next operation task of the current operation task according to the target collaborative state data.

[0144] S820: Provide status feedback on the human-machine collaboration process based on the preparation status data and the personnel status data.

[0145] The preparation state data is used to characterize the situation that the operation object needs to prepare in advance for the next operation task. The preparation state data may be information and requirements related to the next operation task, which need to be prepared in advance before the current operation task is completed. For example, the preparation state data may include at least one of the materials, equipment, tools required for the next operation task, the operation steps, sequence, precautions, etc. required for the next operation task.

[0146] Specifically, after determining the target collaborative state data of the current task, the next task of the current task and the collaborative state data of the next task can be identified through the target state recognition model. The collaborative state data of the next task is analyzed to obtain the preparatory state data required for the next task. The preparatory state data may include at least one of the materials, step instructions, and related information required to perform the next task. The personnel status data may include information on the physiological state, attention level, etc. of the operating object. By analyzing the preparatory state data and personnel status data, various state information in the human-machine collaborative process can be timely understood, and it can be fed back to the operating object and / or operating equipment in an appropriate manner, which helps the operating object to adjust strategies or decisions in a timely manner, and also helps the operating equipment to adjust its own operating state according to the feedback information to better adapt to the current collaborative needs and environment.

[0147] For example, if the personnel status data indicates that the operating object is in an abnormal state (such as fatigue), and the execution mode of the next operation task can be human-led or machine-led. Then, when the personnel status data indicates that the operating object is in an abnormal state, the preparation state data is sent to the operating device, and the next operation task is executed in a machine-led manner, which can reduce the operation risk of the operating object in a fatigued state and improve the safety and reliability of task execution.

[0148] In the above implementation, the preparatory state data of the next work task of the current work task is determined according to the target collaborative state data, and the state feedback of the human-machine collaborative process is performed based on the preparatory state data and the personnel state data, which can make the human-machine collaboration more efficient and reliable.

[0149] In some embodiments, see Fig. 9 , based on the preparation state data and personnel state data, the state feedback of the human-machine collaboration process can include the following steps:

[0150] S910: Determine a target reminder method corresponding to the personnel status data.

[0151] S920: Display the preparation status data in a target reminder manner to remind the operation object to prepare in advance for the next operation task.

[0152] The target reminder method may be to convey the target and preparation status data of the next task to the operation object through a specific method and channel so as to make advance preparations. For example, the target reminder method may be a visual display or a voice prompt.

[0153] Specifically, the personnel status data is analyzed and sorted, including the operator's concentration, fatigue, emotional state, etc., to obtain the analysis results of the personnel status data. According to the analysis results of the personnel status data, a suitable target reminder method (such as visual display, voice prompt, vibration, etc.) is determined to convey the preparatory status data of the next work task to the operator, thereby reminding the operator to prepare for the next work task in advance. Based on the selected target reminder method, the preparatory status data is displayed to the operator in an appropriate form, so that the operator can receive and understand the preparatory status data in a timely manner.

[0154] For example, if the personnel status data indicates that the operator is not in an abnormal state, and the execution mode of the next operation task is human-dominated, then a visual display method can be selected to display the relevant preparation status data on the operation interface to attract the operator's attention, so that the operator can obtain the necessary information in time and dominate the execution of the next operation task.

[0155] If the personnel status data indicates that the operator is in an abnormal state (for example, the operator's task prediction status data is not predicted or fatigued), and the execution mode of the next task is human-dominated. Then the operator can be stimulated by a strong reminder (such as voice prompt) to draw his attention to the preparatory status data. The operator can readjust his attention and concentration to dominate the execution of the next task.

[0156] In some embodiments, augmented reality technology can be used to display task objectives, preparation requirements and other preparation status data to the operating object by superimposing text, images, videos and other forms on the real scene. The operating object obtains the preparation status data in the AR interface through an augmented reality device (such as AR glasses), which helps it to make necessary preparations and adjustments to ensure the smooth execution of the task.

[0157] In other embodiments, virtual reality technology is used to create virtual objects, text, audio and other elements in a virtual environment to present the preparation state data. The operating object can obtain the preparation state data in the VR interface through a virtual reality device (such as a VR helmet).

[0158] In the above implementation, a target reminder method corresponding to the personnel status data is determined, and the preparatory status data is displayed in a target reminder manner to remind the operating object to make advance preparations for the next work task, helping the operating object to avoid missing key steps or required resources and reducing errors and mistakes caused by lack of preparation.

[0159] In some embodiments, the method may further include: performing decision control based on status data fed back from the human-machine collaborative process, so as to execute the next work task in a manner that matches the execution of the next work task of the current work task.

[0160] The execution mode is any one of the human-machine collaborative mode, human-dominated mode, and machine-dominated mode. The human-machine collaborative mode can be a mode in which humans and machines cooperate with each other and work together in the human-machine collaborative process. The human-dominated mode can be a mode in which humans play a leading role in the decision-making and control process, and the way the machine performs tasks depends on human instructions. The machine-dominated mode can be a mode in which machines play a leading role in decision-making and control in the entire human-machine collaborative process, while humans mainly play an auxiliary role.

[0161] Specifically, in the human-machine collaboration process, the status data fed back by the human-machine collaboration process is used to evaluate the situation of the next task, so as to determine the execution mode that matches the next task. If the next task requires a high degree of human-machine collaboration, then the human-machine collaboration mode is selected. If the next task requires guidance and decision-making by the operating object, then the human-dominated mode is selected. If the next task can be executed autonomously by the operating device, then the machine-dominated mode is selected. According to the selected execution mode, a corresponding execution plan is formulated. In the human-machine collaboration mode, the operating object and the operating device need to work together to formulate a suitable execution strategy, while taking into account the advantages and limitations of both parties. In the human-dominated mode, the operating object needs to formulate an execution plan based on the status data and professional knowledge, and convey the task instructions of the execution plan to the operating device. In the machine-dominated mode, the operating device needs to autonomously formulate an execution plan based on the built-in algorithm and rule system without the intervention of the operating object. After determining the execution plan, the next task is executed according to the formulated execution plan.

[0162] In some embodiments, according to the selected execution mode, an augmented reality device is used to superimpose appropriate virtual information in the real scene where the operating object is located to help the operating object better perform the task. For example, in a human-machine collaborative mode, the operating status and task progress of the operating device can be superimposed and displayed in the real scene so that the operating object can adjust the strategy in time. In a human-dominated mode, task execution instructions and operation processes can be superimposed and displayed in the real scene to help the operating object better complete the task. In a machine-dominated mode, the execution status and results of the operating device can be superimposed and displayed in the real scene so that the operating object can understand the operating status and data of the operating device in time.

[0163] In other embodiments, a virtual reality device is used to create a corresponding virtual environment according to the selected execution mode. In the virtual environment, the interaction process between the operating object and the operating device can be simulated, and information such as task execution instructions, operation procedures, and machine status can be displayed. For example, in a human-led mode, the operating object can perform the next operation task according to the information and guidance displayed in the virtual environment.

[0164] In the above implementation, decision control is performed based on the status data fed back from the human-machine collaborative process so that the next work task is executed in a manner that matches the next work task of the current work task. This can effectively balance the decision-making problems of humans and machines, thereby improving the efficiency and safety of the collaborative process.

[0165] The implementation method of this specification also provides a human-machine collaborative state feedback method based on virtual-real fusion. Fig.10a , human factors data include physiological signal data (such as skin electrodermal signal data, heart rate signal data), eye movement signal data, and EEG signal data. Personnel status can be identified based on physiological signal data, eye movement signal data, and EEG signal data to obtain personnel status data. For example, personnel status data may include task prediction status data. It is understandable that early warning feedback can be given to the current work task based on the personnel status data. The current work task corresponds to the current task area range, and the next work task corresponds to the next task area range. For example, please refer to Fig.10b The human-machine collaborative state feedback method based on virtual-reality fusion may include the following steps:

[0166] S1002, obtaining human factor data of the operating object in the human-machine collaborative process of the current operation task, and an operation screen image obtained by shooting the human-machine collaborative process.

[0167] The operation screen image at least includes setting parameters of the operation device and the operation environment.

[0168] S1004. Perform load status identification based on the EEG signal data to obtain load status data of the operation object.

[0169] S1006: Perform emotional state recognition based on the electrodermal signal data and the heart rate signal data to obtain emotional state data of the operation object.

[0170] S1008. Issue an early warning on the working status of the operating object according to the load status data and / or the emotional status data.

[0171] S1010, determining a gaze point scanning path of the operating object based on the eye movement signal data.

[0172] S1012: Determine a change in the gaze position of the operation object based on the gaze point scanning path.

[0173] The gaze position change is used to describe how the gaze position of the operating object changes with the execution time of the current task.

[0174] S1014: Based on the change of gaze position, determine whether at least part of the gaze resource of the operation object is transferred from the current task area range to the next task area range, and obtain task prediction state data.

[0175] S1016: Input the operation screen image into the target state recognition model to recognize the human-machine collaborative state, and obtain the target collaborative state data.

[0176] S1018. Determine the preparation state data of the next work task of the current work task according to the target collaborative state data.

[0177] The preparation state data is used to represent the situation where the operation object needs to make advance preparations for the next operation task.

[0178] S1020: Determine a target reminder method corresponding to the task prediction state data based on the preparation state data and the task prediction state data.

[0179] S1022. Display the preparation status data in a target reminder manner to remind the operation object to make advance preparations for the next operation task.

[0180] This specification provides a human-machine collaborative state feedback device 1100 based on virtual-real fusion. Fig.11 The human-machine collaborative state feedback device 1100 based on virtual-reality fusion includes: a task data acquisition module 1110, a human-machine collaborative state recognition module 1120, a personnel state recognition module 1130, and a state feedback module 1140.

[0181] The task data acquisition module 1110 is used to acquire the human factor data of the operation object in the human-machine collaboration process of the current operation task, and the operation screen image obtained by shooting the human-machine collaboration process, wherein the operation screen image at least includes the setting parameters of the operation device and the operation environment;

[0182] A human-machine collaborative state recognition module 1120 is used to perform human-machine collaborative state recognition based on the operation screen image to obtain target collaborative state data corresponding to the current operation task;

[0183] A personnel status identification module 1130, configured to identify the personnel status based on the human factor data and obtain the personnel status data of the operation object;

[0184] The status feedback module 1140 is used to provide status feedback on the human-machine collaboration process according to the target collaboration status data and the personnel status data.

[0185] In some embodiments, the human factors data includes eye movement signal data, and the personnel status data includes task prediction status data; the personnel status identification module is also used to identify the gaze position based on the eye movement signal data to obtain the gaze position change of the operating object; wherein the gaze position change is used to describe the change of the gaze position of the operating object with the execution time of the current work task; based on the gaze position change, it is judged whether the operating object predicts the next work task of the current work task to obtain the task prediction status data.

[0186] In some embodiments, the human-machine collaborative state feedback device based on virtual-reality fusion also includes: a gaze position change module, which is used to determine the gaze point scanning path of the operating object based on the eye movement signal data; and determine the gaze position change of the operating object based on the gaze point scanning path.

[0187] In some embodiments, the current work task corresponds to the current task area range, and the next work task corresponds to the next task area range; the personnel status identification module is also used to determine whether at least part of the gaze resources of the operation object are transferred from the current task area range to the next task area range based on the change in the gaze position, and obtain the task prediction status data.

[0188] In some embodiments, the human factors data includes at least one of EEG signal data, galvanic skin signal data, and heart rate signal data; the human-machine collaborative state feedback device based on virtual-reality fusion also includes: a working state warning module, which is used to identify the load state based on the EEG signal data to obtain the load state data of the operating object; to identify the emotional state based on the galvanic skin signal data and the heart rate signal data to obtain the emotional state data of the operating object; and to warn the working state of the operating object based on the load state data and / or the emotional state data.

[0189] In some embodiments, the human-machine collaboration state recognition module is also used to perform human-machine collaboration state recognition based on the work screen image to obtain current collaboration state data of the current work task as the target collaboration state data; or to perform human-machine collaboration state recognition based on the work screen image to obtain current collaboration state data of the current work task, and determine the target collaboration state data of the adjacent work task of the current work task based on the current collaboration state data.

[0190] In some embodiments, the human-machine collaborative state recognition module is further used to input the operation screen image into a target state recognition model to recognize the human-machine collaborative state and obtain the target collaborative state data.

[0191] In some embodiments, the target state recognition model is trained by: obtaining a sample screen image obtained by photographing a historical operation process; wherein the label of the sample screen image includes a labeling task identifier, labeling action data, and a labeling tool category corresponding to the labeling action data; inputting the sample screen image into the initial state recognition model for prediction to obtain a predicted task identifier, predicted action data, and a predicted tool category corresponding to the predicted action data; updating the initial state recognition model based on the labeling task identifier, the labeling action data, the labeling tool category, the predicted task identifier, the predicted action data, and the predicted tool category to obtain the target state recognition model.

[0192] In some embodiments, the label of the sample screen image is determined in the following manner: the historical operation process is decomposed into tasks to obtain a number of sample operation tasks sorted in time; the labeling task identifier is determined based on the task identifier of the operation task; the actions involved in the sample operation task are encoded to obtain the labeling action data; the tools used to perform the actions involved in the sample operation task are encoded to obtain the labeling tool category.

[0193] In some embodiments, the status feedback module is also used to determine the preparatory status data of the next work task of the current work task based on the target collaborative status data; wherein the preparatory status data is used to characterize the situation where the operating object needs to make advance preparations for the next work task; and to provide status feedback on the human-machine collaborative process based on the preparatory status data and the personnel status data.

[0194] In some embodiments, the status feedback module is further used to determine a target reminder method corresponding to the personnel status data; and to display the preparatory status data in the target reminder method to remind the operating object to make advance preparations for the next work task.

[0195] In some embodiments, the human-machine collaborative state feedback device based on virtual-reality fusion also includes: a decision control module, which is used to perform decision control based on the state data fed back for the human-machine collaborative process, so as to execute the next work task in an execution mode that matches the next work task of the current work task, wherein the execution mode is any one of a human-machine collaborative mode, a human-dominated mode, and a machine-dominated mode.

[0196] For the specific description of the human-machine collaborative state feedback device based on virtual-reality fusion, please refer to the description of the human-machine collaborative state feedback method based on virtual-reality fusion above, which will not be repeated here.

[0197] In some embodiments, an electronic device is provided, whose internal structure diagram can be as follows: Fig.12 As shown. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a human-computer collaborative state feedback method based on virtual-real fusion is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0198] Those skilled in the art will understand that Fig.12 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme disclosed in this specification, and does not constitute a limitation on the electronic device to which the scheme disclosed in this specification is applied. Specifically, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0199] In some embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method steps in the above embodiments when executing the computer program.

[0200] An embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.

[0201] One embodiment of the present specification provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of the method of any of the above embodiments.

[0202] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

Claims

1. A human-machine collaborative state feedback method based on virtual-reality fusion, characterized in that: The method comprises: Acquire human factor data of the operating object in the human-machine collaboration process of the current operation task, and an operation screen image obtained by photographing the human-machine collaboration process, wherein the operation screen image at least includes setting parameters of the operating device and the operating environment; Performing human-machine collaborative state recognition based on the operation screen image to obtain target collaborative state data corresponding to the current operation task; Performing personnel status identification based on the human factor data to obtain personnel status data of the operation object; Providing status feedback on the human-machine collaboration process according to the target collaboration status data and the personnel status data; The human factor data includes eye movement signal data, and the personnel status data includes task prediction status data; the personnel status identification based on the human factor data to obtain the personnel status data of the operation object includes: The gaze position is identified based on the eye movement signal data to obtain the gaze position change of the operation object; wherein the gaze position change is used to describe the change of the gaze position of the operation object with the execution time of the current task; Based on the change of the gaze position, determining whether the operating object predicts the next operation task of the current operation task, and obtaining the task prediction state data; The current operation task corresponds to the current task area range, and the next operation task corresponds to the next task area range; judging whether the operation object predicts the next operation task of the current operation task based on the change of the gaze position, and obtaining the task prediction state data, includes: Based on the gaze position change, it is determined whether at least part of the gaze resources of the operation object are transferred from the current task area range to the next task area range, so as to obtain the task prediction state data.

2. The method according to claim 1, characterized in that The identifying of the gaze position based on the eye movement signal data to obtain the gaze position change of the operation object includes: Determine a gaze point scanning path of the operating object based on the eye movement signal data; A change in the gaze position of the operating object is determined based on the gaze point scanning path.

3. The method according to claim 1, characterized in that The human factor data includes at least one of electroencephalogram signal data, skin electrical signal data, and heart rate signal data; the method further includes: Perform load state identification based on the EEG signal data to obtain load state data of the operation object; Performing emotional state recognition based on the electrodermal signal data and the heart rate signal data to obtain emotional state data of the operation object; An early warning is issued for the working state of the operating object according to the load state data and / or the emotional state data.

4. The method according to claim 1, characterized in that: The human-machine collaborative state recognition is performed based on the operation screen image to obtain the target collaborative state data corresponding to the current operation task, including: Performing human-machine collaborative state recognition according to the operation screen image to obtain current collaborative state data of the current operation task as the target collaborative state data; or The human-machine collaboration state is identified based on the work screen image to obtain the current collaboration state data of the current work task, and the target collaboration state data of the adjacent work task of the current work task is determined based on the current collaboration state data.

5. The method according to claim 1, characterized in that The human-machine collaborative state recognition is performed based on the operation screen image to obtain the target collaborative state data corresponding to the current operation task, including: The operation screen image is input into a target state recognition model to recognize the human-machine collaborative state, and the target collaborative state data is obtained.

6. The method according to claim 5, characterized in that The target state recognition model is obtained by training in the following manner: Acquire a sample screen image obtained by photographing a historical operation process; wherein the label of the sample screen image includes a labeling task identifier, labeling action data, and a labeling tool category corresponding to the labeling action data; Inputting the sample screen image into the initial state recognition model for prediction, obtaining a prediction task identifier, prediction action data, and a prediction tool category corresponding to the prediction action data; The initial state recognition model is updated based on the labeling task identifier, the labeling action data, the labeling tool category, the prediction task identifier, the prediction action data and the prediction tool category to obtain the target state recognition model.

7. The method according to claim 6, characterized in that The label of the sample screen image is determined in the following manner: Decomposing the historical operation process into tasks to obtain a number of sample operation tasks sorted by time; Determine the marking task identifier based on the task identifier of the operation task; Encoding the actions involved in the sample operation task to obtain the labeled action data; The marking tool category is obtained by encoding the tool used when executing the action involved in the sample job task.

8. The method according to any one of claims 1-3, 5-7, characterized in that: The providing status feedback of the human-machine collaborative process according to the target collaborative status data and the personnel status data includes: Determining the preparatory state data of the next operation task of the current operation task according to the target collaborative state data; wherein the preparatory state data is used to characterize the situation that the operation object needs to make advance preparations for the next operation task; The human-machine collaborative process is fed back with status based on the preparation status data and the personnel status data.

9. The method according to claim 8, characterized in that The providing status feedback of the human-machine collaborative process based on the preparation status data and the personnel status data includes: Determine a target reminder method corresponding to the personnel status data; The preparation status data is displayed in the target reminder manner to remind the operation object to prepare in advance for the next operation task.

10. The method according to claim 1, characterized in that The method further comprises: Decision control is performed based on the status data fed back from the human-machine collaborative process to execute the next work task in an execution mode that matches the next work task of the current work task, wherein the execution mode is any one of a human-machine collaborative mode, a human-dominated mode, and a machine-dominated mode.

11. A human-machine collaborative state feedback device based on virtual-reality fusion, characterized in that: The device comprises: A task data acquisition module, used to acquire human factor data of the operation object in the human-machine collaboration process of the current operation task, and an operation screen image obtained by photographing the human-machine collaboration process, wherein the operation screen image at least includes setting parameters of the operation device and the operation environment; A human-machine collaborative state recognition module is used to perform human-machine collaborative state recognition based on the operation screen image to obtain target collaborative state data corresponding to the current operation task; A personnel status identification module, used to identify the personnel status based on the human factor data to obtain the personnel status data of the operation object; A status feedback module, used for providing status feedback on the human-machine collaboration process according to the target collaboration status data and the personnel status data; The human factor data includes eye movement signal data, and the personnel status data includes task prediction status data; the personnel status identification based on the human factor data to obtain the personnel status data of the operation object includes: The gaze position is identified based on the eye movement signal data to obtain the gaze position change of the operation object; wherein the gaze position change is used to describe the change of the gaze position of the operation object with the execution time of the current task; Based on the change of the gaze position, determining whether the operating object predicts the next operation task of the current operation task, and obtaining the task prediction state data; The current operation task corresponds to the current task area range, and the next operation task corresponds to the next task area range; judging whether the operation object predicts the next operation task of the current operation task based on the change of the gaze position, and obtaining the task prediction state data, includes: Based on the gaze position change, it is determined whether at least part of the gaze resources of the operation object are transferred from the current task area range to the next task area range, so as to obtain the task prediction state data.

12. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images