Target locking method

Abstract

The invention relates to the technical field of intelligent interaction, and discloses a target locking method, which enables a system to intelligently highlight a to-be-selected target by accurately capturing the sight movement of a user on a human-computer interaction interface, enhances the real-time performance and intuition of human-computer interaction, is helpful to quickly screen out a possible operation target, and improves the user experience. The user is prevented from blindly searching in numerous interface elements, and the interaction efficiency is improved. And finally, a final locking target is further determined in combination with a user operation instruction, the subjective intention of the user is fully considered, and effective fusion of man-machine information is realized. Therefore, by implementing the method and the device, the target is preliminarily positioned by using the eye movement information, and confirmation is performed through active operation of the user, so that misoperation possibly caused by relying on eye movement only is avoided, target locking is more accurate and reliable, functions of discrete target selection, multi-target centralized operation and the like are realized, and the user experience is improved. And the flexibility and the precision of the man-machine cooperation system in a multi-target processing scene are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent interaction technology, and in particular to a target locking method. Background Art

[0002] Eye movement is one of the common methods to study human information processing and behavior. It can provide objective and quantitative evidence for the user's visual and (explicit) attention process, and can intuitively reflect the user's current attention range, interest area, information retrieval process, browsing habits, etc. Compared with interactive devices such as mouse, it is closer to what you see is what you get.

[0003] Current eye tracking applications can be roughly divided into two types: diagnostic / evaluation and interactive. In diagnostic / evaluation applications, the application records eye movement data under a specified stimulation paradigm, which is then used to analyze the user's corresponding attention patterns. In interactive applications, eye trackers / gaze tracking devices can be used as an input device by many visually mediated applications.

[0004] Diagnostic / evaluative eye tracking technology usually requires the use of an eye tracker in an unnoticeable state, and generally does not need to change according to the viewer's goals. At this time, the eye tracker / gaze tracking device only needs to simply record the subject's eye movements so that the viewer's gaze can be analyzed offline during the experiment. Based on the collected eye movement information and combined with the experimental design, the subject's attention points, attention preferences, browsing habits, etc. can be analyzed. For example, to test whether the design of certain interactive interfaces is reasonable, the changes in the subject's gaze during actual operation can be tested, such as whether the subject always scans the interface to repeatedly search for certain operations or instructions, and whether a lot of time is spent looking for certain settings. Diagnostic / evaluative eye tracking technology can be used in (but not limited to) fields such as psychology / psychophysics, marketing / advertising, human factors engineering, and ergonomics.

[0005] For interactive systems equipped with eye trackers / gaze tracking devices, it is generally expected that the system can respond or interact with the user's gaze changes. Therefore, the interactive system needs to respond to the user's gaze behavior in some way. This interaction method can be divided into two categories, selective and gaze-dependent. The former can generally be used as a substitute for mouse input, such as activating certain options or selecting certain operating components based on the gaze position, such as enlarging the operation buttons on the interactive interface or expanding the drop-down list. However, this solution is prone to false touches, or in other words, it is impossible to distinguish between gaze selection behavior and normal gaze behavior, and requires additional input signals and fixed time limits for confirmation. In gaze-dependent interactions, the system needs to change the display mode of the interactive interface according to the gaze behavior to achieve acceleration, optimization, and other purposes. For example, in games, the corresponding interval is rendered with higher definition based on the user's gaze position, while the rendering requirements for the surrounding area are reduced. Or in augmented reality (AR), according to the user's position, combined with the target detection model and prior knowledge, the corresponding object name, the operator's operating authority, and whether the current equipment needs to be checked are prompted. For example, AR is used to improve the safety of nuclear power plant operators operating equipment, and to inform operators whether there are equipment that needs to be checked, the order of checking different equipment, and the operations that can be performed. As far as the system is concerned, this type of eye movement interaction can be used as one of a set of multimodal input strategies.

[0006] In existing applications or methods that use eye movement for interactive operations, the accuracy of the eye movement signals collected by the eye tracker / gaze tracking device is mainly relied on, and the errors existing in the eye tracker / gaze tracking device itself are ignored. In selective operations, the target to be selected often needs to have sufficient spatial size to ensure that the gaze position obtained by the device falls within the target interval; at the same time, the existing interaction logic either ignores the unique problem of turning stones into gold in eye movement interaction (that is, the system cannot distinguish between the user's operation intention and random gaze behavior), or adopts eye movement strategies that require specific training; and in gaze-dependent interaction schemes, the directionality of eye movement itself is mainly used to perform local special rendering operations (such as improving the resolution and rendering effect of the corresponding position of the gaze), or as auxiliary information to indicate other operations. For example, in the patent "Robot-assisted hand-eye coordination training system based on eye movement smooth tracking and guiding force field", the gaze signal is mainly used to generate auxiliary torque signals for the control handle to prompt the user's hand coordination movement and manipulation feedback. At the same time, in the existing technology, the interactive behavior combined with eye movements is discrete and independent, and cannot fully utilize the existing artificial intelligence (AI) technology to achieve the performance advantages of the human-computer collaborative system of one person and multiple machines. Summary of the invention

[0007] In view of this, the present invention provides a target locking method to solve the problem that the existing method of using eye movement to perform interactive operations cannot quickly lock the target.

[0008] In a first aspect, the present invention provides a target locking method for a human-machine collaborative system, wherein the human-machine collaborative system includes a human-machine interactive interface; the method includes:

[0009] Acquire a sight data set, where the sight data set is used to characterize the sight movement of the user's eyes in the human-computer interaction interface; control the human-computer interaction interface to highlight a target to be selected based on the sight data set, where the target to be selected is one or more elements in the human-computer interaction interface; respond to a first operation instruction from the user, and control the human-computer interaction interface to lock the target to be selected based on the first operation instruction from the user.

[0010] The target locking method provided by the present invention can accurately capture the user's line of sight action in the human-computer interaction interface by acquiring a line of sight data set. Furthermore, the line of sight data is used to enable the system to intelligently highlight the target to be selected, effectively attract the user's attention and give visual feedback, so that the user can intuitively understand the system's response to its line of sight focus area, enhance the real-time and intuitiveness of human-computer interaction, help to quickly screen out possible operation targets, avoid users from blindly searching among many interface elements, and improve interaction efficiency. Finally, the final locked target is further determined in combination with the user's operation instructions, fully considering the user's subjective intention, and realizing the effective fusion of information from both the human and the machine. Therefore, by implementing the present invention, the eye movement information is used to preliminarily locate the target, and the user's active operation is used to confirm, avoiding the possible misoperation caused by relying solely on eye movement, making the target locking more accurate and reliable, thereby realizing functions such as discrete target selection and centralized operation, and improving the flexibility and accuracy of the human-computer collaborative system in multi-target processing scenarios.

[0011] In an optional implementation, controlling the human-computer interaction interface to highlight the target to be selected based on the line of sight data set includes:

[0012] Based on the line of sight data set, the initial interaction coordinates of the user's line of sight and the human-computer interaction interface are calculated; the initial interaction coordinates are smoothed to obtain the target interaction coordinates; based on the target interaction coordinates, the line of sight coordinate interval is determined in the human-computer interaction interface; based on the line of sight coordinate interval, the human-computer interaction interface is controlled to highlight the target to be selected; or, based on the target interaction coordinates and the line of sight coordinate interval, the human-computer interaction interface is controlled to highlight the target to be selected.

[0013] The target locking method provided by the present invention can accurately determine the starting position of the user's line of sight in the human-computer interaction interface by calculating the initial interaction coordinates of the user's line of sight and the human-computer interaction interface. Furthermore, coordinate smoothing processing can effectively reduce coordinate fluctuations caused by factors such as eye movement jitter and equipment errors, and obtain more stable and reliable target interaction coordinates. Furthermore, the line of sight coordinate interval is determined based on the target interaction coordinates, which provides a clear area definition standard for judging whether the target is within the user's attention range. Furthermore, by determining the target interaction coordinates or the line of sight coordinate interval, possible candidates can be more effectively screened out, reducing unnecessary target search ranges, and improving the speed and efficiency of target locking, especially in a multi-target interface, it is possible to quickly focus on areas that the user may be interested in, and realize rapid target positioning in regional inspection / monitoring functions.

[0014] In an optional implementation, based on the sight line coordinate interval, controlling the human-computer interaction interface to highlight the target to be selected includes:

[0015] Determine whether the sight line coordinate interval overlaps with the target interval and determine the target to be selected; control the human-computer interaction interface to highlight the target to be selected.

[0016] The target locking method provided by the present invention can accurately determine the target to be selected that the user may be concerned about by judging the overlap between the sight coordinate interval and the target interval, making full use of the directivity of eye movement, and can quickly screen out the target related to the user's sight from many interface elements, avoiding the one-by-one investigation of the entire interface elements, greatly improving the speed of target locking, realizing the rapid target positioning in the discrete target selection function, and also providing a clear object for subsequent operations on specific targets. Further, the determined target to be selected is highlighted, which enhances the visualization effect of human-computer interaction, allowing the user to clearly know that the system has identified its possible operation target, and also facilitates the user to confirm whether the target is correct again, improving the accuracy and reliability of the interaction, helping the user to quickly confirm the target and perform subsequent operations in a multi-target scene, and improving the usability and interaction efficiency of the human-computer collaborative system. At the same time, the target locking scheme based on sight can also avoid the problem of moving the mouse pointer to the target to be selected when the target is found, wherein the mouse pointer and other on-screen markers sometimes take a long time to search, and in this process, the dynamic target state may have changed.

[0017] In an optional implementation, the human-machine collaborative system is connected to the unmanned equipment; based on the target interaction coordinates and the sight line coordinate interval, the human-machine interaction interface is controlled to highlight the target to be selected, and further includes:

[0018] Obtain the horizontal field of view and vertical field of view of the unmanned equipment; calculate the horizontal deflection angle and vertical deflection angle of the unmanned equipment based on the target interaction coordinates, the horizontal field of view and the vertical field of view; send the horizontal deflection angle and the vertical deflection angle to the unmanned equipment, so that the unmanned equipment is adjusted based on the horizontal deflection angle and the vertical deflection angle; when the unmanned equipment image sent by the adjusted unmanned equipment is received; based on the line of sight coordinate interval, control the unmanned equipment image to be projected onto the human-computer interaction interface and highlight it.

[0019] The target locking method provided by the present invention realizes the line of sight correlation mapping between the human-computer interaction interface and the unmanned equipment by obtaining the horizontal field of view angle and the vertical field of view angle of the unmanned equipment, and calculating the horizontal deflection angle and the vertical deflection angle of the unmanned equipment based on the target interaction coordinates. Further, the calculated deflection angle is sent to the unmanned equipment, so that it is adjusted based on the deflection angle, and after receiving the adjusted unmanned equipment screen, it is highlighted on the human-computer interaction interface based on the line of sight coordinate interval, thereby realizing the seamless connection between the unmanned equipment viewing angle and the human-computer interaction interface. Therefore, by implementing the present invention, the user can accurately control the direction of the unmanned equipment through the line of sight operation on the human-computer interaction interface, and expand the human-computer collaboration to the interaction field between humans and unmanned equipment, providing an intuitive and convenient way to realize remote control of unmanned equipment, and improving the accuracy and efficiency of unmanned equipment control. Furthermore, the application of the target selection function in the specified area in the unmanned equipment scene is also realized, which improves the target locking and operation execution capabilities of the human-computer collaboration system in cross-device interaction.

[0020] In an optional embodiment, the method further includes:

[0021] Respond to the user's second operation instruction; control the human-computer interaction interface to highlight the selected target based on the user's second operation instruction and the target interaction coordinates; or, control the human-computer interaction interface to highlight the selected target based on the user's second operation instruction and the line of sight coordinate interval; or, control the human-computer interaction interface to highlight the selected target based on the user's second operation instruction, the target interaction coordinates and the line of sight coordinate interval.

[0022] The target locking method provided by the present invention increases the flexibility and diversity of human-computer interaction by combining the user's second operation instruction to control the human-computer interaction interface again to highlight the target to be selected, realizes more sophisticated and personalized human-computer collaborative operation, and improves the system's responsiveness to the user's complex operation intentions.

[0023] In an optional implementation, controlling the human-computer interaction interface to highlight the target to be selected based on the user's second operation instruction and the target interaction coordinates includes:

[0024] Using the target interaction coordinates, the human-computer interaction interface is controlled to generate a first target in the candidate area; based on the user's second operation instruction, the first target in the candidate area is adjusted to obtain a second target in the candidate area; and the human-computer interaction interface is controlled to highlight the second target in the candidate area.

[0025] The target locking method provided by the present invention utilizes the target interactive coordinates to control the human-computer interaction interface to generate a first candidate area target, and adjusts it based on the user's second operation instruction to obtain a second candidate area target, and then highlights the target, thereby realizing dynamic adjustment of the area in the regional inspection / monitoring function, improving the accuracy and flexibility of area selection, and enhancing the target locking and operation execution capabilities of the human-computer collaborative system in the regional operation scenario.

[0026] In an optional implementation, controlling the human-computer interaction interface to highlight the target to be selected based on the user's second operation instruction and the line of sight coordinate interval further includes:

[0027] When there is a deviation between the sight line coordinate interval and the target interval, a target to be selected is determined on the human-computer interaction interface based on the user's second operation instruction and the sight line coordinate interval; and the human-computer interaction interface is controlled to highlight the target in the second candidate area.

[0028] The target locking method provided by the present invention determines and highlights the target to be selected on the human-computer interaction interface based on the user's second operation instruction and the line of sight coordinate interval when there is a deviation in the line of sight coordinate interval, taking into account the possible errors in the line of sight data. At the same time, the inaccurate target determination that may be caused by errors is corrected through the user's operation instructions, thereby improving the accuracy and reliability of target locking. Even when the line of sight data is not very accurate, the correct target can be quickly determined through the user's active intervention, avoiding misjudgment or loss of the target due to errors, and improving the target locking capability of the human-computer collaborative system in complex environments or equipment errors.

[0029] In an optional implementation, controlling the human-computer interaction interface to highlight the target to be selected based on the user's second operation instruction, the target interaction coordinates, and the sight line coordinate interval includes:

[0030] Based on the target interaction coordinates and the user's second operation instruction, the human-computer interaction interface is controlled to be enlarged; based on the line of sight coordinate interval, the enlarged human-computer interaction interface is controlled to highlight the target to be selected.

[0031] The target locking method provided by the present invention uses the target interaction coordinates to perform a zoom operation on the human-computer interaction interface, which allows the user to view the target details more clearly, and is particularly suitable for situations where the target is small or the interface information is complex. Furthermore, the target to be selected is highlighted based on the line of sight coordinate interval in the zoomed-in interface, further emphasizing the key area of ​​user attention, and the target can be quickly located even in the complex interface after zooming in, avoiding the user from getting lost in the target position after zooming in, making the target locking still efficient and accurate under the zooming operation, and improving the target locking capability and interactive experience of the human-computer collaborative system under different display requirements.

[0032] In an optional implementation, the method further includes: determining and executing a target execution task of the target to be selected based on a preset system agreed instruction.

[0033] The target locking method provided by the present invention determines and executes the target execution task of the selected target based on the preset system agreed instructions, realizing the automation and standardization of operations in the human-machine collaborative system. At the same time, the user does not need to perform cumbersome manual operation settings, which improves the efficiency and speed of operation execution, thereby improving the task execution capability and production efficiency of the human-machine collaborative system in specific task scenarios.

[0034] In an optional embodiment, the method further includes:

[0035] Based on the line of sight data set, the human-computer interaction interface is controlled to remove the locked candidate targets and the candidate targets corresponding to the target execution tasks.

[0036] The target locking method provided by the present invention controls the human-computer interaction interface to remove the locked candidate targets and the candidate targets corresponding to the target execution tasks through the line of sight data set, thereby achieving the effect of undoing the operation to release machine resources and improving the utilization rate of system resources. At the same time, it also makes the human-computer interaction more flexible and convenient. The user can correct his or her operation selection at any time to avoid resource waste and operation errors caused by wrong selection, thereby improving the fault tolerance and resource management capabilities of the human-computer collaborative system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 is a flow chart of a target locking method according to an embodiment of the present invention;

[0039] Figure 2is a flow chart of another target locking method according to an embodiment of the present invention;

[0040] Figure 3 is a flow chart of a method for rapid multi-target locking in human-machine collaboration by integrating eye movements according to an embodiment of the present invention;

[0041] Figure 4 is an example diagram for explaining an implementation of a method for rapid multi-target locking by human-machine collaboration with fusion of eye movements according to an embodiment of the present invention;

[0042] Figure 5 is a structural block diagram of a target locking device according to an embodiment of the present invention;

[0043] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0045] The stability and accuracy of eye trackers / gaze tracking devices are far inferior to most manual input devices. Even without considering environmental interference, the eyeball's eye movement behavior and human scanning behavior will cause the line of sight to change within a certain range. On the other hand, in a natural state, the information provided by eye movement signals is limited, and is limited to providing the directional information of the user's current line of sight. Furthermore, it can be considered to provide information about the user's current attention position. The advantage is low load, smoother, more natural, and similar to unconscious operation. Correspondingly, if unnatural operations are taken, additional attention time is required or efforts are made to control the line of sight to run along a specific trajectory, which will reduce the user's attention span and increase the operation load, resulting in blindness, and to a certain extent reduce the user's observation decision-making ability and performance level.

[0046] In terms of machine intelligence, the current level of target detection, tracking or segmentation in video streams is limited by the model scale and model intelligence level. At the same time, there is a lack of understanding of world knowledge, or prior knowledge, and there are phenomena such as missed target recognition or segmentation, and interrupted and discontinuous detection in continuous video streams. Therefore, it is necessary to make judgments and decisions based on human subjective intentions and assist each other to improve the performance level of human-machine collaboration.

[0047] According to an embodiment of the present invention, a target locking method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] In this embodiment, a target locking method is provided for a human-machine collaborative system, the human-machine collaborative system including a human-machine interactive interface; Figure 1 is a flow chart of a target locking method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0049] Step S101, obtaining a sight line dataset.

[0050] Among them, the line of sight data set is used to characterize the line of sight movement of the user's eyes in the human-computer interaction interface, which may include the line of sight angle of the user's eyes, the coordinates of the user's eyes on the human-computer interaction interface, the line of sight angle of the user's eyes, the deflection and displacement of the user's head, the current distance between the eyes and the interaction interface, the direction of the user's eyes, etc.

[0051] Furthermore, the user's binocular line of sight angle represents the deviation angle of each of the user's binocular lines of sight relative to the front; the user's head deflection and displacement represent the horizontal and forward and backward movement distance of the head relative to the initial state, and the angle of head rotation during the line of sight tracking process.

[0052] Specifically, the corresponding sight data set can be obtained by using an eye tracker or sight tracking device: First, the eye tracker or sight tracking device is used to collect information such as the user's binocular sight angle, head deflection and displacement by emitting light to the user's eyes and receiving reflected light or detecting electrical signals generated by eye movement. Then, according to the preset spatial geometric model and calibration parameters inside the device, the coordinates of the intersection of the sight and the interactive interface are calculated. These coordinate information and data such as the sight movement trajectory together constitute the sight data set.

[0053] For example, an eye tracker continuously monitors the user's eye status at a certain frequency per second (such as 30Hz-120Hz), and each piece of collected data is processed and recorded in the line of sight dataset.

[0054] Among them, the eye tracker or gaze tracking device can be worn by the user or attached to the display interface of the human-computer collaborative system. This embodiment does not make any specific limitations on this, as long as it ensures that the user's gaze can be matched with the interactive interface.

[0055] Furthermore, other data sets can be obtained, such as finger movement data, by wearing a special data acquisition device on the finger. Among them, the data acquisition device can be integrated with micro sensors, such as built-in high-precision accelerometers, gyroscopes, and pressure sensors. The accelerometer can accurately measure the acceleration changes of the finger in all directions to track the speed and direction changes of the finger movement; the gyroscope is responsible for recording the rotation angle information of the finger in space to assist in judging the posture changes of the finger; and the pressure sensor can sense the pressure between the finger and the contacted object, which helps to analyze the strength of the finger operation.

[0056] In this embodiment, taking the user's gaze on the human-computer interaction interface as an example, the human-computer interaction interface is implemented based on the human-computer interaction interface with gaze direction-interaction interface positioning as the core.

[0057] Step S102: controlling the human-computer interaction interface to highlight the target to be selected based on the line of sight data set.

[0058] Among them, the target to be selected can be specific orientation information, such as spatial orientation (the unmanned equipment can be moved toward a specified orientation or the camera equipment can adjust the lens toward a specified orientation) and a designated area (the unmanned equipment can be specified to conduct a coverage search for the area or select all target entities or specific types of target entities in the area).

[0059] Furthermore, the target to be selected can also be one or more elements existing in the human-computer interaction interface, such as an entity obtained through target detection or tracking model detection in the image transmitted back by the unmanned equipment, an area defined by the user in the image of the unmanned equipment (the line of sight interval can be regarded as the most basic area), image windows transmitted back by multiple unmanned equipment, and menus and buttons existing in the interaction interface itself.

[0060] Specifically, combined with the line of sight dataset, based on the directionality of gaze and with the gaze direction-interaction interface positioning as the core, the human-computer interaction interface is controlled to highlight the target to be selected.

[0061] In an optional implementation, the human-computer collaborative system analyzes the sight data set to extract sight coordinate information, and then determines which elements are located in areas that intersect with the sight coordinates or are within the sight focus range according to the layout and element distribution of the human-computer interaction interface.

[0062] Furthermore, for these possible targets, the system highlights them by changing their display properties, such as color (changing to bright colors), brightness (increasing brightness), adding borders (bold or flashing borders), etc.

[0063] For example, when a user's eyes move on the interface of an image editing software, if the eyes are close to a certain image element, a flashing blue border will appear around the image element, indicating that it is highlighted as a target to be selected.

[0064] Step S103, responding to the first operation instruction of the user, and controlling the human-computer interaction interface to lock the target to be selected based on the first operation instruction of the user.

[0065] Among them, the user's first operation instruction may include key operations (such as the Enter key, Space key, Ctrl key, etc. on the keyboard), gesture operations (such as click, slide, pinch and other gestures on the touch human-computer interaction interface) and voice commands (such as saying "select this", "lock target", etc.), etc.

[0066] Specifically, after receiving a user operation instruction, the system processes it according to the instruction type and the state of the currently highlighted target to be selected.

[0067] Furthermore, when locking a target to be selected, you can lock one target to be selected, or continuously lock multiple targets to be selected, or lock multiple targets to be selected in sequence. For example, after locking one target to be selected, you can continue to lock the second and third targets to be selected. The embodiment of the present invention does not make specific limitations on this and can be determined according to actual needs.

[0068] Among them, when multiple candidates are locked continuously, the multiple candidates can be formed into a set to wait for specific task instructions; for newly locked targets, they are added to the set of tasks to be executed.

[0069] Specifically, if it is a confirmation command and only one target to be selected is highlighted, then the target is directly locked; if there are multiple highlighted targets to be selected, it is further determined according to the command whether to select all or part of the targets for locking. For example, when the corresponding single target or multiple targets are highlighted in the human-computer interaction interface, the current gaze target can be confirmed to be selected by closing one eye or pressing a key; or when multiple file icons are highlighted as targets to be selected, the user directly says the "lock" command or presses the lock key, and the system locks all the targets corresponding to these file icons and enters the target locking set; if the user says the command "lock person", only the target "person" among the multiple targets to be selected is added to the target locking set, where "person" can be replaced by any other specific target type; the locked target set is used for subsequent batch operations, such as copying, deleting, data matching, historical related target retrieval, related intelligence analysis, task overall allocation, etc.

[0070] The target locking method provided in this embodiment can accurately capture the user's line of sight action in the human-computer interaction interface by acquiring a line of sight data set. Furthermore, the line of sight data is used to enable the system to intelligently highlight the target to be selected, effectively attract the user's attention and give visual feedback, so that the user can intuitively understand the system's response to its line of sight focus area, enhance the real-time and intuitiveness of human-computer interaction, help to quickly screen out possible operation targets, avoid users from blindly searching among many interface elements, and improve interaction efficiency. Finally, the final locked target is further determined in combination with the user's operation instructions, fully considering the user's subjective intention, and realizing the effective fusion of information from both the human and the machine. Therefore, by implementing the present invention, the eye movement information is used to preliminarily locate the target, and the user's active operation is used to confirm, avoiding the possible misoperation caused by relying solely on eye movement, making the target locking more accurate and reliable, thereby realizing functions such as discrete target selection and centralized operation, and improving the flexibility and accuracy of the human-computer collaborative system in multi-target processing scenarios.

[0071] In this embodiment, a target locking method is provided for a human-machine collaborative system, the human-machine collaborative system including a human-machine interactive interface; Figure 2 is a flow chart of a target locking method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0072] Step S201, obtain the sight line dataset. For details, see Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0073] Step S202: controlling the human-computer interaction interface to highlight the target to be selected based on the line of sight data set.

[0074] Specifically, the above step S202 includes:

[0075] Step S2021, based on the line of sight data set, calculate the initial interaction coordinates between the user's line of sight and the human-computer interaction interface.

[0076] Specifically, the intersection coordinates of the binocular sight lines and the human-computer interaction interface can be calculated based on the current eye-to-interface distance, the head deflection angle, and the sight line angle collected by the device in the sight line data set.

[0077] In an optional implementation, taking a two-dimensional plane as an example, the corresponding calculation process is given:

[0078] For the left eye’s line of sight, its deviation angle relative to the front is known (set as α 左 ), the deflection angle of the head (set as θ 头) and the distance between the eye and the interface (set as d). In a two-dimensional rectangular coordinate system (assuming that the plane where the human-computer interaction interface is located is the coordinate system plane, and the origin of the coordinates can be set by yourself, such as the center of the interface), according to the trigonometric function relationship, the horizontal coordinate component x of the left eye's line of sight on the interface can be calculated 左 and the vertical coordinate component y 左 .

[0079] For example, the horizontal coordinate component x of the left eye's line of sight on the interactive interface can be calculated by the following equation (1): 左 :

[0080] x 左 =d×tan(α 左 +θ 头 )

[0081] Further, according to the same method, the horizontal coordinate component x of the right eye's line of sight on the interactive interface can be calculated: 右 and the vertical coordinate component y 右 .

[0082] Finally, the coordinates of the left and right eye sights can be combined according to certain rules. A common method is to take the average value. For example, the final initial interaction coordinates have a horizontal coordinate value of (x 左 +x 右 ) / 2, the vertical coordinate value is (y 左 +y 右 ) / 2, and then the initial interaction coordinates (x, y) between the user's line of sight and the human-computer interaction interface can be obtained.

[0083] Step S2022, performing coordinate smoothing processing on the initial interaction coordinates to obtain target interaction coordinates.

[0084] Specifically, in order to eliminate the influence of eye saccades and device sampling errors, the initial interaction coordinates may be smoothed.

[0085] First, under different resolutions, the distance change of eye saccades on the coordinates is different. Therefore, the eye saccade threshold can be set according to the resolution of the human-computer interaction interface. For example, for the common 1920×1080 resolution screen, after a lot of experiments and data analysis, the eye saccade threshold is determined to be 5 pixels.

[0086] Secondly, the distances between the initial interaction coordinates and the initial interaction coordinates obtained at the previous moment in the horizontal direction and the vertical direction may be calculated.

[0087] Finally, if both the horizontal distance and the vertical distance are lower than the set saccade threshold, the initial interaction coordinates are added to the fixation point set and weighted averaged to obtain the current fixation point coordinates, i.e., the target interaction coordinates.

[0088] Among them, the weight can be set according to the order of joining, such as the weight of the first joining point is 1, and the weight of the last joining sight point is n, where n is the number of all current sight points.

[0089] Furthermore, if both the horizontal distance and the vertical distance are greater than the set eye saccade threshold, the current gaze point set is cleared, and the new sight line coordinates are added as the current gaze point coordinates, i.e., the target interaction coordinates.

[0090] Step S2023, determining the sight line coordinate interval in the human-computer interaction interface based on the target interaction coordinates.

[0091] Specifically, an eye tracker or a gaze tracking device may be used to accurately determine the gaze coordinate deviation α° at the middle position of the interactive interface, wherein the specific range may be 1 / 2 of the accuracy range of the eye tracker or the gaze tracking device or a smaller value.

[0092] Furthermore, the maximum coordinate deviation of the center position of the human-computer interaction interface is set to β ° , the maximum coordinate deviation on the lower side is γ ° , can be confirmed through preliminary data experiments. For example, through a large number of user gaze tracking experiments, it is statistically found that in a specific operation scenario, the maximum coordinate deviation of the upper side of the center position is 3° (i.e., β ° =3°), the maximum coordinate deviation on the lower side is -2° (i.e. γ ° =-2°).

[0093] Furthermore, given that the resolution of the interactive interface is W*H, the distance from the corner of the interactive interface to the center point is

[0094] Furthermore, assuming that the distance between the sight line coordinate position and the center point is x, the current sight line point deviation y can be calculated using the following equation (1):

[0095]

[0096] Finally, with the target interaction coordinates as the center, the upper and lower boundaries of the sight line coordinate interval can be determined according to the upper and lower deviation values ​​y.

[0097] Step S2024, based on the sight line coordinate interval, controlling the human-computer interaction interface to highlight the target to be selected, or, based on the target interaction coordinates and the sight line coordinate interval, controlling the human-computer interaction interface to highlight the target to be selected.

[0098] Specifically, the human-computer interaction interface can be controlled to highlight the target to be selected through the obtained sight line coordinate interval or by combining the target interaction coordinates and the sight line coordinate interval.

[0099] In some optional implementations, when controlling the human-computer interaction interface to highlight the target to be selected based on the line of sight coordinate interval, the above step S2024 includes:

[0100] Step a1, determine whether the sight line coordinate interval overlaps with the target interval and determine the target to be selected.

[0101] Step a2, controlling the human-computer interaction interface to highlight the target to be selected.

[0102] Specifically, each target in the human-computer interaction interface (such as icons, buttons, text areas, and other elements) has its corresponding target range. These target ranges can be pre-set during interface design, and their ranges are usually determined by the boundary coordinates of the target. For example, the target range of a button may be a rectangular area defined by its upper left corner coordinates (x1, y1) and lower right corner coordinates (x2, y2).

[0103] Furthermore, the sight line coordinate interval can be compared with the target interval of each target, and if there is an overlap, the corresponding target is used as a candidate target.

[0104] Furthermore, when only one target's interval overlaps with the sight line coordinate interval, the target is directly highlighted.

[0105] There are many ways to highlight, such as: changing the color of the target, such as changing the original color of the target to a brighter color (such as red); increasing the brightness of the target to make it more eye-catching on the interface; adding a border to the target, such as adding a bold black border.

[0106] Furthermore, when the intervals of multiple targets overlap with the sight line coordinate interval, the degree of highlighting can be adjusted according to the distance between the target and the sight line coordinate center. For example, the closer the target is, the higher the intensity of highlighting is.

[0107] By judging the overlap between the sight coordinate interval and the target interval, the target that the user may focus on can be accurately determined. The directionality of eye movement is fully utilized, and the target related to the user's sight can be quickly screened out from many interface elements, avoiding the one-by-one inspection of the entire interface elements, greatly improving the speed of target locking, and realizing the rapid target positioning in the discrete target selection function. At the same time, it also provides a clear object for subsequent operations on specific targets. Further, the determined target is highlighted, which enhances the visualization effect of human-computer interaction, allowing users to clearly know that the system has identified their possible operation targets, and it is also convenient for users to confirm whether the target is correct again, improving the accuracy and reliability of the interaction, helping users to quickly confirm the target and perform subsequent operations in multi-target scenarios, and improving the usability and interaction efficiency of the human-computer collaborative system. At the same time, the target locking scheme based on sight can also avoid the problem of moving the mouse pointer to the target to be selected when the target is found. Among them, the mouse pointer and other on-screen markers sometimes take a long time to search, and in this process, the dynamic target state may have changed.

[0108] In some optional implementations, the human-machine collaborative system is connected to an unmanned device; when the human-machine interaction interface is controlled to highlight the target to be selected based on the target interaction coordinates and the sight line coordinate interval, the above step S2024 includes:

[0109] Step b1, obtaining the horizontal field of view angle and the vertical field of view angle of the unmanned equipment.

[0110] Step b2, calculating the horizontal deflection angle and vertical deflection angle of the unmanned equipment based on the target interaction coordinates, the horizontal field of view angle and the vertical field of view angle.

[0111] Step b3, sending the horizontal deflection angle and the vertical deflection angle to the unmanned device, so that the unmanned device makes adjustments based on the horizontal deflection angle and the vertical deflection angle.

[0112] Step b4, when receiving the unmanned device picture sent by the adjusted unmanned device.

[0113] Step b5, based on the line of sight coordinate interval, control the unmanned equipment screen to be projected onto the human-computer interaction interface and highlight it.

[0114] Specifically, the human-machine collaborative system may send a request for obtaining the field of view angle information to the unmanned device through a communication link (such as wireless communication), wherein the request may be a data message in a specific format, which includes instructions for requesting to obtain the horizontal field of view angle (HFOV) and the vertical field of view angle (VFOV).

[0115] Further, after receiving the request, the unmanned device obtains the values ​​of the horizontal field of view angle and the vertical field of view angle from its own configuration or sensor data and returns them to the human-machine collaborative system. For example, the camera module of the unmanned device stores parameter information such as the horizontal field of view angle of its lens is 60° and the vertical field of view angle is 40°.

[0116] Furthermore, assuming that the line of sight coordinate position is (x, y), and the coordinate of the center point of the screen is (x c ,y c ), then r1=(xx c ) / x c , r2=(ty c ) / y c Where, the negative sign indicates the deflection direction, and the horizontal deflection angle hδ and the vertical deflection angle vδ are respectively expressed as follows:

[0117] hδ=arctan(r1*tanHFOV) (2)

[0118] vδ=arctan(r2*tanVFOV) (3)

[0119] Furthermore, the human-machine collaborative system can send the obtained horizontal deflection angle hδ and vertical deflection angle vδ to the unmanned equipment.

[0120] Furthermore, the unmanned equipment can be adjusted according to the received horizontal deflection angle hδ and vertical deflection angle vδ by driving a motor or adjusting the flight attitude.

[0121] Among them, the adjustment may include adjusting the direction of the unmanned equipment, adjusting the focal length, moving to a corresponding direction, etc.

[0122] Furthermore, after receiving the unmanned device screen sent by the adjusted unmanned device, the unmanned device screen can be controlled to be projected onto the human-computer interaction interface and highlighted based on the line of sight coordinate interval.

[0123] The specific control process can refer to the description of step a1 and step a2 above, which will not be repeated here.

[0124] By obtaining the horizontal field of view and vertical field of view of the unmanned equipment, and calculating the horizontal deflection angle and vertical deflection angle of the unmanned equipment based on the target interaction coordinates, the line of sight correlation mapping between the human-computer interaction interface and the unmanned equipment is realized. Further, the calculated deflection angle is sent to the unmanned equipment, so that it is adjusted based on the deflection angle, and after receiving the adjusted unmanned equipment screen, it is highlighted on the human-computer interaction interface based on the line of sight coordinate interval, so as to realize the seamless connection between the unmanned equipment perspective and the human-computer interaction interface. Therefore, by implementing the present invention, the user can accurately control the direction of the unmanned equipment through the line of sight operation on the human-computer interaction interface, and expand the human-computer collaboration to the interaction field between humans and unmanned equipment, providing an intuitive and convenient way to realize remote control of unmanned equipment, and improving the accuracy and efficiency of unmanned equipment control. Furthermore, the application of the target selection function in the specified area in the unmanned equipment scene is also realized, which improves the target locking and operation execution capabilities of the human-computer collaboration system in cross-device interaction.

[0125] In some optional implementations, in the target locking method provided by the above embodiment of the present invention, after the above step S2023, step S202 further includes:

[0126] Step S2025, responding to the user's second operation instruction.

[0127] Among them, the user's second operation instruction can refer to the description of the user's first operation instruction in step S103, which will not be repeated here.

[0128] Step S2026, based on the user's second operation instruction and the target interaction coordinates, the human-computer interaction interface is controlled to highlight the selected target; or, based on the user's second operation instruction and the line of sight coordinate interval, the human-computer interaction interface is controlled to highlight the selected target; or, based on the user's second operation instruction, the target interaction coordinates and the line of sight coordinate interval, the human-computer interaction interface is controlled to highlight the selected target.

[0129] In some optional implementations, when the human-computer interaction interface is controlled to highlight the target to be selected based on the user's second operation instruction and the target interaction coordinates, the above step S2026 includes:

[0130] Step c1, using the target interaction coordinates, controlling the human-computer interaction interface to generate a first candidate area target.

[0131] Step c2: adjusting the first candidate area target based on the second operation instruction of the user to obtain a second candidate area target.

[0132] Step c3, controlling the human-computer interaction interface to highlight the second candidate area target.

[0133] Specifically, with the target interaction coordinates as the center, the human-computer interaction interface can be controlled to generate a first target area to be selected based on a preset rule, wherein the preset rule can be to define a circular or rectangular area with the target interaction coordinates as the center.

[0134] Furthermore, the shape or size of the first target in the area to be selected may be adjusted according to the received second operation instruction of the user to form the second target in the area to be selected.

[0135] Furthermore, the second target in the candidate area can be highlighted by changing its visual attributes. The specific process can refer to the description in the above step a2, which will not be repeated here.

[0136] Furthermore, the above method realizes dynamic adjustment of regions in regional inspection / monitoring functions, improves the accuracy and flexibility of region selection, and enhances the target locking and operation execution capabilities of the human-machine collaborative system in regional operation scenarios.

[0137] In some optional implementations, when the human-computer interaction interface is controlled to highlight the target to be selected based on the user's second operation instruction and the line of sight coordinate interval, the above step S2026 includes:

[0138] Step d1, when there is a deviation between the sight line coordinate interval and the target interval, determining a target to be selected in the human-computer interaction interface based on the user's second operation instruction and the sight line coordinate interval.

[0139] Step d2, controlling the human-computer interaction interface to highlight the target to be selected.

[0140] Specifically, it is determined whether there is a deviation between the sight line coordinate interval and the target interval. If there is a deviation, the sight line coordinate offset direction can be indicated according to the received second operation instruction of the user (key press, gesture, mouse movement, etc.).

[0141] For example, the user can press a specific direction key (such as the left arrow key indicates that the line of sight is shifted to the left, the right arrow key indicates that the line of sight is shifted to the right, etc.) to indicate the offset direction of the line of sight coordinates; on the human-computer interaction interface, the user can use a left-sliding gesture to indicate that the line of sight is shifted to the left, and a right-sliding gesture to indicate that the line of sight is shifted to the right, etc.; the user can move the mouse pointer to the vicinity of the direction in which the line of sight coordinates may shift, and the system determines the offset direction of the line of sight coordinates based on the direction of mouse movement.

[0142] Furthermore, the distance between each point in the sight line coordinate interval and the possible target interval can be calculated, and the target closest to the sight line coordinate interval is regarded as the currently watched entity, that is, determined as the target to be selected.

[0143] Further, the human-computer interaction interface is controlled to highlight the target to be selected. The specific process can refer to the description in the above step a2, which will not be repeated here.

[0144] Furthermore, through the above method, possible errors in line of sight data are taken into account. At the same time, inaccurate target determination that may be caused by errors is corrected through the user's operation instructions, which improves the accuracy and reliability of target locking. Even if the line of sight data is not very accurate, the correct target can be quickly determined through the user's active intervention, avoiding misjudgment or loss of the target due to errors, and improving the target locking ability of the human-machine collaborative system in complex environments or equipment errors.

[0145] In some optional implementations, when the human-computer interaction interface is controlled to highlight the target to be selected based on the user's second operation instruction, the target interaction coordinates, and the sight line coordinate interval, the above step S2026 includes:

[0146] Step e1, based on the target interaction coordinates and the second user operation instruction, control the human-computer interaction interface to be enlarged.

[0147] Step e2, based on the sight line coordinate interval, controlling the enlarged human-computer interaction interface to highlight the target to be selected.

[0148] The zoom-in operation may include two execution modes:

[0149] (1) Directly enlarge the interactive interface;

[0150] Specifically, the user inputs the zoom-in instruction by means of keys, gestures, or voice.

[0151] Furthermore, the target interaction coordinates are used as the center position of the magnification, and the magnification ratio can be determined according to the user's second operation instruction and the preset rules. For example, if the magnification is step-by-step, the magnification ratio is pre-agreed by the system. For example, the system sets each magnification operation to magnify the area by 1.5 times. Then, after receiving the magnification instruction, the target interaction coordinates are used as the center, and the surrounding area is magnified by 1.5 times.

[0152] Furthermore, when the sight coordinate interval overlaps with multiple target intervals, the coverage range of the square frame that can cover the multiple target intervals can be determined. For example, the sight coordinate interval overlaps with the target intervals of three icons on the interface, and the system finds a smallest square frame that can include all three icons, and the range of this square frame is the area to be enlarged.

[0153] Further, the range covered by the square box is enlarged in the above manner (such as step-by-step enlargement). If it is found that the enlarged interval will exceed the border of the picture during the enlargement process, the system adopts a strategy of moving to the middle. For example, when the right part of the enlarged square box exceeds the right border of the picture, the system moves the entire square box to the left so that it is completely within the picture, while keeping the enlargement ratio unchanged.

[0154] (2) Control the camera device of the unmanned equipment to adjust the focal length for zooming in. By increasing the focal length of the lens, the field of view angle decreases, the scene range captured by the sensor decreases, and the projection size of the target object in the image increases, thereby achieving a magnification effect.

[0155] The process of controlling the camera device of the unmanned equipment to adjust the focal length can refer to the description of the above steps b2 and b3, which will not be repeated here.

[0156] Specifically, by controlling the camera device of the unmanned equipment to adjust the focal length for zooming in, the field of view can be reduced while the picture remains clear within the field of view.

[0157] Further, an unmanned device image sent by the unmanned device after the focus is adjusted is received.

[0158] Further, based on the sight line coordinate interval, the enlarged human-computer interaction interface is controlled to highlight the target to be selected. The specific process can refer to the description of the above steps a1 and a2, which will not be repeated here.

[0159] Furthermore, through the above method, using the target interaction coordinates to perform the zoom operation of the human-computer interaction interface can allow users to view the target details more clearly, especially for situations where the target is small or the interface information is complex. Furthermore, in the zoomed-in interface, the target to be selected is highlighted based on the line of sight coordinate interval, further emphasizing the key area of ​​user attention, and the target can be quickly located even in the complex interface after zooming in, avoiding the user from getting lost in the target position after zooming in, making the target locking still efficient and accurate under the zooming operation, and improving the target locking ability and interactive experience of the human-computer collaborative system under different display requirements.

[0160] Step S203, respond to the user's first operation instruction, and control the human-computer interaction interface to lock the target to be selected based on the user's first operation instruction. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0161] The target locking method provided in this embodiment can accurately determine the starting position of the user's line of sight in the human-computer interaction interface by calculating the initial interaction coordinates of the user's line of sight and the human-computer interaction interface. Further, the coordinate smoothing process can effectively reduce the coordinate fluctuations caused by factors such as eye movement jitter and device errors, and obtain more stable and reliable target interaction coordinates. Further, the line of sight coordinate interval is determined based on the target interaction coordinates, which provides a clear area definition standard for judging whether the target is in the user's attention range. Further, by determining the target interaction coordinates or the line of sight coordinate interval, possible candidates can be screened out more effectively, reducing unnecessary target search ranges, and improving the speed and efficiency of target locking, especially in a multi-target interface, it can quickly focus on the area that the user may be interested in, and realize rapid target positioning in the regional inspection / monitoring function. Further, by combining the user's second operation instruction to control the human-computer interaction interface again to highlight the target to be selected, the flexibility and diversity of human-computer interaction are increased, more refined and personalized human-computer collaborative operations are achieved, and the system's responsiveness to the user's complex operation intentions is improved.

[0162] In an optional implementation, the target locking method provided by the above embodiment of the present invention further includes: determining and executing the target execution task of the target to be selected based on a preset system agreed instruction.

[0163] Specifically, the target execution task of the target to be selected can be determined and executed according to the specific operation type of the preset system agreed instruction.

[0164] In an optional implementation, when the target to be selected is the target of the second selected area in step c2, the instructions agreed upon by the system can be used to define the specific operations expected to be performed on the area, and the human-machine collaborative system can assign the agent to perform the corresponding tasks. The specific executable tasks may include:

[0165] a) Assign unmanned equipment or intelligent agents to monitor and search the area and perform coverage search tasks;

[0166] b) marking all or specific categories of target entities in the area, which can be processed jointly with the target selection results in step a1 or step d1;

[0167] c) Clear all marked target entities in the area;

[0168] d) Mark the area as containing undetected targets.

[0169] Furthermore, when multiple locked targets to be selected form a set, tasks can be coordinated and arranged for the locked targets through a single instruction.

[0170] For example: after selecting multiple targets, a "tracking" command is issued, and the human-machine collaborative system coordinates the unmanned equipment to track the target, or "information retrieval / search / matching", etc. The human-machine collaborative system can retrieve relevant information and similar targets from the database instead of selecting a target and arranging a task;

[0171] Alternatively, multiple unmanned devices are selected to transmit images as targets, which means locking multiple unmanned devices themselves. The command may require gathering in a certain direction / unmanned device, etc.;

[0172] Alternatively, based on the already locked target, the human-machine collaborative system is controlled to lock all similar targets (for example, if there are multiple similar targets distributed in different unmanned equipment screens, it is slow to select them one by one, and the target names cannot be recalled under stress, one or more targets can be locked through eye movements first, and the rest can be left to the human-machine collaborative system).

[0173] Furthermore, the above method realizes the automation and standardization of operations in the human-machine collaborative system. At the same time, the user does not need to perform tedious manual operation settings, which improves the efficiency and speed of operation execution, thereby improving the task execution capability and production efficiency of the human-machine collaborative system in specific task scenarios.

[0174] Furthermore, when multiple targets to be selected are locked in sequence, for example, after locking one target to be selected, the second and third targets to be selected can be locked continuously, and operations and task allocation can be performed on these locked targets in a centralized manner after the locking is completed.

[0175] In an optional implementation, the target locking method provided in the above embodiment of the present invention further includes: based on the line of sight data set, controlling the human-computer interaction interface to remove the locked candidate targets and the candidate targets corresponding to the target execution tasks.

[0176] Specifically, the line of sight interaction coordinates between the line of sight and the human-computer interaction interface may be determined according to the line of sight data set.

[0177] Further, if the sight interaction coordinates are at the target selected in step a1, the target is removed from the to-be-executed set.

[0178] Furthermore, if the sight interaction coordinates are in the second target area to be selected in step c2, the task to be or already executed in the area is terminated or abandoned, and at the same time, all the targets to be executed in the second target area to be selected are removed from the set to be executed.

[0179] Furthermore, the above method achieves the effect of releasing machine resources by undoing the operation, thereby improving the utilization rate of system resources. At the same time, it also makes human-computer interaction more flexible and convenient. Users can correct their operation choices at any time to avoid resource waste and operational errors caused by wrong choices, thereby improving the fault tolerance and resource management capabilities of the human-computer collaborative system.

[0180] In one example, a human-machine collaborative multi-target fast locking method integrating eye movement is provided. Specifically, the machine collaborative system needs to realize efficient and convenient information communication between humans and machines, and ensure that users can use the system to control remote unmanned equipment in a convenient way. Generally speaking, it is up to humans to determine the tasks and task objects they want to perform, and the human-machine collaborative system performs task splitting and human-machine task allocation according to the brief instructions of humans. Therefore, this example designs a human-machine interaction method based on the human-machine interaction interface, with the gaze direction-interaction interface positioning as the core. In order to clarify the task object, this example decomposes the subjective operation intention of the person into single target or multi-target selection, selection of the area in the picture returned by the unmanned equipment, and control instructions for the unmanned equipment. For the convenience of description, in this example, the so-called target selection includes the entity obtained by target detection or tracking model detection in the picture returned by the unmanned equipment, the area defined by the user in the picture of the unmanned equipment, the picture windows returned by multiple unmanned equipment, and the menus and buttons existing in the interactive interface itself, that is, all elements existing in the interactive interface of the human-machine collaborative system can be regarded as targets to be selected.

[0181] Further, if Figure 3 As shown, the human-machine collaborative multi-target rapid locking method provided in this example by integrating eye movements includes:

[0182] 1. Line of sight coordinate positioning

[0183] 1) Use an eye tracker or gaze tracking device to preliminarily obtain the user's eye gaze angle or the coordinates of the interactive interface. If the user's eye gaze angle is obtained, it is necessary to additionally calculate the intersection coordinates of the eye gaze and the interactive interface:

[0184] (1) The user’s binocular sight angle, i.e., the deviation angle of the user’s binocular sight relative to the front obtained by the sight tracking device;

[0185] (2) Head deflection and displacement, that is, the horizontal and forward and backward movement distance and the angle of head rotation relative to the initial state during the gaze tracking process;

[0186] (3) Based on the current distance between the eyes and the interactive interface, the head deflection angle, and the line of sight angle collected by the device, calculate the intersection coordinates of the binocular line of sight and the interactive interface.

[0187] 2) Smoothing of gaze coordinate position (to eliminate the influence of eye saccades and acquisition errors). A possible gaze coordinate smoothing scheme is as follows:

[0188] (1) Setting the eye saccade threshold according to the resolution of the interactive interface;

[0189] (2) If the distance between the gaze point and the coordinate point obtained at the previous moment is lower than the threshold, the gaze coordinate is added to the gaze point set and weighted average is performed to obtain the current gaze point coordinate. For simplicity, the weight can be set according to the order of addition, such as the weight of the first added point is 1, and the weight of the last added gaze point is n, where n is the number of all current gaze points;

[0190] (3) If the distance between the gaze coordinate point and the coordinate point at the previous moment is greater than the threshold, the current gaze point set is cleared and the new gaze coordinate is added as the current gaze point coordinate;

[0191] 3) Limited by the accuracy of the instrument, the autonomous movement of the eyeball and other factors, the sight interval is defined based on the sight coordinate position in 2):

[0192] (1) Set the center position of the interactive interface, the line of sight coordinate deviation is α°, the specific range is determined according to the accuracy of the eye tracker or line of sight tracking device used, and can be 1 / 2 of the accuracy range or a smaller value; the maximum coordinate deviation on the upper side of the center position is calculated as β ° , the maximum coordinate deviation on the lower side is counted as γ ° , confirmed by preliminary data experiments;

[0193] (2) The distance between the corner of the interactive interface and the center point is L (calculated in pixels). Assuming the resolution of the interactive interface is W*H, then

[0194] (3) Assuming that the distance between the sight line coordinate position and the center point is x, the current sight line point deviation is:

[0195]

[0196] 2. User operation intention realization

[0197] 1) Single target or multiple target selection

[0198] (1) When the sight coordinate interval overlaps with the target interval, the overlapping target is highlighted. If there are multiple overlapping targets, the degree of highlighting is adjusted according to the distance;

[0199] (2) If there is a deviation between the sight coordinate interval and the target interval, the direction of sight coordinate deviation is indicated by pressing a key, making a gesture, or moving the mouse, and the nearest target is regarded as the entity currently being gazed;

[0200] (3) Confirm that the current gaze target is selected by closing one eye or pressing a button;

[0201] (4) If the selected target is an entity in the screen transmitted by the unmanned device, the entity enters the pending execution set and is highlighted in a different way from the gaze retention stage, waiting for further instructions from the user; if the selected target is one of the screens transmitted by multiple unmanned devices, the screen is projected onto the main interactive interface; if the selected target is a menu option in the interactive interface, the operation represented by the menu option is executed normally.

[0202] 2) Region selection

[0203] (1) Taking the sight coordinate as the center, a certain area in the interactive screen is defined as the area to be selected;

[0204] (2) Using gestures, buttons, voice, and other command input methods to adjust the shape and size of the area;

[0205] (3) Using the instructions agreed upon by the system, the specific operations expected to be performed in the area are defined, and the human-machine collaborative system assigns the agent to perform the corresponding tasks. The specific executable tasks include but are not limited to:

[0206] a) Assign unmanned equipment or intelligent agents to monitor and search the area;

[0207] b) marking all or specific categories of target entities in the area, which can be processed jointly with the multi-target selection results in 1);

[0208] c) Clear all marked target entities in the area;

[0209] d) Marking that there is an undetected target in the area;

[0210] 3) Unmanned equipment control

[0211] (1) Spatial positioning

[0212] Calculate the sight coordinates and the deflection angle of the camera of the unmanned equipment. Assume that the horizontal field of view angle of the camera is HFOV, the vertical field of view angle is VFOV, the sight coordinate position is (x, y), and the coordinates of the center point of the screen are (x c ,y c ), r1=(xx c ) / x c , r2=(yy c ) / y c , the positive and negative signs indicate the deflection direction, then:

[0213] The horizontal angle difference is hδ=arctan(r1*tanHFOV)

[0214] The vertical angle difference is vδ=arctan(r2*tanVFOV)

[0215] (2) Executable intent includes but is not limited to:

[0216] a) Adjust the direction of the unmanned equipment, and adjust the horizontal and vertical deflection angles obtained according to (1) to move the corresponding area to the middle area of ​​the picture;

[0217] b) Adjust the focus, zoom in on the target area or include a larger range of the target orientation in the picture. In this case, execute a) first, then adjust the focus of the camera equipment to ensure that the corresponding orientation is clear;

[0218] c) Adjust the unmanned equipment to move toward the corresponding direction, and adjust the direction according to the deflection angle obtained in (1) and the unmanned equipment's own information.

[0219] 4) Zoom in

[0220] (1) Based on the line of sight coordinates, the corresponding area is enlarged, which can be performed in combination with 1) and 2). The enlargement command input method can be keystroke, gesture, voice, etc.;

[0221] (2) When the sight coordinate interval overlaps with multiple target intervals, the range of the square frame that can cover multiple target intervals is enlarged:

[0222] a) Enlarge step by step, and the enlargement ratio is determined by the system in a pre-agreed manner;

[0223] b) Control boundary overflow, that is, when the magnified area exceeds the boundary of the screen, move it to the middle;

[0224] (3) Interval zooming, that is, zooming in only the area selected in 2) and does not involve unmanned equipment operation.

[0225] 5) Revocation

[0226] (1) If the sight coordinate is at the target selected in 1), remove the target from the pending set;

[0227] (2) If the sight coordinates are in the area selected in 2), the task that is about to be or has been executed in the area is terminated or abandoned;

[0228] (3) If a new area is defined using method 2) to cancel the command, all pending targets in the area will be removed from the pending set.

[0229] 6) Confirm the execution of the aforementioned selected goals or tasks, and the collaborative system will carry out task allocation and specific execution.

[0230] This example provides a human-machine collaborative multi-target rapid locking method that integrates eye movements. Based on the directionality of gaze, it combines multiple visual models with human subjective intentions to construct a human-machine collaborative multi-target rapid locking method that integrates eye movements. It realizes functions such as discrete target selection and centralized operation, area inspection / monitoring, target selection in a specified area, and undoing operations to release machine resources.

[0231] In an optional embodiment, the human-machine collaborative system has multiple unmanned devices that can perform tasks, and the images sent back by all devices are displayed in the form of small windows on the system operation screen, and the main area of ​​the operation screen displays the images sent back by a specific unmanned device. Figure 1 As shown, the right side is the screen sent back by multiple unmanned devices, and the left side is the main display area, which is the screen presentation area of ​​the main control unmanned device. In this implementation example, the image screen is processed by the target detection and tracking model to identify all physical targets and categories, and use the mouse, keyboard and voice as additional command signals to clarify the user's operation intention. Furthermore, the buttons, gestures, and specific commands mentioned below are only used for the convenience of description and are not actually limited. The specific implementation and sending method shall be subject to the system agreement.

[0232] Specifically, the sight tracking device calculates the sight coordinates of the user on the operation interface and inputs them into the human-computer collaborative system. The system smoothes the updated sight coordinates based on the current sight coordinate information and the historical coordinate information as the center position of the sight area that the user mainly looks at. Assume that the sight error angle of the user in the center area of ​​the screen obtained by the experiment is α°. For simplicity, only the following sight coordinate errors are considered, and the maximum coordinate deviation on the upper side of the center position is calculated as β ° , the maximum coordinate deviation on the lower side is counted as γ ° , the distribution law of sight error based on the complete screen is not considered, and it is assumed that the sight error increases linearly with the distance from the center point. Let the distance from the center of the screen to the corner be L, and the distance from the sight coordinate to the center be x, then:

[0233]

[0234] Among them, y represents the error angle of the current line of sight.

[0235] Furthermore, when the sight coordinates stay at a certain position of the unmanned equipment screen on the left, the area will be highlighted. The highlighting methods include magnifying a certain multiple, changing the border color, etc. When the user presses the button to confirm, the unmanned equipment screen will be switched to the main screen display area for display.

[0236] Furthermore, the sight stays in the main screen area. If the sight interval (i.e., the area covered by the viewing angle with the sight coordinate as the center plus the deviation) overlaps with the object frame identified by the detection / tracking model, the object is highlighted. The highlighting method includes modifying the frame color, enlarging, covering a layer of transparent template, etc.; if there is a certain deviation between the sight interval and the target interval, the mouse or finger ring is used to slide to determine the deviation direction. At this time, the nearest target is directly included in the target lock set without additional confirmation. A key (such as d, remove from the target lock set) can be pressed; the key is confirmed, and the object is included in the target lock set. The above operation can be performed multiple times, and multiple objects can be locked at the same time. Subsequently, specific task instructions can be sent through voice or other methods agreed with the system. For example, if "tracking target" is sent, the system background automatically assigns unmanned equipment to monitor these locked targets. If "information retrieval" is sent, the system searches from two aspects, one is to retrieve whether different unmanned equipment has monitored the selected target, and the other is to retrieve similar entities and information of related targets from the database; for example, in a military environment, "strike" is sent, and the system arranges automatic strike equipment or sends fixed-point strike notifications to different snipers, and assigns targets and execution times.

[0237] Among them, the selectable targets are not limited to the targets in the current main display area. You can switch between different unmanned equipment screens through your line of sight, perform multi-unmanned equipment and multi-target selection and control, and realize perspective switching and combination.

[0238] Further, with the sight coordinate as the center, press the space bar (use other keys or mouse buttons or direct "area" command and other system agreed forms) to select a certain area, adjust the area size by pressing + or -, adjust the area shape by pressing shift or gestures, and press the enter key to confirm the selection of the area. The user then sends instructions in an agreed form to clarify the operation intention. If the voice is "search", the system arranges unmanned equipment or related intelligent agents to go to the area for search and retrieval (for example, when multiple drones are controlled for maritime rescue at the same time, if a suspected person to be rescued is found in a certain location, but it cannot be successfully detected using the visual model, the drone is arranged to go to the area for observation); if the voice is "selected", all detected entities in the corresponding area are selected, which can be used in the case of gathering of targets to be locked; if the voice "special target" is sent, only the pre-agreed specific type of target is selected, and the agreed method can be category, text description, or specific person visual representation, and there are other entities around the target to cover and block; if "target not recognized" is sent, the system background calls model fine-tuning and other means to improve the detection effect of unrecognized targets by the model.

[0239] Furthermore, when the human-machine collaborative system remotely controls unmanned equipment, it is necessary to control the equipment to move toward a specified direction, or to focus on a specified direction, such as adjusting the focal length to include a larger field of view of that direction, or reducing the field of view of that direction to magnify the object in it. The deflection angle calculation, that is, calculating the angle of movement, can move the spatial position corresponding to the current line of sight to the middle area of ​​the field of view of the unmanned equipment. A spatial positioning calculation method is as follows:

[0240] The horizontal field of view of the camera is HFOV, the vertical field of view is VFOV, the line of sight coordinate position is (x, y), and the coordinates of the center point of the screen are (x c ,y c ), r1=(xx c ) / x c , r2=(yy c ) / y c , the positive and negative signs indicate the deflection direction, then:

[0241] The horizontal angle difference is: hδ = arctan (r1*tanHFOV);

[0242] The vertical angle difference is: vδ = arctan(r2*tanVFOV);

[0243] Furthermore, by pressing the left mouse button, the direction of the unmanned device or the camera device can be adjusted. The specific method is adjusted by the unmanned device itself, such as only moving the camera position, or moving the unmanned device as a whole, that is, the horizontal direction moves at an angle of hδ and the vertical angle moves at vδ. The unmanned device executes the movement strategy according to the command angle.

[0244] Further, by pressing the right button of the mouse, the unmanned device moves toward the line of sight coordinates, and the system sends the horizontal movement angle hδ and the vertical movement angle vδ to the unmanned device. After receiving the command, if necessary, the unmanned device first moves the body to be consistent with the direction of the camera, and then rotates horizontally by hδ to make the machine face the specified direction. The unmanned device adjusts the pitch angle as needed (such as a drone), and then the device moves to the specified direction.

[0245] Furthermore, sliding the roller will adjust the focal length. Moving the roller forward will enlarge the field of view, and moving the roller backward will magnify the objects in the field of view. First, move the angle of the unmanned equipment or camera equipment to ensure that the spatial angle of the positioning is updated to the center of the field of view, and then adjust the focal length according to the instructions.

[0246] Further, if Figure 4As shown in the figure, during the target locking process, some targets are too small and the overlap is too high. At this time, it is difficult to make a unique selection simply by means of sight coordinates. At this time, it is necessary to combine the sight coordinates with the zoom operation to zoom in on the interval of interest and enlarge the area of ​​the non-overlapping interval, thereby promoting the selection of the only target. There are two methods that can be used: Method 1, the sight stays on the target interval to be selected, and the zoom key is directly pressed, then the square box that can cover all the targets covered by the sight interval is directly calculated, and then the square box coverage area is enlarged; Method 2, using area selection, select an area containing the target with the sight coordinates as the center, and then enlarge the area. It should be noted that the zooming process is in-situ zooming. If after multiple zooming, the boundary is displayed in one area, the area is moved to the middle of the display area. The above example is only used as an illustrative use case for the zoom operation, and it is not limited to this application.

[0247] Furthermore, during the target locking process, if an operation error or the development of an event results in some targets not needing to be processed further or new information excludes some targets, then it is necessary to cancel the lock or command processing for the locked targets. Cancel by target, that is, only cancel the selected targets that overlap with the line of sight interval, and the target enters the unlocked state; cancel the lock of multiple targets at the same time, use the area selection function to wrap the target to be canceled into the area, and then press the cancel key to cancel the lock state of all locked targets in it; if the line of sight coordinate interval overlaps with the selected area, the cancellation operation means canceling the locking task for the area.

[0248] Therefore, through this example, combined with machine intelligence, using simple guiding operations, the efficiency of gaze utilization can be improved (for example, if the gaze point is evaluated to have a certain offset, it can be moved to the required position through external instructions; or the gaze position can be used to circle out entities or conditions that are not recognized by machine intelligence, and corresponding optimization requirements are proposed for the background intelligent agent). Furthermore, there is no need for a special operating table for the mouse, nor is there any need to consider the problem of hand sight obstruction during touch screen selection. Furthermore, in the screening of specific targets among multiple entities, problems such as finding mouse marks and switching attention when moving the mouse are reduced.

[0249] In this embodiment, a target locking device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0250] This embodiment provides a target locking device, such as Figure 5 As shown, the device comprises:

[0251] The acquisition module 501 is used to acquire a sight line dataset, where the sight line dataset is used to characterize the sight line movement of the user's eyes in the human-computer interaction interface.

[0252] The first control module 502 is used to control the human-computer interaction interface to highlight the target to be selected based on the line of sight data set, where the target to be selected is one or more elements in the human-computer interaction interface.

[0253] The second control module 503 is used to respond to the first operation instruction of the user, and control the human-computer interaction interface to lock the target to be selected based on the first operation instruction of the user.

[0254] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0255] The target locking device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0256] The embodiment of the present invention also provides a computer device having the above Figure 5 Target locking device shown.

[0257] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0258] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0259] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0260] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0261] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0262] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0263] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0264] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0265] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A target locking method, characterized in that: Used in a human-machine collaborative system, the human-machine collaborative system includes a human-machine interaction interface; the method includes: Acquire a sight line data set, where the sight line data set is used to characterize the sight line movement of the user's eyes on the human-computer interaction interface; Controlling the human-computer interaction interface to highlight a target to be selected based on the line of sight data set, where the target to be selected is specific position information or one or more elements in the human-computer interaction interface; Respond to a first user operation instruction, and control the human-computer interaction interface to lock the target to be selected based on the first user operation instruction.

2. The method according to claim 1, characterized in that Controlling the human-computer interaction interface to highlight the target to be selected based on the line of sight data set includes: Based on the line of sight data set, calculating the initial interaction coordinates between the user's line of sight and the human-computer interaction interface; Performing coordinate smoothing processing on the initial interaction coordinates to obtain target interaction coordinates; Determining a sight line coordinate interval on the human-computer interaction interface based on the target interaction coordinates; Based on the sight line coordinate interval, the human-computer interaction interface is controlled to highlight the target to be selected; or, based on the target interaction coordinates and the sight line coordinate interval, the human-computer interaction interface is controlled to highlight the target to be selected.

3. The method according to claim 2, characterized in that Based on the line of sight coordinate interval, controlling the human-computer interaction interface to highlight the target to be selected includes: Determine whether the sight line coordinate interval overlaps with the target interval and determine the target to be selected; The human-computer interaction interface is controlled to highlight the target to be selected.

4. The method according to claim 3, characterized in that The human-machine collaborative system is connected to an unmanned device; based on the target interaction coordinates and the sight line coordinate interval, the human-machine interaction interface is controlled to highlight the target to be selected, and further includes: Obtaining the horizontal field of view angle and the vertical field of view angle of the unmanned equipment; Calculating a horizontal deflection angle and a vertical deflection angle of the unmanned equipment based on the target interaction coordinates, the horizontal field of view angle, and the vertical field of view angle; Sending the horizontal deflection angle and the vertical deflection angle to the unmanned device, so that the unmanned device is adjusted based on the horizontal deflection angle and the vertical deflection angle; When receiving the adjusted unmanned device picture sent by the unmanned device; Based on the line of sight coordinate interval, the unmanned equipment screen is controlled to be projected onto the human-computer interaction interface and highlighted.

5. The method according to claim 2, characterized in that: The method further comprises: Responding to a second operation instruction from the user; Control the human-computer interaction interface to highlight the target to be selected based on the second user operation instruction and the target interaction coordinates; or, Based on the second user operation instruction and the sight line coordinate interval, the human-computer interaction interface is controlled to highlight the target to be selected; or, The human-computer interaction interface is controlled to highlight the target to be selected based on the second user operation instruction, the target interaction coordinates and the sight line coordinate interval.

6. The method according to claim 5, characterized in that Controlling the human-computer interaction interface to highlight the target to be selected based on the second user operation instruction and the target interaction coordinates includes: Using the target interaction coordinates, controlling the human-computer interaction interface to generate a first candidate area target; Adjusting the first area target to be selected based on the second operation instruction of the user to obtain a second area target to be selected; The human-computer interaction interface is controlled to highlight the second candidate area target.

7. The method according to claim 5, characterized in that The method further comprises: controlling the human-computer interaction interface to highlight the target to be selected based on the second user operation instruction and the sight line coordinate interval; When there is a deviation between the sight line coordinate interval and the target interval, determining the target to be selected on the human-computer interaction interface based on the second user operation instruction and the sight line coordinate interval; The human-computer interaction interface is controlled to highlight the target to be selected.

8. The method according to claim 5, characterized in that Controlling the human-computer interaction interface to highlight the target to be selected based on the second user operation instruction, the target interaction coordinates, and the sight line coordinate interval includes: Based on the target interaction coordinates and the second user operation instruction, controlling the human-computer interaction interface to be enlarged; Based on the sight line coordinate interval, the enlarged human-computer interaction interface is controlled to highlight the target to be selected.

9. The method according to claim 1, characterized in that: The method further comprises: Based on the preset system agreed instructions, determine and execute the target execution task of the selected target.

10. The method according to claim 9, characterized in that The method further comprises: Based on the line of sight data set, the human-computer interaction interface is controlled to remove the locked candidate targets and the candidate targets corresponding to the target execution tasks.

Images