Man-machine interaction method and device for mobile equipment

Abstract

The invention provides a man-machine interaction method and device for mobile equipment, and the method comprises the steps: obtaining a first dynamic behavior feature based on feature extraction, carrying out the recognition of a human motion, and obtaining a completed first interaction motion and a corresponding first target object; analyzing a candidate associated object having a spatial association relationship with the first target object based on the scene structure information, and performing intention pre-judgment based on the candidate associated object and a feature change trend of the first dynamic behavior feature to obtain a pre-judgment interaction intention and a second target object corresponding to the pre-judgment interaction intention; controlling the mobile robot to execute a first response operation matched with the first interaction action; and if the pre-judged interaction intention passes verification based on the real-time response state of the first response operation in combination with a second dynamic behavior feature of the user, controlling the mobile robot to execute a second response operation for a second target object. According to the invention, the fluency of the man-machine interaction process is improved, and the interaction efficiency of man-machine interaction in the continuous interaction scene is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mobile robots, in particular to a human-robot interaction method for mobile devices and a device thereof. BACKGROUND

[0002] In the field of human-robot interaction of mobile robots, the existing mainstream method is an interaction method triggered based on a single visual instruction. The core process of this method is as follows: a mobile robot collects visual information in the surrounding environment in real time through a visual acquisition device carried by itself, when the visual acquisition device captures a preset specific action (such as waving hands, a specific gesture trajectory, etc.) made by a human user, the visual information is transmitted to the control module of the robot, the control module analyzes the visual information, and then triggers the robot to perform a preset response action (such as approaching the user, stopping moving, etc.) corresponding to the specific action.

[0003] However, since the robot response is triggered only by relying on the single visual instruction of the specific action already made by the user, the response of the robot to the user's interaction intention has obvious hysteresis. Specifically, when the user has continuous interaction requirements (such as sequentially instructing the robot to go to multiple target positions), the robot must wait until the user completes the previous specific action and the analysis is completed, and then it can be prepared to receive the next action instruction. The robot cannot predict the user's subsequent interaction intention in advance, which greatly reduces the smoothness of the whole human-robot interaction process, especially in scenarios that require efficient collaboration (such as the user instructing the robot to arrange multiple rooms in home service, or the worker instructing the robot to carry multiple batches of materials in an industrial scenario), the hysteresis will seriously affect the interaction efficiency. SUMMARY

[0004] The present application provides a human-robot interaction method for mobile devices and a device thereof to improve the smoothness of the human-robot interaction process and improve the interaction efficiency of human-robot interaction in continuous interaction scenarios.

[0005] In a first aspect, the present application provides a human-robot interaction method for mobile devices, comprising:

[0006] performing feature extraction based on a continuous visual image sequence of the environment where the user is located to obtain a first dynamic behavior feature, and performing human action recognition based on the first dynamic behavior feature to obtain a first interaction action completed by the user currently and a first target object corresponding thereto;

[0007] analyzing a candidate associated object having a spatial association relationship with the first target object based on the scene structure information of the environment where the user is located, and performing intention prediction based on the candidate associated object and a feature change trend of the first dynamic behavior feature to obtain a predicted interaction intention and a second target object corresponding thereto;

[0008] controlling the movable robot to perform a first response operation matching the first interaction action;

[0009] if the real-time response state based on the first response operation and the second dynamic behavior feature of the user verify that the pre-judgment interaction intention is passed, controlling the movable robot to perform a second response operation for the second target object.

[0010] In a second aspect, the present application further provides a human-machine interaction device for a movable device, which is applied to the human-machine interaction method for a movable device as described in the first aspect; the human-machine interaction device for a movable device comprises:

[0011] an action recognition module, configured to perform feature extraction based on a continuous visual image sequence of an environment where a user is located, to obtain a first dynamic behavior feature, and to perform human body action recognition based on the first dynamic behavior feature, to obtain a first interaction action that a user has completed currently and a first target object corresponding to the first interaction action;

[0012] an intention pre-judgment module, configured to analyze a candidate associated object having a spatial association relationship with the first target object based on scene structure information of an environment where a user is located, and to perform intention pre-judgment based on the candidate associated object and a feature change trend of the first dynamic behavior feature, to obtain a pre-judgment interaction intention and a second target object corresponding to the pre-judgment interaction intention;

[0013] a first operation control module, configured to control a movable robot to perform a first response operation matching the first interaction action;

[0014] a second operation control module, configured to control the movable robot to perform a second response operation for the second target object if the real-time response state based on the first response operation and the second dynamic behavior feature of the user verify that the pre-judgment interaction intention is passed.

[0015] In a third aspect, the present application further provides an electronic device, comprising: a memory, configured to store a computer software program; and a processor, configured to read and execute the computer software program, to realize the human-machine interaction method for a movable device as described above.

[0016] In a fourth aspect, the present application further provides a non-transitory computer readable storage medium, wherein the storage medium stores a computer software program, and the computer software program is executed by a processor to realize the human-machine interaction method for a movable device as described above.

[0017] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to realize the human-machine interaction method for a movable device as described above.

[0018] The human-computer interaction method for the movable device provided in the embodiment of the present application analyzes the candidate associated objects having a spatial association relationship with the first target object based on the scene structure information of the environment where the user is located, and obtains the predicted interaction intention and the corresponding second target object by combining the feature change trend of the first dynamic behavior feature, so that the subsequent interaction intention of the user is perceived in advance. After the first interaction action and the first target object are obtained, the movable robot is controlled to perform the first response operation matched with the first interaction action, and the second target object is locked in advance through the prediction, so that the idle period of waiting for the subsequent response after the first response operation is completed is avoided. If the real-time response state based on the first response operation and the second dynamic behavior feature of the user verify that the predicted interaction intention is passed, the movable robot is controlled to directly perform the second response operation for the second target object, without waiting for the user to make a complete interaction action instruction again, so that the connection time from the first response operation to the second response operation is shortened, the response lag problem caused by the response triggered by only the single action completed by the user is solved, and finally the fluency of the human-computer interaction process is improved and the interaction efficiency of the human-computer interaction in the continuous interaction scene is improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of the human-computer interaction method for the movable device provided in the embodiment of the present application;

[0020] Figure 2 is a structural diagram of the human-computer interaction device provided in the embodiment of the present application;

[0021] Figure 3 is an embodiment diagram of the electronic device provided in the embodiment of the present application;

[0022] Figure 4 is an embodiment diagram of the computer readable storage medium provided in the embodiment of the present application. DETAILED DESCRIPTION

[0023] Optionally, referring to Figure 1 , Figure 1 is a flowchart of the human-computer interaction method for the movable device provided in the embodiment of the present application, the execution subject of the human-computer interaction method for the movable device in the embodiment of the present application is the human-computer interaction device, therefore, the human-computer interaction method for the movable device includes:

[0024] Step 10, feature extraction is performed based on the continuous visual image sequence of the environment where the user is located, the first dynamic behavior feature is obtained, and human action recognition is performed based on the first dynamic behavior feature, so that the first interaction action completed by the user currently and the corresponding first target object are obtained.

[0025] Optionally, the human-computer interaction device calls the image acquisition module (such as a high-definition camera, a depth camera, etc.) equipped therewith to continuously capture the environment where the user is located, and obtain a continuous visual image sequence in a period of time. The continuous visual image sequence refers to a plurality of image frames containing information of the user and the surrounding environment, which are continuously captured in a set time interval (such as capturing one frame every 0.01 seconds).

[0026] Further, the human-computer interaction device starts a feature extraction algorithm to process the obtained continuous visual image sequence. The feature extraction algorithm in the embodiment of the present application first pre-processes each image frame, including image denoising, image enhancement, image segmentation, etc.

[0027] Among them, image denoising is to remove random noise generated in the process of shooting, such as Gaussian noise. Image enhancement is to improve the contrast and definition of the image, which is convenient for subsequent feature recognition. Image segmentation is to separate the user's body region from the background environment region in the image, and to exclude background interference.

[0028] Further, after completing the image pre-processing, the feature extraction algorithm extracts the first dynamic behavior feature for the separated user body region.

[0029] Among them, the motion trajectory of the user's limb action is obtained by tracking the coordinate position change of the key nodes (such as head, shoulder, elbow, wrist, hip, knee, ankle, etc.) of the user's limbs in the continuous image frames. For example, the displacement vector of the key nodes in adjacent frames is calculated by the optical flow method, and then the coordinates of the key nodes in each frame are connected in time sequence to form the motion trajectory of the key nodes, and then the motion trajectory of the overall limb action of the user is obtained.

[0030] The action speed is calculated according to the displacement of the key nodes in a unit of time, that is, the displacement distance of a certain key node in the time interval between adjacent two image frames is divided by the time interval, to obtain the average speed of the node in the time period, and then the speed information of multiple key nodes is integrated to obtain the overall action speed of the user's limb action.

[0031] The action amplitude is determined by calculating the relative angle change or relative distance change between the key nodes of the user's limbs, such as the range of bending angle change of the elbow joint during the action process, the straight-line distance of the arm from the initial position to the maximum extension position, etc., to measure the size of the action amplitude.

[0032] In an embodiment, the image acquisition module of the human-computer interaction device continuously captures 100 images at a frequency of one frame per 0.01 seconds in the living room of the user's home to form a continuous visual image sequence. After preprocessing, the user's body region is separated, and the motion trajectory of the user's right upper limb is extracted: the coordinates of the right shoulder key node start from (100, 200) (unit: pixels) and change to (120, 180), (140, 160)… in turn with time frames, and finally the coordinates are (180, 120) at the 100th frame; the motion speed of the right elbow key node is 5 pixels / 0.01 seconds at the 20th-30th frame, and decreases to 2 pixels / 0.01 seconds at the 50th-60th frame; the motion amplitude of the right upper limb is that the right elbow bending angle gradually decreases from the initial 120° to 60°, and then increases to 100°.

[0033] Further, the human-computer interaction device performs human motion recognition according to the first dynamic behavior feature to obtain a first interaction action completed by the user currently and a corresponding first target object, specifically as the processes of steps 101 to 105.

[0034] Step 20, based on the scene structure information of the environment where the user is located, a candidate associated object having a spatial association relationship with the first target object is analyzed, and an intention is predicted based on the candidate associated object and the feature change trend of the first dynamic behavior feature to obtain a predicted interaction intention and a corresponding second target object.

[0035] Optionally, the human-computer interaction device calls the stored scene structure information of the environment where the user is located, wherein the scene structure information can be obtained by scanning and modeling the environment in advance, including the position coordinates of each object in the environment, the object type, the relative position relationship between objects, etc., and then according to the spatial association relationship, the objects or positions having the above association relationship with the first target object are screened from the scene structure information as candidate associated objects, specifically as the processes of steps 201 to 204.

[0036] Among them, the spatial association relationship includes adjacent relationship, subordinate relationship and path continuity relationship. The adjacent relationship represents that the edge distance between two objects or two positions is less than a preset threshold. The subordinate relationship represents that one object or one position belongs to the component or accessory area of another object or another position. The path continuity relationship represents that there is an unobstructed continuous path of travel from the first target object to another object or another position.

[0037] In an embodiment, assuming that the first target object is a "water cup", the stored living room scene structure information is called, and the information shows that the objects in adjacent relationship with the "water cup" are "water cup tray" (located below the water cup, 1 cm away from the water cup), "tissue box" (located to the right of the water cup, 5 cm away from the water cup); the objects in subordinate relationship are "water cup cover" (originally covered on the water cup, currently placed beside the water cup); the positions in path continuity relationship are "water cup tray position in the extension direction of the path from the current position of the user's hand to the water cup", and the candidate associated objects are filtered out as "water cup tray", "tissue box" and "water cup cover".

[0038] Further, the human-computer interaction device combines the attribute information (such as object function, purpose, etc.) of the candidate associated object and the feature change trend (extension direction of the motion trajectory, change rate of the motion speed) of the first dynamic behavior feature, analyzes the subsequent interaction intention that the user may generate after completing the first interaction action through an intention reasoning algorithm (such as rule-based reasoning), predicts the interaction intention, and determines the object or position pointed by the intention as the second target object, specifically as the process of steps 205 to 208.

[0039] Step 30, controlling the movable robot to perform the first response operation matched with the first interaction action.

[0040] Optionally, after determining the first interaction action completed by the user at present and the corresponding first target object, the human-computer interaction device retrieves the "interaction action-response operation" mapping database stored in it. The database pre-stores the corresponding relationship between various common first interaction actions and the first response operation that the movable robot should perform, wherein the first response operation refers to the operation behavior performed by the movable robot to match the first interaction action completed by the user, which needs to match the purpose or scene demand of the first interaction action.

[0041] Further, after retrieving the first response operation corresponding to the first interaction action, the human-computer interaction device sends a control instruction to the movable robot through a wireless communication module (such as Wi-Fi, Bluetooth, 5G, etc.). The control instruction contains specific execution parameters of the first response operation, such as the type of execution action (such as moving, grabbing, rotating, etc.), the force of execution action (such as the size of clamping force when grabbing an object), the speed of execution action (such as the marching speed when moving), the target position of execution action (such as moving to a specific coordinate position near the first target object), etc.

[0042] Further, after receiving the control instruction, the movable robot starts its own execution module (such as a driving motor, a mechanical arm, a clamping device, etc.), and performs the first response operation according to the execution parameter in the control instruction. In the process of the movable robot performing the first response operation, the man-machine interaction device also monitors the execution progress and execution state of the first response operation in real time through image acquisition module or sensor data (such as position sensor, force sensor, etc.) feedback by the movable robot.

[0043] In an embodiment, if it is determined that the first interaction action is "reaching out to the water cup", and the first target object is "water cup". The man-machine interaction device retrieves the "interaction action-response operation" mapping database, and finds that the first response operation corresponding to the first interaction action "reaching out to the object" is "the movable robot moves to the vicinity of the object, and adjusts its posture for subsequent operation".

[0044] Further, a control instruction is sent to the movable robot, and the instruction content includes: moving to the position (coordinates (X1, Y1, Z1), wherein X1 and Y1 are plane coordinates, and Z1 is height coordinate) of "water cup" on the tea table in the living room 10 cm away; the moving speed is set to 0.2 m / s; after reaching the target position, the mechanical arm is adjusted to the stretched state, and the clamping device at the end of the mechanical arm is opened, preparing for the subsequent possible grabbing operation.

[0045] Further, after receiving the instruction, the movable robot drives the driving motor at the bottom thereof, and moves to the position (X1, Y1, Z1) at the set speed. In the moving process, the movable robot acquires the current position information in real time through the position sensor thereof, compares the current position information with the target position, and continuously adjusts the moving direction, so as to ensure accurate arrival at the target position. When reaching the position 10 cm away from "water cup", the movable robot controls the mechanical arm motor to operate, stretches the mechanical arm to the preset angle, and controls the clamping device motor to drive the clamping claw to open, and the first response operation is completed. The man-machine interaction device observes the action of the movable robot in real time through the image acquisition module, and confirms that the first response operation is executed.

[0046] In step 40, if the predicted interaction intention is verified to be passed based on the real-time response state of the first response operation in combination with the second dynamic behavior characteristic of the user, the movable robot is controlled to perform a second response operation for a second target object.

[0047] Optionally, the human-machine interaction device acquires the real-time response state of the first response operation, which refers to the operation execution related information fed back by the movable robot through its own various sensors (such as position sensors, force sensors, vision sensors, etc.) during and after the execution of the first response operation, including whether the movable robot has reached the specified position, whether the mechanical arm has been adjusted to the preset posture, whether the clamping device has been opened or closed according to the requirements, whether obstacles are encountered during the operation execution (such as encountering tables and chairs during movement), the completion progress of the operation execution (such as how much of the total distance has been moved, how much of the total angle has been stretched by the mechanical arm), etc.

[0048] The human-machine interaction device continuously receives these sensor feedback data through the real-time communication link between the human-machine interaction device and the movable robot, thereby grasping the execution of the first response operation in real time.

[0049] Further, the human-machine interaction device needs to acquire the second dynamic behavior feature of the user. The second dynamic behavior feature refers to the new dynamic behavior feature generated by the user's body movement during and after the execution of the first response operation by the movable robot, and its feature type is consistent with the first dynamic behavior feature, which also includes the movement trajectory, action speed and action amplitude of the user's body movement. The process of acquiring the second dynamic behavior feature is similar to that of acquiring the first dynamic behavior feature: the image acquisition module of the human-machine interaction device continues to take continuous pictures of the user, and acquires the continuous visual image sequence in this period of time, and then through the same feature extraction algorithm, after pre-processing and human body region separation of the image sequence, the new body movement trajectory, action speed and action amplitude of the user are extracted to form the second dynamic behavior feature.

[0050] Further, the human-machine interaction device starts the intention verification algorithm to combine the real-time response state of the first response operation with the second dynamic behavior feature of the user, and verifies the updated pre-judgment interaction intention ("pick up the cup cover") in step 20. In the verification process, the algorithm will first judge whether the real-time response state of the first response operation meets the operation prerequisite requirements corresponding to the pre-judgment interaction intention "pick up the cup cover" - since the cup cover is a second target object, its size is usually smaller than that of the cup, and it may be located beside the cup (such as 5 cm to the right of the cup on the tea table), so the prerequisite requirements include "the movable robot has reached the vicinity of the cup and the mechanical arm posture can cover the position where the cup cover is located" and "the clamping device has been opened to a width suitable for the size of the cup cover"; then, it is analyzed whether the second dynamic behavior feature is consistent with the subsequent action feature that the user may generate under the "pick up the cup cover" intention, for example, the user's hand may fine-tune the trajectory to the cup cover direction, the action amplitude may be more delicate due to the small size of the cup cover, and the action speed may remain flat to ensure accurate pointing, etc.

[0051] If the real-time response state of the first response operation meets the prerequisite requirement, and the matching degree between the second dynamic behavior feature and the pre-judgment action feature exceeds the set threshold (such as 80%), it is determined that the pre-judgment interaction intention verification is passed; otherwise, if the real-time response state does not meet the prerequisite requirement (such as the mechanical arm does not cover the water cup cover position), or the matching degree between the second dynamic behavior feature and the pre-judgment action feature is lower than the threshold (such as the user's hand is still pointing to the water cup instead of the water cup cover), it is determined that the verification is not passed.

[0052] Further, if the pre-judgment interaction intention (“pick up the water cup cover”) verification is passed, the human-computer interaction device generates a second response operation control instruction for the second target object “water cup cover”. The “second response operation” needs to adapt to the attributes of the water cup cover (small size, light weight, may need to be accurately grasped to avoid slipping), and specifically is “the mobile robot adjusts the mechanical arm posture to the water cup cover, controls the clamping device to grasp the water cup cover with appropriate force, and moves to a position where the user's hand can reach”. The control instruction needs to clearly adapt to the execution parameters of the water cup cover: such as the clamping device opening width (such as 3 centimeters, adapting to the common water cup cover diameter), the clamping force (such as 2 Newton, to avoid excessive force to damage the water cup cover or too small to cause slipping), the mechanical arm moving path (straight line trajectory from the current position to the water cup cover, avoiding the water cup obstruction), the final placement position (10 centimeters in front of the user's hand, the height is level with the user's hand) and the like.

[0053] Further, the human-computer interaction device sends the control instruction to the mobile robot through the communication module. After receiving the instruction, the mobile robot starts the drive motor, the mechanical arm motor and the clamping device motor, and executes the second response operation according to the instruction parameters. During the execution process, the human-computer interaction device monitors whether the mechanical arm accurately points to the water cup cover and whether the clamping device stably grasps through the image acquisition module, and receives the force sensor feedback of the mobile robot (to confirm that the clamping force is maintained at about 2 Newton), to ensure that the operation is accurately completed; if the pre-judgment interaction intention verification is not passed, the human-computer interaction device does not send the second response operation instruction, but re-enters step 20 to predict the intention based on the current state (such as the user's hand turning to other objects), or returns to step 10 after waiting for the user's new interaction action.

[0054] In an embodiment, the first response operation is "the movable robot moves to 10 centimeters near the water cup, and adjusts the mechanical arm to the stretched state, and the clamping device is opened", the predicted interaction intention is "to pick up the water cup cover", and the second target object is "the water cup cover" (which is known to be located 5 centimeters to the right of the water cup on the tea table, with a diameter of 3 centimeters and a weight of 50 grams). Therefore, the real-time response state of the first response operation is obtained: through the position sensor feedback of the movable robot, it is confirmed that the current position of the movable robot is 10 centimeters away from the water cup, and only 12 centimeters away from the water cup cover (the stretching range of the mechanical arm can cover the water cup cover); through the attitude sensor feedback of the mechanical arm, it is confirmed that the current stretching angle of the mechanical arm can make the end clamping device move above the water cup cover (with a deviation of less than 3 centimeters); through the displacement sensor feedback of the clamping device, it is confirmed that the opening width of the clamping claw is 4 centimeters (greater than the diameter of 3 centimeters of the water cup cover, suitable for grasping requirements), and no obstacles are encountered during execution (such as no obstruction of the edge of the tea table), it is determined that the real-time response state of the first response operation meets the operation prerequisite requirements of "picking up the water cup cover".

[0055] Further, the human-computer interaction device obtains the second dynamic behavior characteristics of the user: the image acquisition module continues to shoot the user image sequence at a frequency of 0.01 seconds per frame during the execution of the first response operation by the movable robot, and extracts the motion trajectory change of the user's hand - from the coordinates (180, 120) (pixels) pointing to the water cup to (185, 122) (pixels) pointing to the water cup cover, the trajectory extension direction is clearly towards the water cup cover; the hand movement speed remains at 2 pixels / 0.01 seconds, with a change rate of only 5% (the speed is gentle, which meets the fine pointing requirement); the action amplitude is that the hand moves 5 pixels towards the water cup cover, and there is no large swing (because the size of the water cup cover is small, the action is more accurate), these characteristics together form the second dynamic behavior characteristics.

[0056] Further, the human-computer interaction device starts the intention verification algorithm: the real-time response state of the first response operation (the mechanical arm covers the water cup cover, and the clamping device is suitable) is matched with the second dynamic behavior characteristics (the hand points to the water cup cover, and the action is fine and gentle), and the matching degree is calculated to be 88% (more than the set threshold of 80%), and it is determined that the predicted interaction intention "to pick up the water cup cover" is verified.

[0057] A second response operation control instruction is generated for the second target object "cup cover", and the specific parameters are: 1. The mechanical arm moves at a speed of 0.08 m / s from the current position to the position directly above the cup cover (coordinates (X2, Y2, Z2), wherein X2 = cup X coordinate + 5 cm, Y2 = cup Y coordinate, and Z2 = height of the tea table + 2 cm); 2. The clamping device keeps a 4 cm open width, and after being aligned with the cup cover, it is closed at a speed of 0.5 cm / s until the clamping force reaches 2 N (adjusted in real time through a force sensor); 3. After the cup cover is grasped stably, the mechanical arm is lifted 3 cm upward at a speed of 0.1 m / s (to avoid collision with the tea table), and then moved to a position 10 cm in front of the user's hand (the coordinate deviation from the current coordinate of the user's hand is less than 2 cm), and the clamping state is maintained to wait for the user to take it. The human-computer interaction device sends the instruction to the movable robot through Bluetooth. After the movable robot receives the instruction: first, drive the mechanical arm to move accurately above the cup cover, and then control the clamping device to slowly close to a clamping force of 2 N after confirming the alignment through the visual sensor, and then lift and move to the front of the user. The human-computer interaction device observes that the mechanical arm successfully grasps the cup cover and delivers it to the front of the user through the image acquisition module, and receives the "operation completed" feedback signal of the movable robot, and confirms that the second response operation is completed.

[0058] The embodiment of the present application analyzes candidate associated objects having a spatial association relationship with the first target object based on scene structure information of an environment in which the user is located, and predicts an interaction intention and a corresponding second target object by combining a feature change trend of the first dynamic behavior feature, thereby realizing early perception of a subsequent interaction intention of the user. The movable robot is controlled to perform a first response operation matched with the first interaction action, and the second target object is locked in advance through prediction, thereby avoiding a dead time period in which the robot waits for the first response operation to be completely finished before preparing for a subsequent response. If the predicted interaction intention is verified to be passed based on a real-time response state of the first response operation and a second dynamic behavior feature of the user, the movable robot is directly controlled to perform a second response operation for the second target object, without the need to wait for the user to make a complete interaction action instruction again, thereby shortening a connection time from the first response operation to the second response operation, solving a response lag problem caused by a single action trigger response depending on a completed action of the user, and finally improving fluency of a human-computer interaction process and interaction efficiency of human-computer interaction in a continuous interaction scene.

[0059] Optionally, the process of steps 101 to 105 includes:

[0060] In step 101, the motion trajectory coordinate information is determined based on the start point coordinate, the end point coordinate and the coordinate point set of the trajectory, and it is judged whether the motion trajectory of the user's limb action presents a preset action trajectory form based on the motion trajectory coordinate information, to obtain a trajectory form judgment result.

[0061] Optionally, the human-computer interaction device obtains, from the extracted first dynamic behavior features, relevant data of a motion trajectory of the user's limb action, including a starting point coordinate, an ending point coordinate, and a set of coordinate points along the trajectory. The starting point coordinate refers to a two-dimensional pixel coordinate (xs, ys) of a key node (e.g., a wrist) of the user's limb in a first frame of the sequence of continuous visual images when the limb action starts, where xs represents a horizontal pixel position and ys represents a vertical pixel position. The ending point coordinate refers to a two-dimensional pixel coordinate (xe, ye) of the key node in a last frame of the sequence of continuous visual images when the limb action is completed. The set of coordinate points along the trajectory refers to a set of two-dimensional pixel coordinates of the key node in each frame of the sequence of continuous visual images between the starting point coordinate and the ending point coordinate where n is the total number of frames of the sequence of continuous visual images, is the coordinate of the key node in the (i+1)th frame of the image.

[0062] Further, the human-computer interaction device integrates the starting point coordinate, the ending point coordinate, and the set of coordinate points along the trajectory to form complete motion trajectory coordinate information. Further, the human-computer interaction device calls a preset action trajectory form database, which stores standard trajectory form parameters corresponding to various preset interactive actions (e.g., "pointing with an outstretched finger", "waving", "grabbing", etc.), including a relative position relationship between standard starting points and ending points, a curvature range of a standard set of passing points, a length range of a standard trajectory, etc.

[0063] Further, the human-computer interaction device uses a trajectory form matching algorithm to compare and judge the motion trajectory coordinate information and the preset action trajectory form. First, the curvature of the actual motion trajectory is calculated according to the following formula:

[0064] . .

[0065] The curvature of a line segment formed by each adjacent three passing points is calculated The average value of all curvatures is calculated , which is compared with the standard curvature range of the preset action trajectory form. At the same time, the length of the actual motion trajectory is calculated according to the following formula where , The total length L of the trajectory is calculated, which is compared with the standard length range of the preset action trajectory form. In addition, it is determined whether the relative position relationship (e.g., horizontal distance, vertical distance ratio) of the starting point coordinate and the ending point coordinate meets the preset standard. If the curvature, the length, and the relative position relationship all meet the standard range of the preset action trajectory form, the trajectory form judgment result is "compliant"; otherwise, the judgment result is "non-compliant".

[0066] In one embodiment, a user makes a "pointing to a water glass" gesture in the living room. The human-computer interaction device obtains the starting coordinates (xs, ys) = (100, 200) (coordinates in the first frame image) and the ending coordinates (xe, ye) = (180, 120) (coordinates in the 100th frame image) of the right wrist (key node) movement trajectory from the first dynamic behavior features. The set of coordinate points along the trajectory is P = {(120, 180), (140, 160), ..., (170, 130)} (a total of 99 points). The human-computer interaction device calls a preset action trajectory shape database to extract the standard trajectory shape parameters of the "pointing to a glass" action: the standard curvature range is 0.002 ≤ ≤0.005 (approximately straight line trajectory), standard length range is 100≤L≤150 pixels, standard starting point and ending point relative position is horizontal distance greater than vertical distance (horizontal offset / vertical offset>1.2).

[0067] The curvature of three adjacent points along the path is calculated using a trajectory shape matching algorithm. The values ​​were 0.003, 0.004, 0.0035, etc., with average values... =0.0036, within the standard curvature range; total trajectory length L=128 pixels, within the standard length range; horizontal offset between the start and end points is 180-100=80 pixels, vertical offset is 200-120=80 pixels, horizontal offset / vertical offset=1, although slightly lower than the standard value of 1.2, but combined with the overall approximately straight trajectory and length, the trajectory shape is judged to conform to the preset trajectory shape of the "pointing finger" action, and the trajectory shape judgment result is "conforms".

[0068] Step 102: Based on the speed values ​​of the movement speed at different stages of the movement trajectory, determine the phased speed information, and based on the phased speed information, determine whether the change in the user's limb movement speed conforms to the preset movement speed law, and obtain the speed law judgment result.

[0069] Optionally, the human-computer interaction device extracts the user's limb movement speed data from the first dynamic behavior feature. This data is the average speed v1, v2, ..., vn-1 within the corresponding time period of each frame during the movement process, where vi (1≤i≤n-1) represents the average speed within the time period from the i-th frame to the (i+1)-th frame, with units of pixels / frame interval time (in this example, the frame interval time is 0.01 seconds, which can be expressed as pixels / 0.01 seconds). Further, the human-computer interaction device divides the movement process into a starting phase, a constant speed phase, and a deceleration phase according to the phase division rules of the motion trajectory.

[0070] The starting stage refers to a stage in which the speed rises from 0 to a stable speed after the action starts, and the corresponding frame range is frame 1 to frame m1 (a stage in which the speed change rate is usually greater than 0.1 pixel / (0.01 second)2 is the starting stage); the uniform speed stage refers to a stage in which the speed fluctuates within a stable range, and the corresponding frame range is frame m1+1 to frame m2 (a stage in which the absolute value of the speed change rate is less than or equal to 0.05 pixel / (0.01 second)2 is the uniform speed stage); and the deceleration stage refers to a stage in which the speed decreases from the stable speed to 0 or close to 0, and the corresponding frame range is frame m2+1 to frame n (a stage in which the speed change rate is less than -0.1 pixel / (0.01 second)2 is the deceleration stage).

[0071] According to the above stage division, the human-computer interaction device calculates the speed statistical information of each stage to form the stage-based speed information, including the maximum speed vmax1 of the starting stage, the speed rising time frame interval time, the average speed of the uniform speed stage , the uniform speed duration frame interval time, the minimum speed vmin3 of the deceleration stage, and the speed falling time frame interval time.

[0072] Further, a preset action speed rule database is called, which stores standard speed rule parameters corresponding to various preset interaction actions. For example, the standard speed rule of the “stretching hand and pointing” action is: the maximum speed of the starting stage is 4≤vmax1≤6 pixel / 0.01 second, the starting time is 0.1≤t1≤0.2 second; the average speed of the uniform speed stage is 3≤ ≤5 pixel / 0.01 second, the uniform speed time is 0.3≤t2≤0.5 second; the minimum speed of the deceleration stage is 0≤vmin3≤1 pixel / 0.01 second, and the deceleration time is 0.1≤t3≤0.2 second.

[0073] Further, the human-computer interaction device compares the stage-based speed information with the preset action speed rule parameters. If the speed statistical information of each stage is within the corresponding standard range, the speed rule judgment result is “conforms”; otherwise, the judgment result is “does not conform”.

[0074] In step 103, the action amplitude angle information is determined based on the limb activity angle range corresponding to the action amplitude, and whether the action amplitude of the user's limb action matches the preset action amplitude standard is judged based on the action amplitude angle information, to obtain the amplitude standard judgment result.

[0075] Optionally, the human-computer interaction device obtains the action amplitude data of the user's limb action from the first dynamic behavior feature, which is expressed in terms of the limb activity angle range. For different limb joints (such as the elbow and the shoulder), the joint angle change is calculated through the key node coordinates.

[0076] Take the elbow joint as an example, select the shoulder key node A (xA, yA), the elbow key node B (xB, yB), and the wrist key node C (xC, yC), and calculate the elbow joint angle θ by vector method.

[0077] During the action process, the elbow joint angle is calculated once for each frame of image, and the angle sequence θ1, θ2, …, θn is obtained, where is the elbow joint angle in the i-th frame of image. The human-computer interaction device calculates the maximum value θmax and the minimum value θmin of the angle sequence, and the difference between the two is the activity angle range of the elbow joint , which is taken as the action amplitude angle information. If the action involves multiple joints (such as the shoulder and the elbow), the activity angle range of each joint is calculated respectively, and the maximum activity angle range is taken as the overall action amplitude angle information.

[0078] Subsequently, a preset action amplitude standard database is called, which stores the standard amplitude angle range corresponding to each type of preset interactive action, such as the standard amplitude angle range (elbow joint) of the “stretching fingers” action is 30 ≤ Δθ ≤ 60 (the arm gradually extends from the bent state). The human-computer interaction device compares the action amplitude angle information with the preset action amplitude standard, and if it is within the standard range, the amplitude standard judgment result is “conform”; otherwise, the judgment result is “not conform”.

[0079] Optionally, when the action involves multiple joints, the overall amplitude conformity is calculated by weighting, and the formula is: .

[0080] wherein, is the action amplitude comprehensive conformity (the value range is [0, 1], ≥ 0.7 is determined as “conform”); m is the number of joints participating in the action (such as the elbow + shoulder, m = 2); is the weight of the j-th joint (such as the elbow = 0.7, the shoulder = 0.3, and the main active joint is highlighted); is the actual activity angle range of the j-th joint, , is the upper and lower limits of the preset amplitude range of the joint.

[0081] Step 104, if the trajectory form judgment result, the speed law judgment result, and the amplitude standard judgment result all meet the determination conditions of the same preset interactive action, the preset interactive action is determined as the first interactive action completed by the user at present.

[0082] Optionally, the human-computer interaction device integrates the trajectory form determination result of step 101, the speed rule determination result of step 102, and the amplitude standard determination result of step 103 to form a set of three types of determination results {R1, R2, R3, wherein R1, R2, and R3 take values of “comply” or “not comply”.

[0083] Further, the human-computer interaction device calls a preset interaction action determination rule library, which stores the “three results simultaneously comply” determination conditions corresponding to each preset interaction action (such as “stretching fingers to point”, “waving hands”, and “grabbing”) in the library, that is, the effective determination of a certain preset interaction action needs to meet the determination results of the trajectory form standard, the speed rule standard, and the amplitude standard corresponding to the preset interaction action.

[0084] For example, the determination condition of the “stretching fingers to point” action is “R1= comply and R2= comply and R3= comply”, and the determination condition of the “waving hands” action is “its exclusive trajectory form determination result= comply, exclusive speed rule determination result= comply, and exclusive amplitude standard determination result= comply”.

[0085] Further, the human-computer interaction device traverses each action in the preset interaction action determination rule library and verifies whether the determination condition of the action matches the current {R1, R2, R3} set one by one. If there is only one preset interaction action whose determination condition of “trajectory form comply”, “speed rule comply”, and “amplitude standard comply” is completely consistent with the current three types of determination results, the preset interaction action is determined as the first interaction action completed by the user currently; if there are multiple preset interaction actions that meet the determination condition, the human-computer interaction device further calls the action priority rule (such as “the pointing action has a higher priority than the waving hands action without a clear target”) to select the action with the highest priority as the first interaction action; if there is no preset interaction action that meets the determination condition, the result of “no effective interaction action is recognized” is returned, and the feature determination process of steps 101-103 is triggered again (or the user is prompted to make a clear action again).

[0086] Step 105, based on the action type of the first interaction action and the end point coordinates of the motion trajectory, the object or position corresponding to the end point coordinates is located in the visual image sequence to obtain the first target object.

[0087] Optionally, the human-computer interaction device locates the object or position corresponding to the end point coordinates in the visual image sequence based on the action type of the first interaction action and the end point coordinates to obtain the first target object, specifically as the process of steps 1051 to 1054.

[0088] The embodiment of the present application realizes accurate conversion from user limb dynamic characteristics to "explicit interaction intention carrier" by independently judging and jointly matching the standard three-dimensional characteristics of trajectory form-speed law-amplitude, combining the target positioning of action type and trajectory endpoint, accurately identifying the interaction action completed by the user and the corresponding target object, and providing accurate initial interaction information for subsequent scene correlation analysis and robot response control.

[0089] Optionally, the process of steps 1051 to 1054 includes:

[0090] Step 1051, a single-frame visual image containing endpoint coordinates is extracted from a continuous visual image sequence to obtain a target positioning reference image, and a region search is performed in the target positioning reference image according to the target positioning rule corresponding to the action type to obtain a target search area range.

[0091] Optionally, the human-computer interaction device filters out a single-frame visual image containing motion trajectory endpoint coordinates from the continuous visual image sequence obtained in step 10. The single-frame visual image containing the motion trajectory endpoint coordinates refers to the frame image in which the pixel point corresponding to the motion trajectory endpoint coordinates (i.e. (xe, ye) mentioned in step 101) at the completion moment of the user's limb action is located within the frame image range, which is determined as the target positioning reference image.

[0092] Further, the human-computer interaction device calls a preset "action type-target positioning rule" mapping library, which stores target positioning rules corresponding to different first interaction action types. The "action type corresponding target positioning rule" refers to the rule for limiting the search area range of the target object in the target positioning reference image, which is set according to the functional attributes of the interaction action (such as pointing, grabbing, touching, etc.). For example, if the first interaction action type is "stretching hand to point", the corresponding target positioning rule is to expand a circular search area with a radius of 20-30 pixels in the direction away from the user's limb with the motion trajectory endpoint coordinates as the center (because the target of the pointing action is usually in the direction of the finger extension).

[0093] If the action type is "grabbing", the corresponding target positioning rule is "expanding a square search area with a side length of 15-25 pixels in the direction of the user's palm closure with the motion trajectory endpoint coordinates as the center" (because the target of the grabbing action is usually within the range that the palm can cover).

[0094] Further, the human-computer interaction device extracts the corresponding target positioning rule from the mapping library according to the determined first interaction action type, and then divides the area in the target positioning reference image according to the rule to obtain the target search area range, which is expressed in the form of pixel coordinate range.

[0095] Step 1052, based on the target search area range, extract image features of all objects or positions in the target positioning reference image, obtain the image feature set in the region, and remove the image features belonging to the user's own limbs in the image feature set in the region, to obtain the non-limb image feature set.

[0096] Optionally, the human-computer interaction device extracts image features of all objects or positions in the target positioning reference image in the region based on the target search area range obtained in step 1051, to form the image feature set in the region. Image features refer to information capable of representing the visual attributes of objects or positions, including color features (such as RGB color histogram, HSV color space distribution), texture features (such as gray level co-occurrence matrix, LBP local binary pattern), shape features (such as edge contour, circularity, rectangularity), and depth features (if the image acquisition module is a depth camera, it also includes the actual spatial depth value corresponding to each pixel point in the region); the image feature set in the region refers to the image features of all identifiable objects or positions in the target search area range, wherein Fi (1≤i≤k) represents the image features of the i-th object or position in the region, and k is the number of objects or positions initially recognized in the region.

[0097] Further, the human-computer interaction device needs to remove the image features belonging to the user's own limbs from the image feature set in the region. For this purpose, the image features of the user's body region separated in step 10 (i.e. the color, texture, and shape features of the user's body region obtained by image segmentation when the first dynamic behavior features are extracted in step 10) are called as "limb feature template". Through a feature matching algorithm, the similarity of each Fi in the image feature set in the region to the limb feature template is calculated, and if the similarity exceeds a set threshold (such as 85%, indicating that the feature belongs to the user's limbs), the Fi is removed from the image feature set in the region.

[0098] Finally, the remaining image features constitute the non-limb image feature set. The non-limb image feature set refers to the set of image features of objects or positions in the environment in the image feature set in the region, excluding the user's own limb features wherein, represents the image features of the j-th non-limb object or position in the region, and m is the number of non-limb objects or positions (m≤k).

[0099] Step 1053, determine the object or position corresponding to the target image feature of a single object or a single position in the non-limb image feature set as the candidate target object.

[0100] Optionally, the human-computer interaction device performs object / position correlation analysis on the non-limb image feature set obtained in step 1052 to determine whether the object or position corresponding to each image feature in the set is a "single object or single position". A single object or single position refers to an object (such as a single cup, a single remote control) or a position (such as a specific point on a tea table, a marker point on the ground) that has an independent visual boundary and is not visually connected to other objects / positions within the target search area range. The image features of the object or position have uniqueness and independence, and there is no overlap or correlation with other features.

[0101] Specifically, the human-computer interaction device performs boundary detection on the pixel region corresponding to the non-limb image feature in the target search area range by using an image semantic segmentation algorithm. If the pixel region corresponding to a certain image feature F'j has a complete closed boundary, and the pixels within the boundary belong only to the feature (without other features mixed in), and the region does not overlap with the pixel regions corresponding to other features, it is determined that the object or position corresponding to F'j is a "single object or single position", and it is determined as a target image feature.

[0102] Subsequently, the objects or positions corresponding to all target image features are summarized to form a candidate target object set. A candidate target object refers to an object or position within the target search area range that meets the "single object or single position" condition and may be a potential target of a user interaction action , Oq (1≤q≤p) represents the qth candidate target object, and p is the number of candidate target objects (p≤m).

[0103] In step 1054, the first target object is determined based on the adaptability between the first attribute information of each candidate target object and the action type of the first interaction action.

[0104] Optionally, the human-computer interaction device obtains the first attribute information of each candidate target object. The first attribute information refers to attribute data that can represent the adaptability of the function, form, and purpose of the candidate target object to the interaction action, including object type (such as cup, tissue box, book), object function (such as drinkable, wipeable, readable), object form (such as hard, soft, liquid containment type), position attribute (such as table position, ground position, high position), etc. It can be obtained by calling a pre-set "image feature-object attribute" mapping library (which stores the association data of common object image features and corresponding attributes).

[0105] Further, the human-computer interaction device calls a pre-set "action type-attribute adaptation rule" library, which stores the adaptation standards of different first interaction action types and object / position attributes.

[0106] The adaptability refers to the degree of matching of the first attribute information of the candidate target object with the functional requirement of the first interaction action type, for example, the "stretching hand and pointing" action is usually matched with the object attribute of "being operable and usable" (such as a water cup being drinkable and a remote controller being controllable), and the adaptability with the object attribute of "being inoperable" (such as a wall surface and a floor decoration) is low; the "grabbing" action is usually matched with the object attribute of "having a moderate volume and a light weight and having a grabbing point" (such as a water cup and a mobile phone), and the adaptability with the object attribute of "having an excessively large volume and having no grabbing point" (such as a tea table and a sofa) is low.

[0107] The human-computer interaction device quantifies the adaptability degree of the first attribute information of each candidate target object Oq with the first interaction action type through the adaptability calculation algorithm, selects the candidate target object with the highest adaptability degree, and determines the candidate target object as the first target object. If there are multiple candidate target objects with the same adaptability degree and the highest adaptability degree, the motion trajectory end point coordinates and the pixel distances of the candidate target objects are further combined (the closest distance is selected), and finally the unique first target object is determined.

[0108] The embodiment of the application realizes the accurate screening of the unique first target object corresponding to the user interaction action from the complex environment image through the progressive target positioning logic of the reference image positioning-non-limb feature screening-single target screening-attribute adaptability judgment, and realizes the accurate association of the action and the target.

[0109] Optionally, the process of steps 201 to 204 comprises:

[0110] In step 201, the position coordinates of all objects and the spatial identification information of all positions are extracted from the scene structure information, and the environmental object position set and the environmental spatial identification set are obtained.

[0111] Optionally, the human-computer interaction device calls the scene structure information of the environment in which the user is located, which refers to the structured data set containing the spatial positions, shapes, categories of all objects in the environment and the identification, functions and the like of all physical positions, and the association relationship between the objects and the positions (such as "the water cup is on the tea table" and "the sofa is close to the tea table") obtained through the early environment scanning (such as laser radar scanning, depth camera modeling and the like).

[0112] Further, the human-computer interaction device performs data extraction and classification processing on the scene structure information, and separates two types of data including the position coordinates of all objects and the spatial identification information of all positions.

[0113] Wherein, the position coordinates of all objects refer to the three-dimensional coordinate values of each identifiable object (such as a cup, a tea table, a sofa, a tissue box, etc.) in a pre-set spatial coordinate system (such as a three-dimensional rectangular coordinate system with the corner of the room as the origin, the X-axis along the horizontal direction, the Y-axis along the vertical direction, and the Z-axis along the depth direction) in the environment, and the coordinate values are accurate to the centimeter level, used to represent the specific position of the object in space.

[0114] The spatial identification information of all positions refers to the exclusive identification and functional description corresponding to each physical position (such as the surface of a tea table, the ground beside a sofa, the area of a dining table, the position of a window sill, etc.) in the environment. The spatial identification can be a custom number (such as "Position 001" "Position 002") or a semantic name (such as "tea table surface position" "sofa right side ground position"), and the "functional description" refers to the purpose of the position (such as "for placing objects" "for people to sit" "for temporary placement of objects").

[0115] Further, the human-computer interaction device arranges the extracted position coordinates of all objects into an ordered data set according to object categories (such as "container category" "furniture category" "daily necessities category"), forming an environment object position set. The environment object position set refers to a structured data set containing the names, categories, and corresponding three-dimensional position coordinates of all objects in the environment. Each data in the set contains three core fields: "object name-object category-three-dimensional coordinates (X coordinate value, Y coordinate value, Z coordinate value)".

[0116] At the same time, the spatial identification information of all positions extracted is arranged into an ordered data set according to the position function category (such as "placement type position" "activity type position" "storage type position"), forming an environment spatial identification set. The environment spatial identification set refers to a structured data set containing the spatial identification, functional description, and corresponding spatial range (such as the spatial range of "tea table surface position" is X-axis 50-150 cm, Y-axis 70-80 cm, Z-axis 0-100 cm) of all physical positions in the environment.

[0117] Step 202, based on the second attribute information of the first target object, the position coordinates or spatial identification corresponding to the first target object are located in the environment object position set and the environment spatial identification set, and the target object spatial information is obtained.

[0118] Optionally, the human-computer interaction device obtains the second attribute information of the first target object.

[0119] The second attribute information of the first target object refers to feature information that can uniquely identify the first target object and matches the object or location attribute in the scene structure information, including the object category of the first target object (such as “the glass cup belongs to the container subcategory in the daily necessities category”), the object name (such as “glass cup”), the object morphological feature (such as “height 15 cm, diameter 8 cm, transparent material”), and the physical location to which the first target object belongs (such as “located on the tea table surface”), which is obtained and stored synchronously when the first target object is determined in step 1054.

[0120] Further, the human-computer interaction device performs matching retrieval in the environment object location set and the environment space identifier set obtained in step 201 respectively with the second attribute information as the retrieval condition. In the environment object location set, the object entry that completely matches the first target object is found by comparing the “object name” and “object category” fields with the corresponding content in the second attribute information, and the three-dimensional location coordinates corresponding to the entry are extracted; in the environment space identifier set, the corresponding location entry is found by comparing the “space identifier” and “function description” with the description of the physical location to which the first target object belongs (such as “tea table surface”), and the space identifier and space range corresponding to the entry are extracted.

[0121] Further, the human-computer interaction device integrates the three-dimensional location coordinates, the corresponding space identifier, and the space range of the first target object retrieved to form the target object space information. The target object space information refers to an information set that can completely represent the location and the physical region to which the first target object belongs in the environment space, including three core parts: the three-dimensional location coordinates of the first target object itself, the space identifier of the physical location to which the first target object belongs, and the space range of the physical location to which the first target object belongs.

[0122] In step 203, based on the target object space information, the location coordinates of all objects and the space identifier information of all locations within a preset distance range centered on the first target object are extracted from the scene structure information to obtain the target object surrounding space information.

[0123] Optionally, the human-computer interaction device determines the preset distance range. The preset distance range refers to a spatial distance range centered on the first target object, which is set according to the type of the first target object and the common environment interaction scene, and is used to define the spatial boundary for searching for surrounding associated objects and locations. For example, if the first target object is a “cup” (small daily necessities), the preset distance range is set as a “spherical space range with a radius of 50 cm”; if the first target object is a “sofa” (large furniture), the preset distance range is set as a “spherical space range with a radius of 150 cm”, and the numerical value of the preset distance range is stored in the “target type-preset distance” mapping table of the human-computer interaction device and can be adjusted according to the actual scene.

[0124] Further, the human-computer interaction device takes the three-dimensional position coordinates of the first target object itself in the target object space information obtained in step 202 as the center of a sphere, and takes the radius of the preset distance range as the radius to construct a three-dimensional spherical space region, which is a preset distance range centered on the first target object.

[0125] Further, the human-computer interaction device filters out object entries of all objects whose three-dimensional position coordinates are within the three-dimensional spherical space region from the set of environmental object positions obtained in step 201, and extracts the object names, object categories and three-dimensional position coordinates of these entries.

[0126] Meanwhile, the human-computer interaction device filters out position entries whose spatial ranges overlap with the three-dimensional spherical space region (i.e., part or all of the space of the position is within the spherical region) from the set of environmental space identifiers, and extracts the space identifiers, function descriptions and spatial ranges of these entries.

[0127] Further, the human-computer interaction device integrates the surrounding object information (object name, category, three-dimensional coordinates) and the surrounding position information (space identifier, function description, spatial range) filtered and extracted to form target object surrounding space information. The target object surrounding space information refers to a set of position information of all objects within the preset distance range of the first target object and identifier information of all positions, which does not include the information of the first target object itself, but only includes the surrounding associated object and position information.

[0128] Step 204, based on the adjacent relationship, the subordinate relationship and the path continuity relationship, the target object space information and the target object surrounding space information are analyzed to obtain a candidate associated object.

[0129] Optionally, the human-computer interaction device analyzes the target object space information and the target object surrounding space information according to the adjacent relationship, the subordinate relationship and the path continuity relationship to obtain a candidate associated object, specifically as the processes of steps 2041 to 2044.

[0130] Through the process of environmental data extraction-target space positioning-surrounding range screening-association relationship determination, the embodiment of the application can accurately screen out a candidate associated object that has a spatial association with the first target object from the overall environmental scene, avoiding irrelevant target interference with the intention judgment.

[0131] Optionally, the processes of steps 2041 to 2044 include:

[0132] Step 2041, based on the spatial parameters of each object and each position in the target object space information and the target object surrounding space information, a spatial comparison analysis is performed, and objects or positions that meet the adjacent relationship are determined as an adjacent relationship candidate object set.

[0133] Optionally, the human-computer interaction device extracts the spatial parameters of the first target object from the target object space information, and extracts the spatial parameters of each surrounding object and each surrounding location from the target object surrounding space information. The spatial parameters refer to key data capable of representing the spatial boundaries and locations of objects or locations. For objects, the spatial parameters include the three-dimensional location coordinates (X coordinate value, Y coordinate value, Z coordinate value) of the objects, the three-dimensional dimensions (length, width, height) of the objects, and the spatial boundary coordinates of the objects (calculated from the three-dimensional location coordinates and the three-dimensional dimensions, such as the front end boundary X coordinate of the object = the X coordinate of the object + length / 2, the rear end boundary X coordinate of the object = the X coordinate of the object - length / 2, the left side boundary Y coordinate of the object = the Y coordinate of the object + width / 2, the right side boundary Y coordinate of the object = the Y coordinate of the object - width / 2, the top boundary Z coordinate of the object = the Z coordinate of the object + height / 2, and the bottom boundary Z coordinate of the object = the Z coordinate of the object - height / 2); for locations, the spatial parameters include the three-dimensional center coordinates (X coordinate value, Y coordinate value, Z coordinate value) of the locations and the spatial range boundary coordinates of the locations (such as the front end boundary X coordinate of the location, the rear end boundary X coordinate of the location, the left side boundary Y coordinate of the location, the right side boundary Y coordinate of the location, the top boundary Z coordinate of the location, and the bottom boundary Z coordinate of the location).

[0134] Further, the human-computer interaction device starts a spatial comparison analysis process, and calculates the edge distance of each surrounding object and each surrounding location from the first target object. The edge distance refers to the shortest straight line distance between the spatial boundaries of the first target object and the spatial boundaries of the surrounding objects / locations. The calculation method is as follows: the boundary distances of the first target object and the surrounding objects / locations in the X-axis direction, the Y-axis direction, and the Z-axis direction are calculated respectively, and the maximum value of the three direction boundary distances is taken as the final edge distance (if there is an overlap between the boundaries of the two in a certain direction, the boundary distance in that direction is taken as 0). For example, in the X-axis direction, the boundary distance = max (the rear end boundary X coordinate of the first target object - the front end boundary X coordinate of the surrounding object / location, the rear end boundary X coordinate of the surrounding object / location - the front end boundary X coordinate of the first target object, 0); the boundary distances in the Y-axis direction and the Z-axis direction are calculated in the same way, and the final edge distance is the maximum value of the three direction boundary distances.

[0135] Further, the human-computer interaction device calls a preset adjacency relationship threshold, the preset threshold refers to an edge distance critical value for determining whether the adjacency relationship is established according to the environmental interaction scene, and is usually set to 5-15 cm (for example, 10 cm), and the specific value is stored in a spatial correlation relationship determination parameter library of the human-computer interaction device. If the edge distance between a certain peripheral object or peripheral position and the first target object is less than the preset threshold, it is determined that the peripheral object or peripheral position meets the adjacency relationship, and is included in the adjacency relationship candidate object set. The adjacency relationship candidate object set refers to a structured data set composed of all peripheral objects and peripheral positions meeting the adjacency relationship with the first target object, and each piece of data in the set includes three core fields of “object name-object type (object / position)-edge distance from the first target object”.

[0136] In step 2042, attribute comparison and analysis are performed based on the attributes of each object and each position in the target object spatial information and the target object peripheral spatial information, and objects or positions meeting the subordinate relationship are determined as the subordinate relationship candidate object set.

[0137] Alternatively, the human-computer interaction device extracts the attributes of the first target object from the target object spatial information, and extracts the attributes of each peripheral object and each peripheral position from the target object peripheral spatial information. The attributes of the first target object include the type (such as “object” or “position”) of the first target object, the category (such as “container category” or “furniture category” for object category, and “placement type position” or “active type position” for position category), the function description (such as “liquid containing” or “supporting articles” for object function, and “placing small articles” or “standing for people” for position function), and the information of the superior object to which it belongs (such as “whether it belongs to a component part of a certain furniture” or “whether it belongs to an attached position of a certain area”). The attributes of the peripheral object / position are consistent with the attributes of the first target object, and also include the type, category, function description, and information of the superior object to which it belongs, and are synchronously obtained from the environmental object position set and the environmental space identifier set in step 201.

[0138] Further, the human-computer interaction device determines whether the surrounding objects / positions and the first target object meet the subordinate relationship. The specific determination logic is divided into two cases: in the first case, if the first target object is an object and the surrounding object is an object, it is determined whether the "belonging superior object information" of the surrounding object contains the name of the first target object, and whether the function of the surrounding object is to assist the first target object to realize the core function (for example, the first target object is a "water cup", the surrounding object is a "water cup cover", the belonging superior object information of the "water cup cover" is a "water cup", and the function is to "seal the water cup and prevent liquid overflow", which assists the "water cup" to realize the core function of "containing liquid"); if the surrounding object is a position, it is determined whether the "belonging superior object information" of the position contains the name of the first target object, and whether the position is the exclusive placement position of the first target object (for example, the first target object is a "water cup", the surrounding position is a "water cup exclusive placement position", the belonging superior object information of the position is a "water cup", and the function is to "place the water cup", which is determined to meet the subordinate relationship). In the second case, if the first target object is a position and the surrounding object is an object, it is determined whether the "belonging superior object information" of the object contains the name of the first target object, and whether the object is placed in the position for a long time (for example, the first target object is a "tea table surface position", the surrounding object is a "remote control on the tea table", the "remote control" has the "tea table surface position" as the belonging superior object information, and is placed in the position for a long time, which is determined to meet the subordinate relationship); if the surrounding object is a position, it is determined whether the "belonging superior object information" of the surrounding position contains the name of the first target object, and whether the surrounding position is a subdivision area of the first target object (for example, the first target object is a "living room area position", the surrounding position is a "living room tea table side position", the "living room tea table side position" has the "living room area position" as the belonging superior object information, and is a subdivision area of the "living room area position", which is determined to meet the subordinate relationship).

[0139] Further, if a certain surrounding object or surrounding position meets the determination logic of the subordinate relationship through the above attribute comparison and analysis, it is included in the subordinate relationship candidate object set. Therefore, the subordinate relationship candidate object set refers to a structured data set composed of all surrounding objects and surrounding positions that meet the subordinate relationship with the first target object, and each data in the set contains three core fields of "object name-object type (object / position)-subordinate relationship determination basis (such as 'the belonging superior object is the first target object' and 'the function assists the first target object')".

[0140] In step 2043, the path continuity relationship candidate object set is determined based on the obstacle distribution information in the scene structure information and the path continuity relationship between the first target object and the surrounding space information of the target object.

[0141] Optionally, the human-computer interaction device extracts obstacle distribution information from the scene structure information, which refers to the spatial parameters (three-dimensional position coordinates, three-dimensional dimensions, spatial boundary coordinates) and location passage restriction information (such as “no passage area” and “personnel-only passage area”) of the objects recorded in the scene structure information that will hinder the passage path.

[0142] Further, the theoretical straight-line path is planned in the three-dimensional space with the spatial boundary of the first target object as the starting point and the spatial boundary of each peripheral object / location as the ending point; then, it is judged whether the theoretical straight-line path overlaps with the spatial boundary of the obstacle in the obstacle distribution information (i.e., whether the path is blocked by the obstacle), and if there is no overlap and the location passed by the path has no passage restriction, it is determined that the peripheral object / location meets the path continuity relationship with the first target object; finally, all the peripheral objects / locations meeting the path continuity relationship are included in the path continuity relationship candidate object set, and the process is specifically as steps 20431 to 20435.

[0143] In step 2044, the objects or locations repeatedly appearing in the adjacent relationship candidate object set, the subordinate relationship candidate object set, and the path continuity relationship candidate object set are removed to obtain a candidate object preliminary set, and the candidate associated object is determined based on the adaptability of the second attribute information to the attributes of each object and each location in the candidate object preliminary set.

[0144] Optionally, the human-computer interaction device performs a deduplication process on the adjacent relationship candidate object set, the subordinate relationship candidate object set, and the path continuity relationship candidate object set. Specifically, the “object name” and “object type” of all objects in the three sets are extracted, a temporary comparison table is established, and if the “object name + object type” combination of an object repeatedly appears in multiple sets (such as “water cup cover” (object) existing in both the adjacent relationship candidate object set and the subordinate relationship candidate object set), only one record of the object is retained (the record in the subordinate relationship candidate object set is retained because it contains the determination basis of “functionally assisting the first target object”), and the other repeated records are removed. After the deduplication process, the candidate object preliminary set is obtained. The candidate object preliminary set refers to a structured data set composed of all objects meeting any one of the spatial association relationships after removing the repeated objects in the three relationship candidate object sets, and each data in the set contains four core fields of “object name-object type (object / location)-association relationship type (adjacent / subordinate / path continuity)-determination basis”.

[0145] Further, the human-computer interaction device acquires second attribute information of the first target object (containing core attributes such as type, category, and function of the first target object, which are acquired synchronously in step 202) and extracts attributes of each object in the preliminary candidate object set (which are consistent with the attributes of the surrounding objects extracted in step 2042, including type, category, and function description), and starts attribute adaptability analysis.

[0146] The attribute adaptability refers to the matching degree of the attributes of the candidate object and the second attribute information of the first target object in terms of function and category. For example, the second attribute information of the first target object is “object type-water cup, category-container, and function-liquid containment”, the attributes of the candidate object “water cup cover” are “object type-water cup cover, category-container accessory, and function-sealing of the water cup”, and the two are highly matched in terms of category (container and container accessory) and function (liquid containment and sealing assistance), so the adaptability is high. The attributes of the candidate object “paper towel box” are “object type-paper towel box, category-daily necessities, and function-storage of paper towels”, and the matching degree with the category and function of the first target object is low, so the adaptability is low.

[0147] Further, the human-computer interaction device filters out objects with high adaptability to the second attribute information of the first target object from the preliminary candidate object set according to adaptability determination rules (such as “category belongs to the same subcategory under the same category” and “function has a direct auxiliary or correlation relationship”), and determines these objects as candidate associated objects. Therefore, the candidate associated object refers to an object with high attribute adaptability to the first target object, which is filtered out from the preliminary candidate object set.

[0148] The embodiment of the application can accurately filter out candidate associated objects with both spatial association and attribute adaptability to the first target object from the spatial information around the target object through the processes of three kinds of spatial association relationship, de-duplication integration, and attribute adaptability secondary filtering, and provide a precise and non-redundant associated object set for subsequent prediction of user intention in combination with dynamic behavior characteristics.

[0149] Optionally, the processes of steps 20431 to 20435 include:

[0150] In step 20431, the spatial center point coordinates of the first target object are analyzed based on the spatial information of the target object, the target object center point information is obtained, and the spatial center point coordinates of each object and the spatial center point coordinates of each position are extracted based on the spatial information around the target object, and a surrounding object center point set is obtained.

[0151] Optionally, the human-computer interaction device extracts the spatial parameters of the first target object from the target object space information, which include the three-dimensional position coordinates (X coordinate value, Y coordinate value, Z coordinate value) and three-dimensional dimensions (length, width, height) of the first target object. The spatial center point coordinates refer to three-dimensional coordinates that can represent the geometric center of an object or a location, and the calculation method is determined according to the type (object or location) of the first target object: if the first target object is an object, the X coordinate value of the spatial center point coordinates = the X coordinate value of the three-dimensional position coordinates of the object, the Y coordinate value = the Y coordinate value of the three-dimensional position coordinates of the object, and the Z coordinate value = the Z coordinate value of the three-dimensional position coordinates of the object (because the three-dimensional position coordinates of the object are usually set as the geometric center coordinates); if the first target object is a location, the spatial center point coordinates need to be calculated through the spatial range boundary coordinates of the location, i.e., the X coordinate value = (the front end boundary X coordinate value of the location + the rear end boundary X coordinate value of the location) / 2, the Y coordinate value = (the left side boundary Y coordinate value of the location + the right side boundary Y coordinate value of the location) / 2, and the Z coordinate value = (the top boundary Z coordinate value of the location + the bottom boundary Z coordinate value of the location) / 2. The human-computer interaction device integrates the calculated spatial center point coordinates of the first target object and the corresponding object identifier (the name of the first target object) to form target object center point information. The target object center point information refers to structured data containing the name of the first target object and the three-dimensional coordinates of the spatial center point.

[0152] Further, the human-computer interaction device extracts the spatial parameters of each surrounding object and each surrounding location from the target object peripheral space information, which are consistent with the dimensions of the spatial parameters of the first target object, including three-dimensional position coordinates (or spatial range boundary coordinates) and three-dimensional dimensions. The same method as calculating the spatial center point coordinates of the first target object is used to calculate the spatial center point coordinates of each surrounding object (the object type is determined directly according to the three-dimensional position coordinates of the object, and the location type is calculated according to the spatial range boundary coordinates), and each calculation result is associated with the corresponding peripheral object identifier (the name of the surrounding object or the name of the surrounding location). The human-computer interaction device organizes the identifiers of all surrounding objects and their corresponding spatial center point three-dimensional coordinates into an ordered data set to form a surrounding object center point set. The surrounding object center point set refers to a structured data set containing the names of all objects and locations around the target object and their respective spatial center point three-dimensional coordinates, which is used for subsequent end positioning of path planning.

[0153] Step 20432, for each target surrounding object center point in the surrounding object center point set, based on the target object center point information and the surrounding object center point set, connecting the target object center point and the target surrounding object center point to obtain an initial straight line path set.

[0154] Optionally, the human-computer interaction device extracts the identification of each peripheral object and the corresponding spatial center point three-dimensional coordinates from the set of peripheral object center points, and defines the spatial center point of each peripheral object as a target peripheral object center point. The target peripheral object center point refers to the spatial center point corresponding to a single peripheral object (object or location) in the set of peripheral object center points, which is the endpoint of the subsequent initial straight line path planning.

[0155] Further, the human-computer interaction device starts the path generation process, taking the first target object spatial center point coordinate in the target object center point information as the starting point, and sequentially taking each target peripheral object center point coordinate in the set of peripheral object center points as the endpoint, to construct a straight line segment in three-dimensional space. The initial straight line path refers to a straight line segment in three-dimensional space connecting the first target object spatial center point (starting point) and a single target peripheral object center point (endpoint). The mathematical expression of the straight line segment is determined by the three-dimensional coordinates of the starting point and the endpoint, and also contains the length information of the path (the straight line distance between the starting point and the endpoint).

[0156] Further, the human-computer interaction device associates each generated initial straight line path with the corresponding path identification (such as "path 001", composed of the starting point object identification + the endpoint object identification, such as "first target object - peripheral object A"), the starting point coordinate (first target object spatial center point coordinate), the endpoint coordinate (target peripheral object center point coordinate), and the path length. All the relevant information of the initial straight line paths is sorted into an ordered data set to form an initial straight line path set. The initial straight line path set refers to a structured data set containing all initial straight line paths connecting the first target object and the peripheral object and their associated information (path identification, starting point coordinate, endpoint coordinate, path length).

[0157] Step 20433, based on the obstacle distribution information, extract the spatial contour coordinates of all obstacles to obtain an obstacle contour set, and determine whether each initial straight line path in the initial straight line path set intersects with any obstacle contour in the obstacle contour set to obtain a path-obstacle intersection judgment result.

[0158] Optionally, the human-computer interaction device extracts obstacle distribution information from the scene structure information, which contains object identification (obstacle name such as "chair", "cabinet", "floor lamp", etc.), obstacle type (such as "fixed obstacle", "movable obstacle"), and corresponding spatial parameters (three-dimensional position coordinates, three-dimensional size, spatial boundary coordinates) of all obstacles in the environment. Obstacle refers to an object in the environment where the user is located that physically blocks the path of travel, and the spatial range of the obstacle overlaps with the path, causing the path to be unable to travel. Spatial contour coordinates refer to the three-dimensional coordinate set of the spatial boundary of the obstacle, including the extreme coordinates of the front end, rear end, left side, right side, top, and bottom boundaries of the obstacle (such as the maximum X coordinate of the front end boundary, the minimum X coordinate of the rear end boundary, etc.), as well as the three-dimensional coordinates of the key contour points of the obstacle surface (used to accurately determine the intersection of the path and the obstacle). The human-computer interaction device processes the spatial parameters of each obstacle, extracts its spatial contour coordinates, and associates the corresponding obstacle identification to organize an ordered data set and form an obstacle contour set. The obstacle contour set refers to a structured data set containing the names and spatial contour three-dimensional coordinates of all obstacles in the environment.

[0159] Further, the human-computer interaction device starts the intersection judgment process of the path and the obstacle, and compares each initial straight line path in the initial straight line path set with the spatial contour coordinates of each obstacle in the obstacle contour set one by one. The judgment logic is based on three-dimensional space geometric operation: first, determine the spatial range of the initial straight line path (the value interval of the straight line segment on the X, Y, Z three coordinate axes determined by the start and end coordinates), and then determine the spatial range of a single obstacle (the value interval on the X, Y, Z three coordinate axes determined by its spatial contour coordinates); if the spatial ranges of the two paths do not overlap on the three coordinate axes (i.e., there is no intersection between the value interval of the path and the value interval of the obstacle on a certain coordinate axis), then it is directly determined that the initial straight line path does not intersect with the obstacle; if there is overlap on the three coordinate axes, then further through the intersection detection algorithm of the line segment and the obstacle space model, it is determined whether the straight line segment of the initial straight line path passes through the spatial contour of the obstacle (i.e., whether there is a point on the straight line segment whose three-dimensional coordinates belong to the spatial contour range of the obstacle). Further, the human-computer interaction device records the intersection judgment result ("with intersection" or "without intersection") of each initial straight line path with all obstacles and associates the corresponding obstacle identification, organizes these results into structured data, and forms the path-obstacle intersection judgment result. The path-obstacle intersection judgment result refers to a structured data set containing the identification of each initial straight line path and the intersection state (with / without intersection) with each obstacle.

[0160] At step 20434, the initial straight line path without intersection is determined based on the intersection judgment result of the path obstacle, the initial path set without occlusion is obtained, and whether the corresponding peripheral object of each initial path without occlusion in the initial path set without occlusion is in the passable area of the environment where the user is located is judged based on the initial path set without occlusion to obtain the passable area judgment result of the peripheral object.

[0161] Optionally, the human-computer interaction device analyzes the path obstacle intersection judgment result, checks the intersection state of each initial straight line path with all obstacles in the obstacle contour set: if the intersection state of the initial straight line path with all obstacles is “no intersection”, it is determined that the initial straight line path is not blocked by any obstacle, and it is included in the initial path set without occlusion. The initial path set without occlusion refers to a structured data set containing all initial straight line paths without intersection with any obstacle in the environment and their associated information (path identifier, starting point coordinates, end point coordinates, path length). These paths are not blocked by obstacles in the physical space and have the possibility of passing through.

[0162] Further, the human-computer interaction device extracts the passable area information of the environment where the user is located from the scene structure information. The passable area refers to the space area in the environment that allows personnel or movable robots to move freely. The spatial range of the area is determined by the “area attribute” field in the scene structure information (for example, “living room central area” and “corridor area” are marked as passable areas, and “furniture internal area” and “wall area” are marked as non-passable areas), and the spatial boundary coordinates (X, Y, Z coordinate axis value interval) of the passable area are also included.

[0163] Further, the human-computer interaction device extracts the corresponding end point object (i.e. the peripheral object associated with the path, including peripheral objects or peripheral positions) of each initial path without occlusion in the initial path set without occlusion, determines the spatial range of the end point object (obtained from the spatial parameters in the target object peripheral space information, such as the spatial boundary of the three-dimensional size of the object or the spatial range boundary of the position), and judges whether the spatial range of the end point object is completely within the spatial boundary coordinate range of the passable area: if the value interval of the end point object on the X, Y, Z coordinate axis is completely contained in the value interval of the passable area, it is determined that the peripheral object is in the passable area; if the value interval of the end point object on any one coordinate axis exceeds the value interval of the passable area, it is determined that the peripheral object is not in the passable area.

[0164] Further, the human-computer interaction device records whether the corresponding surrounding object is in the passable area (‘yes’ or ‘no’) for each unobstructed initial path, associates the corresponding surrounding object identifier, arranges these results into structured data, and forms a surrounding object passable area judgment result. The “surrounding object passable area judgment result” refers to a structured data set containing the identifier of each unobstructed initial path, the corresponding surrounding object identifier, and the state of whether the surrounding object is in the passable area.

[0165] In step 20435, the target surrounding object in the passable area is determined based on the surrounding object passable area judgment result, and a path continuity relationship candidate object set is obtained.

[0166] Optionally, the human-computer interaction device filters the surrounding object passable area judgment result and extracts the records with a judgment result of ‘yes’, i.e., the unobstructed initial path records in which the corresponding surrounding object is in the passable area. The corresponding surrounding object identifier (surrounding object name or surrounding location name) is extracted from these records, and the attribute information (type, category, and function description, obtained from the target object surrounding space information) of the surrounding object is associated. Further, the human-computer interaction device performs uniqueness verification on the filtered surrounding objects. If the same surrounding object is filtered multiple times due to multiple paths (such as repeated paths caused by multiple starting point or ending point deviations), only one associated record is retained to avoid duplication. The human-computer interaction device arranges the identifier, attribute information, and corresponding unobstructed initial path information (path identifier and path length) of the finally filtered surrounding objects into an ordered data set, and forms a path continuity relationship candidate object set. The path continuity relationship candidate object set refers to a structured data set containing all surrounding objects and surrounding locations that have unobstructed continuous passable paths with the first target object and are in the passable area.

[0167] Through the analysis of center point positioning-initial path generation-obstacle intersection judgment-passable area verification-candidate object filtering, the embodiments of the present application can accurately filter the surrounding objects that have unobstructed continuous passable paths with the first target object and are in the passable area, form a path continuity relationship candidate object set, and provide an associated target basis that meets the actual passable scenario for the subsequent determination of complete candidate associated objects.

[0168] Optionally, the process of steps 205 to 208 includes:

[0169] In step 205, based on the trajectory extension pointing area corresponding to the extension direction and the attribute information of each candidate associated object, it is determined whether each candidate associated object is in the trajectory extension pointing area, and a pointing matching result is obtained.

[0170] Optionally, the human-computer interaction device extracts the extension direction of the motion trajectory from the feature variation trend of the first dynamic behavior feature. The extension direction of the motion trajectory refers to the tangent direction of the motion trajectory in the approaching completion stage of the user's limb motion, which can reflect the pointing trend of the user's motion, and is usually calculated by the coordinates of the key nodes (such as the wrist and the fingertip) in the continuous three frames of images near the end point of the motion trajectory (such as fitting a straight line by the last three frames of coordinates, and the direction of the straight line is the extension direction).

[0171] Further, based on the extracted extension direction, the human-computer interaction device divides the trajectory extension pointing area in the target positioning reference image (the image determined in step 1051). The trajectory extension pointing area refers to a fan-shaped area or a conical area formed by extending from the end point coordinate of the motion trajectory along the extension direction, and the angle range of the area is set according to the type of the motion (such as setting the angle of the pointing area of the "stretching and pointing" motion to 30° to ensure covering a reasonable range of the possible pointing of the fingers), and the length range of the area is set to extend 50-100 pixels from the end point coordinate to the extension direction (adjusted according to the image resolution and the actual size of the object in the scene), which is used to define the target range that the user's motion can point to.

[0172] Further, the human-computer interaction device extracts the attribute information of each candidate associated object, wherein the attribute information related to the pointing judgment includes the pixel coordinate range of the candidate associated object in the target positioning reference image (obtained by converting the environmental object position set and the environmental space identifier set in step 201 into image pixel coordinates), and the contour pixel boundary of the candidate associated object (used for accurately judging whether it overlaps with the pointing area).

[0173] Further, the human-computer interaction device checks whether the pixel coordinate range of each candidate associated object overlaps with the trajectory extension pointing area one by one through the image area overlap judgment algorithm: if more than 50% of the pixel coordinate range of the candidate associated object is in the trajectory extension pointing area, or the core pixel area (such as the pixel area corresponding to the geometric center of the object) of the candidate associated object is completely in the pointing area, it is determined that the candidate associated object is "in the trajectory extension pointing area", and the pointing matching result is "match"; if the pixel coordinate range of the candidate associated object does not overlap with the pointing area, or the overlapping area accounts for less than 50%, the pointing matching result is "not match".

[0174] The human-computer interaction device records the corresponding pointing matching result for each candidate associated object and associates the candidate associated object identifier to form a pointing matching result data set. The "pointing matching result" refers to a structured data set containing the candidate associated object identifier, the overlapping proportion of the pixel coordinate range and the trajectory extension pointing area, and the matching state (match / not match).

[0175] In step 206, based on the speed change type according to the speed change rate and the attribute information of each candidate associated object, the candidate associated objects matching the speed change type are screened to obtain a speed matching candidate set.

[0176] Optionally, the human-computer interaction device extracts a change rate of the action speed from the feature change trend of the first dynamic behavior feature. The change rate of the action speed refers to a degree of change of the action speed over time, and is obtained by calculating a ratio of a difference between the action speed in two continuous frames and a time interval (e.g., a difference between the speed in the nth frame and the speed in the (n-1)th frame, divided by the time interval of the two frames, to obtain the speed change rate in the time period). The speed change rate can be positive or negative. A positive value indicates that the speed increases, a negative value indicates that the speed decreases, and a zero value indicates that the speed is stable.

[0177] Further, based on the numerical distribution of the speed change rate, the human-computer interaction device determines a speed change type. The speed change type refers to a state of the action speed according to the overall trend of the speed change rate, and includes three core types: the first type is an acceleration type, which means that the speed change rate is positive for a plurality of continuous frames, and the average change rate is greater than a set acceleration threshold (e.g., 0.5 pixel / (0.01 second)2), indicating that the action speed continuously increases; the second type is a deceleration type, which means that the speed change rate is negative for a plurality of continuous frames, and the absolute value of the average change rate is greater than a set deceleration threshold, indicating that the action speed continuously decreases; and the third type is a constant speed type, which means that the absolute value of the speed change rate for a plurality of continuous frames is less than a set stability threshold, indicating that the action speed is basically stable.

[0178] Further, the human-computer interaction device extracts attribute information of each candidate associated object, wherein the attribute information related to the speed change type includes operation difficulty (e.g., “easy operation” and “difficult operation”, based on object weight, volume, and operation required force) and operation required action amplitude (e.g., “small amplitude operation” and “large amplitude operation”, based on the range of limb activity required for object function implementation) of the candidate associated object. For example, a candidate object (e.g., a water cup cover) with “easy operation” and “small amplitude operation” is usually matched with a “deceleration type” speed change (the user slows down to accurately operate when approaching the target); and a candidate object (e.g., a remote controller far away) that needs to be quickly approached can be matched with an “acceleration type” speed change.

[0179] Further, the human-computer interaction device calls a preset "speed change type-object attribute matching rule library", which stores matching relationships between different speed change types and candidate object attributes (for example, "deceleration type" matches "easy operation and small amplitude operation" attribute, "acceleration type" matches "quick approach and large amplitude operation" attribute, and "uniform speed type" matches "regular operation and medium amplitude operation" attribute). The attributes of each candidate associated object are compared one by one with the matching rules of the speed change type. If the candidate object attribute meets the matching rule corresponding to the current speed change type, it is included in the speed matching candidate set. The "speed matching candidate set" refers to a structured data set containing all candidate associated object identifiers, attribute information and matching basis that match the current action speed change type.

[0180] In step 207, based on the pointing matching result and the speed matching candidate set, the candidate associated objects that meet the conditions of being in the trajectory extension pointing area and belonging to the speed matching candidate set are screened to obtain a double matching candidate set.

[0181] Optionally, the human-computer interaction device obtains the pointing matching result of step 205 and the speed matching candidate set of step 206, and establishes a data association relationship between the two with the candidate associated object identifier as the association key.

[0182] Further, a double screening process is started: first, the candidate associated objects with a matching state of "match" are screened from the pointing matching result to obtain a pointing matching candidate subset, which contains all candidate objects in the trajectory extension pointing area; second, the intersection operation is performed between the pointing matching candidate subset and the speed matching candidate set, that is, the candidate associated objects that exist in both the pointing matching candidate subset and the speed matching candidate set are screened out. Further, the human-computer interaction device performs uniqueness verification on the screened candidate associated objects. If the same candidate associated object is screened out multiple times due to data duplication (such as multiple attribute matching entries), only one record containing complete attribute information and matching state is retained. The final screening result is arranged into an ordered data set to form a double matching candidate set.

[0183] In step 208, based on the double matching candidate set, the interaction intent is pre-judged in combination with a preset interaction intent mapping rule to obtain a pre-judged interaction intent, and the candidate associated object that has a direct association with the pre-judged interaction intent is selected from the double matching candidate set to obtain a second target object.

[0184] Optionally, the human-computer interaction device calls a preset interaction intention mapping rule library, in which a corresponding relationship of "dual matching candidate set features - pre-judgment interaction intention" is stored, wherein the dual matching candidate set features include attribute combination of the candidate object (such as attribute combination of "water cup + water cup cover"), number of the candidate object (single or multiple), association type of the candidate object and the first target object (such as subordinate association or adjacent association), and the pre-judgment interaction intention refers to a subsequent interaction purpose (such as "picking up the water cup to drink water", "opening the water cup cover" or "taking the paper towel beside the water cup") that the user is likely to generate.

[0185] For example, if the dual matching candidate set contains "water cup cover" (with attributes of "water cup subordinate accessory and easy to operate") and the first target object is "water cup", the mapping rule is "pre-judgment interaction intention = opening the water cup cover"; if the dual matching candidate set contains "paper towel box" (with attributes of "water cup adjacent object and used for wiping"), the mapping rule is "pre-judgment interaction intention = taking the paper towel".

[0186] Further, the human-computer interaction device extracts the features (attribute combination, object number and association type with the first target object) of the dual matching candidate set, performs matching retrieval in the preset interaction intention mapping rule library, and finds the unique corresponding pre-judgment interaction intention.

[0187] If there are multiple matching rules (such as the dual matching candidate set contains multiple objects corresponding to multiple intentions), the intention priority rule (such as "the intention corresponding to the subordinate relationship object has higher priority than the intention corresponding to the adjacent relationship object" or "the intention corresponding to the object with low operation difficulty has higher priority than the intention corresponding to the object with high operation difficulty") is called to select the intention with the highest priority as the final pre-judgment interaction intention.

[0188] Further, the human-computer interaction device screens the candidate associated objects in the dual matching candidate set that have direct association with the pre-judgment interaction intention. Direct association refers to the attribute of the candidate associated object being directly related to the implementation of the pre-judgment interaction intention, and the candidate associated object being the core operation object for implementing the intention. For example, if the pre-judgment interaction intention is "opening the water cup cover", the "water cup cover" is the direct associated object; if the pre-judgment interaction intention is "taking the paper towel", the "paper towel box" is the direct associated object. If there are multiple direct associated objects in the dual matching candidate set (such as there are two "water cup covers", which usually does not exist in actual scenarios, and this is an extreme case), the distance between the candidate object and the first target object is considered (the closest one is selected), and finally a unique candidate associated object is determined, which is taken as the second target object.

[0189] The embodiment of the present application can accurately predict the subsequent interaction intention of the user and the corresponding second target object based on the candidate associated object and the first dynamic behavior characteristic trend through the progressive process of pointing matching screening-speed trend matching screening-dual condition integration-regularized intention mapping, realize the conversion from the behavior characteristics + associated object to the explicit subsequent intention, and provide the decision basis for the advance response of the movable robot.

[0190] Further, the human-machine interaction device for movable equipment provided by the present application is described below, and the human-machine interaction device for movable equipment described below can be correspondingly referred to the human-machine interaction method for movable equipment described above.

[0191] Optionally, referring to Figure 2 , Figure 2 is a structural diagram of the human-machine interaction device for movable equipment provided by the present application, and the human-machine interaction device for movable equipment comprises:

[0192] The action recognition module 210 is configured to perform feature extraction based on the continuous visual image sequence of the environment where the user is located to obtain the first dynamic behavior characteristic, and perform human body action recognition based on the first dynamic behavior characteristic to obtain the first interaction action completed by the user currently and the corresponding first target object;

[0193] The intention prediction module 220 is configured to analyze the candidate associated object having a spatial association relationship with the first target object based on the scene structure information of the environment where the user is located, and perform intention prediction based on the candidate associated object and the feature change trend of the first dynamic behavior characteristic to obtain the predicted interaction intention and the corresponding second target object;

[0194] The first operation control module 230 is configured to control the movable robot to perform the first response operation matched with the first interaction action;

[0195] The second operation control module 240 is configured to control the movable robot to perform the second response operation for the second target object if the predicted interaction intention is verified to be correct based on the real-time response state of the first response operation and the second dynamic behavior characteristic of the user.

[0196] The embodiment of the present application does not need to wait for the user to make a complete interaction action instruction again, shortens the connection time from the first response operation to the second response operation, and finally improves the fluency of the human-machine interaction process and improves the interaction efficiency of the human-machine interaction in the continuous interaction scene.

[0197] Optionally, referring to Figure 3 , Figure 3 is an embodiment diagram of the electronic device provided by the embodiment of the present application. As Figure 3As shown in FIG. 4, the embodiment of the present application provides an electronic device 300, which comprises a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320, and the processor 320 implements the processes of steps 10 to 40 when executing the computer program 311.

[0198] As shown in FIG. 4, the embodiment of the present application provides an electronic device 300, which comprises a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320, and the processor 320 implements the processes of steps 10 to 40 when executing the computer program 311. Figure 4 Figure 4 As shown in FIG. 4, the embodiment of the present application provides an electronic device 300, which comprises a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320, and the processor 320 implements the processes of steps 10 to 40 when executing the computer program 311. Figure 4

[0199] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable the computer to perform the human-computer interaction method for a mobile device provided by the above-mentioned methods, which comprises the processes of steps 10 to 40.

[0200] The above-mentioned apparatus embodiments are only schematic, wherein the units as shown can or can not be physically separate, and the components as shown can or can not be physical components, i.e., can be located in one place or distributed on multiple network components. Some or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments. Those of ordinary skill in the art can understand and implement without creative labor.

[0201] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the above-mentioned embodiments are described in detail, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the above-mentioned embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.​​

Claims

1. A human-machine interaction method for a mobile device, characterized in that, The method comprises: performing feature extraction based on a continuous visual image sequence of an environment where a user is located to obtain a first dynamic behavior feature, and performing human action recognition based on the first dynamic behavior feature to obtain a first interaction action that the user has completed currently and a corresponding first target object; analyzing candidate associated objects that have a spatial correlation relationship with the first target object based on scene structure information of the environment where the user is located, and performing intention prediction based on the candidate associated objects and a feature change trend of the first dynamic behavior feature to obtain a predicted interaction intention and a corresponding second target object; controlling a movable robot to perform a first response operation that matches the first interaction action; if the predicted interaction intention is verified to be correct based on a real-time response state of the first response operation in combination with a second dynamic behavior feature of the user, controlling the movable robot to perform a second response operation for the second target object.

2. The human-machine interaction method for a mobile device according to claim 1, wherein, The feature change trend comprises an extension direction of the motion trajectory and a speed change rate of the motion speed; The intention prediction based on the candidate associated objects and the feature change trend of the first dynamic behavior feature to obtain the predicted interaction intention and the corresponding second target object comprises: determining whether each candidate associated object is in the trajectory extension pointing area based on attribute information of each candidate associated object and a trajectory extension pointing area corresponding to the extension direction to obtain a pointing matching result; screening candidate associated objects that match the speed change type based on attribute information of each candidate associated object and a speed change type of the speed change rate to obtain a speed matching candidate set; screening candidate associated objects that meet the conditions of being in the trajectory extension pointing area and belonging to the speed matching candidate set based on the pointing matching result and the speed matching candidate set to obtain a double matching candidate set; performing intention prediction based on the double matching candidate set in combination with a preset interaction intention mapping rule to obtain the predicted interaction intention, and selecting a candidate associated object that has a direct correlation with the predicted interaction intention from the double matching candidate set to obtain the second target object.

3. The human-machine interaction method for a mobile device according to claim 1, wherein, The first dynamic behavior feature comprises a motion trajectory, a motion speed and a motion amplitude of a user's limb action; The human action recognition based on the first dynamic behavior feature to obtain the first interaction action that the user has completed currently and the corresponding first target object comprises: determining motion trajectory coordinate information based on a start point coordinate, an end point coordinate and a coordinate point set that the motion trajectory passes through, and determining whether the motion trajectory of the user's limb action presents a preset motion trajectory form based on the motion trajectory coordinate information to obtain a trajectory form determination result; determining stage-by-stage speed information based on speed values of the motion speed at different stages of the motion trajectory, and determining whether the motion speed change of the user's limb action conforms to a preset motion speed rule based on the stage-by-stage speed information to obtain a speed rule determination result; determine motion amplitude angle information based on the limb activity angle range corresponding to the motion amplitude, and determine whether the motion amplitude of the user limb motion matches a preset motion amplitude standard based on the motion amplitude angle information, to obtain an amplitude standard determination result; if the trajectory form determination result, the speed law determination result, and the amplitude standard determination result all satisfy the determination condition of the same preset interaction action, the preset interaction action is determined as a first interaction action that the user has completed currently; based on the motion type of the first interaction action and the end point coordinate of the motion trajectory, locate an object or a position corresponding to the end point coordinate in the visual image sequence to obtain a first target object.

4. The human-machine interaction method for a mobile device according to claim 3, wherein, The method for locating the object or the position corresponding to the end point coordinate in the visual image sequence based on the motion type of the first interaction action and the end point coordinate of the motion trajectory to obtain the first target object comprises: extract a single-frame visual image containing the end point coordinate in the continuous visual image sequence to obtain a target positioning reference image, and perform region search in the target positioning reference image according to a target positioning rule corresponding to the motion type to obtain a target search region range; extract image features of all objects or positions in the target positioning reference image based on the target search region range to obtain an intra-region image feature set, and remove image features belonging to the user's own limbs in the intra-region image feature set to obtain a non-limb image feature set; determine an object or a position corresponding to target image features existing in the non-limb image feature set as a candidate target object; determine the first target object based on the adaptability between the first attribute information of each candidate target object and the motion type of the first interaction action.

5. The human-machine interaction method for a mobile device according to claim 1, wherein, The spatial correlation relationship comprises a neighboring relationship, a subordinate relationship, and a path continuity relationship. The method for analyzing candidate associated objects having a spatial correlation relationship with the first target object based on the scene structure information of the environment in which the user is located comprises: extract position coordinates of all objects and spatial identification information of all positions in the scene structure information to obtain an environment object position set and an environment spatial identification set; locate position coordinates or spatial identification information corresponding to the first target object in the environment object position set and the environment spatial identification set based on the second attribute information of the first target object to obtain target object spatial information; extract position coordinates of all objects and spatial identification information of all positions within a preset distance range centered on the first target object from the scene structure information based on the target object spatial information to obtain target object peripheral spatial information; analyze the target object spatial information and the target object peripheral spatial information based on the neighboring relationship, the subordinate relationship, and the path continuity relationship to obtain the candidate associated objects.

6. The human-machine interaction method for a mobile device according to claim 5, wherein, The method for analyzing the target object spatial information and the target object peripheral spatial information based on the neighboring relationship, the subordinate relationship, and the path continuity relationship to obtain the candidate associated objects comprises: performing spatial comparison analysis on the spatial parameters of each object and each position in the target object space information and the target object peripheral space information, and determining objects or positions that meet adjacent relationships as an adjacent relationship candidate object set; an adjacent relationship represents that the edge distance between two objects or two positions is less than a preset threshold; performing attribute comparison analysis on the attributes of each object and each position in the target object space information and the target object peripheral space information, and determining objects or positions that meet subordinate relationships as a subordinate relationship candidate object set; a subordinate relationship represents that one object or one position belongs to the component or accessory area of another object or another position; analyzing the passing path of each object and each position in the first target object and the target object peripheral space information based on the obstacle distribution information in the scene structure information, and determining objects or positions that meet path continuity relationships as a path continuity relationship candidate object set; a path continuity relationship represents that there is a continuous passing path without occlusion from the first target object to another object or another position; removing objects or positions that repeatedly appear in the adjacent relationship candidate object set, the subordinate relationship candidate object set, and the path continuity relationship candidate object set to obtain a preliminary candidate object set, and determining the candidate association object based on the adaptability of the second attribute information and the attributes of each object and each position in the preliminary candidate object set.

7. The human-machine interaction method for a mobile device according to claim 6, wherein, The step of determining the path continuity relationship candidate object set comprises: analyzing the spatial center point coordinates of the first target object based on the target object space information to obtain target object center point information, and extracting the spatial center point coordinates of each object and the spatial center point coordinates of each position based on the target object peripheral space information to obtain a peripheral object center point set; for each target peripheral object center point in the peripheral object center point set, connecting the target object center point and the target peripheral object center point based on the target object center point information and the peripheral object center point set to obtain an initial straight line path set; extracting the spatial contour coordinates of all obstacles based on the obstacle distribution information to obtain an obstacle contour set, and determining whether each initial straight line path in the initial straight line path set intersects with any one of the obstacle contour set to obtain a path obstacle intersection judgment result; determining the initial straight line path that does not exist intersection based on the path obstacle intersection judgment result to obtain an unoccluded initial path set, and determining whether the peripheral object corresponding to each unoccluded initial path in the unoccluded initial path set is in the passable area of the environment of the user based on the unoccluded initial path set to obtain a peripheral object passable area judgment result; determining the target peripheral object that is in the passable area based on the peripheral object passable area judgment result to obtain the path continuity relationship candidate object set.

8. A human-machine interaction device for a mobile device, characterized in that, The device is applied to the human-computer interaction method for a movable device as claimed in any one of claims 1 to 7; the device comprises: The action recognition module is configured to perform feature extraction based on the collected continuous visual image sequence of the environment where the user is located to obtain a first dynamic behavior feature, and perform human action recognition based on the first dynamic behavior feature to obtain a first interaction action completed by the user currently and a corresponding first target object; The intention prediction module is configured to analyze a candidate associated object having a spatial association relationship with the first target object based on scene structure information of the environment where the user is located, and perform intention prediction based on the candidate associated object and a feature change trend of the first dynamic behavior feature to obtain a predicted interaction intention and a corresponding second target object; The first operation control module is configured to control the movable robot to perform a first response operation matched with the first interaction action; The second operation control module is configured to control the movable robot to perform a second response operation for the second target object if the predicted interaction intention is verified to be correct based on a real-time response state of the first response operation in combination with a second dynamic behavior feature of the user.

9. An electronic device comprising: The memory is configured to store a computer software program; The processor is configured to read and execute the computer software program, and when the processor executes the computer software program, the method for human-machine interaction facing a movable device according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer readable storage medium having stored therein a computer software program, characterized in that, The computer software program is executed by the processor, and the method for human-machine interaction facing a movable device according to any one of claims 1 to 7 is implemented.

Images