Humanoid robot adaptive interaction method and system for dynamic scene

Abstract

The application discloses a dynamic scene-oriented humanoid robot adaptive interaction method and system, relates to the technical field of robot intelligent sensing and autonomous interaction, and comprises the following steps: acquiring continuous visual data and depth data in a dynamic scene where a humanoid robot is located; determining a dynamic region according to depth changes of continuous frames; and performing feature extraction and alignment processing on the visual data and the depth data by using a convolutional neural network with a time sequence consistency constraint of the dynamic region to obtain an image sequence; and for the image sequence, using an edge detection operator to extract pixel outlines of various entities in a picture to obtain a target entity outline set. The dynamic scene-oriented humanoid robot adaptive interaction method and system improve the rationality of interaction timing and interaction range determination in a dynamic scene, and enhance the adaptive interaction capability of the humanoid robot in a complex dynamic environment.

Description

Technical Field

[0001] This invention relates to the field of robot intelligent perception and autonomous interaction technology, specifically to an adaptive interaction method and system for humanoid robots in dynamic scenarios. Background Technology

[0002] Robot autonomous navigation and environmental interaction technologies are widely used in fields such as intelligent manufacturing, warehousing and logistics, and intelligent services. When performing tasks such as handling, collaboration, guidance, and assisted operation, humanoid robots need to continuously perceive the status of surrounding people, equipment, materials, and other obstacles, and adjust their behavior in a timely manner according to environmental changes. Compared with static environments, the positions and motion states of people, mobile devices, and operable objects in dynamic scenes are constantly changing. Therefore, humanoid robots not only need to identify entities in the current environment, but also need to continuously understand the changes of entities in time and space to support subsequent autonomous interactive decisions.

[0003] Existing environmental perception methods typically acquire environmental information through visual sensors, depth sensors, etc., and determine the category and location of objects within the field of view based on image recognition or object detection. However, in dynamic scenes, the movement of entities themselves and the robot can cause significant changes in continuously acquired data. Instable spatial correspondences may also exist between different types of sensor data, making it difficult to maintain consistent characteristics of the same dynamic entity in continuous data. Especially when personnel, material carts, and other objects approach, obstruct, or intersect each other, entity boundaries can easily become confused, affecting the accurate judgment of different entity outlines and spatial states. Furthermore, existing technologies often focus more on determining the category and spatial location of entities, while insufficiently considering the physical properties of the entities themselves related to robot interaction. For humanoid robots, different entities have different effects on their interaction behavior; for example, some entities can be grasped, pushed, or carried, while others only need to be avoided or kept at a safe distance. Judging solely based on entity category or current location makes it difficult to accurately determine whether an entity is suitable for interaction, and also makes it difficult to select appropriate subsequent processing methods for different types of entities. For dynamic entities, existing technologies typically estimate their position and trajectory based on continuous observations. However, entities in complex scenes may experience changes in velocity, direction of motion, or short-term occlusion. When the state estimation process cannot adapt to changes in the entity's motion characteristics, trajectory drift or state estimation errors can easily occur, further affecting the judgment of the entity's motion trend. Simultaneously, if only independent position or trajectory information is obtained at each moment, without a unified organization of the entity's motion state, physical attributes, and their temporal relationships, it is difficult to form environmental information that continuously reflects the evolution of the dynamic scene. Furthermore, the interaction between humanoid robots and moving entities depends not only on the target entity's current position but also on the target entity's future motion trend and the robot's own motion capabilities. In dynamic scenes, even if an interactive object is identified, if the positional changes of the object over a future period and the spatial range within which the robot can effectively approach and interact cannot be accurately determined, situations may arise such as inappropriate timing of interaction, the target leaving the interactive range, or a mismatch between the robot's actions and the target's motion, thereby reducing the stability and adaptability of the interaction behavior. Summary of the Invention

[0004] The purpose of this invention is to provide an adaptive interaction method and system for humanoid robots in dynamic scenarios, thereby solving the problems existing in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an adaptive interaction method for humanoid robots in dynamic scenes, comprising:

[0006] S1. Obtain continuous visual and depth data in the dynamic scene where the humanoid robot is located, and obtain an image sequence;

[0007] S2. For the image sequence, edge detection operators are used to extract the pixel contours of various entities in the image to obtain the target entity contour set;

[0008] S3. Based on the target entity outline set, extract the physical attributes of the corresponding entity. If the physical attributes meet the preset operable conditions, then determine that the entity is an interactive object and obtain the interactive object set.

[0009] S4. For the set of interactive objects, obtain the motion trajectory of the interactive objects;

[0010] S5. Obtain the motion trajectory and calculate the change in spatial position of the interactive object on the continuous time axis. If the change in spatial position is greater than the preset displacement threshold, it is determined that the interactive object is in a dynamic change state and dynamic spatiotemporal evolution information is obtained.

[0011] S6. Based on dynamic spatiotemporal evolution information and a set of interactive objects, spatially map the motion states and physical attributes with time stamps to construct a real-time updated environment map.

[0012] S7. Predict the future motion state of the interactive object based on the spatiotemporal evolution information contained in the environmental map, and dynamically intersect the predicted trajectory with the reachable space of the humanoid robot to determine the interactive reachable domain that changes over time. Adaptively adjust the interaction strategy according to the migration of the interactive reachable domain to generate interaction decisions for the interactive object.

[0013] Preferably, step S1 includes acquiring continuous visual data and continuous depth data in the dynamic scene where the humanoid robot is located; performing synchronous matching processing on the continuous visual data and the continuous depth data to obtain a synchronous visual frame sequence and a synchronous depth frame sequence; calculating the depth difference based on the synchronous depth frame sequence to determine the initial dynamic region; calculating the temporal consistency score based on the motion trajectory of the initial dynamic region to obtain the target dynamic region; using the target dynamic region as a constraint, performing feature extraction on the synchronous visual frame sequence and the synchronous depth frame sequence through a convolutional neural network to obtain a visual feature map and a depth feature map; performing alignment processing on the visual feature map and the depth feature map; and generating an image sequence based on the aligned feature map.

[0014] Preferably, step S2 includes performing gradient calculation on the first image sequence to generate a gradient feature map sequence; performing non-maximum suppression and double threshold classification on the gradient feature map sequence to obtain a classified edge image sequence containing strong edge points and weak edge points; filtering weak edge points according to the connectivity of strong edge points in the classified edge image sequence to obtain a continuous edge image sequence; performing a closing operation on the continuous edge image sequence and extracting the pixel contours of various entities in the continuous edge image sequence to obtain a target entity contour set.

[0015] Preferably, step S3 includes acquiring a set of target entity outlines; determining the initial physical properties of the target entity based on the set of target entity outlines; simulating the stress state based on the initial physical properties to obtain the current stress state value; determining whether the current stress state value meets preset operability conditions, including determining whether the current stress state value is greater than a preset interaction threshold; if the current stress state value is greater than the preset interaction threshold, then determining that the target entity meets the preset operability conditions; if the target entity meets the preset operability conditions, then determining that the target entity is an interactive object, and summarizing at least one interactive object to obtain an interactive object set.

[0016] Preferably, step S4 includes acquiring the physical attributes and contour change data of the interactive object; extracting initial kinematic features based on the physical attributes and contour change data; calculating a state transition matrix based on the initial kinematic features; updating the Kalman filter algorithm using the state transition matrix to obtain the filter gain; performing state estimation using the filter gain to obtain a state estimate; and if the state estimate is greater than a preset threshold, generating a spatial coordinate sequence based on the state estimate to fit the motion trajectory of the interactive object.

[0017] Preferably, step S5 includes acquiring the initial motion trajectory of the interactive object, segmenting the initial motion trajectory to obtain a discrete coordinate sequence and a timestamp set; calculating the spatial position change of the interactive object based on the discrete coordinate sequence and timestamp set; if the spatial position change is greater than a preset displacement threshold, determining that the interactive object is in a dynamic change state and generating state identification data; extracting multidimensional evolution features based on the state identification data to construct a spatiotemporal mapping matrix and obtain dynamic spatiotemporal evolution information.

[0018] Preferably, step S6 includes acquiring dynamic spatiotemporal evolution information and extracting motion state and physical attributes from a set of interactive objects; performing spatial mapping using a multi-sensor fusion algorithm based on the motion state and physical attributes to obtain a fusion mapping matrix; if the coordinate offset of the fusion mapping matrix is ​​greater than a preset threshold, then resampling the fusion mapping matrix to determine the target mapping matrix; and constructing a topology update graph based on the target mapping matrix to render and generate a real-time updated environment map.

[0019] Preferably, step S7 includes acquiring the spatiotemporal evolution characteristics of the environmental map to obtain the predicted trajectory of the interactive object; dynamically intersecting the predicted trajectory with the reachable space to determine the interactive reachable domain.

[0020] Preferably, step S7 further includes calculating a migration vector based on the centroid coordinates of the interactive reachable domain; if the migration vector is greater than a preset threshold, obtaining the migration direction and velocity of the interactive reachable domain; adjusting the driving torque based on the migration direction and velocity; and generating an interaction decision for the interactive object based on the adjusted driving torque.

[0021] An adaptive interaction system for humanoid robots in dynamic scenarios, used to implement the steps of the aforementioned adaptive interaction method for humanoid robots in dynamic scenarios, the system comprising:

[0022] The image sequence generation module acquires continuous visual and depth data in the dynamic scene where the humanoid robot is located, determines the dynamic region based on the depth changes in consecutive frames, and uses the temporal consistency constraint of the dynamic region to constrain the convolutional neural network to perform feature extraction and alignment processing on the visual and depth data to obtain an image sequence.

[0023] The target entity contour acquisition module uses edge detection operators to extract the pixel contours of various entities in the image sequence to obtain a set of target entity contours.

[0024] The interactive object determination module extracts the physical attributes of the corresponding entities based on the target entity outline set. If the physical attributes meet the preset operable conditions, the entity is determined to be an interactive object, and a set of interactive objects is obtained.

[0025] The motion trajectory estimation module, for the set of interactive objects, adaptively corrects the state transition parameters of the Kalman filter based on the physical properties of the interactive objects and the contour changes at adjacent time points, and performs state estimation in the continuous time dimension to obtain the motion trajectory of the interactive objects.

[0026] The dynamic spatiotemporal evolution information acquisition module acquires the motion trajectory and calculates the change in spatial position of the interactive object on a continuous time axis. If the change in spatial position is greater than a preset displacement threshold, it determines that the interactive object is in a dynamic change state and obtains dynamic spatiotemporal evolution information.

[0027] The environmental map construction module uses a multi-sensor fusion algorithm to spatially map the time-stamped motion state and physical attributes based on dynamic spatiotemporal evolution information and a set of interactive objects, thereby constructing a real-time updated environmental map.

[0028] The interaction decision generation module acquires an environmental map, predicts the future motion state of interactive objects based on the spatiotemporal evolution information contained in the environmental map, dynamically intersects the predicted trajectory with the reachable space of the humanoid robot, determines the interaction reachable domain that changes over time, adaptively adjusts the interaction strategy based on the migration of the interaction reachable domain, and generates interaction decisions for interactive objects.

[0029] As can be seen from the above technical solution, the present invention has the following beneficial effects:

[0030] This adaptive interaction method and system for humanoid robots in dynamic scenes continuously acquires visual and depth data and uses the temporal consistency of dynamic regions to constrain the multimodal feature extraction and alignment process, which helps improve the correlation stability of continuous environmental information and the accuracy of entity contour recognition in dynamic scenes. By combining entity physical attributes to determine interactive objects and adaptively correcting the state estimation process based on physical attributes and contour changes at adjacent times, it helps reduce the impact of dynamic entity motion state changes on trajectory estimation and improves the continuous perception of the motion trajectory and dynamic changes of interactive objects. By spatially mapping time-stamped motion states with physical attributes, an environmental map that updates in real time with scene changes can be formed, enabling the humanoid robot to obtain a dynamic environmental representation that takes into account entity spatial position, motion state, and interaction attributes. Furthermore, the future motion state of interactive objects is predicted based on the environmental map, and the predicted trajectory is dynamically correlated with the humanoid robot's reachable space to determine the interaction reachable domain that changes over time. Based on this, the interaction strategy is adaptively adjusted, thus comprehensively considering the dynamic changes of interactive objects and the humanoid robot's own motion capabilities, improving the rationality of determining the timing and range of interaction in dynamic scenes, and enhancing the humanoid robot's adaptive interaction capabilities in complex dynamic environments. Attached Figure Description

[0031] Figure 1 This is a flowchart of the overall method of the present invention;

[0032] Figure 2 A bar chart comparing the trajectory tracking errors of the four methods in different dynamic scenarios of the present invention;

[0033] Figure 3 This is a graph showing the change in tracking position error over time in a sudden turning scenario according to the present invention.

[0034] Figure 4 This is a schematic diagram showing the comparison between the predicted trajectory and the actual trajectory, as well as the interactive reachable domain of the present invention.

[0035] Figure 5 This is a curve comparing the trajectory tracking error under different target motion speeds according to the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Example 1:

[0038] As shown in the figure, this invention provides a technical solution: an adaptive interaction method for humanoid robots in dynamic scenes, comprising:

[0039] S1. Obtain continuous visual and depth data in the dynamic scene where the humanoid robot is located, determine the dynamic region based on the depth changes of the continuous frames, and use the convolutional neural network of the dynamic region to perform feature extraction and alignment processing on the visual and depth data to obtain an image sequence.

[0040] S2. For the image sequence, edge detection operators are used to extract the pixel contours of various entities in the image to obtain the target entity contour set;

[0041] S3. Based on the target entity outline set, extract the physical attributes of the corresponding entity. If the physical attributes meet the preset operable conditions, then determine that the entity is an interactive object and obtain the interactive object set.

[0042] S4. For the set of interactive objects, the state transition parameters of the Kalman filter are adaptively corrected according to the physical properties of the interactive objects and the contour changes at adjacent times, and the state is estimated in the continuous time dimension to obtain the motion trajectory of the interactive objects.

[0043] S5. Obtain the motion trajectory and calculate the change in spatial position of the interactive object on the continuous time axis. If the change in spatial position is greater than the preset displacement threshold, it is determined that the interactive object is in a dynamic change state and dynamic spatiotemporal evolution information is obtained.

[0044] S6. Based on the dynamic spatiotemporal evolution information and the set of interactive objects, a multi-sensor fusion algorithm is used to spatially map the motion state and physical attributes with time stamps, and construct a real-time updated environmental map.

[0045] S7. Obtain the environment map, predict the future motion state of the interactive object based on the spatiotemporal evolution information contained in the environment map, and dynamically intersect the predicted trajectory with the reachable space of the humanoid robot to determine the interaction reachable domain that changes over time. Adaptively adjust the interaction strategy based on the migration of the interaction reachable domain to generate interaction decisions for the interactive object.

[0046] In this embodiment, the humanoid robot acquires continuous visual and depth data in a dynamic scene and establishes a temporal correspondence between the visual and depth data according to the acquisition time.

[0047] Specifically, visual data is used to characterize the color, texture, edge, and appearance structure of entities in the scene, while depth data is used to characterize the spatial depth of the corresponding pixel position. For depth data at two consecutive time points, the depth values ​​at the corresponding spatial positions are compared, and dynamic regions are determined based on the depth change results. When the depth change of the same spatial region in consecutive frames meets the preset dynamic region determination conditions, the spatial region is determined as a dynamic region, and temporal consistency constraints are established based on the positional correspondence of the dynamic region in consecutive frames.

[0048] Furthermore, the temporal consistency constraint of the dynamic region is applied to the feature extraction and alignment process of the convolutional neural network.

[0049] Specifically, the convolutional neural network extracts image features from visual data and depth features from depth data, and performs feature alignment based on the spatial correspondence between visual data and depth data. For data belonging to the same dynamic region and located at adjacent time points, the corresponding features are associated according to the temporal continuity of the dynamic region, so that the visual features, depth features and spatial location of the same entity in continuous time remain in correspondence. After feature extraction and alignment are completed, an image sequence is generated in chronological order.

[0050] In this embodiment, edge detection operators are used to extract edges for each frame of the image sequence.

[0051] Specifically, the edge detection operator determines edge pixels based on grayscale changes, color changes, or fusion feature changes between adjacent pixels, and forms entity boundaries according to the spatial connectivity between edge pixels; for edge pixels belonging to the same entity boundary, they are grouped according to their connectivity to obtain pixel contours corresponding to the entity; and they are classified and recorded according to the pixel contours corresponding to different entities in the image to form a set of target entity contours.

[0052] In addition, based on the target entity contour set, a mapping relationship is established between each target entity contour and the corresponding depth data, and the physical attributes of the corresponding entity are extracted based on the contour information and depth information.

[0053] Specifically, the physical attributes include parameters used to characterize the spatial state and interaction conditions of entities, and the specific parameters are configured according to preset operable conditions; the physical attributes of each entity are judged against the preset operable conditions one by one, and when the physical attributes of an entity meet the preset operable conditions, the entity is determined as an interactive object; when the physical attributes of an entity do not meet the preset operable conditions, the entity is not determined as an interactive object; the entities that meet the preset operable conditions are grouped together to obtain a set of interactive objects.

[0054] Furthermore, for each interactive object in the set of interactive objects, the target entity contour corresponding to the current time and the target entity contour corresponding to the adjacent time are obtained, and the contour change between adjacent time is calculated.

[0055] Specifically, the contour change includes at least one of contour position change, contour size change, or contour shape change; at the same time, the physical attributes of the corresponding interactive object are obtained, and the physical attributes and the contour changes at adjacent time points are used together as the basis for correcting the Kalman filter state transition parameters.

[0056] In addition, based on the physical properties obtained at the current moment and the contour changes at adjacent moments, the state transition parameters in the Kalman filter are updated to match the motion state of the currently interactive object.

[0057] Specifically, the updated state transition parameters are used to calculate the predicted state at the current time based on the state estimation result of the previous time step, and the predicted state is corrected based on the observation results obtained at the current time step to obtain the state estimation result at the current time step. The state prediction, state correction and state update are repeated in time sequence so that each interactive object forms a continuous state estimation result in the continuous time dimension.

[0058] Furthermore, the state estimation results corresponding to each time point are arranged in chronological order, and the spatial location corresponding to each state estimation result is extracted.

[0059] By associating the spatial locations of interactive objects at consecutive moments according to time markers, the motion trajectory of the interactive object is formed. This motion trajectory is used to characterize the spatial location change process of the interactive object in the continuous time dimension and serves as the data basis for dynamic state determination and future motion state prediction.

[0060] Furthermore, after obtaining the motion trajectory of the interactive object, the spatial positions corresponding to adjacent moments or preset time intervals are extracted according to the continuous time axis, and the change in spatial position is calculated based on the coordinate difference between the two spatial positions.

[0061] The calculated change in spatial position is compared with a preset displacement threshold. When the change in spatial position is greater than the preset displacement threshold, the interactive object is determined to be in a dynamic state. When the change in spatial position is not greater than the preset displacement threshold, the object state corresponding to that time period is not determined to be in a dynamic state.

[0062] For interactive objects that are determined to be in a dynamic state, their time stamp, spatial location, motion state, and corresponding physical attributes are associated and recorded in chronological order.

[0063] By connecting the states of objects at different times through continuous time markers, dynamic spatiotemporal evolution information is formed; the dynamic spatiotemporal evolution information is used to characterize the state changes of interactive objects in the time dimension and the position migration in the spatial dimension.

[0064] Furthermore, dynamic spatiotemporal evolution information and a set of interactive objects are input into the multi-sensor fusion processing.

[0065] For data from different sensors, a time correspondence is first established based on time stamps. Then, based on the spatial coordinate transformation relationship between different sensors and the humanoid robot, the data from each sensor is mapped to a unified spatial coordinate system. After completing the time correspondence and spatial coordinate transformation, the motion state, physical attributes, and time stamps of each interactive object are mapped to a unified environmental space.

[0066] As a preferred embodiment, an environment map is constructed based on the interactive object data after spatial mapping is completed.

[0067] Specifically, the environment map records the spatial location, physical attributes, motion state, and time stamp of interactive objects; with the input of continuous visual and depth data, the corresponding information in the environment map is updated according to the new object detection results and state estimation results; when a corresponding interactive object already exists in the environment map, the spatial location, motion state, and physical attributes of the object are updated according to the data obtained at the current moment; when a new interactive object is detected at the current moment, the corresponding object information is written into the environment map; by continuously executing the above update process, a real-time updated environment map is formed.

[0068] Furthermore, dynamic spatiotemporal evolution information corresponding to interactive objects is obtained from the environmental map, and the motion state of the interactive object in subsequent moments is predicted based on its spatial location and motion state at consecutive moments.

[0069] Calculate the spatial location of the interactive object at each prediction time according to the prediction time axis, and connect the spatial locations corresponding to each prediction time in chronological order to form the prediction trajectory of the interactive object.

[0070] Furthermore, the reachable space of the humanoid robot is obtained, and the predicted trajectory of the interactive object is dynamically intersected with the reachable space of the humanoid robot according to the prediction time axis.

[0071] Specifically, for each predicted moment in the predicted trajectory, it is determined whether the predicted position corresponding to that moment is within the reachable space of the humanoid robot; when the predicted position is within the reachable space, the corresponding spatial position and time marker are recorded as interactive reachable information; interactive reachable information that satisfies the intersection condition in continuous time is associated to form an interactive reachable domain that changes with time.

[0072] In addition, as the predicted trajectory progresses over time and the environmental map is updated in real time, the intersection between the predicted trajectory and the humanoid robot's reachable space is recalculated, and the interactive reachable domain is updated.

[0073] By comparing the positional changes, spatial range changes, and temporal correspondences of the interactive reachable domain at consecutive time points, the migration state of the interactive reachable domain is determined; the humanoid robot's interaction strategy for interactive objects is updated based on the migration state of the interactive reachable domain, and corresponding interaction decisions are generated based on the updated interaction strategy.

[0074] In this embodiment, firstly, dynamic regions are determined and image sequences are generated using continuous visual data and depth data; secondly, a set of target entity contours is obtained through edge detection, and a set of interactive objects is obtained based on physical properties and preset operable conditions; subsequently, the motion trajectory of the interactive objects is obtained by adaptively correcting the Kalman filter state transition parameters, and dynamic spatiotemporal evolution information is determined based on spatial position changes and preset displacement thresholds; then, a real-time updated environmental map is constructed using a multi-sensor fusion algorithm; finally, the future motion state of the interactive objects is predicted based on the spatiotemporal evolution information in the environmental map, and the interactive reachable domain is determined based on the dynamic intersection result between the predicted trajectory and the reachable space of the humanoid robot. The interaction strategy is adjusted based on the migration of the interactive reachable domain, thereby generating interaction decisions for the interactive objects.

[0075] Design experiments to verify this application:

[0076] Hardware platform: The experiment used a humanoid robot prototype (165cm tall, dual 6-DOF robotic arms, with a two-finger gripper at the end); an onboard RGB-D depth camera (Intel RealSense D435, 1280×720 resolution, 30fps); and the main control system used an NVIDIA Jetson AGX Orin embedded platform.

[0077] Experimental environment: The indoor experimental area is 6m×6m with stable lighting; two experimenters play the roles of pedestrian and cart pusher, respectively, and move around in the area according to a pre-arranged script. The experiment is repeated 10 times and the average value is taken.

[0078] Dynamic scene design (5 categories, corresponding to the main types of dynamic scenes):

[0079] A pedestrian is approaching from the opposite direction (at a speed of approximately 0.8 m / s).

[0080] Pedestrians weave in from the side;

[0081] The cart is approaching (a cargo handcart, weighing 10kg).

[0082] The pedestrian suddenly turned (90° turn while walking);

[0083] Multi-objective mixed (two people + one vehicle moving simultaneously).

[0084] Comparison method:

[0085] Fixed Kalman filter: Trajectory estimation is performed using the standard Kalman filter without time consistency constraints or adaptive correction of state transition parameters;

[0086] Timing consistency constraint + fixed filtering: Only the S1 dynamic region timing consistency constraint is enabled, and the filtering parameters remain fixed.

[0087] Adaptive filtering (no timing constraints): Only enables adaptive correction of S4 state transition parameters, without timing consistency constraints;

[0088] The proposed method fully implements the S1–S7 process.

[0089] Evaluation indicators:

[0090] Trajectory tracking error RMSE (cm): The root mean square error between the estimated trajectory and the true trajectory of the Vicon motion capture system;

[0091] Interactive object recognition accuracy (%): The proportion of real interactive objects among entities identified as interactive objects;

[0092] Interaction success rate (%): The percentage of interactive actions that are successfully executed on the target (effective gripping / contact by the gripper);

[0093] Single frame processing time (ms): The average processing time of a single frame by the algorithm.

[0094] Experimental Results and Comparison:

[0095] Comparison of trajectory tracking errors (different dynamic scenarios);

[0096] Table 1 shows the RMSE of trajectory tracking for the four methods in five types of dynamic scenarios:

[0097] Table 1. Trajectory tracking error RMSE (cm) of the four methods under different dynamic scenarios.

[0098]

[0099] like Figure 2As shown in Table 1, the trajectory tracking error of the proposed method is significantly lower than that of the three comparative methods in all five scenarios. The pedestrian sudden turning scenario best demonstrates the value of adaptive filtering: in this scenario, the target motion model changes abruptly, and the fixed Kalman filter error is as high as 12.4 cm, while the proposed method still maintains 4.8 cm (a reduction of about 61%). Compared with the two intermediate methods that only use a single module (average 7.1 cm and 6.9 cm), the proposed method has an average error of 4.0 cm, indicating that the synergistic effect of the temporal consistency constraint and adaptive correction filtering modules is better than either single module.

[0100] like Figure 3 As shown, taking a pedestrian suddenly turning scene as an example, the position tracking error change curves of the two methods within 40 frames are plotted to illustrate the dynamic perception capability of the proposed method after the target motion state changes abruptly.

[0101] For the first 20 frames, the target moves at a constant speed, and the errors of both methods remain stable. After the target suddenly turns in the 20th frame, the fixed Kalman filter's error rapidly rises to about 13.8 cm and remains high due to the failure of the state transition model. The proposed method, on the other hand, adaptively corrects the state transition parameters with changes in physical properties and the contours of adjacent frames, resulting in an error that only rises slightly to about 4.9 cm before falling back. The results show that the proposed method can quickly converge the state estimation after a sudden change in the target's motion pattern, and its ability to continuously perceive dynamic entities is significantly stronger.

[0102] like Figure 4 As shown, taking a pedestrian approaching from the front as an example, the trajectory of the interactive object (pedestrian) is predicted, and the predicted trajectory, the actual trajectory, and the reachable domain of the interaction are drawn.

[0103] In the figure, the solid black line represents the actual trajectory of the pedestrian, the dashed blue line represents the predicted trajectory of the fixed filter, and the dashed red line represents the predicted trajectory of the proposed method. It can be seen that the prediction of the fixed filter deviates significantly from the actual trajectory in the later part of the trajectory (the maximum deviation is about 0.9m); the predicted trajectory of the proposed method is highly consistent with the actual trajectory, with a maximum deviation of about 0.08m. The red ellipse represents the interactive reachable domain obtained by dynamically intersecting the predicted trajectory with the reachable space of the robotic arm, and its centroid (red star) basically coincides with the actual position reached by the pedestrian. The results show that the proposed method can accurately predict the future state of the interactive object and determine the reasonable interaction timing and interaction range accordingly, avoiding invalid interactions that are too early / too late, too close / too far.

[0104] like Figure 5 As shown, to examine the adaptability of the proposed method under different target motion speeds, the target is set to move at a constant speed of 0.2–1.2 m / s, and the tracking RMSE of the two methods is compared.

[0105] As the target speed increases, the tracking error of both methods increases. The fixed Kalman filter error increases from 3.9 cm to 16.8 cm in the speed range of 0.2 m / s to 1.2 m / s, while the proposed method only increases from 2.6 cm to 6.7 cm, with a significantly slower error growth rate. The results show that the proposed method has a strong adaptability to target speed changes by adaptively correcting the filter parameters through contour changes, and can maintain stable tracking accuracy over a wide speed range.

[0106] In a mixed test (10 repetitions) of five types of dynamic scenes, the accuracy of interactive object recognition, the success rate of interaction, and the processing time per frame were statistically analyzed. The results are shown in Table 2.

[0107] Table 2 Comparison of Recognition Accuracy, Interaction Success Rate and Processing Time

[0108]

[0109] The proposed method achieves an interactive object recognition accuracy of 94.6%, which is 16.1 percentage points higher than that of fixed Kalman filtering. This indicates that the temporal consistency constraint effectively enhances the correlation stability of entity contours between consecutive frames, reducing missed detections and false detections. The interaction success rate reaches 81.0%, which is about 17–30 percentage points higher than the comparison method. This shows that the interaction strategy based on the interaction reachability domain can integrate the dynamic changes of the target and the reachability of the robotic arm, making the interaction timing and range more reasonable. The processing time per frame is 47.2 ms (about 21 fps), which meets the real-time requirements and only increases by about 4 ms compared to the comparison method (mainly due to contour association and reachability domain calculation).

[0110] Example 2:

[0111] S1 includes acquiring continuous visual data and continuous depth data in the dynamic scene where the humanoid robot is located; performing synchronous matching processing on the continuous visual data and the continuous depth data to obtain a synchronous visual frame sequence and a synchronous depth frame sequence; calculating the depth difference based on the synchronous depth frame sequence to determine the initial dynamic region, and calculating the temporal consistency score based on the motion trajectory of the initial dynamic region to obtain the target dynamic region; using the target dynamic region as a constraint, performing feature extraction on the synchronous visual frame sequence and the synchronous depth frame sequence through a convolutional neural network to obtain a visual feature map and a depth feature map; performing alignment processing on the visual feature map and the depth feature map, and generating an image sequence based on the aligned feature map.

[0112] In this embodiment, the humanoid robot continuously acquires visual data of the dynamic scene through the visual acquisition unit and continuously acquires corresponding depth data through the depth acquisition unit.

[0113] Specifically, the visual acquisition unit and the depth acquisition unit add acquisition time markers to each frame of data they output; during synchronization matching, they read the acquisition time of the current visual frame and the acquisition time of the current depth frame, calculate the absolute time interval between the two acquisition times, and compare the absolute time interval with a preset synchronization time threshold; when the absolute time interval is not greater than the preset synchronization time threshold, the corresponding visual frame and depth frame are determined to be the same synchronized data group; when the absolute time interval is greater than the preset synchronization time threshold, no synchronization relationship is established between the visual frame and the depth frame; all the matched data are arranged from front to back according to the acquisition time, forming a synchronized visual frame sequence and a synchronized depth frame sequence respectively.

[0114] For a synchronized depth frame sequence, the current depth frame and the previous depth frame are selected sequentially according to time, and pixels with the same spatial position in the two depth data frames are compared pixel by pixel.

[0115] For any pixel location, read the current depth value and the previous depth value, subtract the two depth values ​​and take the absolute value to obtain the depth difference corresponding to the pixel location; compare the depth difference of each pixel location with a preset depth difference threshold. When the depth difference is greater than the preset depth difference threshold, the pixel is determined as a dynamic pixel; when the depth difference is not greater than the preset depth difference threshold, the pixel is determined as a non-dynamic pixel.

[0116] In addition, the identified dynamic pixels are divided into connected regions according to their spatial adjacency, and dynamic pixels with vertical, horizontal or preset neighborhood connections are grouped into the same candidate region.

[0117] After completing the connected region division, the number of dynamic pixels contained in each candidate region is counted and compared with the preset region pixel count threshold. When the number of dynamic pixels in a candidate region reaches the preset region pixel count threshold, the candidate region is determined as the initial dynamic region. Candidate regions that do not reach the preset region pixel count threshold are not used as initial dynamic regions.

[0118] For the initial dynamic regions obtained at consecutive time points, calculate the centroid position of each pixel in the initial dynamic region.

[0119] The x-coordinate of the pixel centroid position is obtained by summing the x-coordinates of all pixels in the region and dividing by the total number of pixels in the region. The y-coordinate of the pixel centroid position is obtained by summing the y-coordinates of all pixels in the region and dividing by the total number of pixels in the region. Then, the centroid position of the initial dynamic region at the current moment is compared with the centroid positions of each initial dynamic region at the previous moment.

[0120] When calculating the centroid displacement between two initial dynamic regions, the difference between the two centroid abscissas is first calculated and squared, then the difference between the two centroid ordinates is calculated and squared, the two squared values ​​are added together and the square root is taken as the centroid displacement between the two initial dynamic regions.

[0121] At the same time, calculate the overlapping area between the initial dynamic region at the current moment and the initial dynamic region at the previous moment, and calculate the area of ​​each of the two initial dynamic regions respectively; divide the overlapping area by the sum of the areas of the two regions and subtract the overlapping area to obtain the area overlap ratio between the two initial dynamic regions.

[0122] When the centroid displacement between two initial dynamic regions is not greater than a preset association distance threshold, and the region overlap ratio reaches a preset overlap threshold, the two initial dynamic regions are determined as the corresponding regions of the same dynamic region at adjacent times.

[0123] Connect the initial dynamic regions that satisfy the above correspondence in a chronological order, and the motion trajectory of the initial dynamic region is formed by the centroid position corresponding to each moment.

[0124] Furthermore, after obtaining the motion trajectory of the initial dynamic region, the temporal consistency score is calculated based on the motion trajectory.

[0125] First, the centroid displacement between every two adjacent moments in the trajectory is calculated sequentially. Then, the absolute difference between two adjacent displacement values ​​is calculated, and this absolute difference is used to characterize the displacement change during the continuous motion process of the dynamic region. The displacement change is compared with a preset motion change threshold, and the displacement change is converted into a motion continuity value according to a preset conversion relationship. When the displacement change is 0, the motion continuity value takes a preset upper limit value. When the displacement change reaches the preset motion change threshold, the motion continuity value takes a preset lower limit value. When the displacement change is between 0 and the preset motion change threshold, the corresponding motion continuity value is determined according to the proportion of the displacement change to the preset motion change threshold.

[0126] Subsequently, the region overlap ratio is calculated for the initial dynamic region at every two adjacent moments in the motion trajectory, and each region overlap ratio is used as a region continuous value.

[0127] The arithmetic mean of all continuous motion values ​​in the motion trajectory is calculated to obtain the continuous motion result; the arithmetic mean of all continuous regional values ​​is calculated to obtain the continuous regional result; the continuous motion result and the continuous regional result are weighted according to preset weights respectively, and the two weighted results are added together to obtain the temporal consistency score corresponding to the initial dynamic region; wherein, the sum of the two preset weights is 1.

[0128] The timing consistency score is compared with a preset consistency threshold.

[0129] Specifically, when the time sequence consistency score reaches the preset consistency threshold, the corresponding initial dynamic region is determined as the target dynamic region; when the time sequence consistency score does not reach the preset consistency threshold, the initial dynamic region is not determined as the target dynamic region. Thus, the target dynamic region entering the subsequent feature extraction process simultaneously satisfies the depth change condition, the cross-frame position association condition, and the temporal continuity condition.

[0130] Furthermore, after determining the target dynamic region, a region mask is generated based on the pixel range of the target dynamic region in the synchronized depth frame, and the region mask is mapped to the corresponding position in the synchronized visual frame according to the spatial correspondence between the synchronized visual frame and the synchronized depth frame.

[0131] Pixels belonging to the target dynamic region in the region mask are marked as retained, while pixels not belonging to the target dynamic region are marked as masked. The region mask is applied to the synchronized visual frame and the synchronized depth frame respectively, so that the data in the target dynamic region enters the feature extraction process of the convolutional neural network.

[0132] When extracting features from synchronized visual frames, the convolutional neural network moves position by position in the image plane according to a preset convolution window.

[0133] At each position, the pixel values ​​within the coverage area of ​​the convolution window are multiplied one by one with the weights at the corresponding positions in the convolution kernel. The sum of all the product results is then superimposed with the corresponding bias value to obtain the convolution response at that position. After continuously performing convolution processing, nonlinear transformation, and downsampling processing according to the hierarchical structure of the convolutional neural network, a visual feature map is obtained.

[0134] In addition, when extracting features from the synchronous depth frame, the depth values ​​in the target dynamic region are convolved according to the network processing flow corresponding to the visual data.

[0135] Specifically, each convolutional position obtains a convolutional response by multiplying the depth value with the convolutional kernel weights, summing the products, and stacking biases. This response is then used by subsequent network layers to form a depth feature map. The visual feature map is used to characterize the image structure features of the target dynamic region, while the depth feature map is used to characterize the spatial depth distribution features of the target dynamic region.

[0136] Furthermore, after obtaining the visual feature map and the depth feature map, spatial alignment is performed based on the pre-completed calibration results between the visual acquisition unit and the depth acquisition unit.

[0137] For any target pixel in the depth data, first read the image x-coordinate, image y-coordinate and depth value of the pixel, and then convert the pixel into three-dimensional spatial coordinates in the depth acquisition unit coordinate system according to the focal length parameter and principal point coordinates of the depth acquisition unit.

[0138] Subsequently, based on the rotation and translation relationships obtained from the calibration between the vision acquisition unit and the depth acquisition unit, the three-dimensional spatial coordinates are transformed from the depth acquisition unit coordinate system to the vision acquisition unit coordinate system.

[0139] After the coordinate transformation is completed, the transformed three-dimensional spatial coordinates are projected onto the visual image plane according to the focal length parameters and principal point coordinates of the visual acquisition unit to obtain the corresponding visual image position; the depth pixels in the target dynamic area are traversed according to the same processing procedure to establish the spatial position correspondence between depth data and visual data.

[0140] The above spatial location correspondence is further mapped to visual feature maps and depth feature maps.

[0141] For each target feature location in the depth feature map, the projection position in the visual image is determined based on its corresponding original depth pixel position. Then, according to the scaling relationship between the visual feature map and the original visual image, the projection position is converted into the corresponding feature position in the visual feature map.

[0142] When the transformed position is between adjacent feature points, calculate the horizontal and vertical distances between the position and the surrounding feature points respectively. Determine the interpolation weights of each surrounding feature point based on the ratio of the distance to the distance between adjacent feature points. Then, multiply the feature values ​​of each surrounding feature point by the corresponding interpolation weights and sum them to obtain the visual feature value corresponding to the position.

[0143] Through the above processing, the target features in the depth feature map and the corresponding target features in the visual feature map are made to have the same spatial positional relationship.

[0144] Furthermore, after alignment, the visual and depth features corresponding to the same spatial location are connected in the order of feature channels to form the fused feature map corresponding to that moment.

[0145] For each synchronized visual frame and synchronized depth frame, the target dynamic region determination, feature extraction and feature alignment processes are repeatedly performed, and the fused feature maps are arranged in the time order of the synchronized data group to finally obtain the image sequence.

[0146] Example 3:

[0147] S2 includes performing gradient calculation on the first image sequence to generate a gradient feature map sequence; performing non-maximum suppression and double threshold classification on the gradient feature map sequence to obtain a classified edge image sequence containing strong edge points and weak edge points; filtering weak edge points based on the connectivity of strong edge points in the classified edge image sequence to obtain a continuous edge image sequence; performing closing operation on the continuous edge image sequence and extracting the pixel contours of various entities in the continuous edge image sequence to obtain a target entity contour set.

[0148] In this embodiment, after acquiring the first image sequence, gradient calculation is performed frame by frame according to the temporal order of the image frames. For any frame image, a neighborhood window of a fixed size is established with the pixel to be processed as the center, and the pixel changes in the horizontal and vertical directions are calculated respectively.

[0149] Specifically, the pixel changes in the horizontal direction are obtained by weighting and summing the pixel values ​​in the left and right neighborhoods of the pixel to be processed according to the weights of the edge detection operator, and then calculating the difference between the two weighted results; the pixel changes in the vertical direction are obtained by performing the same processing on the pixel values ​​in the upper and lower neighborhoods of the pixel to be processed.

[0150] Subsequently, the horizontal and vertical pixel changes are squared respectively, and the two squared results are added together and the square root is taken to obtain the gradient intensity of the pixel to be processed. Then, the gradient direction is determined according to the directional relationship between the vertical and horizontal pixel changes. After traversing all pixels in the current frame, the gradient intensity and gradient direction are recorded according to the coordinate position of each pixel in the original image to form the gradient feature map corresponding to the current frame. The same processing is performed on each frame in the first image sequence to obtain the gradient feature map sequence.

[0151] When performing non-maximum suppression on the gradient feature map sequence, the gradient direction of the current pixel to be processed is used as the comparison direction. Two adjacent positions on both sides of the current pixel are selected in this direction, and the gradient intensity of the two adjacent positions is read respectively.

[0152] If the gradient intensity of the current pixel is not less than the gradient intensity of two adjacent positions, the current pixel is retained; if the gradient intensity of the current pixel is less than the gradient intensity of any of the adjacent positions, the current pixel is marked as a non-candidate edge point. Through the above processing, only the local maximum response positions corresponding to the gradient direction are retained in the same gradient change region as candidate edge points for subsequent double threshold classification.

[0153] Furthermore, after performing nonmaximum suppression, the retained candidate edge points are classified using a double threshold method.

[0154] Specifically, the dual thresholds include a high gradient threshold and a low gradient threshold, where the high gradient threshold is greater than the low gradient threshold. The high gradient threshold is determined based on the average and standard deviation of all non-zero gradient intensities after non-maximum suppression in the current frame. Specifically, the arithmetic mean of all non-zero gradient intensities is calculated first, then the square of the difference between each non-zero gradient intensity and the average is calculated, the average of all squared differences is taken, and the square root is used to obtain the standard deviation. The sum of the average and the standard deviation is used as the high gradient threshold. The low gradient threshold is determined by half of the high gradient threshold. Using the above method, the dual thresholds can be generated based on the actual gradient distribution of the current frame without the need for manual frame-by-frame adjustment.

[0155] In the classification process, the gradient intensity of each candidate edge point is compared with the high gradient threshold and the low gradient threshold in turn.

[0156] Specifically, when the gradient intensity reaches the high gradient threshold, the candidate edge point is marked as a strong edge point; when the gradient intensity is lower than the high gradient threshold but reaches the low gradient threshold, the candidate edge point is marked as a weak edge point; when the gradient intensity is lower than the low gradient threshold, the candidate edge point is deleted; strong edge points and weak edge points are recorded according to the coordinate position of each pixel in the current frame, thereby forming the classification edge image corresponding to the current frame, and further forming a classification edge image sequence.

[0157] In addition, weak edge points in the classified edge image are not directly retained as the final edge, but are filtered based on whether there is a continuous connection path between them and strong edge points.

[0158] In the specific processing, each strong edge point is used as a starting point, and weak edge points are searched within its 8-neighborhood. When a weak edge point is detected, it is marked as a connected edge point, and the search continues from that weak edge point as the new search position, searching for subsequent weak edge points within its 8-neighborhood. A weak edge point is retained if it can be ultimately connected to any strong edge point through one or more consecutive adjacent weak edge points. After completing the connectivity search for all strong edge points, weak edge points that have not established a continuous connection path with any strong edge point are deleted. The strong edge points and the weak edge points retained after connectivity filtering together form a continuous edge image sequence.

[0159] When performing closing operations on a sequence of consecutive edge images, dilation is performed first, followed by erosion. The structuring element used is a rectangular structuring element with a side length of 3 pixels.

[0160] During dilation, the center of the structuring element is sequentially aligned with the pixel positions in the continuous edge image. When at least one edge pixel exists within the coverage area of ​​the structuring element, the position corresponding to the center of the structuring element is set as the edge pixel, thus connecting the breakpoints between adjacent edges with an interval of no more than one pixel. After dilation, erosion is performed using the same structuring element. Only the edge pixel corresponding to the center of the structuring element is retained if the pixels within the coverage area meet the edge region preservation condition; otherwise, the position is deleted. By dilation followed by erosion, edge breakpoints with an interval of no more than one pixel are connected, while simultaneously restoring the pixel range added to the outer edge during dilation.

[0161] Furthermore, after completing the closing operation, the pixels are scanned from the top left corner of the edge image in a left-to-right, top-to-bottom order.

[0162] Specifically, when an unmarked edge pixel is detected, it is used as the starting pixel of the current contour, and the search continues along its 8-neighborhood for the next adjacent edge pixel. Each time a new edge pixel is identified, its coordinates are recorded, and the search continues within that pixel's 8-neighborhood for unvisited adjacent edge pixels until the search path returns to the starting pixel or no new adjacent edge pixels exist; this yields one pixel contour. The process then continues scanning for unvisited edge pixels in the current frame until all edge pixels have been contoured.

[0163] Specifically, when determining the entity contour of each obtained pixel contour, the number of pixels contained in each contour is counted, and the area of ​​the region enclosed by the contour is calculated.

[0164] Specifically, the outline pixel count threshold is determined based on the image resolution of the first image sequence, and the number of pixels corresponding to a fixed proportion of the total number of pixels in the image is taken as the outline pixel count threshold; the region area threshold is calculated according to a fixed proportion of the total area of ​​the same image.

[0165] During implementation, the above ratios are pre-written into the processing parameters after the acquisition equipment is calibrated, and remain unchanged under the same resolution conditions; only when the number of contour pixels reaches the contour pixel number threshold and the area enclosed by the contour reaches the area threshold, the contour is determined as a valid entity contour; contours that do not reach either threshold are discarded as isolated edges or local noise contours.

[0166] For valid contours that have an inclusion relationship in the same frame, the correspondence between the outer contour and the inner contour is established according to the spatial enclosing relationship of the contour, and the outermost closed contour is used as the outer pixel boundary of the corresponding entity.

[0167] The effective entity contours in each frame are recorded in chronological order according to the first image sequence, and the pixel coordinates, frame number and region position of each contour are associated to obtain the target entity contour set.

[0168] Example 4:

[0169] S3 includes obtaining a set of target entity outlines; determining the initial physical properties of the target entity based on the target entity outline set; simulating the stress state based on the initial physical properties to obtain the current stress state value; determining whether the current stress state value meets preset operability conditions, including determining whether the current stress state value is greater than a preset interaction threshold; if the current stress state value is greater than the preset interaction threshold, then the target entity is determined to meet the preset operability conditions; if the target entity meets the preset operability conditions, then the target entity is determined to be an interactive object, and at least one interactive object is summarized to obtain an interactive object set.

[0170] In this embodiment, the target entity contour set is first read, and the image area occupied by each target entity is determined based on the pixel coordinates of each target entity contour in the corresponding image frame.

[0171] Specifically, the depth data corresponding to the image frame is called to convert the depth values ​​of each effective pixel within the contour range into the three-dimensional coordinate system of the humanoid robot; for the same target entity, the maximum and minimum coordinate values ​​of all three-dimensional space points in the front-back, left-right, and vertical directions are counted respectively, and the maximum coordinate value in the same direction is subtracted from the minimum coordinate value to obtain the length, width, and height of the target entity.

[0172] The spatial center of the target entity is calculated based on all valid three-dimensional spatial points within the contour.

[0173] Specifically, the coordinate values ​​of all three-dimensional spatial points in the three coordinate directions are summed and then divided by the number of effective three-dimensional spatial points. The three average coordinates are used as the spatial center of the target entity. For target entities that are in contact with the ground, tabletop, or support platform, three-dimensional spatial points whose vertical coordinates are within the bottom boundary of the entity are selected, and the selected spatial points are projected onto the corresponding support plane. The outermost connection boundary of the projected points is determined as the support area of ​​the target entity.

[0174] In addition, the volume of the target entity is determined based on the occupied area formed by its three-dimensional spatial points.

[0175] Specifically, the three-dimensional space corresponding to the target entity is divided into cubic units of fixed size. If a three-dimensional space point corresponding to the target entity exists inside a cubic unit, the cubic unit is recorded as an occupied unit. The total number of occupied units is counted, and the number of occupied units is multiplied by the actual volume of a single cubic unit. The result is used as the equivalent volume of the target entity. The side length of the cubic unit is determined according to the spatial sampling interval corresponding to the depth acquisition unit within the robot's working distance, and remains fixed when the depth acquisition parameters do not change.

[0176] In addition, for the mass parameters required for simulating the stress state, a physical parameter table corresponding to the target entity category is pre-established. The physical parameter table stores the equivalent density and contact friction coefficient of each entity category.

[0177] When determining the target entity category, the aspect ratio, outline area, minimum bounding rectangle area, and outline perimeter are calculated based on the target entity outline, and compared with the outline parameters corresponding to each entity category in the physical parameter table. For each candidate category, the absolute difference between each outline parameter of the target entity and the corresponding parameter of the candidate category is calculated, and then each absolute difference is weighted and summed according to a preset fixed weight. The candidate category with the smallest weighted sum is determined as the target entity category, and the equivalent density and contact friction coefficient corresponding to the category are read.

[0178] Furthermore, the equivalent volume of the target entity is multiplied by the equivalent density of the corresponding category to obtain the equivalent mass of the target entity; then, the gravity acting on the target entity is determined based on the equivalent mass and the gravitational acceleration corresponding to the current working location.

[0179] The initial physical properties of the target entity consist of its length, width, height, spatial center, supporting area, equivalent volume, equivalent mass, gravity, and contact friction coefficient, and these parameters are used as inputs for simulating the stress state.

[0180] In the process of simulating the stress state, the critical forces corresponding to the target entity entering the slip state and the overturning state under the interaction are calculated respectively.

[0181] For the slip state, first determine the normal bearing force on the horizontal bearing surface based on the weight of the target entity, then multiply the normal bearing force by the corresponding contact friction coefficient, and the result is used as the critical force corresponding to the target entity entering the slip state from the static state.

[0182] In this embodiment, for the overturning state, the spatial center of the target entity is first projected onto the supporting plane along the direction of gravity, and the distance from the projected position to each boundary of the supporting area is calculated. The minimum distance is taken as the gravity arm when the entity resists overturning. Then, the contact position of the humanoid robot acting on the target entity is determined according to the current interaction action, and the vertical distance from the contact position to the supporting plane is calculated. This distance is taken as the arm of the interaction force. The gravity acting on the target entity is multiplied by the gravity arm to obtain the anti-overturning moment that keeps the target entity from overturning. The anti-overturning moment is then divided by the arm of the interaction force, and the result is taken as the critical force corresponding to the target entity entering the overturning state.

[0183] Furthermore, the critical force corresponding to the slip state is compared with the critical force corresponding to the overturning state, and the smaller value is taken as the current force state value.

[0184] This value represents the upper limit of the interaction force that the target entity can withstand while maintaining its current stress state under the current spatial dimensions, mass distribution, support state, and contact conditions; when the interaction force applied by the humanoid robot reaches this value, the target entity will first enter a sliding state or an overturning state.

[0185] As a preferred embodiment, the preset interaction threshold is pre-calibrated based on the actual force required for the humanoid robot to perform the target interaction action.

[0186] During calibration, the humanoid robot is made to act on the force measuring device with the same execution part, contact direction, contact position, and movement speed as in the formal interaction, and to continuously perform 10 complete interactive actions. For each interactive action, the maximum force value from the start of contact to the end of the action is recorded. After comparing the 10 maximum force values, the result with the largest value is selected and the upper limit of the measurement error specified in the calibration parameters of the force measuring device is added. The obtained value is determined as the interaction threshold corresponding to the interactive action. The above calibration process is performed for different interactive actions, and the obtained interaction threshold is stored after establishing a mapping relationship with the corresponding interactive action.

[0187] When determining the operability conditions of a target entity, the corresponding interaction threshold is first read based on the interaction action to be executed, and then the current force state value is compared with the interaction threshold.

[0188] When the current force value is greater than the interaction threshold, it indicates that the target entity has not yet reached the slip or overturning critical state while bearing the force required to complete the interaction action. Based on this, the target entity is determined to meet the preset operable conditions. When the current force value is not greater than the interaction threshold, the target entity is not determined to meet the preset operable conditions.

[0189] For each target entity in the target entity contour set, the above-mentioned physical attribute extraction, stress state simulation and operability condition judgment are performed sequentially.

[0190] For a target entity that meets the preset operable conditions, an object identifier is created for it, and its corresponding spatial location, spatial size, equivalent mass, contact friction coefficient and current force state value are associated with it. The target entity is then identified as an interactive object. After all target entities are identified, at least one interactive object is summarized to obtain a set of interactive objects.

[0191] Example 5:

[0192] S4 includes acquiring the physical attributes and contour change data of the interactive object, extracting initial kinematic features based on the physical attributes and contour change data, calculating the state transition matrix based on the initial kinematic features, updating the Kalman filter algorithm using the state transition matrix to obtain the filter gain, performing state estimation using the filter gain to obtain the state estimate value, and if the state estimate value is greater than a preset threshold, generating a spatial coordinate sequence based on the state estimate value to fit the motion trajectory of the interactive object.

[0193] In this embodiment, after obtaining the interactive object, the spatial position, spatial size, equivalent mass, force state value, and contour change data of the object in the current moment are read.

[0194] Specifically, the contour change data is obtained by comparing the contour at the current moment with the contour at the previous moment, and includes at least the changes in the contour centroid position, contour width, contour height, and depth direction position. For the contour centroid position, the differences between the coordinates of the centroid at the current moment in the three spatial directions and the corresponding coordinates at the previous moment are calculated, and the coordinate differences in each direction are divided by the actual sampling time interval between two frames of data to obtain the motion velocity in the three spatial directions.

[0195] Further read the velocity obtained in the previous moment, subtract the current velocity from the velocity in the previous moment, and divide by the corresponding time interval to obtain the acceleration in each spatial direction.

[0196] Furthermore, the contour size change is calculated based on the outer contour size of the same object in two adjacent frames.

[0197] Specifically, the contour width at the current moment is subtracted from the contour width at the previous moment, and the contour height at the current moment is subtracted from the contour height at the previous moment. The average depth change at the two moments is combined to determine whether the scale change of the target in the imaging plane originates from actual spatial displacement. The current spatial position, the velocity in the three spatial directions, the acceleration in the three spatial directions, and the contour scale change result are combined to form the initial kinematic features.

[0198] Among them, spatial position represents the current position of the object, velocity represents the position change per unit time, acceleration represents the velocity change per unit time, and contour scale change is used to check the change in projected size when the object moves along the depth direction.

[0199] When determining the state transition matrix based on the initial kinematic characteristics, the actual sampling time interval between two adjacent state calculations is used as the time parameter.

[0200] The parameters used to update the spatial position in the matrix are set according to the time recursion relationship of position, velocity, and acceleration. That is, the position prediction result at the current moment is jointly determined by the position at the previous moment, the displacement generated by the velocity at the previous moment within the current sampling interval, and the displacement generated by the acceleration at the previous moment within the current sampling interval; the velocity prediction result at the current moment is jointly determined by the velocity increment generated by the velocity and acceleration at the previous moment within the current sampling interval; and the acceleration at the current moment is recalculated based on the velocity changes at the latest two moments.

[0201] In addition, when the contour change data indicates that the object's current motion direction is consistent with the previous moment, the corresponding motion parameters in the state transition matrix are updated based on the velocity and acceleration calculated in this time; when the positive or negative velocity direction changes in any of the three spatial directions, the state transition parameters corresponding to that direction are replaced with the velocity and acceleration recalculated at the current moment.

[0202] Therefore, the state transition matrix is ​​updated as the interactive object moves.

[0203] When performing Kalman filtering using the updated state transition matrix, the predicted state at the current time step is first calculated based on the state estimation results of the previous time step.

[0204] Specifically, the process involves multiplying each state component from the previous moment according to the corresponding parameters in the state transition matrix, and summing the multiplication results corresponding to the same current state component to obtain the current position prediction value, current velocity prediction value, and current acceleration prediction value.

[0205] Simultaneously, the state estimation error is predicted. Based on the error covariance of the previous time step, the error is propagated according to the current state transition matrix, and the process noise covariance is superimposed to obtain the prediction error covariance at the current time step.

[0206] The process noise covariance is calibrated before the robot is put into operation. During calibration, a calibration target with a known trajectory is driven through the robot's sensing area at a fixed speed, and 20 frames of data are continuously collected. The predicted position obtained in each frame based on the state at the previous moment is compared with the actual position of the calibration target at the corresponding moment, and the prediction error in three spatial directions is obtained. Then, the average value of the prediction error in each direction is calculated, and the square of the difference between each prediction error and the corresponding average value is calculated. The sum of all the squared differences is divided by the corresponding number of errors, and the process noise parameters in each direction are determined from the results.

[0207] Furthermore, after completing the state prediction, the actual spatial location of the interactive object obtained through visual and depth data at the current moment is read and used as the current observation value.

[0208] First, the predicted spatial position is subtracted from the actual spatial position to obtain the observation residuals in the three spatial directions. Then, the filter gain is determined based on the prediction error covariance and the observation noise covariance. The prediction error covariance characterizes the error range of the current prediction state, and the observation noise covariance characterizes the error range generated by the sensor's spatial position measurement. During calculation, the error component corresponding to the spatial position observation and the observation noise component in the prediction state are added together to obtain the total observation error. Then, the error component corresponding to the prediction state is divided by this total observation error to obtain the filter gain corresponding to each spatial direction.

[0209] As a preferred embodiment, the observation noise covariance is also obtained through pre-calibration.

[0210] During calibration, the calibration object is fixed within the robot's working area. 20 frames of depth data are continuously acquired while the spatial position of the calibration object remains unchanged. The deviation between the measured spatial coordinates of each frame and the average spatial coordinates of the 20 frames is calculated. The deviations in each spatial direction are squared, and the sum of the squared results in the same direction is divided by the corresponding number of samples. The result is used as the observation noise parameter for that direction. The same observation noise parameter is used when the installation position of the depth acquisition unit, the acquisition resolution, and the sampling frequency remain unchanged.

[0211] Furthermore, once the filter gain is determined, the current prediction state is corrected.

[0212] The observation residuals in the three spatial directions are multiplied by the corresponding filter gain, and the resulting correction is added to the predicted spatial position to obtain the corrected spatial position at the current time. The velocity and acceleration are updated synchronously according to the time relationship between the current corrected position and the previous corrected position. Then, the prediction error covariance is corrected using the filter gain, and the corrected state estimation result and error covariance are saved as the input for the state prediction at the next time step.

[0213] In this embodiment, the state estimate is determined based on the spatial position after filtering correction at two consecutive time points.

[0214] Specifically, the coordinate differences between the current time and the previous time in three spatial directions are calculated respectively. The three coordinate differences are squared respectively, and the sum of the three squared results is taken as the square root. The resulting spatial distance is used as the state estimate of the current time. Thus, the state estimate is used to characterize the actual spatial displacement of the interactive object between two adjacent effective observation times after Kalman filtering.

[0215] The preset threshold used to determine the state estimate is determined by static calibration.

[0216] During calibration, a calibration object matching the robot's normal recognition range is fixed within the working area. With the sensors maintaining normal operating parameters, 30 frames of data are continuously acquired. These 30 frames are processed using the same Kalman filtering procedure as in actual operation to obtain state estimates between adjacent frames. The maximum value among all state estimates is selected, and the upper limit of spatial measurement error given in the depth acquisition unit calibration file is added to obtain the preset threshold. Using this method, the preset threshold corresponds to the upper limit of displacement formed by sensor error, image contour jitter, and state estimation error under static object conditions.

[0217] Furthermore, the current state estimate is compared with a preset threshold.

[0218] When the state estimate is greater than the preset threshold, the corrected spatial coordinates corresponding to that moment are determined as valid motion coordinates, and the valid motion coordinates are associated with and saved with the corresponding time marker; when the state estimate is not greater than the preset threshold, the current spatial coordinates are not written into the valid motion coordinate sequence used for trajectory fitting.

[0219] Once at least three valid motion coordinates are obtained within a continuous time period, a spatial coordinate sequence is constructed in chronological order. When fitting the motion trajectory, least-squares fitting is performed on the changes of coordinates in the three spatial directions over time.

[0220] The specific process involves first selecting the fitting order corresponding to the current trajectory segment, then calculating the corresponding fitted coordinate values ​​based on each sampling time, squaring the difference between each fitted coordinate value and the estimated coordinate of the actual state at that time, summing the squared differences of all sampling points, and adjusting the fitting parameters until the sum of the squared differences reaches its minimum value. After fitting is completed in the three spatial directions, the three fitted coordinates are combined according to the same time marker to obtain the continuous motion trajectory of the interactive object in three-dimensional space.

[0221] Example 6:

[0222] S5 includes obtaining the initial motion trajectory of the interactive object, segmenting the initial motion trajectory to obtain a discrete coordinate sequence and a timestamp set; calculating the spatial position change of the interactive object based on the discrete coordinate sequence and timestamp set; if the spatial position change is greater than a preset displacement threshold, determining that the interactive object is in a dynamic change state and generating state identification data; extracting multi-dimensional evolution features based on the state identification data to construct a spatiotemporal mapping matrix and obtain dynamic spatiotemporal evolution information.

[0223] In this embodiment, the initial motion trajectory of the interactive object, formed after the aforementioned Kalman filtering process, is obtained. The initial motion trajectory consists of three-dimensional spatial coordinates arranged in chronological order and the time information corresponding to each three-dimensional spatial coordinate.

[0224] When segmenting the initial motion trajectory, the time interval between two consecutive state updates of the Kalman filter is used as the trajectory sampling interval. Starting from the initial motion trajectory, the sampling time is determined sequentially according to this sampling interval. If the sampling time is consistent with the time of existing coordinates in the initial motion trajectory, the corresponding three-dimensional spatial coordinates are directly read. If the sampling time is between the times corresponding to two existing coordinates, the time interval between the sampling time and the two times before and after is calculated respectively. Then, according to the proportion of the time interval to the total time interval of the two times before and after, the three-dimensional spatial coordinates of the two times before and after are linearly interpolated to obtain the three-dimensional spatial coordinates corresponding to the sampling time. The obtained coordinates are recorded sequentially according to the sampling time to form a discrete coordinate sequence, and the sampling times are combined into a timestamp set.

[0225] The change in spatial location is calculated segment by segment based on the discrete coordinate sequence and the timestamp set.

[0226] Specifically, for two discrete coordinates that are adjacent in time, the coordinate differences between the latter and the former coordinates in three spatial directions are calculated. Each of the three coordinate differences is squared, and the sum of the three squared results is taken as the square root to obtain the spatial displacement between two adjacent sampling times. For all adjacent coordinate segments contained within the same decision time window, the spatial displacements of each segment are accumulated in chronological order, and the accumulated displacement is used as the spatial position change corresponding to that decision time window. The decision time window is fixed at 5 consecutive sampling intervals, ensuring that each state determination is based on data of the same time length.

[0227] As a preferred embodiment, the preset displacement threshold is determined through static calibration before the robot is put into operation.

[0228] Specifically, the calibration object is fixed within the robot's normal working area. Under the condition that the vision acquisition unit, depth acquisition unit, sampling frequency, and trajectory processing parameters are consistent with those in actual operation, data for 30 decision time windows are continuously collected. Following the same trajectory segmentation and spatial position change calculation process as in actual operation, the cumulative displacement corresponding to each of the 30 decision time windows is obtained. The 30 cumulative displacements are compared, and the cumulative displacement with the largest value is taken. This value is then added to the upper limit of spatial measurement error recorded in the depth acquisition unit calibration document, and the resulting value is used as the preset displacement threshold. This displacement threshold remains unchanged when the sensor installation position, sampling frequency, and trajectory segmentation parameters remain unchanged.

[0229] Furthermore, after calculating the spatial position change within the current judgment time window, the obtained spatial position change is compared with a preset displacement threshold.

[0230] When the change in spatial position is greater than the preset displacement threshold, the corresponding interactive object is determined to be in a dynamic change state within the time window, and status identification data is generated; when the change in spatial position is not greater than the preset displacement threshold, the corresponding state of the time window is recorded as a non-dynamic change state.

[0231] Furthermore, a correspondence is established between the status identifier data, the interactive object identifier, and the current judgment time window.

[0232] When generating status identifier data, the object number, dynamic status category, start time, end time, start spatial coordinates, and end spatial coordinates of the judgment time window are recorded. If two consecutive judgment time windows are both judged as dynamic change states, the two status identifiers are associated based on the object number and time continuity. If the subsequent judgment time window no longer meets the dynamic change conditions, the end time of the previous judgment time window is taken as the end time of this dynamic change process.

[0233] When extracting multidimensional evolution features based on state identifier data, position change features, velocity features, velocity change features, motion direction features, and duration features are obtained sequentially from the discrete coordinates corresponding to the time window that is determined to be in a dynamic state.

[0234] The position change feature is obtained by calculating the three-dimensional spatial distance between the starting and ending coordinates of the time window. Specifically, the coordinate difference in the three spatial directions is calculated, the squares of each coordinate difference are summed and the square root is taken. The velocity feature is obtained by removing the spatial position between two adjacent discrete coordinates to obtain the time difference between the corresponding two timestamps, and the arithmetic mean of the velocities of each segment within the current time window is calculated as the motion velocity of the time window.

[0235] In this embodiment, the speed change characteristics are determined by the difference between the speed of movement in the current time window and the speed of movement in the previous time window.

[0236] Specifically, the motion direction feature is obtained based on the coordinate increase and decrease relationship of the start and end coordinates of the time window in three spatial directions, and the motion direction is jointly characterized by the coordinate changes in the three spatial directions; the duration feature is obtained by subtracting the start time from the end time of the current dynamic change process; for continuous dynamic change processes, the corresponding duration is updated synchronously for each additional judgment time window that meets the dynamic change conditions.

[0237] In addition, the spatiotemporal mapping matrix is ​​established in chronological order.

[0238] Each row in the matrix corresponds to one decision time window, and each column is used to store the object number, start time of the time window, end time of the time window, start coordinates of the three spatial directions, end coordinates of the three spatial directions, spatial position change, movement speed, speed change, coordinate change of the three spatial directions, and dynamic status identifier. For the same interactive object, the data corresponding to each decision time window is written into the matrix in the order of timestamps, so that the changes in the object's spatial position and movement status form a continuous record.

[0239] Furthermore, after completing the construction of the spatiotemporal mapping matrix, matrix data belonging to the same interactive object are extracted according to the object number, and continuous dynamic state records are connected in chronological order to obtain the dynamic spatiotemporal evolution information corresponding to the object.

[0240] The dynamic spatiotemporal evolution information reflects the changes in the spatial position, speed, and direction of movement of the interactive object over a continuous period of time, as well as the continuous process of its dynamic state.

[0241] Example 7:

[0242] S6 includes acquiring dynamic spatiotemporal evolution information and extracting motion state and physical attributes from a set of interactive objects; performing spatial mapping using a multi-sensor fusion algorithm based on motion state and physical attributes to obtain a fusion mapping matrix; if the coordinate offset of the fusion mapping matrix is ​​greater than a preset threshold, resampling the fusion mapping matrix to determine the target mapping matrix; and constructing a topology update graph based on the target mapping matrix to render and generate a real-time updated environment map.

[0243] In this embodiment, the dynamic spatiotemporal evolution information is first mapped to the set of interactive objects based on the object identifier.

[0244] For the same interactive object, the three-dimensional spatial position, motion speed, motion direction and dynamic state duration corresponding to the current moment are read from the dynamic spatiotemporal evolution information, and the spatial size, equivalent mass, support area and force state values ​​are read from the interactive object set. The above data are merged into the object state record at the current moment according to the object identifier and timestamp, and used as the input for subsequent spatial mapping.

[0245] When spatially mapping object state data obtained from different sensors, the robot base coordinate system is used as a unified coordinate reference; for the three-dimensional coordinates obtained by the depth acquisition unit, coordinate transformation is performed based on the pre-calibrated rotation and translation relationships between the depth acquisition unit and the robot base.

[0246] In the specific processing, the coordinate components in the sensor coordinate system are first multiplied and accumulated according to the rotation parameters in the three spatial directions to obtain the coordinates after the direction transformation. Then, the translation amounts in the three spatial directions are added respectively to obtain the spatial coordinates in the robot base coordinate system. For the target contour obtained by the vision acquisition unit, the three-dimensional spatial coordinates are first recovered by combining the depth value corresponding to the contour, and then transformed to the robot base coordinate system according to the same calibration relationship.

[0247] In addition, data from different sensors are matched according to timestamps before being fused.

[0248] Specifically, the time difference between the timestamp of each sensor data and the current fusion time is calculated, and the data with the smallest time difference is selected to participate in the current fusion. The maximum time difference allowed to participate in the fusion is the maximum value among the sampling periods of each sensor. When the time difference of a certain sensor data exceeds this value, the data is not written into the current fusion result.

[0249] When the same interactive object has more than two valid spatial coordinates, the fusion weight is determined based on the coordinate measurement error of each sensor.

[0250] The coordinate measurement error is obtained during the sensor calibration stage. A calibration object with known spatial coordinates is fixed in the robot's working area, and 30 frames of data are continuously collected. The absolute difference between the measured coordinates and the known coordinates in each frame is calculated in three spatial directions. Then, the arithmetic mean of the 30 absolute differences in the same direction is calculated, and the result is taken as the measurement error of the sensor in the corresponding direction. When performing data fusion, the reciprocal of the measurement error of each sensor is taken first, and then each reciprocal is divided by the sum of all the reciprocals to obtain the fusion weight of the corresponding sensor. The coordinates of each sensor in the same spatial direction are multiplied by the corresponding fusion weight and then summed to obtain the fused coordinates in that direction. The above processing is performed on the three spatial directions to obtain the fused three-dimensional coordinates of the interactive object.

[0251] Furthermore, after completing the spatial location fusion, the fused 3D coordinates, motion speed, motion direction, spatial size, equivalent mass, force state value and timestamp corresponding to the same interactive object are written into the same data unit, and the data units are arranged in order according to the object identifier to obtain the fusion mapping matrix.

[0252] Each data unit in the fusion mapping matrix corresponds to one interactive object, so that the object's spatial position, motion state, and physical attributes are in the same data record.

[0253] Furthermore, after the fusion mapping matrix is ​​formed, its coordinate offset relative to the target mapping matrix at the previous time step is calculated.

[0254] Specifically, for interactive objects with the same object identifier at two different time points, the coordinate differences between the current fused coordinates and the coordinates at the previous time point are calculated in three spatial directions. The three coordinate differences are squared, and the sum of the three squared results is taken as the square root to obtain the spatial offset distance corresponding to the object. The spatial offset distances are calculated for all interactive objects at the current time point that have a corresponding relationship with the previous time point. The spatial offset distances are then added together and divided by the number of objects involved in the calculation. The result is used as the coordinate offset of the current fused mapping matrix.

[0255] As a preferred embodiment, the preset threshold is determined by static spatial mapping calibration.

[0256] During calibration, three fixed calibration objects with known spatial coordinates are set within the robot's working area. These three calibration objects are located at the front, middle, and rear of the robot's effective sensing distance, respectively. The robot body remains stationary and performs 30 consecutive spatial mappings using the same sensor acquisition parameters and fusion processing flow as during normal operation. After each mapping, the spatial distance between the fused coordinates of the three calibration objects and their known coordinates is calculated, and the arithmetic mean of the three spatial distances is taken to obtain the coordinate error corresponding to that mapping. The 30 coordinate errors are compared, and the maximum value is taken. This value is then added to the upper limit of the positioning error recorded in the robot pose detection unit calibration file, and the resulting value is set as the preset threshold corresponding to the coordinate offset.

[0257] Specifically, when the coordinate offset of the fused mapping matrix is ​​not greater than a preset threshold, the current fused mapping matrix is ​​directly used as the target mapping matrix; when the coordinate offset is greater than the preset threshold, resampling is performed on the current fused mapping matrix.

[0258] Resampling establishes a three-dimensional grid with fixed side lengths based on the robot base coordinate system. The grid side length is taken as the spatial sampling interval corresponding to the depth acquisition unit within the current working distance, and remains fixed while the sensor acquisition parameters remain unchanged.

[0259] In addition, for each interactive object in the fusion mapping matrix, its grid position is determined based on its fused 3D coordinates.

[0260] If the same object has multiple fused coordinates in the same grid, the arithmetic mean of the coordinates in the three spatial directions is calculated, and the average coordinates are used as the resampled coordinates of the object in that grid. If a missing grid appears in the continuous trajectory due to mapping offset, the valid grids that are directly adjacent to the missing grid in the three spatial directions and have the same object identifier are selected as the interpolation source. The spatial distance from the center of the missing grid to the center of each valid grid is calculated, and the reciprocal of each spatial distance is taken and normalized as the interpolation weight. The coordinates in each valid grid are multiplied by the corresponding weight and summed to obtain the resampled coordinates corresponding to the missing grid. After completing the grid resampling of all objects, the data is rearranged according to the object identifier and grid index to obtain the target mapping matrix.

[0261] In this embodiment, when constructing the topology update graph based on the target mapping matrix, the data unit corresponding to each interactive object is established as a topology node, and the object identifier, three-dimensional spatial position, movement speed, movement direction and physical attributes are written into the node attributes.

[0262] For any two nodes, calculate their coordinate differences in the three spatial directions, square the three coordinate differences, sum them, and take the square root to obtain the spatial distance between the two nodes. Compare this spatial distance with the preset adjacency distance. If the spatial distance is not greater than the preset adjacency distance, establish a connection between the two nodes.

[0263] In this embodiment, the preset adjacency distance is determined based on the maximum straight-line reachable distance of the end effector when the humanoid robot performs interactive actions.

[0264] After the robot completes the mechanical parameter calibration, the maximum linear distance that the end effector can reach relative to the robot base is calculated within the joint range of motion, and this distance is used as the preset adjacency distance. The preset adjacency distance remains unchanged if the robot's mechanical structure and joint range of motion do not change.

[0265] During topology update, the current target mapping matrix is ​​matched with the topology update graph of the previous time step based on the object identifier.

[0266] For existing object nodes, update the node's 3D spatial position, motion speed, motion direction, and physical attributes, and recalculate the connection relationship with other nodes based on the updated spatial position; for newly appearing objects in the current target mapping matrix, create new nodes and generate corresponding connections according to the preset adjacency distance; for objects that have not appeared in the target mapping matrix for three consecutive fusion cycles, delete their corresponding nodes and associated connections.

[0267] Furthermore, after completing the topology update, the three-dimensional spatial coordinates of each node are converted into map coordinates according to the spatial resolution of the environment map, and the occupied area is determined based on the spatial size of the corresponding object.

[0268] For objects in a dynamic state, their movement direction and speed are associated with the corresponding map area; for objects whose positions are updated, the occupied area is updated according to the latest coordinates in the current target mapping matrix; after refreshing each node and connection relationship according to the timestamp of the current fusion time, a real-time updated environment map is generated, so that the environment map continuously records the position changes, movement state, physical attributes and mutual spatial adjacency relationships of interactive objects.

[0269] Example 8:

[0270] S7 includes acquiring the spatiotemporal evolution features of the environmental map to obtain the predicted trajectory of the interactive object; dynamically intersecting the predicted trajectory with the reachable space to determine the interactive reachable domain; calculating the migration vector based on the centroid coordinates of the interactive reachable domain; if the migration vector is greater than a preset threshold, obtaining the migration direction and velocity of the interactive reachable domain; adjusting the driving torque based on the migration direction and velocity; and generating an interaction decision for the interactive object based on the adjusted driving torque.

[0271] In this embodiment, the spatiotemporal evolution features corresponding to the currently interactive objects are first extracted from the real-time updated environment map.

[0272] Specifically, the spatiotemporal evolution features include three-dimensional spatial coordinates of continuous moments, timestamps corresponding to each moment, motion speed, and motion direction; when performing trajectory prediction, the spatial coordinates of the three consecutive sampling moments before the current moment are read in chronological order, the coordinate differences of two adjacent sampling moments in the three spatial directions are calculated respectively, and divided by the time interval between the corresponding timestamps to obtain the motion speed in each spatial direction; then the motion speed of the current sampling moment is subtracted from the motion speed of the previous sampling moment, and divided by the time interval between the two speed calculations to obtain the motion acceleration in each spatial direction.

[0273] Furthermore, using the current spatial coordinates as the prediction starting point, subsequent prediction times are set sequentially according to the update interval of the environmental map.

[0274] Specifically, for each predicted moment, the current velocity is first multiplied by the time interval from the current moment to the predicted moment to obtain the position change caused by the velocity; then, the acceleration is multiplied by the square of the time interval and divided by 2 to obtain the position change caused by the acceleration; these two position changes are sequentially added to the current spatial coordinates to obtain the spatial position at the corresponding predicted moment; the predicted positions are arranged in chronological order and connected to form the predicted trajectory of the interactive object. Each time the environmental map is updated, the predicted trajectory is recalculated using the data corresponding to the latest three sampling moments.

[0275] When determining the reachable domain, the current reachable space is first calculated based on the current joint angles, range of motion of each joint, and geometric dimensions of each link of the humanoid robot.

[0276] Specifically, the process involves generating joint pose combinations within the allowable angle range of each joint, calculating the three-dimensional coordinates of the end effector corresponding to each set of joint poses based on the robot's forward kinematics, and then mapping all the end effector three-dimensional coordinates onto the spatial grid of the environment map. The reachable space at the current moment is composed of the reachable grids.

[0277] Then, the spatial coordinates of each predicted trajectory are read in the order of prediction time, and it is determined whether the spatial raster where the predicted coordinate is located belongs to the reachable space corresponding to the same prediction time. If the predicted coordinate is located in the reachable space, the spatial raster corresponding to the predicted coordinate is recorded as an interactive reachable raster, and the corresponding prediction time is recorded. If the predicted coordinate is located outside the reachable space, the interactive reachable result is not written. The interactive reachable rasters corresponding to consecutive prediction times are collected in the order of time to obtain the interactive reachable domain updated over time.

[0278] Furthermore, after obtaining the interactive reachable domain, its centroid coordinates are calculated.

[0279] The coordinates of the center points of all grids in the current interactive reachable domain in the three spatial directions are summed, and then divided by the number of grids contained in the interactive reachable domain. The resulting three average coordinate values ​​are used as the centroid coordinates of the current interactive reachable domain.

[0280] In this embodiment, the migration vector is calculated based on the centroid coordinates of the interactive reachable domain obtained from two consecutive environmental map updates.

[0281] Specifically, the coordinates of the centroid at the current moment in each of the three spatial directions are subtracted from the corresponding coordinates of the centroid at the previous moment to obtain the coordinate changes in the three directions. The three coordinate changes together determine the migration direction from the previous centroid position to the current centroid position. The three coordinate changes are then squared, and the sum of the three squared results is taken as the square root. The resulting spatial distance is used as the numerical value of the migration vector.

[0282] In this embodiment, the preset threshold used to determine the migration vector is determined by static calibration.

[0283] During calibration, the interactive calibration object is fixed within the robot's reachable space. With the object's position unchanged, 30 consecutive interactive reachability calculations are performed according to the environmental map update interval during actual operation. The spatial distance between the centroids of two adjacent interactive reachable domains is calculated sequentially, resulting in 29 centroid offset values. The maximum value among these 29 centroid offset values ​​is selected, and then the environmental map is added... Figure 1 The distance corresponding to the side length of each spatial grid is used as the preset threshold for migration determination. The same preset threshold is used when the environmental map resolution, sensor installation location, and update time interval do not change.

[0284] Specifically, when the value of the migration vector is greater than a preset threshold, it is determined that the interactive reachable domain has migrated, and the migration direction is determined according to the coordinate increase or decrease relationship of the current centroid relative to the previous centroid in three spatial directions; the migration speed is determined according to the spatial distance between the current centroid and the previous centroid and the time difference between the corresponding timestamps of the two centroid calculations, specifically by dividing the spatial distance by the time interval between the two timestamps to obtain the migration speed of the interactive reachable domain.

[0285] Furthermore, after obtaining the migration direction and speed, the joint driving torques of the humanoid robot participating in interactive actions are adjusted.

[0286] First, the target spatial position of the end effector in the next control cycle is determined based on the current centroid coordinates and migration direction of the interactive reachable domain. Then, the target joint angles of each participating joint are obtained through robot inverse kinematics calculation. During the calculation, the target spatial position is input into the robot joint geometry model, and the solution is obtained step by step according to the length of each link and the joint connection relationship, so that the joint angle corresponding to the end effector reaching the target spatial position is taken as the target joint angle.

[0287] Then, the position control quantity is calculated based on the angle difference between the current joint angle and the target joint angle, and the speed control quantity is calculated based on the speed difference between the current joint angular velocity and the target joint angular velocity.

[0288] The target joint angular velocity is determined based on the migration speed of the interactive reachable domain. That is, the spatial migration distance of the interactive reachable domain within one control cycle is first converted into the target displacement of the end effector. Then, according to the inverse kinematics relationship, it is converted into the angular change that each joint needs to complete within the same control cycle. The angular change is divided by the control cycle to obtain the target angular velocity of the corresponding joint.

[0289] As a preferred embodiment, the driving torque adjustment value of each joint is determined by both position control and speed control.

[0290] Specifically, the joint angle difference is multiplied by the pre-calibrated position control gain, and the joint angular velocity difference is multiplied by the pre-calibrated speed control gain. The two results are then added together, and the resulting value is used as the driving torque adjustment value corresponding to the current control cycle. The position control gain and speed control gain are determined during the robot joint control parameter calibration stage. During calibration, the corresponding joint is made to perform 10 movements from the initial angle to the target angle under rated load conditions. The angle tracking error and speed tracking error under each set of control parameters are recorded. The control parameter that ensures the angle tracking error does not exceed the angle error corresponding to one counting unit of the joint encoder and the speed tracking error does not exceed the speed error upper limit set by the controller is selected as the control gain of the joint.

[0291] Furthermore, the driving torque adjustment value is increased to the current driving torque of the joint to obtain the adjusted driving torque.

[0292] If the adjusted driving torque exceeds the rated allowable output torque of the joint, the driving torque will be limited to the rated allowable output torque; if the adjusted driving torque does not exceed the rated allowable output torque, the calculated driving torque will be used as the control output for the current control cycle.

[0293] After adjusting the driving torque of each participating joint, an interaction decision is generated based on the interactive object identifier, predicted trajectory, interactive reachable domain, target interaction position, target arrival time, and the adjusted driving torque of each joint.

[0294] The interaction decision is used to determine the start time, target position, joint movement direction and driving torque of the humanoid robot to perform an interaction action on the current interactive object; as the environment map is continuously updated, the predicted trajectory, interactive reachable domain and its migration state are recalculated, and the driving torque and interaction decision corresponding to the subsequent control cycle are updated accordingly.

[0295] Example 9:

[0296] A humanoid robot adaptive interaction system for dynamic scenes is also provided, used to implement the steps of the humanoid robot adaptive interaction method for dynamic scenes, the system comprising:

[0297] The image sequence generation module acquires continuous visual and depth data in the dynamic scene where the humanoid robot is located, determines the dynamic region based on the depth changes in consecutive frames, and uses the temporal consistency constraint of the dynamic region to constrain the convolutional neural network to perform feature extraction and alignment processing on the visual and depth data to obtain an image sequence.

[0298] The target entity contour acquisition module uses edge detection operators to extract the pixel contours of various entities in the image sequence to obtain a set of target entity contours.

[0299] The interactive object determination module extracts the physical attributes of the corresponding entities based on the target entity outline set. If the physical attributes meet the preset operable conditions, the entity is determined to be an interactive object, and a set of interactive objects is obtained.

[0300] The motion trajectory estimation module, for the set of interactive objects, adaptively corrects the state transition parameters of the Kalman filter based on the physical properties of the interactive objects and the contour changes at adjacent time points, and performs state estimation in the continuous time dimension to obtain the motion trajectory of the interactive objects.

[0301] The dynamic spatiotemporal evolution information acquisition module acquires the motion trajectory and calculates the change in spatial position of the interactive object on a continuous time axis. If the change in spatial position is greater than a preset displacement threshold, it determines that the interactive object is in a dynamic change state and obtains dynamic spatiotemporal evolution information.

[0302] The environmental map construction module uses a multi-sensor fusion algorithm to spatially map the time-stamped motion state and physical attributes based on dynamic spatiotemporal evolution information and a set of interactive objects, thereby constructing a real-time updated environmental map.

[0303] The interaction decision generation module acquires an environmental map, predicts the future motion state of interactive objects based on the spatiotemporal evolution information contained in the environmental map, dynamically intersects the predicted trajectory with the reachable space of the humanoid robot, determines the interaction reachable domain that changes over time, adaptively adjusts the interaction strategy based on the migration of the interaction reachable domain, and generates interaction decisions for interactive objects.

[0304] In this embodiment, the image sequence generation module continuously receives visual data and depth data, determines dynamic regions based on the depth changes at corresponding positions in adjacent depth frames, and establishes temporal consistency constraints based on the position changes of dynamic regions at continuous time intervals. Under this constraint, visual features and depth features are extracted by a convolutional neural network, and aligned according to temporal and spatial correspondences to generate an image sequence.

[0305] The target entity contour acquisition module performs gradient calculation, edge filtering, and contour connection processing on the image sequence to obtain the pixel contours corresponding to each entity, and forms a target entity contour set according to the entity set; the interactive object determination module extracts spatial size, spatial position, and force-related physical attributes based on the contours of each target entity and the corresponding depth information, and compares the physical attributes with preset operable conditions. Target entities that meet the conditions are determined as interactive objects.

[0306] The motion trajectory estimation module extracts kinematic features based on the spatial position, contour changes, and physical properties of the interactive object at continuous time intervals, and updates the state transition parameters of the Kalman filter accordingly. It obtains the state estimation results at each time interval through continuous state prediction and observation correction, and then connects the corresponding spatial coordinates in chronological order to form the motion trajectory.

[0307] The dynamic spatiotemporal evolution information acquisition module calculates the spatial position change over a continuous time period based on the motion trajectory and compares it with a preset displacement threshold.

[0308] When the change in spatial position exceeds a preset displacement threshold, the corresponding interactive object is marked as dynamically changing, and associated with time, spatial position, speed and direction of movement to form dynamic spatiotemporal evolution information.

[0309] The environment map construction module maps dynamic spatiotemporal evolution information to the physical attributes of interactive objects based on object identifiers and time markers. It also uses a multi-sensor fusion method to convert data acquired by different sensors into a unified robot coordinate system, forming a spatial mapping result that includes object position, motion state, and physical attributes, and continuously updates the environment map accordingly.

[0310] The interactive decision generation module extracts the continuous spatiotemporal state of interactive objects from the environment map, predicts their subsequent spatial position based on their position, speed and direction of motion, and forms a predicted trajectory; then, the predicted trajectory is intersected with the current reachable space of the humanoid robot in chronological order to obtain the interactive reachable domain.

[0311] The migration state is determined based on the positional changes of the reachable domain at continuous intervals, and the humanoid robot's interaction position, movement direction, and joint control parameters are adjusted accordingly to ultimately generate interaction decisions for the corresponding interactive object.

[0312] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An adaptive interaction method for humanoid robots in dynamic scenarios, characterized in that, include: S1. Obtain continuous visual and depth data in the dynamic scene where the humanoid robot is located, and obtain an image sequence; S2. For the image sequence, edge detection operators are used to extract the pixel contours of various entities in the image to obtain the target entity contour set; S3. Based on the target entity outline set, extract the physical attributes of the corresponding entity. If the physical attributes meet the preset operable conditions, then determine that the entity is an interactive object and obtain the interactive object set. S4. For the set of interactive objects, obtain the motion trajectory of the interactive objects; S5. Obtain the motion trajectory and calculate the change in spatial position of the interactive object on the continuous time axis. If the change in spatial position is greater than the preset displacement threshold, it is determined that the interactive object is in a dynamic change state and dynamic spatiotemporal evolution information is obtained. S6. Based on dynamic spatiotemporal evolution information and a set of interactive objects, spatially map the motion states and physical attributes with time stamps to construct a real-time updated environment map. S7. Predict the future motion state of the interactive object based on the spatiotemporal evolution information contained in the environmental map, and dynamically intersect the predicted trajectory with the reachable space of the humanoid robot to determine the interactive reachable domain that changes over time. Adaptively adjust the interaction strategy according to the migration of the interactive reachable domain to generate interaction decisions for the interactive object.

2. The adaptive interaction method for humanoid robots in dynamic scenes according to claim 1, characterized in that: S1 includes: Continuous visual data and continuous depth data in the dynamic scene where the humanoid robot is located are acquired, and the continuous visual data and the continuous depth data are synchronously matched to obtain a synchronous visual frame sequence and a synchronous depth frame sequence. The depth difference is calculated based on the synchronous depth frame sequence to determine the initial dynamic region, and the temporal consistency score is calculated based on the motion trajectory of the initial dynamic region to obtain the target dynamic region. Using the target dynamic region as a constraint, a convolutional neural network is used to extract features from the synchronized visual frame sequence and the synchronized depth frame sequence to obtain visual feature maps and depth feature maps. The visual feature map and the depth feature map are aligned, and an image sequence is generated based on the aligned feature map.

3. The adaptive interaction method for humanoid robots in dynamic scenarios according to claim 1, characterized in that: S2 includes: Gradient calculation is performed on the first image sequence to generate a gradient feature map sequence; Non-maximum suppression and double threshold classification are performed on the gradient feature map sequence to obtain a classified edge image sequence containing strong edge points and weak edge points; Weak edge points are filtered out based on the connectivity of strong edge points in the classified edge image sequence to obtain a continuous edge image sequence; A closing operation is performed on a continuous edge image sequence, and the pixel contours of various entities in the continuous edge image sequence are extracted to obtain a set of target entity contours.

4. The adaptive interaction method for humanoid robots in dynamic scenes according to claim 1, characterized in that: S3 includes: Obtain the target entity outline set, and determine the initial physical properties of the target entity based on the target entity outline set; Simulate the force state based on the initial physical properties to obtain the current force state value; Determine whether the current force state value meets the preset operable conditions, including whether the current force state value is greater than the preset interaction threshold; If the current force value is greater than the preset interaction threshold, the target entity is determined to meet the preset operable conditions. If the target entity meets the preset operable conditions, the target entity is determined to be an interactive object, and at least one interactive object is aggregated to obtain an interactive object set.

5. The adaptive interaction method for humanoid robots in dynamic scenes according to claim 1, characterized in that: S4 includes: Obtain the physical properties and contour change data of interactive objects, and extract initial kinematic features based on the physical properties and contour change data; The state transition matrix is ​​calculated based on the initial kinematic characteristics, and the Kalman filter algorithm is updated using the state transition matrix to obtain the filter gain. The state estimate is obtained by performing state estimation through the filter gain; If the state estimate is greater than the preset threshold, the motion trajectory of the interactive object is obtained by fitting a spatial coordinate sequence based on the state estimate.

6. The adaptive interaction method for humanoid robots in dynamic scenes according to claim 1, characterized in that: S5 includes: Obtain the initial motion trajectory of the interactive object, segment the initial motion trajectory, and obtain a discrete coordinate sequence and a set of timestamps; The spatial position change of interactive objects is calculated based on the discrete coordinate sequence and timestamp set. If the change in spatial position is greater than the preset displacement threshold, the interactive object is determined to be in a dynamic change state and status identification data is generated. Multidimensional evolution features are extracted from state identifier data to construct a spatiotemporal mapping matrix and obtain dynamic spatiotemporal evolution information.

7. The adaptive interaction method for humanoid robots in dynamic scenes according to claim 1, characterized in that: S6 includes: Obtain dynamic spatiotemporal evolution information and interactive object set to extract motion state and physical properties; Based on the motion state and physical properties, a multi-sensor fusion algorithm is used to perform spatial mapping to obtain a fusion mapping matrix; If the coordinate offset of the fused mapping matrix is ​​greater than a preset threshold, the fused mapping matrix is ​​resampled to determine the target mapping matrix. A topology update graph is constructed based on the target mapping matrix to render and generate a real-time updated environment map.

8. The adaptive interaction method for humanoid robots in dynamic scenes according to claim 1, characterized in that: S7 includes: To obtain the spatiotemporal evolution characteristics of the environment map and thus predict the trajectory of interactive objects; The predicted trajectory is dynamically intersected with the reachable space to determine the interactive reachable domain.

9. The adaptive interaction method for humanoid robots in dynamic scenes according to claim 8, characterized in that: The S7 also includes: The migration vector is calculated based on the centroid coordinates of the interactive reachable domain. If the migration vector is greater than a preset threshold, the migration direction and velocity of the interactive reachable domain are obtained. The driving torque is adjusted according to the migration direction and speed, and the adjusted driving torque is used to generate interaction decisions for interactive objects.

10. A humanoid robot adaptive interaction system for dynamic scenes, used to implement the steps of the humanoid robot adaptive interaction method for dynamic scenes according to any one of claims 1-9, characterized in that, The system includes: The image sequence generation module acquires continuous visual and depth data in the dynamic scene where the humanoid robot is located, determines the dynamic region based on the depth changes in consecutive frames, and uses the temporal consistency constraint of the dynamic region to constrain the convolutional neural network to perform feature extraction and alignment processing on the visual and depth data to obtain an image sequence. The target entity contour acquisition module uses edge detection operators to extract the pixel contours of various entities in the image sequence to obtain a set of target entity contours. The interactive object determination module extracts the physical attributes of the corresponding entities based on the target entity outline set. If the physical attributes meet the preset operable conditions, the entity is determined to be an interactive object, and a set of interactive objects is obtained. The motion trajectory estimation module, for the set of interactive objects, adaptively corrects the state transition parameters of the Kalman filter based on the physical properties of the interactive objects and the contour changes at adjacent time points, and performs state estimation in the continuous time dimension to obtain the motion trajectory of the interactive objects. The dynamic spatiotemporal evolution information acquisition module acquires the motion trajectory and calculates the change in spatial position of the interactive object on a continuous time axis. If the change in spatial position is greater than a preset displacement threshold, it determines that the interactive object is in a dynamic change state and obtains dynamic spatiotemporal evolution information. The environmental map construction module uses a multi-sensor fusion algorithm to spatially map the time-stamped motion state and physical attributes based on dynamic spatiotemporal evolution information and a set of interactive objects, thereby constructing a real-time updated environmental map. The interaction decision generation module acquires an environmental map, predicts the future motion state of interactive objects based on the spatiotemporal evolution information contained in the environmental map, dynamically intersects the predicted trajectory with the reachable space of the humanoid robot, determines the interaction reachable domain that changes over time, adaptively adjusts the interaction strategy based on the migration of the interaction reachable domain, and generates interaction decisions for interactive objects.

Images