A real-time human-computer interaction intention strength recognition method

Abstract

This invention relates to the field of robot interaction technology, specifically to a method for recognizing the intensity of human-computer interaction intentions in real time. The method includes the following steps: S1, acquiring video of pedestrians through a collection unit mounted on the robot, performing frame extraction on the video, and using a 3D skeleton extraction model to extract the skeleton information of each pedestrian in each frame in chronological order; S2, based on the skeleton information of each pedestrian in each frame, using a preset lightweight motion behavior recognition model to identify the interaction reference information of each pedestrian; S3, analyzing and processing the three types of interaction reference information—straight-line distance, orientation information, and motion behavior—to obtain the real-time interaction intention intensity of the pedestrian relative to the robot. This invention can accurately identify the interaction intentions of pedestrians, thereby improving the initiative and naturalness of social robots when interacting with humans.

Description

Technical Field

[0001] This invention relates to the field of robot interaction technology, and more specifically to a method for recognizing the intensity of human-computer interaction intentions in real time. Background Technology

[0002] The rise of social robots has brought new opportunities to the development of various industries in society. People can obtain the information or help they need by communicating with these social robots.

[0003] Currently, most interactions between social robots and pedestrians are triggered by verbal commands. While robots can understand verbal commands and recognize human intentions, this interaction mode requires humans to constantly issue explicit commands to the robot, which contradicts the original design intent of social robots. In other words, most existing social robots can only passively respond to user requests. Furthermore, because most users are unfamiliar with the specific functions of social robots and may not know if they can function properly, many users are hesitant to consult them, rendering them largely ineffective. Additionally, for social robots with realistic human appearances, such as Sophia and Nadine, unnatural interactions can frighten people. To improve the naturalness of these social robot interactions, it is necessary not only to enhance the smoothness of their movements but also to further improve their social environment awareness to increase their proactive interaction.

[0004] Therefore, a real-time human-computer interaction intent intensity recognition method is needed to accurately identify the interaction intent of pedestrians, thereby improving the initiative and naturalness of social robots when interacting with people, enabling social robots to better respond to users' needs or psychological states, and thus enhancing the naturalness and harmony of human-computer interaction. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, this invention provides a real-time human-computer interaction intent intensity recognition method, which can accurately identify the interaction intent of pedestrians, thereby improving the initiative and naturalness of social robots when interacting with people.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A method for real-time human-computer interaction intent strength recognition includes the following steps:

[0008] S1. After collecting videos of pedestrians through the acquisition unit set on the robot, the video is processed by frame extraction, and the 3D skeleton extraction model is used to extract the skeleton information of each pedestrian in each frame in chronological order.

[0009] S2, based on the skeleton information of each pedestrian in each frame of the image, a preset lightweight motion behavior recognition model is used to identify the interaction reference information of each pedestrian; the interaction reference information includes the straight-line distance from the robot, orientation information, and motion behavior; the motion behavior includes approaching, moving away, passing by, and standing.

[0010] S3 analyzes and processes three types of interactive reference information of pedestrians: straight-line distance, orientation information, and movement behavior, to obtain the real-time interaction intent intensity of pedestrians relative to robots.

[0011] S4. Based on the strength of the obtained interaction intent, control the robot to execute the corresponding interaction decision; the interaction decision includes no interaction, tentative interaction, and active interaction.

[0012] Basic working principle and beneficial effects:

[0013] This invention provides a method for identifying the intensity of pedestrian interaction intentions. Specifically, by acquiring dynamic video of pedestrians and extracting frames, interaction reference information, including the straight-line distance to the robot, orientation information, and movement behavior, is identified from each frame in chronological order. Then, these three types of interaction reference information are fused to obtain the real-time interaction intention intensity of the pedestrian relative to the robot. Through comprehensive analysis of distance, orientation information, and movement behavior, the robot can accurately understand the interaction intention intensity of each pedestrian within its field of vision. Based on the interaction intention intensity, the robot can determine whether to initiate interaction and, if so, what interaction decision to make (e.g., tentative interaction or direct proactive interaction).

[0014] This approach takes into account the dynamic characteristics of pedestrians over a period of time. Compared with some existing methods that rely on static single-frame image recognition (such as recognizing pedestrians' expressions and postures to understand their interaction intentions), it offers higher reliability and greater precision (able to provide specific interaction decisions). Furthermore, this invention utilizes a lightweight neural network model to obtain interaction reference information such as the pedestrian's straight-line distance, orientation, and movement behavior. The algorithm runs quickly and can be used even by mobile robots with limited computing power, ensuring recognition efficiency while maximizing the naturalness of robot-human interaction.

[0015] In summary, this invention can accurately identify the interaction intentions of travelers, thereby improving the initiative and naturalness of social robots when interacting with people.

[0016] Preferably, in S2, the lightweight motion behavior recognition model is an improved ConvLSTM network model with only one ConvLSTM layer and a higher-order feature extraction layer between the input layer and the ConvLSTM layer; the higher-order feature extraction layer consists of a higher-order feature extractor and a gate function; there are three higher-order feature extractors, which are used to extract the straight-line distance, orientation information and inter-frame change feature information after processing the skeleton information of each frame; the gate function is used to select the feature information to be sent to the ConvLSTM network layer.

[0017] Beneficial effects: Having only one ConvLSTM layer ensures the lightweight nature of the motion behavior recognition model, thereby guaranteeing the real-time nature of interactive reference information. With the addition of a higher-order feature extraction layer, three higher-order feature extractors enable the ConvLSTM to learn deeper information from the dataset, while the gate function can select the specific feature information fed into the ConvLSTM network layer, ensuring the effectiveness of the lightweight motion behavior recognition model's recognition results.

[0018] Preferably, the working process of the high-order feature extractor for extracting straight-line distance includes: extracting the three-dimensional coordinate information of the pedestrian skeleton in a preset camera coordinate system; the three-dimensional coordinate system takes the position of the acquisition unit as the origin, the y-axis represents the position information in the depth direction of the acquisition unit, the x-axis represents the position information in the horizontal direction perpendicular to the y-axis, and the z-axis represents the position information in the vertical direction;

[0019] The pedestrian's position is represented by the coordinates of the neck in the skeleton points; after treating the height of the acquisition unit and the head of all interactive objects as the same height, the straight-line distance of the pedestrian is calculated using the following formula:

[0020]

[0021] In the formula, The straight-line distance. Let x be the coordinates of the key points of a person's neck on the x-axis in a three-dimensional coordinate system. r These are the coordinates of the acquisition unit on the x-axis in a three-dimensional coordinate system. Let y be the coordinates of the key points of a person's neck in a three-dimensional coordinate system on the y-axis. r This represents the coordinates of the acquisition unit on the y-axis in a three-dimensional coordinate system.

[0022] Beneficial effects: This setup ensures the accuracy of extracting the straight-line distance between pedestrians and the robot while simplifying the processing, resulting in high stability and efficiency. Furthermore, treating the height of the acquisition unit and the head of all interactive objects as the same height not only provides a more intuitive reflection of the straight-line distance during human-computer interaction but also eliminates the influence of pedestrian height on the interaction distance.

[0023] Preferably, the working process of the high-order feature extractor for extracting orientation information includes: calculating the angle of the pedestrian's body orientation relative to the robot using the three-dimensional coordinates of the pedestrian's left and right shoulders. Then, calculate the coordinates of the intersection point of the ray pointing towards the pedestrian's body and the z-plane where the robot is located. Recalculate Horizontal distance between the robot's position and through Indicates the direction of pedestrians;

[0024] In the formula, a o Indicates the scaling factor, The x-axis represents the coordinates of the intersection point of the pedestrian's body orientation and the vertical plane where the data acquisition unit is located. r This represents the coordinate value of the acquisition unit on the x-axis in a three-dimensional coordinate system.

[0025] Beneficial effects: In existing research on human-computer interaction, pedestrian orientation information relative to the robot is usually directly used... This is used to represent the strength of an interaction intention. However, assuming all other factors are equal, using orientation angle to reflect the strength of an interaction intention is not very objective. For example, pedestrian 1 and pedestrian 2 may be at the same distance and have the same body orientation angle relative to the robot, but they may simply be located on the same arc centered on the robot's data collection unit. Obviously, their interaction intentions will be completely different in this case. This invention introduces... To better describe the pedestrian's orientation information, specifically the intersection of the ray pointing towards the body and the robot's z-plane. Simultaneously, using... The horizontal distance between the robot's position and its location (which the applicant also refers to as orientation offset) is used to represent the relative intensity of interaction intent indicated by different orientations, and a specific calculation method is provided as a formula. This approach allows for a more realistic description of pedestrian orientation information.

[0026] Preferably, when At that time, set The maximum value is 100, when At that time, set The value is uniformly set to 104.

[0027] Beneficial effects: To prevent Too large, while increasing its distinguishability under different behaviors. Set when... hour, The maximum value is 100; while when At this time, the pedestrian is facing away from the robot, and their interaction intention is very weak, therefore this setting is... The value is uniformly set to 104. This ensures the validity of the extracted orientation information.

[0028] Preferably, the working process of the high-order feature extractor for extracting inter-frame variations includes:

[0029] Calculate the change in distance of pedestrians along the x-axis and the change in distance of pedestrians along the y-axis.

[0030]

[0031] Where k represents the current frame number, The x-axis coordinates of key points on the pedestrian's neck. Let a be the y-axis coordinate of the key points on the pedestrian's neck. nx With a ny Let a be the scaling factor, and a nx =a ny =10;

[0032] like Then the motion behavior is stationary; if and Then the movement behavior is close; if and Then the movement behavior is to move away; if Then the movement behavior is a process.

[0033] Beneficial effects: These four different movement behaviors—stationary, approaching, moving away, and passing by— and These two features differ significantly, and the introduction of... and These two features can effectively reflect the characteristics of different motion behaviors, enabling accurate and rapid training of the corresponding algorithm model and ensuring the accuracy of motion behavior recognition in practical applications.

[0034] Preferably, in S3, the Mamdani fuzzy inference system is used to fuse three types of interactive reference information: the straight-line distance, orientation information, and motion behavior of the pedestrian. The working process of the Mamdani fuzzy inference system includes: fuzzifying the input straight-line distance, orientation information, and motion behavior according to preset fuzzification rules; inferring and analyzing the pedestrian's interaction intention according to preset fuzzy inference rules to obtain the fuzzy value of the pedestrian's interaction intention; and then defuzzifying the fuzzy value of the interaction intention using a defuzzification algorithm to obtain the precise value of the pedestrian's interaction intention.

[0035] The fuzzy inference rules include dynamic inference rules and static inference rules; static rules are used to determine the intensity level of interaction intent when a pedestrian is standing, based on distance and orientation information; dynamic rules are used to determine the intensity level of interaction intent when a pedestrian is approaching, passing by, or moving away, based on straight-line distance and action behavior.

[0036] Beneficial effects: This invention utilizes fuzzy reasoning to calculate the intensity of interaction intentions of different pedestrians in real time, which can further enhance the naturalness of robot-human interaction. Furthermore, this processing method can effectively integrate straight-line distance, orientation information, and motion behavior, and the final result (the precise value of the pedestrian's interaction intention) can serve as specific reference information for the robot's interaction decisions, allowing for rapid analysis of the interaction decisions the robot should execute.

[0037] By using dynamic reasoning rules to analyze whether a pedestrian is standing or not, compared to using a single set of reasoning rules for all states, the reasoning rules can better match the specific state of the pedestrian and have better accuracy. On the other hand, the reasoning rules can also be simpler, clearer, and easier to create.

[0038] Preferably, the fuzzy values ​​of the interactive intent are defuzzified using the area centroid method:

[0039]

[0040] In the formula, y is the precise value of the pedestrian's interaction intent after deblurring. for The point where the maximum value is obtained. The membership function represents the output quantity of a fuzzy rule, fuzzy set Bi.

[0041] Beneficial effects: This algorithm can quickly and reliably obtain accurate values ​​of pedestrian interaction intentions.

[0042] Preferably, in S3, when defuzzifying the fuzzy value of the interaction intention, the range of the obtained precise value of the pedestrian's interaction intention is normalized to [0,1].

[0043] In S4, if the precise value of the pedestrian's interaction intention is in the range of [0-0.4], the interaction decision is to not interact; if the precise value of the pedestrian's interaction intention is in the range of (0.4-0.7], the interaction decision is to attempt an interaction; if the precise value of the pedestrian's interaction intention is in the range of (0.7-1.0], the interaction decision is to initiate an interaction.

[0044] Beneficial effects: Normalization makes the precise values ​​of pedestrian interaction intentions more concise and clear, facilitating subsequent analysis of specific interaction decisions. If the precise value of the pedestrian's interaction intention falls within [0-0.4], it indicates a weak interaction intention, therefore the robot does not generate any proactive interaction behavior; if it falls within (0.4-0.7), it indicates a possibility of interaction between the pedestrian and the robot, therefore the robot performs tentative interaction behavior, such as greeting the pedestrian or asking if they need help; if it falls within (0.7-1.0), it indicates a clear willingness of the pedestrian to interact with the robot, and the robot performs proactive interaction behavior. This setting allows the robot to select appropriate interaction targets and make reasonable interaction decisions.

[0045] Preferably, in step S3, a preset DOM model is used to analyze and process three types of interactive reference information: the pedestrian's straight-line distance, orientation information, and movement behavior, to obtain the real-time interaction intent intensity of the pedestrian relative to the robot; the DOM model includes mutually perpendicular S-axis and D-axis; the S-axis represents static information, including the pedestrian's distance intent value. and orientation intention value The D-axis represents dynamic information, including the pedestrian's movement intention value.

[0046] S3 includes:

[0047] S31. Calculate the corresponding distance target value based on the straight-line distance of the pedestrian.

[0048]

[0049] Among them, b d It is the maximum amplitude, and b d =10; σ is the straight-line distance for pedestrians. d These are preset scaling parameters used to adjust the horizontal axis span;

[0050] S32. Calculate the corresponding orientation intention value based on the pedestrian's orientation information.

[0051]

[0052] Among them, b o For the maximum amplitude and b o =10, σ represents the direction information of pedestrians. o These are preset scaling parameters used to adjust the horizontal axis span;

[0053] S33. Calculate the corresponding motion intention value based on the pedestrian's motion behavior.

[0054]

[0055]

[0056] in, Human movement behavior, b m For the amplitude scaling factor and b m =5;

[0057] S34. Using the preset DOM model, calculate the pedestrian's interaction intent strength value Fip:

[0058]

[0059] Among them, c d With c o For the preset proportional parameters, when At that time, c d =1,c o =0; when At that time, c d =c o =0.5;

[0060] And based on the pedestrian's interaction intent strength value F i p To determine the intensity of pedestrians' interaction intentions.

[0061] Beneficial effects: This processing method first processes the pedestrian's straight-line distance, orientation information, and movement behavior separately to obtain their corresponding distance intention value. Orientation Intent Value and motion intention value It can reveal the interaction intent of pedestrians in a single dimension, including distance, orientation, and movement behavior. Then, by analyzing distance intent values... and orientation intention value Process it as static information and convert the motion intent value After processing into dynamic information, the intensity of the pedestrian's interaction intent can be expressed as F by using the area of ​​a rectangle composed of dynamic and static information. i p To provide a more accurate quantitative representation. For example, F i p The value range is [0, 100]. Then, by using preset interaction intent intensity ranges, such as [0, 55) for no interaction, [55, 75) for probing, and [75, 100] for active interaction, the intensity of the pedestrian's interaction intent is understood. This facilitates the robot's corresponding interaction decisions in S4. Attached Figure Description

[0062] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0063] Figure 1 This is a flowchart illustrating Example 1;

[0064] Figure 2 A schematic diagram illustrating the structure and working process of a lightweight motion behavior recognition model;

[0065] Figure 3 This is a schematic diagram of the three-dimensional coordinate axes in the embodiment;

[0066] Figure 4 A schematic diagram showing the body orientation of a pedestrian relative to a robot;

[0067] Figure 5 A schematic diagram representing four types of motion behavior;

[0068] Figure 6 A schematic diagram illustrating the relationship between the number of network layers and model performance;

[0069] Figure 7 This is a flowchart illustrating step S3 in Example 1;

[0070] Figure 8 This is a schematic diagram of the fuzzy representation of straight-line distance (distance value) in Example 1;

[0071] Figure 9 This is a schematic diagram of the fuzzy orientation information (orientation value) in Example 1;

[0072] Figure 10 This is a schematic diagram illustrating the blurring of interaction intent intensity in Example 1;

[0073] Figure 11 This is a diagram illustrating the intensity of pedestrian interaction intent recognition in real-time during the verification experiment in Example 1;

[0074] Figure 12 This is a schematic diagram of the DOM model in Example 2. Detailed Implementation

[0075] The following detailed explanation illustrates the specific implementation methods:

[0076] Existing methods for identifying the intensity of user interaction intent are all static. That is, they analyze the user's interaction intent relative to the robot using single-frame data (such as the user's facial expressions and posture in a static image). These methods have very limited applicability, only applicable after the user has actively interacted with the robot. Furthermore, in real-world applications, human interaction intent changes dynamically over time, making single-frame data identification methods unsuitable for dynamic monitoring. Existing technologies also neglect the impact of changes in user behavior on the intensity of interaction intent. This invention proposes a method for dynamically identifying the intensity of human-computer interaction intent by processing spatiotemporal sequence information.

[0077] Example 1:

[0078] like Figure 1 As shown, this embodiment discloses a real-time human-computer interaction intent strength recognition method, including the following steps:

[0079] S1. After collecting videos of pedestrians through the acquisition unit set on the robot, the video is processed by frame extraction, and the 3D skeleton extraction model is used to extract the skeleton information of each pedestrian in each frame in chronological order.

[0080] S2, based on the skeleton information of each pedestrian in each frame of the image, a preset lightweight motion behavior recognition model is used to identify the interaction reference information of each pedestrian; the interaction reference information includes the straight-line distance from the robot, orientation information, and motion behavior; the motion behavior includes approaching, moving away, passing by, and standing.

[0081] The structure and working process of the lightweight motion behavior recognition model are as follows: Figure 2 As shown, the lightweight motion behavior recognition model is an improved ConvLSTM network model with only one ConvLSTM layer and a higher-order feature extraction layer between the input layer and the ConvLSTM layer. The higher-order feature extraction layer consists of a higher-order feature extractor (HLFE) and a gate function (GF). There are three higher-order feature extractors (HLFE), which are used to extract the straight-line distance, orientation information and inter-frame change feature information after processing the skeleton information of each frame. The gate function (GF) is used to select the feature information to be fed into the ConvLSTM network layer.

[0082] In practice, the process of extracting high-order features for straight-line distances includes: extracting the three-dimensional coordinate information of the pedestrian skeleton in a preset camera coordinate system; such as... Figure 3As shown, the three-dimensional coordinate system has the position of the acquisition unit as the origin, the y-axis represents the position information in the depth direction of the acquisition unit, the x-axis represents the position information in the horizontal direction perpendicular to the y-axis, and the z-axis represents the position information in the vertical direction. The position of the pedestrian is represented by the coordinate information of the neck in the skeleton points; after considering the height of the acquisition unit and the head of all interactive objects as the same height, the straight-line distance of the pedestrian is calculated by the following formula:

[0083]

[0084] In the formula, The straight-line distance. Let x be the coordinates of the key points of a person's neck on the x-axis in a three-dimensional coordinate system. r These are the coordinates of the acquisition unit on the x-axis in a three-dimensional coordinate system. Let y be the coordinates of the key points of a person's neck in a three-dimensional coordinate system on the y-axis. r This represents the coordinates of the acquisition unit on the y-axis in a three-dimensional coordinate system.

[0085] The process of a high-order feature extractor for extracting orientation information includes: calculating the angle of the pedestrian's body orientation relative to the robot using the three-dimensional coordinates of the pedestrian's left and right shoulders. Then, calculate the coordinates of the intersection point of the ray pointing towards the pedestrian's body and the z-plane where the robot is located. Recalculate Horizontal distance between the robot's position and through Indicates the direction of pedestrians; In the formula, a o Indicates the scaling factor, The x-axis represents the coordinates of the intersection point of the pedestrian's body orientation and the vertical plane where the data acquisition unit is located. r This represents the coordinate value of the acquisition unit on the x-axis in a three-dimensional coordinate system. It can represent the relative intensity of the interaction intent indicated by different orientations. The larger the value, the weaker the interactive intent. It should be noted that when... At that time, set The maximum value is 100, when At that time, set The value is uniformly set to 104.

[0086] like Figure 4 As shown. In previous studies of human-computer interaction, the orientation information of the human body relative to the robot was usually directly used... This can be expressed as an angle. However, all other things being equal, using orientation angle to reflect the strength of an interaction's intent is not very objective. For example, if pedestrian 1 and pedestrian 2 are at the same distance from the robot and their body orientation angles are the same, then... and However, compared to pedestrian 1, pedestrian 2's region of interest (i.e., the focal point of the ray facing their body and the robot's z-plane) is more concentrated on the robot. For pedestrians 3 and 4, although... and However, their regions of interest are the same. Therefore, this invention introduces features... To better describe the orientation characteristics, That is, the intersection of the ray pointing towards the human body and the z-plane where the robot is located. Simultaneously, using... Horizontal distance between the robot's position (The applicant refers to this as orientation offset) to represent the relative intensity of interaction intent represented by different orientations, and provides its specific calculation method. The larger the value, the weaker the interaction intent. Figure 4 In the middle, it can be calculated that...

[0087] The working process of a high-order feature extractor for extracting inter-frame variations includes:

[0088] Calculate the change in distance of pedestrians along the x-axis and the change in distance of pedestrians along the y-axis.

[0089]

[0090] Where k represents the current frame number, The x-axis coordinates of key points on the pedestrian's neck. Let a be the y-axis coordinate of the key points on the pedestrian's neck. nx With a ny Let a be the scaling factor, and a nx =a ny =10;

[0091] like Then the motion behavior is stationary; if and Then the movement behavior is close; if and Then the movement behavior is to move away; if Then the movement behavior is a process.

[0092] It should be noted that, in defining motion behavior and constructing related models, existing publicly available data did not meet the requirements of this invention. Furthermore, since the motion behaviors to be identified in this invention are not very complex and only fall into four categories, the applicant created its own dataset. Details are as follows:

[0093] A lightweight multi-person 3D skeleton extraction model is used to extract the 3D skeleton information of the human body, thereby obtaining the 3D spatial coordinate information of 18 feature points of the human body.

[0094] The applicant selected 10 participants aged 18-25 for the experiment. In a spacious indoor environment, one participant entered at a time and performed four actions relative to the robot: approaching, passing by, standing, and moving away. Simultaneously, cameras on the robot captured video information and recorded key points about the participants' bodies, saving the data locally. To ensure the dataset fully reflects real-world conditions, participants were encouraged to perform each type of movement in different postures and orientations, such as... Figure 5 As shown. Each person recorded data for an average of 5 minutes. The following sections will detail the definitions of these four movement behaviors and some aspects of data collection.

[0095] (a) close to

[0096] Definition as follows Figure 5 The first behavioral illustration shows the behavior as "approach," which is characterized by: 1) the distance between the pedestrian and the robot decreases; and 2) the pedestrian is facing almost directly at the robot. To achieve this, participants are asked to walk towards the robot from different positions within its field of vision.

[0097] (b) After

[0098] The definition of the "passing" behavior is as follows: Figure 5 The second type of behavior is illustrated in the diagram. Its salient feature is that the pedestrian's body orientation is neither directly facing the robot nor with their back to it. To make the dataset of this type of behavior more comprehensive, participants were asked to "pass by" the robot at different angles each time, with the minimum angle controlled by the pedestrian themselves.

[0099] (c) Away from

[0100] "Avoidance" behavior is the complete opposite of "approach" behavior, such as Figure 5 The third type of behavior is illustrated in the diagram, and its significant features are: 1) the distance between the pedestrian and the robot increases; 2) the pedestrian walks with their back to the robot.

[0101] (d) Standing

[0102] "Standing" refers to a pedestrian standing still in one place, while the direction of their body can be arbitrary, such as... Figure 5 The fourth type of behavior is illustrated in the diagram. Similarly, participants are asked to stand in different positions to obtain a more comprehensive dataset.

[0103] Evaluation of lightweight motion behavior recognition models

[0104] Because the lightweight motion behavior recognition model uses a relatively basic network structure, performance comparison experiments were not conducted with some existing advanced networks. Instead, the focus was on differences in feature sets, specifically the impact of different datasets on model performance. To verify the effectiveness of adding higher-order features to the datasets, three datasets were divided according to their feature sets, as shown in Table 1. Specifically, D_54 contains all original features, D_58 is a combination of higher-order features and all original features, and D_7 is a combination of higher-order features and the human neck coordinate information from the original features. Furthermore, to verify that the ConvLSTM network has superior performance compared to the traditional LSTM network, comparative experiments were performed. The model's accuracy and runtime on the test set were used as evaluation metrics.

[0105] Table 1. Feature set allocation for the three datasets.

[0106]

[0107] Table 2 shows the test results. From the evaluation results in the table, we can see that:

[0108] ConvLSTM networks have a faster prediction time than LSTM networks.

[0109] When the dataset contains higher-order feature sets, the model performance is significantly improved, which shows that the addition of higher-order feature sets is effective.

[0110] The evaluation results of the model on dataset D_7 show that the model still performs well even with only 7 dimensions of feature set. This indicates that in recognition tasks, most of the skeleton information is redundant, and the human body can be treated as a whole while ignoring local movements.

[0111] Reducing the dimensionality of the dataset can reduce the complexity of the model, thereby improving the real-time performance of the recognition algorithm, which is very beneficial for building lightweight models.

[0112] Table 2 shows the model's performance evaluation on three datasets.

[0113]

[0114]

[0115] In addition, the applicant was also concerned about whether increasing network depth could effectively improve model performance for the original dataset. To this end, the applicant conducted experiments and obtained the following results: Figure 6As shown, increasing the number of LSTM and ConvLSTM layers does not effectively improve the performance of the recognition model, and the prediction time for both types of models increases with the number of layers. This indicates that simply increasing the number of network layers does not effectively improve model performance. Therefore, adding higher-order feature layers has significant advantages.

[0116] S3 analyzes and processes three types of interactive reference information of pedestrians: straight-line distance, orientation information, and movement behavior, to obtain the intensity of the pedestrian's real-time interactive intention relative to the robot.

[0117] In this embodiment, as Figure 7 As shown, the Mamdani fuzzy inference system is used to fuse three types of interactive reference information of pedestrians: straight-line distance, orientation information, and motion behavior, and infer the intensity of the pedestrian's real-time interactive intention relative to the robot.

[0118] Specifically, the working process of the Mamdani fuzzy inference system includes: according to the preset fuzzification rules, the input straight-line distance, orientation information and motion behavior are fuzzified; the pedestrian's interaction intention is inferred and analyzed according to the preset fuzzy inference rules to obtain the fuzzy value of the pedestrian's interaction intention; and then the fuzzy value of the interaction intention is defuzzified by the defuzzification algorithm to obtain the precise value of the pedestrian's interaction intention.

[0119] Regarding blurring

[0120] Fuzzy logic maps precise observations in the input space to fuzzy sets in the input universe of discourse. Specifically, it converts precise observations of three types of input information—linear distance, orientation information, and motion behavior—into fuzzy linguistic values. Each fuzzy linguistic value corresponds to a fuzzy set, which is described by a membership function. The shape of the membership function significantly impacts the performance of the fuzzy inference system. A narrower membership function graph indicates a more sensitive system to input, while a wider graph indicates a less sensitive system.

[0121] Blurring of straight-line distance (distance value)

[0122] The straight-line distance between humans and robots The distance is blurred into 6 levels: D1: very close, D2: close, D3: relatively close, D4: relatively far, D5: far, D6: very far, with values ​​ranging from [0,5], and the unit is meters. In real life, the human-computer distance distribution should be relatively uniform; therefore, the membership function should be constructed as an equidistant distribution. Simultaneously, considering the influence of distance on interaction intent, a right-angled triangular function is used to blur the straight-line distance, constructing membership functions μ(D1), μ(D2), μ(D3), μ(D4), μ(D5), and μ(D6), as follows: Figure 8 As shown.

[0123] Blurring of orientation information (orientation value)

[0124] Orientation Information The orientation information is blurred into three levels: O1: small, O2: medium, and O3: large, with values ​​ranging from [0, 104]. These three levels correspond to the orientation information of three types of motion behaviors: approaching, passing by, and moving away. Based on the content of 3.3.1 and the distribution pattern of orientation information in the applicant's collected dataset, triangular and trapezoidal membership functions are used to blur the orientation information, and membership functions μ(O1), μ(O2), and μ(O3) are constructed, as follows: Figure 9 As shown.

[0125] motor behavior

[0126] The movement behaviors are represented by four discrete, clear values, requiring no additional membership functions for fuzzification and can be directly used for subsequent inference rules. Among these four movement behaviors, "standing" does not exhibit a clear interaction intention like the other behaviors. For example, "approaching" shows a strong interaction intention, "moving away" shows a very weak interaction intention, and "passing by" falls somewhere in between. Therefore, the "standing" behavior cannot be directly involved in the formulation of inference rules; when a pedestrian is "standing," other factors are needed to determine the interaction intention. "Approaching," "passing by," and "moving away" are denoted into three levels: M1: strong, M2: medium, and M3: weak.

[0127] Blurring of Interaction Intent Strength

[0128] The interaction intent strength is blurred into 11 levels: F0-F10. The value range is normalized to [0,1], where F0 is the weakest and F11 is the strongest. Triangular membership functions are selected to blur the interaction intent strength, resulting in uniformly distributed membership functions μ(F0)-μ(F10), as follows: Figure 10 As shown.

[0129] In specific implementation, the fuzzy inference rules include dynamic inference rules and static inference rules; static rules are used to determine the intensity level of interaction intent when a pedestrian is standing based on distance and orientation information; dynamic rules are used to determine the intensity level of interaction intent when a pedestrian is approaching, passing by, or moving away based on straight-line distance and action behavior.

[0130] Fuzzy reasoning rules are an effective use of subjective experience. Subjective experience tells us:

[0131] 1) The closer the human-computer interaction distance, the stronger the interaction intent;

[0132] 2) The smaller the orientation value, the stronger the interaction intent;

[0133] 3) The intensity of interaction intent for the three behaviors of approaching, passing by, and moving away is ranked as follows: approaching > passing by > moving away;

[0134] 4) When the distances are similar, approaching the robot indicates a stronger interaction intent than standing facing the robot.

[0135] 5) When the distance difference is not large, maintaining the facing angle when passing by and standing still is slightly stronger in terms of interaction intent than continuously passing by the robot, because the pedestrian standing is more likely to turn towards the robot.

[0136] 6) When a pedestrian has their back to the robot, they have almost no intention to interact with the robot. The intention to interact is weaker when the pedestrian moves away from the robot than when they stand with their back to the robot, because the distance does not increase when the pedestrian stands.

[0137] Based on the above experience, the applicant developed two sets of inference rule bases: a static one and a dynamic one. The static rule base is used to determine the intensity of a pedestrian's interaction intention when they are standing; in this case, distance and orientation are used to formulate inference rules. The dynamic rule base is used to determine the intensity of a pedestrian's interaction intention when they are approaching, passing by, or moving away; in this case, distance and these three movement states are used to formulate inference rules, while orientation is not needed because it is included in the movement state. Furthermore, the static and dynamic rule bases are linked through fuzzy interaction intention intensity levels.

[0138] As can be seen from the above analysis, both rule bases correspond to a two-input single-output (TISO) system, which can be represented as:

[0139] R:IF x1 is A1 and x2 is A2,THEN y is B

[0140] Where R represents the rule base, x1 and x2 are input values, y is the output value, and A1, A2 and B are the fuzzy values ​​corresponding to x1, x2 and y, respectively.

[0141] Based on the above empirical rules, two sets of reasoning rules were formulated as shown in Tables 3 and 4.

[0142] Table 3 Dynamic Reasoning Rules

[0143]

[0144] Table 4 Static Reasoning Rules

[0145]

[0146] The Mamdani fuzzy inference algorithm was proposed by E.H. Mamdani in 1974. For the aforementioned multi-rule TISO system, let the fuzzy rules describing the input / output relationship be:

[0147]

[0148] Where j = 1, 2, ..., m, m represents the total number of rules, A is the speech variable value of x, and B is the linguistic variable value of y. Then, for a given input x, the applicability of each rule generated from fuzzy relations is:

[0149]

[0150] The total fuzzy set of output quantities can be obtained by using the max-min inference synthesis rule:

[0151]

[0152] Let Bi be the membership function of the fuzzy set Bi, which is the output of a fuzzy rule.

[0153] The interaction intent strength obtained through fuzzy inference is a fuzzy value. To achieve a more refined recognition effect of interaction intent strength, defuzzification is required. Therefore, this embodiment uses the area centroid method for defuzzification. The centroid method takes the centroid of the area enclosed by the membership function curve and the horizontal axis as the final output value of the fuzzy inference. To reduce computational complexity, a simplified formula of this method is used for calculation, as shown in the following formula:

[0154]

[0155] In the formula, y is the precise value of the pedestrian's interaction intent after deblurring. for The point where the maximum value is obtained. The output of each fuzzy rule is a fuzzy set B. i Membership function.

[0156] When defuzzing the fuzzy value of the interaction intent, the range of the precise value of the pedestrian's interaction intent is normalized to [0,1]. Through normalization, the precise value of the pedestrian's interaction intent becomes more concise and clear, facilitating subsequent analysis of specific interaction decisions.

[0157] S4. Based on the obtained interaction intent strength, control the robot to execute the corresponding interaction decision; the interaction decision includes no interaction, tentative interaction, and active interaction. In specific implementation, if the precise value of the pedestrian's interaction intent is within [0-0.4], the interaction decision is no interaction; if the precise value of the pedestrian's interaction intent is within (0.4-0.7], the interaction decision is tentative interaction; if the precise value of the pedestrian's interaction intent is within (0.7-1.0], the interaction decision is active interaction.

[0158] If the precise value of the pedestrian's interaction intention falls within [0-0.4], it indicates a weak interaction intention, and therefore the robot will not engage in any proactive interaction. If it falls within (0.4-0.7), it indicates a possibility of interaction between the pedestrian and the robot, and the robot will perform tentative interaction behaviors, such as greeting the pedestrian or asking if they need help. If it falls within (0.7-1.0), it indicates a clear willingness to interact with the robot, and the robot will engage in proactive interaction behaviors. This setting allows the robot to select appropriate interaction targets and make reasonable interaction decisions.

[0159] Based on these three intervals, the applicant designed Algorithm 1 to enable the robot to select appropriate interaction objects and make reasonable interaction decisions.

[0160]

[0161]

[0162] In this algorithm, `Median_filter(data,s)` is the median filter function, where `data` is the data to be processed and `s` is the step size. `Max()` is the function to retrieve the maximum value. The `Index()` function is used to obtain the index of the current value, i.e., the pedestrian's ID number. Algorithm 1 calculates the average interaction intent intensity of all pedestrians within the robot's field of view over 10 frames, uses median filtering to remove outliers, and then performs interval judgment on the maximum value of the current interaction intent intensity to determine whether the robot should interact with the pedestrian. The judgment result is determined by O. r Output is then provided. When the robot needs to make an interactive response, the pedestrian's ID number i is used. m It will be output.

[0163] This invention provides a method for identifying the intensity of pedestrian interaction intentions. Specifically, by acquiring dynamic video of pedestrians and extracting frames, interaction reference information, including straight-line distance to the robot, orientation information, and motion behavior, is identified from each frame in chronological order. Then, the three types of interaction reference information are fused using a Mamdani fuzzy inference system to infer the real-time interaction intention intensity of the pedestrian relative to the robot. Through comprehensive analysis of distance, orientation information, and motion behavior, the robot can accurately understand the interaction intention intensity of each pedestrian within its field of vision. Based on the interaction intention intensity, the robot can determine whether to initiate interaction and, if so, what interaction decision to make (e.g., tentative interaction or direct proactive interaction). This approach takes into account the dynamic characteristics of pedestrians over a period of time. Compared with some existing methods that rely on static single-frame image recognition (such as recognizing pedestrians' expressions and postures to understand their interaction intentions), this method offers higher reliability and greater precision (able to provide specific interaction decisions). Furthermore, this invention utilizes a lightweight neural network model to obtain interaction reference information such as the pedestrian's straight-line distance, orientation, and movement behavior. The algorithm runs quickly, making it usable even for mobile robots with limited computing power. In addition, this invention uses fuzzy inference to calculate the intensity of different pedestrians' interaction intentions in real time, ensuring recognition efficiency while maximizing the naturalness of robot-human interaction.

[0164] experiment

[0165] To verify the effectiveness of this invention, specifically whether the selection of the strongest interactive intent in the current environment is correct, the applicant placed a humanoid robot in the lobby of a university office building. A camera on the robot's head was equipped with real-time video capture capabilities, and the data collected by the camera was transmitted to a PC. When a pedestrian appeared in the camera's view, the algorithm of this invention extracted various feature data of the pedestrian and identified the strength of their interactive intent.

[0166] Camera data was collected from 11:00 am to 1:00 pm on a weekday, a time of high pedestrian traffic. After filtering, 85 minutes of valid data were obtained (the number of people in the footage was greater than 1, the pedestrians moved at a slow speed, and the pedestrians were not obscured by large areas). Figure 11The graph shows the pedestrian interaction intent intensity recognition results for a certain time period. F represents the interaction intent intensity value of each pedestrian, ranging from [0,1]. The larger the F value, the stronger the pedestrian's interaction intent relative to the robot. It is evident that the method proposed in this invention can effectively distinguish the interaction intent intensity of different pedestrians, and the results are consistent with reality. Furthermore, the strongest interaction intent among all collected valid data is manually labeled, on average once every 10 frames. By comparing the results of the interaction intent intensity recognition algorithm, the final accuracy rate of the algorithm is 92.63%.

[0167] Example 2

[0168] Unlike Embodiment 1, in S3 of this embodiment, a preset DOM model is used to analyze and process three types of interaction reference information: the pedestrian's straight-line distance, orientation information, and movement behavior, to obtain the real-time interaction intent intensity of the pedestrian relative to the robot; such as Figure 12 As shown, the DOM model includes mutually perpendicular S-axis and D-axis; the S-axis represents static information, including pedestrian distance intention values. and orientation intention value The D-axis represents dynamic information, including the pedestrian's movement intention value. The area of ​​the rectangle formed by the pedestrian's static and dynamic information is used as the intensity value of the pedestrian's interactive intent.

[0169] S3 includes:

[0170] S31. Calculate the corresponding distance target value based on the straight-line distance of the pedestrian.

[0171]

[0172] Among them, b d It is the maximum amplitude, and b d =10; σ is the straight-line distance for pedestrians. d The preset scaling parameter is used to adjust the horizontal axis span. In this embodiment, σ d =2, so that the distribution can better conform to the theory of interpersonal distance.

[0173] S32. Calculate the corresponding orientation intention value based on the pedestrian's orientation information.

[0174]

[0175] Among them, b o For the maximum amplitude and b o =10, σ represents the direction information of pedestrians.o The preset scaling parameter is used to adjust the horizontal axis span. In this embodiment, σ o =1, so that the distribution is more in line with reality.

[0176] S33. Calculate the corresponding motion intention value based on the pedestrian's motion behavior.

[0177]

[0178]

[0179] in, Human movement behavior, b m For the amplitude scaling factor and b m =5;

[0180] S34. Using a preset DOM model, calculate the intensity value of the pedestrian's interaction intent.

[0181]

[0182] Among them, c d With c o As a proportional parameter, it must satisfy c d +c o =1; when At that time, c d =1,c o =0, at this point the orientation information is redundant because it can be included by the motion behavior; that is, the interaction intent strength can be determined solely by motion behavior information and distance information; when At that time, c d =c o =0.5, when the pedestrian is not standing, the contribution level of distance information and orientation information is the same;

[0183] And based on the pedestrian's interaction intent strength value F i p To determine the intensity of pedestrians' interaction intentions.

[0184] This processing method first processes the pedestrian's straight-line distance, orientation information, and movement behavior separately to obtain their corresponding distance intention value. Orientation Intent Value and motion intention value It can reveal the interaction intent of pedestrians in a single dimension, including distance, orientation, and movement behavior. Then, by analyzing distance intent values... and orientation intention value Process it as static information and convert the motion intent value After processing into dynamic information, the intensity of the pedestrian's interaction intent can be expressed as F by using the area of ​​a rectangle composed of dynamic and static information. i p A more accurate quantification representation is achieved. In this embodiment, F i The value of p ranges from [0, 100]. Then, by using preset interaction intent intensity ranges, such as [0, 55) for no interaction, [55, 75) for probing, and [75, 100] for active interaction, the intensity of the pedestrian's interaction intent is determined. This facilitates the robot's corresponding interaction decisions in S4.

[0185] In other embodiments, when the robot collects more than one number of pedestrians, it can calculate the interaction intent strength value of all pedestrians, find the pedestrian with the highest interaction intent value, and then determine the interaction intent strength of that pedestrian and take the corresponding interaction decision.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A method for real-time human-computer interaction intent strength recognition, characterized in that, Includes the following steps: S1. After collecting videos of pedestrians through the acquisition unit set on the robot, the video is processed by frame extraction, and the 3D skeleton extraction model is used to extract the skeleton information of each pedestrian in each frame in chronological order. S2, based on the skeleton information of each pedestrian in each frame of the image, a preset lightweight motion behavior recognition model is used to identify the interaction reference information of each pedestrian; the interaction reference information includes the straight-line distance from the robot, orientation information, and motion behavior; the motion behavior includes approaching, moving away, passing by, and standing. S3 analyzes and processes three types of interactive reference information of pedestrians: straight-line distance, orientation information, and movement behavior, to obtain the real-time interaction intent intensity of pedestrians relative to robots. S4. Based on the strength of the obtained interaction intent, control the robot to execute the corresponding interaction decision; the interaction decision includes no interaction, tentative interaction, and active interaction; In S2, the lightweight motion behavior recognition model is an improved ConvLSTM network model with only one ConvLSTM layer and a higher-order feature extraction layer between the input layer and the ConvLSTM layer. The higher-order feature extraction layer consists of higher-order feature extractors and gate functions. There are three higher-order feature extractors, which are used to extract the straight-line distance, orientation information, and inter-frame change feature information after processing the skeleton information of each frame. The gate function is used to select the feature information to be fed into the ConvLSTM network layer. In S3, a pre-defined DOM model is used to analyze and process three types of interactive reference information: the pedestrian's straight-line distance, orientation information, and movement behavior, to obtain the real-time interaction intent intensity of the pedestrian relative to the robot. The DOM model includes mutually perpendicular S-axis and D-axis; the S-axis represents static information, including the pedestrian's distance intent value. and orientation intention value The D-axis represents dynamic information, which includes the pedestrian's movement intention value. ; Specifically, the Mamdani fuzzy inference system is used to fuse three types of interactive reference information for pedestrians: straight-line distance, orientation information, and movement behavior. The working process of the Mamdani fuzzy inference system includes: fuzzifying the input straight-line distance, orientation information, and movement behavior according to preset fuzzification rules; inferring and analyzing the pedestrian's interaction intention according to preset fuzzy inference rules to obtain the fuzzy value of the pedestrian's interaction intention; and then defuzzifying the fuzzy value of the interaction intention using a defuzzification algorithm to obtain the precise value of the pedestrian's interaction intention. The fuzzy inference rules include dynamic inference rules and static inference rules; static rules are used to determine the intensity level of interaction intent when a pedestrian is standing, based on distance and orientation information; dynamic rules are used to determine the intensity level of interaction intent when a pedestrian is approaching, passing by, or moving away, based on straight-line distance and action behavior. S3 includes: S31. Calculate the corresponding distance target value based on the straight-line distance of the pedestrian. : ； in, It is the maximum amplitude, and =10; The straight-line distance for pedestrians; These are preset scaling parameters used to adjust the horizontal axis span; S32. Calculate the corresponding orientation intention value based on the pedestrian's orientation information. : ； in, The maximum amplitude and =10, Indicates the direction information of pedestrians, These are preset scaling parameters used to adjust the horizontal axis span; S33. Calculate the corresponding motion intention value based on the pedestrian's motion behavior. : ； ； in, Human movement behavior, For the amplitude scaling factor and =5; S34. Using a preset DOM model, calculate the intensity value of the pedestrian's interaction intent. : ； in, and For the preset proportional parameters, when When ≠1, =1, =0; when When =1, = =0.5; And based on the intensity value of the pedestrian's interaction intent To determine the intensity of pedestrians' interaction intentions.

2. The real-time human-computer interaction intent strength recognition method as described in claim 1, characterized in that: The working process of the high-order feature extractor for extracting straight-line distance includes: extracting the three-dimensional coordinate information of the pedestrian skeleton in a preset camera coordinate system; the three-dimensional coordinate system takes the position of the acquisition unit as the origin, the y-axis represents the position information in the depth direction of the acquisition unit, the x-axis represents the position information in the horizontal direction perpendicular to the y-axis, and the z-axis represents the position information in the vertical direction; The pedestrian's position is represented by the coordinates of the neck in the skeleton points; after treating the height of the acquisition unit and the head of all interactive objects as the same height, the straight-line distance of the pedestrian is calculated using the following formula: ； In the formula, The straight-line distance. These are the coordinates of the key points of a pedestrian's neck on the x-axis in a three-dimensional coordinate system. These are the coordinates of the acquisition unit on the x-axis in a three-dimensional coordinate system. These are the y-coordinates of the key points of a pedestrian's neck in a three-dimensional coordinate system. This represents the coordinates of the acquisition unit on the y-axis in a three-dimensional coordinate system.

3. The real-time human-computer interaction intent strength recognition method as described in claim 2, characterized in that: The process of a high-order feature extractor for extracting orientation information includes: calculating the angle of the pedestrian's body orientation relative to the robot using the three-dimensional coordinates of the pedestrian's left and right shoulders. Then, calculate the coordinates of the intersection point of the ray pointing towards the pedestrian's body and the z-plane where the robot is located. , then calculate Horizontal distance between the robot's position and through Indicates the direction of pedestrians; In the formula, Indicates the scaling factor, This represents the x-axis value of the intersection point between the pedestrian's body orientation and the vertical plane where the data acquisition unit is located. This represents the coordinate value of the acquisition unit on the x-axis in a three-dimensional coordinate system.

4. The real-time human-computer interaction intent strength recognition method as described in claim 3, characterized in that: when When <90°, set The maximum value is 100, when When ≥90°, set The value is uniformly set to 104.

5. The real-time human-computer interaction intent strength recognition method as described in claim 4, characterized in that: The working process of a high-order feature extractor for extracting inter-frame variations includes: Calculate the change in distance of pedestrians along the x-axis and the change in distance of pedestrians in the y-axis direction. : ； ； Where k represents the current frame number, The x-axis coordinates of key points on the pedestrian's neck. Let y be the y-coordinate of the key points on the pedestrian's neck. and This is the scaling factor, and = =10; like = =0, then the motion behavior is static; if =0 and If <0, the movement behavior is close; if =0 and If >0, the movement behavior is to move away; if If the value is greater than 0, then the motion behavior is considered as having passed through.

6. The real-time human-computer interaction intent strength recognition method as described in claim 5, characterized in that: Defuzzification of fuzzy values ​​of interactive intent is achieved using the area centroid method: ； In the formula, y is the precise value of the pedestrian's interaction intent after deblurring. for The point where the maximum value is obtained. The membership function represents the output quantity of a fuzzy rule, fuzzy set Bi.

7. The real-time human-computer interaction intent strength recognition method as described in claim 6, characterized in that: In S3, when defuzzifying the fuzzy value of the interaction intention, the range of the obtained precise value of the pedestrian's interaction intention is normalized to [0,1]. In S4, if the precise value of the pedestrian's interaction intention is in the range of [0-0.4], the interaction decision is to not interact; if the precise value of the pedestrian's interaction intention is in the range of (0.4-0.7], the interaction decision is to attempt an interaction; if the precise value of the pedestrian's interaction intention is in the range of (0.7-1.0], the interaction decision is to initiate an interaction.

Images