A multimodal accompanying robot intention understanding method and system
By acquiring and analyzing the hand skeleton line image sets of the elderly, identifying high-value and low-value points, combining local feature information, using neural networks and speech recognition for intention understanding, the accuracy and efficiency of accompanying robots in the elderly are solved. The low-level problem improves the accuracy and efficiency of human-computer interaction.
Patent Information
- Application Number
- CN202410917009.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-10
AI Technical Summary
Existing escort robots are not accurate and efficient in the elderly’s intention understanding, especially when identifying the elderly’s physical movements and behaviors, resulting in poor human-computer interaction.
By obtaining the image set of hand skeleton lines of the elderly, high value points and low value points are determined based on the timing change information, target feature points are determined based on local neighborhood feature information, and intention understanding is used using neural networks and speech recognition probability sets.
It improves the accuracy and efficiency of intention understanding, optimizes feature point selection, and enhances the fit and accuracy of human-computer interaction.
Smart Images

Figure CN118470802B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image data processing, and in particular to a multimodal accompanying robot intention understanding method and system in the technical field of image data processing. Background Art
[0002] Through actual research in nursing homes and studies of companion robots, we've found that robotic companionship is increasingly accepted by the elderly. However, robotic companion systems are not very accurate and efficient in understanding the elderly's intentions. In particular, when robots identify the elderly's movements, they often misinterpret the meaning of body movements, resulting in poor human-machine interaction and a less-than-satisfactory user experience. Summary of the Invention
[0003] In order to solve the technical problem of low accuracy and efficiency of intention understanding in accompanying systems, the purpose of the present invention is to provide a multimodal accompanying robot intention understanding method and system. The technical solutions adopted are as follows:
[0004] In a first aspect, an embodiment of the present invention provides a method for understanding intentions of a multimodal accompanying robot, the method comprising:
[0005] Obtaining a skeleton line image set of the hand of the person to be accompanied;
[0006] determining high-value points and low-value points among the skeleton pixel points of the hand based on temporal variation information corresponding to the skeleton line image set;
[0007] Determining target feature points among the high-value points and the low-value points based on a local neighborhood of the low-value points in the corresponding skeleton line image;
[0008] Based on the target feature points, the intention of the person to be accompanied is understood to obtain the target intention of the person to be accompanied.
[0009] In conjunction with the first aspect above, in some possible implementations, obtaining a skeleton line image set of the hand of the person to be accompanied includes:
[0010] Obtaining a hand movement video of the person to be accompanied;
[0011] Performing binary segmentation on the action video to obtain a binary image;
[0012] Perform skeleton extraction on the binary image to obtain the skeleton line image set.
[0013] In the above method, by performing binary segmentation on the action video of the hand of the person being cared for, and extracting the skeleton from the binary image, an image with skeleton lines can be obtained quickly and accurately.
[0014] In combination with the first aspect above, in some possible implementations, performing skeleton extraction on the binary image to obtain the skeleton line image set includes:
[0015] Taking any point in the binary image as the center of a circle, determining a target disk image in the binary image;
[0016] In the target disk image, determining skeleton pixel points tangent to the edge of the hand;
[0017] Based on the skeleton pixel points, the skeleton line image set is determined.
[0018] In the above manner, by determining the largest target disk image in the binary image, the points on the skeleton can be analyzed in the target disk image, and the skeleton line image set can be accurately obtained.
[0019] In conjunction with the first aspect above, in some possible implementations, determining high-value points and low-value points in the skeleton pixel points of the hand based on the temporal change information corresponding to the skeleton line image set includes:
[0020] Assigning a value coefficient to the skeleton pixel points based on the temporal variation information corresponding to the skeleton line image set;
[0021] Based on the value coefficient, the high value point and the low value point are determined.
[0022] In the above method, high-value points and low-value points can be quickly divided in the skeleton line image through the value coefficient of the skeleton pixel point.
[0023] In conjunction with the first aspect above, in some possible implementations, assigning value coefficients to pixels on the skeleton based on temporal change information corresponding to the skeleton line image set includes:
[0024] For any skeleton line image in the skeleton line image set, determining a first distance between a candidate pixel point of a skeleton line in a previous frame of skeleton line image and the skeleton line image based on temporal change information corresponding to the skeleton line image; wherein the candidate pixel point includes an endpoint or an intersection of a skeleton line;
[0025] Determining an initial value coefficient of a candidate pixel point of the skeleton line based on the first distance, the number of frames of the skeleton line image set, and the frequency of the candidate image; wherein the candidate image is a skeleton line image in which the candidate pixel point is displaced;
[0026] determining a value weight of a candidate pixel point in any one of the skeleton line images based on a second distance between endpoints of the skeleton lines in any one of the skeleton line images;
[0027] Based on the initial value coefficient and the value weight, the value coefficient of the candidate pixel point in any skeleton line image is obtained.
[0028] In the above method, by combining the value weights of the endpoints or intersections of the skeleton lines and the initial value coefficients, the value coefficients of the endpoints or intersections of the skeleton lines in each skeleton line image can be obtained more accurately, thereby facilitating the determination of whether the endpoints or intersections of the skeleton lines are high-value points.
[0029] In conjunction with the first aspect above, in some possible implementations, determining the initial value coefficients of the candidate pixels of the skeleton line based on the first distance, the number of frames of the skeleton line image set, and the frequency of the candidate images includes:
[0030] Determining the frequency of the image frames in which the candidate pixel points of a single skeleton line are displaced based on the number of frames of the skeleton line image set and the frequency of the candidate images;
[0031] The first distance and the frequency are fused to obtain the initial value coefficients of the candidate pixel points of the skeleton line.
[0032] In the above method, by combining the first distance and the frequency of the number of image frames at which the candidate pixel points of a single skeleton line are displaced, the initial value coefficient of the candidate pixel point can be accurately obtained, thereby facilitating rapid determination of whether the candidate pixel point is a high-value point.
[0033] In combination with the first aspect above, in some possible implementations, determining the target feature point among the high-value points and the low-value points based on the local neighborhood of the low-value point in the skeleton line image to which it belongs includes:
[0034] Determining local feature change information in the local neighborhood;
[0035] Based on the local feature change information, the value coefficient of the low-value point is modified to obtain a modified value point with an updated value coefficient;
[0036] The target feature point is determined among the revised value point and the high value point based on the updated value coefficient of the revised value point and the value coefficient of the high value point.
[0037] In the above method, the value coefficient of the low-value point is corrected by the local feature change information of the local neighborhood of the low-value point, so that the high-value point can be divided more accurately, and the key information contained in the target feature point is richer.
[0038] In conjunction with the foregoing first aspect, in some possible implementations, determining the local feature change information in the local neighborhood includes:
[0039] For any low-value point on any skeleton line, determine the frequency of occurrence of a tuple consisting of the grayscale value of the low-value point and the grayscale value of the local neighborhood;
[0040] Based on the occurrence frequency and a neighborhood scale corresponding to the local neighborhood, local feature change information in the local neighborhood is determined.
[0041] In the above method, by calculating the frequency of occurrence of the binary group consisting of the low-value point and the grayscale value in the local neighborhood and the neighborhood scale, the local feature change information in the local neighborhood can be obtained, thereby accurately reflecting the temporal displacement, posture change and grayscale change of the hand.
[0042] In conjunction with the first aspect above, in some possible implementations, understanding the intention of the person to be accompanied based on the target feature points to obtain the target intention of the person to be accompanied includes:
[0043] Using a trained neural network to perform gesture recognition on a skeleton line image set carrying the target feature points to obtain a gesture probability set;
[0044] Fusing the preset speech recognition probability set and the gesture probability set to obtain a fused probability set;
[0045] Based on the fused probability set, the intention of the person to be accompanied is understood to obtain the target intention of the person to be accompanied.
[0046] In the above method, by fusing the preset voice recognition probability set and the gesture probability set, the intention of the fused probability set is understood, so that the target intention is more accurate and more in line with the needs of the person to be accompanied.
[0047] In a second aspect, a multimodal accompanying robot intention understanding system is provided, the system comprising:
[0048] An acquisition module, used for acquiring a skeleton line image set of the hand of the person to be accompanied;
[0049] a first determining module, configured to determine high-value points and low-value points among the skeleton pixel points of the hand based on temporal variation information corresponding to the skeleton line image set;
[0050] A second determining module is configured to determine a target feature point among the high-value points and the low-value points based on a local neighborhood of the low-value point in the corresponding skeleton line image;
[0051] The intention understanding module is used to understand the intention of the object to be accompanied based on the target feature points to obtain the target intention of the object to be accompanied.
[0052] In a third aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect.
[0053] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.
[0054] The present invention has the following beneficial effects: by acquiring a skeleton line image set of the hand of the person to be accompanied, high-value points and low-value points are determined among the skeleton pixels of the hand according to the time-series change information corresponding to the skeleton line image set; and based on the local neighborhood of the low-value point in the skeleton line image to which it belongs, target feature points are determined among the high-value points and the low-value points; according to the target feature points, the intention of the person to be accompanied is understood, and the target intention of the person to be accompanied is obtained. In this way, by determining the target feature points through the high-value points and the low-value points, the selection of feature points can be optimized, so that the feature points for intention understanding are as many as possible containing high-value pixels, thereby improving the efficiency and accuracy of intention understanding. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0056] Figure 1 This is a schematic diagram of a process for implementing a multimodal accompanying robot intention understanding method provided by an embodiment of the present invention;
[0057] Figure 2 1 is another implementation flowchart of a multimodal accompanying robot intention understanding method provided by an embodiment of the present invention;
[0058] Figure 3 This is another implementation flowchart of a multimodal accompanying robot intention understanding method provided by an embodiment of the present invention;
[0059] Figure 4Schematic diagram of a template structure of a multimodal accompanying robot intention understanding method provided by an embodiment of the present invention;
[0060] Figure 5 1 is another schematic diagram of a template structure of a multimodal accompanying robot intention understanding method provided by an embodiment of the present invention;
[0061] Figure 6 1 is another implementation flowchart of a multimodal accompanying robot intention understanding method provided by an embodiment of the present invention;
[0062] Figure 7 1 is a schematic diagram of the structure of a multimodal accompanying robot intention understanding system provided by an embodiment of the present invention;
[0063] Figure 8 This is a structural diagram of a multimodal accompanying robot intention understanding device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0064] To further illustrate the technical means and effectiveness of the present invention in achieving its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail a multimodal care robot intention understanding method according to the present invention, including its specific implementation, structure, features, and effectiveness. In the following description, references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.
[0065] In the description of the embodiments of the present invention, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present invention, "multiple" refers to two or more than two.
[0066] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.
[0067] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
[0068] The specific scheme of the multimodal accompanying robot intention understanding method provided by the present invention is described in detail below with reference to the accompanying drawings.
[0069] See also Figure 1 , which shows a flowchart of a multimodal accompanying robot intention understanding method provided by one embodiment of the present invention, the method comprising:
[0070] 101. Obtain a skeleton line image set of the hand of the person to be accompanied.
[0071] Here, the person to be accompanied can be any user, for example, an elderly person who needs to be accompanied.
[0072] In some possible implementations, the above step 101 can be performed by Figure 2 The steps shown achieve:
[0073] 201. Obtain a hand movement video of the person to be accompanied.
[0074] Here, the detection camera of the accompanying robot is used to shoot and record a hand movement in real time to obtain the action video.
[0075] 202 , performing binary segmentation on the action video to obtain a binary image.
[0076] Here, the action video is binarized frame by frame using the Otsu thresholding method to segment the image of the hand, thereby obtaining a clear binary image. This binary image is the hand image, which facilitates the extraction of the hand skeletal features from the hand image. The image processing mentioned in this embodiment is based on the first frame of the video.
[0077] 203 : Perform skeleton extraction on the binary image to obtain the skeleton line image set.
[0078] For the human body, skeletal motion can well reflect the specific characteristics and meaning of movement. Therefore, morphological skeleton extraction is performed on the binary image to obtain a corresponding skeleton line image set. By performing binary segmentation on the hand motion video of the accompanying person and extracting the skeleton from the binary image, an image representing the skeleton line can be quickly and accurately obtained.
[0079] In some possible implementations, the skeleton line image set is extracted from the binary image by using the maximum inscribed circle method, that is, the above step 203 can be implemented by the following process:
[0080] First, a target circular image is determined in the binary image, with any point in the binary image as the center. Second, skeleton pixel points tangent to the hand edge are determined in the target circular image. Finally, the skeleton line image set is determined based on the skeleton pixel points.
[0081] Here, the maximum inscribed circle method is used for skeleton extraction. At any given point inside the binary image, if there is a maximum disk with this point as the center, its entirety is inside the image, and at least two points are tangent to the edge of the hand (i.e., skeleton pixels tangent to the edge of the hand), then this skeleton pixel point is a point on the skeleton, and thus a skeleton line is formed through the centers of all the maximum disks to obtain a skeleton line image. In some possible implementations, a video tracking algorithm is used to perform similarity matching on the endpoints and intersections in the images of consecutive frames. With the image of the initial frame as the target, the corresponding endpoints and intersections in all images are marked, thereby obtaining a skeleton line image set of marked skeleton line endpoints and intersections. In this way, by determining the largest target disk image in the binary image, it is possible to analyze the points on the skeleton in the target disk image, and thus accurately obtain the skeleton line image set.
[0082] 102. Determine high-value points and low-value points among the skeleton pixel points of the hand based on the temporal variation information corresponding to the skeleton line image set.
[0083] Here, the temporal variation information corresponding to the skeleton line image set includes temporal variation amplitude and temporal position variation. By analyzing the temporal variation information corresponding to the skeleton line image set, high-value points containing high-value information and low-value points containing low-value information are analyzed from the skeleton pixels.
[0084] In some possible implementations, the value coefficients of the skeleton similarity points are determined by analyzing the time series change information, thereby dividing the high-value points and the low-value points. That is, the above step 102 can be performed by Figure 3 The steps shown achieve:
[0085] 301. Assign value coefficients to the skeleton pixels based on temporal variation information corresponding to the skeleton line image set.
[0086] In some possible implementations, step 301 may be implemented through the following process:
[0087] First, for any skeleton line image in the skeleton line image set, based on the temporal variation information corresponding to the any skeleton line image, a first distance between the any skeleton line image and a candidate pixel point of the skeleton line in the previous frame of the skeleton line image is determined.
[0088] Among them, the candidate pixel points include: endpoints or intersections of skeleton lines. In the intention understanding method and system of the multimodal accompanying robot, whether a neural network or a method based on intention probability is used, it is necessary to analyze based on the body movement performance as a feature. Specifically, selecting appropriate points on the limb skeleton as feature points can reflect the details of its limb movement. In the hand skeleton line image, the skeleton lines of the five fingers and the skeleton line corresponding to the palm are included. In the skeleton image topology analysis, the endpoints and intersections of the skeleton are particularly important. In this embodiment, a hit-and-miss algorithm is used to extract these key points on the skeleton image. Among them, the endpoint extraction template is as follows: Figure 4 As shown, from Figure 4 From the template structures 41 and 42, we can see that only lines with at least two pixels are matched, which reduces the impact of image noise on endpoint extraction to a certain extent. The matched pixels are "covered" or "surrounded" by background pixels. The extraction template of the intersection is as follows Figure 5 As shown, from Figure 5 From template structures 51, 52, and 53, we can see that the skeleton intersection template can find the intersection points of three or more lines. Thus, all endpoints and intersections of the hand skeleton line image are obtained using the endpoint extraction template and the intersection extraction template.
[0089] The first distance between a candidate pixel point of a skeleton line in any skeleton line image and a previous skeleton line image is the Euclidean distance between endpoints or intersections in two adjacent skeleton line images.
[0090] Secondly, based on the first distance, the number of frames of the skeleton line image set and the frequency of the candidate image, the initial value coefficients of the candidate pixel points of the skeleton line are determined.
[0091] Wherein, the candidate image is a skeleton line image in which the candidate pixel points are displaced. For a single skeleton line in the image, if the posture changes, such as the bending of the finger, that is, the distance between the two end points of the corresponding skeleton line changes, then the newly appeared intersection on the skeleton line is considered to be a high-value point with a high probability. If the position of the skeleton line changes in consecutive frames, such as the movement of the finger or palm position, that is, the endpoint corresponding to the skeleton line changes position in the consecutive images, then the endpoint is considered to be a high-value point with a high probability. It should be noted that the change in position in the continuous image mentioned here refers to the continuous change in the endpoint position throughout the entire time sequence. The pixel points that change position only in some frames will be discussed in subsequent steps. The remaining points other than the high-value points are recorded as low-value points.
[0092] In some possible implementations, the frequency of displacement of candidate pixels for a single skeleton line in the image frames is determined based on the frame count of the skeleton line image set and the frequency of the candidate image. The first distance and the frequency are then combined to obtain an initial value coefficient for the candidate pixel for the skeleton line. For example, the initial value coefficient can be obtained by multiplying the first distance by the frequency. In this way, by combining the first distance with the frequency of displacement of candidate pixels for a single skeleton line in the image frames, the initial value coefficient for the candidate pixel can be accurately obtained, thereby facilitating the rapid determination of whether the candidate pixel is a high-value point.
[0093] Among them, the initial value coefficient is shown in formula (1):
[0094] (1);
[0095] in, Represents a single skeleton line in the image The initial value coefficients of the endpoints or intersections, Indicates the first frame, Indicates the total number of video frames. Indicates the The coordinates of the endpoints or intersections of the skeleton lines in the frame, Indicates the frequency of frames in which endpoints or intersections in a video appear to move. A single skeleton line in a video can represent a finger in the original video. Represents a single skeleton line The initial value coefficients of the endpoints or intersections, The expression consists of two parts, the first of which is Indicates: A single skeleton line of a frame The average Euclidean distance between the endpoint or intersection of the video and the corresponding point in the previous frame. The larger the value, the closer the single skeleton line in the video. The larger the amplitude of the corresponding finger displacement, the more likely the endpoint or intersection is a high-value point. Represents a single skeleton line The frequency of the number of frames in which the endpoints or intersections of the skeleton line appear displaced in the video. The larger the value, the longer the skeleton line continues to displace in time, and the more likely its endpoints or intersections are high-value points.
[0096] Thirdly, based on the second distance between the endpoints of the skeleton lines in any one of the skeleton line images, the value weight of the candidate pixel point in any one of the skeleton line images is determined.
[0097] Among them, the value weight is shown in formula (2):
[0098] (2);
[0099] in, Represents a single skeleton line The value weight of the upper endpoint or intersection, Indicates the Skeleton line in frame The distance between the two endpoints of and Respectively represent the two endpoints of the skeleton line E in the i-th frame, is the normalization function. A single skeleton line The value weight of the upper endpoint or intersection, where and Indicates skeleton line The greater the difference between the maximum and minimum values of the Euclidean distance between the two end points, the stronger the skeleton line. The greater the change in the corresponding finger posture in the video, the greater the value weight of the corresponding endpoint or intersection. If there is no intersection on the skeleton line, that is, the finger posture in the video does not change, then Equal to 0, then it is considered No Have an impact, so , therefore, it is necessary to Add 1 before.
[0100] Finally, based on the initial value coefficient and the value weight, the value coefficient of the candidate pixel point in any skeleton line image is obtained.
[0101] Here, the value weight is multiplied by the initial value coefficient, that is, , get the value coefficient of each endpoint or intersection The larger the value, the greater the feature value that the point in the video may contain. In this way, by combining the value weight of the skeleton line endpoint or intersection and the initial value coefficient, we can more accurately obtain the value coefficient of each skeleton line endpoint or intersection in the skeleton line image, and then facilitate the determination of whether the skeleton line endpoint or intersection is a high-value point.
[0102] 302. Determine the high-value point and the low-value point based on the value coefficient.
[0103] Here, for the value coefficient The points with the value coefficient of the skeleton pixel are recorded as high-value points, and the rest of the points are recorded as low-value points. In this way, the high-value points and low-value points can be quickly divided in the skeleton line image through the value coefficient of the skeleton pixel point.
[0104] 103 : Determine target feature points among the high-value points and the low-value points based on the local neighborhood of the low-value points in the skeleton line image to which they belong.
[0105] Here, the local neighborhood of the low-value point in the skeleton line image to which it belongs is the local neighborhood of the low-value point in the skeleton line image. The value coefficient of the low-value point is corrected by the two-dimensional entropy of the low-value point in the local neighborhood, so as to redetermine whether the point is a high-value point according to the corrected value coefficient. And the high-value point finally determined is used as the target feature point. In this embodiment, for any pixel point, the method for obtaining its local neighborhood is: with the pixel point as the center point, a window of a preset size is constructed, and the area where the pixel points except the center point are located in the window is used as the local neighborhood of the pixel point. In this embodiment, the preset size is In specific applications, implementers can set it according to specific circumstances.
[0106] In some possible implementations, the above step 103 can be performed by Figure 6 The steps shown achieve:
[0107] 601: Determine local feature change information in the local neighborhood.
[0108] Here, for any low-value point on any skeleton line, the frequency of occurrence of a tuple consisting of the grayscale value of the any low-value point and the grayscale value of the local neighborhood is determined; and based on the frequency of occurrence and the neighborhood scale corresponding to the local neighborhood, the local feature change information in the local neighborhood is determined.
[0109] In some possible implementations, first count the skeleton lines Gray value corresponding to the jth low value point The frequency of occurrence of a feature binary consisting of a low-value point and the grayscale mean of its local neighborhood is calculated. The square value of the neighborhood scale corresponding to the local neighborhood is calculated and divided by this frequency to obtain the local feature change information in the local neighborhood. In this way, by calculating the frequency of occurrence of a binary consisting of a low-value point and the grayscale value in the local neighborhood and the neighborhood scale, the local feature change information in the local neighborhood can be obtained, thereby accurately reflecting the displacement, posture changes, and grayscale changes of the hand in time.
[0110] 602. Based on the local feature change information, the value coefficient of the low value point is corrected to obtain a corrected value point with an updated value coefficient.
[0111] 603 : Determine the target feature point among the revised value point and the high value point based on the updated value coefficient of the revised value point and the value coefficient of the high value point.
[0112] In steps 601 to 603 above, the most important points to focus on among the selected low-value points are those that change position only in a portion of the frames. Although the temporal change is small, they may still contain key information. For example, the grayscale fluctuations around the connected domain in the original grayscale image before and after the change are large. Therefore, the calculated value coefficient needs to be corrected, as shown in formula (3):
[0113] (3);
[0114] in, represents the modified value coefficient; , Indicates skeleton line The two-dimensional entropy of the low-value points in the local neighborhood of any frame grayscale image, that is, the local feature change information of the low-value points in the local neighborhood; Indicates skeleton line The two-dimensional entropy of the low-value points in the local neighborhood of the next frame of grayscale image in any frame of grayscale image, Expression and The parameters in the expression of are the same. , Indicates skeleton line Gray value corresponding to the jth low value point The frequency of occurrence of a feature binary group composed of the grayscale mean of its local neighborhood. is the scale of the local neighborhood.
[0115] The skeleton line whose position changes in time series is calculated by formula (3) Corresponding to the change of local grayscale features before and after the change in the original grayscale image, here is In this embodiment, what needs to be calculated is the two-dimensional entropy of the neighborhood around the low-value point. Therefore, analogous to the two-dimensional entropy of the image, the skeleton line is calculated. The low-value points on the local neighborhood The two-dimensional entropy within the video is calculated, and the change of the two-dimensional entropy between adjacent frames in the entire video is calculated. , and sum them up. The summation result shows the change of grayscale features of the local neighborhood on the grayscale image caused by the displacement of the original low-value point. The larger the value, the more significant the grayscale information change caused by the point in the local neighborhood, and the corresponding value coefficient should be relatively higher. Based on this, according to the temporal displacement, posture change, and grayscale change of the hand, high-value points on the corresponding skeleton line are determined. These points can reflect the key features of most information in the hand video. In this way, the value coefficients of low-value points are corrected based on the local feature change information of their local neighborhood, allowing for more accurate identification of high-value points, thereby enriching the key information contained in the target feature points.
[0116] 104 , based on the target feature points, understand the intention of the person to be accompanied to obtain the target intention of the person to be accompanied.
[0117] Here, by using a trained neural network to perform gesture recognition on the target feature points, and combining it with a preset speech recognition probability set, the intention of the accompanying person can be accurately understood to obtain the target intention.
[0118] In some possible implementations, step 104 may be implemented through the following process:
[0119] First, a trained neural network is used to perform gesture recognition on a skeleton line image set containing the target feature points, obtaining a gesture probability set. Here, a trained neural network is a neural network used for gesture recognition, such as a convolutional neural network or a residual neural network. The target feature points are marked on the skeleton line image set, and the skeleton line image set containing the target feature points is used as input to the trained neural network. Gesture recognition is performed using the trained neural network, resulting in a recognition result, i.e., a gesture probability set.
[0120] Next, the preset speech recognition probability set and the gesture probability set are fused to obtain a fused probability set. Here, the preset speech recognition probability set and the gesture probability set of gesture recognition are intentionally fused according to a fusion algorithm based on a confusion matrix to obtain a fused probability set.
[0121] Finally, based on the fused probability set, the intent of the person to be accompanied is understood to obtain the target intent of the person to be accompanied. Here, under different intent classifications, the F1 score is used to calculate the weight ratio of different intents when the two probability sets are fused, and then the target intent of the person to be accompanied is determined. In this way, by fusing the preset speech recognition probability set with the gesture probability set and understanding the intent of the fused probability set, the obtained target intent is more accurate and better suited to the needs of the person to be accompanied.
[0122] In an embodiment of the present invention, high-value and low-value points are determined from the skeleton pixel points of the hand based on the temporal variation information corresponding to the skeleton line image set. Target feature points are then determined from the high-value and low-value points based on the local neighborhood of the low-value points in the skeleton line image to which they belong. Based on the target feature points, the intention of the person to be cared for is understood, and the target intention of the person to be cared for is obtained. In this way, by determining the target feature points based on the high-value and low-value points, the selection of feature points can be optimized, so that the feature points for intention understanding are as close as possible to those containing high-value pixels, thereby improving the efficiency and accuracy of intention understanding.
[0123] The embodiment of the present invention provides a multimodal accompanying robot intention understanding system, such as Figure 7 As shown, the system 700 includes:
[0124] An acquisition module 701 is used to acquire a skeleton line image set of the hand of the person to be accompanied; a first determination module 702 is used to determine high-value points and low-value points in the skeleton pixel points of the hand based on the temporal change information corresponding to the skeleton line image set; a second determination module 703 is used to determine target feature points in the high-value points and the low-value points based on the local neighborhood of the low-value points in the skeleton line image to which they belong; an intention understanding module 704 is used to understand the intention of the person to be accompanied based on the target feature points to obtain the target intention of the person to be accompanied.
[0125] In some possible implementations, the acquisition module 701 is further used to acquire a hand movement video of the person to be accompanied; perform binary segmentation on the movement video to obtain a binary image; and perform skeleton extraction on the binary image to obtain the skeleton line image set.
[0126] In some possible implementations, the acquisition module 701 is further used to determine a target circular disk image in the binary image with any point in the binary image as the center of the circle; determine the skeleton pixel points tangent to the edge of the hand in the target circular disk image; and determine the skeleton line image set based on the skeleton pixel points.
[0127] In some possible implementations, the first determination module 702 is further configured to assign a value coefficient to the skeleton pixel points based on temporal variation information corresponding to the skeleton line image set; and to determine the high-value points and the low-value points based on the value coefficient.
[0128] In some possible implementations, the first determination module 702 is further used to determine, for any skeleton line image in the skeleton line image set, a first distance between the candidate pixel points of the skeleton line in the previous frame of the skeleton line image and the temporal change information corresponding to the any skeleton line image; wherein the candidate pixel points include: endpoints or intersections of the skeleton line; based on the first distance, the number of frames of the skeleton line image set and the frequency of the candidate images, determine the initial value coefficient of the candidate pixel points of the skeleton line; wherein the candidate image is a skeleton line image in which the candidate pixel points are displaced; based on the second distance between the endpoints of the skeleton lines in the any skeleton line image, determine the value weight of the candidate pixel points in the any skeleton line image; based on the initial value coefficient and the value weight, determine the value coefficient of the candidate pixel points in the any skeleton line image.
[0129] In some possible implementations, the first determination module 702 is further used to determine the frequency of the image frames in which the candidate pixel points of a single skeleton line are displaced based on the number of frames of the skeleton line image set and the frequency of the candidate image; and to fuse the first distance and the frequency to obtain the initial value coefficient of the candidate pixel points of the skeleton line.
[0130] In some possible implementations, the second determination module 703 is further used to determine local feature change information in the local neighborhood; based on the local feature change information, the value coefficient of the low-value point is corrected to obtain a corrected value point carrying an updated value coefficient; based on the updated value coefficient of the corrected value point and the value coefficient of the high-value point, the target feature point is determined between the corrected value point and the high-value point.
[0131] In some possible implementations, the second determination module 703 is further used to determine, for any low-value point on any skeleton line, the frequency of occurrence of a tuple consisting of the grayscale value of the any low-value point and the grayscale value of the local neighborhood; and determine the local feature change information in the local neighborhood based on the frequency of occurrence and the neighborhood scale corresponding to the local neighborhood.
[0132] In some possible implementations, the intention understanding module 704 is further used to use a trained neural network to perform gesture recognition on the skeleton line image set carrying the target feature points to obtain a gesture probability set; fuse the preset voice recognition probability set and the gesture probability set to obtain a fused probability set; based on the fused probability set, understand the intention of the person to be cared for to obtain the target intention of the person to be cared for.
[0133] Optionally, the transmission medium may be a wired link (such as, but not limited to, coaxial cable, optical fiber, and digital subscriber line (DSL)) or a wireless link (such as, but not limited to, wireless Fidelity (WIFI), Bluetooth, and mobile device network). It should be noted that the apparatus provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be distributed to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the shopping guide apparatus provided in the above embodiment and the image extension method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0134] Figure 8 : is a structural diagram of a multimodal accompanying robot intention understanding device provided by an embodiment of the present invention. For example, Figure 8 As shown, the device 800 includes: a memory 801, a processor 802, and a computer program 803 stored in the memory 801 and running on the processor 802. When the processor 802 executes the computer program 803, the device can perform any of the multimodal care robot intention understanding methods described above. In addition, an embodiment of the present invention also protects a device that may include a memory and a processor, wherein the memory stores executable program code, and the processor is configured to call and execute the executable program code to perform a multimodal care robot intention understanding method provided by an embodiment of the present invention.
[0135] This embodiment can divide the functional modules of the device according to the above-mentioned method example. For example, it can correspond to each functional module, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation. In the case of dividing each module corresponding to each function, the device may also include a signal upload module, a determination module, an adjustment module, etc. It should be noted that all relevant contents of each step involved in the above-mentioned method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.
[0136] It should be understood that the apparatus provided in this embodiment is used to implement the aforementioned method for understanding the intentions of a multimodal care robot, and thus can achieve the same effects as the aforementioned implementation method. When an integrated unit is employed, the apparatus may include a processing module and a storage module. When the apparatus is applied to a device, the processing module can be used to control and manage the device's actions. The storage module can be used to support the device in executing program code, etc. The processing module can be a processor or controller, which can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory. Furthermore, the apparatus provided in the embodiments of the present invention can specifically be a chip, component, or module. The chip may include a connected processor and memory; the memory is used to store instructions. When the processor calls and executes the instructions, the chip can execute the method for understanding the intentions of a multimodal care robot provided in the aforementioned embodiment. This embodiment also provides a computer-readable storage medium having a computer program code stored therein. When the computer program code is executed on a computer, the computer executes the above-mentioned related method steps to implement a multimodal care robot intention understanding method provided in the above embodiment. This embodiment also provides a computer program product. When the computer program product is executed on a computer, the computer executes the above-mentioned related steps to implement a multimodal care robot intention understanding method provided in the above embodiment. Among them, the device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here. Through the description of the above embodiments, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual application, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In the embodiments provided by the present invention, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.On the other hand, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, which may be electrical, mechanical or other forms.
[0137] It should be noted that the above-mentioned order of the embodiments of the present invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-tasking and parallel processing are also possible or may be advantageous. The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. The above content is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered within the scope of protection of the present invention.
Claims
1. A multimodal care robot intention understanding method, characterized in that: The method comprises: Acquire a skeleton line image set of the hand of the person to be accompanied; Based on the temporal variation information corresponding to the skeleton line image set, determining high-value points and low-value points among the skeleton pixel points of the hand; Determine a target feature point among the high-value points and the low-value points based on a local neighborhood of the low-value points in the corresponding skeleton line image; Based on the target feature points, the intention of the person to be accompanied is understood to obtain the target intention of the person to be accompanied; The determining of high-value points and low-value points in the skeleton pixel points of the hand based on the time-series change information corresponding to the skeleton line image set includes: Assigning value coefficients to the skeleton pixels based on the temporal variation information corresponding to the skeleton line image set; Based on the value coefficient, determining the high value point and the low value point; The assigning of value coefficients to the pixels on the skeleton based on the temporal variation information corresponding to the skeleton line image set includes: For any skeleton line image in the skeleton line image set, based on the temporal change information corresponding to the any skeleton line image, determine a first distance between the any skeleton line image and a candidate pixel point of a skeleton line in a previous frame of the skeleton line image; wherein the candidate pixel point includes: an endpoint or an intersection of a skeleton line; Determining the initial value coefficient of the candidate pixel point of the skeleton line based on the first distance, the number of frames of the skeleton line image set and the frequency of the candidate image; wherein the candidate image is the skeleton line image in which the candidate pixel point is displaced; Determining a value weight of a candidate pixel point in any one of the skeleton line images based on a second distance between endpoints of the skeleton lines in any one of the skeleton line images; Based on the initial value coefficient and the value weight, obtaining the value coefficient of the candidate pixel point in any one of the skeleton line images; The step of determining the initial value coefficient of the candidate pixel point of the skeleton line based on the first distance, the number of frames of the skeleton line image set and the frequency of the candidate image comprises: Based on the number of frames of the skeleton line image set and the frequency of the candidate image, determining the frequency of the image frames in which the candidate pixel points of a single skeleton line are displaced; The first distance and the frequency are combined to obtain an initial value coefficient of the candidate pixel point of the skeleton line; The determining of target feature points among the high-value points and the low-value points based on the local neighborhood of the low-value points in the corresponding skeleton line image comprises: Determining local feature change information in the local neighborhood; Based on the local feature change information, the value coefficient of the low value point is corrected to obtain a corrected value point carrying an updated value coefficient; The target feature point is determined among the revised value point and the high value point based on the updated value coefficient of the revised value point and the value coefficient of the high value point.
2. A method for understanding intention of a multimodal accompanying robot according to claim 1, characterized in that: The step of obtaining a skeleton line image set of the hand of the person to be accompanied includes: Acquire a hand movement video of the person to be accompanied; Binarizing and segmenting the action video to obtain a binary image; The skeleton is extracted from the binary image to obtain the skeleton line image set.
3. A multimodal accompanying robot intention understanding method according to claim 2, characterized in that: The step of extracting the skeleton of the binary image to obtain the skeleton line image set includes: Taking any point in the binary image as the center of a circle, determining a target disk image in the binary image; In the target disk image, determining skeleton pixel points tangent to the edge of the hand; Based on the skeleton pixel points, the skeleton line image set is determined.
4. A method for understanding intention of a multimodal accompanying robot according to claim 1, characterized in that: The determining of the local feature change information in the local neighborhood includes: For any low-value point on any skeleton line, determine the frequency of occurrence of a tuple consisting of the gray value of the any low-value point and the gray value of the local neighborhood; Based on the occurrence frequency and the neighborhood scale corresponding to the local neighborhood, local feature change information in the local neighborhood is determined.
5. The method for understanding intention of a multimodal accompanying robot according to claim 1, characterized in that: The step of understanding the intention of the object to be accompanied based on the target feature point to obtain the target intention of the object to be accompanied includes: Using a trained neural network to perform gesture recognition on a skeleton line image set carrying the target feature points to obtain a gesture probability set; Fusing the preset speech recognition probability set with the gesture probability set to obtain a fused probability set; Based on the fused probability set, the intention of the object to be accompanied is understood to obtain the target intention of the object to be accompanied.
6. A multimodal companion robot intention understanding system, characterized in that: The system comprises: An acquisition module, used for acquiring a skeleton line image set of the hand of the person to be accompanied; A first determination module, configured to determine high-value points and low-value points among the skeleton pixel points of the hand based on the temporal variation information corresponding to the skeleton line image set; A second determination module is used to determine a target feature point among the high-value points and the low-value points based on a local neighborhood of the low-value points in the corresponding skeleton line image; An intention understanding module, used to understand the intention of the object to be accompanied based on the target feature points, and obtain the target intention of the object to be accompanied; The determining of high-value points and low-value points in the skeleton pixel points of the hand based on the time-series change information corresponding to the skeleton line image set includes: Assigning value coefficients to the skeleton pixels based on the temporal variation information corresponding to the skeleton line image set; Based on the value coefficient, determining the high value point and the low value point; The assigning of value coefficients to the pixels on the skeleton based on the temporal variation information corresponding to the skeleton line image set includes: For any skeleton line image in the skeleton line image set, based on the temporal change information corresponding to the any skeleton line image, determine a first distance between the any skeleton line image and a candidate pixel point of a skeleton line in a previous frame of the skeleton line image; wherein the candidate pixel point includes: an endpoint or an intersection of a skeleton line; Determining the initial value coefficient of the candidate pixel point of the skeleton line based on the first distance, the number of frames of the skeleton line image set and the frequency of the candidate image; wherein the candidate image is the skeleton line image in which the candidate pixel point is displaced; Determining a value weight of a candidate pixel point in any one of the skeleton line images based on a second distance between endpoints of the skeleton lines in any one of the skeleton line images; Based on the initial value coefficient and the value weight, obtaining the value coefficient of the candidate pixel point in any one of the skeleton line images; The step of determining the initial value coefficient of the candidate pixel point of the skeleton line based on the first distance, the number of frames of the skeleton line image set and the frequency of the candidate image comprises: Based on the number of frames of the skeleton line image set and the frequency of the candidate image, determining the frequency of the image frames in which the candidate pixel points of a single skeleton line are displaced; The first distance and the frequency are combined to obtain an initial value coefficient of the candidate pixel point of the skeleton line; The determining of target feature points among the high-value points and the low-value points based on the local neighborhood of the low-value points in the corresponding skeleton line image comprises: Determining local feature change information in the local neighborhood; Based on the local feature change information, the value coefficient of the low value point is corrected to obtain a corrected value point carrying an updated value coefficient; The target feature point is determined among the revised value point and the high value point based on the updated value coefficient of the revised value point and the value coefficient of the high value point.
