A service robot artificial labeling data screening method for elderly users

Abstract

The application discloses a kind of service robot artificial marking data screening methods for old users, belong to intelligent robot technical field.Service robot when entering certain old user home environment, according to new category data collected in user home environment, need select the image most easily to old user labeling with most learning value, present it to user for labeling.The specific screening method is: robot is with three kinds of attributes (perception brightness, contrast, size) of image as feature, in combination with image labeling difficulty prediction model, obtain the labeling difficulty prediction of image for old user.Combining the labeling difficulty prediction value of image with other three kinds of index describing image learning value, calculate the screening value measure of image, and then according to the measure select a certain proportion of value measure maximum image, as request old user to give out labeling and require robot learning data.

Description

TECHNICAL FIELD

[0001] The application relates to a service robot artificial annotation data screening method for an elderly user and belongs to the technical field of intelligent robots. BACKGROUND

[0002] The elderly population in the world is steadily growing, and it is estimated that by 2050, one sixth of the world's population will be over 65 years old. At the same time, the degree of population aging in China has become increasingly apparent since the 1990s of last century. Therefore, in the near future, endowment will become an urgent problem that the whole world society faces. Due to better privacy protection and autonomy, most elderly people prefer to choose home endowment. Due to the degradation of physical and cognitive functions caused by aging, elderly users urgently need the help of service robots to improve their quality of life.

[0003] Generally, a service robot entering an elderly user's home environment will be equipped with a neural network model trained according to a predefined training data set of the manufacturer, which is used as the starting local model and is used for the execution of some advanced perception tasks. At the same time, new categories that are not in the predefined training data set and are not learned by the starting local model of the robot often appear in the user's home environment. This requires the service robot to collect new data in the user's home environment and request the user to provide annotations for the new data to learn the new data. For the service object of the service robot, the elderly user, the visual ability and judgment ability will decline with age, and some images that are easy to distinguish for ordinary people may be very difficult for the elderly user, and even some mislabeling may occur. In addition, the energy of the elderly user is also very limited, and the amount of data annotation that can be provided is often small. Therefore, how to select a small part of data from the large amount of new data collected by the robot, which is most beneficial to the robot to learn new categories and is most easily labeled by the elderly user, is an urgent problem to be solved. SUMMARY

[0004] In order to solve the above technical problems, the application provides a service robot artificial annotation data screening method for an elderly user, which is used for screening a small part of data most easily labeled by the elderly user and most valuable for learning from new data collected by the robot. In the screening process, an image screening value measurement method is used, which combines image labeling difficulty and three image learning value measurements. The screening value of the new data collected by the robot is calculated by the measurement method, and a small part of images with the largest screening value are selected as the data that need to be labeled by the elderly user and learned by the robot.

[0005] To solve the technical problems, the solution of the application is:

[0006] The application discloses a service robot artificial labeling data screening method for the elderly, extracts a feature vector related to image labeling difficulty, and predicts the labeling difficulty of an image for the elderly according to an image labeling difficulty prediction model; an image screening value measurement method is adopted to comprehensively measure the image labeling difficulty prediction value and the image learning value, and part of image data with the highest labeling difficulty for the elderly and the highest learning value is screened from a large amount of new category image data collected from a user's home environment.

[0007] Further, the prediction method of the labeling difficulty of the image for the elderly is as follows:

[0008] (1) three image attributes closely related to the image labeling difficulty, namely, perceived brightness, contrast and size, are determined, and the three image attributes are used to form a labeling difficulty feature vector;

[0009] (2) the image labeling difficulty is divided into different levels, a part of images is selected from an open source image data set as survey images, and the labeling difficulty true value of the survey images for the elderly is counted; the labeling difficulty true value is the average value of the labeling difficulty of all the elderly participating in the survey under a certain image;

[0010] (3) all the survey images are divided into a training set and a test set, and an image labeling difficulty prediction model capable of predicting the labeling difficulty of an image for the elderly according to the feature vector of the image is trained according to the labeling difficulty feature vector and the labeling difficulty true value of the training set images;

[0011] (4) the labeling difficulty feature vector of the test set images is used as the input of the image labeling difficulty prediction model in step (3), and the labeling difficulty prediction value of the test images for the elderly is obtained.

[0012] The significance of this data screening method is that the image data is considered in terms of the labeling difficulty of the elderly users and the learning value of the robot neural network model. On the one hand, when the robot selects the images that need to be manually labeled by the elderly users, the selected images should be easy for the elderly users to label, so as to reduce the occurrence of false labeling and avoid the damage of too much mislabeled data to the learning process of the robot; and the brain labor burden of the data labeling task on the elderly users is reduced, and the acceptability of the data labeling task for the elderly users is ensured. On the other hand, the robot also needs to ensure that the selected images have high learning value, since the amount of manual labeling that can be completed by the elderly users is very limited, so the valuable data selection capacity should be concentrated on the data that is most uncertain for the local model of the robot, most similar to the selected data, and most similar to other unselected data. The data screening method proposed in the present application considers both requirements, and ensures that the selected data is easy for the elderly users to label while ensuring their learning value.

[0013] Compared with the prior art, the present application has the following advantages:

[0014] (1) A feature vector composed of three image attributes closely related to the labeling difficulty of the image is proposed, which is used to predict the labeling difficulty of a new image for the elderly users; the design of this feature vector is derived from existing medical research on the vision of the elderly, and has strong interpretability;

[0015] (2) An image screening value measurement that takes into account the labeling difficulty of the image and the learning value of the image is designed, and a screening method is designed for the value measurement, which is used to screen a small part of images that are easy for the elderly users to label and have the highest learning value from a large number of new data collected by the robot. Compared with the traditional random screening method and the screening method that only considers the learning value of the data without considering the labeling difficulty of the data, the prediction accuracy of the robot neural network model on the new class after learning is obviously improved under the condition that the elderly users provide data labeling. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The age distribution diagram of the elderly volunteers participating in the survey shown in the embodiment of the present application;

[0017] Figure 2 The image and its question examples presented to the elderly volunteers during the online survey shown in the embodiment of the present application;

[0018] Figure 3 The distribution diagram of the labeling difficulty true value of the training set images obtained according to the survey shown in the embodiment of the present application;

[0019] Figure 4After using the method proposed in the present application and the data screened out by random screening and only considering the data learning value to complete learning, the test accuracy of the robot neural network model is compared;

[0020] Figure 5 A flowchart of the artificial labeling data screening method for the service robot for the elderly user is shown in the embodiment of the present application. DETAILED DESCRIPTION

[0021] The present application will be further described in detail below with reference to the accompanying drawings.

[0022] When the service robot enters a certain home environment of an elderly user, it needs to learn new categories of data collected in the user's home environment and combine the corresponding data given by the user to learn these new categories. However, the number and difficulty of the labeling tasks that can be completed by the elderly user are very limited, so the robot needs to select the images that are most valuable for learning and most easily labeled by the elderly user. The present application is a data screening method for comprehensively measuring the image labeling difficulty and image learning value for a service robot serving an elderly user, which needs to request the elderly user to give new data labeling. As shown in Figure 5 First, the feature vector related to the image labeling difficulty is extracted to predict the labeling difficulty of the image for the elderly user. Then, the image screening value measurement method that combines the image labeling difficulty prediction value and the image learning value is used to screen out some images that are most easily labeled by the elderly user and most valuable for learning from a large number of image data collected from the user's home environment.

[0023] Considering the prediction of the labeling difficulty of the image for the elderly user, the image properties closely related to the labeling difficulty need to be determined first. After excluding the influence of eye diseases, the most common and key three visual problems of the elderly are summarized as visual decline, contrast sensitivity decline, and sensitivity decline to dark light from the existing medical research literature on the visual problems of the elderly. These three visual problems correspond to the three properties of the image size, contrast, and perceived brightness, that is, for the elderly user, the image size, contrast, and perceived brightness are negatively correlated with the labeling difficulty in theory. Therefore, the normalized results of the perceived brightness, contrast, and size are used as the feature vector for predicting the labeling difficulty of the image.

[0024] To train the image annotation difficulty prediction model, the embodiment selects a part of images from the open-source robot image dataset ARID (Autonomous Robotic Indoor Dataset, an indoor object image dataset collected by a robot) as survey images, and uses an online questionnaire survey method to count the annotation difficulty true value of the survey images for the elderly users. The embodiment simultaneously surveys 69 elderly users, the age distribution of these elderly volunteers is as shown in Figure 1 The embodiment finally determines the annotation difficulty true value of 290 indoor object images, and uses 190 images and their annotation difficulty true value as a training set, and the remaining 100 images as a test set to test the effect of the model. Using the prior knowledge in the training set, a regression model is constructed that maps the annotation difficulty feature vector of an image to its annotation difficulty, and the effect of the regression model is tested by the test set. The regression model that meets the test requirements is used as the image annotation difficulty prediction model.

[0025] In the online questionnaire survey process, the total number of survey images allocated to each elderly volunteer is 29, and the 29 images are divided into three batches according to the number of 10-10-9, and are presented to the user with an interval of 1 month to ensure the enthusiasm of the elderly volunteers participating in the survey, and to avoid the influence of the survey results of the previous batch of images on the survey of the next batch of images. As shown in Figure 2 Each survey image corresponds to two questions that need to be answered by the user. First, ask the user which category the object in the image belongs to, and second, ask the user to give the difficulty level of judging the category of the object in the image (the level range is 1 to 10, 1 represents the easiest, and 10 represents the most difficult). When the elderly volunteer completes the answers to the two questions of a certain image, the annotation difficulty of the image for the user will be calculated in the following way: if the elderly volunteer gives the correct category judgment of the object in the image, the annotation difficulty of the image for the user is the annotation difficulty level given by the user, otherwise, it is directly counted as the highest difficulty level (i.e. difficulty level 10). The annotation difficulty true value of a survey image is the average of the annotation difficulties of all elderly volunteers participating in the survey.

[0026] In a specific implementation of the present application, when constructing the image annotation difficulty prediction model, the decision tree regression algorithm is used as the base learner, the Bagging regression method is used, the mean squared error (MSE) is used as the loss function, and the grid search optimal hyperparameter combination method is used to train a regression model that can predict the image annotation difficulty according to the image feature vector designed by the present application. Based on the trained regression model, the difficulty feature vector of the image to be predicted is only needed as input, and the annotation difficulty prediction value for the elderly user is obtained according to the above regression model.

[0027] The annotation difficulty prediction value of the obtained new image is combined with the learning value metric calculated based on the other three feature vectors and the output probability vector extracted by the image on the current neural network model of the robot, and the screening value metric of the image is calculated in the following manner:

[0028]

[0029]

[0030]

[0031] BvSB(x)=max(g(f(x)))-submax(g(f(x)))

[0032] wherein Metric(x) is the screening value metric of the image x, Sim(x,D U ) represents the average similarity between the image x and the unselected images, as the representative metric result; ||·|| represents the modulus, f(x) is the feature extracted by the image x on the feature extraction layer of the robot neural network, g(·) represents the classification layer and the softmax layer of the robot neural network, and accordingly, g(f(x)) represents the probability vector output when the image x is input into the robot neural network; and represent the selected images and the other unselected images, respectively, represents the number of images in the selected image set, represents the number of images in the unselected image set, the function Diff(·) represents the image annotation difficulty prediction model, ε is a very small value to prevent the denominator from being 0, submax(·) represents the second maximum value, and max(·) represents the maximum value.

[0033] For all new data collected by the robot from the home environment of the elderly user, a small portion of the most easily labeled by the elderly user and the most valuable learning images are dynamically selected according to the image screening value metric defined above. In a specific implementation of the present application, the specific implementation method of the artificial labeling data screening method of the service robot for the elderly user is: after the three basic attributes (perceived brightness, contrast, size) of the image are normalized, they are used as the input of the above-mentioned labeling difficulty regression model to obtain the predicted value of the labeling difficulty for the elderly user. And according to the current neural network model of the robot, the feature vector extracted by the image on the neural network and the output probability vector are obtained, and then the uncertainty metric of the image, the average similarity between the image and the selected images (diversity metric of the image), and the average similarity between the image and the unselected images (representativeness metric of the image) are calculated, and finally the screening value metric of the image is calculated by combining the above-mentioned image labeling difficulty prediction value. Then select the image with the largest screening value from the unselected images and move it to the selected image set, and repeat this process until the number of selected images reaches a certain proportion of the total number of images.

[0034] Table 1 Pseudocode of the data screening method of the present application

[0035]

[0036] As shown in Table 1, the following steps are included:

[0037] Step 1, the new category image data set collected by the service robot for the elderly user is denoted as The layers in the neural network model built in the service robot are denoted as feature extraction layer f(·), classification and softmax layer g(·), and the image labeling difficulty prediction model is denoted as Diff(·);

[0038] Step 2, establish the selected image set Initialize to empty set; establish the unselected image set Initialize to

[0039] Step 3, judge whether the number of images in the selected image set satisfies the requirement: is the screening ratio. If yes, the image data in is the screening result, which is presented to the elderly user for labeling; if not, step 4 is entered;

[0040] Step 4, for extracting a perceptual brightness, a contrast, a size feature of the image and normalizing to obtain a labeling difficulty feature vector, and calculating a labeling difficulty prediction value Diff(x) of the image by using an image labeling difficulty prediction model;

[0041] Step 5, calculating an uncertainty measure value BvSB(x), a diversity measure value and a representative measure value Sim(x, D U ) of each image x;

[0042] Step 6, normalizing the image uncertainty measure value, the labeling difficulty prediction value, the diversity measure value and the representative measure value, and calculating a screening value measure Metric(x) of each image x;

[0043] Step 7, putting the image with the maximum screening value measure into a set and removing it from the image set ; returning to Step 3.

[0044] Figure 4 In the scenario that the old users provide manual labeling for the data screened by the robot, and the robot learns according to the screened data and the manual labeling thereof, the robot neural network performance obtained after data screening, manual labeling and learning by using the data screening method proposed in the present application, the random screening method and the screening method considering only data learning value but not data labeling difficulty is compared, and it can be obviously seen from the figure that the data screening method proposed has advantages.

[0045] The above only describes the preferred embodiments of the present application, and it should be noted that several improvements and modifications can be made without departing from the technical principles of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A service robot artificial annotation data screening method for elderly users, characterized by, The feature vector related to the image labeling difficulty is extracted, and the image labeling difficulty for the elderly users is predicted according to an image labeling difficulty prediction model; and an image screening value measurement method is used to screen part of the image data most easily labeled by the elderly users and most valuable for learning from a large amount of new category image data collected from the home environment of the users. The image screening value measurement method integrates the image labeling difficulty and three kinds of image learning values, and the calculation formula is: ； ； ； ； wherein, is a screening value metric of images, is an average similarity between images and images that have been selected as a diversity metric result; is an average similarity between images and images that have not been selected as a representativeness metric result; is a modulo operation, is a feature extracted from a feature extraction layer of a service robot neural network model, is a classification and softmax layer of a service robot neural network model, whose output probability vector is ; and represent a set of images that have been screened and a set of images that have not been selected, respectively, represents the number of images in the set of images that have been screened, represents the number of images in the set of images that have not been selected, represents an image annotation difficulty prediction model, is to prevent a minimum value of a denominator being 0, represents taking a second maximum value, represents taking a maximum value.​​ 2. The method of claim 1, wherein the method is a method of filtering artificial annotation data for a service robot for an elderly user, characterized by, The prediction method of the image labeling difficulty for the elderly users is: (1) three kinds of image attributes closely related to the image labeling difficulty, namely, perceived brightness, contrast and size, are determined, and the three kinds of image attributes are used to form a labeling difficulty feature vector; (2) the image labeling difficulty is divided into different levels, a part of images are selected from an open source image data set as investigation images, and the labeling difficulty true value of the investigation images for the elderly users is counted; the labeling difficulty true value uses the average value of the labeling difficulty of all the elderly users participating in the investigation under a certain image; (3) all the investigation images are divided into a training set and a test set, and an image labeling difficulty prediction model capable of predicting the labeling difficulty of an image for the elderly users according to the feature vector of the image is trained according to the labeling difficulty feature vector and the labeling difficulty true value of the training set images; (4) the labeling difficulty feature vector of the test set images is used as the input of the image labeling difficulty prediction model in step (3), and the labeling difficulty prediction value of the test images for the elderly users is obtained.

3. The method of claim 1, wherein the method is a method of filtering artificial annotation data for a service robot for an elderly user, characterized by, The image data most easily labeled by the elderly users and most valuable for learning are screened from a large amount of new category image data collected from the home environment of the users, and the specific method is: Step 1, a new category image dataset collected by the service robot for the elderly user is denoted as Each layer in the service robot neural network model is denoted as a feature extraction layer , a classification and softmax layer An image labeling difficulty prediction model is denoted as ; Step 2, Establishing the set of images that have been filtered out , initialized to the empty set; establishing the set of images that have not been selected , initialized to ; Step 3: Determine the set of images that have been filtered. Does the number of images in the database meet the requirements? , For the selection ratio, if so, then... The image data in the image is used as the filtering result and presented to elderly users for annotation; otherwise, proceed to step 4. Step 4, for Each image in The image's perceived brightness, contrast, and size features are extracted and normalized to obtain a labeling difficulty feature vector. The image labeling difficulty prediction model is then used to calculate the predicted labeling difficulty value for the image. And calculate each image Features output by the feature extraction layer in the neural network model built into the service robot and the probability vector output by the softmax layer. ; Step 5, calculating uncertainty metric values for each image ;​​​ Step 6, normalizing the image uncertainty measure, the annotation difficulty prediction, the diversity measure, and the representativeness measure, and calculating a screening value measure for each image ;​ Step 7, put the image with the largest measure of screening value into the set and remove it from the image set ; go to step 3.

Images