A viewpoint estimation method based on uncertainty

Abstract

The present invention discloses a viewpoint estimation method based on uncertainty, which relates to the field of computer vision and includes the following steps: obtaining a data set and performing preprocessing; constructing a convolutional neural network, including a scene saliency feature extraction network, a line of sight feature extraction network, a feature fusion network, a temporal information extraction network, and a viewpoint decoding network; constructing an overall neural network to output the viewpoint position prediction coordinates and their uncertainty; designing a loss function to minimize the loss function of the overall neural network model; calculating the loss value between the viewpoint coordinate prediction value and the viewpoint coordinate label true value according to a supervised learning strategy network, and completing the training of the overall neural network model using a gradient backpropagation algorithm; using the trained overall neural network to perform viewpoint estimation on test image data and output uncertainty. The present invention uses an uncertainty extraction network, takes fused features as input, outputs uncertainty, and optimizes the uncertainty extraction network through the loss function.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a viewpoint estimation method based on uncertainty. Background Art

[0002] Human vision contains a wealth of information. The viewpoint is the intersection of the human line of sight and the object of attention. Viewpoint estimation is a key research topic in computer vision, encompassing the understanding of human attention and scene information. Viewpoint estimation has significant applications in both scientific research and commercial applications. It can be applied to analyzing the visual attention of children with autism, analyzing the gaze behavior of multiple people in multi-person social scenarios, and analyzing user interaction intent when embedded in robotic devices.

[0003] Viewpoint estimation is a method for estimating the gaze position of a subject in an image containing the subject and the scene they are in. Existing viewpoint estimation methods mainly use the entire image and the subject's head image as input data, the two-dimensional coordinates of the subject's gaze point in the image as output data, a neural network as the mapping model, and the gradient backpropagation algorithm to optimize the established neural network model. Existing gaze estimation methods are mainly divided into two categories. One category is viewpoint estimation methods based on single-frame images, which only extract features from a single frame image to predict the subject's viewpoint output in the current frame image. References: Recasens, A., Khosla, A., Vondrick, C., & Torralba, A. (2015). Where are they looking? .Advances in neural information processing systems, 28.; Another type is the viewpoint estimation method based on multi-frame images, that is, extracting features in units of continuous multi-frame images to predict the viewpoint output of the subject in the multi-frame images. References Chong, E., Wang, Y., Ruiz, N., & Rehg, JM (2020). Detecting attended visual targets in video. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition (pp. 5396-5406).

[0004] In recent years, viewpoint estimation methods have become increasingly mature, but they can only output predicted values, not the uncertainty of the predicted values, and lack interpretability.

[0005] Therefore, those skilled in the art are committed to developing a viewpoint estimation method based on uncertainty. Summary of the Invention

[0006] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is that the viewpoint estimation method in the prior art can only output a predicted value but cannot output the uncertainty of the predicted value.

[0007] To achieve the above object, the present invention provides a viewpoint estimation method based on uncertainty, characterized in that the method comprises the following steps:

[0008] S101: Acquire a data set and preprocess the data set;

[0009] S103: Constructing a convolutional neural network, wherein the convolutional neural network includes a scene saliency feature extraction network, a line of sight feature extraction network, a feature fusion network, a temporal information extraction network, and a viewpoint decoding network;

[0010] S105: constructing a total neural network, wherein the total neural network realizes output of viewpoint position prediction coordinates and their uncertainty;

[0011] S107: Designing a loss function to minimize the loss function of the total neural network model;

[0012] S109: According to the supervised learning strategy network, the loss value between the viewpoint coordinate prediction value and the viewpoint coordinate label true value is calculated according to the loss function, and the gradient back propagation algorithm is used to complete the training of the overall neural network model;

[0013] S111: Using the trained total neural network, perform viewpoint estimation on the test image data and output uncertainty.

[0014] Furthermore, in step S101, the data set includes training data, and the training data includes a scene image containing a subject and annotation information. According to the annotation information, the subject's face in a single-frame entire scene image is cropped, and a face position binary mask image is generated according to the position of the subject's face in the entire scene image.

[0015] Furthermore, the pre-processing of the data set in step S101 includes the following sub-steps:

[0016] S1011: cropping the face image in the scene image according to the annotation information of the rectangular frame circumscribing the face of the subject;

[0017] S1012: generating a face position binary mask image according to the position of the subject's face in the scene image;

[0018] S1013: Adjust the sizes of the scene image, the face image, and the face position binary mask image, and normalize the image pixel values.

[0019] Furthermore, in step S1013, the sizes of the scene image, the face image and the face position binary mask image are adjusted to 224×224, and the image pixel values ​​are normalized to the range of [-1, 1].

[0020] Furthermore, step S103 includes the following sub-steps:

[0021] S1031: Constructing the scene saliency feature extraction network, wherein the scene saliency feature extraction network is composed of a first ResNet-50 convolutional neural network and two bottleneck layers;

[0022] S1032: Constructing the sight feature extraction network, wherein the sight feature extraction network is composed of a second ResNet-50 convolutional neural network and two bottleneck layers;

[0023] S1033: Constructing the feature fusion network, wherein the feature fusion network is composed of two convolutional layers, two batch normalization layers and a ReLU activation function;

[0024] S1034: Constructing an uncertainty extraction network, wherein the uncertainty extraction network is composed of two pooling layers, a linear layer, and a Sigmoid activation function, and the output of the uncertainty extraction network is uncertainty;

[0025] S1035: Construct the viewpoint decoding network, which is composed of two pooling layers, a linear layer, and a Sigmoid activation function. The output of the viewpoint decoding network is the viewpoint position prediction coordinates.

[0026] Furthermore, the input data of the scene saliency feature extraction network is the data obtained by superimposing the preprocessed scene image and the face position binary mask image along the channel direction. The input data is first processed by the first ResNet-50 convolutional neural network and then processed by two bottleneck layers in sequence. The output of the scene saliency feature extraction network is the scene saliency feature f s , the input dimension of the first ResNet-50 convolutional neural network is 224×224×4.

[0027] Furthermore, the input data of the sight feature extraction network is the preprocessed face image data, the input data is first processed by the second ResNet-50 convolutional neural network, and then processed by two bottleneck layers in sequence, and the output of the sight feature extraction network is the sight feature f g, the input dimension of the second ResNet-50 convolutional neural network is 224×224×3.

[0028] Furthermore, the input of the feature fusion network is the scene saliency feature f s With the sight feature f g The feature vector after superposition along the channel direction, the output of the feature fusion network is the fusion feature f f .

[0029] Furthermore, the overall neural network includes the scene saliency feature extraction network, the sight feature extraction network, the feature fusion network, the uncertainty extraction network and the viewpoint decoding network, and the overall neural network transforms the feature fusion vector f f Input the uncertainty extraction network and the viewpoint decoding network respectively to obtain the uncertainty and the viewpoint coordinate prediction value

[0030] Furthermore, the loss function is:

[0031]

[0032] Among them, i is the sample number, n is the number of samples, y i is the true value of the viewpoint coordinate label of the i-th sample, is the viewpoint coordinate prediction value of the total neural network for the i-th sample, is the uncertainty of the i-th sample output of the total neural network, y i and The Euclidean norm of .

[0033] In a preferred embodiment of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention proposes an uncertainty extraction network that outputs the uncertainty of the model for a sample to indicate the degree of certainty of the model's predicted viewpoint coordinates for the sample;

[0035] 2. The uncertainty extraction network proposed in this invention takes the fusion feature as input and outputs uncertainty. The loss function designed adds Optimize the uncertainty extraction network;

[0036] 3. For a given scene picture containing a person and surrounding information, the present invention can output the coordinates of the person's gaze point and the uncertainty.

[0037] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 1 is a flow chart of a viewpoint estimation method according to a preferred embodiment of the present invention;

[0039] Figure 2 1 is a schematic diagram of a scene salient feature extraction network according to a preferred embodiment of the present invention;

[0040] Figure 3 1 is a schematic diagram of a line of sight feature extraction network according to a preferred embodiment of the present invention;

[0041] Figure 4 Schematic diagram of the bottleneck layer in the scene saliency feature extraction network of a preferred embodiment of the present invention

[0042] Figure 5 1 is a schematic diagram of a feature fusion network according to a preferred embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of uncertainty extraction in a preferred embodiment of the present invention;

[0044] Figure 7 1 is a schematic diagram of a viewpoint decoding network according to a preferred embodiment of the present invention;

[0045] Figure 8 Schematic diagram of the overall neural network of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following describes several preferred embodiments of the present invention with reference to the accompanying drawings to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0047] In the drawings, components with identical structures are denoted by the same reference numerals, and components with similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrary and are not limited by the present invention. For clarity, the thickness of components in some places in the drawings is appropriately exaggerated.

[0048] Existing viewpoint estimation methods only output a predicted gaze point coordinate for a given sample, not the model's degree of certainty about the sample. This results in the same degree of certainty for a low-quality sample with significant noise as for a high-quality sample without noise, leading to low reliability in applications based on these methods.

[0049] Aiming at this field and taking the above shortcomings into consideration, the present invention proposes a viewpoint estimation method based on uncertainty and an uncertainty extraction network, which outputs the uncertainty of the model for the sample to indicate the degree of certainty of the model's predicted value of the viewpoint coordinates of the sample, and can output the confidence of the predicted value while ensuring accuracy.

[0050] like Figure 1 As shown, an embodiment of the present invention provides a viewpoint estimation method based on uncertainty, comprising the following steps:

[0051] S101: Obtain a data set and preprocess the data set.

[0052] The acquired data set includes training data, which includes a scene image containing a subject and annotation information. According to the annotation information, the subject's face in a single-frame entire scene image is cropped, and a face position binary mask image is generated according to the position of the subject's face in the entire scene image.

[0053] The following sub-steps are included:

[0054] S1011: cropping the face image in the scene image according to the annotation information of the rectangular frame surrounding the subject's face;

[0055] S1012: generating a face position binary mask image according to the position of the subject's face in the scene image;

[0056] S1013: resizing the scene image, the face image, and the face position binary mask image, and normalizing the image pixel values.

[0057] The sizes of the scene image, face image, and face position binary mask image are adjusted to 224×224, and the image pixel values ​​are normalized to the range [-1, 1] for subsequent processing.

[0058] S103: Construct a convolutional neural network, which includes a scene saliency feature extraction network, a line of sight feature extraction network, a feature fusion network, a temporal information extraction network and a viewpoint decoding network.

[0059] When building a convolutional neural network, the following sub-steps are included:

[0060] S1031: Construct a scene saliency feature extraction network. The scene saliency feature extraction network consists of a ResNet-50 convolutional neural network and two bottleneck layers.

[0061] The input data of the scene saliency feature extraction network is the preprocessed scene image and the face position binary mask image, which are superimposed along the channel direction. The above input data is first processed by the ResNet-50 convolutional neural network and then processed by two bottleneck layers in sequence. The output of the scene saliency feature extraction network is the scene saliency feature f s , the input dimension of the ResNet-50 convolutional neural network is 224×224×4.

[0062] S1032: Build a gaze feature extraction network. The gaze feature extraction network consists of a ResNet-50 convolutional neural network and two bottleneck layers.

[0063] The input data of the gaze feature extraction network is the preprocessed face image data. The input data is first processed by the ResNet-50 convolutional neural network and then processed by two bottleneck layers in sequence. The output of the gaze feature extraction network is the gaze feature f g The input dimension of the ResNet-50 convolutional neural network is 224×224×3. The input dimension of the ResNet-50 convolutional neural network used in the gaze feature extraction network is different from that of the scene saliency feature extraction network. The two bottleneck layers used in the gaze feature extraction network are the same as those in the scene saliency feature extraction network.

[0064] S1033: Construct a feature fusion network. The feature fusion network consists of two convolutional layers, two batch normalization layers, and a ReLU activation function.

[0065] The input of the feature fusion network is the scene saliency feature f s With the sight feature f g The feature vector after superposition along the channel direction, the output of the feature fusion network is the fusion feature f f .

[0066] S1034: Construct an uncertainty extraction network. The uncertainty extraction network consists of two pooling layers, a linear layer, and a Sigmoid activation function. The output of the uncertainty extraction network is uncertainty.

[0067] S1035: Construct a viewpoint decoding network. The viewpoint decoding network consists of two pooling layers, a linear layer, and a Sigmoid activation function. The output of the viewpoint decoding network is the predicted viewpoint position coordinates.

[0068] S105: Constructing a total neural network. The total neural network outputs the predicted viewpoint position coordinates and their uncertainty.

[0069] The total neural network includes scene saliency feature extraction network, sight feature extraction network, feature fusion network, uncertainty extraction network and viewpoint decoding network. The total neural network transforms the feature fusion vector f f Input uncertainty extraction network and viewpoint decoding network respectively to obtain uncertainty And the viewpoint coordinate prediction value

[0070] S107: Design a loss function to minimize the loss function of the overall neural network model.

[0071] The loss function designed by the present invention is:

[0072]

[0073] Among them, i is the sample number, n is the number of samples, y i is the true value of the viewpoint coordinate label of the i-th sample, is the viewpoint coordinate prediction value of the total neural network for the i-th sample, is the uncertainty of the i-th sample of the total neural network output, y i and The Euclidean norm of .

[0074] S109: According to the supervised learning strategy network, the loss value between the viewpoint coordinate prediction value and the true value of the viewpoint coordinate label is calculated according to the loss function, and the gradient back propagation algorithm is used to complete the training of the total neural network model.

[0075] S111: Using the trained total neural network, perform viewpoint estimation on the test image data and output uncertainty.

[0076] The existing viewpoint estimation method can only output the predicted value but not the uncertainty of the predicted value, which lacks interpretability. The uncertainty extraction network proposed in this invention takes the fusion feature as input. For a given scene picture containing a person and surrounding information, this method can output the coordinates of the person's gaze point and the uncertainty. At the same time, the invention adds a loss function to the designed loss function. Realize the optimization of uncertainty extraction network.

[0077] The present invention will be described in detail below in conjunction with the preferred embodiments of the present invention.

[0078] In view of the problem in the prior art that for a given sample, only the predicted value of the gaze point coordinates can be output, but the degree of certainty of the model for the current sample cannot be output, the present invention proposes a viewpoint estimation method based on uncertainty to solve the problem that the viewpoint estimation method in the prior art cannot output uncertainty. This method first selects a neural network as the basic framework, and the training data consists of a scene image and annotation containing the subject. According to the annotation information, the subject's face in a single frame of the entire scene image is cropped, and a face position binary mask image is generated according to the position of the subject's face in the entire scene image. The cropped face image is then input into the sight line encoding module to obtain the sight line feature, and the face position mask image and the scene image are superimposed along the channel direction and input into the scene saliency encoding module to obtain the scene saliency feature. The sight line feature and the scene saliency feature are then superimposed and input into the feature fusion module to obtain the fused feature. The fused feature is input into the uncertainty encoding module and the viewpoint coordinate decoding module to obtain the viewpoint position prediction coordinates and their uncertainty, respectively, as shown in FIG. Figure 8 shown.

[0079] The present invention provides a viewpoint estimation method based on uncertainty, which includes:

[0080] Step 1: Preprocess the dataset.

[0081] A dataset is obtained, which consists of images and corresponding annotation information. First, the face image in the scene image is cropped according to the annotation information of the subject's face circumscribed rectangular frame, and a face position binary mask image is generated according to the position of the face in the scene image; the size of the scene image, face image and face position binary mask image is adjusted to 224×224, and the image pixel values ​​are normalized to the range [-1, 1].

[0082] Step 2: Construct a convolutional neural network, which includes a scene saliency feature extraction network, a line of sight feature extraction network, a feature fusion network, a temporal information extraction network, and a viewpoint decoding network.

[0083] 1) Construct a scene saliency feature extraction network. The scene saliency feature extraction network consists of a ResNet-50 and two bottleneck layers. The scene saliency feature extraction network is as follows: Figure 2 As shown, the bottleneck layer is Figure 4 Its input is the data obtained by superimposing the preprocessed scene image and the binary mask image along the channel direction, and the output is the scene saliency feature.

[0084] 2) Construct a sight feature extraction network. The sight feature extraction network consists of a ResNet-50 and two bottleneck layers. Figure 3 As shown, the bottleneck layer is Figure 4Its input is pre-processed face image data, and its output is gaze features.

[0085] 3) Construct a feature fusion network. The feature fusion network consists of two convolutional layers, two batch normalization layers and a Relu activation function. The feature fusion network is as follows: Figure 5 Its input is the feature vector of the superposition of scene saliency features and line of sight features along the channel direction, and its output is the fusion feature.

[0086] 4) Construct uncertainty extraction network. The uncertainty extraction network consists of two pooling layers, a linear layer, and a Sigmoid activation function. The output is uncertainty. The uncertainty extraction network is as follows: Figure 6 shown.

[0087] 5) Construct the viewpoint decoding network. The viewpoint decoding network consists of two pooling layers, a linear layer, and a Sigmoid activation function. The output is the predicted coordinates of the viewpoint position. The uncertainty extraction network is as follows: Figure 7 shown.

[0088] Step 3: Construct the total neural network. Use the saliency feature extraction network and the sight feature extraction network in step 2 to extract the scene feature vector f from the preprocessed data. s and the line of sight feature vector f g ; The feature vector f s and f g Input into the feature fusion network to obtain the feature fusion vector f f , and then the feature fusion vector f f Input uncertainty extraction network and viewpoint decoding network respectively to obtain uncertainty And the viewpoint coordinate prediction value The total neural network is as follows Figure 8 shown.

[0089] Step 4: Design the loss function. i is the sample number, n is the number of samples in a batch, is the viewpoint coordinate prediction value of the total neural network for the i-th sample, y i is the true value of the viewpoint coordinate label of the i-th sample, is the uncertainty of the total neural network output sample i, y i and The Euclidean norm of , the loss function can be expressed as:

[0090]

[0091] The goal of the entire model is to minimize the loss function in the above formula.

[0092] Step 5: According to the supervised learning strategy network, the loss value between the viewpoint coordinate prediction value and the viewpoint coordinate label true value is calculated according to the loss function in formula (2), and the gradient back propagation algorithm is used to complete the training of the model.

[0093] Step 6: Use the trained total neural network to estimate the viewpoint of the test image data and output the uncertainty.

[0094] The uncertainty extraction network proposed in this paper takes the fusion feature as input and outputs uncertainty. The loss function designed is added with The uncertainty extraction network can be optimized. Given a scene image containing a person and surrounding information, this method can output the coordinates of the person's gaze point and the uncertainty.

[0095] The preferred embodiments of the present invention have been described in detail above. It should be understood that numerous modifications and variations based on the concepts of the present invention are possible without inventive effort by those skilled in the art. Therefore, any technical solution that can be derived by one skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A viewpoint estimation method based on uncertainty, characterized in that: The method comprises the following steps: S101: Acquire a data set and preprocess the data set; S103: Constructing a convolutional neural network, wherein the convolutional neural network includes a scene saliency feature extraction network, a line of sight feature extraction network, a feature fusion network, an uncertainty extraction network, and a viewpoint decoding network; S105: constructing a total neural network, wherein the total neural network realizes output of viewpoint position prediction coordinates and their uncertainty; S107: Designing a loss function to minimize the loss function of the total neural network model; S109: According to the supervised learning strategy network, the loss value between the viewpoint coordinate prediction value and the viewpoint coordinate label true value is calculated according to the loss function, and the gradient back propagation algorithm is used to complete the training of the overall neural network model; S111: using the trained total neural network to perform viewpoint estimation on the test image data and output uncertainty; in, In step S101, the data set includes training data, the training data including a scene image containing a subject and annotation information, the subject's face in a single frame of the entire scene image is cropped according to the annotation information, and a face position binary mask image is generated according to the position of the subject's face in the entire scene image; The input data of the scene saliency feature extraction network is the data obtained by superimposing the preprocessed scene image and the face position binary mask image along the channel direction. The input data is first processed by the first ResNet-50 convolutional neural network and then processed by two bottleneck layers in sequence. The output of the scene saliency feature extraction network is the scene saliency feature. , the input dimension of the first ResNet-50 convolutional neural network is 224×224×4; The input data of the sight feature extraction network is the pre-processed face image data. The input data is first processed by the second ResNet-50 convolutional neural network and then processed by two bottleneck layers in sequence. The output of the sight feature extraction network is the sight feature. , the input dimension of the second ResNet-50 convolutional neural network is 224×224×3; The input of the feature fusion network is the scene saliency feature With the sight feature The feature vector after superposition along the channel direction, the output of the feature fusion network is the feature fusion vector .

2. The method according to claim 1, wherein The pre-processing of the data set in step S101 includes the following sub-steps: S1011: cropping the face image in the scene image according to the annotation information of the rectangular frame circumscribing the face of the subject; S1012: generating a face position binary mask image according to the position of the subject's face in the scene image; S1013: Adjust the sizes of the scene image, the face image, and the face position binary mask image, and normalize the image pixel values.

3. The method according to claim 2, wherein In step S1013, the sizes of the scene image, the face image and the face position binary mask image are adjusted to 224×224, and the image pixel values ​​are normalized to the range of [-1, 1].

4. The method according to claim 2, wherein The step S103 includes the following sub-steps: S1031: Constructing the scene saliency feature extraction network, wherein the scene saliency feature extraction network is composed of a first ResNet-50 convolutional neural network and two bottleneck layers; S1032: Constructing the sight feature extraction network, wherein the sight feature extraction network is composed of a second ResNet-50 convolutional neural network and two bottleneck layers; S1033: Constructing the feature fusion network, wherein the feature fusion network is composed of two convolutional layers, two batch normalization layers and a ReLU activation function; S1034: Constructing an uncertainty extraction network, wherein the uncertainty extraction network is composed of two pooling layers, a linear layer, and a Sigmoid activation function, and the output of the uncertainty extraction network is uncertainty; S1035: Construct the viewpoint decoding network, which is composed of two pooling layers, a linear layer, and a Sigmoid activation function. The output of the viewpoint decoding network is the viewpoint position prediction coordinates.

5. The method according to claim 4, wherein The overall neural network includes the scene saliency feature extraction network, the sight feature extraction network, the feature fusion network, the uncertainty extraction network and the viewpoint decoding network. The overall neural network transforms the feature fusion vector Input the uncertainty extraction network and the viewpoint decoding network respectively to obtain the uncertainty and the viewpoint coordinate prediction value .

6. The method according to claim 1, wherein The loss function is: in, is the sample serial number, is the number of samples, For the The true value of the viewpoint coordinate label of samples, The total neural network is The viewpoint coordinate prediction value of samples, is the total neural network output The uncertainty of the sample, for and The Euclidean norm of .

Images