Target-oriented glancing path prediction method

Abstract

The invention discloses a target-oriented glancing path prediction method, and belongs to the field of path prediction. The method comprises the following steps: extracting target semantic information and multi-level visual information of a scene image; fusing the target semantic information and the multi-level visual information into joint feature information; inputting the query information and the joint feature information of all fixation points into a glancing path decoder, and predicting hidden state information of all fixation points in a parallel mode; inputting the hidden state information of all fixation points into a glancing path head, and converting the hidden state information into parameters such as a coordinate value X, a coordinate value Y, duration T and a termination probability value V; and finally, traversing all fixation points, if the termination probability value of the current fixation point is less than 0.5, adding the coordinate value and the duration of the fixation point into the glancing path, otherwise, stopping traversing. The model provided by the invention can generate a more real and reliable glancing path in a target-oriented scene.

Description

Technical Field

[0001] The present invention relates to the technical field of path prediction, and in particular to a target-oriented scanning path prediction method. Background Art

[0002] Gaze path prediction is a process of dynamically predicting human attention. With the development of technologies such as augmented reality (AR) and virtual reality (VR), the field of glance path prediction has received increasing attention in recent years because it can predict human attention and can be applied to viewport rendering, fatigue detection, and image / video quality assessment.

[0003] Unlike free viewing, goal-oriented scan path prediction is task-driven and purposeful. It is often relatively complex because it needs to consider and model the semantic information of the target and the scene image information. Currently, there are few studies on goal-oriented scan path prediction, but this scenario is very common in reality (for example, parents may search for their children in a complex crowd in an amusement park). Therefore, the study of goal-oriented scan path prediction is of great significance to enriching and improving the theory of scan path prediction.

[0004] Early goal-oriented scanning path prediction methods often use semantic guidance graphs to encode target information. However, these methods often rely on a trained target detection model to encode the target's text information, and are often limited to the categories included in the training set of the target detection model, which is often relatively small. When it is necessary to predict unseen categories, it is necessary to first collect a data set containing the corresponding category and then retrain the target detection model. Recently, some methods have proposed to use visual-semantic matching (i.e., using the good expressive power of pre-trained language models and visual models) to predict goal-oriented scanning paths. However, the existing methods often use independent visual encoders and semantic encoders, and do not fully consider the alignment relationship between visual-semantic information. This will result in a large distance difference between the same object in different feature spaces, and often requires a large amount of training data to learn the correspondence between the two expressions. Therefore, the existing methods based on visual-semantic matching cannot produce good results for unseen categories, and the results for seen categories with sufficient training sets are often poor. Summary of the invention

[0005] The present invention provides a target-oriented scanning path prediction method, which can generate a more realistic and reliable scanning path in a target-oriented scenario.

[0006] An embodiment of the present invention provides a target-oriented scanning path prediction method, comprising the following steps:

[0007] Step 1: Use CLIP (Contrastive Language-Image Pre-training) text / visual encoder to extract target semantic information;

[0008] Step 2, using CLIP visual encoder to extract multi-level visual information of the scene image;

[0009] Step 3: Using a hierarchical visual-semantic feature fusion module, the target semantic information and multi-level visual information are fused into joint feature information;

[0010] Step 4: input the query information and joint feature information of all fixation points into the scan path decoder, and predict the hidden state information of all fixation points in parallel;

[0011] Step 5, input the hidden state information of all fixation points into a scanning path head, and convert them into parameters of coordinate value X, coordinate value Y, duration T and termination probability value V;

[0012] Step 6, traverse all fixation points. If the termination probability value of the current fixation point is less than or equal to 0.5, add the coordinate value and duration of the fixation point to the scanning path. Otherwise, stop traversal.

[0013] Optionally, in one embodiment of the present invention, in step 1, the target semantic information is extracted using the CLIP text / visual encoder as follows:

[0014] F target =CLIP(target)

[0015] Where target is the textual cue or visual cue representing the target, CLIP represents the corresponding CLIP text / visual encoder, and F target Embedding that represents the semantic information of the target.

[0016] Optionally, in one embodiment of the present invention, in step 2, the multi-level visual information of the scene image is extracted using the CLIP visual encoder:

[0017] A1,A2,...,A N =CLIP visual (image)

[0018] Among them, image represents the two-dimensional plane image of the scene, CLIP visual represents the CLIP visual encoder, A1, A2, ..., A NThey represent the feature map embeddings output by N different feature layers in the CLIP visual encoder.

[0019] Optionally, in one embodiment of the present invention, step 3 specifically includes the following steps:

[0020] Step 301: The target semantic information F extracted in step 1 is target After N independent linear mappings, the dimension is transformed into the internal dimension D of the subsequent Transformer architecture, and N new target feature vectors are embedded in F. target1 ,F target2 ,...,F targetN ;

[0021] Step 302: The visual feature graphs A1, A2, ..., A of the multi-level visual information extracted in step 2 are N After N independent linear mappings, the dimension is transformed into the internal dimension D of the subsequent Transformer architecture, and N new visual feature maps A1', A'2, ..., A' are obtained. N ;

[0022] Step 303: embed the first target feature vector into F target1 And the last visual feature map A' N After fusion through a Film structure, the fused feature map F is obtained joint1 ;

[0023] Step 304: obtain the fusion feature map F joint1 Add a two-dimensional position encoding to get a new feature map F j ' oint1 ;

[0024] Step 305: transform the feature map F j ' oint1 After a standard Transformer encoding layer, the fused features are made aware of the corresponding relationship between each other, and a new feature map F is obtained. j ” oint1 ;

[0025] Step 306: Get the feature map F″ joint1 and the penultimate visual feature map A' N-1 Directly add them together to get the new feature map F″′ joint1 ;

[0026] Step 307: The second target feature vector F target2 Replace the first target feature vector F in step 303 target1 Then use the obtained feature map F″′ joint1Replace the last visual feature map A' in step 303 N And obtain a new fusion feature map F through the new Film structure joint2 And so on, repeat steps 303 to 307 until the final joint feature information F″ is obtained. jointN .

[0027] Optionally, in one embodiment of the present invention, in step 4, the scan path decoder generates the hidden states of all gaze points of the scan path in parallel.

[0028] Optionally, in one embodiment of the present invention, step 4 specifically includes the following steps:

[0029] Step 401, all gaze point query information Q0, Q1, ..., Q N-1 The value of is initialized to a vector of all zeros, and then the dimension is mapped to the internal dimension D of the Transformer architecture in the scan path decoder to obtain the new query information representation Q′0,Q′1,...,Q' N-1 ;

[0030] Step 402, query information Q'0, Q'1, ..., Q' for the mapped gaze points N-1 Add one-dimensional position coding information to obtain new query information Q″0,Q″1,...,Q″ N-1 ;

[0031] Step 403, query information Q″0, Q″1, ..., Q″ N-1 And the fused joint feature information F″ jointN Input M Transformer decoding layers together, in this process, all mask information in the Transformer decoding layer is set to empty, and finally the hidden states Z0, Z1, ..., Z of all gaze points are obtained N-1 .

[0032] Optionally, in one embodiment of the present invention, in step 5, the hidden state information of all gaze points is input into a scanning path header, and the formula for converting into coordinate value X, coordinate value Y, duration T and termination probability value V is:

[0033] {X i ,Y i ,T i ,V i} = f head (Z i θ head ),i∈[0,1,...,N-1]

[0034] Among them, f headrepresents the scanning path head, which consists of four independent one-layer linear fully connected networks, Z i represents the hidden state of the i-th fixation point, θ head Represents the parameters of the scanning path head, X i represents the X-axis coordinate value of the i-th gaze point, Y i represents the Y-axis coordinate value of the i-th gaze point, T i represents the duration of the i-th fixation point, V i Represents the termination probability value of the i-th fixation point.

[0035] Optionally, in one embodiment of the present invention, in step 6, in the process of traversing all the gaze points, the specific steps include:

[0036] The scanning path queue needs to be set to empty first, and then traverse from fixation point i=0 to fixation point i=N-1 in sequence;

[0037] Check the termination probability value V of the current fixation point i ,i∈[0,1,...,N-1], if the termination probability value V i ≤0.5, the coordinate value X of the gaze point i , coordinate value Y i and duration T i Add it to the scanning path queue; otherwise, stop traversal. Eventually, all the fixation point information in the scanning path queue constitutes a complete scanning path.

[0038] Optionally, in one embodiment of the present invention, after step 3, the method further comprises the following steps:

[0039] Before using the scan path dataset for training, image segmentation is used as a proxy task for pre-training. The proxy task converts the joint feature information obtained in step 3 into an instance segmentation map of the specified target in the scene image, and obtains a better initial feature fusion module through learning. In order to perform pre-training on the proxy task, a segmentation head is added after the hierarchical visual-semantic feature fusion module, and training is performed on the corresponding image segmentation dataset. During the pre-training process, the parameters of the CLIP text / visual encoder, the scan path decoder, and the scan path head are not involved in the training, and the hierarchical visual-semantic feature fusion module and the segmentation head are involved in the training. The pre-training process satisfies the formula:

[0040] Image seg =SegHead(F joint )

[0041] Among them, Image segIt represents the segmentation image of the target in the scene image, SegHead represents the segmentation head, and it consists of two upsampling layers.

[0042] Optionally, in one embodiment of the present invention, when the model is trained on the corresponding scanning path data set, the parameters of the CLIP text / visual encoder and the segmentation head are not involved in the training, and the hierarchical visual-semantic feature fusion module, the scanning path decoder and the scanning path head are involved in the training;

[0043] The loss function used in the training process is the position time loss L xyt , effectiveness loss L val and Soft-DTW loss L Soft-DTW The sum of the three loss functions satisfies the formula:

[0044]

[0045] Among them, M represents the batch size, and the position time loss uses L1 regression loss as the loss function;

[0046] Position-time loss function for the jth scanning path Expressed as:

[0047]

[0048] Among them, l j represents the actual length of the j-th scanning path, Respectively represent the actual coordinate value X, coordinate value Y and duration T of the i-th fixation point in the j-th scanning path, Respectively represent the predicted coordinate value X, coordinate value Y and duration T of the i-th fixation point in the j-th scanning path;

[0049] The effectiveness loss uses the binary cross entropy loss function. The effectiveness loss function for the j-th scanning path is Expressed as:

[0050]

[0051] Among them, α is a balancing factor used to balance the weights of the end point and the middle point. and They represent the true value and predicted value of the probability value of the end point of the i-th fixation point in the j-th scanning path, respectively, and L is the maximum length of the scanning path;

[0052] The Soft-DTW loss function for the j-th scanning path is expressed as:

[0053]

[0054] Among them, ρ and They represent the true j-th scanning path and the predicted j-th scanning path respectively, and Soft-DTW represents the Soft-DTW loss function.

[0055] The target-oriented scanning path prediction method of the embodiment of the present invention has the following beneficial effects:

[0056] 1. The present invention uses CLIP to extract text information and visual information. Since CLIP has been trained in an aligned manner on a large number of data sets, the present invention can extract more aligned initial features;

[0057] 2. The present invention extracts multi-level visual information and fuses it with the target semantic information, which helps to consider visual information of different scales and can focus on both large and small targets at the same time;

[0058] 3. The present invention designs a novel feature fusion module. A Transformer encoder is added, which helps to consider more global visual information; two-dimensional position encoding information is added, which helps to locate objects more accurately when predicting the gaze point;

[0059] 4. The present invention adopts a parallel Transformer decoder structure to solve the sequence generation problem, which can pay attention to the relationship between all gaze points, consider the relationship between the gaze points and the complete image sequence, and generate a complete scanning path more quickly.

[0060] 5. In addition to predicting the location, duration and other information of the fixation point, the present invention also predicts the probability value of the fixation point being the end point, which helps to determine when the scanning path should stop.

[0061] 6. The present invention uses image segmentation as a proxy task and can perform pre-training on a large-scale image segmentation dataset, so that the hierarchical visual-semantic fusion module can better learn the correspondence between text and image. This helps to solve the problem of few categories and small number of scanning path datasets, thereby improving the effect of scanning path prediction.

[0062] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0064] Figure 1 A flowchart of a target-oriented scanning path prediction method provided by an embodiment of the present invention;

[0065] Figure 2 A flowchart of a target-oriented scanning path prediction method provided by an embodiment of the present invention;

[0066] Figure 3 A detailed algorithm flow chart of the hierarchical visual-semantic fusion structure provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0068] Figure 1 A flowchart of a target-oriented scanning path prediction method provided by an embodiment of the present invention.

[0069] like Figure 1 As shown, the target-oriented scanning path prediction method includes the following steps:

[0070] Step 1: Use CLIP text / visual encoder to extract target semantic information.

[0071] In an embodiment of the present invention, the implementation process of step 1 is: passing the target text / visual prompt through the CLIP text / visual encoder, and extracting the final layer output of the CLIP text / visual encoder for the target encoding information. The process can be expressed as:

[0072] F target =CLIP(target)

[0073] Where target is the textual cue or visual cue representing the target, CLIP represents the corresponding CLIP text / visual encoder, and F target The semantic information embedding representing the target, d represents the real number R target Matrix of order.

[0074] In step 1, the semantic information of the target can be extracted from either textual cues or visual cues, wherein the textual cues are extracted using the CLIP text encoder, and the visual cues are extracted using the CLIP visual encoder.

[0075] Step 2: Use the CLIP visual encoder to extract multi-level visual information of the scene image.

[0076] In an embodiment of the present invention, the implementation process of step 2 is: firstly scale the size of the input image so that the width of the feature map extracted after the CLIP visual encoder is W and the height is H, and then use the CLIP visual encoder to extract the multi-level visual information of the scene image. The process can be expressed as:

[0077] A1,A2,...,A N =CLIP visual (image)

[0078] Among them, image represents the two-dimensional plane image of the scene, CLIP visual represents the CLIP visual encoder, A1, A2, ..., A N They represent the feature map embeddings output by N different feature layers in the CLIP visual encoder. In the specific implementation process, H×W×d representing a real number R image Matrix of order.

[0079] Step 3: Use a hierarchical visual-semantic feature fusion module to fuse the target semantic information and multi-level visual information into joint feature information.

[0080] In an embodiment of the present invention, the implementation process of step 3 is:

[0081] Step 301: Transform the target semantic information F target After N independent linear mappings, its dimension is transformed into the internal dimension D of the subsequent Transformer architecture, and N new target feature vectors are embedded in F. target1 ,F target2 ,...,F targetN , R D represents a D-order matrix of real numbers R;

[0082] Step 302: embed the extracted visual feature map into A1, A2, ..., A N After N independent linear mappings, its dimension is transformed into the internal dimension D of the subsequent Transformer architecture to obtain N new visual feature maps A1', A'2, ..., A' N ∈R H×W×D , R H×W×D A matrix of order H×W×D representing the real number R;

[0083] Step 303: embed the first target feature into F target1 And the last visual feature map A' N After fusion through a Film structure, the fused feature map F is obtained joint1 ;

[0084] Step 304: The obtained fusion feature map F joint1 Add a two-dimensional position encoding to get a new feature map F′ joint1 ;

[0085] Step 305: transform the feature graph vector F′ joint1 After a standard Transformer encoding layer, the fused features can perceive the corresponding relationship between each other and obtain a new feature map F″ joint1 ;

[0086] Step 306: Get the feature map F″ joint1 and the penultimate visual feature map A' N-1 Directly add them together to get the new feature map F″′ joint1 ;

[0087] Step 307: The second target feature vector F target2 Replace the first target feature vector F in step 303 target1 And use the obtained feature map F″′ joint1 Replace the last visual feature map A' in step 303 N , and then pass them through the new Film structure to get a new fusion feature map F joint2 And so on, repeat steps 303 to 307 until the final joint feature information F is obtained. j ” ointN The present invention then uses F joint To represent F j ” ointN , that is, F joint =F j ” ointN .

[0088] Step 4: Input the query information and joint feature information of all fixation points into the scan path decoder together, and predict the hidden state information of all fixation points in parallel.

[0089] In an embodiment of the present invention, in step 4, the scan path decoder generates the hidden states of all fixation points of the scan path in parallel.

[0090] In an embodiment of the present invention, the implementation process of step 4 is:

[0091] Step 401: query information of all gaze points Q0, Q1, ..., Q N-1 The value of is initialized to a zero vector, and then its dimension is mapped to the internal dimension D of the Transformer architecture in the scan path decoder to obtain the new query information representation Q′0,Q′1,...,Q' N-1 ∈RD , R D A D-order matrix representing real numbers R.

[0092] Step 402: Query information Q′0, Q′1, ..., Q′ for the mapped gaze points N-1 Add one-dimensional position coding information to obtain new query information Q″0,Q″1,...,Q” N-1 .

[0093] Step 403: query information Q″0, Q″1, ..., Q″ N-1 and the fused visual-semantic information Input M Transformer decoding layers together. In this process, all mask information in the Transformer decoding layer needs to be set to empty. Finally, the hidden states Z0, Z1, ..., Z of all gaze points can be obtained. N-1 ∈R D , R D A D-order matrix representing real numbers R.

[0094] Step 5: Input the hidden state information of all fixation points into a scanning path head and convert them into parameters of coordinate value X, coordinate value Y, duration T and termination probability value V.

[0095] In an embodiment of the present invention, the calculation formula of step 5 is expressed as:

[0096] {X i ,Y i ,T i ,V i} = f head (Z i θ head ),i∈[0,1,...,N-1]

[0097] Among them, f head represents the scanning path head, which consists of four independent one-layer linear fully connected networks, Z i represents the hidden state of the i-th fixation point, θ head Represents the parameters of the scanning path head, X i represents the X-axis coordinate value of the i-th gaze point, Y i represents the Y-axis coordinate value of the i-th gaze point, T i represents the duration of the i-th fixation point, V i Represents the termination probability value of the i-th fixation point.

[0098] Step 6, traverse all fixation points. If the termination probability value of the current fixation point is less than or equal to 0.5, add the coordinate value and duration of the fixation point to the scanning path. Otherwise, stop traversal.

[0099] In one embodiment of the present invention, in step 6, the scanning path queue needs to be set to empty first, and then traverse from the fixation point i=0 to the fixation point i=N-1 in sequence. Check the termination probability value V of the current fixation point i ,i∈[0,1,...,N-1], if the probability value V i ≤0.5, the coordinate value X of the gaze point i , coordinate value Y i and duration T i Add it to the scanning path queue; otherwise, stop traversal. Finally, all the fixation point information in the scanning path queue constitutes a complete scanning path.

[0100] It should be noted that, for the convenience of training, the lengths of all the scanning paths in the training set need to be unified to L. For scanning paths whose lengths are shorter than L, they should be padded so that their lengths are equal to L.

[0101] In one embodiment of the present invention, before the model is trained using a dataset of scanning paths, the model may use image segmentation as a proxy task.

[0102] The proxy task can use the joint feature information F obtained in step 3 joint It is converted into an instance segmentation image of the target in the scene image, and a better initial feature fusion module can be learned. In order to perform pre-training on this proxy task, it is necessary to add a segmentation head after the hierarchical visual-semantic feature fusion module and train it on the corresponding image segmentation dataset (such as the PhraseCut dataset). During the pre-training process, the parameters of the CLIP text / visual encoder, the scan path decoder, and the scan path head are not trained, and the hierarchical visual-semantic feature fusion module and the segmentation head are trained. The process satisfies the formula:

[0103] Image seg =SegHead(F joint )

[0104] Among them, Image seg It represents the segmentation image of the target in the scene image, and SegHead represents the segmentation head, which consists of two upsampling layers.

[0105] In one embodiment of the present invention, during the training of the model using the scan path dataset, the parameters of the CLIP text / visual encoder and the segmentation head are not involved in the training, while the hierarchical visual-semantic feature fusion module, the scan path decoder and the scan path head are involved in the training. The loss function used in the training process is the position time loss L xyt , effectiveness loss L valand Soft-DTW loss L Soft-DTW The sum of the three loss functions satisfies the formula:

[0106]

[0107] Among them, M represents the batch size, the position time loss uses L1 regression loss as the loss function, and the position time loss function for the j-th scanning path is It can be expressed as:

[0108]

[0109] Among them l j Represents the actual length of the j-th scanning path. Respectively represent the actual coordinate value X, coordinate value Y and duration T of the i-th fixation point in the j-th scanning path, Respectively represent the predicted coordinate value X, coordinate value Y and duration T of the i-th fixation point in the j-th scanning path. The validity loss uses the binary cross entropy loss (negative log-likelihood loss) function. The validity loss function for the j-th scanning path is It can be expressed as:

[0110]

[0111] Among them, α is a balancing factor used to balance the weights of the end points and the intermediate points. This is because the number of intermediate points in a complete scanning path is often greater than the number of end points (usually 1). Here it is set to the ratio of the number of intermediate points to the number of end points. and Represent the true value and predicted value of the probability value of the end point of the i-th fixation point in the j-th scanning path. The Soft-DTW loss function is a differentiable variant of the DTW (Dynamic Time Warping) function. Adding this part can make up for the defect that the position time loss focuses too much on the distance between the fixation point sequences, and further considers the influence of the shape relationship between the two fixation point sequences (because different scanning paths to find the same object may contain different numbers of fixations). The Soft-DTW loss function for the j-th scanning path can be expressed as:

[0112]

[0113] where ρ and They represent the true j-th scanning path and the predicted j-th scanning path respectively, and Soft-DTW represents the Soft-DTW loss function.

[0114] like Figure 2 and Figure 3As shown, the present invention mainly comprises six parts, namely, CLIP text encoder, CLIP visual encoder, visual-semantic fusion module, scan path decoder, scan path head and segmentation head. The CLIP text encoder is responsible for extracting the semantic information of the target, the CLIP visual encoder is responsible for extracting the multi-scale visual information of the target, the visual-semantic fusion module is responsible for the fusion of the target semantic information and the multi-level visual information, the scan path decoder is responsible for integrating the gaze point query information and the fused visual-semantic information and generating the hidden state of the gaze point, the scan path head is responsible for converting the hidden state of the gaze point into detailed information such as the location, duration and validity of the gaze point, and the segmentation head is used to take image segmentation as a proxy task and perform pre-training.

[0115] The target-oriented scanning path prediction method of the present invention is described in detail below through a specific embodiment. The target-oriented scanning path prediction method comprises the following steps:

[0116] Step S1: Use the CLIP text / visual encoder to extract the semantic information of the target.

[0117] Specifically, the present invention uses the CLIP text / visual encoder based on ViT-L / 14@336px to extract the final encoding output of the CLIP text / visual encoder for the target semantic information. The process can be expressed as:

[0118] F target =CLIP(target)

[0119] Where target is the text / visual cue representing the target, CLIP represents the corresponding CLIP text / visual encoder, and F target Represents the semantic information embedding of the target, F target ∈R 768 , R 768 Represents a 768-order matrix of real numbers R.

[0120] Step S2: Use the CLIP visual encoder to extract multi-level visual information of the scene image.

[0121] First, the size of the input image is scaled so that the width of the input image is 448 and the height is 280. Then it is passed through the CLIP visual encoder based on ViT-L / 14@336px, and the width and height of the extracted feature map are 32 and 20, and then the CLIP visual encoder is used to extract the multi-level visual information of the scene image. There is the following relationship:

[0122] A1, A2, A3=CLIP visual (image)

[0123] Among them, image represents the two-dimensional plane image of the scene, CLIP visual A1, A2, A3 represent the feature maps of the output of the three different feature layers in the CLIP visual encoder. In the specific implementation process, the invention extracts the output of the 6th, 14th, and 18th feature maps of the CLIP visual encoder based on ViT-L / 14@336px. A1, A2, A3∈R 20×32×1024 , R 20×32×1024 A 20×32×1024 matrix representing the real number R.

[0124] Step S3: using a hierarchical visual-semantic feature fusion module, the target semantic information and multi-level visual information are fused into joint feature information.

[0125] Specifically, the process of fusing target semantic information and multi-level visual information into joint feature information includes the following steps:

[0126] S301, target semantic information F target After three independent linear mappings, its dimension is converted to the internal dimension D = 1024 of the subsequent Transformer architecture, and three new target feature vectors are embedded in F target1 ,F target2 ,F target3 ∈R 1024 , R 1024 Represents a 1024-order matrix of real numbers R;

[0127] S302: The extracted visual feature maps A1, A2, and A3 are transformed into the internal dimension D = 1024 of the subsequent Transformer architecture through three independent linear mappings to obtain three new visual feature maps A′1, A′2, and A′3∈R 20 ×32×1024 , R 20×32×1024 A 20×32×1024-order matrix representing the real number R;

[0128] S303, embed the first target feature into F target1 The last visual feature map A'3 is fused through a Film structure to obtain the fused feature map F joint1 ;

[0129] S304, the obtained fusion feature map F joint1 Add a two-dimensional position encoding to get a new feature map F′ joint1 ;

[0130] S305, feature graph vector F′ joint1After passing through a standard Transformer encoding layer, the obtained fusion features can perceive the corresponding relationship between each other in the context, and obtain a new feature map F″ joint1 ;

[0131] S306, the obtained feature map F″ joint1 Directly add it to the penultimate visual feature map A'2 to get the new feature map F' joint1 ;

[0132] S307, the second target feature vector F target2 Replace the first target feature vector F in step S303 target1 Then use the obtained feature map F″′ joint1 Replace the last visual feature map A'3 in step S303 and pass them through the new Film structure to obtain a new fused feature map F joint2 And so on, repeat steps S303-S307 until the final joint feature information F″′ is obtained joint3 The present invention then uses F joint To represent F″ joint3 , that is, F joint =F″ joint3 .

[0133] Step S4: input the query information and joint feature information of all gaze points into the scan path decoder, and predict the hidden state information of all gaze points in parallel.

[0134] Specifically, predicting the hidden state information of all gaze points in parallel includes the following steps:

[0135] S401, the values ​​of all gaze query information Q0, Q1, ..., Q6 are initialized to all zero vectors, and then their dimensions are mapped to the internal dimension D = 1024 of the Transformer architecture in the scan path decoder to obtain new query information expressions Q′0, Q′1, ..., Q′6∈R D＝1024 , R 1024 Represents a 1024-order matrix of real numbers R.

[0136] S402, adding one-dimensional position coding information to the mapped gaze point query information Q′0, Q′1, ..., Q′6 to obtain new query information Q″0, Q″1, ..., Q″6.

[0137] S403, query information Q″0, Q″1, ..., Q″6 and fused visual-semantic information F jointInput M = 6 Transformer decoding layers together. In this process, all mask information in the Transformer decoding layer needs to be set to empty. Finally, the hidden states of all gaze points Z0, Z1, ..., Z6∈R 1024 , R 1024 Represents a 1024-order matrix of real numbers R.

[0138] Step S5, input the hidden state information of all fixation points into a scanning path header, and convert it into parameters such as coordinate value X, coordinate value Y, duration T and termination probability value V;

[0139] Specifically, the hidden state information of all fixation points is input into a scanning path head, and converted into parameters such as coordinate value X, coordinate value Y, duration T and termination probability value V. The calculation formula is expressed as:

[0140] {X i ,Y i ,T i ,V i} = f head (Z i θ head ),i∈[0,1,...,6]

[0141] Among them, f head represents the scanning path head, which includes four independent one-layer linear fully connected networks, Z i represents the state of the i-th gaze point, θ head Represents the parameters of the scanning path head, X i represents the X-axis coordinate value of the i-th gaze point, Y i represents the Y-axis coordinate value of the i-th gaze point, T i represents the duration of the ith fixation, V i Represents the termination probability value of the i-th fixation point.

[0142] The scanning path head contains 4 independent one-layer linear fully connected networks. After the first layer, when calculating the mean X, mean Y and duration T of each fixation point, the present invention converts the fixation point hidden state into a vector with a dimension of 1 through a linear operation, and then uses the relu function to convert it into a corresponding value; when calculating the fixation point termination probability value V, the present invention converts the fixation point hidden state into a vector with a dimension of 2 through a linear operation after the first layer, and then uses the logsoftmax function to convert it into the probability of the middle point and the probability value of the termination point, and finally judges whether the current fixation point is the end point of the scanning path by judging which of the probability of the middle point and the probability of the termination point is greater.

[0143] Step S6, in the process of generating the scanning path, traverse all the fixation points, if the termination probability value of the current fixation point is less than or equal to 0.5, then add the coordinate value and duration of the fixation point to the scanning path, otherwise, stop traversing.

[0144] In one embodiment of the present invention, in step S6, the scanning path queue needs to be set to empty first, and then traverse from the fixation point i=0 to the fixation point i=6 in sequence. Check the termination probability value V of the current fixation point i ,i∈[0,1,...,6], if the probability value V i ≤0.5, the coordinate value X of the gaze point i , coordinate value Y i and duration T i Add it to the scanning path queue; otherwise, stop traversal. Finally, all the fixation point information in the scanning path queue constitutes a complete scanning path.

[0145] The present invention proposes a target-oriented scanning path prediction method, which first uses CLIP to extract the semantic information of the target and the multi-level visual information of the image scene, which is helpful to obtain more aligned initial features; multi-level visual information and target semantic information are used for fusion, which is helpful to pay attention to object information of different scales; a visual-semantic fusion module that integrates Transformer encoder and two-dimensional position encoding is used, which is helpful to pay attention to more global visual information and more accurate position information; a parallel structure Transformer decoder is used to generate a scanning path, which can fully consider the relationship between the fixation points, the relationship between the fixation points and the complete image, and generate a complete scanning path faster; a scanning path head is used to predict the position, time and validity of the fixation point, and the distribution information, time information and stop information of the fixation point in the scanning path can be more reasonably predicted; image segmentation is used as a proxy task, and the corresponding relationship between the image and the text can be better learned in the pre-training process. Through the reasonable combination of these modules, the present invention can generate a more realistic and reliable scanning path for the target-oriented scanning path prediction task.

[0146] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0147] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0148] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

Claims

1. A goal-oriented scanning path prediction method, characterized in that: The following steps are involved: Step 1: Use CLIP text / visual encoder to extract target semantic information; Step 2, using CLIP visual encoder to extract multi-level visual information of the scene image; Step 3: Using a hierarchical visual-semantic feature fusion module, the target semantic information and multi-level visual information are fused into joint feature information; Step 4: input the query information and joint feature information of all fixation points into the scan path decoder, and predict the hidden state information of all fixation points in parallel; Step 5, input the hidden state information of all fixation points into a scanning path head, and convert them into parameters of coordinate value X, coordinate value Y, duration T and termination probability value V; Step 6, traverse all fixation points. If the termination probability value of the current fixation point is less than or equal to 0.5, add the coordinate value and duration of the fixation point to the scanning path. Otherwise, stop traversal.

2. The target-oriented scanning path prediction method according to claim 1, characterized in that: In step 1, the CLIP text / visual encoder is used to extract the target semantic information as: F target =CLIP(target) Where target is the textual cue or visual cue representing the target, CLIP represents the corresponding CLIP text / visual encoder, and F target Embedding that represents the semantic information of the target.

3. The target-oriented scanning path prediction method according to claim 1, characterized in that: In step 2, the CLIP visual encoder is used to extract the multi-level visual information of the scene image as follows: A1,A2,...,A N =CLIP visual (image) Among them, image represents the two-dimensional plane image of the scene, CLIP visual represents the CLIP visual encoder, A1, A2, ..., A N They represent the feature map embeddings output by N different feature layers in the CLIP visual encoder.

4. The target-oriented scanning path prediction method according to claim 3, characterized in that: Step 3 specifically includes the following steps: Step 301: The target semantic information F extracted in step 1 is target After N independent linear mappings, the dimension is transformed into the internal dimension D of the subsequent Transformer architecture, and N new target feature vectors are embedded in F. target1 ,F target2 ,...,F targetN ; Step 302: The visual feature graphs A1, A2, ..., A of the multi-level visual information extracted in step 2 are N After N independent linear mappings, the dimension is transformed into the internal dimension D of the subsequent Transformer architecture, and N new visual feature maps A1', A'2, ..., A' are obtained. N ; Step 303: embed the first target feature vector into F target1 And the last visual feature map A' N After fusion through a Film structure, the fused feature map F is obtained joint1 ; Step 304: obtain the fusion feature map F joint1 Add a two-dimensional position encoding to get a new feature map F j ' oint1 ; Step 305: transform the feature map F j ' oint1 After a standard Transformer encoding layer, the fused features are made aware of the corresponding relationship between each other, and a new feature map F is obtained. j ” oint1 ; Step 306: Get the feature map F j ” oint1 and the penultimate visual feature map A' N-1 Directly add to get the new feature map F j ”' oint1 ; Step 307: The second target feature vector F target2 Replace the first target feature vector F in step 303 target1 Then use the obtained feature map F j ”' oint1 Replace the last visual feature map A' in step 303 N And obtain a new fusion feature map F through the new Film structure joint2 And so on, repeat steps 303 to 307 until the final joint feature information F is obtained. j ” ointN .

5. The target-oriented scanning path prediction method according to claim 1, characterized in that: In step 4, the glance path decoder generates the hidden states of all fixation points of the glance path in parallel.

6. The target-oriented scanning path prediction method according to claim 4, characterized in that: Step 4 specifically includes the following steps: Step 401, all gaze point query information Q0, Q1, ..., Q N-1 The value of is initialized to a vector of all zeros, and then the dimension is mapped to the internal dimension D of the Transformer architecture in the scan path decoder to obtain the new query information representation Q0', Q1', ..., Q' N-1 ; Step 402, query information Q0', Q1', ..., Q' for the mapped gaze points N-1 Add one-dimensional position coding information to obtain new query information Q0”, Q1”, ..., Q” N-1 ; Step 403, query information Q0", Q1", ..., Q" N-1 And the fused joint feature information F j ” ointN Input M Transformer decoding layers together, in this process, all mask information in the Transformer decoding layer is set to empty, and finally the hidden states Z0, Z1, ..., Z of all gaze points are obtained N-1 .

7. The target-oriented scanning path prediction method according to claim 4, characterized in that: In step 5, the hidden state information of all fixation points is input into a scanning path head and converted into the coordinate value X, coordinate value Y, duration T and termination probability value V as follows: {X i ,Y i ,T i ,V i }=f head (Z i ;θ head ),i∈[0,1,…,N-1] Among them, f head represents the scanning path head, which consists of four independent one-layer linear fully connected networks, Z i represents the hidden state of the i-th fixation point, θ head Represents the parameters of the scanning path head, X i represents the X-axis coordinate value of the i-th gaze point, Y i represents the Y-axis coordinate value of the i-th gaze point, T i represents the duration of the i-th fixation point, V i Represents the termination probability value of the i-th fixation point.

8. The target-oriented scanning path prediction method according to claim 1, characterized in that: In step 6, in the process of traversing all the fixation points, the specific steps include: The scanning path queue needs to be set to empty first, and then traverse from fixation point i=0 to fixation point i=N-1 in sequence; Check the termination probability value V of the current fixation point i ,i∈[0,1,...,N-1], if the termination probability value V i ≤0.5, the coordinate value X of the gaze point i , coordinate value Y i and duration T i Add it to the scanning path queue; otherwise, stop traversal. Eventually, all the fixation point information in the scanning path queue constitutes a complete scanning path.

9. The target-oriented scanning path prediction method according to claim 1, characterized in that: After step 3, the method further comprises the following steps: Before using the scan path dataset for training, image segmentation is used as a proxy task for pre-training. The proxy task converts the joint feature information obtained in step 3 into an instance segmentation map of the specified target in the scene image, and obtains a better initial feature fusion module through learning. In order to perform pre-training on the proxy task, a segmentation head is added after the hierarchical visual-semantic feature fusion module, and training is performed on the corresponding image segmentation dataset. During the pre-training process, the parameters of the CLIP text / visual encoder, the scan path decoder, and the scan path head are not involved in the training, and the hierarchical visual-semantic feature fusion module and the segmentation head are involved in the training. The pre-training process satisfies the formula: Image seg =SegHead(F joint ) Among them, Image seg It represents the segmentation image of the target in the scene image, SegHead represents the segmentation head, and it consists of two upsampling layers.

10. The target-oriented scanning path prediction method according to claim 1, characterized in that: When the model is trained on the corresponding scan path dataset, the parameters of the CLIP text / visual encoder and segmentation head are not trained, while the hierarchical visual-semantic feature fusion module, the scan path decoder and the scan path head are trained; The loss function used in the training process is the position time loss L xyt , effectiveness loss L val and Soft-DTW loss L Soft-DTW The sum of the three loss functions satisfies the formula: Among them, M represents the batch size, and the position time loss uses L1 regression loss as the loss function; Position-time loss function for the jth scanning path Expressed as: Among them, l j represents the actual length of the j-th scanning path, Respectively represent the actual coordinate value X, coordinate value Y and duration T of the i-th fixation point in the j-th scanning path, Respectively represent the predicted coordinate value X, coordinate value Y and duration T of the i-th fixation point in the j-th scanning path; The effectiveness loss uses the binary cross entropy loss function. The effectiveness loss function for the j-th scanning path is Expressed as: Among them, α is a balancing factor used to balance the weights of the end point and the middle point. and They represent the true value and predicted value of the probability value of the end point of the i-th fixation point in the j-th scanning path, respectively, and L is the maximum length of the scanning path; The Soft-DTW loss function for the j-th scanning path is expressed as: Among them, ρ and They represent the true j-th scanning path and the predicted j-th scanning path respectively, and Soft-DTW represents the Soft-DTW loss function.

Images