A goal-oriented scanning path prediction method

Abstract

The present invention discloses a target-oriented scanning path prediction method, which belongs to the field of path prediction. The present invention extracts target semantic information and multi-level visual information of a scene image; fuses the target semantic information and the multi-level visual information into joint feature information; inputs the query information and joint feature information of all fixations into a scanning path decoder, and predicts the hidden state information of all fixations in parallel; inputs the hidden state information of all fixations into a scanning path header, and converts it into parameters such as coordinate value X, coordinate value Y, duration T, and termination probability value V; finally, traverses all fixations, and if the termination probability value of the current fixation point is less than 0.5, the coordinate value and duration of the fixation point are added to the scanning path, otherwise, the traversal is stopped. The model proposed by the present invention can generate a more realistic and reliable scanning path in a target-oriented scene.

Description

Technical Field

[0001] The present invention relates to the technical field of path prediction, and in particular to a target-oriented scanning path prediction method. Background Art

[0002] Gaze path prediction is a process of dynamically predicting human attention. With the development of technologies such as augmented reality (AR) and virtual reality (VR), the field of glance path prediction has received increasing attention in recent years because it can predict human attention and can be applied to various aspects such as viewport rendering, fatigue detection, and image / video quality assessment.

[0003] Unlike free viewing, goal-oriented scan path prediction is task-driven and purposeful. It is often relatively complex because it needs to consider and model the semantic information of the target and the scene image information. Currently, there is little research on goal-oriented scan path prediction, but this scenario is very common in reality (for example, parents may search for their children in a complex crowd at an amusement park). Therefore, the research on goal-oriented scan path prediction is of great significance for enriching and improving the theory of scan path prediction.

[0004] Early goal-directed saccadic path prediction methods often used semantic guidance graphs to encode target information. However, these methods often rely on pre-trained target detection models to encode target textual information and are often limited to the categories included in the target detection model's training set, which is often relatively small. When predicting unseen categories, it is necessary to first collect a dataset containing the corresponding categories and then retrain the target detection model. Recently, some methods have proposed using visual-semantic matching (i.e., leveraging the expressive power of pre-trained language models and visual models) to predict goal-directed saccadic paths. However, existing methods often use independent visual encoders and semantic encoders and do not fully consider the alignment between visual and semantic information. This results in large distances between different feature spaces for the same object, often requiring a large amount of training data to learn the correspondence between the two representations. Therefore, existing methods based on visual-semantic matching do not produce good results for unseen categories and often perform poorly for seen categories with sufficient training sets. Summary of the Invention

[0005] The present invention provides a target-oriented scanning path prediction method, which can generate a more realistic and reliable scanning path in a target-oriented scenario.

[0006] An embodiment of the present invention provides a target-oriented scanning path prediction method, comprising the following steps:

[0007] Step 1: Use CLIP (Contrastive Language-Image Pre-training) text / visual encoder to extract target semantic information;

[0008] Step 2: Use CLIP visual encoder to extract multi-level visual information of the scene image;

[0009] Step 3: Using a hierarchical visual-semantic feature fusion module, the target semantic information and multi-level visual information are fused into joint feature information;

[0010] Step 4: Input the query information and joint feature information of all fixations into the scan path decoder, and predict the hidden state information of all fixations in parallel;

[0011] Step 5: Input the hidden state information of all fixations into a scan path header and convert it into parameters of coordinate value X, coordinate value Y, duration T, and termination probability value V;

[0012] Step 6: traverse all fixation points. If the termination probability value of the current fixation point is less than or equal to 0.5, add the coordinate value and duration of the fixation point to the scanning path. Otherwise, stop traversal.

[0013] Optionally, in one embodiment of the present invention, in step 1, the target semantic information is extracted using the CLIP text / visual encoder as follows:

[0014] F target =CLIP(target)

[0015] Where target is the textual cue or visual cue representing the target, CLIP represents the corresponding CLIP text / visual encoder, and F target Embedding that represents the semantic information of the target.

[0016] Optionally, in one embodiment of the present invention, in step 2, the CLIP visual encoder is used to extract multi-level visual information of the scene image as follows:

[0017] A1,A2,...,A N =CLIP visual (image)

[0018] Among them, image represents the two-dimensional plane image of the scene, CLIP visual Represents CLIP visual encoder, A1, A2, ..., A NThey represent the feature map embeddings output by N different feature layers in the CLIP visual encoder.

[0019] Optionally, in one embodiment of the present invention, step 3 specifically includes the following steps:

[0020] Step 301: The target semantic information F extracted in step 1 is target After N independent linear mappings, the dimension is transformed into the internal dimension D of the subsequent Transformer architecture, and N new target feature vectors are embedded in F. target1 ,F target2 ,...,F targetN ;

[0021] Step 302: The visual feature graphs A1, A2, ..., A N After N independent linear mappings, the dimension is transformed into the internal dimension D of the subsequent Transformer architecture, and N new visual feature maps A1', A'2, ..., A' are obtained. N ;

[0022] Step 303: embed the first target feature vector into F target1 And the last visual feature map A' N After fusion through a Film structure, the fusion feature map F is obtained joint1 ;

[0023] Step 304: The obtained fusion feature map F joint1 Add a two-dimensional position encoding to get a new feature map F j ' oint1 ;

[0024] Step 305: transform the feature map F j ' oint1 After a standard Transformer encoding layer, the fused features are made aware of the corresponding relationship between each other and a new feature map F is obtained. j ” oint1 ;

[0025] Step 306: Get the feature map F″ joint1 and the penultimate visual feature map A' N-1 Directly add to get the new feature map F″′ joint1 ;

[0026] Step 307: The second target feature vector F target2 Replace the first target feature vector F in step 303 target1 Then use the obtained feature map F″′ joint1Replace the last visual feature map A' in step 303 N And obtain a new fusion feature map F through the new Film structure joint2 And so on, repeat steps 303 to 307 until the final joint feature information F″ is obtained jointN .

[0027] Optionally, in one embodiment of the present invention, in step 4, the scan path decoder generates the hidden states of all gaze points of the scan path in parallel.

[0028] Optionally, in one embodiment of the present invention, step 4 specifically includes the following steps:

[0029] Step 401: query all gaze point information Q0, Q1, ..., Q N-1 The value of is initialized to a zero vector, and then the dimension is mapped to the internal dimension D of the Transformer architecture in the scan path decoder to obtain the new query information representation Q′0,Q′1,...,Q' N-1 ;

[0030] Step 402: query information Q′0, Q′1, ..., Q′ for the mapped gaze points N-1 Add one-dimensional position coding information to obtain new query information Q″0, Q″1,..., Q″ N-1 ;

[0031] Step 403: query information Q″0, Q″1, ..., Q″ N-1 And the fused joint feature information F″ jointN Input M Transformer decoding layers together, in this process all the mask information in the Transformer decoding layer is set to empty, and finally the hidden states Z0, Z1, ..., Z of all gaze points are obtained. N-1 .

[0032] Optionally, in one embodiment of the present invention, in step 5, the hidden state information of all gaze points is input into a scanning path header and converted into the coordinate value X, coordinate value Y, duration T and termination probability value V by the formula:

[0033] {X i ,Y i ,T i ,V i}=f head (Z i θ head ),i∈[0,1,...,N-1]

[0034] Among them, f headRepresents the scanning path head, which consists of four independent one-layer linear fully connected networks, Z i represents the hidden state of the i-th gaze point, θ head Represents the parameters of the scanning path head, X i represents the X-axis coordinate value of the i-th gaze point, Y i represents the Y-axis coordinate value of the i-th gaze point, T i represents the duration of the i-th fixation point, V i Represents the termination probability value of the i-th fixation point.

[0035] Optionally, in one embodiment of the present invention, in step 6, during the process of traversing all gaze points, the specific steps include:

[0036] The scanning path queue needs to be set to empty first, and then traverse from gaze point i = 0 to gaze point i = N-1 in sequence;

[0037] Check the termination probability value V of the current gaze point i ,i∈[0,1,...,N-1], if the termination probability value V i ≤0.5, the coordinate value X of the gaze point i , coordinate value Y i and duration T i Add it to the scanning path queue; otherwise, stop traversal. Eventually, all the gaze point information in the scanning path queue constitutes a complete scanning path.

[0038] Optionally, in one embodiment of the present invention, after step 3, the method further comprises the following steps:

[0039] Before using the scan path dataset for training, image segmentation is used as a proxy task for pre-training. The proxy task converts the joint feature information obtained in step 3 into an instance segmentation map of the specified target in the scene image, and obtains a better initial feature fusion module through learning. In order to perform pre-training on the proxy task, a segmentation head is added after the hierarchical visual-semantic feature fusion module and trained on the corresponding image segmentation dataset. During the pre-training process, the parameters of the CLIP text / visual encoder, scan path decoder and scan path head are not trained, and the hierarchical visual-semantic feature fusion module and segmentation head are trained. The pre-training process satisfies the formula:

[0040] Image seg =SegHead(F joint )

[0041] Among them, Image segIt represents the segmentation image of the target in the scene image, SegHead represents the segmentation head, and is composed of two upsampling layers.

[0042] Optionally, in one embodiment of the present invention, when the model is trained on the corresponding scan path dataset, the parameters of the CLIP text / visual encoder and the segmentation head are not trained, and the hierarchical visual-semantic feature fusion module, the scan path decoder, and the scan path head are trained;

[0043] The loss function used in the training process is the position time loss L xyt , effectiveness loss L val and Soft-DTW loss L Soft-DTW The sum of the three loss functions satisfies the formula:

[0044]

[0045] Among them, M represents the batch size, and the position time loss uses L1 regression loss as the loss function;

[0046] Position-time loss function for the j-th scanning path Expressed as:

[0047]

[0048] Among them, l j represents the true length of the j-th scanning path, Represent the actual coordinate value X, coordinate value Y and duration T of the i-th fixation point in the j-th scanning path, respectively. Respectively represent the predicted coordinate value X, coordinate value Y and duration T of the i-th fixation point in the j-th saccade path;

[0049] The effectiveness loss uses the binary cross entropy loss function, and the effectiveness loss function for the j-th scanning path is Expressed as:

[0050]

[0051] Among them, α is a balancing factor used to balance the weights of the end point and the intermediate point. and represent the true value and predicted value of the endpoint probability of the i-th fixation point in the j-th saccade path, respectively, and L is the maximum length of the saccade path;

[0052] The Soft-DTW loss function for the j-th scanning path is expressed as:

[0053]

[0054] Among them, ρ and They represent the true j-th scanning path and the predicted j-th scanning path respectively, and Soft-DTW represents the Soft-DTW loss function.

[0055] The goal-oriented scanning path prediction method according to the embodiment of the present invention has the following beneficial effects:

[0056] 1. The present invention uses CLIP to extract textual and visual information. Since CLIP has been trained in an aligned manner on a large number of datasets, the present invention can extract more aligned initial features.

[0057] 2. This invention extracts multi-level visual information and fuses it with target semantic information, which helps to consider visual information of different scales and can focus on both large and small targets at the same time;

[0058] 3. This paper designs a novel feature fusion module. It adds a Transformer encoder, which helps to consider more global visual information; it also adds two-dimensional position encoding information, which helps to more accurately locate objects when predicting the gaze point;

[0059] 4. The present invention adopts a parallel Transformer decoder structure to solve the sequence generation problem, which can pay attention to the relationship between all gaze points, consider the relationship between the gaze points and the complete image sequence, and generate the complete scanning path more quickly.

[0060] 5. In addition to predicting the location, duration and other information of the gaze point, the present invention also predicts the probability value of the gaze point being the end point, which helps to determine when the scanning path should stop.

[0061] 6. This invention uses image segmentation as a proxy task, enabling pre-training on large-scale image segmentation datasets. This allows the hierarchical visual-semantic fusion module to better learn the correspondence between text and images. This helps address the limited number of categories and the size of glance path datasets, thereby improving the effectiveness of glance path prediction.

[0062] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0064] Figure 1 A flowchart of a target-oriented scanning path prediction method provided by an embodiment of the present invention;

[0065] Figure 2 A flowchart of a target-oriented scanning path prediction method provided by an embodiment of the present invention;

[0066] Figure 3 Detailed algorithm flow chart of the hierarchical visual-semantic fusion structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0068] Figure 1 A flowchart of a target-oriented scanning path prediction method provided by an embodiment of the present invention.

[0069] like Figure 1 As shown, the target-oriented scanning path prediction method includes the following steps:

[0070] Step 1: Use CLIP text / visual encoder to extract target semantic information.

[0071] In an embodiment of the present invention, the implementation process of step 1 is: passing the target text / visual prompt through the CLIP text / visual encoder, and extracting the final layer output of the CLIP text / visual encoder for the target encoding information. This process can be expressed as:

[0072] F target =CLIP(target)

[0073] Where target is the textual cue or visual cue representing the target, CLIP represents the corresponding CLIP text / visual encoder, and F target Embedding of semantic information representing the target, d represents the real number R target Matrix of order.

[0074] In step 1, the semantic information of the target can be extracted from either textual cues or visual cues. Textual cues are extracted using the CLIP text encoder, while visual cues are extracted using the CLIP visual encoder.

[0075] Step 2: Use the CLIP visual encoder to extract multi-level visual information of the scene image.

[0076] In an embodiment of the present invention, the implementation process of step 2 is as follows: first, the size of the input image is scaled so that the width of the feature map extracted after the CLIP visual encoder is W and the height is H, and then the CLIP visual encoder is used to extract the multi-level visual information of the scene image. This process can be expressed as:

[0077] A1,A2,...,A N =CLIP visual (image)

[0078] Among them, image represents the two-dimensional plane image of the scene, CLIP visual Represents CLIP visual encoder, A1, A2, ..., A N They represent the feature map embeddings output by N different feature layers in the CLIP visual encoder. In the specific implementation process, H×W×d representing a real number R image Matrix of order.

[0079] Step 3: Use a hierarchical visual-semantic feature fusion module to fuse the target semantic information and multi-level visual information into joint feature information.

[0080] In an embodiment of the present invention, the implementation process of step 3 is:

[0081] Step 301: The target semantic information F target After N independent linear mappings, its dimension is transformed into the internal dimension D of the subsequent Transformer architecture to obtain N new target feature vector embeddings F target1 ,F target2 ,...,F targetN , R D represents a D-order matrix of real numbers R;

[0082] Step 302: embed the extracted visual feature map into A1, A2, ..., A N After N independent linear mappings, its dimension is converted into the internal dimension D of the subsequent Transformer architecture to obtain N new visual feature maps A1', A'2, ..., A' N ∈R H×W×D , R H×W×D Represents the H×W×D matrix of real numbers R;

[0083] Step 303: embed the first target feature into F target1 And the last visual feature map A' N After fusion through a Film structure, the fusion feature map F is obtained joint1 ;

[0084] Step 304: The obtained fusion feature map F joint1 Add a two-dimensional position encoding to get a new feature map F′ joint1 ;

[0085] Step 305: transform the feature graph vector F′ joint1 After a standard Transformer encoding layer, the fused features can perceive the corresponding relationship between each other and obtain a new feature map F″ joint1 ;

[0086] Step 306: Get the feature map F″ joint1 and the penultimate visual feature map A' N-1 Directly add to get the new feature map F″′ joint1 ;

[0087] Step 307: The second target feature vector F target2 Replace the first target feature vector F in step 303 target1 , and use the obtained feature map F″′ joint1 Replace the last visual feature map A' in step 303 N , and then pass them through the new Film structure to get a new fusion feature map F joint2 And so on, repeat steps 303 to 307 until the final joint feature information F is obtained. j ” ointN The present invention then uses F joint To represent F j ” ointN , that is, F joint =F j ” ointN .

[0088] In step 4, the query information and joint feature information of all gaze points are input into the scan path decoder, and the hidden state information of all gaze points is predicted in parallel.

[0089] In an embodiment of the present invention, in step 4, the scan path decoder generates the hidden states of all gaze points of the scan path in a parallel manner.

[0090] In an embodiment of the present invention, the implementation process of step 4 is as follows:

[0091] Step 401: All gaze point query information Q0, Q1, ..., Q N-1 The value of is initialized to an all-zero vector, and then its dimension is mapped to the internal dimension D of the Transformer architecture in the scan path decoder to obtain the new query information representation Q′0,Q′1,...,Q' N-1 ∈RD , R D Represents a D-order matrix of real numbers R.

[0092] Step 402: Query information Q′0, Q′1, ..., Q′ for the mapped gaze points N-1 Add one-dimensional position coding information to obtain new query information Q″0, Q″1,..., Q” N-1 .

[0093] Step 403: query information Q″0, Q″1, ..., Q″ N-1 and the fused visual-semantic information Input M Transformer decoding layers together. In this process, all mask information in the Transformer decoding layer needs to be set to empty. Finally, the hidden states Z0, Z1, ..., Z of all gaze points can be obtained. N-1 ∈R D , R D Represents a D-order matrix of real numbers R.

[0094] Step 5: Input the hidden state information of all gaze points into a scanning path header and convert it into parameters of coordinate value X, coordinate value Y, duration T and termination probability value V.

[0095] In an embodiment of the present invention, the calculation formula of step 5 is expressed as:

[0096] {X i ,Y i ,T i ,V i}=f head (Z i θ head ),i∈[0,1,...,N-1]

[0097] Among them, f head Represents the scanning path head, which consists of four independent one-layer linear fully connected networks, Z i represents the hidden state of the i-th gaze point, θ head Represents the parameters of the scanning path head, X i represents the X-axis coordinate value of the i-th gaze point, Y i represents the Y-axis coordinate value of the i-th gaze point, T i represents the duration of the i-th fixation point, V i Represents the termination probability value of the i-th fixation point.

[0098] Step 6: traverse all fixation points. If the termination probability value of the current fixation point is less than or equal to 0.5, add the coordinate value and duration of the fixation point to the scanning path. Otherwise, stop traversal.

[0099] In one embodiment of the present invention, in step 6, the scanning path queue needs to be set to empty first, and then traverse from the fixation point i=0 to the fixation point i=N-1 in sequence. Check the termination probability value V of the current fixation point i ,i∈[0,1,...,N-1], if the probability value V i ≤0.5, the coordinate value X of the gaze point i , coordinate value Y i and duration T i Add it to the scanning path queue; otherwise, stop traversal. Finally, all the gaze point information in the scanning path queue constitutes a complete scanning path.

[0100] It should be noted that, in order to facilitate training, the lengths of all scanning paths in the training set need to be unified to L. For scanning paths shorter than L, they should be padded to ensure that their length is equal to L.

[0101] In one embodiment of the present invention, before the model is trained using a dataset of scan paths, the model may use image segmentation as a proxy task.

[0102] The agent task can use the joint feature information F obtained in step 3 joint It is converted into an instance segmentation image of the target in the scene image, and a better initial feature fusion module can be learned. In order to perform pre-training on this proxy task, it is necessary to add a segmentation head after the hierarchical visual-semantic feature fusion module and train it on the corresponding image segmentation dataset (such as the PhraseCut dataset). During the pre-training process, the parameters of the CLIP text / visual encoder, the scan path decoder, and the scan path head are not trained, while the hierarchical visual-semantic feature fusion module and the segmentation head are trained. This process satisfies the formula:

[0103] Image seg =SegHead(F joint )

[0104] Among them, Image seg Represents the segmentation image of the target in the scene image, SegHead represents the segmentation head, which consists of two upsampling layers.

[0105] In one embodiment of the present invention, during the training of the model using the scan path dataset, the parameters of the CLIP text / visual encoder and the segmentation head are not trained, while the hierarchical visual-semantic feature fusion module, the scan path decoder and the scan path head are trained. The loss function used in the training process is the position-time loss L xyt , effectiveness loss L valand Soft-DTW loss L Soft-DTW The sum of the three loss functions satisfies the formula:

[0106]

[0107] Among them, M represents the batch size, the position time loss uses L1 regression loss as the loss function, and the position time loss function for the j-th scanning path is It can be expressed as:

[0108]

[0109] where l j Represents the true length of the j-th scanning path. Represent the actual coordinate value X, coordinate value Y and duration T of the i-th fixation point in the j-th scanning path, respectively. Represent the predicted coordinate value X, coordinate value Y and duration T of the i-th fixation point in the j-th scanning path. The validity loss uses the binary cross entropy loss (negative log-likelihood loss) function. The validity loss function for the j-th scanning path is It can be expressed as:

[0110]

[0111] Here, α is a balancing factor used to balance the weights of the endpoints and intermediate points. This is because the number of intermediate points in a complete scanning path is often greater than the number of endpoints (usually 1). Here, it is set as the ratio of the number of intermediate points to the number of endpoints. and Represent the true value and predicted value of the endpoint probability of the i-th fixation point in the j-th saccade path, respectively. The Soft-DTW loss function is a differentiable variant of the DTW (Dynamic Time Warping) function. Adding this part can make up for the defect that the position time loss focuses too much on the distance between fixation sequences. It also takes into account the influence of the shape relationship between two fixation sequences (because different saccade paths to find the same object may contain different numbers of fixations). The Soft-DTW loss function for the j-th saccade path can be expressed as:

[0112]

[0113] where ρ and They represent the true j-th scanning path and the predicted j-th scanning path respectively, and Soft-DTW represents the Soft-DTW loss function.

[0114] like Figure 2 and Figure 3As shown, the present invention mainly includes six parts: CLIP text encoder, CLIP visual encoder, visual-semantic fusion module, scan path decoder, scan path head and segmentation head. Among them, the CLIP text encoder is responsible for extracting the semantic information of the target, the CLIP visual encoder is responsible for extracting the multi-scale visual information of the target, the visual-semantic fusion module is responsible for fusing the target semantic information with the multi-level visual information, the scan path decoder is responsible for integrating the gaze point query information and the fused visual-semantic information and generating the hidden state of the gaze point, the scan path head is responsible for converting the hidden state of the gaze point into detailed information such as the location, duration and validity of the gaze point, and the segmentation head is used to use image segmentation as a proxy task and perform pre-training.

[0115] The target-oriented scanning path prediction method of the present invention is described in detail below by a specific embodiment. The target-oriented scanning path prediction method comprises the following steps:

[0116] Step S1: Use the CLIP text / visual encoder to extract the semantic information of the target.

[0117] Specifically, the present invention uses the CLIP text / visual encoder based on ViT-L / 14@336px to extract the final encoding output of the CLIP text / visual encoder for the target semantic information. The process can be expressed as:

[0118] F target =CLIP(target)

[0119] Where target is the text / visual cue representing the target, CLIP represents the corresponding CLIP text / visual encoder, and F target Represents the semantic information embedding of the target, F target ∈R 768 , R 768 Represents the 768-order matrix of real numbers R.

[0120] Step S2: Use the CLIP visual encoder to extract multi-level visual information of the scene image.

[0121] First, the input image is scaled to a width of 448 and a height of 280. Then, it is passed through a CLIP visual encoder based on ViT-L / 14@336px. The extracted feature map has a width of 32 and a height of 20. The CLIP visual encoder is then used to extract multi-level visual information of the scene image. The following relationship exists:

[0122] A1, A2, A3=CLIP visual (image)

[0123] Among them, image represents the two-dimensional plane image of the scene, CLIP visual Represents the visual encoder of the CLIP model, A1, A2, A3 respectively represent the feature maps output by the three different feature layers in the CLIP visual encoder. In the specific implementation process, the invention extracts the output of the 6th, 14th, and 18th layer feature maps of the CLIP visual encoder based on ViT-L / 14@336px, A1, A2, A3∈R 20×32×1024 , R 20×32×1024 A 20-by-32-by-1024 matrix representing real numbers R.

[0124] Step S3: Using a hierarchical visual-semantic feature fusion module, the target semantic information and multi-level visual information are fused into joint feature information.

[0125] Specifically, the process of fusing target semantic information and multi-level visual information into joint feature information includes the following steps:

[0126] S301, target semantic information F target After three independent linear mappings, its dimension is converted to the internal dimension D = 1024 of the subsequent Transformer architecture, and three new target feature vectors are embedded in F target1 ,F target2 ,F target3 ∈R 1024 , R 1024 Represents a 1024-order matrix of real numbers R;

[0127] S302, the extracted visual feature maps A1, A2, A3 are transformed into the internal dimension D = 1024 of the subsequent Transformer architecture through three independent linear mappings, and three new visual feature maps A′1, A′2, A′3∈R are obtained. 20 ×32×1024 , R 20×32×1024 A 20×32×1024 matrix representing the real number R;

[0128] S303, embed the first target feature into F target1 The last visual feature map A'3 is fused through a Film structure to obtain the fused feature map F joint1 ;

[0129] S304, the obtained fusion feature map F joint1 Add a two-dimensional position encoding to get a new feature map F′ joint1 ;

[0130] S305, the feature map vector F′ joint1After passing through a standard Transformer encoding layer, the fused features can perceive the corresponding relationship between each other in the context, and a new feature map F″ is obtained. joint1 ;

[0131] S306, the obtained feature map F″ joint1 Directly add it to the penultimate visual feature map A'2 to get the new feature map F' joint1 ;

[0132] S307, the second target feature vector F target2 Replace the first target feature vector F in step S303 target1 Then use the obtained feature map F″′ joint1 Replace the last visual feature map A'3 in step S303 and pass them through the new Film structure to obtain a new fusion feature map F joint2 And so on, repeat steps S303-S307 until the final joint feature information F″′ is obtained joint3 The present invention then uses F joint To represent F″ joint3 , that is, F joint =F″ joint3 .

[0133] Step S4: input the query information and joint feature information of all gaze points into the scan path decoder, and predict the hidden state information of all gaze points in parallel.

[0134] Specifically, predicting the hidden state information of all gaze points in parallel includes the following steps:

[0135] S401, the values ​​of all gaze query information Q0, Q1, ..., Q6 are initialized to all zero vectors, and then their dimensions are mapped to the internal dimension D = 1024 of the Transformer architecture in the scan path decoder to obtain the new query information representation Q′0, Q′1, ..., Q′6∈R D＝1024 , R 1024 Represents a 1024-order matrix of real numbers R.

[0136] S402 , adding one-dimensional position coding information to the mapped gaze point query information Q′0, Q′1, ..., Q′6 to obtain new query information Q″0, Q″1, ..., Q″6.

[0137] S403, query information Q″0, Q″1, ..., Q″6 and fused visual-semantic information F jointInput M = 6 Transformer decoding layers together. In this process, all mask information in the Transformer decoding layer needs to be set to empty. Finally, the hidden states Z0, Z1, ..., Z6∈R of all gaze points can be obtained 1024 , R 1024 Represents a 1024-order matrix of real numbers R.

[0138] Step S5: Input the hidden state information of all gaze points into a scanning path header, and convert it into parameters such as coordinate value X, coordinate value Y, duration T, and termination probability value V;

[0139] Specifically, the hidden state information of all gaze points is input into a scanning path header, and converted into parameters such as coordinate value X, coordinate value Y, duration T and termination probability value V. The calculation formula is expressed as:

[0140] {X i ,Y i ,T i ,V i}=f head (Z i θ head ),i∈[0,1,...,6]

[0141] Among them, f head Represents the scanning path head, which includes four independent one-layer linear fully connected networks, Z i represents the state of the i-th gaze point, θ head Represents the parameters of the scanning path head, X i represents the X-axis coordinate value of the i-th gaze point, Y i represents the Y-axis coordinate value of the i-th gaze point, T i represents the duration of the ith fixation, V i Represents the termination probability value of the i-th fixation point.

[0142] The scan path header consists of four independent, one-layer linear, fully connected networks. After the first layer, when calculating the mean X, mean Y, and duration T of each fixation point, the present invention converts the fixation point hidden state into a vector of dimension 1 through linear operations, and then uses the ReLU function to convert it into the corresponding value. When calculating the fixation point termination probability value V, the present invention converts the fixation point hidden state into a vector of dimension 2 through linear operations after the first layer, and then uses the logsoftmax function to convert it into the probability of the intermediate point and the probability of the terminal point. Finally, the probability of the intermediate point or the terminal point is determined to determine whether the current fixation point is the end point of the scan path.

[0143] Step S6, in the process of generating the scanning path, traverse all the fixation points, if the termination probability value of the current fixation point is less than or equal to 0.5, then add the coordinate value and duration of the fixation point to the scanning path, otherwise, stop traversing.

[0144] In one embodiment of the present invention, in step S6, the scanning path queue needs to be set to empty first, and then traverse from the fixation point i=0 to the fixation point i=6 in sequence. Check the termination probability value V of the current fixation point i ,i∈[0,1,...,6], if the probability value V i ≤0.5, the coordinate value X of the gaze point i , coordinate value Y i and duration T i Add it to the scanning path queue; otherwise, stop traversal. Finally, all the gaze point information in the scanning path queue constitutes a complete scanning path.

[0145] The present invention proposes a target-oriented scanning path prediction method, which first uses CLIP to extract the semantic information of the target and the multi-level visual information of the image scene, which helps to obtain more aligned initial features; uses multi-level visual information and target semantic information for fusion, which helps to pay attention to object information of different scales; uses a visual-semantic fusion module that integrates Transformer encoder and two-dimensional position encoding, which helps to pay attention to more global visual information and more accurate position information; uses a parallel structure Transformer decoder to generate the scanning path, which can fully consider the relationship between the gaze points, the relationship between the gaze points and the complete image, and generate the complete scanning path faster; uses the scanning path head to predict the position, time and validity parameters of the gaze point, which can more reasonably predict the distribution information, time information and stop information of the gaze point in the scanning path; uses image segmentation as a proxy task, which can better learn the correspondence between image and text during pre-training. Through the reasonable combination of these modules, the present invention can generate a more realistic and reliable scanning path for the target-oriented scanning path prediction task.

[0146] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0147] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0148] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or N executable instructions for implementing a custom logical function or step of a process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

Claims

1. A goal-oriented scanning path prediction method, characterized in that: The following steps are involved: Step 1: Use CLIP text / visual encoder to extract target semantic information; Step 2: Use CLIP visual encoder to extract multi-level visual information of the scene image; Step 3: Using a hierarchical visual-semantic feature fusion module, the target semantic information and multi-level visual information are fused into joint feature information. Step 3 specifically includes the following steps: Step 301: The target semantic information F extracted in step 1 is target After N independent linear mappings, the dimension is transformed into the internal dimension D of the subsequent Transformer architecture, and N new target feature vectors are embedded in F. target1 ,F target2 ,...,F targetN ; Step 302: The visual feature graphs A1, A2, ..., A N After N independent linear mappings, the dimension is transformed into the internal dimension D of the subsequent Transformer architecture, and N new visual feature maps A1', A'2, ..., A' are obtained. N ; Step 303: embed the first target feature vector into F target1 And the last visual feature map A' N After fusion through a Film structure, the fusion feature map F is obtained joint1 ; Step 304: The obtained fusion feature map F joint1 Add a two-dimensional position encoding to get a new feature map F′ joint1 ; Step 305: transform the feature map F′ joint1 After a standard Transformer encoding layer, the fused features are made aware of the corresponding relationship between each other and a new feature map F is obtained. j ” oint1 ; Step 306: Get the feature map F″ joint1 and the penultimate visual feature map A' N-1 Directly add to get the new feature map F j ”' oint1 ; Step 307: The second target feature vector F target2 Replace the first target feature vector F in step 303 target1 Then use the obtained feature map F″′ joint1 Replace the last visual feature map A' in step 303 N And obtain a new fusion feature map F through the new Film structure joint2 And so on, repeat steps 303 to 307 until the final joint feature information F is obtained. j ” ointN ; Step 4: Input the query information and joint feature information of all fixations into the scan path decoder, and predict the hidden state information of all fixations in parallel; Step 5: Input the hidden state information of all fixations into a scan path header and convert it into parameters of coordinate value X, coordinate value Y, duration T, and termination probability value V; Step 6: traverse all fixation points. If the termination probability value of the current fixation point is less than or equal to 0.5, add the coordinate value and duration of the fixation point to the scanning path. Otherwise, stop traversal.

2. The target-oriented scanning path prediction method according to claim 1, wherein In step 1, the CLIP text / visual encoder is used to extract the target semantic information as: F target =CLIP(target) Where target is the textual cue or visual cue representing the target, CLIP represents the corresponding CLIP text / visual encoder, and F target Embedding that represents the semantic information of the target.

3. The target-oriented scanning path prediction method according to claim 1, wherein In step 2, the CLIP visual encoder is used to extract the multi-level visual information of the scene image as follows: A1,A2,...,A N =CLIP visual (image) Among them, image represents the two-dimensional plane image of the scene, CLIP visual Represents CLIP visual encoder, A1, A2, ..., A N They represent the feature map embeddings output by N different feature layers in the CLIP visual encoder.

4. The target-oriented scanning path prediction method according to claim 1, wherein In step 4, the glance path decoder generates the hidden states of all gaze points in the glance path in parallel.

5. The target-oriented scanning path prediction method according to claim 1, wherein: Step 4 specifically includes the following steps: Step 401: query all gaze point information Q0, Q1, ..., Q N-1 The value of is initialized to a zero vector, and then the dimension is mapped to the internal dimension D of the Transformer architecture in the scan path decoder to obtain the new query information representation Q0', Q1', ..., Q' N-1 ; Step 402: query information Q0', Q1', ..., Q' for the mapped gaze points N-1 Add one-dimensional position coding information to obtain new query information Q0”, Q1”, ..., Q” N-1 ; Step 403, query information Q0", Q1", ..., Q" N-1 And the fused joint feature information F j ” ointN Input M Transformer decoding layers together, in this process all the mask information in the Transformer decoding layer is set to empty, and finally the hidden states Z0, Z1, ..., Z of all gaze points are obtained. N-1 .

6. The target-oriented scanning path prediction method according to claim 1, characterized in that: In step 5, the hidden state information of all fixations is input into a scan path header and converted into the coordinate value X, coordinate value Y, duration T and termination probability value V as follows: {X i ,Y i ,T i ,V i }=f head (Z i ;θ head ),i∈[0,1,…,N−1] Among them, f head Represents the scanning path head, which consists of four independent one-layer linear fully connected networks, Z i represents the hidden state of the i-th gaze point, θ head Represents the parameters of the scanning path head, X i represents the X-axis coordinate value of the i-th gaze point, Y i represents the Y-axis coordinate value of the i-th gaze point, T i represents the duration of the i-th fixation point, V i Represents the termination probability value of the i-th fixation point.

7. The target-oriented scanning path prediction method according to claim 1, characterized in that: In step 6, in the process of traversing all the fixation points, the specific steps include: The scanning path queue needs to be set to empty first, and then traverse from gaze point i = 0 to gaze point i = N-1 in sequence; Check the termination probability value V of the current gaze point i ,i∈[0,1,...,N-1], if the termination probability value V i ≤0.5, the coordinate value X of the gaze point i , coordinate value Y i and duration T i Add it to the scanning path queue; otherwise, stop traversal. Eventually, all the gaze point information in the scanning path queue constitutes a complete scanning path.

8. The target-oriented scanning path prediction method according to claim 1, characterized in that: After step 3, the method further comprises the following steps: Before using the scan path dataset for training, image segmentation is used as a proxy task for pre-training. The proxy task converts the joint feature information obtained in step 3 into an instance segmentation map of the specified target in the scene image, and obtains a better initial feature fusion module through learning. In order to perform pre-training on the proxy task, a segmentation head is added after the hierarchical visual-semantic feature fusion module and trained on the corresponding image segmentation dataset. During the pre-training process, the parameters of the CLIP text / visual encoder, scan path decoder and scan path head are not trained, and the hierarchical visual-semantic feature fusion module and segmentation head are trained. The pre-training process satisfies the formula: Image seg =SegHead(F joint ) Among them, Image seg Represents the segmentation image of the target in the scene image, SegHead represents the segmentation head, which consists of two upsampling layers, F joint =F j ” ointN .

9. The target-oriented scanning path prediction method according to claim 1, characterized in that: When the model is trained on the corresponding scan path dataset, the parameters of the CLIP text / visual encoder and segmentation head are not trained, while the hierarchical visual-semantic feature fusion module, scan path decoder, and scan path head are trained; The loss function used in the training process is the position time loss L xyt , effectiveness loss L val and Soft-DTW loss L Soft-DTW The sum of the three loss functions satisfies the formula: Among them, M represents the batch size, and the position time loss uses L1 regression loss as the loss function; Position-time loss function for the j-th scanning path Expressed as: Among them, l j represents the true length of the j-th scanning path, Represent the actual coordinate value X, coordinate value Y and duration T of the i-th fixation point in the j-th scanning path, respectively. Respectively represent the predicted coordinate value X, coordinate value Y and duration T of the i-th fixation point in the j-th saccade path; The effectiveness loss uses the binary cross entropy loss function, and the effectiveness loss function for the j-th scanning path is Expressed as: Among them, α is a balancing factor used to balance the weights of the end point and the intermediate point. and represent the true value and predicted value of the endpoint probability of the i-th fixation point in the j-th saccade path, respectively, and L is the maximum length of the saccade path; The Soft-DTW loss function for the j-th scanning path is expressed as: Among them, ρ and They represent the true j-th scanning path and the predicted j-th scanning path respectively, and Soft-DTW represents the Soft-DTW loss function.

Images