Method, system, terminal and medium for constructing visual attention prediction model

Abstract

The application provides a construction method of a visual attention prediction model, and faces an autism population, and comprises the following steps: constructing a visual attention prediction model based on atypical salient region enhancement; pre-training the visual attention prediction model based on atypical salient region enhancement by using a known eye movement data set, and correcting the model by using an eye movement data set of an autism population, so as to complete end-to-end training of the visual attention prediction model based on atypical salient region enhancement; testing the trained visual attention prediction model based on atypical salient region enhancement by using test images in the known eye movement data set, and constructing a final visual attention prediction model. Meanwhile, a corresponding construction system, an application method, a terminal and a medium are provided. The application is started from the special visual preference of autism patients, has the characteristics of high prediction efficiency, low cost, easy implementation, and very flexible deployment, and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual attention prediction, in particular to a construction method and system of a visual attention prediction model for autism groups, a terminal and a medium. BACKGROUND

[0002] The human visual system can quickly select and focus on important areas in visual stimuli, and this ability enables humans to selectively process a large amount of information entering the field of view, so as to efficiently receive and process main information while ignoring irrelevant information. This selective mechanism is called visual attention mechanism. Visual attention prediction (also called visual saliency prediction and gaze point prediction) is a technology that simulates the visual attention mechanism of the human eye, and the final calculated saliency map can quantitatively represent the attention distribution, wherein the higher the brightness of the region, the greater the probability of attracting the attention of the human eye. It has very important applications in object segmentation, target tracking, image compression, video compression and many other vision-related tasks.

[0003] Autism spectrum disorder is a genetic neurodevelopmental disorder. Currently, there is computational model and neuroimaging evidence to prove that autism patients exhibit atypical visual attention behavior when facing visual stimuli, which is different from that of normal control groups. In short, when observing a scene, the normal control group tends to focus on objects with high-level semantic attributes, such as human faces, texts, etc.; while autism patients are often attracted by background areas and some regions with low-level attributes, which are called atypical salient regions. The existing visual attention prediction methods for autism patients are mostly inspired by general visual attention prediction methods for regular groups, ignoring the atypical gaze behavior and special visual preference of autism patients, which are significantly different from normal control groups. Only simple migration is performed on the eye movement data set of autism patients, which lacks pertinence and has poor prediction performance of gaze points. SUMMARY

[0004] The present application provides a construction method, system, terminal and medium of a visual attention prediction model for autism groups to solve the above problems in the prior art.

[0005] According to one aspect of the present application, a construction method of a visual attention prediction model is provided, comprising:

[0006] constructing a visual attention prediction model based on atypical salient region enhancement;

[0007] The atypical saliency region enhanced visual attention prediction model is pre-trained by using a known eye movement data set, and is corrected by using an eye movement data set of an autism group, so as to complete end-to-end training of the atypical saliency region enhanced visual attention prediction model.

[0008] The atypical saliency region enhanced visual attention prediction model after training is tested by using test images in the known eye movement data set, and a final visual attention prediction model is constructed.

[0009] Optionally, the atypical saliency region enhanced visual attention prediction model is constructed, and includes:

[0010] A feature extraction network layer is constructed, which is used for extracting features of an input image and outputting multi-order feature maps.

[0011] A multi-scale enhancement network layer is constructed, which is used for performing multi-scale enhancement on a highest-order feature map in the multi-order feature maps, so as to improve the detection ability of saliency regions of different scales.

[0012] An atypical saliency region enhancement network layer is constructed, which is used for taking the highest-order feature map as an initial prediction result, performing residual fusion on the multi-order feature maps from top to bottom, and obtaining an enhanced atypical saliency region feature map.

[0013] A global semantic stream network layer is constructed, which is used for extracting context semantic information of the highest-order feature map from spatial and channel angles respectively, obtaining a global semantic stream, and introducing the global semantic stream into the atypical saliency region enhancement network layer, so as to guide the residual fusion and simultaneously adaptively supplement diluted semantic information.

[0014] A saliency map reading network layer is constructed, which is used for compressing and normalizing the enhanced atypical saliency region feature map along a channel dimension, and obtaining a visual attention prediction result.

[0015] Optionally, the feature extraction network layer adopts a full convolution form of a pre-trained object recognition network based on deep learning to construct a backbone network of the feature extraction network layer, which is used for extracting features of the input image and outputting multi-order feature maps; wherein the feature extraction network layer includes a convolution layer, a pooling layer and a ReLu activation layer.

[0016] Optionally, the multi-scale enhancement network layer adopts a plurality of parallel convolution layers with different sizes of convolution kernels to explicitly introduce multi-scale information, and performs multi-scale enhancement on the highest-order feature map, and the enhancement result acts on the residual fusion process, so as to improve the feature extraction ability of saliency regions of different scales.

[0017] Optionally, the atypical salient region enhancement network layer comprises: a background enhancement network layer, a foreground enhancement network layer and a residual fusion network layer; and the residual fusion is performed on the features in the multi-order feature maps from top to bottom to obtain the enhanced atypical salient region feature map, using the highest-order feature map as the initial prediction result, comprising:

[0018] The background feature is obtained by performing negation on the high-order feature in the multi-order feature maps using the background enhancement network layer, and then the background weight map is obtained by normalizing the background feature;

[0019] The enhanced low-order feature is obtained by performing foreground weighting on the low-order feature adjacent to the high-order feature using the foreground enhancement network layer;

[0020] The residual feature is obtained by performing weighted fusion on the enhanced low-order feature using the background weight map through the residual fusion network layer; and the new prediction result is obtained by adaptively fusing the high-order feature and the residual feature;

[0021] The new prediction result is taken as the new high-order feature, and the residual fusion is continuously performed on the new high-order feature and the adjacent low-order feature until the final enhanced atypical salient region feature map is obtained.

[0022] Optionally, the global semantic flow network layer comprises a channel enhancement network layer and a spatial position enhancement network layer; and the context semantic information of the feature map is extracted from the spatial and channel angles to obtain the global semantic flow, comprising:

[0023] The network layer input feature is obtained by transforming and compressing the channel number of the highest-order feature in the feature map through the convolution layer;

[0024] The channel enhancement network layer adopts global average pooling to obtain global prior, and the channel weighting map of the network layer input feature is obtained by transforming and normalizing through the 1*1 convolution layer, and the channel enhancement feature is obtained by performing channel enhancement on the network layer input feature using the channel weighting map;

[0025] The spatial position enhancement network layer adopts a self-attention mechanism to fully capture the correlation between the pixels of the network layer input feature, to obtain the spatial position weighting map, and the position enhancement feature is obtained by performing position enhancement on the network layer input feature using the spatial position weighting map;

[0026] The global semantic flow is obtained by fusing the channel enhancement feature and the position enhancement feature;

[0027] The global semantic stream is introduced into the atypical saliency region enhancement network layer, and by adaptively adjusting the weight of the global semantic stream, global information is adaptively supplemented in the residual fusion process.

[0028] Optionally, the saliency map reading network layer comprises a 3*3 convolution layer and a sigmoid activation function.

[0029] Optionally, the atypical saliency region enhancement-based visual attention prediction model is pre-trained by using a known eye movement dataset, and is corrected by using an eye movement dataset of an autism group, and end-to-end training of the atypical saliency region enhancement-based visual attention prediction model is completed, comprising:

[0030] The atypical saliency region enhancement-based visual attention prediction model is pre-trained by using a public eye movement dataset SALICON and MIT1003, and is corrected by using an eye movement dataset Saliency4ASD of an autism group;

[0031] The initialization parameters of the atypical saliency region enhancement-based visual attention prediction model are set;

[0032] The loss function of the atypical saliency region enhancement-based visual attention prediction model is determined;

[0033] The related hyperparameters in the atypical saliency region enhancement-based visual attention prediction model are determined;

[0034] The end-to-end training of the atypical saliency region enhancement-based visual attention prediction model is completed through the above steps.

[0035] Optionally, the atypical saliency region enhancement-based visual attention prediction model is pre-trained by using a public eye movement dataset SALICON and MIT1003, and is corrected by using an eye movement dataset Saliency4ASD of an autism group, comprising:

[0036] The public eye movement datasets SALICON and MIT1003 and the eye movement dataset Saliency4ASD of an autism group are obtained, the eye movement position sampling points of the image data in the eye movement datasets are clustered to generate a mat file containing fixation points, the mat file is normalized to be converted to generate a fixation point density map as a true value map;

[0037] The image of the eye movement dataset SALICON and MIT1003 is taken as a model input, and the ground truth map corresponding to the image of the eye movement dataset SALICON and MIT1003 is taken as a label, so as to train the visual attention prediction model based on atypical salient region enhancement in an end-to-end manner, so that the model automatically learns the mapping relationship between the original image and the ground truth map, and obtains a feature distribution related to the human eye.

[0038] The image of the eye movement dataset Saliency4ASD of the autism group is taken as a model input, and the ground truth map corresponding to the image of the eye movement dataset Saliency4ASD of the autism group is taken as a label, so as to fine-tune the visual attention prediction model based on atypical salient region enhancement in an end-to-end manner, so that the model automatically learns the mapping relationship between the original image and the ground truth map, and obtains an eye movement feature of the autism group, and corrects the model.

[0039] Optionally, the setting of the initialization parameter of the visual attention prediction model based on atypical salient region enhancement comprises:

[0040] The visual attention prediction model based on atypical salient region enhancement comprises a feature extraction network layer, a multi-scale enhancement network layer, an atypical salient region enhancement network layer, a global semantic stream network layer, and a saliency map readout network layer.

[0041] The feature extraction network layer adopts the parameters obtained by pre-training on the ImageNet dataset as the initialization parameters; and the initial parameters of the other network layers are random initialization parameters.

[0042] Optionally, the determination of the loss function of the visual attention prediction model based on atypical salient region enhancement comprises:

[0043] The loss function adopts a weighted linear combination of three saliency performance evaluation indexes KL, CC and NSS.

[0044] Optionally, the determination of the related hyperparameters in the visual attention prediction model based on atypical salient region enhancement comprises:

[0045] In the pre-training process, a stochastic gradient descent algorithm is adopted, the initial learning rate is 10 -4 , the Batchsize size is 10, and the pre-training process needs to be iterated for 20 epochs until the model converges.

[0046] Optionally, the testing of the trained visual attention prediction model based on atypical salient region enhancement by using the test image in the known eye movement dataset comprises:

[0047] The trained non-typical salient region enhanced visual attention prediction model is tested by using the benchmark provided in the public data sets SALICON, MIT1003 and Saliency4ASD, so as to evaluate the performance of the model.

[0048] According to another aspect of the present application, a system for constructing a visual attention prediction model is provided, comprising:

[0049] a prediction model constructing module configured to construct a non-typical salient region enhanced visual attention prediction model;

[0050] a model training module configured to pre-train the non-typical salient region enhanced visual attention prediction model by using known eye movement data sets, and correct the non-typical salient region enhanced visual attention prediction model by using eye movement data sets of autism groups, so as to complete end-to-end training of the non-typical salient region enhanced visual attention prediction model;

[0051] a model testing module configured to test the trained non-typical salient region enhanced visual attention prediction model by using known test images, and evaluate the performance of the constructed model.

[0052] According to a third aspect of the present application, a visual attention prediction method is provided, which uses the visual attention prediction model constructed by the method or system according to any one of the above-mentioned aspects, takes any image as input of the model, and outputs a visual attention prediction result.

[0053] According to a fourth aspect of the present application, a computer terminal is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the program to perform the method according to any one of the above-mentioned aspects.

[0054] According to a fifth aspect of the present application, a computer readable storage medium is provided, which stores a computer program executable on a processor to perform the method according to any one of the above-mentioned aspects.

[0055] Thanks to the above technical solutions, the present application has at least one of the following beneficial effects compared with the prior art:

[0056] The application provides a visual attention prediction model construction method, system, terminal and medium.

[0057] The application provides a visual attention prediction model construction method, system, terminal and medium, which utilizes global semantic flow technology to guide residual fusion of each stage, reduces adverse effects of noise contained in low-order features on model performance, and improves accuracy and robustness of the visual attention prediction model.

[0058] The application provides a visual attention prediction model construction method, system, terminal and medium, which is efficient, low in cost, easy to implement, very flexible, and can be deployed on a main network with less parameters (considering efficiency) or a main network with better performance (considering performance) according to actual needs. BRIEF DESCRIPTION OF DRAWINGS

[0059] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the following drawings:

[0060] Figure 1 A workflow diagram of a visual attention prediction method in a preferred embodiment of the application.

[0061] Figure 2 A component module schematic diagram of a visual attention prediction system in a preferred embodiment of the application.

[0062] Figure 3 Part of subjective experimental results of a visual attention prediction method and system provided in a preferred embodiment of the application on a public data set in the field of visual attention prediction dedicated to autism. DETAILED DESCRIPTION

[0063] The embodiments of the application are described in detail below: The embodiments are implemented on the premise of the technical solutions of the application, and detailed implementation modes and specific operation processes are given. It should be noted that, for ordinary skilled persons in the art, without departing from the concept of the application, a number of modifications and improvements can be made, which all belong to the protection scope of the application.

[0064] An embodiment of the present application provides a method for constructing a visual attention prediction model, which is oriented to an autism group, and is based on special visual preferences of autism patients to realize construction of a visual attention prediction model based on non-typical salient region enhancement.

[0065] As shown in the figure, the method for constructing the visual attention prediction model provided by the embodiment can include: Figure 1

[0066] S1, constructing a visual attention prediction model based on non-typical salient region enhancement;

[0067] S2, pre-training the visual attention prediction model based on non-typical salient region enhancement by using a known eye movement data set, and correcting the visual attention prediction model based on non-typical salient region enhancement by using an eye movement data set of the autism group, to complete end-to-end training of the visual attention prediction model based on non-typical salient region enhancement;

[0068] S3, testing the trained visual attention prediction model based on non-typical salient region enhancement by using test images in the known eye movement data set, to construct a final visual attention prediction model.

[0069] In a preferred embodiment of S1, constructing the visual attention prediction model based on non-typical salient region enhancement can include:

[0070] The visual attention prediction model based on non-typical salient region enhancement mainly includes a feature extraction network layer, a multi-scale enhancement network layer, a non-typical salient region enhancement network layer, a global semantic stream network layer, and a saliency map reading network layer, and performs end-to-end visual attention prediction; wherein:

[0071] S101, constructing the feature extraction network layer, which is used to extract features of an input image and output multi-order feature maps. The network layer uses a pre-trained object recognition network based on deep learning (removing the subsequent fully connected layer and only retaining the front full convolution part) as a backbone network to extract features, mainly composed of convolution layers, pooling layers, and ReLu activation layers, with an image as input and a feature map as output. For ease of illustration, the outputs of the five convolution blocks of the backbone network are denoted as: F i (i∈{1,2,3,4,5}). In addition, since the non-typical salient region enhancement network layer and the global semantic stream network layer proposed in the embodiment of the present application can be flexibly deployed on different backbone networks, the visual attention prediction model can select VGG with less parameter amount or DenseNet with better performance as the backbone network according to actual needs;

[0072] ​S102, a multi-scale enhancement network layer is constructed for multi-scale enhancement of the highest order feature map in the multi-order feature map to improve the detection ability of the salient region of different scales. In this network layer, a multi-scale enhancement module is explicitly introduced to improve the detection ability of the model for salient regions of different scales. In a specific application example, an Inception-like structure is used, i.e., a plurality of parallel convolution layers with different sizes of convolution kernels are used to introduce multi-scale information. In order to control the parameter amount, the network layer is implemented using a dilated convolution with different dilated rates. The highest order feature F 5 As the input of the multi-scale enhancement network layer, the multi-scale enhanced feature F 5 is output.

[0073] S103, a non-typical salient region enhancement network layer is constructed for residual fusion of the multi-order feature map from top to bottom with the highest order feature map as the initial prediction result to obtain the enhanced non-typical salient region feature map. The network layer mainly consists of a foreground enhancement network layer of low-order features, a background enhancement network layer across orders, and a residual fusion network layer. Specifically, since the feature extraction network layer as the backbone network is modified from a pre-trained object recognition network, the output of the backbone network from the shallow layer to the deep layer is more and more focused on objects with high-order semantic attributes. For the general population, in most cases, objects with high-order semantic attributes (such as human faces, texts, etc.) are visual salient regions, so the semantic bias property of the features output by the backbone network promotes the improvement of the performance of the general visual attention prediction model for the general population. However, the visual attention of autism patients is dominated by atypical saliency, and this group pays more attention to background regions and other regions without social attributes than the general population, so there is a big gap between the semantic bias property of the features output by the backbone network and the atypical visual attention of autism patients, and the commonly used direct fusion method performs poorly in the visual attention prediction task for the autism population. In this step, the highest order output of the backbone network is taken as the initial prediction result, and residual fusion is performed from top to bottom to enhance the non-typical salient region. The background enhancement network layer acts between two adjacent order features. In a specific application example, F 5 and F 4 are taken as examples:

[0074] First, F 5 is upsampled to the same size as the adjacent low-order feature F 4 to obtain Then, the background enhancement network layer takes to get the background feature Bf 5 , and then the background weight map BAM 5 is obtained by normalization:

[0075] BAM 5= σ (Bf 5 )

[0076] where σ denotes sigmoid activation function.

[0077] Then, the residual fusion network layer uses the background weight map to weight and fuse the adjacent low-order features, and the residual features obtained contain the atypical salient regions that are not detected in the initial prediction but are successfully detected by the adjacent low-order features.

[0078] It is worth noting that considering that the low-order features contain more noise, and in this residual fusion manner, the adverse effects of noise will be more obvious. Therefore, before the background weighting, the foreground enhancement network layer is used to weight the low-order features, enhance the foreground salient regions and weaken the influence of the noise contained in the low-order features, to obtain the enhanced low-order features:

[0079] FAM 4 = σ (Conv (F 4 ))

[0080] where FAM 4 is the obtained enhanced low-order feature, and Conv(.) denotes convolution operation.

[0081] Finally, the residual fusion network layer adaptively fuses the original high-order feature (i.e. the original prediction result ) and the residual feature to obtain a new prediction result F 4 ':

[0082]

[0083]

[0084] where F 5 is obtained by processing F 4 through the multi-scale enhancement network layer described in S202 and upsampling, and , and and respectively denote pixel-by-pixel multiplication, pixel-by-pixel addition and channel dimension splicing operation, and T represents 3 consecutive convolution-batch normalization-ReLu activation operations.

[0085] After that, F 4 ' as the new prediction result, continues to repeat the above operation with the adjacent low-order feature F 3 until the final prediction result is obtained.

[0086] Therefore, compared with the direct fusion method commonly used in general visual attention prediction models for regular groups, the atypical salient region enhancement module proposed in this embodiment of the invention makes the model pay more attention to the salient regions that were not detected before. As a result, in the top-down residual fusion process, the atypical salient regions are gradually detected, and the prediction results are more complete.

[0087] S104, construct a global semantic flow network layer to extract contextual semantic information from the highest-order feature map from both spatial and channel perspectives, obtaining a global semantic flow. This global semantic flow is then introduced into an atypical salient region enhancement network layer to guide residual fusion and adaptively supplement diluted semantic information. Considering that semantic information from deeper layers is continuously diluted during top-down feature fusion, while the adverse effects of shallow noise become increasingly significant, this step constructs a global semantic flow network layer to extract contextual semantic information from both spatial and channel perspectives, thereby guiding residual fusion in the feature fusion stage and adaptively supplementing diluted semantic information. This network layer mainly consists of a channel enhancement network layer and a spatial location enhancement network layer. First, the highest-order feature F... 5 The number of channels is obtained by compressing the convolutional layer. This serves as the input to the network layer. For the channel enhancement network layer, global average pooling is first used to obtain the global prior, followed by a 1x1 convolutional layer transformation and normalization to obtain the channel-weighted map (CAM). 5 And based on channel-weighted graph CAM 5 For the original features Channel enhancement is performed to obtain output channel enhancement features.

[0088]

[0089]

[0090] Here, GAP represents global average pooling operation. This represents a channel-wise multiplication operation.

[0091] For the spatial location enhancement network layer, a self-attention mechanism is used to fully capture the correlation between pixels to obtain the spatial location weighted map (SM). 5 And based on the spatial location weighted graph SM 5 The enhanced location enhancement features are obtained.

[0092]

[0093]

[0094] Wherein Q, K, V are original features After convolutional transformation.

[0095] Finally, the two parts of features respectively after channel and spatial position weighting are fused to obtain a global semantic stream

[0096]

[0097] The global semantic stream is used to guide the residual fusion operation of each stage, and the fusion operation in S203 is modified as follows:

[0098]

[0099] Wherein, is obtained by upsampling, and b is a learnable parameter for adaptively adjusting the weight of the global semantic stream, thereby adaptively supplementing global information.

[0100] S105, a saliency map reading network layer is constructed, which is used to compress and normalize the enhanced atypical salient region feature map along the channel dimension to obtain a visual attention prediction result. The network layer is composed of a 3*3 convolution layer and a sigmoid activation function. The role is to compress and normalize the output of the above module along the channel dimension to obtain the final prediction result.

[0101] In a preferred embodiment of S2, the visual attention prediction model based on atypical salient region enhancement is pre-trained using known eye movement data sets, and the visual attention prediction model based on atypical salient region enhancement is modified using eye movement data sets of autism groups, and the end-to-end training of the visual attention prediction model based on atypical salient region enhancement is completed, which can include:

[0102] S201, the visual attention prediction model based on atypical salient region enhancement is pre-trained using public data sets SALICON and MIT1003, and then fine-tuned (modified) on the autism-specific eye movement data set Saliency4ASD data set. Wherein:

[0103] Obtain public eye movement data sets SALICON and MIT1003 and eye movement data sets Saliency4ASD of autism groups, and generate mat files containing fixation points by clustering the eye movement position sampling points of image data in the above eye movement data sets. In order to facilitate processing, the mat file is normalized to generate a fixation point density map as a true value map and train the constructed model in the subsequent steps;

[0104] ​The image of the eye movement dataset SALICON and MIT1003 is taken as input, and the true value map corresponding to the image is taken as a label. The proposed visual attention prediction model based on non-typical salient region enhancement is trained in an end-to-end manner, so that the mapping relationship between the original image and the true value map is automatically learned, and the feature distribution related to the human eye is obtained. The image of the eye movement dataset Saliency4ASD of the autism group is taken as the input of the model, and the true value map corresponding to the image of the eye movement dataset Saliency4ASD of the autism group is taken as a label. The visual attention prediction model based on non-typical salient region enhancement is fine-tuned in an end-to-end manner, so that the model automatically learns the mapping relationship between the original image and the true value map, and the eye movement features of the autism group are obtained. The model is corrected. Specifically:

[0105] SALICON is a general eye movement dataset that simulates the human eye fixation point using a mouse click, and it is also the largest public dataset in the field of visual attention prediction. Therefore, the proposed non-typical visual attention prediction model is first pre-trained on this dataset to provide good initialization parameters for subsequent training. The MIT1003 dataset is a dataset collected and constructed using an eye tracker. The model pre-trained on SALICON is further trained on this dataset to enable the model to learn the feature distribution related to the human eye. Finally, because the Saliency4ASD dataset is relatively small, to prevent overfitting, the model is first trained using the eye movement datasets SALICON and MIT1003, and then fine-tuned using the Saliency4ASD dataset to enable the model to fully learn the eye movement features of the autism group.

[0106] S202, set initialization parameters for the model. The parameters of the backbone network pre-trained on the ImageNet dataset are used as the initialization parameters, and the initial parameters of the other network layers are randomly initialized.

[0107] S203, determine the loss function. The loss function used in the model training is a weighted linear combination of three saliency performance evaluation indicators KL, CC, and NSS. The weights of the indicators are determined according to the experimental results to better balance the influence of each indicator on the performance of the model. For the KL indicator, the better the performance of the model, the smaller the value of the KL indicator, so the coefficient of the KL indicator in the loss function is negative, and the coefficients of the remaining indicators are positive.

[0108] S204, determine the related hyperparameters in the model. The gradient descent algorithm used in the training process is stochastic gradient descent, the initial learning rate is 10 -4 , and it decreases by 10 times every 3 epochs. The Batchsize size is 10, and the training generally needs to iterate 20 epochs to converge the model.

[0109] In a preferred embodiment of S3, the trained visual attention prediction model based on atypical salient region enhancement is tested by using test images in known eye movement data sets, and the final visual attention prediction model is constructed, which can include:

[0110] S301, the performance of the model proposed in the application is tested by using three commonly used public data sets in the field of visual attention prediction, namely SALICON, MIT1003 and Saliency4ASD. These three public data sets provide benchmarks for researchers to make fair performance comparisons. When testing, the test image is input into the trained visual attention prediction model based on atypical salient region enhancement in S2 to obtain the prediction result, which is compared with the corresponding true value map and the performance is calculated, and the final visual attention prediction model is constructed.

[0111] An embodiment of the application provides a visual attention prediction model construction system.

[0112] As shown in Figure 2 The visual attention prediction model construction system provided by the embodiment can include:

[0113] A prediction model construction module, which is configured to construct a visual attention prediction model based on atypical salient region enhancement;

[0114] A model training module, which is configured to pre-train the visual attention prediction model based on atypical salient region enhancement by using known eye movement data sets, and correct the visual attention prediction model based on atypical salient region enhancement by using eye movement data sets of autism groups, and complete end-to-end training of the visual attention prediction model based on atypical salient region enhancement;

[0115] A model testing module, which is configured to test the trained visual attention prediction model based on atypical salient region enhancement by using known test images, and evaluate the performance of the constructed model.

[0116] In a preferred embodiment, the visual attention prediction model based on atypical salient region enhancement can include:

[0117] A feature extraction module configured to extract a feature map of an input image and output a multi-order feature map;

[0118] A multi-scale enhancement module configured to perform multi-scale enhancement on the highest-order feature map in the multi-order feature map to improve the detection ability of different scale salient regions;

[0119] The atypical salient region enhancement module is used for taking the highest order feature map as an initial prediction result, performing residual fusion on features in the feature map from top to bottom, and obtaining an enhanced atypical salient region feature map.

[0120] The global semantic stream module is used for extracting context semantic information of the highest order feature map from the spatial and channel angles respectively, obtaining a global semantic stream, and introducing the global semantic stream into the atypical salient region enhancement network layer, which is used for guiding the residual fusion and simultaneously adaptively supplementing the diluted semantic information.

[0121] The saliency map reading module is used for compressing the enhanced atypical salient region feature map along the channel dimension and performing normalization to obtain a visual attention prediction result.

[0122] It should be noted that the steps in the method provided by the present application can be realized by corresponding modules, devices, units, etc. in the system, and those skilled in the art can refer to the technical solution of the method to realize the composition of the system, that is, the embodiments in the method can be understood as preferred examples of constructing the system, which will not be described here.

[0123] An embodiment of the present application provides a visual attention prediction method, which adopts the visual attention prediction model constructed by the construction method or the construction system of any one of the above embodiments to obtain a visual attention prediction result by taking any image as an input of the model.

[0124] An embodiment of the present application provides a computer terminal, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor can be used to execute the method or the system of any one of the above embodiments of the present application when executing the program.

[0125] Optionally, a memory is configured to store a program; the memory can include volatile memory (e.g., random-access memory (RAM), such as static random-access memory (SRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), etc.), and / or non-volatile memory (e.g., flash memory). The memory is configured to store computer programs (e.g., application programs, functional modules, etc. for implementing the above-described methods), computer instructions, etc. The computer programs, computer instructions, etc. described above can be stored in one or more memories in a partitioned manner. Furthermore, the computer programs, computer instructions, data, etc. described above can be invoked by the processor.

[0126] The computer programs, computer instructions, etc. described above can be stored in one or more memories in a partitioned manner. Furthermore, the computer programs, computer instructions, data, etc. described above can be invoked by the processor.

[0127] The processor is configured to execute the computer programs stored in the memory, so as to implement the various steps in the methods or the various modules of the systems described in the above embodiments. For details, refer to the related descriptions in the above method and system embodiments.

[0128] The processor and the memory can be independent structures, or can be integrated into an integrated structure. When the processor and the memory are independent structures, the memory and the processor can be coupled and connected through a bus.

[0129] An embodiment of the present application provides a computer readable storage medium, which stores a computer program. The program is executed by a processor, and can be used to execute the method or the system of any one of the above-described embodiments of the present application.

[0130] The construction method, system, terminal and medium of the visual attention prediction model provided by the above embodiments of the present application construct a visual attention prediction model based on atypical salient region enhancement, the multi-scale enhancement part is used to improve the detection ability of the model to different scale salient regions, the atypical salient region enhancement part uses the semantic object biasing characteristics of the backbone network to perform cross-order residual fusion between the multi-order features output by the backbone network to enhance the atypical salient region, and thus better simulate the atypical visual attention specific to autism patients, the global semantic stream part is used to guide the residual fusion in the feature fusion stage and simultaneously adaptively supplement the diluted semantic information, and the saliency map reading part compresses and normalizes the output of the above modules along the channel dimension to obtain the final prediction result.

[0131] The construction method, system, terminal and medium of the visual attention prediction model provided by the above embodiments of the present application are different from the direct feature fusion method commonly used in general visual attention prediction methods, and according to the characteristics of the feature extraction network layer and the visual attention mode specific to autism patients, an atypical salient region enhancement technology is proposed to effectively perform residual fusion on cross-order features, so that the model pays more attention to the atypical salient regions that are not initially detected. In addition, the global semantic stream technology is also proposed in the above embodiments of the present application, which extracts context semantic information from the spatial and channel dimensions and guides feature fusion, and thus in the top-down residual fusion process, the atypical salient regions are gradually detected, and the obtained prediction result is more complete.

[0132] The construction method, system, terminal and medium of the visual attention prediction model provided by the above embodiments of the present application can be oriented to the autism group, and through the constructed visual attention prediction model based on atypical salient region enhancement, the visual attention prediction efficiency is high, the cost is low, it is easy to implement, and it is very flexible, and can be deployed on a backbone network with less parameters (considering efficiency) or a backbone network with better performance (considering performance) according to actual needs.

[0133] The flow diagrams in the drawings show the functionality and the operation of a preferred embodiment of methods according to this application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the flow diagram, and combinations of blocks in the flow diagram, can be realized by a dedicated hardware-based system which performs the specified functions or actions, or a combination of hardware- and software-based systems.

[0134] Those skilled in the art will realize that the systems provided by the present application and the various apparatuses thereof can be implemented in a variety of ways, including by means of logic programmed to perform the methods steps described herein, or by means of dedicated hardware, or by a combination of both. Therefore, the systems provided by the present application and the various apparatuses thereof can be viewed as a hardware component, and the means for performing the various functions described herein can be viewed as structures within the hardware component. The means for performing the various functions described herein can also be viewed as software modules that are implemented in a hardware component.

[0135] The details of the above-described embodiments of the present application are not to be taken as limiting the present application. Other variations and modifications of the embodiments disclosed herein can be made based on the concepts disclosed herein without departing from the scope of the present application.

[0136] The specific embodiments of the present application have been described. It is to be understood, however, that based on the overall disclosure provided herein, various modifications and variations can be made by those of ordinary skill in the art in light of the above teachings. It is, therefore, to be understood that changes can be made in form, detail, and / or arrangement of parts without departing from the scope of the application as set forth in the appended claims.

Claims

1. A method for constructing a visual attention prediction model, characterized in that, The application relates to a visual attention prediction model based on atypical salient region enhancement. The visual attention prediction model based on atypical salient region enhancement is pre-trained by using a known eye movement data set, and is modified by using an eye movement data set of an autism group, so that end-to-end training of the visual attention prediction model based on atypical salient region enhancement is completed. The trained visual attention prediction model based on atypical salient region enhancement is tested by using test images in the known eye movement data set, and a final visual attention prediction model is constructed. The visual attention prediction model based on atypical salient region enhancement comprises the following steps. A feature extraction network layer is constructed to extract features of an input image and output multi-order feature maps. A multi-scale enhancement network layer is constructed to perform multi-scale enhancement on a highest-order feature map in the multi-order feature maps, so as to improve the detection ability of salient regions of different scales. An atypical salient region enhancement network layer is constructed to take the highest-order feature map as an initial prediction result, perform residual fusion on the multi-order feature maps from top to bottom, and obtain an enhanced atypical salient region feature map. A global semantic stream network layer is constructed to extract context semantic information of the highest-order feature map from spatial and channel angles respectively, obtain a global semantic stream, and introduce the global semantic stream into the atypical salient region enhancement network layer to guide the residual fusion and adaptively supplement diluted semantic information. A salient map reading network layer is constructed to compress and normalize the enhanced atypical salient region feature map along a channel dimension, and obtain a visual attention prediction result. The feature extraction network layer adopts a full convolution form of a pre-trained object recognition network based on deep learning to construct a backbone network of the feature extraction network layer, so as to extract features of the input image and output multi-order feature maps. The multi-scale enhancement network layer adopts a plurality of parallel convolution layers with different sizes of convolution kernels to explicitly introduce multi-scale information, perform multi-scale enhancement on the highest-order feature map, and use the enhancement result in the residual fusion process to improve the feature extraction ability of salient regions of different scales.

2. The method of claim 1, wherein, The atypical salient region enhancement network layer comprises a background enhancement network layer, a foreground enhancement network layer and a residual fusion network layer. The background enhancement network layer is used to take inverse of high-order features in the multi-order feature maps to obtain background features, and then normalize the background features to obtain a background weight map. The foreground enhancement network layer is used to perform foreground weighting on low-order features adjacent to the high-order features to obtain enhanced low-order features. ​ ​ ​ The residual fusion network layer is used to fuse the enhanced low-order features by using the background weight map to obtain residual features; and the high-order features and the residual features are adaptively fused to obtain a new prediction result; The new prediction result is taken as a new high-order feature, and the residual fusion is continued with adjacent low-order features until a final enhanced atypical salient region feature map is obtained; The global semantic stream network layer includes a channel enhancement network layer and a spatial position enhancement network layer; the context semantic information of the feature map is extracted from the spatial and channel angles respectively to obtain a global semantic stream, which includes: The highest-order feature in the feature map is transformed and compressed in the channel number by a convolution layer to obtain a network layer input feature; The channel enhancement network layer uses global average pooling to obtain global priori, and the global priori is transformed and normalized by a 1*1 convolution layer to obtain a channel weight map of the network layer input feature; and the channel weight map is used to enhance the channel of the network layer input feature to obtain a channel enhanced feature; The spatial position enhancement network layer uses a self-attention mechanism to fully capture the correlation between pixels of the network layer input feature to obtain a spatial position weight map; and the spatial position weight map is used to enhance the position of the network layer input feature to obtain a position enhanced feature; The channel enhanced feature and the position enhanced feature are fused to obtain a global semantic stream; The global semantic stream is introduced into the atypical salient region enhancement network layer, and the weight of the global semantic stream is adaptively adjusted to adaptively supplement global information in the residual fusion process; The saliency map reading network layer includes a 3*3 convolution layer and a sigmoid activation function. 3.The method of claim 1, wherein, The atypical salient region enhanced visual attention prediction model is pre-trained by using a known eye movement data set, and the atypical salient region enhanced visual attention prediction model is corrected by using an eye movement data set of an autism group to complete end-to-end training of the atypical salient region enhanced visual attention prediction model, including: The atypical salient region enhanced visual attention prediction model is pre-trained by using a public eye movement data set SALICON and MIT1003, and the atypical salient region enhanced visual attention prediction model is corrected by using an eye movement data set Saliency4ASD of an autism group; The initialization parameters of the atypical salient region enhanced visual attention prediction model are set; The loss function of the atypical salient region enhanced visual attention prediction model is determined; The related hyperparameters in the atypical salient region enhanced visual attention prediction model are determined; The end-to-end training of the atypical salient region enhanced visual attention prediction model is completed through the above steps.

4. The method of claim 3, wherein, Further comprising any one or any multiple of the following: - the pre-training and modification of the visual attention prediction model based on atypical salient region enhancement using the public eye movement dataset SALICON and MIT1003 and the eye movement dataset Saliency4ASD of the autism population, comprising: obtaining the public eye movement dataset SALICON and MIT1003 and the eye movement dataset Saliency4ASD of the autism population, clustering the eye movement position sampling points of the image data in the eye movement dataset to generate a mat file containing the fixation points; normalizing the mat file to generate a fixation point density map as the ground truth map; using the images of the eye movement dataset SALICON and MIT1003 as model input and the ground truth map corresponding to the images of the eye movement dataset SALICON and MIT1003 as label, training the visual attention prediction model based on atypical salient region enhancement in an end-to-end manner, so that the model automatically learns the mapping relationship between the original image and the ground truth map, and obtains the feature distribution related to the human eye; using the images of the eye movement dataset Saliency4ASD of the autism population as model input and the ground truth map corresponding to the images of the eye movement dataset Saliency4ASD of the autism population as label, fine-tuning the visual attention prediction model based on atypical salient region enhancement in an end-to-end manner, so that the model automatically learns the mapping relationship between the original image and the ground truth map, and obtains the eye movement features of the autism population, and modifies the model; - setting the initialization parameters of the visual attention prediction model based on atypical salient region enhancement, comprising: the visual attention prediction model based on atypical salient region enhancement comprises a feature extraction network layer, a multi-scale enhancement network layer, an atypical salient region enhancement network layer, a global semantic stream network layer, and a saliency map reading network layer; wherein: the feature extraction network layer uses the parameters obtained by pre-training on the ImageNet dataset as the initialization parameters; the initial parameters of the other network layers are random initialization parameters; - determining the loss function of the visual attention prediction model based on atypical salient region enhancement, comprising: the loss function adopts a weighted linear combination of three saliency performance evaluation indexes KL, CC, and NSS; - determining the related hyperparameters in the visual attention prediction model based on atypical salient region enhancement, comprising: The random gradient descent algorithm is adopted in the pre-training process, and the initial learning rate is 10 -4 , and the Batchsize size is 10, and the pre-training process needs to iterate 20 epochs until the model converges. 5.The method of claim 1, wherein, testing the trained visual attention prediction model based on atypical salient region enhancement using test images in known eye movement datasets, comprising: testing the trained visual attention prediction model based on atypical salient region enhancement using the benchmark provided in the public datasets SALICON, MIT1003, and Saliency4ASD, so as to evaluate the performance of the model. 6.A system for constructing a visual attention prediction model, comprising: comprising: a prediction model construction module configured to construct a visual attention prediction model enhanced by atypical salient region; a model training module configured to pre-train the visual attention prediction model enhanced by atypical salient region using a known eye movement dataset, and correct the visual attention prediction model enhanced by atypical salient region using an eye movement dataset of an autism population, to complete end-to-end training of the visual attention prediction model enhanced by atypical salient region; a model testing module configured to test the trained visual attention prediction model enhanced by atypical salient region using known test images, and evaluate the performance of the constructed model; wherein: the prediction model construction module is configured to construct a visual attention prediction model enhanced by atypical salient region, comprising: a feature extraction network layer configured to extract features of an input image and output multi-order feature maps; a multi-scale enhancement network layer configured to perform multi-scale enhancement on a highest-order feature map in the multi-order feature maps, to improve the detection ability of salient regions of different scales; an atypical salient region enhancement network layer configured to take the highest-order feature map as an initial prediction result, and perform residual fusion on the multi-order feature maps from top to bottom, to obtain an enhanced atypical salient region feature map; a global semantic stream network layer configured to extract context semantic information of the highest-order feature map from spatial and channel perspectives respectively, to obtain a global semantic stream, and introduce the global semantic stream into the atypical salient region enhancement network layer, to guide the residual fusion and adaptively supplement diluted semantic information; a saliency map reading network layer configured to compress and normalize the enhanced atypical salient region feature map along the channel dimension, to obtain a visual attention prediction result.

7. A visual attention prediction method characterized by, The visual attention prediction model constructed by the method of any one of claims 1-5 or the system of claim 6 is used to input any image into the model, and output a visual attention prediction result.

8. A computer terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method of any one of claims 1-5 or 7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-5 or 7.

Images