A method and system for visual saliency prediction based on synthesized images
Patent Information
- Application Number
- CN202311371569.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-23
- Publication Date
- 2026-10-30
- Estimated Expiration
- 2043-10-23
AI Technical Summary
由于音视频作为长时间有序序列，其相邻帧之间具有必然的关联性和联系性，但是虚拟人脸图像是由生成模型或三维模型渲染得到，与真实人脸图像具有不同的特征分布，因此目前的方法不适用于虚拟人脸图像的显著性预测，无法有效提取虚拟图像的多层特征
[0036]通过上述技术方案，采用基于Swim网络的模型对合成图像进行特征提取处理，提取合成图像的多尺度特征块，并将合成图像的像素位置坐标在传递至偏移网络之前映射至预设的高维空间，确定合成图像的空间位置坐标的正弦编码，显著提高表示细粒度细节的性能，提高模型的表征能力；采用信道对准方式和渐进求和方式对合成图像的多尺度特征块和空间位置坐标的正弦编码的特征映射进行融合处理，有效提高合成人脸显著性预测的性能，并有效预测合成人脸图像的显著性分布。
Smart Images
Figure CN117437139B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of multimedia visual saliency technology, and more specifically, to a method and system for visual saliency prediction based on synthetic images. Background Technology
[0002] In deep learning tasks, the biggest challenge is not choosing the right model, but finding suitable learning data. With the rapid development and widespread use of the internet, many application areas, including internet telephony, video conferencing, and distance education, face multimedia data processing challenges. Particularly in the field of human-related computer vision, collecting facial videos is particularly challenging due to privacy concerns and the difficulty in comprehensively covering a wide range of scenarios. Recently, synthetic data has emerged as a promising approach for simulating real-world scenarios and creating data repositories for challenging problems. By generating synthetic data instead of collecting and labeling real data, perfect labels can be guaranteed without any annotation noise. It also generates rich labels that are impossible to obtain manually, allowing for complete control over the variability and diversity of the dataset. For example, real-world traffic scenarios can capture more natural driving behaviors of autonomous driving systems, such as safety situations and unavoidable accidents in the physical world.
[0003] Synthetic data can help overcome restrictions related to regulated or sensitive data and allows data to be customized to match conditions that real data does not permit. Furthermore, synthetic data can facilitate compliance with data privacy regulations; its anonymity determines its suitability for dissemination, which is a significant advantage for constructing face datasets.
[0004] Human saliency refers to human visual attention to specific regions or objects in an image or video. Based on the inherent characteristics of the human visual system, it tends to focus on unique regions with distinctive features. The significance of human saliency lies in its ability to mimic human attention and provide valuable insights into how humans perceive and interact with their environment. Notably, an increasing number of face-related applications require the identification of attentional cues in facial videos. For example, in video conferencing, focusing on salient areas can optimize the audio and visual quality of conversations while improving the effectiveness of the communication system; for beauty cameras, identifying visually appealing areas can enhance target facial features, creating aesthetically pleasing overall images. From a technical perspective, human saliency can be used to identify and track key facial landmarks, including the eyes, mouth, nose, and eyebrows. These features are crucial for emotional expression and the communication of cues. More specifically, human attention to synthetic faces has many graphical applications, such as mesh simplification. When a facial mesh is expressed, it can guide mesh simplification to retain more triangles around salient areas and fewer triangles in other less noticeable areas, reducing the computationally burdensome rendering load while maintaining visual appeal.
[0005] In the context of analyzing visual attention in facial images, research shows that visual attention attracted by facial features increases with facial size; dynamic changes in attention distribution in monitored facial videos reveal variations in face and facial feature attentional distribution with face size and mouth movement; and sound plays a significant role in human attention during speech transitions in multi-face videos. Researchers analyzed their saliency database and found that audio influences people's gaze and the shift of their attention between faces.
[0006] Existing facial saliency databases typically consist of images collected from the internet, often displaying random facial expressions that are difficult to annotate with appropriate labels indicating specific emotions. One approach to address this challenge is to establish photo sessions with models and photographers to capture images of a variety of facial expressions. However, this method also introduces other problems; environmental factors such as lighting and weather can negatively impact image quality during the photographic process. Furthermore, the accuracy of the facial expressions depicted in these images is limited by the model's own capabilities and movement.
[0007] Current research on face saliency primarily focuses on saliency prediction and analysis methods for real human images and videos. Since audio and video are long-term ordered sequences, adjacent frames inherently possess correlation and connection. However, virtual face images are rendered from generative or 3D models and have different feature distributions than real face images. Therefore, current methods are not suitable for saliency prediction of virtual face images and cannot effectively extract multi-layered features from virtual images. Summary of the Invention
[0008] In view of the deficiencies in the prior art, the purpose of this disclosure is to provide a method and system for visual saliency prediction based on synthetic images.
[0009] To achieve the above objectives, according to one aspect of this disclosure, a method for visual saliency prediction based on synthetic images is provided, comprising:
[0010] Feature extraction processing is performed on the RGB image of the synthesized image to determine the multi-scale feature blocks of the synthesized image;
[0011] The pixel position coordinates of the synthesized image are encoded to determine the sinusoidal encoding of the spatial position coordinates of the synthesized image;
[0012] The multi-scale feature blocks of the synthesized image and the sinusoidal encoded feature maps of the spatial location coordinates are fused using a channel alignment method and an asymptotic summation method to determine the saliency image of the predicted synthesized image.
[0013] Optionally, the step of performing feature extraction processing on the RGB image of the synthesized image to determine the multi-scale feature blocks of the synthesized image includes:
[0014] The Swin-B network is initialized using a preset dataset and pre-trained weights, and the initialized Swin-B network is determined.
[0015] The RGB image of the synthesized image is input into the patch partition of the initialized Swin-B network to determine the non-overlapping image patches of the synthesized image;
[0016] Each non-overlapping image patch of the synthesized image is marked and concatenated with the RGB values of the original pixels of the synthesized image;
[0017] The non-overlapping image patch is fed into a Swing-Transformer block with a self-attention mechanism to output the multi-scale features of the synthesized image.
[0018] Optionally, the step of encoding the pixel position coordinates of the synthesized image to determine the sinusoidal encoding of the spatial position coordinates of the synthesized image includes:
[0019] The pixel position coordinates of the synthesized image are mapped to a preset high-dimensional space to determine the sinusoidal encoding of the spatial position coordinates of the synthesized image. The sinusoidal encoding represents a vector of sine curves with different frequencies.
[0020] Optionally, the method further includes:
[0021] The sinusoidal encoding of the spatial location coordinates of the synthesized image is passed to the offset network.
[0022] Optionally, the method further includes:
[0023] The sinusoidal encoding of the spatial location coordinates of the synthesized image passed to the offset network is converted into a feature map of the sinusoidal encoding of the spatial location coordinates.
[0024] Optionally, the step of fusing the multi-scale features of the synthesized image and the sinusoidal coding of the spatial location using a channel alignment method and an asymptotic summation method to determine the saliency image of the predicted synthesized image includes:
[0025] The multi-scale feature blocks of the synthesized image are integrated using the channel alignment method and the progressive summation method to determine the summation features of the synthesized image.
[0026] The summation features of the synthesized image and the sinusoidal encoded feature map of the spatial location coordinates of the synthesized image are fused together to generate a predicted saliency image of the synthesized image.
[0027] Optionally, the step of fusing the multi-scale features of the synthesized image and the sinusoidal coding of the spatial location coordinates using channel alignment and progressive summation methods to determine the saliency image of the predicted synthesized image further includes:
[0028] The multi-scale feature blocks of the synthesized image are processed using a preset resolution processing method to determine the multi-scale feature blocks of the synthesized image after resolution processing.
[0029] Optionally, the resolution processing involves adjusting the size of the multi-scale feature blocks of the synthesized image to a preset size.
[0030] Optionally, the multi-scale feature block includes shallow blocks and deep blocks, wherein the shallow blocks are used to capture image texture features and / or patch color features, and the deep blocks are used to learn image content features of the synthesized image.
[0031] According to a second aspect of this disclosure, a visual saliency prediction system based on synthetic images is provided, comprising:
[0032] The depth feature extraction module is used to perform feature extraction processing on the RGB image of the synthesized image to determine the multi-scale feature blocks of the synthesized image;
[0033] The channel compensation module is used to encode the pixel position coordinates of the synthesized image to determine the sinusoidal code of the spatial position coordinates of the synthesized image;
[0034] The target prediction module is used to fuse the feature maps of the multi-scale feature blocks of the synthetic image and the sinusoidal encoding of the spatial location coordinates using a channel alignment method and an incremental summation method to determine the saliency image of the predicted synthetic image.
[0035] Compared with the prior art, the embodiments disclosed herein have at least one of the following beneficial effects:
[0036] The above technical solution employs a Swim network-based model to extract features from the synthesized image, extracting multi-scale feature blocks and mapping the pixel coordinates of the synthesized image to a preset high-dimensional space before passing them to the offset network. This determines the sinusoidal encoding of the spatial coordinates of the synthesized image, significantly improving the performance in representing fine-grained details and enhancing the model's representational ability. Furthermore, the channel alignment method and the progressive summation method are used to fuse the feature mappings of the multi-scale feature blocks and the sinusoidal encoding of the spatial coordinates of the synthesized image, effectively improving the performance of synthesized face saliency prediction and effectively predicting the saliency distribution of the synthesized face image. Attached Figure Description
[0037] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0038] Figure 1 This is a flowchart illustrating a visual saliency prediction method based on synthetic images according to an exemplary embodiment.
[0039] Figure 2 This is an overall flowchart illustrating a visual saliency prediction method based on synthetic images, according to an exemplary embodiment.
[0040] Figure 3 This is a block diagram illustrating a visual saliency prediction system based on synthetic images, according to an exemplary embodiment. Detailed Implementation
[0041] The present disclosure will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present disclosure, but do not limit the present disclosure in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present disclosure. These all fall within the protection scope of the present disclosure.
[0042] The present invention provides a method for predicting the visual saliency of synthetic images, which can predict the visual saliency distribution of rendered face images and generated face model images.
[0043] Figure 1 This is a flowchart illustrating a visual saliency prediction method based on synthetic images according to an exemplary embodiment. Figure 2 This is an overall flowchart illustrating a visual saliency prediction method based on synthetic images, according to an exemplary embodiment.
[0044] like Figure 1 , Figure 2 As shown, a visual saliency prediction method based on synthetic images includes S11 to S13.
[0045] S11, perform feature extraction processing on the RGB image of the synthesized image to determine the multi-scale feature blocks of the synthesized image.
[0046] In one possible embodiment, feature extraction processing is performed on the RGB image of the synthesized image to determine the multi-scale feature blocks of the synthesized image, including S21 to S24.
[0047] S21. Initialize the Swin-B network using a preset dataset and pre-trained weights to determine the initialized Swin-B network.
[0048] The preset dataset can be the ImageNet dataset.
[0049] S22, input the RGB image of the synthesized image into the patch partition of the initialized Swin-B network to determine the non-overlapping image patches of the synthesized image.
[0050] For example, the patch partition can be a Vision Transformer.
[0051] Non-overlapping image patches are those that do not overlap with each other.
[0052] S23, each non-overlapping image patch of the synthesized image is marked and concatenated with the RGB values of the original pixels of the synthesized image.
[0053] As an example, each image patch can be viewed as a "tag" and each image patch can be encoded as a concatenation of the RGB values of the original pixels of the composite image, that is, concatenated with the RGB values of the original pixels of the composite image.
[0054] S24 feeds the non-overlapping image patch into the Swin-Transformer block with a self-attention mechanism to output the multi-scale features of the synthesized image.
[0055] For example, embedded non-overlapping image patches are fed into a Swin-Transformer block with a modified self-attention mechanism in four sub-Stage modules, and multi-scale features of the synthesized image are extracted within the four sub-Stage modules.
[0056] The multi-scale feature blocks include shallow blocks and deep blocks. Shallow blocks are used to capture image texture features and / or patch color features, while deep blocks are used to learn image content features of the synthesized image. Thus, a top-down perception mechanism is used to generate a set of feature blocks for predicting the target.
[0057] S12, encode the pixel position coordinates of the synthesized image to determine the sinusoidal encoding of the spatial position coordinates of the synthesized image.
[0058] In this process, the pixel coordinates of the synthesized image are encoded, that is, before the pixel coordinates of the synthesized image are passed to the prediction network, the pixel coordinates of the synthesized image are mapped to a higher-dimensional space.
[0059] In this disclosure, the pixel position coordinates of the synthesized image can be mapped to a preset high-dimensional space to determine the sinusoidal encoding of the spatial position coordinates of the synthesized image. The sinusoidal encoding represents a vector of sine curves with different frequencies.
[0060] In one possible embodiment, the method for predicting the visual saliency of synthetic images further includes:
[0061] The sinusoidal encoding of the spatial location coordinates of the synthesized image is passed to the offset network.
[0062] In another possible embodiment, the method for predicting the visual saliency of synthetic images further includes:
[0063] After passing the sinusoidal encoding of the spatial location coordinates of the synthesized image to the offset network, the sinusoidal encoding of the spatial location coordinates of the synthesized image passed to the offset network is converted into a feature map of the sinusoidal encoding of the spatial location coordinates.
[0064] This disclosure aggregates the feature maps of the spatial location coordinates of the synthesized image with the feature images of the synthesized image, thereby significantly improving the performance of representing fine-grained details and obtaining more effective saliency prediction results.
[0065] In some possible embodiments, following the example above, the short-time features of the video signal and the short-time features of the audio signal are input into the gated recurrent neural network in time sequence to learn the correlation between adjacent video frames and obtain the long-time features of the video signal and the long-time features of the audio signal, respectively.
[0066] S13, the feature maps of the multi-scale feature blocks and the sinusoidal coding of the spatial location coordinates of the synthetic image are fused using the channel alignment method and the progressive summation method to determine the saliency image of the predicted synthetic image.
[0067] In one possible embodiment, S13 includes S31 to S32.
[0068] S31, the multi-scale feature blocks of the synthesized image are integrated using channel alignment and progressive summation methods to determine the total features of the synthesized image.
[0069] S32, the summation features of the synthesized image and the feature mapping of the sinusoidal encoding of the spatial location coordinates of the synthesized image are fused to generate the saliency image of the predicted synthesized image.
[0070] Specifically, the summation features of the synthesized image and the feature mapping of the sinusoidal encoding of the spatial location coordinates of the synthesized image are fused to obtain comprehensive features. The comprehensive features are then passed through a fusion layer to generate a prediction result with integrated features, and the saliency image of the predicted synthesized image is output.
[0071] In another possible embodiment, S13 further includes S33.
[0072] S33, a preset resolution processing method is used to process the multi-scale feature blocks of the synthesized image to determine the multi-scale feature blocks of the synthesized image after resolution processing.
[0073] In this disclosure, step S33 is performed before step S31.
[0074] The preset resolution processing method includes the pixel shuffle method. Resolution processing involves adjusting the size of the multi-scale feature blocks of the synthesized image to a preset size to facilitate subsequent saliency prediction.
[0075] The above technical solution employs a Swim network-based model to extract features from the synthesized image, extracting multi-scale feature blocks and mapping the pixel coordinates of the synthesized image to a preset high-dimensional space before passing them to the offset network. This determines the sinusoidal encoding of the spatial coordinates of the synthesized image, significantly improving the performance in representing fine-grained details and enhancing the model's representational ability. Furthermore, the channel alignment method and the progressive summation method are used to fuse the feature mappings of the multi-scale feature blocks and the sinusoidal encoding of the spatial coordinates of the synthesized image, effectively improving the performance of synthesized face saliency prediction and effectively predicting the saliency distribution of the synthesized face image.
[0076] To verify the effectiveness of the visual saliency prediction method based on synthetic images disclosed herein, the visual saliency prediction method based on synthetic images disclosed herein was compared with several state-of-the-art saliency prediction methods on the constructed MetaFace dataset. The saliency prediction methods used included SalNet, UNet, MobileNet, ResNet, DenseNet, PNASNet, TransalNet_Res, and TransalNet_Dense.
[0077] The images in the MetaFace dataset are randomly divided into two parts: the first part, consisting of 80% of the images, serves as the training set, and the second part, consisting of 20% of the images, serves as the test set.
[0078] The original image size was 1920*1080, which was adjusted to 512*288 according to a preset ratio. All the methods described above were implemented using an NVIDIA GeForce RTX 3090 and an Intel i7 processor. The overall performance of the significance model was quantified using six widely used metrics, including correlation coefficient (CC), similarity (SIM), normalized scan path significance (NSS), area under the ROC curve (AUC), AUC-Judd, and Kullback-Leibler divergence (KLD).
[0079]
[0080] Table 1
[0081] As shown in Table 1, Table 1 presents the quantitative performance of the visual saliency prediction method based on synthetic images and other methods on the dataset, with the best results highlighted in bold. The method of this disclosure achieves scores of 0.857, 0.726, and 4.011 in CC, SIM, and NSS, respectively, ranking highest among all compared methods in terms of perception-based metrics. Furthermore, for non-perception-based metrics, the method of this disclosure demonstrates competitive performance in KLD and outperforms other methods in AUC and AUC-J, although there are no significant differences among all methods in AUC and AUC-J. Comparative analysis shows that the reconstruction results of this disclosure are closer to the true labeled results.
[0082] Figure 3 This is a block diagram illustrating a visual saliency prediction system based on synthetic images, according to an exemplary embodiment.
[0083] Based on the same concept, this disclosure also provides a visual saliency prediction system based on synthetic images, referring to... Figure 3 A visual saliency prediction system 100 based on synthetic images includes a depth feature extraction module 110, a channel compensation module 120, and a target prediction module 130.
[0084] The depth feature extraction module 110 is used to perform feature extraction processing on the RGB image of the synthesized image to determine the multi-scale feature blocks of the synthesized image;
[0085] The channel compensation module 120 is used to encode the pixel position coordinates of the synthesized image to determine the sinusoidal encoding of the spatial position coordinates of the synthesized image.
[0086] The target prediction module 130 is used to perform fusion processing on the multi-scale feature blocks of the synthetic image and the sinusoidal encoded feature maps of the spatial location coordinates using a channel alignment method and an incremental summation method to determine the saliency image of the predicted synthetic image.
[0087] In this disclosure, the deep feature extraction module 110 is used to extract semantic features of different scales of the synthesized image; the channel compensation module 120 is used to supplement the semantic features of different scales as compensation information; and the target prediction module 130 is used to reconstruct the semantic features of different scales into saliency prediction results through channel alignment and progressive summation.
[0088] Regarding the embodiments of the above system, the specific ways in which each module performs operations have been described in detail in the embodiments of the method, and will not be elaborated here.
[0089] The specific embodiments of this disclosure have been described above. It should be understood that this disclosure is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this disclosure. The above-described preferred features can be used in any combination without conflict.
Claims
1. A method for visual saliency prediction based on synthetic images, characterized in that, include: Feature extraction processing is performed on the RGB image of the synthesized image to determine the multi-scale feature blocks of the synthesized image; The pixel position coordinates of the synthesized image are encoded to determine the sinusoidal encoding of the spatial position coordinates of the synthesized image; The multi-scale feature blocks of the synthetic image and the sinusoidal encoded feature maps of the spatial location coordinates are fused using a channel alignment method and an incremental summation method to determine the saliency image of the predicted synthetic image. The method further includes: passing the sinusoidal encoding of the spatial position coordinates of the synthesized image to an offset network; and converting the sinusoidal encoding of the spatial position coordinates of the synthesized image passed to the offset network into a feature map of the sinusoidal encoding of the spatial position coordinates. The process of fusing the multi-scale features of the synthesized image and the sinusoidal encoded feature mapping of the spatial location using channel alignment and progressive summation methods to determine the saliency image of the predicted synthesized image includes: The multi-scale feature blocks of the synthesized image are integrated using the channel alignment method and the progressive summation method to determine the summation features of the synthesized image. The summation features of the synthesized image and the sinusoidal encoded feature mapping of the spatial location coordinates of the synthesized image are fused to generate a predicted saliency image of the synthesized image. The multi-scale feature blocks include shallow blocks and deep blocks; The feature mappings of the deep block after resolution processing and the sinusoidal encoding of the spatial location coordinates are added and fused to determine the first fused feature; The first fused feature is added to and fused with the next-level feature block that has undergone the resolution processing to determine the second fused feature. The process is repeated step by step until the shallow blocks that have undergone the resolution processing are added and fused together to determine the total features of the synthesized image.
2. The method according to claim 1, characterized in that, The step of performing feature extraction processing on the RGB image of the synthesized image to determine the multi-scale feature blocks of the synthesized image includes: The Swin-B network is initialized using a preset dataset and pre-trained weights, and the initialized Swin-B network is determined. The RGB image of the synthesized image is input into the patch partition of the initialized Swin-B network to determine the non-overlapping image patches of the synthesized image; Each non-overlapping image patch of the synthesized image is marked and concatenated with the RGB values of the original pixels of the synthesized image; The non-overlapping image patch is fed into a Swing-Transformer block with a self-attention mechanism to output the multi-scale features of the synthesized image.
3. The method according to claim 1, characterized in that, The step of encoding the pixel position coordinates of the synthesized image to determine the sinusoidal encoding of the spatial position coordinates of the synthesized image includes: The pixel position coordinates of the synthesized image are mapped to a preset high-dimensional space to determine the sinusoidal encoding of the spatial position coordinates of the synthesized image. The sinusoidal encoding represents a vector of sine curves with different frequencies.
4. The method according to claim 1, characterized in that, The step of fusing the multi-scale features of the synthesized image and the sinusoidal coding of the spatial location coordinates using channel alignment and progressive summation methods to determine the saliency image of the predicted synthesized image further includes: The multi-scale feature blocks of the synthesized image are processed using a preset resolution processing method to determine the multi-scale feature blocks of the synthesized image after resolution processing.
5. The method according to claim 4, characterized in that, The resolution processing involves adjusting the size of the multi-scale feature blocks of the synthesized image to a preset size.
6. The method according to claim 1, characterized in that, The multi-scale feature block includes shallow blocks and deep blocks. The shallow blocks are used to capture image texture features and / or patch color features, while the deep blocks are used to learn image content features of the synthesized image.
7. A visual saliency prediction system based on synthetic images, employing the visual saliency prediction method based on synthetic images as described in any one of claims 1-6, characterized in that, include: The depth feature extraction module is used to perform feature extraction processing on the RGB image of the synthesized image to determine the multi-scale feature blocks of the synthesized image; The channel compensation module is used to encode the pixel position coordinates of the synthesized image to determine the sinusoidal code of the spatial position coordinates of the synthesized image; The target prediction module is used to fuse the feature maps of the multi-scale feature blocks of the synthetic image and the sinusoidal encoding of the spatial location coordinates using a channel alignment method and an incremental summation method to determine the saliency image of the predicted synthetic image.