Model training method, human eye gaze region prediction method and device

Abstract

The application provides a model training method, a human eye gaze area prediction method and device, wherein the model training method comprises the following steps: obtaining a sample two-view video data set, the sample two-view video data set comprising a plurality of sample two-view video data shot in a three-dimensional dynamic scene; training an initial human eye gaze area prediction model based on the sample two-view video data set to determine a preset human eye gaze area prediction model; wherein the initial human eye gaze area prediction model comprises a bottom-up attention mechanism submodel established based on a convolutional neural network and a top-down attention mechanism submodel established based on a long short-term memory network. The application achieves the purpose of establishing a visual attention mechanism model suitable for a three-dimensional dynamic scene, greatly improves the prediction accuracy of the human eye gaze area prediction result, and also greatly improves the application range of the preset human eye gaze area prediction model obtained by training.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a model training method, a method and apparatus for predicting the human eye's gaze region. Background Technology

[0002] As is well known, a human eye fixation region prediction model is a model that predicts the region that the human eye may fixate on based on image or video information and the human eye's attention mechanism. Furthermore, using a human eye fixation region prediction model incorporating the human eye's attention mechanism not only allows limited computational resources to be allocated to important targets during processing but also produces results that meet the requirements of human visual cognition. Therefore, how to construct the prediction results for the human eye fixation region in a human eye fixation region prediction model is crucial.

[0003] In related technologies, a human eye gaze region prediction model containing a bottom-up human eye attention mechanism is used to predict the human eye gaze region of human eye gaze image data collected in a two-dimensional scene, and the human eye gaze region prediction result output by the human eye gaze region prediction model can be obtained.

[0004] However, existing human eye gaze region prediction models are only applicable to two-dimensional scenes, and the human eye gaze region prediction process only focuses on color information in the human eye gaze image data, resulting in limited information content. This leads to low prediction accuracy and limited applicability of the human eye gaze region prediction results. Summary of the Invention

[0005] This invention provides a model training method, a method and apparatus for predicting human eye gaze regions, to address the shortcomings of existing human eye gaze region prediction models, which are only applicable to two-dimensional scenes and only focus on color information in the human eye gaze image data during the prediction process, resulting in low prediction accuracy and limited applicability. By training a bottom-up attention mechanism sub-model based on a convolutional neural network and a top-down attention mechanism sub-model based on a long short-term memory network, this invention not only achieves the goal of establishing a visual attention mechanism model applicable to three-dimensional dynamic scenes, but also ensures that when predicting the human eye gaze region in three-dimensional dynamic scenes, it can pay attention to multiple aspects of information such as color and depth, thereby significantly improving the prediction accuracy of the human eye gaze region prediction results and also significantly expanding the applicability of the trained preset human eye gaze region prediction model.

[0006] This invention provides a model training method, comprising: Obtain a sample two-viewpoint video dataset, which includes multiple sample two-viewpoint video data captured in a three-dimensional dynamic scene; Based on the sample two-viewpoint video dataset, the initial human eye gaze region prediction model is trained, and the preset human eye gaze region prediction model is determined. The initial human eye gaze region prediction model includes a bottom-up attention mechanism sub-model based on a convolutional neural network and a top-down attention mechanism sub-model based on a long short-term memory network.

[0007] According to a model training method provided by the present invention, before training an initial human eye gaze region prediction model based on the sample two-viewpoint video dataset to determine a preset human eye gaze region prediction model, the method further includes: Determine the sample human eye gaze density image set corresponding to the sample two-viewpoint human eye gaze coordinate dataset for each of the sample two-viewpoint video data; Determine the linear correlation coefficient between every two consecutive sample human eye fixation density images in the sample human eye fixation density image set; Determine the object motion intensity sorting sequence set for each sample of two-viewpoint video data and the pixel gaze probability sorting sequence set for each sample of human eye gaze density image set; Based on the linear correlation coefficient, the object motion intensity ranking sequence set, and the pixel gaze probability ranking sequence set, the sample two-viewpoint video dataset is determined to be a dataset that meets the model training requirements.

[0008] According to a model training method provided by the present invention, the step of training an initial human eye gaze region prediction model based on the sample two-viewpoint video dataset to determine a preset human eye gaze region prediction model includes: The initial human eye fixation region prediction model is trained using the sample two-viewpoint video dataset, and the intermediate model loss of the intermediate human eye fixation region prediction model after a preset number of training iterations is obtained. Based on the successful matching result between the intermediate model loss and the preset model loss, the intermediate human eye gaze region prediction model corresponding to the intermediate model loss is determined as the preset human eye gaze region prediction model.

[0009] According to a model training method provided by the present invention, the step of training the initial human eye gaze region prediction model using the sample two-viewpoint video dataset and obtaining the intermediate model loss of the intermediate human eye gaze region prediction model after a preset number of training iterations includes: For each training session, the intermediate bottom-up attention mechanism sub-model after the previous training is trained based on the two-view video data of the sample participating in this training session, and the multi-scale spatial fusion feature image set of the sample output by the intermediate bottom-up attention mechanism sub-model after this training session is determined. Based on the sample multi-scale spatial fusion feature image set, the intermediate top-down attention mechanism sub-model after the previous training is trained, and the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model after this training is determined. Based on the sample human eye gaze region prediction result set and the sample human eye gaze density image set corresponding to the sample two-viewpoint human eye gaze dataset of the sample two-viewpoint video data, the intermediate model loss of the intermediate human eye gaze region prediction model after this training is determined.

[0010] According to a model training method provided by the present invention, determining the sample two-viewpoint video dataset as a dataset that meets the model training requirements based on the linear correlation coefficient, the object motion intensity ranking sequence set, and the pixel gaze probability ranking sequence set includes: The linear correlation coefficient is matched with a preset coefficient threshold, and the degree of agreement between the object motion intensity sorting sequence set and the pixel gaze probability sorting sequence set is matched with a preset degree of agreement threshold. Based on the successful matching results of the linear correlation coefficient and the preset coefficient threshold, and the successful matching results of the consistency degree and the preset consistency threshold, the sample two-viewpoint video dataset is determined to be a dataset that meets the model training requirements.

[0011] According to a model training method provided by the present invention, the step of training a previously trained bottom-up attention mechanism sub-model based on the two-viewpoint video data of the samples participating in the current training, and determining the multi-scale spatial fusion feature image set of the samples output by the currently trained bottom-up attention mechanism sub-model, includes: The sample two-viewpoint video data participating in this training is input into the intermediate bottom-up attention mechanism sub-model after the previous training. Sample depth images are extracted from the sample two-viewpoint video frame image data. The sample depth images and the corresponding sample two-viewpoint video frame image data are convolved and fused multiple times to determine the sample multi-scale spatial fusion feature image set output by the intermediate bottom-up attention mechanism sub-model after this training.

[0012] According to a model training method provided by the present invention, the step of training the previously trained intermediate top-down attention mechanism sub-model based on the sample multi-scale spatial fusion feature image set, and determining the sample human eye gaze region prediction result set output by the currently trained intermediate top-down attention mechanism sub-model, includes: Obtain the intermediate memory vector and intermediate human eye gaze region prediction results of the intermediate top-down attention mechanism sub-model after the previous training; The sample multi-scale spatial fusion feature image set, the intermediate memory vector, and the intermediate human eye gaze region prediction result are input into the previously trained intermediate top-down attention mechanism sub-model for two layers of weighting and deconvolution operations to determine the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model after this training.

[0013] The present invention also provides a method for predicting the human eye's gaze region, comprising: Acquire target two-viewpoint video data and a preset human eye gaze region prediction model trained according to the aforementioned model training method; The target two-viewpoint video data is input into the preset human eye gaze region prediction model, and the target human eye gaze region prediction result of the target two-viewpoint video data is determined based on the output result of the preset human eye gaze region prediction model.

[0014] The present invention also provides a model training apparatus, comprising: The sample acquisition module is used to acquire a sample two-view video dataset, which includes multiple sample two-view video data captured in a three-dimensional dynamic scene. The model training module is used to train the initial human eye gaze region prediction model based on the sample two-viewpoint video dataset and determine the preset human eye gaze region prediction model. The initial human eye gaze region prediction model includes a bottom-up attention mechanism sub-model based on a convolutional neural network and a top-down attention mechanism sub-model based on a long short-term memory network.

[0015] The present invention also provides a human eye fixation region prediction device, comprising: The test video acquisition module is used to acquire target two-viewpoint video data and a preset human eye gaze region prediction model determined by the aforementioned model training method. The human eye gaze region prediction module is used to input the target two-viewpoint video data into the preset human eye gaze region prediction model, and determine the target human eye gaze region prediction result of the target two-viewpoint video data based on the output result of the preset human eye gaze region prediction model.

[0016] The present invention provides a model training method, a human eye gaze region prediction method, and a device. The model training method involves a terminal device training an initial human eye gaze region prediction model using multiple sample two-viewpoint video data captured in a 3D dynamic scene. This achieves the goal of establishing a visual attention mechanism model applicable to 3D dynamic scenes. Furthermore, since the initial human eye gaze region prediction model includes a bottom-up attention mechanism sub-model based on a convolutional neural network and a top-down attention mechanism sub-model based on a long short-term memory network, training these two sub-models ensures that the prediction of the human eye gaze region in a 3D dynamic scene considers multiple aspects such as color and depth. This significantly improves the prediction accuracy of the human eye gaze region prediction results and also greatly expands the applicability of the trained preset human eye gaze region prediction model. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the model training method provided by the present invention; Figure 2 This is a schematic diagram of the task-driven 3D dynamic scene viewing experiment provided by the present invention; Figure 3 A schematic diagram of the sample dataset provided by this invention; Figure 4 A schematic diagram illustrating the process of multi-scale spatial fusion for the bottom-up attention mechanism sub-model provided by this invention; Figure 5 A schematic diagram of the DRB-based fusion process provided by the present invention; Figure 6 A schematic diagram illustrating the feature extraction process of the convolutional long short-term memory network provided by this invention; Figure 7 This is a schematic diagram of the human eye fixation region prediction results provided by the present invention; Figure 8 This is a flowchart illustrating the human eye fixation region prediction method provided by the present invention; Figure 9 This is a schematic diagram of the structure of the model training device provided by the present invention; Figure 10This is a schematic diagram of the structure of the human eye fixation area prediction device provided by the present invention; Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] The following is combined with Figures 1-11 This invention describes a model training method, a method for predicting human eye gaze regions, and an apparatus. The execution subject of the model training method can be a terminal device, which can be a personal computer (PC), portable device, laptop computer, smartphone, tablet computer, portable wearable device, or other electronic device. It is understood that the execution subject of the model training method can also be a server, which can refer to a single server or a server cluster composed of multiple servers, a cloud computing center, etc. This invention does not limit the specific form of the terminal device or server. The following method embodiments are described using a terminal device as an example.

[0021] Figure 1 This is a flowchart illustrating the model training method provided by the present invention, as shown below. Figure 1 As shown, the model training method includes the following steps: Step 110: Obtain the sample two-viewpoint video dataset, which includes multiple sample two-viewpoint video data captured in a 3D dynamic scene.

[0022] Specifically, each sample two-viewpoint video data in the sample two-viewpoint video dataset can be obtained based on experimental video footage. The experimental video footage can include elements such as the movement of the ball and the person's position, and the change in the person's depth, when shooting two-viewpoint ball-throwing videos using a camera array. The camera array here can be two cameras shooting left-view and right-viewpoint videos, and the camera array can be set up on the terminal device or set up independently on the terminal device and communicate with it. No specific limitations are made here. Furthermore, each sample two-viewpoint video data can be distinguished based on different factors such as the number of people playing ball-throwing, the number of people throwing the ball, and / or video duration; for example, the sample two-viewpoint video data can be a video of two people playing ball-throwing, a video of three people playing ball-throwing, or a video of four people playing ball-throwing. No specific limitations are made here either.

[0023] Step 120: Based on the sample two-viewpoint video dataset, train the initial human eye gaze region prediction model to determine the preset human eye gaze region prediction model.

[0024] The initial human eye fixation region prediction model includes a bottom-up attention mechanism sub-model based on a convolutional neural network and a top-down attention mechanism sub-model based on a long short-term memory network.

[0025] Specifically, in order to improve the prediction effect of the human eye's gaze area, a bottom-up attention mechanism sub-model can be constructed by adding depth information from the three-dimensional scene to the traditional two-dimensional display modeling method. That is, a bottom-up attention mechanism sub-model based on convolutional neural networks.

[0026] Furthermore, each sample's two-viewpoint video data can be used for viewing, and viewers can be assigned viewing tasks. In this case, the viewer's attention mechanism is task-driven; the brain gives commands, and the visual system responds accordingly. Therefore, when modeling the task-driven attention mechanism, it is crucial to ensure that the Long Short-Term Memory (LSTM) network can learn, as much as possible, the object transformation relationships across video frames under task-driven conditions, and the temporal correlation of gazed objects between video frames at the pixel level. This yields a top-down attention mechanism sub-model based on the LTM network.

[0027] At this point, the initial human eye gaze region prediction model is trained using a sample two-viewpoint video dataset until training is complete, and the intermediate human eye gaze region prediction model at the end of training is determined as the preset human eye gaze region prediction model. During training, gradient descent is used to optimize the model, and the learning rate is set to 1e. -3 The weight decays to 5×10 -4 .

[0028] The model training method provided by this invention enables a terminal device to train an initial human eye gaze region prediction model using multiple sample two-viewpoint video data captured in a 3D dynamic scene. This achieves the goal of establishing a visual attention mechanism model applicable to 3D dynamic scenes. Furthermore, since the initial human eye gaze region prediction model includes a bottom-up attention mechanism sub-model based on a convolutional neural network and a top-down attention mechanism sub-model based on a long short-term memory network, training these two sub-models ensures that the prediction of the human eye gaze region in a 3D dynamic scene can consider multiple aspects such as color and depth. This significantly improves the prediction accuracy of the human eye gaze region prediction results and also greatly expands the applicability of the trained preset human eye gaze region prediction model.

[0029] Optionally, prior to step 120, the model training method provided by this invention may further include: First, determine the sample human eye fixation density image set corresponding to the sample two-viewpoint human eye fixation coordinate dataset for each sample two-viewpoint video data set; then determine the linear correlation coefficient between every two consecutive sample human eye fixation density images in the sample human eye fixation density image set; further determine the object motion intensity ranking sequence set for each sample two-viewpoint video data set and the pixel fixation probability ranking sequence set for each sample human eye fixation density image set; then, based on the linear correlation coefficient, the object motion intensity ranking sequence set, and the pixel fixation probability ranking sequence set, determine that the sample two-viewpoint video dataset meets the requirements for model training.

[0030] Specifically, to verify whether the sample two-viewpoint video dataset can be used for subsequent model training, we can first determine the sample human eye fixation density image set corresponding to each sample two-viewpoint video data, and then perform quantitative analysis to determine whether the quantitative analysis results meet the model training requirements. That is, referring to... Figure 2 This is a schematic diagram of a task-driven 3D dynamic scene viewing experiment provided by the present invention. Figure 2In this experiment, where each sample of two-viewpoint video data is acquired by capturing two-viewpoint ball-throwing videos using a camera array, the experimenter can assign viewing tasks to viewers of each sample's two-viewpoint video data. The number of viewers can be multiple, for example, up to 20. The viewing task for each viewer can be to instruct them to determine the number of times the person throwing the ball does so within the duration of watching the corresponding sample's two-viewpoint video data. That is, when a sample of two-viewpoint video data depicts two people playing ball-throwing, the viewing task can be to determine the number of times one of the two people throws the ball; when another sample of two-viewpoint video data depicts three people playing ball-throwing, the viewing task can be to determine the number of times one of the three people throws the ball; and when yet another sample of two-viewpoint video data depicts four people playing ball-throwing, the viewing task can be to determine the number of times one of the four people throws the ball, or the number of times another of the four people throws the ball.

[0031] Based on this, the terminal device can instruct the eye tracker to collect sample two-viewpoint eye fixation coordinate datasets for each viewer watching different sample two-viewpoint video data for different tasks. Then, by performing visualization data processing on each sample two-viewpoint eye fixation coordinate dataset, the corresponding sample eye fixation density image set can be determined. Each sample two-viewpoint video data contains one sample two-viewpoint eye fixation coordinate dataset and one corresponding sample eye fixation density image set. The number of images in each two-viewpoint video frame is the same as the number of sample eye fixation density images in the corresponding sample eye fixation density image set, and they correspond one-to-one. Therefore, a sample dataset representing a one-to-one correspondence between the sample two-viewpoint video data and the sample eye fixation density image set can be established, such as... Figure 3 The diagram shown illustrates the sample dataset. Figure 3 In the image, the left side shows two sample two-view video frame images from the sample two-view video data, and the right side shows the sample human eye fixation density images corresponding to the two sample two-view video frame images.

[0032] At this point, the terminal device can first verify the temporal correlation of attention on consecutive frames in the sample dataset, that is, calculate the linear correlation coefficient between every two consecutive sample human eye fixation density images in the sample human eye fixation density image set; secondly, verify the correlation between human eye attention and motion physics in the sample dataset, that is, use the optical flow method based on convolutional neural networks to measure the motion intensity of objects in all sample two-viewpoint video frame image data in each sample two-viewpoint video data and sort them from largest to smallest, thereby obtaining the object motion intensity sorting sequence set for each sample two-viewpoint video data; then use each object motion intensity sorting sequence set to sort each sample human eye fixation density image in the corresponding sample human eye fixation density image set from largest to smallest pixel value, thereby obtaining the pixel fixation probability sorting sequence set for each sample human eye fixation density image set. Each pixel fixation probability sorting sequence can reflect the probability that each pixel in the corresponding sample human eye fixation density image is fixated, and the pixels with the highest sorting must highly match the viewer's fixation area.

[0033] Based on this, the terminal device can determine the linear correlation coefficient between every two consecutive sample human eye fixation density images in the sample human eye fixation density image set, the object motion intensity ranking sequence set of each sample two-viewpoint video data, and the pixel fixation probability ranking sequence set of each sample human eye fixation density image set. These are then matched against pre-set linear correlation conditions, object motion intensity conditions, and pixel fixation probability conditions. If the match is successful, the sample two-viewpoint video dataset is determined to be a dataset that meets the model training requirements. Conversely, if the match fails, the sample two-viewpoint video dataset is reacquired and usability verification is performed. The terminal device can also display the linear correlation coefficient, object motion intensity ranking sequence set, and pixel fixation probability ranking sequence set to the experimenter, receive feedback from the experimenter, and determine whether the sample two-viewpoint video dataset is a dataset that meets the model training requirements or whether the usability verification fails based on the feedback information. No specific limitations are set here.

[0034] The model training method provided by this invention involves a terminal device performing usability verification on the sample two-view video dataset by analyzing linear correlation coefficients, object motion intensity ranking sequences, and pixel gaze probability ranking sequences for each sample two-view video dataset and its corresponding sample human eye gaze density image set. This improves the accuracy and reliability of the sample two-view video dataset and provides a reliable guarantee for the accuracy of subsequent model training.

[0035] Optionally, based on the linear correlation coefficient, the object motion intensity ranking sequence set, and the pixel gaze probability ranking sequence set, the sample two-viewpoint video dataset is determined to be a dataset that meets the model training requirements. The specific implementation process may include: The linear correlation coefficient is matched with a preset coefficient threshold, and the consistency between the object motion intensity sorted sequence set and the pixel gaze probability sorted sequence set is matched with a preset consistency threshold. Based on the successful matching results of the linear correlation coefficient and the preset coefficient threshold, and the successful matching results of the consistency with the preset consistency threshold, the sample two-viewpoint video dataset is determined to be a dataset that meets the model training requirements.

[0036] Specifically, the linear correlation coefficient between every two consecutive samples of human eye fixation density images in each determined sample human eye fixation density image set is matched with a preset coefficient threshold. If the number of linear correlation coefficients in each sample human eye fixation density image set that successfully match the preset coefficient threshold exceeds a preset number, then the multiple linear correlation coefficients of the corresponding sample human eye fixation density image set are determined to be successfully matched with the preset coefficient threshold; otherwise, the matching fails.

[0037] Furthermore, the degree of agreement between the determined object motion intensity ranking sequence set of each sample's two-viewpoint video data and the pixel gaze probability ranking sequence set of the corresponding sample's human eye gaze density image set is matched with a preset degree of agreement. This can be achieved by first determining the degree of agreement between each object motion intensity ranking sequence set and the corresponding pixel gaze probability ranking sequence set in the pixel gaze probability ranking sequence set, and then matching each determined degree of agreement with a preset degree of agreement threshold. If the number of successful matches reaches the preset threshold, it indicates that the object motion intensity ranking sequence set and the pixel gaze probability ranking sequence set of the corresponding sample's two-viewpoint video data have been successfully matched; otherwise, the match has failed.

[0038] For example, for the sample human eye fixation density image set 2 corresponding to the sample two-viewpoint video image 1, the object motion intensity ranking sequence set 4 of the sample two-viewpoint video data 3, and the pixel fixation probability ranking sequence set 5 of the sample human eye fixation density image set 2, if at least 8 of the 10 linear correlation coefficients in the sample human eye fixation density image set 2 match a preset coefficient threshold, then it is determined that the 10 linear correlation coefficients in the sample human eye fixation density image set 2 successfully match the preset coefficient threshold; and if the 10 object motion intensity ranking sequences and pixel fixation in the object motion intensity ranking sequence set 4... If there are 10 coincidences among the 10 pixel gaze probability ranking sequences in probability ranking sequence set 5, and at least 8 of the 10 coincidences match the preset coincidence, then it is determined that the object motion intensity ranking sequence set 4 and the pixel gaze probability ranking sequence set 5 of the sample two-viewpoint video data 3 are successfully matched. At this time, it can be determined that the sample human eye gaze density image set 2 is usable. In this way, usability verification is performed on each sample human eye gaze density image set. When each sample two-viewpoint video data passes the usability verification, it can be determined that the current sample two-viewpoint video dataset is a dataset that meets the model training requirements.

[0039] It should be noted that if multiple linear correlation coefficients of the corresponding sample human eye gaze density image set fail to match the preset coefficient threshold, and / or if the object motion intensity ranking sequence set of the corresponding sample two-viewpoint video data fails to match the pixel gaze probability ranking sequence set, usability verification can be performed on the next sample two-viewpoint video data. If all sample two-viewpoint video data with a preset probability or higher (including the preset probability) in the sample two-viewpoint video dataset pass the usability verification, then the sample two-viewpoint video dataset is determined to be a dataset that meets the model training requirements; otherwise, a new sample two-viewpoint video dataset needs to be obtained and usability verification performed. The preset probability can be 80%.

[0040] The model training method provided by this invention determines the sample two-viewpoint video dataset as a dataset that meets the requirements of model training by matching the linear correlation coefficient between every two consecutive sample human eye fixation density images in each sample human eye fixation density image set with a preset coefficient threshold, and matching the consistency between the object motion intensity sorting sequence set of each sample two-viewpoint video data and the pixel fixation probability sorting sequence set of the corresponding sample human eye fixation density image set with a preset consistency. This improves the usability and reliability of the sample two-viewpoint video dataset and lays the foundation for the accuracy of subsequent model training.

[0041] Optionally, the specific implementation process of step 120 may include: First, the initial human eye gaze region prediction model is trained using a sample two-viewpoint video dataset, and the intermediate model loss of the intermediate human eye gaze region prediction model after a preset number of training iterations is obtained. Then, based on the successful matching results between the intermediate model loss and the preset model loss, the intermediate human eye gaze region prediction model corresponding to the intermediate model loss is determined as the preset human eye gaze region prediction model.

[0042] The preset number of training iterations can be the number of training iterations included in one round of training on the two-viewpoint video dataset. Furthermore, the preset model loss can be an empirical loss value, a loss threshold range, or an ideal loss level achievable after training on a preset number of iterations based on experience. No specific limitations are specified here.

[0043] Specifically, after the terminal device trains the initial human eye gaze region prediction model a preset number of times using the sample two-viewpoint video dataset, it can determine whether the intermediate model loss of the intermediate human eye gaze region prediction model after the preset number of training times matches the preset model loss. If the terminal model loss of the intermediate human eye gaze region prediction model after the preset number of training times matches the preset model loss, then the model training is stopped, and the intermediate human eye gaze region prediction model corresponding to the intermediate model loss at the time of model training stop is determined as the trained preset human eye gaze region prediction model.

[0044] It should be noted that if the terminal model loss of the intermediate human eye fixation region prediction model after a preset number of training iterations does not match the preset model loss, at least one new sample two-view video dataset can be acquired and combined with the original sample two-view video dataset to form a new sample two-view video dataset. This new sample two-view video dataset can then be used to continue training the current parameter-updated intermediate human eye fixation region prediction model. Alternatively, the original sample two-view video dataset can be preprocessed and used as the new sample two-view video dataset. The specific method for obtaining the new sample two-view video dataset is not limited here.

[0045] The model training method provided by this invention improves the training speed and reduces the training complexity by using a sample two-viewpoint video dataset to train the initial human eye gaze region prediction model a preset number of times, thereby effectively improving the training accuracy.

[0046] Optionally, the initial human eye fixation region prediction model can be trained using a sample two-viewpoint video dataset to obtain the intermediate model loss of the intermediate human eye fixation region prediction model after a preset number of training iterations. The specific implementation process may include: For each training iteration, the intermediate bottom-up attention mechanism sub-model from the previous training iteration is trained based on the two-viewpoint video data of the samples participating in this training, determining the sample multi-scale spatial fusion feature image set output by the intermediate bottom-up attention mechanism sub-model after this training. Then, the intermediate top-down attention mechanism sub-model from the previous training iteration is further trained based on the sample multi-scale spatial fusion feature image set, determining the sample human eye fixation region prediction result set output by the intermediate top-down attention mechanism sub-model after this training. Finally, based on the sample human eye fixation region prediction result set and the sample human eye fixation density image set corresponding to the sample two-viewpoint human eye fixation dataset of the sample two-viewpoint video data, the intermediate model loss of the intermediate human eye fixation region prediction model after this training is determined.

[0047] Specifically, since depth images of objects in the real world have a significant impact on human eye attention and provide spatial orientation information, effectively fusing depth images and color images in a 3D scene can improve the prediction of the eye's gaze region. Therefore, a data-driven bottom-up attention mechanism sub-model can be established based on convolutional neural networks. Furthermore, because depth images and color images are complementary, a simple linear combination cannot account for differences in cross-patterns. Therefore, when the two-viewpoint video data used in this training is input into the intermediate bottom-up attention mechanism sub-model after previous training, deep learning methods can be used to perform multi-scale fusion of the sample depth images and sample color images from the two-viewpoint video data used in this training, while extracting spatial and semantic information from various dimensions. This determines the multi-scale spatial fusion feature image set output by the intermediate bottom-up attention mechanism sub-model after this training.

[0048] Furthermore, when the sample multi-scale spatial fusion feature image set output by the intermediate bottom-up attention mechanism sub-model after this training is input into the intermediate top-down attention mechanism sub-model after the previous training, a long short-term memory network can be used to learn as much as possible the object transformation relationship between cross-sample multi-scale spatial fusion feature images in the sample multi-scale spatial fusion feature image set driven by the task, and to consider the temporal correlation of objects between sample multi-scale spatial fusion feature images at the pixel level; thereby determining the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model after this training.

[0049] At this point, the intermediate model loss can be determined by using the sample human eye gaze density image set corresponding to the sample two-viewpoint video data participating in this training for supervision. That is, the intermediate model loss of the intermediate human eye gaze region prediction model after this training is determined by using the sample human eye gaze density image set corresponding to the sample two-viewpoint video data participating in this training, the sample multi-scale spatial fusion feature image set output by the intermediate bottom-up attention mechanism sub-model after this training, and the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model after this training.

[0050] The model training method provided by this invention determines the intermediate model loss of the intermediate human eye gaze region prediction model after training by using the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model after training, the sample human eye gaze density image set corresponding to the sample two-viewpoint video data participating in this training, and so on. In combination with the multi-scale spatial fusion feature extraction of convolutional neural network, the temporal feature extraction of long short-term memory network, and the supervision technology of real human eye gaze density image set, the method achieves the goal of continuously optimizing the model and ensuring the model prediction accuracy, thereby effectively improving the accuracy and reliability of model training.

[0051] Optionally, the intermediate bottom-up attention mechanism sub-model trained previously can be trained based on the two-viewpoint video data of the samples used in this training, to determine the multi-scale spatial fusion feature image set of the samples output by the intermediate bottom-up attention mechanism sub-model after this training. The implementation process may include: The sample two-viewpoint video data used in this training is input into the intermediate bottom-up attention mechanism sub-model after the previous training. Sample depth images are extracted from the sample two-viewpoint video frame image data. The sample depth images and the corresponding sample two-viewpoint video frame image data are convolved and fused multiple times to determine the sample multi-scale spatial fusion feature image set output by the intermediate bottom-up attention mechanism sub-model after this training.

[0052] The number of convolutions and the number of fusions are the same and correspond one-to-one. That is, the number of convolutions is the same as the number of times the features of the two convolutional layers are fused.

[0053] Specifically, refer to Figure 4 The diagram illustrates the process of multi-scale spatial fusion using the bottom-up attention mechanism sub-model, as shown below. Figure 4As shown, firstly, optical flow estimation is used to calculate the sample depth image of each sample two-view video frame image data in the sample two-view video data participating in this training. Then, while performing multiple convolutions on each sample two-view video frame image data in the sample two-view video data participating in this training, multiple convolutions are also performed on each sample depth image. The two layers of features obtained after each convolution are fused based on the Depth Refinement Block (DRB). This process continues until multiple convolutions and multiple fusions are performed, determining the sample multi-scale spatial fusion feature image set output by the intermediate bottom-up attention mechanism sub-model after this training, as shown. Figure 4 The feature map in the image.

[0054] It should be noted that the sample depth image can be extracted based on the sparse optical flow method. This is because when the target moves, each pixel in the sample two-viewpoint video data will be displaced. Therefore, the partial derivative of each pixel between two sample two-viewpoint video frames can be calculated using the sparse optical flow estimation method as the sample depth information. For video footage of sample two-viewpoint video data, the sample depth information can be obtained by performing optical flow estimation on the same frame of sample two-viewpoint video frame data.

[0055] Furthermore, since each sample's two-viewpoint video frame image data is a color image, and depth in a 3D scene can provide the human eye with spatial orientation information of targets, depth images and color images are complementary. Therefore, it is necessary to effectively integrate the two to improve fixation point prediction performance. Based on this, we can refer to... Figure 5 The diagram shown illustrates the DRB-based fusion process for multi-scale spatial fusion. Figure 5 In this process, residual connections are used to fully extract and fuse the paired, mutually exclusive information of each layer. That is, the two inputs represent the color feature maps of the i-th layer samples output after the video frame image data from the two viewpoints of each sample passes through the i-th convolutional layer. And the sample depth feature map of the i-th layer output after the sample depth image corresponding to the two viewpoint video frame image data of each sample is processed by the i-th convolutional layer. The color feature map of the i-th layer sample After two layers of convolution and activation, a sample depth residual vector is obtained. This sample depth residual vector can be approximated as complementary information extracted from the corresponding sample depth image. This residual vector is then compared with the sample depth feature map of the i-th layer. After addition, the sample fusion features obtained after the i-th DRB-based fusion can be obtained. .

[0056] To better fuse features from multiple layers, the size of the sample fusion features in each layer is adjusted in the transform layer (i.e., the reshape layer). Specifically, bilinear interpolation can be used to increase the size of one sample fusion feature, or max pooling can be used to decrease the size of another sample fusion feature, ultimately ensuring that the sample fusion features of all layers are adjusted to the same size. Then, a regular residual module is used to combine the values ​​of multiple sample fusion features of the same size, thus better adapting to the addition of multi-layer sample fusion features. A 1*1 convolution is then used for full connection, resulting in multi-layer sample feature maps of uniform size. Finally, these multi-layer sample feature maps of uniform size are fused to obtain the color feature map of the i-th layer. and the depth feature map of the i-th layer samples Fusion results after DRB-based fusion This allows for the extraction of both high-dimensional semantic information and low-dimensional spatial information from the two-viewpoint video frame image data of each sample. Ultimately, the multi-scale spatial fusion feature image set of the samples output by the intermediate bottom-up attention mechanism sub-model after this training is obtained.

[0057] The model training method provided by this invention achieves the training objective of the bottom-up attention mechanism sub-model by extracting sample depth images from the two-viewpoint video frame image data of each sample, and performing multiple convolutions and multiple fusions on the sample depth images and the corresponding two-viewpoint video frame image data. In combination with fully extracting the high-dimensional semantic information and low-dimensional spatial information of the two-viewpoint video frame image data of each sample, the accuracy and reliability of the multi-scale spatial fusion feature image of each sample can be greatly improved, thereby also improving the accuracy of model training.

[0058] Optionally, the intermediate top-down attention mechanism sub-model trained previously can be trained based on a multi-scale spatial fusion feature image set of the samples to determine the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model after this training. The implementation process may include: First, the intermediate memory vector and the intermediate human eye gaze region prediction result of the intermediate top-down attention mechanism sub-model after the previous training are obtained. Then, the sample multi-scale spatial fusion feature image set, intermediate memory vector and intermediate human eye gaze region prediction result are input into the intermediate top-down attention mechanism sub-model after the previous training for two layers of weighting and deconvolution operation to determine the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model after this training.

[0059] Among them, the top-down attention mechanism sub-model can be a model obtained by modeling the top-down visual attention mechanism of the human eye through a convolutional long short-term memory network (Conv Long Short-Term Memory, ConvLSTM), and specifically can include two layers of ConvLSTM, so as to achieve the purpose of extracting temporal features and learning the temporal transfer relationship of the human eye fixation area under task-driven conditions.

[0060] Specifically, due to the high temporal correlation of attention in continuous video frame image data, the intermediate bottom-up attention mechanism sub-model after this training outputs the multi-scale spatial fusion feature image of each sample. Then, utilizing the high-dimensional semantic information and low-dimensional spatial information contained in each scale of the deep convolutional layers, two layers of ConvLSTM are used to further consider the temporal correlation of the gazed object between the two viewpoint video frame image data of continuous samples at the pixel level. This learns the object transformation relationship between the two viewpoint video frame image data across samples under task-driven learning. The process can be referenced... Figure 6 The diagram shown illustrates the feature extraction process of a convolutional long short-term memory network, as follows: Figure 6 As shown, when the sample two-view video data participating in this training includes T sample two-view video frame images, Frame 1 is the multi-scale spatial fusion feature image of the first sample two-view video frame image, Frame t is the multi-scale spatial fusion feature image of the t-th sample two-view video frame image, and Frame T is the multi-scale spatial fusion feature image of the T-th sample two-view video frame image. After Frame 1 enters the first layer of ConvLSTM for this training, it can output the intermediate memory vector accordingly. Prediction results of the intermediate eye fixation area Intermediate eye fixation area prediction results After re-entering the second layer of ConvLSTM for this training, the intermediate memory vector can be output accordingly. Prediction results of the intermediate eye fixation area This achieves the purpose of performing two-layer weighting on Frame 1; finally, the prediction results of the intermediate human eye gaze region are analyzed. Perform deconvolution to restore the size, and output the sample human eye gaze region prediction result for the corresponding gaze time of the first sample two-viewpoint video frame image data after this training.

[0061] While Frame t is entering the first ConvLSTM layer, the intermediate memory vector and the intermediate human eye gaze region prediction result of the intermediate top-down attention mechanism submodel trained previously can be obtained. That is, the intermediate memory vector corresponding to the output of Frame t-1 after passing through the first ConvLSTM layer can be obtained. Prediction results of the intermediate eye fixation area And the intermediate memory vector output by Frame t-1 after passing through the second layer of ConvLSTM. Prediction results of the intermediate eye fixation area Frame t, intermediate memory vector Prediction results of the intermediate eye fixation area After inputting the data into the first layer of the ConvLSTM for this training, the intermediate memory vector can be output accordingly. Prediction results of the intermediate eye fixation area Then, the prediction results of the intermediate human eye fixation area are used. Intermediate memory vector Prediction results of the intermediate eye fixation area After inputting the data into the second layer of the ConvLSTM for this training, the intermediate memory vector can be output accordingly. Prediction results of the intermediate eye fixation area This achieves the purpose of performing two-layer weighting on Frame t; finally, the prediction results of the intermediate human eye gaze region are analyzed. Perform deconvolution to restore the size, and output the sample human eye gaze region prediction result for the corresponding gaze time of the t-th sample two-viewpoint video frame image data after this training. Similarly, when Frame T enters the first ConvLSTM layer, we can first obtain the intermediate memory vector of the corresponding output of Frame T-1 after passing through the first ConvLSTM layer. Prediction results of the intermediate eye fixation area And the intermediate memory vector output by FrameT-1 after passing through the second layer of ConvLSTM. Prediction results of the intermediate eye fixation area Frame T, intermediate memory vector Prediction results of the intermediate eye fixation area After inputting the data into the first layer of the ConvLSTM for this training, the corresponding output can be the prediction result of the intermediate human eye gaze region. Then, the prediction results of the intermediate human eye fixation area are used. Intermediate memory vector Prediction results of the intermediate eye fixation area After inputting the data into the second layer of the ConvLSTM for this training, the predicted intermediate human eye gaze region can be output accordingly. This achieves the goal of performing two-layer weighting on Frame T; finally, the prediction results for the intermediate human eye gaze region are analyzed. Perform deconvolution to restore the size, and output the sample human eye gaze region prediction result of the corresponding gaze time of the T-th sample two-viewpoint video frame image data after this training; thereby determine the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model after this training.

[0062] Reference Figure 7 The diagram shown illustrates the prediction results of the human eye's gaze region. Figure 7 In this context, *stimuli* represents the 107th, 141st, 205th, and 254th sample two-view video frames. The ground truth represents the actual ground truth for each of these three sample two-view video frames. The modeling prediction results represent the predicted ground truth regions for each of these three sample two-view video frames. This demonstrates that the qualitative prediction of the ground truth regions is quite accurate. Figure 7 The second row contains real sample human eye gaze density images, and the current linear correlation coefficient between the model's prediction results and the real sample human eye gaze density images exceeds 80%. At this point, training can be stopped, and the preset human eye gaze region prediction model corresponding to the training stop time can be determined.

[0063] The model training method provided by this invention achieves the training objective of a top-down attention mechanism sub-model by performing two-layer weighting and deconvolution operations on the multi-scale spatial fusion feature image of the samples, the intermediate memory vector obtained from the previous training, and the intermediate human eye gaze region prediction result obtained from the previous training. In combination with depth information, it provides accurate target spatial orientation information and temporal correlation to simulate the inter-frame attention transfer relationship driven by human eye tasks to the greatest extent. This ensures that the human attention mechanism is simulated as closely as possible, accurately predicts the gaze region, and thus further improves the training accuracy of the model.

[0064] Reference Figure 8 This is a flowchart illustrating the human eye fixation region prediction method provided by the present invention. The execution entity of the human eye fixation region prediction method can also be a terminal device or a server. The present invention does not limit the specific form of the terminal device or server. The following method embodiments are described using a terminal device as an example.

[0065] like Figure 8As shown, the method for predicting the human eye's gaze region includes the following steps: Step 810: Obtain target two-viewpoint video data and train a preset human eye gaze region prediction model determined by the model training method provided in any embodiment of the present invention.

[0066] Specifically, the terminal device acquires the target two-viewpoint video, which can be based on the target video material of the current experiment. This target video material can also include elements such as the movement of the ball and the person's position, and the change in the person's depth, when shooting two-viewpoint ball-throwing videos using a camera array. Here, the camera array can be two cameras shooting left-view and right-viewpoint videos. Furthermore, the number of target two-viewpoint videos can be one or multiple. When multiple target two-viewpoint videos are captured, they can be distinguished based on differences in the number of people playing ball-throwing, the number of people throwing the ball, and / or video duration, etc. For example, multiple target two-viewpoint video data can include videos of two people playing ball-throwing, videos of three people playing ball-throwing, and videos of four people playing ball-throwing, etc. No specific limitations are imposed here.

[0067] Step 820: Input the target two-viewpoint video data into the preset human eye gaze region prediction model, and determine the target human eye gaze region prediction result of the target two-viewpoint video data based on the output result of the preset human eye gaze region prediction model.

[0068] Specifically, the target two-viewpoint video data is input into a preset human eye gaze region prediction model to predict the human eye gaze region. This involves extracting the target multi-scale spatial fusion feature image of each target two-viewpoint video frame image data in the target two-viewpoint video data, and combining the high-dimensional semantic information and low-dimensional spatial information contained in each target multi-scale spatial fusion feature image to predict the location that the human eye may gaze at. Thus, based on the output of the preset human eye gaze region prediction model, the target human eye gaze region prediction result of the target two-viewpoint video data can be determined.

[0069] The human eye gaze region prediction method provided by this invention achieves the goal of quickly and accurately predicting the human eye gaze region by inputting the target two-viewpoint video data into a preset human eye gaze region prediction model. This significantly improves the prediction accuracy of the human eye gaze region and also greatly expands the applicability of the trained preset human eye gaze region prediction model.

[0070] The model training apparatus provided by the present invention is described below. The model training apparatus described below and the model training method described above can be referred to in correspondence.

[0071] Reference Figure 9 This is a schematic diagram of the structure of the model training device provided by the present invention, as shown below. Figure 9 As shown, the model training device 900 includes: The sample acquisition module 910 is used to acquire a sample two-view video dataset, which includes multiple sample two-view video data captured in a three-dimensional dynamic scene. The model training module 920 is used to train the initial human eye gaze region prediction model based on the sample two-viewpoint video dataset and determine the preset human eye gaze region prediction model. The initial human eye fixation region prediction model includes a bottom-up attention mechanism sub-model based on a convolutional neural network and a top-down attention mechanism sub-model based on a long short-term memory network.

[0072] Optionally, the model training apparatus provided by the present invention may further include a sample determination module, used to determine a sample human eye gaze density image set corresponding to the sample two-viewpoint human eye gaze coordinate dataset of each sample two-viewpoint video data; determine the linear correlation coefficient between every two consecutive sample human eye gaze density images in the sample human eye gaze density image set; determine the object motion intensity ranking sequence set of each sample two-viewpoint video data and the pixel gaze probability ranking sequence set of each sample human eye gaze density image set; and determine the sample two-viewpoint video dataset as a dataset that meets the requirements of model training based on the linear correlation coefficient, the object motion intensity ranking sequence set, and the pixel gaze probability ranking sequence set.

[0073] Optionally, the sample determination module can also be used to match the linear correlation coefficient with a preset coefficient threshold, and to match the consistency between the object motion intensity sorted sequence set and the pixel gaze probability sorted sequence set with a preset consistency threshold; based on the successful matching results of the linear correlation coefficient with the preset coefficient threshold and the successful matching results of the consistency with the preset consistency threshold, the sample two-viewpoint video dataset is determined to be a dataset that meets the model training requirements.

[0074] Optionally, the model training module 920 can be used to train the initial human eye gaze region prediction model using a sample two-viewpoint video dataset, obtain the intermediate model loss of the intermediate human eye gaze region prediction model after a preset number of training iterations, and determine the intermediate human eye gaze region prediction model corresponding to the intermediate model loss as the preset human eye gaze region prediction model based on the successful matching result between the intermediate model loss and the preset model loss.

[0075] Optionally, the model training module 920 can also be used to train the intermediate bottom-up attention mechanism sub-model after the previous training based on the sample two-view video data participating in the current training for each training session, and determine the sample multi-scale spatial fusion feature image set output by the intermediate bottom-up attention mechanism sub-model after the current training; train the intermediate top-down attention mechanism sub-model after the previous training based on the sample multi-scale spatial fusion feature image set, and determine the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model after the current training; and determine the intermediate model loss of the intermediate human eye gaze region prediction model after the current training based on the sample human eye gaze region prediction result set and the sample human eye gaze density image set corresponding to the sample two-view human eye gaze dataset of the sample two-view video data.

[0076] Optionally, the model training module 920 can also be used to input the sample two-viewpoint video data participating in this training into the intermediate bottom-up attention mechanism sub-model after the previous training, to extract the sample depth image of each sample two-viewpoint video frame image data, and to perform multiple convolutions and multiple fusions between the sample depth image and the corresponding sample two-viewpoint video frame image data, so as to determine the sample multi-scale spatial fusion feature image set output by the intermediate bottom-up attention mechanism sub-model after this training.

[0077] Optionally, the model training module 920 can also be used to obtain the intermediate memory vector and intermediate human eye gaze region prediction results of the intermediate top-down attention mechanism sub-model after the previous training; input the sample multi-scale spatial fusion feature image set, intermediate memory vector and intermediate human eye gaze region prediction results into the intermediate top-down attention mechanism sub-model after the previous training for two layers of weighting and deconvolution operations, and determine the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model after this training.

[0078] The eye fixation region prediction device provided by the present invention is described below. The eye fixation region prediction device described below can be referred to in correspondence with the eye fixation region prediction method described above.

[0079] Reference Figure 10 This is a schematic diagram of the structure of the human eye fixation region prediction device provided by the present invention, as shown below. Figure 10 As shown, the human eye fixation area prediction device 1000 includes: The test video acquisition module 1010 is used to acquire target two-viewpoint video data and a preset human eye gaze region prediction model trained and determined according to the model training method provided in any embodiment of the present invention. The human eye gaze region prediction module 1020 is used to input the target two-viewpoint video data into a preset human eye gaze region prediction model, and determine the target human eye gaze region prediction result of the target two-viewpoint video data based on the output result of the preset human eye gaze region prediction model.

[0080] Figure 11 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 11 As shown, the electronic device 1100 may include: a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute a model training method, which includes: Obtain a sample two-viewpoint video dataset, which includes multiple sample two-viewpoint video data captured in a 3D dynamic scene; Based on a sample two-viewpoint video dataset, an initial human eye gaze region prediction model is trained to determine a preset human eye gaze region prediction model. The initial human eye gaze region prediction model includes a bottom-up attention mechanism sub-model based on a convolutional neural network and a top-down attention mechanism sub-model based on a long short-term memory network. Alternatively, a human eye gaze region prediction method may be implemented, comprising: Acquire target two-viewpoint video data and train a preset human eye gaze region prediction model determined by the model training method provided in any embodiment of the present invention; The target two-viewpoint video data is fed into a preset human eye fixation region prediction model, and the target human eye fixation region prediction result is determined based on the output of the preset human eye fixation region prediction model.

[0081] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer being able to execute the model training method provided by the above methods, the method comprising: Obtain a sample two-viewpoint video dataset, which includes multiple sample two-viewpoint video data captured in a 3D dynamic scene; Based on a sample two-viewpoint video dataset, an initial human eye gaze region prediction model is trained to determine a preset human eye gaze region prediction model. The initial human eye gaze region prediction model includes a bottom-up attention mechanism sub-model based on a convolutional neural network and a top-down attention mechanism sub-model based on a long short-term memory network. Alternatively, a human eye gaze region prediction method may be implemented, comprising: Acquire target two-viewpoint video data and train a preset human eye gaze region prediction model determined by the model training method provided in any embodiment of the present invention; The target two-viewpoint video data is fed into a preset human eye fixation region prediction model, and the target human eye fixation region prediction result is determined based on the output of the preset human eye fixation region prediction model.

[0083] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the model training methods provided by the methods described above, the method comprising: Obtain a sample two-viewpoint video dataset, which includes multiple sample two-viewpoint video data captured in a 3D dynamic scene; Based on a sample two-viewpoint video dataset, an initial human eye gaze region prediction model is trained to determine a preset human eye gaze region prediction model. The initial human eye gaze region prediction model includes a bottom-up attention mechanism sub-model based on a convolutional neural network and a top-down attention mechanism sub-model based on a long short-term memory network. Alternatively, a human eye gaze region prediction method may be implemented, comprising: Acquire target two-viewpoint video data and train a preset human eye gaze region prediction model determined by the model training method provided in any embodiment of the present invention; The target two-viewpoint video data is fed into a preset human eye fixation region prediction model, and the target human eye fixation region prediction result is determined based on the output of the preset human eye fixation region prediction model.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training method, characterized in that, include: Obtain a sample two-viewpoint video dataset, which includes multiple sample two-viewpoint video data captured in a three-dimensional dynamic scene; Based on the sample two-view video dataset, an initial human eye fixation region prediction model is trained to determine a preset human eye fixation region prediction model. This includes: training the initial human eye fixation region prediction model using the sample two-view video dataset, and obtaining the intermediate model loss of the intermediate human eye fixation region prediction model after a preset number of training iterations. This includes: for each training iteration, training the intermediate bottom-up attention mechanism sub-model from the previous training iteration based on the sample two-view video data participating in this training iteration, and determining the sample multi-scale spatial fusion feature image set output by the intermediate bottom-up attention mechanism sub-model after this training iteration; based on the sample multi-scale spatial fusion... The intermediate top-down attention mechanism sub-model trained from the previous training is trained using a set of combined feature images to determine the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model after this training. Based on the sample human eye gaze region prediction result set and the sample human eye gaze density image set corresponding to the sample two-viewpoint human eye gaze dataset of the sample two-viewpoint video data, the intermediate model loss of the intermediate human eye gaze region prediction model after this training is determined. Based on the successful matching result of the intermediate model loss and the preset model loss, the intermediate human eye gaze region prediction model corresponding to the intermediate model loss is determined to be the preset human eye gaze region prediction model. The initial human eye gaze region prediction model includes a bottom-up attention mechanism sub-model based on a convolutional neural network and a top-down attention mechanism sub-model based on a long short-term memory network.

2. The model training method according to claim 1, characterized in that, Before training the initial human eye gaze region prediction model based on the sample two-viewpoint video dataset to determine the preset human eye gaze region prediction model, the method further includes: Determine the sample human eye gaze density image set corresponding to the sample two-viewpoint human eye gaze coordinate dataset for each of the sample two-viewpoint video data; Determine the linear correlation coefficient between every two consecutive sample human eye fixation density images in the sample human eye fixation density image set; Determine the object motion intensity sorting sequence set for each sample of two-viewpoint video data and the pixel gaze probability sorting sequence set for each sample of human eye gaze density image set; Based on the linear correlation coefficient, the object motion intensity ranking sequence set, and the pixel gaze probability ranking sequence set, the sample two-viewpoint video dataset is determined to be a dataset that meets the model training requirements.

3. The model training method according to claim 2, characterized in that, The step of determining the sample two-viewpoint video dataset as a dataset that meets the model training requirements based on the linear correlation coefficient, the object motion intensity ranking sequence set, and the pixel gaze probability ranking sequence set includes: The linear correlation coefficient is matched with a preset coefficient threshold, and the degree of agreement between the object motion intensity sorting sequence set and the pixel gaze probability sorting sequence set is matched with a preset degree of agreement threshold. Based on the successful matching results of the linear correlation coefficient and the preset coefficient threshold, and the successful matching results of the consistency degree and the preset consistency threshold, the sample two-viewpoint video dataset is determined to be a dataset that meets the model training requirements.

4. The model training method according to claim 1, characterized in that, The process involves training the previously trained bottom-up attention mechanism sub-model based on the two-viewpoint video data of the samples used in this training, and determining the multi-scale spatial fusion feature image set output by the currently trained bottom-up attention mechanism sub-model, including: The sample two-viewpoint video data participating in this training is input into the intermediate bottom-up attention mechanism sub-model after the previous training. Sample depth images are extracted from the sample two-viewpoint video frame image data. The sample depth images and the corresponding sample two-viewpoint video frame image data are convolved and fused multiple times to determine the sample multi-scale spatial fusion feature image set output by the intermediate bottom-up attention mechanism sub-model after this training.

5. The method for predicting the human eye's fixation region according to claim 1, characterized in that, The intermediate top-down attention mechanism sub-model trained based on the sample multi-scale spatial fusion feature image set is used to train the previously trained intermediate top-down attention mechanism sub-model, and the sample human eye gaze region prediction result set output by the currently trained intermediate top-down attention mechanism sub-model is determined, including: Obtain the intermediate memory vector and intermediate human eye gaze region prediction results of the intermediate top-down attention mechanism sub-model after the previous training; The sample multi-scale spatial fusion feature image set, the intermediate memory vector, and the intermediate human eye gaze region prediction result are input into the previously trained intermediate top-down attention mechanism sub-model for two layers of weighting and deconvolution operations to determine the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model after this training.

6. A method for predicting the human eye's gaze region, characterized in that, include: The method involves acquiring target two-viewpoint video data and a preset human eye gaze region prediction model trained according to any one of the model training methods described in claims 1 to 5. The preset human eye gaze region prediction model is determined as follows: the initial human eye gaze region prediction model is trained using the sample two-viewpoint video dataset, and the intermediate model loss of the intermediate human eye gaze region prediction model after a preset number of training iterations is obtained. This includes: for each training iteration, training the intermediate bottom-up attention mechanism sub-model after the previous training iteration based on the sample two-viewpoint video data participating in this training iteration, and determining the sample multi-scale spatial fusion feature image set output by the intermediate bottom-up attention mechanism sub-model after this training iteration. The intermediate top-down attention mechanism sub-model trained previously is trained based on the sample multi-scale spatial fusion feature image set to determine the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model trained this time. Based on the sample human eye gaze region prediction result set and the sample human eye gaze density image set corresponding to the sample two-viewpoint human eye gaze dataset of the sample two-viewpoint video data, the intermediate model loss of the intermediate human eye gaze region prediction model trained this time is determined. Based on the successful matching result of the intermediate model loss and the preset model loss, the intermediate human eye gaze region prediction model corresponding to the intermediate model loss is determined to be the preset human eye gaze region prediction model. The target two-viewpoint video data is input into the preset human eye gaze region prediction model, and the target human eye gaze region prediction result of the target two-viewpoint video data is determined based on the output result of the preset human eye gaze region prediction model.

7. A model training device, characterized in that, include: The sample acquisition module is used to acquire a sample two-view video dataset, which includes multiple sample two-view video data captured in a three-dimensional dynamic scene. The model training module is used to train an initial human eye fixation region prediction model based on the sample two-viewpoint video dataset, and to determine a preset human eye fixation region prediction model. This includes: training the initial human eye fixation region prediction model using the sample two-viewpoint video dataset, and obtaining the intermediate model loss of the intermediate human eye fixation region prediction model after a preset number of training iterations. This includes: for each training iteration, training the intermediate bottom-up attention mechanism sub-model from the previous training iteration based on the sample two-viewpoint video data participating in this training iteration, and determining the sample multi-scale spatial fusion feature image set output by the intermediate bottom-up attention mechanism sub-model after this training iteration; based on the sample multi-scale... The intermediate top-down attention mechanism sub-model trained from the previous training is trained using a set of feature images fused in the degree space to determine the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model after this training. Based on the sample human eye gaze region prediction result set and the sample human eye gaze density image set corresponding to the sample two-viewpoint human eye gaze dataset of the sample two-viewpoint video data, the intermediate model loss of the intermediate human eye gaze region prediction model after this training is determined. Based on the successful matching result of the intermediate model loss and the preset model loss, the intermediate human eye gaze region prediction model corresponding to the intermediate model loss is determined to be the preset human eye gaze region prediction model. The initial human eye gaze region prediction model includes a bottom-up attention mechanism sub-model based on a convolutional neural network and a top-down attention mechanism sub-model based on a long short-term memory network.

8. A device for predicting the gaze region of the human eye, characterized in that, include: The test video acquisition module is used to acquire target two-viewpoint video data and a preset human eye gaze region prediction model determined by training according to any one of the model training methods described in claims 1 to 5. The preset human eye gaze region prediction model is determined in the following way: the initial human eye gaze region prediction model is trained using the sample two-viewpoint video dataset, and the intermediate model loss of the intermediate human eye gaze region prediction model after a preset number of training times is obtained, including: for each training, the intermediate bottom-up attention mechanism sub-model after the previous training is trained based on the sample two-viewpoint video data participating in this training, and the sample multi-scale spatial fusion feature image set output by the intermediate bottom-up attention mechanism sub-model after this training is determined; The intermediate top-down attention mechanism sub-model trained previously is trained based on the sample multi-scale spatial fusion feature image set to determine the sample human eye gaze region prediction result set output by the intermediate top-down attention mechanism sub-model trained this time. Based on the sample human eye gaze region prediction result set and the sample human eye gaze density image set corresponding to the sample two-viewpoint human eye gaze dataset of the sample two-viewpoint video data, the intermediate model loss of the intermediate human eye gaze region prediction model trained this time is determined. Based on the successful matching result of the intermediate model loss and the preset model loss, the intermediate human eye gaze region prediction model corresponding to the intermediate model loss is determined to be the preset human eye gaze region prediction model. The human eye gaze region prediction module is used to input the target two-viewpoint video data into the preset human eye gaze region prediction model, and determine the target human eye gaze region prediction result of the target two-viewpoint video data based on the output result of the preset human eye gaze region prediction model.

Images