Model training method and naked eye 3D display screen visual quality evaluation method based on interactive perception network

Abstract

The invention provides a model training method and a naked eye 3D display screen visual quality evaluation method based on an interactive perception network. The model training method comprises the steps of obtaining training data; inputting the left view, the right view and the binocular color difference image into a crosstalk sensing module of the initial model to obtain a left view feature map containing crosstalk and a right view feature map containing crosstalk; inputting the crosstalk-containing left view feature map and the crosstalk-containing right view feature map into a multi-domain binocular interaction module of the initial model to obtain a target binocular feature map; inputting the target binocular feature map into a regression module of the initial model to obtain a prediction score; using a preset loss function to obtain a loss value based on the prediction score and the score corresponding to the binocular view; and adjusting the initial model based on the loss value to obtain a trained visual quality evaluation model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically to a model training method and a naked-eye 3D display screen visual quality evaluation method based on an interactive perception network. Background Art

[0002] With the rapid development of stereoscopic imaging and display technology, three-dimensional display is widely used in multimedia applications such as virtual reality, telemedicine, and 3D (three-dimensional) television for its realistic stereoscopic and interactive sense. The popular 3D display screens on the market at this stage require polarization or shutter glasses to separate the left and right views, which is not convenient for viewers. Therefore, naked-eye 3D display screens were born, which use parallax barriers or lens systems to directly separate binocular views, effectively improving the visual experience.

[0003] For naked-eye 3D displays, since crosstalk is a key hardware parameter in naked-eye 3D displays, the visual quality of the display is generally reflected by predicting the crosstalk level. However, this method ignores the impact of other parameters of naked-eye 3D displays and is not accurate in evaluating the visual quality of naked-eye 3D displays. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a model training method and a naked-eye 3D display screen visual quality evaluation method based on an interactive perception network.

[0005] According to a first aspect of the present disclosure, a model training method is provided, comprising: acquiring training data, the training data comprising a binocular view and a score corresponding to the binocular view, the binocular view being acquired based on a naked-eye 3D display screen, the binocular view comprising a left view and a right view; obtaining a binocular color difference map based on the left view and the right view; inputting the left view, the right view, and the binocular color difference map into a crosstalk perception module of an initial model to obtain a left view feature map containing crosstalk and a right view feature map containing crosstalk, wherein the crosstalk perception module is used to add interference or aliasing data caused by the opposite side view to the left view and the right view based on the color difference feature, the color difference feature being obtained based on the binocular color difference map, and the left view feature map containing crosstalk. The crosstalk data caused by the right view is included in the right view feature map containing crosstalk, and the crosstalk data caused by the left view is included in the right view feature map containing crosstalk; the left view feature map containing crosstalk and the right view feature map containing crosstalk are input into the multi-domain binocular interaction module of the initial model to obtain the target binocular feature map, and the multi-domain binocular interaction module is used to integrate the attribute information of the naked-eye 3D display screen into the initial model based on the view features, and the attribute information affects the visual experience of the naked-eye 3D display screen; the target binocular feature map is input into the regression module of the initial model to obtain a predicted score; using a preset loss function, a loss value is obtained based on the predicted score and the score corresponding to the binocular view; the initial model is adjusted based on the loss value to obtain a trained visual quality evaluation model.

[0006] According to an embodiment of the present disclosure, a crosstalk perception module includes a chromatic aberration-guided enhancement submodule and a crosstalk learning submodule, and a left view, a right view, and a binocular chromatic aberration map are input into the crosstalk perception module of the initial model to obtain a left view feature map containing crosstalk and a right view feature map containing crosstalk, including: a feature encoder based on a preset weight performs feature extraction on the left view, the right view, and the binocular chromatic aberration map to obtain a left view feature map, a right view feature map, and a binocular chromatic aberration feature map; the left view feature map, the right view feature map, and the binocular chromatic aberration feature map are input into the chromatic aberration-guided enhancement submodule to obtain an enhanced left view feature map and an enhanced right view feature map, and the chromatic aberration-guided enhancement submodule is used to enhance the features of chromatic aberration-significant areas in the left view feature map and the right view feature map; the enhanced left view feature map and the right view feature map are input into the crosstalk learning submodule to obtain a right view feature map containing crosstalk; and the enhanced right view feature map and the left view feature map are input into the crosstalk learning submodule to obtain a left view feature map containing crosstalk.

[0007] According to an embodiment of the present disclosure, the chromatic aberration-guided enhancement submodule includes a local representation unit and a global representation unit, and the left view feature map, the right view feature map and the binocular chromatic aberration feature map are input into the chromatic aberration-guided enhancement submodule to obtain an enhanced left view feature map and an enhanced right view feature map, including: inputting the binocular chromatic aberration feature map into the local representation unit to obtain a local representation map of chromatic aberration features; inputting the binocular chromatic aberration feature map into the global representation unit to obtain a global representation map of chromatic aberration features; aggregating the local representation map of chromatic aberration features and the global representation map of chromatic aberration features to obtain a comprehensive chromatic aberration feature map; obtaining an enhanced left view feature map based on the left view feature map and the comprehensive chromatic aberration feature map; and obtaining an enhanced right view feature map based on the right view feature map and the comprehensive chromatic aberration feature map.

[0008] According to an embodiment of the present disclosure, a multi-domain binocular interaction module includes a full-channel interaction submodule and a co-localized space interaction submodule. A left view feature map containing crosstalk and a right view feature map containing crosstalk are input into the multi-domain binocular interaction module of the initial model to obtain a target binocular feature map, including: inputting the left view feature map containing crosstalk and the right view feature map containing crosstalk into the full-channel interaction submodule to obtain a left-view dominated binocular feature map and a right-view dominated binocular feature map; inputting the left-view dominated binocular feature map and the right-view dominated binocular feature map into the co-localized space interaction submodule to obtain a target binocular feature map.

[0009] According to an embodiment of the present disclosure, a left view feature map containing crosstalk and a right view feature map containing crosstalk are input into a full-channel interaction submodule to obtain a left view-dominated binocular feature map and a right view-dominated binocular feature map, including: obtaining a correlation matrix between binocular view channels based on the binocular view feature map, the binocular view feature map is obtained by splicing a left view feature map containing crosstalk and a right view feature map containing crosstalk; dividing the correlation matrix into a left view channel autocorrelation matrix, a right view channel autocorrelation matrix, a left view to right view channel mutual correlation matrix, and a right view to left view channel mutual correlation matrix based on the view channel correspondence; obtaining an intra-view autocorrelation weight based on the left view channel autocorrelation matrix and the right view channel autocorrelation matrix; and obtaining an intra-view autocorrelation weight based on the left view channel autocorrelation matrix and the right view channel autocorrelation matrix. The inter-view cross-correlation weights are obtained from the left view to right view channel cross-correlation matrix and the right view to left view channel cross-correlation matrix; the left view enhanced feature map and the right view enhanced feature map are obtained based on the intra-view autocorrelation weights, the left view feature map including crosstalk and the right view feature map including crosstalk respectively; the left view interactive feature fusion map and the right view interactive feature fusion map are obtained based on the inter-view cross-correlation weights, the left view feature map including crosstalk and the right view feature map including crosstalk respectively; the left view enhanced feature map and the left view interactive feature fusion map are obtained based on the channel shuffling algorithm, the left view dominated binocular feature map is obtained based on the left view enhanced feature map and the left view interactive feature fusion map; the right view dominated binocular feature map is obtained based on the right view enhanced feature map and the right view interactive feature fusion map using the channel shuffling algorithm.

[0010] According to an embodiment of the present disclosure, a binocular feature map dominated by a left view and a binocular feature map dominated by a right view are input into a co-localization space interaction submodule to obtain a target binocular feature map, including: segmenting the binocular feature map dominated by the left view to obtain a left first half feature view and a left second half feature view; segmenting the binocular feature map dominated by the right view to obtain a right first half feature view and a right second half feature view; aggregating the left first half feature view and the right first half feature view to obtain a first half binocular feature map; aggregating the left second half feature view and the right second half feature view to obtain a second half binocular feature map; and using a horizontal recombination algorithm to obtain a target binocular feature map based on the first half binocular feature map and the second half binocular feature map.

[0011] According to a second aspect of the present disclosure, a method for evaluating the visual quality of a naked-eye 3D display screen based on an interactive perception network is provided, comprising: obtaining a binocular view of the naked-eye 3D display screen; inputting the binocular view into a visual quality evaluation model to generate a score for characterizing the visual quality of the naked-eye 3D display screen, wherein the visual quality evaluation model is obtained according to the method of the first aspect described above.

[0012] A third aspect of the present disclosure provides a model training device, including: an acquisition module, used to acquire training data, the training data including binocular views and scores corresponding to the binocular views, the binocular views are acquired based on a naked-eye 3D display screen, and the binocular views include a left view and a right view; a binocular color difference module, used to obtain a binocular color difference map based on the left view and the right view; a binocular crosstalk module, used to input the left view, the right view, and the binocular color difference map into a crosstalk perception module of an initial model to obtain a left view feature map containing crosstalk and a right view feature map containing crosstalk, wherein the crosstalk perception module is used to add interference or aliasing data caused by the opposite side view to the left view and the right view based on the color difference feature, the color difference feature is obtained based on the binocular color difference map, and the left view feature map containing crosstalk includes interference or aliasing data caused by the right view. The crosstalk data caused by the left view is included in the right view feature map containing crosstalk; the binocular feature map acquisition module is used to input the left view feature map containing crosstalk and the right view feature map containing crosstalk into the multi-domain binocular interaction module of the initial model to obtain the target binocular feature map, and the multi-domain binocular interaction module is used to integrate the attribute information of the naked-eye 3D display screen into the initial model based on the view features, and the attribute information affects the visual experience of the naked-eye 3D display screen; the prediction module is used to input the target binocular feature map into the regression module of the initial model to obtain a predicted score; the loss value calculation module is used to use a preset loss function to obtain a loss value based on the predicted score and the score corresponding to the binocular view; the adjustment module is used to adjust the initial model based on the loss value to obtain a trained visual quality evaluation model.

[0013] The fourth aspect of the present disclosure provides a naked-eye 3D display screen visual quality evaluation device based on an interactive perception network, comprising: a binocular view acquisition module, used to acquire a binocular view of the naked-eye 3D display screen; a scoring module, used to input the binocular view into a visual quality evaluation model to generate a score for characterizing the visual quality of the naked-eye 3D display screen, wherein the visual quality evaluation model is obtained by the method of the first aspect above.

[0014] A fifth aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0015] The sixth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0016] The seventh aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0017] According to the embodiments of the present disclosure, crosstalk data is introduced into the left and right views based on the color difference features through the crosstalk perception module, and at the same time, the attribute information that can affect the visual perception of the naked-eye 3D display screen is integrated into the initial model based on the view features through the multi-domain binocular interaction module, so as to accurately evaluate the visual quality of the naked-eye 3D display screen. Furthermore, the color difference features are used to guide the initial model to learn features that are highly correlated with crosstalk, thereby improving the accuracy of the visual quality assessment of the naked-eye 3D display screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0019] Figure 1 The following schematically shows an application scenario diagram of the model training method and device according to an embodiment of the present disclosure;

[0020] Figure 2 A flowchart of a model training method according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 3 A schematic diagram of a process flow of a model training method according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 4 A schematic diagram of a crosstalk sensing module according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 5 A schematic diagram of a chromatic aberration guided enhancement submodule according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 6 A schematic diagram of a crosstalk learning submodule according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 7 A schematic diagram of a multi-domain binocular interaction module according to an embodiment of the present disclosure is schematically shown;

[0026] Figure 8 A flowchart of a naked-eye 3D display screen visual quality evaluation method based on an interactive perception network according to an embodiment of the present disclosure is schematically shown;

[0027] Fig. 9 The structure block diagram of the model training device according to the embodiment of the present disclosure is schematically shown;

[0028] Fig.10 The structure block diagram of the naked-eye 3D display screen visual quality evaluation device based on the interactive perception network according to an embodiment of the present disclosure is schematically shown;

[0029] Fig.11 A block diagram of an electronic device suitable for implementing a model training method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0031] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0032] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0033] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0034] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0035] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0036] With the rapid development of stereo imaging and display technology, three-dimensional display is widely used in multimedia applications such as virtual reality, telemedicine, and 3D TV for its realistic stereoscopic and interactive sense. At present, the popular 3D display screens on the market usually require polarization or shutter glasses to separate the left and right views. While providing a real 3D experience, it also brings various inconveniences to viewers and has an adverse effect on the stereoscopic visual experience. In order to avoid these problems, naked-eye 3D display screens use parallax barriers or lens systems to directly separate binocular views, which effectively improves the visual experience. However, naked-eye 3D display screens introduce optical distortion and visual illusions during the imaging process, causing visual fatigue and discomfort to viewers. Quantifying these discomfort phenomena is very important for optimizing display performance. To this end, visual quality evaluation technology for naked-eye 3D display screens will inevitably become an important research topic in industry and academia.

[0037] In order to evaluate the visual quality of naked-eye 3D displays, some existing subjective evaluation methods recruit viewers and collect their subjective experience scores of the display to explore the impact of various factors on the visual quality of the display. However, these subjective methods are time-consuming and labor-intensive, which seriously hinders the performance optimization process of naked-eye 3D displays. In order to automatically and efficiently evaluate the visual quality of naked-eye 3D displays, researchers began to explore objective evaluation methods for the visual quality of naked-eye 3D displays. Considering that crosstalk is one of the most critical hardware parameters of naked-eye 3D displays, the visual artifacts caused by it have a significant damage to the stereoscopic visual experience. Therefore, some objective methods reflect the visual quality of the display by predicting the crosstalk level. However, the visual quality of naked-eye 3D displays is determined by a combination of multiple display parameters, and a single display parameter, such as crosstalk, cannot accurately reflect the visual quality of the display.

[0038] In view of this, an embodiment of the present disclosure provides a model training method, the method comprising: acquiring training data, the training data comprising a binocular view and a score corresponding to the binocular view, the binocular view being acquired based on a naked-eye 3D display screen, the binocular view comprising a left view and a right view; obtaining a binocular color difference map based on the left view and the right view; inputting the left view, the right view, and the binocular color difference map into a crosstalk perception module of an initial model to obtain a left view feature map containing crosstalk and a right view feature map containing crosstalk, wherein the crosstalk perception module is used to add interference or aliasing data caused by the opposite side view to the left view and the right view based on the color difference feature, the color difference feature being obtained based on the binocular color difference map, and the left view feature map containing crosstalk. The figure includes crosstalk data caused by the right view, and the right view feature map containing crosstalk includes crosstalk data caused by the left view; the left view feature map containing crosstalk and the right view feature map containing crosstalk are input into the multi-domain binocular interaction module of the initial model to obtain the target binocular feature map, and the multi-domain binocular interaction module is used to integrate the attribute information of the naked-eye 3D display screen into the initial model based on the view features, and the attribute information affects the visual experience of the naked-eye 3D display screen; the target binocular feature map is input into the regression module of the initial model to obtain a predicted score; using a preset loss function, a loss value is obtained based on the predicted score and the score corresponding to the binocular view; the initial model is adjusted based on the loss value to obtain a trained visual quality evaluation model.

[0039] Figure 1 The application scenario diagram of the model training method and device according to the embodiments of the present disclosure is schematically shown.

[0040] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0041] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0042] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0043] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0044] It should be noted that the model training method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the model training device provided in the embodiment of the present disclosure can generally be set in the server 105. The model training method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the model training device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0045] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0046] The following will be based on Figure 1 The scene described by Figure 2~Figure 7 The model training method of the disclosed embodiment is described in detail.

[0047] Figure 2 A flowchart of a model training method according to an embodiment of the present disclosure is schematically shown.

[0048] like Figure 2 As shown, the model training method of this embodiment includes operations S210 to S270.

[0049] In operation S210, training data is acquired.

[0050] The training data includes binocular views and scores corresponding to the binocular views. The binocular views are acquired based on a naked-eye 3D display screen. The binocular views include a left view and a right view.

[0051] According to an embodiment of the present disclosure, the present disclosure adopts a TJU (Tianjin University) naked-eye 3D display screen data set, which includes 800 high-quality images with a resolution of 1152×864, and simulates naked-eye 3D display screens under different display parameter conditions such as crosstalk, screen size, and screen resolution. The data set is randomly divided, and 80% of the samples are used as training data, and the remaining 20% ​​of the samples are used as test data.

[0052] According to an embodiment of the present disclosure, in order to retain the inherent properties of the naked-eye 3D display screen, the image size in the data set is not adjusted, and images of the original acquisition size are used as training data.

[0053] In operation S220, a binocular color difference map is obtained based on the left view and the right view.

[0054] According to an embodiment of the present disclosure, the binocular color difference map may be generated by comparing La*b* values ​​between colors at corresponding positions of the left view and the right view in the CIE (Commission Internationale de l'éclairage) La*b* color space, for example.

[0055] In operation S230, the left view, the right view, and the binocular color difference map are input into the crosstalk perception module of the initial model to obtain a left view feature map containing crosstalk and a right view feature map containing crosstalk.

[0056] Among them, the crosstalk perception module is used to add interference or aliasing data caused by the opposite side view in the left view and the right view based on the color difference feature. The color difference feature is obtained based on the binocular color difference map. The left view feature map containing crosstalk includes the crosstalk data caused by the right view, and the right view feature map containing crosstalk includes the crosstalk data caused by the left view.

[0057] In operation S240, the left view feature map containing crosstalk and the right view feature map containing crosstalk are input into a multi-domain binocular interaction module of the initial model to obtain a target binocular feature map.

[0058] Among them, the multi-domain binocular interaction module is used to integrate the attribute information of the naked-eye 3D display screen into the initial model based on view features, and the attribute information affects the visual experience of the naked-eye 3D display screen.

[0059] In operation S250, the target binocular feature map is input into the regression module of the initial model to obtain a prediction score.

[0060] In operation S260 , a loss value is obtained based on the predicted score and the score corresponding to the binocular view using a preset loss function.

[0061] According to an embodiment of the present disclosure, the preset loss function may be, for example, an L1 loss function, which calculates the deviation between the predicted score and the true score and performs back propagation to update the network model parameters, thereby minimizing the prediction error on the test set.

[0062] In operation S270, the initial model is adjusted based on the loss value to obtain a trained visual quality assessment model.

[0063] According to the embodiment of the present disclosure, Adam (Adaptive Moment Estimation) can be used as an optimizer, the learning rate of the initial model is set to 0.00005, 60 iterations of training are performed, and 10 random experiments are performed to verify the robustness of the model.

[0064] According to an embodiment of the present disclosure, the above-mentioned visual quality assessment model adopts an interactive perception network.

[0065] According to the embodiments of the present disclosure, crosstalk data is introduced into the left and right views based on the color difference features through the crosstalk perception module, and at the same time, the attribute information that can affect the visual perception of the naked-eye 3D display screen is integrated into the initial model based on the view features through the multi-domain binocular interaction module, so as to accurately evaluate the visual quality of the naked-eye 3D display screen. Furthermore, the color difference features are used to guide the initial model to learn features that are highly correlated with crosstalk, thereby improving the accuracy of the visual quality assessment of the naked-eye 3D display screen.

[0066] Figure 3A flow chart of a model training method according to an embodiment of the present disclosure is schematically shown.

[0067] like Figure 3 As shown in the figure, the left view and the right view are passed through the CCAM (Color-informed Crosstalk Aware Module) to obtain a left view feature map containing crosstalk and a right view feature map containing crosstalk, and then the left view feature map containing crosstalk and the right view feature map containing crosstalk are input into the MBIM (Multi-domain Binocular Interaction Module) to obtain a target binocular feature map, and then the target binocular feature map is input into a regression module consisting of a fully connected layer of 192 neurons and a fully connected layer of 1 neuron to obtain a prediction score.

[0068] According to an embodiment of the present disclosure, a crosstalk perception module includes a chromatic aberration-guided enhancement submodule and a crosstalk learning submodule, and a left view, a right view, and a binocular chromatic aberration map are input into the crosstalk perception module of the initial model to obtain a left view feature map containing crosstalk and a right view feature map containing crosstalk, including: a feature encoder based on a preset weight performs feature extraction on the left view, the right view, and the binocular chromatic aberration map to obtain a left view feature map, a right view feature map, and a binocular chromatic aberration feature map; the left view feature map, the right view feature map, and the binocular chromatic aberration feature map are input into the chromatic aberration-guided enhancement submodule to obtain an enhanced left view feature map and an enhanced right view feature map, and the chromatic aberration-guided enhancement submodule is used to enhance the features of chromatic aberration-significant areas in the left view feature map and the right view feature map; the enhanced left view feature map and the right view feature map are input into the crosstalk learning submodule to obtain a right view feature map containing crosstalk; and the enhanced right view feature map and the left view feature map are input into the crosstalk learning submodule to obtain a left view feature map containing crosstalk.

[0069] Figure 4 A schematic diagram of a crosstalk sensing module according to an embodiment of the present disclosure is schematically shown.

[0070] like Figure 4 As shown, I l and I r Represents the left view and the right view respectively, I c Represents the binocular color difference map, the left view I l 、Right View I r 、Binocular color difference map I c Input into the crosstalk perception module of the initial model, the left view feature map I is obtained through three weight-sharing feature encoders. l F , right view feature map I r F And binocular color difference feature map Ic F , and then the left view feature map I l F And binocular color difference feature map I c F Input to CDEUnit (Color Difference-guided EnhancementUnit, color difference guided enhancement submodule) to obtain the enhanced left view feature map L r , which indicates the information leakage from the left view to the right view, and the right view feature map I r F And binocular color difference feature map I c F Input to CDEUnit (Color Difference-guided Enhancement Unit) to obtain the enhanced right view feature map L l , which indicates the information leakage from the right view to the left view. The enhanced left view feature map and the right view feature map are input into CLUnit (Crosstalk Learning Unit, crosstalk learning submodule) to obtain the right view feature map F containing crosstalk r , the enhanced right view feature map and the left view feature map are input into CLUnit (Crosstalk Learning Unit, crosstalk learning submodule) to obtain the left view feature map F containing crosstalk l .

[0071] According to an embodiment of the present disclosure, the feature encoder maps the left and right views and the color difference map from the pixel domain to the feature domain by weight sharing.

[0072] According to an embodiment of the present disclosure, the above-mentioned chromatic aberration-guided enhancement submodule includes a local representation unit and a global representation unit, and the left view feature map, the right view feature map and the binocular chromatic aberration feature map are input into the chromatic aberration-guided enhancement submodule to obtain an enhanced left view feature map and an enhanced right view feature map, including: inputting the binocular chromatic aberration feature map into the local representation unit to obtain a local representation map of chromatic aberration features; inputting the binocular chromatic aberration feature map into the global representation unit to obtain a global representation map of chromatic aberration features; aggregating the local representation map of chromatic aberration features and the global representation map of chromatic aberration features to obtain a comprehensive chromatic aberration feature map; obtaining an enhanced left view feature map L based on the left view feature map and the comprehensive chromatic aberration feature map r ; and obtaining an enhanced right view feature map L based on the right view feature map and the comprehensive color difference feature map l .

[0073] Figure 5 A schematic diagram of a chromatic aberration guided enhancement submodule according to an embodiment of the present disclosure is schematically shown.

[0074] like Figure 5 As shown, the right view feature map I r F And binocular color difference feature map I c F Take as an example, where 501 represents the local representation unit and 502 represents the global representation unit. The local representation unit 501 consists of 3 consecutive dilated convolutions, whose dilation rates are set to 1, 2 and 4 respectively, to learn color difference features with multi-level receptive fields. The unit designs dense jump connections between different dilated convolutions to capture and understand more detailed color difference information by promoting the flow of information between different scales. The global representation unit 502 consists of a global average pooling, a 1×1 convolution and a bilinear upsampling to extract the global contextual information of color difference. Considering that spatial-level feature modulation can improve the representation ability of features through fine-grained control, CDEUnit aggregates the local and global representations of color difference, and generates a shift factor and a scale factor through two independent convolution blocks to modulate and strengthen the view features. Finally, the right view feature map I r F Combined with the comprehensive color difference feature map, the enhanced right view feature map L is obtained l .

[0075] The workflow of CDEUnit can be expressed by the following formula (1).

[0076]

[0077] in, and Represent the i-th dilated convolution and its output features respectively; Represents the color difference feature aggregated from the local and global representations of color difference; represents the enhanced right view feature map, which indicates the information leakage from the right view to the left view; L r represents the enhanced left view feature map, which indicates the information leakage from the left view to the right view; represents a bilinear upsampling operation; represents the global average pooling operation; and They represent convolutional layers with kernel sizes of 1×1 and 3×3 respectively; and These are the convolution blocks that generate the scale factor and shift factor respectively; and Represent element-wise addition and element-wise multiplication operations, respectively.

[0078] Figure 6 A schematic diagram of a crosstalk learning submodule according to an embodiment of the present disclosure is schematically shown.

[0079] like Figure 6 As shown in the figure, the crosstalk learning submodule consists of an information receiving stage and an information retaining stage to enhance the right view feature map L l and left view feature map I l F For example, in the information receiving stage, the left view feature map I l F and from the enhanced right view feature map L l The leakage aggregation is used to obtain the features related to crosstalk. At this time, part of the left view feature map I is lost in the generated features. l F The original view information is then aggregated with the coarse crosstalk features in the information preservation stage to obtain the left view feature map F containing the crosstalk. l , which is highly correlated with crosstalk.

[0080] The workflow of CLUnit can be expressed by the following formula (2).

[0081]

[0082] in, and represents a left view feature map including crosstalk and a right view feature map including crosstalk; Indicates a cascade operation; Represents a convolution layer with a convolution kernel size of 3×3; represents the enhanced right view feature map, which indicates the information leakage from the right view to the left view; represents the enhanced left view feature map, which indicates the information leakage from the left view to the right view.

[0083] According to an embodiment of the present disclosure, the crosstalk learning submodule is used to simulate information leakage between the left and right views, and learns features that are highly correlated with crosstalk in a coarse-to-fine manner.

[0084] According to the embodiments of the present disclosure, the local representation unit and the global representation unit in the chromatic aberration-guided enhancement submodule are used to learn chromatic aberration features in an all-round way, thereby improving the crosstalk discrimination capability, and further being able to more effectively learn the view features related to crosstalk, thereby improving the accuracy of the visual quality assessment of the naked-eye 3D display screen.

[0085] According to an embodiment of the present disclosure, the multi-domain binocular interaction module includes a full-channel interaction submodule and a co-localized space interaction submodule. The left view feature map containing crosstalk and the right view feature map containing crosstalk are input into the multi-domain binocular interaction module of the initial model to obtain a target binocular feature map, including: inputting the left view feature map containing crosstalk and the right view feature map containing crosstalk into the full-channel interaction submodule to obtain a left-view dominated binocular feature map and a right-view dominated binocular feature map; inputting the left-view dominated binocular feature map and the right-view dominated binocular feature map into the co-localized space interaction submodule to obtain a target binocular feature map.

[0086] According to an embodiment of the present disclosure, the above-mentioned inputting of the left view feature map containing crosstalk and the right view feature map containing crosstalk into the full channel interaction submodule to obtain the left view-dominated binocular feature map and the right view-dominated binocular feature map includes: obtaining a correlation matrix between binocular view channels based on the binocular view feature map, the binocular view feature map is obtained by splicing the left view feature map containing crosstalk and the right view feature map containing crosstalk; dividing the correlation matrix into a left view channel autocorrelation matrix, a right view channel autocorrelation matrix, a left view to right view channel mutual correlation matrix and a right view to left view channel mutual correlation matrix based on the view channel correspondence; obtaining the intra-view autocorrelation weight based on the left view channel autocorrelation matrix and the right view channel autocorrelation matrix; and obtaining the intra-view autocorrelation weight based on the left view channel autocorrelation matrix and the right view channel autocorrelation matrix. The inter-view cross-correlation weights are obtained from the left view to right view channel cross-correlation matrix and the right view to left view channel cross-correlation matrix; the left view enhanced feature map and the right view enhanced feature map are obtained based on the intra-view autocorrelation weights, the left view feature map containing crosstalk and the right view feature map containing crosstalk respectively; the left view interactive feature fusion map and the right view interactive feature fusion map are obtained based on the inter-view cross-correlation weights, the left view feature map containing crosstalk and the right view feature map containing crosstalk respectively; the left view dominated binocular feature map is obtained based on the left view enhanced feature map and the left view interactive feature fusion map by using the channel shuffling algorithm; the right view dominated binocular feature map is obtained based on the right view enhanced feature map and the right view interactive feature fusion map by using the channel shuffling algorithm.

[0087] According to an embodiment of the present disclosure, the above-mentioned inputting of the binocular feature map dominated by the left view and the binocular feature map dominated by the right view into the co-location space interaction submodule to obtain the target binocular feature map includes: segmenting the binocular feature map dominated by the left view to obtain a left first half feature view and a left second half feature view; segmenting the binocular feature map dominated by the right view to obtain a right first half feature view and a right second half feature view; aggregating the left first half feature view and the right first half feature view to obtain a first half binocular feature map; aggregating the left second half feature view and the right second half feature view to obtain a second half binocular feature map; and using a horizontal recombination algorithm to obtain a target binocular feature map based on the first half binocular feature map and the second half binocular feature map.

[0088] Figure 7 A schematic diagram of a multi-domain binocular interaction module according to an embodiment of the present disclosure is schematically shown.

[0089] like Figure 7 As shown, first, the left view feature map containing crosstalk And the right view feature map including crosstalk The left view feature map with a preset number of channels is obtained through the convolution layer And the right view feature map with preset number of channels ,Will and The binocular view feature map is concatenated and input into two independent convolutional layers to obtain query Q and key K respectively. Then, matrix multiplication is performed on Q and K and input into the SoftMax activation function to obtain the correlation matrix between all channels. The aforementioned correlation matrix is ​​divided into four correlation sub-matrices according to the channel correspondence, namely, the left view channel autocorrelation matrix A, the right view to left view channel mutual correlation matrix B, the left view to right view channel mutual correlation matrix C and the right view channel autocorrelation matrix D, and the above four correlation sub-matrices are summed row by row to obtain the intra-view autocorrelation weights and the cross-correlation weights between views ,in, represents the autocorrelation weight within the left view, represents the autocorrelation weight within the right view, represents the cross-correlation weight of the left view to the right view, represents the cross-correlation weight of the right view to the left view. and the cross-correlation weights between views The corresponding view features containing crosstalk are multiplied to obtain the corresponding view enhancement feature map and view interaction feature fusion map. , which can be compared with the left view feature map with a preset number of channels Multiply to get the left view enhanced feature map Finally, the in-view enhanced feature map is fused with the corresponding interactive feature map through channel shuffling to obtain the left view dominated binocular feature map and the right view dominated binocular feature map.

[0090] According to an embodiment of the present disclosure, the above-mentioned preset number of channels may be 192, for example.

[0091] The workflow of weight allocation in the full-channel interaction submodule can be expressed by the following formula (3).

[0092]

[0093] in, and They represent the binocular feature map dominated by the left view and the binocular feature map dominated by the right view respectively; Represents a channel shuffle operation; and They represent the autocorrelation weight within the left view and the cross-correlation weight between the left view and the right view respectively; and They represent the autocorrelation weight within the right view and the cross-correlation weight between the right view and the left view respectively; and Respectively represent a left view feature map with a preset number of channels and a right view feature map with a preset number of channels; Represents an element-wise multiplication operation.

[0094] like Figure 7 As shown in the figure, after obtaining the binocular feature map dominated by the left view and the binocular feature map dominated by the right view, they can be vertically segmented to obtain four half-side feature views, namely the left first half-side feature view, the left second half-side feature view, the right first half-side feature view and the right second half-side feature view. Then, the views on the same side are spliced ​​together, and two independent convolutional layers are used to aggregate the binocular features dominated by the left and right views from the same side, thereby obtaining the first half-side binocular feature map (left binocular feature) and the second half-side binocular feature map (right binocular feature). Finally, the first half-side binocular feature map and the second half-side binocular feature map are horizontally reorganized to obtain the target binocular feature map, thereby reconstructing the complete field of view.

[0095] The workflow of the co-localization spatial interaction submodule can be expressed by the following formula (4).

[0096]

[0097] in, , They respectively represent the left first half feature view and the left second half feature view generated after segmentation of the binocular feature map dominated by the left view; , They respectively represent the right first half feature view and the right second half feature view generated after segmentation of the binocular feature map dominated by the right view; , Respectively represent the first half binocular feature map and the second half binocular feature map; Indicates a cascade operation; Represents a convolution layer with a convolution kernel size of 3×3; Represents a spatial reorganization operation; Represents the target binocular feature map.

[0098] According to the embodiments of the present disclosure, the correlation between all channels in the binocular view is calculated through the full-channel interaction submodule to realize full-channel feature interaction, and the binocular interaction process is simulated in the spatial domain through the co-located spatial interaction submodule to capture the visual impact of the naked-eye 3D display screen properties. Furthermore, the target binocular feature map is re-weighted by introducing an attention block to further optimize the binocular features.

[0099] According to the embodiment of the present disclosure, the performance of the visual quality evaluation model can be verified by test data. The present disclosure uses SROCC (Spearman Rank Order Correlation Coefficient), PLCC (Pearson Linear Correlation Coefficient), KROCC (Kendall Rank Order Correlation Coefficient), RMSE (Root Mean Squared Error) and MAE (Mean Absolute Error) to evaluate the visual quality evaluation model. The test results of the present disclosure are shown in Table 1 below.

[0100] Table 1

[0101]

[0102] From the above Table 1, we can see that the visual quality evaluation model proposed in the present disclosure has obtained better test results in all test indicators compared with the comparative scheme. Compared with the comparative scheme 10 with the best performance among the comparative schemes, the visual quality evaluation model proposed in the present disclosure has achieved performance gains of 0.027 and 0.025 on SROCC and PLCC respectively.

[0103] Based on the above model training method, the present disclosure also provides a naked-eye 3D display screen visual quality evaluation method based on an interactive perception network.

[0104] Figure 8 The flowchart of the method for evaluating the visual quality of a naked-eye 3D display screen based on an interactive perception network according to an embodiment of the present disclosure is schematically shown.

[0105] like Figure 8 As shown, the naked-eye 3D display screen visual quality evaluation method based on an interactive perception network of this embodiment includes operations S810 to S820.

[0106] In operation S810, a binocular view of a naked-eye 3D display screen is acquired.

[0107] In operation S820 , the binocular view is input into a visual quality assessment model to generate a score for characterizing the visual quality of the naked-eye 3D display screen.

[0108] The visual quality assessment model is obtained according to any one of the above-mentioned model training methods.

[0109] Based on the above-mentioned naked-eye 3D display screen visual quality evaluation method based on interactive perception network, the present disclosure also provides a naked-eye 3D display screen visual quality evaluation device based on interactive perception network. Fig. 9 The device is described in detail.

[0110] Fig. 9 The structural block diagram of the model training device according to an embodiment of the present disclosure is schematically shown.

[0111] like Fig. 9 As shown, the model training device 900 of this embodiment includes an acquisition module 910, a binocular chromatic aberration module 920, a binocular crosstalk module 930, a binocular feature map acquisition module 940, a prediction module 950, a loss value calculation module 960, and an adjustment module 970.

[0112] The acquisition module 910 is used to acquire training data, the training data including binocular views and scores corresponding to the binocular views, the binocular views are acquired based on the naked eye 3D display screen, and the binocular views include left views and right views. In one embodiment, the acquisition module 910 can be used to perform the operation S210 described above, which will not be repeated here.

[0113] The binocular color difference module 920 is used to obtain a binocular color difference map based on the left view and the right view. In one embodiment, the binocular color difference module 920 can be used to perform the operation S220 described above, which will not be repeated here.

[0114] The binocular crosstalk module 930 is used to input the left view, the right view, and the binocular color difference map into the crosstalk perception module of the initial model to obtain a left view feature map containing crosstalk and a right view feature map containing crosstalk, wherein the crosstalk perception module is used to add interference or aliasing data caused by the opposite side view to the left view and the right view based on the color difference feature, the color difference feature is obtained based on the binocular color difference map, the left view feature map containing crosstalk includes the crosstalk data caused by the right view, and the right view feature map containing crosstalk includes the crosstalk data caused by the left view. In one embodiment, the binocular crosstalk module 930 can be used to perform the operation S230 described above, which will not be repeated here.

[0115] The binocular feature map acquisition module 940 is used to input the left view feature map containing crosstalk and the right view feature map containing crosstalk into the multi-domain binocular interaction module of the initial model to obtain the target binocular feature map. The multi-domain binocular interaction module is used to integrate the attribute information of the naked-eye 3D display screen into the initial model based on the view features. The attribute information affects the visual experience of the naked-eye 3D display screen. In one embodiment, the binocular feature map acquisition module 940 can be used to perform the operation S240 described above, which will not be repeated here.

[0116] The prediction module 950 is used to input the target binocular feature map into the regression module of the initial model to obtain a prediction score. In one embodiment, the prediction module 950 can be used to perform the operation S250 described above, which will not be repeated here.

[0117] The loss value calculation module 960 is used to obtain the loss value based on the predicted score and the score corresponding to the binocular view using a preset loss function. In one embodiment, the loss value calculation module 960 can be used to perform the operation S260 described above, which will not be repeated here.

[0118] The adjustment module 970 is used to adjust the initial model based on the loss value to obtain a trained visual quality assessment model. In one embodiment, the adjustment module 970 can be used to perform the operation S270 described above, which will not be described in detail here.

[0119] According to an embodiment of the present disclosure, any multiple modules among the acquisition module 910, the binocular color difference module 920, the binocular crosstalk module 930, the binocular feature map acquisition module 940, the prediction module 950, the loss value calculation module 960, and the adjustment module 970 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 910, the binocular color difference module 920, the binocular crosstalk module 930, the binocular feature map acquisition module 940, the prediction module 950, the loss value calculation module 960, and the adjustment module 970 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by an appropriate combination of any of them. Alternatively, at least one of the acquisition module 910, the binocular color difference module 920, the binocular crosstalk module 930, the binocular feature map acquisition module 940, the prediction module 950, the loss value calculation module 960, and the adjustment module 970 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0120] Fig.10 The structural block diagram of a naked-eye 3D display screen visual quality assessment device based on an interactive perception network according to an embodiment of the present disclosure is schematically shown.

[0121] like Fig.10 As shown, the naked-eye 3D display screen visual quality assessment device 1000 based on an interactive perception network of this embodiment includes a binocular view acquisition module 1010 and a scoring module 1020 .

[0122] The binocular view acquisition module 1010 is used to acquire the binocular view of the naked-eye 3D display screen. In one embodiment, the binocular view acquisition module 1010 can be used to perform the operation S810 described above, which will not be described in detail here.

[0123] The scoring module 1020 is used to input the binocular view into the visual quality evaluation model to generate a score for characterizing the visual quality of the naked-eye 3D display screen, wherein the visual quality evaluation model is obtained according to any of the above-mentioned model training methods. In one embodiment, the binocular chromatic aberration module 1020 can be used to perform the operation S820 described above, which will not be repeated here.

[0124] Fig.11A block diagram of an electronic device suitable for implementing a model training method according to an embodiment of the present disclosure is schematically shown.

[0125] like Fig.11 As shown, the electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage part 1108 to a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include an onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0126] In RAM 1103, various programs and data required for the operation of electronic device 1100 are stored. Processor 1101, ROM 1102 and RAM 1103 are connected to each other through bus 1104. Processor 1101 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1102 and / or RAM 1103. It should be noted that the program can also be stored in one or more memories other than ROM 1102 and RAM 1103. Processor 1101 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0127] According to an embodiment of the present disclosure, the electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to the bus 1104. The electronic device 1100 may further include one or more of the following components connected to the input / output (I / O) interface 1105: an input portion 1106 including a keyboard, a mouse, etc.; an output portion 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1108 including a hard disk, etc.; and a communication portion 1109 including a network interface card such as a LAN card, a modem, etc. The communication portion 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the input / output (I / O) interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed, so that a computer program read therefrom is installed into the storage portion 1108 as needed.

[0128] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0129] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1102 and / or RAM 1103 described above and / or one or more memories other than ROM 1102 and RAM 1103.

[0130] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the model training method provided by the embodiment of the present disclosure.

[0131] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1101. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0132] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1109, and / or installed from a removable medium 1111. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0133] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0134] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0135] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0137] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A model training method, characterized in that: The method comprises: Acquire training data, where the training data includes a binocular view and a score corresponding to the binocular view, where the binocular view is acquired based on a naked-eye 3D display screen, and the binocular view includes a left view and a right view; Obtaining a binocular color difference map based on the left view and the right view; Inputting the left view, the right view, and the binocular color difference map into a crosstalk perception module of an initial model to obtain a left view feature map containing crosstalk and a right view feature map containing crosstalk, wherein the crosstalk perception module is used to add interference or aliasing data caused by the opposite side view to the left view and the right view based on a color difference feature, the color difference feature is obtained based on the binocular color difference map, the left view feature map containing crosstalk includes crosstalk data caused by the right view, and the right view feature map containing crosstalk includes crosstalk data caused by the left view; Inputting the left view feature map containing crosstalk and the right view feature map containing crosstalk into the multi-domain binocular interaction module of the initial model to obtain a target binocular feature map, wherein the multi-domain binocular interaction module is used to integrate the attribute information of the naked-eye 3D display screen into the initial model based on the view features, wherein the attribute information affects the visual experience of the naked-eye 3D display screen; Inputting the target binocular feature map into the regression module of the initial model to obtain a prediction score; Using a preset loss function, a loss value is obtained based on the predicted score and the score corresponding to the binocular view; The initial model is adjusted based on the loss value to obtain a trained visual quality assessment model.

2. The method according to claim 1, characterized in that The crosstalk perception module includes a color difference guidance enhancement submodule and a crosstalk learning submodule, and the crosstalk perception module inputs the left view, the right view, and the binocular color difference map into the initial model to obtain a left view feature map containing crosstalk and a right view feature map containing crosstalk, including: A feature encoder based on a preset weight performs feature extraction on the left view, the right view, and the binocular color difference map to obtain a left view feature map, a right view feature map, and a binocular color difference feature map; Inputting the left view feature map, the right view feature map and the binocular color difference feature map into the color difference guidance enhancement submodule to obtain an enhanced left view feature map and an enhanced right view feature map, wherein the color difference guidance enhancement submodule is used to enhance the color difference significant area features in the left view feature map and the right view feature map; Inputting the enhanced left view feature map and the right view feature map into the crosstalk learning submodule to obtain a right view feature map containing crosstalk; and The enhanced right view feature map and the left view feature map are input into the crosstalk learning submodule to obtain a left view feature map containing crosstalk.

3. The method according to claim 2, characterized in that The color difference guidance enhancement submodule includes a local representation unit and a global representation unit, and the inputting the left view feature map, the right view feature map and the binocular color difference feature map into the color difference guidance enhancement submodule to obtain an enhanced left view feature map and an enhanced right view feature map includes: Inputting the binocular color difference feature map into the local representation unit to obtain a color difference feature local representation map; Inputting the binocular color difference feature map into the global representation unit to obtain a color difference feature global representation map; Aggregating the local color difference feature representation map and the global color difference feature representation map to obtain a comprehensive color difference feature map; Obtaining an enhanced left-view feature map based on the left-view feature map and the comprehensive color difference feature map; and An enhanced right view feature map is obtained based on the right view feature map and the comprehensive color difference feature map.

4. The method according to claim 1, characterized in that: The multi-domain binocular interaction module includes a full-channel interaction submodule and a co-location space interaction submodule, and the multi-domain binocular interaction module inputs the left view feature map containing crosstalk and the right view feature map containing crosstalk into the initial model to obtain a target binocular feature map, including: Inputting the left view feature map containing crosstalk and the right view feature map containing crosstalk into the full-channel interaction submodule to obtain a left view-dominated binocular feature map and a right view-dominated binocular feature map; The binocular feature map dominated by the left view and the binocular feature map dominated by the right view are input into the co-location space interaction submodule to obtain a target binocular feature map.

5. The method according to claim 4, characterized in that The step of inputting the left view feature map containing crosstalk and the right view feature map containing crosstalk into the full channel interaction submodule to obtain a left view-dominated binocular feature map and a right view-dominated binocular feature map comprises: Obtaining a correlation matrix between the binocular view channels based on a binocular view feature map, wherein the binocular view feature map is obtained by concatenating the left view feature map containing crosstalk and the right view feature map containing crosstalk; Dividing the correlation matrix into a left view channel autocorrelation matrix, a right view channel autocorrelation matrix, a left view to right view channel cross-correlation matrix and a right view to left view channel cross-correlation matrix based on the view channel correspondence relationship; Obtaining intra-view autocorrelation weights based on the left view channel autocorrelation matrix and the right view channel autocorrelation matrix; Obtaining inter-view cross-correlation weights based on the left view to right view channel cross-correlation matrix and the right view to left view channel cross-correlation matrix; Obtaining a left view enhanced feature map and a right view enhanced feature map respectively based on the intra-view autocorrelation weight, the left view feature map containing crosstalk, and the right view feature map containing crosstalk; Obtaining a left view interactive feature fusion map and a right view interactive feature fusion map respectively based on the inter-view cross-correlation weight, the left view feature map containing crosstalk, and the right view feature map containing crosstalk; Using a channel shuffling algorithm, a left-view-dominated binocular feature map is obtained based on the left-view enhanced feature map and the left-view interactive feature fusion map; A channel shuffling algorithm is used to obtain a right view-dominated binocular feature map based on the right view enhanced feature map and the right view interactive feature fusion map.

6. The method according to claim 4, characterized in that The step of inputting the left view-dominated binocular feature map and the right view-dominated binocular feature map into the co-location space interaction submodule to obtain a target binocular feature map comprises: Segmenting the binocular feature map dominated by the left view to obtain a left first half feature view and a left second half feature view; Segmenting the binocular feature map dominated by the right view to obtain a right first half feature view and a right second half feature view; Aggregating the left first half feature view and the right first half feature view to obtain a first half binocular feature map; Aggregating the left second half feature view and the right second half feature view to obtain a second half binocular feature map; The target binocular feature map is obtained based on the first half-side binocular feature map and the second half-side binocular feature map by using a horizontal recombination algorithm.

7. A naked-eye 3D display screen visual quality evaluation method based on an interactive perception network, characterized in that: The method comprises: Obtaining a binocular view of a naked-eye 3D display; The binocular view is input into a visual quality evaluation model to generate a score for characterizing the visual quality of the naked-eye 3D display screen, wherein the visual quality evaluation model is obtained according to the method according to any one of claims 1-6.

8. A model training device, characterized in that: The device comprises: An acquisition module, used to acquire training data, wherein the training data includes a binocular view and a score corresponding to the binocular view, wherein the binocular view is acquired based on a naked-eye 3D display screen, and the binocular view includes a left view and a right view; A binocular color difference module, used for obtaining a binocular color difference map based on the left view and the right view; a binocular crosstalk module, configured to input the left view, the right view, and the binocular color difference map into a crosstalk perception module of an initial model to obtain a left view feature map containing crosstalk and a right view feature map containing crosstalk, wherein the crosstalk perception module is configured to add interference or aliasing data caused by the opposite side view to the left view and the right view based on color difference features, the color difference features are obtained based on the binocular color difference map, the left view feature map containing crosstalk includes crosstalk data caused by the right view, and the right view feature map containing crosstalk includes crosstalk data caused by the left view; a binocular feature map acquisition module, configured to input the left view feature map containing crosstalk and the right view feature map containing crosstalk into the multi-domain binocular interaction module of the initial model to obtain a target binocular feature map, wherein the multi-domain binocular interaction module is configured to integrate the attribute information of the naked-eye 3D display screen into the initial model based on the view features, wherein the attribute information affects the visual experience of the naked-eye 3D display screen; A prediction module, used for inputting the target binocular feature map into the regression module of the initial model to obtain a prediction score; A loss value calculation module, configured to obtain a loss value based on the predicted score and the score corresponding to the binocular view by using a preset loss function; An adjustment module is used to adjust the initial model based on the loss value to obtain a trained visual quality assessment model.

9. A naked-eye 3D display screen visual quality evaluation device based on an interactive perception network, characterized in that: The device comprises: A binocular view acquisition module is used to acquire a binocular view of a naked-eye 3D display screen; A scoring module is used to input the binocular view into a visual quality evaluation model to generate a score for characterizing the visual quality of the naked-eye 3D display screen, wherein the visual quality evaluation model is obtained according to the method according to any one of claims 1-6.

10. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

Images