A dynamic vision training method, system, electronic device, and storage medium based on naked-eye 3D imaging.

Abstract

This invention belongs to the field of visual training and discloses a dynamic visual training method, system, electronic device, and storage medium based on naked-eye 3D imaging. The method includes: acquiring real-time eye movement information of a user and analyzing the real-time movement information to obtain the user's viewing angle change parameters; based on the viewing angle change parameters, using a disparity estimation algorithm to extract and match features from input left and right view images to obtain pixel-level virtual views; dynamically rendering based on the pixel-level disparity map to obtain a 3D view; generating a training task, selecting the 3D view based on the training task, and training the user's dynamic vision. This invention adopts a modular architecture design for artificial intelligence algorithms, supports online learning and updates, and realizes intelligent and personalized visual training, which can effectively improve the user's visual cognitive ability.

Description

Technical Field

[0001] This invention relates to the field of visual training technology, and specifically to a dynamic visual training method, system, electronic device, and storage medium based on naked-eye 3D imaging. Background Technology

[0002] In a dynamic vision training system using naked-eye 3D imaging, achieving dynamic changes in the visual scene is a key technical challenge. Traditional 3D imaging technologies typically employ fixed parallax settings, resulting in stereoscopic images with limited depth information, which is insufficient for dynamic vision training. To create realistic stereoscopic dynamic scenes, parallax needs to be adjusted in real-time based on changes in the user's perspective, generating corresponding stereoscopic images. This requires the system to quickly and accurately capture the user's head and eye movement information and calculate the optimal parallax parameters based on this information. Simultaneously, generating high-quality stereoscopic images places higher demands on computational power. Achieving low-latency, high-frame-rate dynamic 3D image rendering with limited computing resources is another pressing technical challenge. Furthermore, during dynamic vision training, the training content and difficulty need to be dynamically adjusted based on user feedback and training progress. This requires the system to analyze user behavior data in real-time and adaptively generate training tasks based on a specific algorithm model. The collection and analysis of user behavior data, as well as the automatic generation of training tasks, involve complex artificial intelligence algorithms; designing and implementing these algorithms is also a significant technical challenge. Summary of the Invention

[0003] To address the above technical problems, this invention provides a dynamic vision training method based on naked-eye 3D imaging, the method comprising:

[0004] The system acquires real-time eye movement information and analyzes the real-time movement information to obtain parameters of the user's perspective change.

[0005] Based on the aforementioned viewpoint change parameters, a disparity estimation algorithm is used to extract and match features from the input left and right view images to obtain a pixel-level virtual view.

[0006] Dynamic rendering is performed based on the pixel-level disparity map to obtain a 3D view;

[0007] A training task is generated, and the 3D view is selected based on the training task to train the user's dynamic vision.

[0008] Preferably, the method for obtaining the viewpoint change parameters includes:

[0009] The user's real-time eye movement information is acquired, and the real-time eye movement information is denoised and cleaned of outliers to obtain cleaned eye movement data.

[0010] The cleaned eye-tracking data was subjected to feature extraction and analysis using a convolutional neural network model to obtain a feature vector of viewpoint changes.

[0011] For the aforementioned viewpoint change feature vectors, a clustering algorithm is used to classify them, and similar feature vectors are aggregated together to obtain cluster centers representing different viewpoint change patterns.

[0012] Calculate the distance between the viewpoint change feature vector and each of the cluster centers, find the closest cluster center, and take the corresponding viewpoint change pattern as the current user's viewpoint change trend;

[0013] Based on the stated viewpoint change trend, the corresponding quantization parameter is found in the viewpoint change parameter mapping table to obtain the viewpoint change parameter.

[0014] Preferably, the method for obtaining the pixel-level virtual view includes:

[0015] Based on the left and right view images and the viewpoint change parameters, a convolutional neural network model is used to extract features from the left and right view images to obtain feature maps of the left and right view images.

[0016] The feature map is matched using a feature matching algorithm to calculate the position offset of each pixel in the left and right view images, and an initial disparity map is generated.

[0017] The initial disparity map is denoised to obtain a smooth disparity map;

[0018] The smoothed disparity map is combined with the pixel information of the left and right view images, and the depth information of the scene is calculated by a 3D reconstruction algorithm to generate a pixel-level depth map.

[0019] The pixel-level depth map is post-processed and fused with the original left view image to obtain the pixel-level virtual view.

[0020] Preferably, the training method includes:

[0021] Based on the training objectives and requirements, select several task templates with high matching degree from the pre-built task template library as the base templates;

[0022] The basic template is parsed to extract key elements, and the training task is generated based on the key elements.

[0023] Based on the training task, the 3D view is selected to generate a training scene;

[0024] The user's dynamic vision is trained based on the training task, combined with the training scenario.

[0025] The present invention also provides a dynamic vision training system based on naked-eye 3D imaging. The system is used to implement the method described in any one of the above, including: a viewpoint change parameter acquisition module, a virtual view generation module, a 3D view rendering module, and a training module.

[0026] The perspective change parameter acquisition module is used to acquire real-time motion information of the user's eyeballs and analyze the real-time motion information to obtain the user's perspective change parameters.

[0027] The virtual view generation module uses the disparity estimation algorithm to extract and match features from the input left and right view images based on the viewpoint change parameters to obtain a pixel-level virtual view.

[0028] The 3D view rendering module performs dynamic rendering based on the pixel-level disparity map to obtain a 3D view;

[0029] The training module is used to generate training tasks, select the 3D view based on the training tasks, and train the user's dynamic vision.

[0030] Preferably, the viewpoint change parameter acquisition module includes: a preprocessing unit, a first feature extraction unit, a clustering unit, a change trend determination unit, and a parameter lookup unit;

[0031] The preprocessing unit is used to acquire real-time eye movement information of the user, and to perform noise reduction and outlier cleaning on the real-time eye movement information to obtain cleaned eye movement data.

[0032] The first feature extraction unit uses a convolutional neural network model to extract and analyze features from the cleaned eye movement data to obtain a viewpoint change feature vector;

[0033] The clustering unit classifies the viewpoint change feature vectors using a clustering algorithm, aggregating similar feature vectors together to obtain cluster centers representing different viewpoint change patterns.

[0034] The trend determination unit is used to calculate the distance between the view change feature vector and each of the cluster centers, find the closest cluster center, and take the corresponding view change pattern as the view change trend of the current user.

[0035] The parameter lookup unit searches for the corresponding quantized parameter in the viewpoint change parameter mapping table based on the viewpoint change trend to obtain the viewpoint change parameter.

[0036] Preferably, the virtual view generation module includes: a second feature extraction unit, a matching unit, a denoising unit, a 3D reconstruction unit, and a fusion unit;

[0037] The second feature extraction unit extracts features from the left and right view images using a convolutional neural network model based on the left and right view images and the viewpoint change parameters, and obtains feature maps of the left and right view images.

[0038] The matching unit uses a feature matching algorithm to match the feature map, calculates the position offset of each pixel in the left and right view images, and generates an initial disparity map.

[0039] The denoising unit is used to denoise the initial disparity map to obtain a smooth disparity map;

[0040] The 3D reconstruction unit combines the smoothed disparity map with the pixel information of the left and right view images, calculates the depth information of the scene through a 3D reconstruction algorithm, and generates a pixel-level depth map.

[0041] The fusion unit is used to post-process the pixel-level depth map and fuse it with the original left view image to obtain the pixel-level virtual view.

[0042] Preferably, the training module includes: a task template confirmation unit, a training task generation unit, a scene construction unit, and a training unit;

[0043] The task template confirmation unit selects several task templates with high matching degree from the pre-built task template library as basic templates according to the training objectives and requirements.

[0044] The training task generation unit is used to parse the basic template, extract key elements from the template, and generate the training task based on the key elements.

[0045] The scene construction unit selects the 3D view based on the training task to generate a training scene;

[0046] The training unit, in conjunction with the training scenario, trains the user's dynamic vision based on the training task.

[0047] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described dynamic vision training method based on naked-eye 3D imaging.

[0048] The present invention also provides a computer-readable storage medium storing a computer program that, when executed, implements the above-described dynamic vision training method based on naked-eye 3D imaging.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] This invention acquires real-time user head and eye movement information, analyzes eye movement data using a convolutional neural network to obtain perspective change parameters, and then introduces a deep learning disparity estimation algorithm to generate pixel-level virtual views, which are then rendered as 3D images. These rendered 3D images provide an immersive visual training environment. Training tasks are automatically generated, and personalized task optimization is achieved by combining rule templates and learning models. This invention employs a modular architecture design for its artificial intelligence algorithms, supporting online learning and updates, enabling intelligent and personalized visual training that effectively improves users' visual cognitive abilities. Attached Figure Description

[0051] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Attached image description:

[0055] 1010, Processor; 1020, Memory; 1030, Input / Output Interface; 1040, Communication Interface; 1050, Bus. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0058] Example 1

[0059] In this embodiment, as Figure 1 As shown, a dynamic vision training method based on naked-eye 3D imaging includes the following steps:

[0060] S1. Obtain real-time eye movement information of the user and analyze the real-time movement information to obtain the user's perspective change parameters.

[0061] The method for obtaining perspective change parameters includes: acquiring real-time eye movement information of the user, denoising and cleaning outliers from the real-time movement information to obtain cleaned eye movement data; using a convolutional neural network model to extract and analyze features from the cleaned eye movement data to obtain perspective change feature vectors; classifying the perspective change feature vectors using a clustering algorithm, aggregating similar feature vectors together to obtain cluster centers representing different perspective change patterns; calculating the distance between the perspective change feature vectors and each cluster center, finding the closest cluster center, and using the corresponding perspective change pattern as the current user's perspective change trend; and looking up the corresponding quantization parameter from the perspective change parameter mapping table based on the perspective change trend to obtain the perspective change parameters.

[0062] In this embodiment, an infrared camera is used to capture reflected light from the eyeball, combined with head posture sensor data, to achieve high-precision eye tracking and obtain real-time eye movement information of the user. The raw real-time motion information often contains noise, such as blinking and head movements, requiring preprocessing through filtering and outlier detection. Specifically, median filtering can be used to remove sudden noise, followed by Kalman filtering to smooth the data trajectory. The cleaned eye-tracking data is then input into a pre-trained convolutional neural network model to extract features reflecting changes in viewing angle. The convolutional neural network model can contain multiple convolutional and pooling layers for extracting local features and dimensionality reduction, and finally, a fully connected layer outputs a feature vector. The extracted features... Vector clustering analysis can identify different viewpoint change patterns. Commonly used clustering algorithms, such as K-means, can group similar feature vectors into one class. Assuming a cluster size of 5, 5 cluster centers can be obtained, representing typical viewpoint change patterns such as upward, downward, leftward, rightward, and gaze. By calculating the Euclidean distance between the current feature vector and each cluster center, the user's current viewpoint change trend can be determined. A mapping table is established to map different viewpoint change patterns to specific quantized values. In this embodiment, upward viewing corresponds to +2, downward viewing corresponds to -2, leftward and rightward viewing correspond to +1 and -1 respectively, and gaze corresponds to 0. This quantization allows the system to accurately measure and respond to the degree of change in the user's viewpoint. When the quantized value exceeds a preset threshold, the system can trigger corresponding operations.

[0063] S2. Based on the viewpoint change parameters, the disparity estimation algorithm is used to extract and match features from the input left and right view images to obtain pixel-level virtual views.

[0064] The method for obtaining a pixel-level virtual view includes: extracting features from the left and right view images using a convolutional neural network model based on the left and right view images and viewpoint change parameters to obtain feature maps of the left and right view images; matching the feature maps using a feature matching algorithm to calculate the positional offset of each pixel in the left and right view images to generate an initial disparity map; denoising the initial disparity map to obtain a smoothed disparity map; combining the smoothed disparity map with the pixel information of the left and right view images, and calculating the depth information of the scene using a 3D reconstruction algorithm to generate a pixel-level depth map; post-processing the pixel-level depth map and fusing it with the original left view image to obtain a pixel-level virtual view.

[0065] In this embodiment, for the 640×480 resolution left and right view images, a ResNet50 network model is used to extract feature maps with 2048 channels, each channel representing a high-level semantic feature, resulting in feature maps for the left and right views. Then, the SIFT algorithm is used to establish a correspondence between the left and right views by detecting and describing key points in the image. Assuming 1000 key points are detected in the left view, 800 pairs of valid matching points can be obtained by matching them with the right view. The positional offsets of these matching points constitute the initial disparity map. Next, bilateral filtering is used to denoise the initial disparity map. For an 8-bit grayscale disparity map, the spatial domain standard deviation can be set to 5 pixels, and the value domain standard deviation to 20 gray levels, effectively removing noise and mismatches in the disparity map while preserving the sharpness of object edges. Finally, a 3D reconstruction algorithm is used to convert the 2D disparity information into 3D depth information.

[0066] Z = fB / d

[0067] Where f is the camera focal length, B is the baseline distance, and d is the disparity value. By performing similar calculations on each pixel, a complete pixel-level depth map can be generated. After obtaining the depth map, the median filtering method is used to fill holes and optimize edges. Finally, the optimized depth map is fused with the original left view image, and a pixel-level virtual view under the new view is generated through the viewpoint transformation algorithm based on the viewpoint change parameters.

[0068] S3. Dynamic rendering is performed based on pixel-level disparity maps to obtain a 3D view.

[0069] In this embodiment, to address the high computational resource requirements of dynamic 3D rendering, a cloud-based distributed rendering architecture is adopted, distributing rendering tasks across multiple graphics processor server nodes to achieve parallel computing. Specifically, upon receiving a dynamic 3D rendering request, the rendering task is decomposed into multiple subtasks, and a load balancing algorithm is used to distribute the subtasks to the appropriate nodes based on their load. Each graphics processor server node, upon receiving its assigned rendering subtask, utilizes its local computing resources for parallel rendering processing to generate the corresponding rendering result data. A scheduling server is set up in the cloud architecture to track the rendering progress of each node. If the progress of a node lags behind a preset threshold, the task allocation is dynamically adjusted, reallocating some tasks to other nodes with lower loads. After a rendering subtask is completed, the rendering result data is uploaded to the cloud-based distributed storage system, and a hash algorithm is used to index the result data for subsequent querying and retrieval. After all rendering subtasks are completed, the scheduling server aggregates and stitches the rendering result data distributed across different nodes according to the task allocation to obtain the complete rendering result, i.e., the 3D view.

[0070] S4. Generate a training task, select a 3D view based on the training task, and train the user's dynamic vision.

[0071] The training method includes: selecting several task templates with high matching degree from a pre-built task template library as base templates according to the training objectives and requirements; parsing the base templates, extracting key elements from the templates, and generating training tasks based on the key elements; selecting 3D views to generate training scenes based on the training tasks; and training the user's dynamic vision based on the training scenes and training tasks.

[0072] In this embodiment, based on the training objectives and requirements, one or more highly matched task templates are selected from a pre-built task template library as base templates. The selected base templates are parsed to extract key elements such as task type, difficulty level, and knowledge coverage. Then, the extracted template attribute features are input into a pre-trained task generation model. The model inferences to generate a batch of candidate task optimization schemes. A rule engine is used to filter and sort the generated candidate task optimization schemes, eliminating non-compliant or redundant schemes to obtain a subset of high-quality and usable task optimization schemes. From the selected subset of task optimization schemes, combined with the learner's individual learning characteristics and ability level, the most suitable optimization scheme is selected, and the base template is dynamically adjusted and optimized. The optimized task template is instantiated, and a specific personalized training task is generated based on the task description and parameter indicators in the template. A 3D view is selected based on the personalized training task to generate a training scene, and combined with the training scene, the user's dynamic vision is trained based on the training task.

[0073] Example 2

[0074] Based on the same inventive concept, corresponding to any of the above-described embodiments, this disclosure also provides a dynamic vision training system based on naked-eye 3D imaging, including: a viewpoint change parameter acquisition module, a virtual view generation module, a 3D view rendering module, and a training module.

[0075] The perspective change parameter acquisition module is used to acquire real-time motion information of the user's eyeballs and analyze the real-time motion information to obtain the user's perspective change parameters.

[0076] The perspective change parameter acquisition module includes: a preprocessing unit, a first feature extraction unit, a clustering unit, a change trend determination unit, and a parameter lookup unit. The preprocessing unit acquires real-time eye movement information from the user, performs noise reduction and outlier cleaning on the real-time movement information, and obtains cleaned eye movement data. The first feature extraction unit uses a convolutional neural network model to extract and analyze features from the cleaned eye movement data, obtaining perspective change feature vectors. The clustering unit classifies the perspective change feature vectors using a clustering algorithm, aggregating similar feature vectors together to obtain cluster centers representing different perspective change patterns. The change trend determination unit calculates the distance between the perspective change feature vectors and each cluster center, finds the closest cluster center, and uses the corresponding perspective change pattern as the current user's perspective change trend. The parameter lookup unit searches for the corresponding quantized parameters from the perspective change parameter mapping table based on the perspective change trend to obtain the perspective change parameters.

[0077] The virtual view generation module uses a disparity estimation algorithm to extract and match features from the input left and right view images based on the viewpoint change parameters, and obtains a pixel-level virtual view.

[0078] The virtual view generation module includes: a second feature extraction unit, a matching unit, a denoising unit, a 3D reconstruction unit, and a fusion unit. The second feature extraction unit extracts features from the left and right view images using a convolutional neural network model based on the left and right view images and viewpoint change parameters, obtaining feature maps of the left and right view images. The matching unit uses a feature matching algorithm to match the feature maps, calculates the positional offset of each pixel in the left and right view images, and generates an initial disparity map. The denoising unit denoises the initial disparity map to obtain a smoothed disparity map. The 3D reconstruction unit combines the smoothed disparity map with the pixel information of the left and right view images, calculates the depth information of the scene using a 3D reconstruction algorithm, and generates a pixel-level depth map. The fusion unit performs post-processing on the pixel-level depth map and fuses it with the original left view image to obtain a pixel-level virtual view.

[0079] The 3D view rendering module performs dynamic rendering based on pixel-level parallax maps to obtain a 3D view.

[0080] The training module is used to generate training tasks, select 3D views based on the training tasks, and train the user's dynamic vision.

[0081] The training module includes: a task template confirmation unit, a training task generation unit, a scene construction unit, and a training unit. The task template confirmation unit selects several highly matched task templates from a pre-built task template library as base templates based on the training objectives and requirements. The training task generation unit parses the base templates, extracts key elements from the templates, and generates training tasks based on these key elements. The scene construction unit selects 3D views based on the training tasks to generate training scenes. The training unit combines the training scenes with the training tasks to train the user's dynamic vision.

[0082] The system described in the above embodiments is used to implement the corresponding dynamic visual training method based on naked-eye 3D imaging in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0083] It should be noted that the dynamic vision training system based on naked-eye 3D imaging is embodied in the form of functional units. The term "module" here can be implemented in software and / or hardware, without specific limitations.

[0084] For example, a "module" can be a software program, hardware circuit, or a combination of both that implements the above functions. Hardware circuits may include application-specific integrated circuits (ASICs), electronic circuits, processors (e.g., shared processors, proprietary processors, or group processors) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functions.

[0085] Example 3

[0086] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the dynamic visual training method based on naked-eye 3D imaging as described in any of the above embodiments.

[0087] Figure 2 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0088] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0089] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0090] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0091] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB (Universal Serial Bus), network cable, etc.) or wireless means (such as mobile network, WIFI (Wireless Fidelity), Bluetooth, etc.).

[0092] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0093] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0094] The system described in the above embodiments is used to implement the corresponding dynamic visual training method based on naked-eye 3D imaging in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0095] Example 4

[0096] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the dynamic vision training method based on naked-eye 3D imaging as described in any of the above embodiments.

[0097] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0098] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the dynamic vision training method based on naked-eye 3D imaging as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0099] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0100] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuitry) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0101] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0102] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0103] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A dynamic visual training system based on naked eye 3D imaging, characterized in that, The application relates to a method for generating a 3D view based on a user's dynamic vision, and a device thereof. The method comprises the following steps: a view angle change parameter acquisition module, a virtual view generation module, a 3D view rendering module and a training module; the view angle change parameter acquisition module is used for acquiring real-time movement information of a user's eyeball, and analyzing the real-time movement information to obtain a view angle change parameter of the user; the virtual view generation module is used for performing feature extraction and matching on input left and right view images based on a parallax estimation algorithm, and obtaining a pixel-level virtual view based on the view angle change parameter; the 3D view rendering module is used for performing dynamic rendering based on the pixel-level virtual view to obtain a 3D view; the training module is used for generating a training task, selecting the 3D view based on the training task, and training the dynamic vision of the user; the view angle change parameter acquisition module comprises a preprocessing unit, a first feature extraction unit, a clustering unit, a change trend determination unit and a parameter searching unit; the preprocessing unit is used for acquiring real-time movement information of a user's eyeball, and performing denoising and outlier cleaning on the real-time movement information to obtain cleaned eye movement data; the first feature extraction unit is used for performing feature extraction and analysis on the cleaned eye movement data by using a convolutional neural network model to obtain a view angle change feature vector; the clustering unit is used for classifying the view angle change feature vector by using a clustering algorithm, and aggregating similar feature vectors together to obtain clustering centers representing different view angle change modes; the change trend determination unit is used for calculating distances between the view angle change feature vector and each of the clustering centers, finding out the nearest clustering center, and taking the corresponding view angle change mode as a view angle change trend of the current user; the parameter searching unit is used for searching for corresponding quantization parameters from a view angle change parameter mapping table based on the view angle change trend to obtain the view angle change parameter; the virtual view generation module comprises a second feature extraction unit, a matching unit, a denoising unit, a three-dimensional reconstruction unit and a fusion unit; the second feature extraction unit is used for performing feature extraction on left and right view images by using a convolutional neural network model according to the left and right view images and the view angle change parameter to obtain feature maps of the left and right view images; the matching unit is used for matching the feature maps by using a feature matching algorithm, calculating position offsets of each pixel point in the left and right view images, and generating an initial parallax map; the denoising unit is used for denoising the initial parallax map to obtain a smooth parallax map; the three-dimensional reconstruction unit is used for combining the smooth parallax map and pixel information of the left and right view images, calculating depth information of a scene by using a three-dimensional reconstruction algorithm, and generating a pixel-level depth map; the fusion unit is used for post-processing the pixel-level depth map, fusing the pixel-level depth map with an original left view image, and obtaining the pixel-level virtual view; the training module comprises a task template confirmation unit, a training task generation unit, a scene construction unit and a training unit; the task template confirmation unit is used for selecting a plurality of task templates with high matching degrees as basic templates from a pre-constructed task template library according to a training target and requirements; The training task generation unit is configured to parse the basic template, extract key elements in the template, and generate the training task based on the key elements; The scene construction unit selects the 3D view generation training scene based on the training task; The training unit trains the dynamic vision of the user based on the training task in combination with the training scene.

Images