3D Imaging Correction Method and Device Based on Neural Network

Abstract

The present invention provides a three-dimensional imaging correction method and device based on a neural network. The method includes the following steps: inputting the original primitive image array of the object to be imaged into a pre-trained convolutional neural network to obtain a calibrated restoration image array; obtaining a corrected light vector field of the three-dimensional imaging system according to the calibrated restoration image array, and obtaining corrected position coordinates of the camera lens center according to the corrected light vector field; collecting a disparity image of the object to be imaged according to the corrected position coordinates of the camera lens center and synthesizing a corrected primitive image array. The present invention can eliminate light reconstruction error factors in the three-dimensional imaging system, such as inherent error factors like pixel stray light and optical device aberrations, as well as external error factors like manufacturing errors, surface damage, and system assembly deviations, and ensure the light field display quality.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional imaging, and in particular to a three-dimensional imaging correction method and device based on a neural network. Background Art

[0002] Naked-eye three-dimensional imaging is an important branch in the technical field of three-dimensional imaging. The naked-eye three-dimensional imaging technology first records the light rays in the original light field at the acquisition end, then compresses and encodes the recorded light information at the data processing end, and finally performs optical decoding using an optical light control device at the display end to reconstruct the light rays, realizing the restoration of the direction and intensity of the recorded light rays. Among them, the display end generally uses a flat panel display and a light control device to cooperate to achieve naked-eye three-dimensional imaging.

[0003] However, in the existing naked-eye three-dimensional imaging process, in most cases, the restoration results of the light intensity and direction information are not satisfactory. The main reason is that there are errors in the light ray reconstruction.

[0004] The light ray reconstruction errors come from inherent error factors such as pixel stray light and optical device aberration, as well as external error factors such as manufacturing errors, surface damage, and system assembly deviation. The light ray reconstruction errors will cause the intensity and direction of the reproduced light rays to be inconsistent with the recorded light rays, seriously deteriorating the three-dimensional display quality, resulting in problems such as distortion, blurring, and incorrect occlusion relationships in the three-dimensional image, and the degree of deterioration will become more serious as the viewing angle, display depth of field, and viewpoint density increase. The light ray reconstruction errors caused by external error factors have randomness and discreteness, which lead to high-order and discrete errors in three-dimensional imaging and are difficult to be corrected. The existence of light ray reconstruction errors restricts the further development of the naked-eye three-dimensional display technology.

[0005] Therefore, there is an urgent need for a method to correct the light ray reconstruction errors in three-dimensional imaging. Summary of the Invention

[0006] Aiming at the problems existing in the prior art, the present invention provides a three-dimensional imaging correction method and device based on a neural network.

[0007] The present invention provides a three-dimensional imaging correction method based on a neural network, including the following steps:

[0008] Input the original primitive image array of the object to be imaged into a pre-trained convolutional neural network to obtain a calibrated restoration image array;

[0009] Obtain the corrected light vector field of the three-dimensional imaging system according to the calibrated restoration image array, and obtain the corrected position coordinates of the camera lens center according to the corrected light vector field;

[0010] Collect the parallax image of the object to be imaged according to the corrected position coordinates of the camera lens center and synthesize the corrected primitive image array.

[0011] According to a three-dimensional imaging correction method based on a neural network provided by the present invention, the step of obtaining the corrected optical vector field includes:

[0012] Obtain the ideal optical vector field of the three-dimensional imaging system according to the light control device information of the three-dimensional imaging system and the original primitive image array;

[0013] Obtain the real optical vector field of the three-dimensional imaging system according to the light control device information of the three-dimensional imaging system and the calibrated restoration map array;

[0014] Obtain the corrected optical vector field according to the ideal optical vector field and the real optical vector field.

[0015] According to a three-dimensional imaging correction method based on a neural network provided by the present invention, the step of obtaining the corrected optical vector field according to the ideal optical vector field and the real optical vector field specifically includes:

[0016] Obtain the angle between the ideal optical vector and the real optical vector corresponding to each pixel in the original primitive image array. If the angle is less than the preset human eye visual threshold, the corrected optical vector is empty. If the angle is greater than or equal to the preset human eye visual threshold, the corrected optical vector is the real optical vector. The corrected optical vectors of each pixel form the corrected optical vector field.

[0017] According to a three-dimensional imaging correction method based on a neural network provided by the present invention, the step of obtaining the ideal optical vector field of the three-dimensional imaging system includes:

[0018] Perform a difference operation on the coordinates of each pixel in the original primitive image array and the center coordinates of the light control device of the three-dimensional imaging system corresponding to each pixel to obtain the ideal optical vector. The ideal optical vectors of each pixel in the original primitive image array form the ideal optical vector field;

[0019] The step of obtaining the real optical vector field of the three-dimensional imaging system includes:

[0020] Perform a difference operation on the pixel coordinates of the calibrated restoration map array corresponding to each pixel in the original primitive image array and the center coordinates of the light control device of the three-dimensional imaging system to obtain the real optical vector. The real optical vectors of each pixel in the original primitive image array form the real optical vector field.

[0021] According to a three-dimensional imaging correction method based on a neural network provided by the present invention, the step of obtaining the corrected position coordinates of the camera lens center according to the corrected optical vector field includes:

[0022] Perform a subtraction operation on the pixel coordinates of the calibrated restoration map array corresponding to each pixel in the original primitive image array and the corrected optical vector in the corrected optical vector field to obtain the corrected camera lens center correction position coordinates.

[0023] According to a three-dimensional imaging correction method based on a neural network provided by the present invention, the convolutional neural network includes a calibration convolutional neural network unit and a point spread function unit. The original primitive image array input into the convolutional neural network is processed by the calibration convolutional neural network unit and then subjected to a convolution operation with the point spread function unit to obtain a calibrated restoration map array. The point spread function unit is a point spread function array of the light control device of the three-dimensional imaging system.

[0024] According to a three-dimensional imaging correction method based on a neural network provided by the present invention, the training steps of the convolutional neural network include:

[0025] Obtain multiple side images of the three-dimensional image of the object as a control set;

[0026] Input the original primitive image array of the object into the convolutional neural network to obtain a calibrated restoration map array;

[0027] Perform a fast Fourier transform on the calibrated restoration map array and the data in the control set respectively, and perform a loss function operation on the data after the fast Fourier transform of the control set and the data after the fast Fourier transform of the calibrated restoration map array to complete the training and calibration of the parameters of the convolutional neural network.

[0028] The present invention also provides a three-dimensional imaging correction device based on a neural network, including:

[0029] A neural network module for inputting the original primitive image array of the object to be imaged into a pre-trained convolutional neural network to obtain a calibrated restoration map array;

[0030] A camera correction module for obtaining the corrected optical vector field of the three-dimensional imaging system according to the calibrated restoration map array, and obtaining the corrected camera lens center correction position coordinates according to the corrected optical vector field;

[0031] A correction module for collecting the disparity image of the object to be imaged according to the corrected camera lens center correction position coordinates and synthesizing a corrected primitive image array.

[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the three-dimensional imaging correction method based on a neural network as described in any one of the above.

[0033] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the three-dimensional imaging correction method based on a neural network as described in any one of the above.

[0034] The three-dimensional imaging correction method and device based on a neural network provided by the present invention obtain a calibration repair map array containing three-dimensional imaging error information caused by light reconstruction error factors through a convolutional neural network, and then obtain a corrected light vector field that overcomes the light reconstruction error according to the calibration repair map array. The parallax image is collected and synthesized according to the corrected position coordinates of the camera lens center corrected by the corrected light vector field, which can eliminate the light reconstruction error factors in the three-dimensional imaging system and improve the three-dimensional imaging display quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0036] Figure 1 is a flowchart of the three-dimensional imaging correction method based on a neural network provided by the present invention;

[0037] Figure 2 is an algorithm framework diagram of the convolutional neural network provided by the present invention;

[0038] Figure 3 is a network architecture diagram of the calibration convolutional neural network unit provided by the present invention;

[0039] Figure 4 is a schematic diagram of the spatial position of the calibration restoration map array provided by the present invention;

[0040] Figure 5 is a schematic diagram of the reconstruction principle of the light vector field provided by the present invention;

[0041] Figure 6 is a schematic diagram of the acquisition principle of the light vector field provided by the present invention;

[0042] Figure 7 is a schematic diagram of the structure of the three-dimensional imaging correction device based on a neural network provided by the present invention;

[0043] Figure 8 is a schematic diagram of the structure of the electronic device provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0045] First, the following are some technical terms involved in this invention:

[0046] Parallax: Multiple slightly different images of the same scene obtained by using a stereo camera are called parallax images, and the combination of two-dimensional display information presented by these parallax images is parallax.

[0047] Primitive image array: an image array composed of several disparity sequence images.

[0048] Viewpoint: The two-dimensional light information seen when observing an object from a certain angle.

[0049] Light field: refers to the vector field composed of light emitted by the object itself, which contains light intensity and direction information.

[0050] Figure 1 This is a flow chart of the neural network-based three-dimensional imaging correction method provided by the present invention, such as Figure 1 As shown, the method includes:

[0051] S1: Input the original primitive image array of the object to be imaged into the pre-trained convolutional neural network to obtain the calibration restoration image array;

[0052] S2: Obtaining a corrected light vector field of the three-dimensional imaging system according to the calibration restoration image array, and obtaining a corrected position coordinate of the camera lens center according to the corrected light vector field;

[0053] S3: Collect the parallax image of the object to be imaged according to the corrected position coordinates of the camera lens center after correction and synthesize the correction primitive image array.

[0054] The three-dimensional imaging system used in the present invention includes a flat panel display and a light control device arranged in parallel and spaced apart. The light control device is a lens array, a cylindrical lens array, a slit grating or other light control device capable of realizing three-dimensional imaging.

[0055] It should be noted that the execution subject of the above method can be an electronic device, a component in the electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The present invention does not make specific limitations.

[0056] The correction of the light reconstruction error requires obtaining the light reconstruction error first. The present invention uses a pre-trained convolutional neural network to obtain a calibrated restoration map array. The pixel information stored in the calibrated restoration map array includes three-dimensional imaging error information caused by the light reconstruction error. Based on the calibrated restoration map array, the light reconstruction error can be corrected.

[0057] Further, in one embodiment, as Figure 2 shown, in order to obtain the calibrated restoration map array, the training steps of the convolutional neural network include:

[0058] Obtaining multiple side images of the three-dimensional image of the object as a control set;

[0059] Inputting the original primitive image array of the object into the convolutional neural network to obtain the calibrated restoration map array;

[0060] Performing a fast Fourier transform on the data in the calibrated restoration map array and the control set respectively, and performing a loss function operation on the data of the control set after the fast Fourier transform and the data of the calibrated restoration map array after the fast Fourier transform to complete the training and calibration of the parameters of the convolutional neural network.

[0061] The training process of the convolutional neural network of the present invention can be based on the fast Fourier transform, can obtain the output data of the calibrated restoration map and the control set in the frequency domain, and then calculate the similarity between the two in the frequency domain based on the loss function. Processing in the frequency domain can further represent the spatial position relationship between the pixels on the calibrated restoration map and the control set, effectively improving the accuracy of the calibrated restoration map; and processing in the frequency domain can add a frequency domain filter after the fast Fourier transform to filter the noise in the two data sets, improving the training efficiency of the convolutional neural network while improving the accuracy of the calibrated restoration map.

[0062] Specifically, the specific formula for performing the loss function operation in this application is as follows:

[0063]

[0064] Among them, f decoding (x i ) is the pixel output result of the calibrated restoration image predicted by the network, y i is the pixel of the control set, n is the number of pixels in the primitive image array, w t (x i ) is the actual pixel value y of the network in the t-th iteration cycle i and the predicted pixel value f decoding (x i ) The weight function of the difference, w i (t) can balance the gradient propagation of the network according to the dynamic changes of the training effect.

[0065] Training samples can be constructed using multiple different objects to form a control set for training.

[0066] Furthermore, in one embodiment, the naked-eye three-dimensional imaging effect of an object is captured using a camera array, and the images of all sides of the three-dimensional image are obtained as the control set of the neural network for training.

[0067] Furthermore, in one embodiment, as Figure 2 shown, the convolutional neural network includes a calibration convolutional neural network unit and a point spread function unit. The original primitive image array input into the convolutional neural network is processed by the calibration convolutional neural network unit and then convolved with the point spread function unit to obtain a calibrated restoration image array. The point spread function unit is the point spread function array of the light control device of the three-dimensional imaging system.

[0068] The convolutional neural network processes the primitive image array through the calibration convolutional neural network unit, introduces the point spread function unit, and uses the point spread function unit to simulate the influence of the light control device in the three-dimensional imaging system on the calibrated restoration image, thereby improving the accuracy of the calibrated restoration image output by the convolutional neural network.

[0069] The present invention adopts a calibration convolutional neural network unit, which constructs a deep network architecture with multiple feature extraction channels by stacking single convolutional feature layers, and while achieving high-precision fitting of the high-order non-linear light decoding function, enables the training process to converge quickly and has strong generalization ability.

[0070] Specifically, the network architecture of the calibration convolutional neural network unit of the present invention includes, but is not limited to, convolutional neural network architectures such as Feature Pyramid Network, VGGet Network, ResNet Network, etc., as Figure 3As shown, taking the Feature Pyramid Network as an example, the calibration convolutional neural network units respectively include Channel 1, Channel 2, Channel 3, and Channel 4. The primitive image array input to the calibration convolutional neural network unit is processed through Channel 1, and then goes through downsampling step by step from top to bottom. After that, it undergoes convolution through Channels 2, 3, and 4. Then, the outputs of Channels 2, 3, and 4 are combined step by step from bottom to top. After upsampling and stitching, they are stitched with the output of Channel 1 to output the calibrated restoration image array.

[0071] Taking the full-parallax display with M×N viewpoints as an example, the calibrated restoration image array is on the viewing plane, representing the spatial positions on the viewing plane where the actual reconstructed light rays pass through, such as Figure 4 shown. The resolution of the calibrated restoration images in the calibrated restoration image array is n×m, and their spatial arrangement order on the viewing plane is as Figure 4 shown. The central positions of the calibrated restoration images on the viewing plane are coaxial with the camera center positions of the camera array used in the light vector field acquisition process. Before calibration, the camera array arranged according to the camera center positions acquires the parallax image of the object to be imaged, and then the parallax image is synthesized into the original primitive image array through a synthesis algorithm.

[0072] Specifically, in step S2, based on ray tracing to confirm the corrected position coordinates of the camera lens center, it can be divided into:

[0073] S21: Obtain the corrected light vector field of the three-dimensional imaging system according to the calibrated restoration image array;

[0074] S22: And obtain the corrected position coordinates of the camera lens center according to the corrected light vector field.

[0075] Furthermore, in one embodiment, the steps of obtaining the corrected light vector field in step S21 include:

[0076] S211: Obtain the ideal light vector field of the three-dimensional imaging system according to the light control device information of the three-dimensional imaging system and the original primitive image array;

[0077] S212: Obtain the real light vector field of the three-dimensional imaging system according to the light control device information of the three-dimensional imaging system and the calibrated restoration image array;

[0078] S213: Obtain the corrected light vector field according to the ideal light vector field and the real light vector field.

[0079] Specifically, taking a flat panel display with a resolution of n×m as an example, the ideal light vector field reconstructed by the present invention is The reconstructed real light vector field with errors is The corrected light vector field is

[0080] Further, in one embodiment, step S211 specifically includes: performing a difference operation on the coordinates of each pixel in the original primitive image array and the center coordinates of the light control device of each pixel corresponding three-dimensional imaging system to obtain an ideal light vector, and the ideal light vectors of each pixel in the original primitive image array form an ideal light vector field.

[0081] Taking the light control device as a lens array as an example, the formula for the ideal light vector is:

[0082]

[0083] where, (x k , y l , z4) MP is the center coordinates of the k-th row and the l-th column light control unit on the light control device with a periodic light control unit structure corresponding to the pixel (i, j) in the original primitive image array, where k = floor(i / M), l = floor(j / N), floor() is the floor function, (x k , y l , z4) MP is the known light control device parameter preset in the three-dimensional imaging system design stage, (x k , y l , z4) MP is used to modulate the light emitted by the pixel (i, j) in the primitive image array loaded on the flat panel display; (u′ (i，j) , v′ (i，j) , z3) EIA is the coordinate of the pixel (i, j) in the original primitive image array loaded on the flat panel display;

[0084] The formula of the above ideal light vector is described by taking the lens array as an example in this embodiment. When the light control device is a light control device such as a cylindrical lens array or a slit grating that can achieve three-dimensional imaging, those skilled in the art can also obtain the corresponding ideal light vector field based on the steps of step S211.

[0085] Further, in one embodiment, step S212 specifically includes: performing a difference operation on the pixel coordinates of the calibrated restoration image array corresponding to each pixel in the original primitive image array and the center coordinates of the light control device of the three-dimensional imaging system to obtain a real light vector, and the real light vectors of each pixel in the original primitive image array form a real light vector field.

[0086] Taking the light control device as a lens array as an example, the formula for the real light vector is:

[0087]

[0088] where, (s′ (i，j) , t′ (i，j), z2) DPI is the pixel coordinate in the calibrated and restored image array corresponding to the pixel (i, j) in the original primitive image array; (x k , y l , z4) MP is the central coordinate of the k-th row and l-th column of the light control unit on the light control device with a periodic light control unit structure corresponding to the pixel (i, j) in the original primitive image array, (x k , y l , z4) MP is used to modulate the light emitted from the pixel (i, j) in the primitive image array loaded on the flat panel display;

[0089] Further, in one embodiment, step S213 specifically includes:

[0090] Obtain the angle between the ideal light vector and the real light vector corresponding to each pixel in the original primitive image array. If the angle is less than the preset human eye visual threshold, the corrected light vector is null. If the angle is greater than or equal to the preset human eye visual threshold, the corrected light vector is the real light vector. The corrected light vectors of each pixel form a corrected light vector field.

[0091] Specifically, the calculation formula for the corrected light vector is:

[0092]

[0093] where null represents being null, is the angle between the ideal light vector and the real light vector corresponding to the pixel (i, j) in the original primitive image array, θ eye is the preset human eye visual threshold.

[0094] Specifically, further, in one embodiment, the step of obtaining the corrected camera lens center correction position coordinates according to the corrected light vector field in step S22 includes:

[0095] Perform a difference operation on the pixel coordinates of the calibrated and restored image array corresponding to each pixel in the original primitive image array and the corrected light vectors in the corrected light vector field to obtain the corrected camera lens center correction position coordinates.

[0096] Specifically, the calculation formula for the corrected camera lens center correction position coordinates is:

[0097]

[0098] where, (u (i，j) , v (i，j) , z1) camerais the corrected position coordinate of the camera lens center corresponding to the pixel (i, j), (s′ (i，j) , t′ (i，j) , z2) DPI is the pixel coordinate in the calibrated restoration image array corresponding to the pixel (i, j) in the original primitive image array, is the corrected light vector.

[0099] For the sake of convenience of explanation, as Figure 5 shown, the coordinates are explained as follows:

[0100] As Figure 5 shown, for the plane where the flat panel display is located, (u′ (i，j) , v′ (i，j) , z3) EIA is the coordinate of the pixel (i, j) in the original primitive image array loaded by the flat panel display; as Figure 6 shown, the light rays emitted by the pixel (i, j) on the flat panel display pass through the (x k , y l , z4) MP coordinates on the light control device plane and finally hit the (s′ (i，j) , t′ (i，j) , z2) DPI coordinates on the viewing plane, and the pixels hitting the viewing plane form the calibrated restoration image array. Among them, z3, z4, and z2 are the plane coordinates of the flat panel display, the light control device plane, and the viewing plane respectively.

[0101] Finally, specifically, step S3 is based on the corrected position coordinates of the camera lens center, and a new camera array is built to collect the disparity image of the object to be imaged. Through the synthesis algorithm, the newly collected disparity image is synthesized into the corrected primitive image array, that is, the correction of the light ray reconstruction error in the original primitive image array is completed.

[0102] Specifically, as Figure 6 shown, the light rays emitted by the object to be imaged pass through the camera array lens plane and the camera array CCD plane in sequence and are then captured by the camera array to complete the acquisition of the disparity image of the object to be imaged.

[0103] Furthermore, in one embodiment, to build a new camera array, the arrangement of each camera array can be reconstructed so that the lens plane of the new camera array satisfies the corrected position coordinates of the camera lens center; in another implementation manner, the coordinate points where the corrected position coordinates of the camera lens center are different from the position coordinates of the camera lens center of the original camera array can be obtained first, and new cameras are set at these different coordinate points to complete the construction of the camera array.

[0104] The three-dimensional imaging correction device based on a neural network provided by the present invention will be described below. The three-dimensional imaging correction device based on a neural network described below can be correspondingly referred to the three-dimensional imaging correction method based on a neural network described above.

[0105] The three-dimensional imaging correction device based on a neural network provided by the present invention, as Figure 7 shown, includes:

[0106] A neural network module 701, configured to input the original primitive image array of the object to be imaged into a pre-trained convolutional neural network to obtain a calibrated restoration map array;

[0107] A camera correction module 702, configured to obtain a corrected light vector field of the three-dimensional imaging system according to the calibrated restoration map array, and obtain corrected camera lens center correction position coordinates according to the corrected light vector field;

[0108] A correction module 703, configured to collect a parallax image of the object to be imaged according to the corrected camera lens center correction position coordinates and synthesize a corrected primitive image array.

[0109] Further, in an embodiment, the camera correction module 702 specifically includes: an ideal light vector field acquisition module, configured to obtain an ideal light vector field of the three-dimensional imaging system according to the light control device information of the three-dimensional imaging system and the original primitive image array; a real light vector field acquisition module, configured to obtain a real light vector field of the three-dimensional imaging system according to the light control device information of the three-dimensional imaging system and the calibrated restoration map array; a corrected light vector field acquisition module, configured to obtain a corrected light vector field according to the ideal light vector field and the real light vector field.

[0110] Further, in an embodiment, the corrected light vector field acquisition module may specifically include: an included angle acquisition unit, configured to obtain an included angle between the ideal light vector and the real light vector corresponding to each pixel in the original primitive image array; a discrimination correction unit, configured to judge the size of the included angle and the human visual threshold and confirm the corrected light vector. If the included angle is less than a preset human eye visual threshold, the corrected light vector is empty. If the included angle is greater than or equal to the preset human eye visual threshold, the corrected light vector is the real light vector, and the corrected light vectors of each pixel form a corrected light vector field.

[0111] Further, in an embodiment, the specific processing steps of the ideal light vector field acquisition module include: performing a difference operation on the coordinates of each pixel in the original primitive image array and the center coordinates of the light control device of the three-dimensional imaging system corresponding to each pixel to obtain an ideal light vector, and the ideal light vectors of each pixel in the original primitive image array form an ideal light vector field;

[0112] The specific processing steps of the true light vector field acquisition module include: performing a difference operation on the pixel coordinates of the calibrated restoration image array corresponding to each pixel in the original primitive image array and the center coordinates of the light control device of the three-dimensional imaging system to obtain the true light vector, and the true light vectors of each pixel in the original primitive image array form the true light vector field.

[0113] Further, in one embodiment, the camera correction module 702 further includes a calibrated camera coordinate acquisition module, which is used to perform a difference operation on the pixel coordinates of the calibrated restoration image array corresponding to each pixel in the original primitive image array and the corresponding calibrated light vector in the calibrated light vector field to obtain the corrected camera lens center correction position coordinates.

[0114] Further, in one embodiment, the convolutional neural network includes a calibration convolutional neural network unit and a point spread function unit. The original primitive image array input into the convolutional neural network is processed by the calibration convolutional neural network unit and then subjected to a convolution operation with the point spread function unit to obtain the calibrated restoration image array, and the point spread function unit is the point spread function array of the light control device of the three-dimensional imaging system.

[0115] Further, in one embodiment, the device further includes a pre-training module, which is used to obtain multiple side images of the three-dimensional image of the object as a control set; input the original primitive image array of the object into the convolutional neural network to obtain the calibrated restoration image array; perform a fast Fourier transform on the calibrated restoration image array and the data in the control set respectively, and perform a loss function operation on the data obtained by the fast Fourier transform of the control set and the data obtained by the fast Fourier transform of the calibrated restoration image array to complete the training and calibration of the parameters of the convolutional neural network.

[0116] Figure 8 An example of the physical structure diagram of an electronic device is shown in Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840. Among them, the processor 810, the communication interface 820, and the memory 830 complete communication with each other through the communication bus 840. The processor 810 can call the logical instructions in the memory 830 to execute the three-dimensional imaging correction method based on the neural network, and the method includes: inputting the original primitive image array of the object to be imaged into the pre-trained convolutional neural network to obtain the calibrated restoration image array; obtaining the calibrated light vector field of the three-dimensional imaging system according to the calibrated restoration image array, and obtaining the corrected camera lens center correction position coordinates according to the calibrated light vector field; collecting the parallax image of the object to be imaged according to the corrected camera lens center correction position coordinates and synthesizing the corrected primitive image array.

[0117] In addition, when the logical instructions in the above-mentioned memory 830 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, external hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0118] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the three-dimensional imaging correction method based on a neural network provided by the above-mentioned various methods. The method includes: inputting the original primitive image array of the object to be imaged into a pre-trained convolutional neural network to obtain a calibrated restoration image array; obtaining the corrected light vector field of the three-dimensional imaging system according to the calibrated restoration image array, and obtaining the corrected position coordinates of the camera lens center according to the corrected light vector field; collecting the parallax image of the object to be imaged according to the corrected position coordinates of the camera lens center and synthesizing a corrected primitive image array.

[0119] On yet another hand, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the three-dimensional imaging correction method based on a neural network provided by the above-mentioned various methods. The method includes: inputting the original primitive image array of the object to be imaged into a pre-trained convolutional neural network to obtain a calibrated restoration image array; obtaining the corrected light vector field of the three-dimensional imaging system according to the calibrated restoration image array, and obtaining the corrected position coordinates of the camera lens center according to the corrected light vector field; collecting the parallax image of the object to be imaged according to the corrected position coordinates of the camera lens center and synthesizing a corrected primitive image array.

[0120] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

[0121] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional imaging correction method based on a neural network, characterized in that It includes the following steps: Input the original primitive image array of the object to be imaged into a pre-trained convolutional neural network to obtain a calibrated restoration image array; Obtain the corrected light vector field of the three-dimensional imaging system according to the calibrated restoration image array, and obtain the corrected position coordinates of the camera lens center according to the corrected light vector field; Collect the parallax images of the object to be imaged according to the corrected position coordinates of the camera lens center and synthesize a corrected primitive image array.

2. The three-dimensional imaging correction method based on a neural network according to claim 1, wherein, The step of obtaining the corrected light vector field includes: Obtain the ideal light vector field of the three-dimensional imaging system according to the light control device information of the three-dimensional imaging system and the original primitive image array; Obtain the real light vector field of the three-dimensional imaging system according to the light control device information of the three-dimensional imaging system and the calibrated restoration image array; Obtain the corrected light vector field according to the ideal light vector field and the real light vector field.

3. The three-dimensional imaging correction method based on a neural network according to claim 2, wherein, The step of obtaining the corrected light vector field according to the ideal light vector field and the real light vector field specifically includes: Obtain the angle between the ideal light vector and the real light vector corresponding to each pixel in the original primitive image array. If the angle is less than the preset human eye vision threshold, the corrected light vector is empty. If the angle is greater than or equal to the preset human eye vision threshold, the corrected light vector is the real light vector. The corrected light vectors of each pixel form the corrected light vector field.

4. A three-dimensional imaging correction method based on a neural network according to claim 2, characterized in that, The step of obtaining the ideal light vector field of the three-dimensional imaging system includes: Perform a difference operation on the coordinates of each pixel in the original primitive image array and the center coordinates of the light control device of the three-dimensional imaging system corresponding to each pixel to obtain the ideal light vector. The ideal light vectors of each pixel in the original primitive image array form the ideal light vector field; The step of obtaining the real light vector field of the three-dimensional imaging system includes: Perform a difference operation on the pixel coordinates of the calibrated restoration image array corresponding to each pixel in the original primitive image array and the center coordinates of the light control device of the three-dimensional imaging system to obtain the real light vector. The real light vectors of each pixel in the original primitive image array form the real light vector field.

5. A three-dimensional imaging correction method based on a neural network according to claim 1, characterized in that, The step of obtaining the corrected position coordinates of the camera lens center according to the corrected light vector field includes: Perform a difference operation on the pixel coordinates of the calibrated restoration image array corresponding to each pixel in the original primitive image array and the corrected light vector corresponding to the corrected light vector field to obtain the corrected position coordinates of the camera lens center.

6. A three-dimensional imaging correction method based on a neural network according to claim 1, characterized in that, The convolutional neural network includes a calibration convolutional neural network unit and a point spread function unit. The original primitive image array input into the convolutional neural network is processed by the calibration convolutional neural network unit and then convolved with the point spread function unit to obtain the calibrated restoration image array. The point spread function unit is the point spread function array of the light control device of the three-dimensional imaging system.

7. A three-dimensional imaging correction method based on a neural network according to claim 1, characterized in that The training step of the convolutional neural network includes: Obtain multiple side images of the three-dimensional image of the object as a control set; Input the original primitive image array of the object into the convolutional neural network to obtain the calibrated restoration image array; Perform fast Fourier transforms on the calibrated restoration image array and the data in the control set respectively, and perform a loss function operation on the data after the fast Fourier transform of the control set and the data after the fast Fourier transform of the calibrated restoration image array to complete the training and calibration of the convolutional neural network parameters.

8. A three-dimensional imaging correction device based on a neural network, characterized in that, Including: A neural network module for inputting the original primitive image array of the object to be imaged into a pre-trained convolutional neural network to obtain a calibrated restoration image array; A camera correction module for obtaining a corrected optical vector field of the three-dimensional imaging system according to the calibrated restoration image array, and obtaining corrected camera lens center correction position coordinates according to the corrected optical vector field; A correction module for collecting a disparity image of the object to be imaged and synthesizing a corrected primitive image array according to the corrected camera lens center correction position coordinates.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a three-dimensional imaging correction method based on a neural network according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a processor, it implements a three-dimensional imaging correction method based on a neural network according to any one of claims 1 to 7.

Images