Three-dimensional reconstruction method and device of visual target, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a three-dimensional reconstruction method and device for a visual target, electronic equipment and a storage medium, and relates to the technical field of visual perception, and the method comprises the steps: obtaining a video sequence corresponding to an initial three-dimensional model; sampling the video sequence to obtain an original image corresponding to the initial stereoscopic model; obtaining a convolution processing network for the original image, wherein the convolution processing network at least comprises a convolution coding network and a plurality of convolution decoding networks; performing feature extraction on the original image according to the convolutional coding network to obtain coding features corresponding to the original image; each convolutional decoding network is adopted to carry out progressive decoding processing on the coding features, and attribute elements corresponding to each convolutional decoding network are obtained; according to the method, the element sequence corresponding to each attribute element is obtained, progressive three-dimensional reconstruction is performed on the initial three-dimensional model according to the element sequence, and the target three-dimensional model corresponding to the initial three-dimensional model is obtained, so that the implementation cost is reduced, and the reconstruction output effect is ensured.

Description

Technical Field

[0001] The present invention relates to the field of visual perception technology, and in particular to a method for stereoscopic reconstruction of a visual target, a device for stereoscopic reconstruction of a visual target, an electronic device, and a computer-readable storage medium. Background Art

[0002] The goal of stereo reconstruction of visual targets is to use two-dimensional images as input to build a three-dimensional model of the visual targets (such as vehicles and faces). This technology has important application value in many fields. Among them, stereo reconstruction of visual targets requires stereo reconstruction of different visual targets in complex environments and perspectives, so this task is very challenging. Especially under the condition that the contour of the visual target is rough, the performance of current methods and systems is difficult to guarantee. In the process of stereo reconstruction of visual targets, stereo reconstruction of visual targets can be performed by using multi-view supervision or based on prior assumptions. Among them, the method based on multi-view supervision needs to take visual target images of multiple perspectives as input, so the input of this type of method is complex and the implementation cost is high; the method based on prior assumptions uses information such as symmetry and contour as the prior for visual target reconstruction, but rough contours and symmetry prediction errors usually result in poor accuracy. Summary of the invention

[0003] The embodiments of the present invention provide a method, device, electronic device and computer-readable storage medium for stereoscopic reconstruction of a visual target, so as to solve or partially solve the problems of high implementation cost and poor reconstruction output effect in the process of stereoscopic reconstruction of a visual target.

[0004] The embodiment of the present invention discloses a method for stereoscopic reconstruction of a visual target, comprising:

[0005] Obtaining a video sequence corresponding to the initial stereo model;

[0006] Sampling the video sequence to obtain an original image corresponding to the initial stereo model;

[0007] Acquire a convolution processing network for the original image, wherein the convolution processing network at least includes a convolution encoding network and a plurality of convolution decoding networks;

[0008] Extracting features of the original image according to the convolutional coding network to obtain coding features corresponding to the original image;

[0009] Using each of the convolutional decoding networks to perform progressive decoding processing on the encoded features to obtain attribute elements corresponding to each of the convolutional decoding networks;

[0010] The element sequence corresponding to each of the attribute elements is obtained, and the initial stereo model is progressively reconstructed in stereo mode according to the element sequence to obtain a target stereo model corresponding to the initial stereo model.

[0011] The embodiment of the present invention further discloses a stereoscopic reconstruction device of a visual target, comprising:

[0012] A video acquisition module, used for acquiring a video sequence corresponding to the initial stereo model;

[0013] A sampling module, used for sampling the video sequence to obtain an original image corresponding to the initial stereo model;

[0014] A network acquisition module, used to acquire a convolutional processing network for the original image, wherein the convolutional processing network at least includes a convolutional encoding network and a plurality of convolutional decoding networks;

[0015] A coding module, used for performing feature extraction on the original image according to the convolutional coding network to obtain coding features corresponding to the original image;

[0016] A decoding module, used to use each of the convolutional decoding networks to perform progressive decoding processing on the encoded features to obtain attribute elements corresponding to each of the convolutional decoding networks;

[0017] The reconstruction module is used to obtain the element sequence corresponding to each of the attribute elements, and to perform progressive stereoscopic reconstruction on the initial stereoscopic model according to the element sequence to obtain a target stereoscopic model corresponding to the initial stereoscopic model.

[0018] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0019] The memory is used to store computer programs;

[0020] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0021] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0022] The embodiments of the present invention include the following advantages:

[0023] In an embodiment of the present invention, a video sequence corresponding to an initial stereo model is obtained, and then the video sequence is sampled to obtain an original image corresponding to the initial stereo model, and a convolution processing network for the original image is obtained, wherein the convolution processing network at least includes a convolution coding network and a plurality of convolution decoding networks. Then, the original image is firstly feature extracted according to the convolution coding network to obtain a coding feature corresponding to the original image, and then each convolution decoding network is used to progressively decode the coding feature to obtain an attribute element corresponding to each convolution decoding network, and then an element sequence corresponding to each attribute element is obtained. The initial stereo model is progressively reconstructed according to the element sequence to obtain a target stereo model corresponding to the initial stereo model. Thus, in the process of reconstructing the stereo model, a single original image is input for processing without inputting multiple images, thereby effectively reducing the implementation cost. Meanwhile, progressive stereo reconstruction through encoding and multiple attribute elements can achieve a finer-grained modeling effect of multiple dimensions such as shape, texture, affine transformation and background of the visual target, thereby effectively ensuring the reconstruction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flowchart of the steps of a method for stereoscopic reconstruction of a visual target provided in an embodiment of the present invention;

[0025] Figure 2 is a schematic diagram of a process of stereo reconstruction provided in an embodiment of the present invention;

[0026] Figure 3 is a network structure diagram of a convolutional network provided in an embodiment of the present invention;

[0027] Figure 4 is a flowchart of an application scenario provided in an embodiment of the present invention;

[0028] Figure 5 is a schematic diagram of a reconstruction system provided in an embodiment of the present invention;

[0029] Figure 6 is a structural block diagram of a stereoscopic reconstruction device for a visual target provided in an embodiment of the present invention;

[0030] Figure 7 It is a block diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] As an example, methods based on multi-view supervision require visual target images from multiple viewpoints as input, so the input of such methods is complex and the implementation cost is high. Methods based on prior assumptions use information such as symmetry and contour as priors for visual target reconstruction, but rough contours and symmetry prediction errors usually result in very poor accuracy.

[0033] In this regard, in the present invention, by obtaining a video sequence corresponding to the initial stereo model, then sampling the video sequence, obtaining the original image corresponding to the initial stereo model, and obtaining a convolutional processing network for the original image, the convolutional processing network at least includes a convolutional coding network and a plurality of convolutional decoding networks, and then firstly extracting features of the original image according to the convolutional coding network to obtain coding features corresponding to the original image, and then using each convolutional decoding network to perform progressive decoding processing on the coding features to obtain attribute elements corresponding to each convolutional decoding network, and then obtaining the element sequence corresponding to each attribute element, and performing progressive stereo reconstruction on the initial stereo model according to the element sequence to obtain a target stereo model corresponding to the initial stereo model, so that in the process of reconstructing the stereo model, a single original image is input for processing without inputting multiple images, which effectively reduces the implementation cost, and at the same time, progressive stereo reconstruction is performed through encoding and multiple attribute elements, which can achieve a more fine-grained modeling effect of multiple dimensions such as shape, texture, affine transformation and background of the visual target, and effectively ensure the reconstruction effect.

[0034] Reference Figure 1 , shows a flowchart of a method for stereoscopic reconstruction of a visual target provided in an embodiment of the present invention, which may specifically include the following steps:

[0035] Step 101, obtaining a video sequence corresponding to an initial stereo model;

[0036] In the embodiment of the present invention, for the visual target that needs to be stereoscopically reconstructed, a video sequence corresponding to the initial stereoscopic model can be obtained, so as to use the video sequence as basic data for stereoscopic reconstruction. The visual target may include a vehicle, a human face, etc., the initial stereoscopic model may be a model corresponding to the vehicle, the human face, etc., and the video sequence may be a video containing the initial stereoscopic model.

[0037] Step 102, sampling the video sequence to obtain an original image corresponding to the initial stereo model;

[0038] In an embodiment of the present invention, the video sequence may be sampled to obtain an original image corresponding to the initial stereo model. Specifically, corresponding video frames may be extracted from the video sequence at intervals, and the RGB image obtained from each video frame is the original image corresponding to the sampled frame, so that stereo reconstruction is performed by extracting the original image.

[0039] Step 103, obtaining a convolution processing network for the original image, wherein the convolution processing network at least includes a convolution encoding network and a plurality of convolution decoding networks;

[0040] After the original image is determined, a convolution processing network for the original image can be further obtained. The convolution processing network may include a convolution encoding network and several convolution decoding networks. The convolution encoding network may be used to extract features from the original image, and the convolution decoding network may be used to progressively process the features extracted by the convolution encoding network to obtain corresponding attribute elements, and then perform stereoscopic reconstruction of the visual target based on the attribute elements.

[0041] Step 104, extracting features of the original image according to the convolutional coding network to obtain coding features corresponding to the original image;

[0042] In an embodiment of the present invention, the convolutional coding network includes at least a convolutional layer, an activation layer and a batch normalization layer. The original image can be input into the convolutional layer for convolution to extract local features of the original image, and then the local features are input into the activation layer for mapping, nonlinear features are introduced on the local features, and corresponding mapping features are output. Then, the mapping features are input into the batch normalization layer for normalization to obtain the coding features corresponding to the original image, and the coding features are used to characterize the initial stereo model in the feature space.

[0043] Among them, the convolutional coding network can be used as a deep learning model for extracting features from the original image, which may include a convolutional layer, an activation layer, and a batch normalization layer. Through the combination of these layers, the convolutional coding network can extract useful features from the original image for characterizing the initial stereo model in the feature space. In a specific implementation, the convolutional layer can perform a convolution operation on the input image through a convolution kernel (filter) to extract local features of the image, and then the activation layer applies a nonlinear activation function to the output of the convolutional layer to increase the nonlinear expression ability of the network and output the mapping features. Then the batch normalization layer normalizes the mapping features of the activation layer to accelerate the training process and improve the generalization ability of the model, and finally outputs the coding features of the original image.

[0044] Step 105, using each of the convolutional decoding networks to perform progressive decoding processing on the encoded features to obtain attribute elements corresponding to each of the convolutional decoding networks;

[0045] In an embodiment of the present invention, since there are different convolutional decoding networks, after the corresponding coding features are extracted, the coding features can be progressively decoded through each convolutional decoding network in sequence according to a preset order to obtain attribute elements corresponding to each convolutional decoding network.

[0046] In a specific implementation, the convolutional decoding network includes at least a shape decoding network, a material decoding network, an affine transformation network, and a background decoding network. The encoded features include at least three-dimensional vertices and shape codes. The attribute elements include at least object shape attribute elements, texture attribute elements, affine transformation attribute elements, and background attribute elements. The three-dimensional vertices and shape codes can be first input into the shape decoding network for decoding to obtain the corresponding object shape attribute elements. The object shape attribute elements are used to characterize the vertex positions of the initial three-dimensional model. Then, the texture code for the object shape attribute elements is determined, and the texture code and the object shape attribute elements are input into the material decoding network for decoding to generate a target image, and the spherical coordinates corresponding to the target image are obtained. Then, the target image is mapped to a preset sphere through the spherical coordinates, and the two-dimensional pixel points on the target image are associated with each vertex in the sphere to obtain the corresponding texture attribute elements. The texture attribute elements are used Characterize the texture information of the initial three-dimensional model; then, the affine transformation parameters for the texture attribute elements can be obtained, and the affine transformation parameters and the texture attribute elements can be input into the affine transformation decoding network for affine transformation to obtain the corresponding affine transformation attribute elements, and the affine transformation attribute elements and the background codes can be input into the background decoding network, a background image corresponding to the background code is generated through the background decoding network, and the affine transformation attribute elements are superimposed on the background image to obtain the corresponding background attribute elements, and the background attribute elements are used to characterize the background information of the initial three-dimensional model, and progressive three-dimensional reconstruction can be performed through encoding and multiple attribute elements, so as to achieve a finer-grained modeling effect in multiple dimensions such as shape, texture, affine transformation and background of the visual target, and effectively guarantee the reconstruction effect.

[0047] It should be noted that for the shape decoding network, it needs to meet the following requirements: given a three-dimensional vertex x of the ellipsoid, the shape deformation module Among them, s θ is a multilayer perceptron that takes a 3D vertex x and a shape code z sh as input.

[0048] For the material decoding network, it needs to satisfy: given a texture code z tx , using the convolutional network t θ Generate image t θ (z tx ), mapped onto a sphere using spherical coordinates, associating a 2D point with each vertex of the ellipsoid, and thus with each vertex of the shape mesh.

[0049] For the affine transformation decoding network, scaling and the most likely 6D pose can be combined in the network. a = {s, r, t}, where s, r, t correspond to the three scales of anisotropic scaling, the three Euler angles of a rotation, and the three coordinates of a translation. A three-dimensional point x on the grid is represented by A a (x) = rot(r) diag(s) x + t transformation, where rot(r) is the rotation matrix associated with r and diag(s) is the diagonal matrix associated with s. The affine transformation is applied to all points of the texture mesh T, resulting in a pose mesh P = A a (T).

[0050] For the background decoding network, it can be composed of the background code z bg Generate, specifically a differentiable module In this background image b θ (z bg ) to obtain a reconstructed image

[0051] Step 106 , obtaining the element sequence corresponding to each of the attribute elements, and performing progressive stereoscopic reconstruction on the initial stereoscopic model according to the element sequence to obtain a target stereoscopic model corresponding to the initial stereoscopic model.

[0052] After obtaining each attribute element, the element order corresponding to the attribute element can be obtained, and then each attribute element is used in turn according to the element order to perform progressive stereo reconstruction on the initial stereo model to obtain a target stereo model corresponding to the initial stereo model. Therefore, in the process of reconstructing the stereo model, a single original image is input for processing without inputting multiple images, thereby reducing the implementation cost. At the same time, progressive stereo reconstruction through encoding and multiple attribute elements can achieve finer-grained modeling effects in multiple dimensions such as shape, texture, affine transformation and background of the visual target, effectively ensuring the reconstruction effect.

[0053] In an embodiment of the present invention, progressive stereo reconstruction means taking the initial stereo model as input, sequentially using the corresponding attribute elements to perform stereo reconstruction on the model of the previous stage, and using the output model as the reconstruction input of the next stage, until all four groups of attribute elements are involved in the reconstruction and the final stereo model is obtained.

[0054] In a specific implementation, the initial stereo model is an ellipsoid model, and the order of elements may be shape attribute element>texture attribute element>affine change attribute element>background attribute element. Then, for the stereo reconstruction process, the object shape attribute element may be used to perform shape processing on the ellipsoid model to obtain a first intermediate model corresponding to the object shape attribute element. Then, the texture attribute element may be used to perform texture processing on the first intermediate model to obtain a second intermediate model corresponding to the texture attribute element. Then, the affine transformation attribute element may be used to perform affine transformation on the second intermediate model to obtain a third intermediate model corresponding to the affine change attribute element. Finally, the background attribute element may be used to perform background processing on the third intermediate model to obtain a target stereo model corresponding to the initial stereo model.

[0055] It should be noted that the embodiments of the present invention include but are not limited to the above examples. It is understandable that those skilled in the art can also make settings according to actual needs under the guidance of the ideas of the embodiments of the present invention, and the present invention is not limited to this.

[0056] In an embodiment of the present invention, a video sequence corresponding to an initial stereo model is obtained, and then the video sequence is sampled to obtain an original image corresponding to the initial stereo model, and a convolution processing network for the original image is obtained, wherein the convolution processing network at least includes a convolution coding network and a plurality of convolution decoding networks. Then, the original image is firstly feature extracted according to the convolution coding network to obtain a coding feature corresponding to the original image, and then each convolution decoding network is used to progressively decode the coding feature to obtain an attribute element corresponding to each convolution decoding network, and then an element sequence corresponding to each attribute element is obtained. The initial stereo model is progressively reconstructed according to the element sequence to obtain a target stereo model corresponding to the initial stereo model. Thus, in the process of reconstructing the stereo model, a single original image is input for processing without inputting multiple images, thereby effectively reducing the implementation cost. Meanwhile, progressive stereo reconstruction through encoding and multiple attribute elements can achieve a finer-grained modeling effect of multiple dimensions such as shape, texture, affine transformation and background of the visual target, thereby effectively ensuring the reconstruction effect.

[0057] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are used for exemplary description:

[0058] Reference Figure 2 , shows a schematic diagram of the process of stereo reconstruction provided in an embodiment of the present invention, which specifically includes the following steps: taking a video sequence as input,

[0059] Step 1: Image sampling S1, performing image sampling on the video sequence to obtain a sampled frame as the original image;

[0060] Step 2: Convolutional coding network processing S2, using the convolutional coding network to process the original image and obtain coding features;

[0061] Step 3: Convolutional decoding network processing S3, using four different convolutional decoding networks to progressively process the encoded features, and obtain a feature set containing four groups of attribute elements. Among them, the four different convolutional decoding networks include a shape decoding network, a material decoding network, an affine transformation decoding network, and a background decoding network. The feature set contains four groups of attribute elements, which are object shape attribute elements, texture attribute elements, affine transformation attribute elements, and background attribute elements in order;

[0062] Step 4: Progressive stereo reconstruction S4, progressively reconstruct the initialized stereo model in the order of the attribute elements in the element set, and finally obtain a stereo model after the original image is reconstructed. Progressive stereo reconstruction means taking the initialized stereo model as input, sequentially using the corresponding attribute elements to perform stereo reconstruction on the previous stage model, and outputting the model as the next stage reconstruction input, until all four sets of attribute elements participate in the reconstruction and obtain the final stereo model.

[0063] Reference Figure 3 , shows a network structure diagram of a convolutional network provided in an embodiment of the present invention, thereby illustrating the data input and output of each step of operation;

[0064] S1: The input is a video, i.e. a 3-channel RGB color image sequence of length L, so the data dimension is 3×L×W×H. A single frame is extracted from the video as the original image, so the data dimension is 3×W×H;

[0065] S2: The convolutional coding network processes the sampled original image to obtain corresponding coding features. The convolutional coding network includes a convolution layer, an activation layer, a batch normalization layer, a pooling layer, etc. The input is the original image of the sampled frame, and the output is the frame-level coding features as the visual semantic representation of the original image;

[0066] S3: Four different convolutional decoding networks process the encoded features as output, including shape decoding network, material decoding network, affine transformation decoding network and background decoding network. The feature set includes object shape attribute elements, texture attribute elements, affine transformation attribute elements and background attribute elements;

[0067] S4: Progressive stereo reconstruction takes the feature set and the initialized stereo model as input, uses the corresponding attribute elements in turn to reconstruct the previous stage model, and outputs the model as the input for the next stage reconstruction, until all four sets of attribute elements are involved in the reconstruction and the final stereo model is output.

[0068] Reference Figure 4, shows a flow chart of an application scenario provided in an embodiment of the present invention, which includes successively reconstructing the initialized three-dimensional model according to shape attribute elements, reconstructing according to texture attribute elements, reconstructing according to affine transformation elements and reconstructing according to background attribute elements, and finally obtaining a reconstructed three-dimensional model.

[0069] Reference Figure 5 , shows a schematic diagram of a reconstruction system provided in an embodiment of the present invention, which can be used for stereoscopic reconstruction of visual targets in video signals or image sequences. The system may at least include:

[0070] An image sampling unit, a coding network processing unit, a decoding network processing unit, a stereo reconstruction unit, and a stereo model output unit. The image sampling unit is used to sample the video sequence to obtain a sampling frame as the original image; the coding network processing unit is used to abstract the original image using a convolutional coding network to obtain a coding feature representing the original image; the decoding network processing unit is used to process the coding feature using four different convolutional decoding networks to obtain an element set including four groups of attribute elements; the stereo reconstruction unit is used to perform stereo reconstruction on the initialized stereo model according to the four groups of attribute elements; and the stereo model output unit is used to output the stereo model reconstructed from the original image. Specifically, the output of the image sampling unit is used as the input of the coding network processing unit; the output of the coding network processing unit is used as the input of the decoding network processing unit; the output of the decoding network processing unit is used as the input of the stereo reconstruction unit; and the output of the stereo reconstruction unit is used as the input of the stereo model output unit.

[0071] Compared with the prior art, the embodiments of the present invention have at least the following advantages and beneficial effects:

[0072] The input of the method of the present invention is only a single, single-view sampling image. Compared with the mainstream method, it does not need to input multiple, multi-view images as supervision information, so the implementation cost is lower;

[0073] The method of the present invention does not require prior assumptions such as symmetry and contour during stereo reconstruction, and can avoid adverse reconstruction effects caused by prior errors.

[0074] Compared with the traditional method, the method of the present invention adopts an innovative progressive stereo reconstruction strategy, which can achieve a more fine-grained modeling effect on the shape, texture, affine transformation and background of the visual target;

[0075] The method of the present invention is highly flexible, and different hyperparameters are reserved for setting. A more optimized set of hyperelements can be selected according to specific scenarios and visual objectives.

[0076] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0077] Reference Figure 6 , shows a structural block diagram of a stereoscopic reconstruction device for a visual target provided in an embodiment of the present invention, which may specifically include the following modules:

[0078] The video acquisition module 601 is used to acquire a video sequence corresponding to the initial stereo model;

[0079] A sampling module 602 is used to sample the video sequence to obtain an original image corresponding to the initial stereo model;

[0080] A network acquisition module 603 is used to acquire a convolutional processing network for the original image, wherein the convolutional processing network at least includes a convolutional encoding network and a plurality of convolutional decoding networks;

[0081] The encoding module 604 is used to extract features of the original image according to the convolutional encoding network to obtain encoding features corresponding to the original image;

[0082] A decoding module 605, configured to use each of the convolutional decoding networks to perform progressive decoding processing on the encoded features to obtain attribute elements corresponding to each of the convolutional decoding networks;

[0083] The reconstruction module 606 is used to obtain the element sequence corresponding to each of the attribute elements, and perform progressive stereoscopic reconstruction on the initial stereoscopic model according to the element sequence to obtain a target stereoscopic model corresponding to the initial stereoscopic model.

[0084] In some feasible implementations, the convolutional coding network includes at least a convolutional layer, an activation layer, and a batch normalization layer, and the encoding module 604 is specifically used to:

[0085] Inputting the original image into the convolution layer for convolution to extract local features of the original image;

[0086] Inputting the local features into the activation layer for mapping, introducing nonlinear features on the local features, and outputting corresponding mapping features;

[0087] The mapping features are input into the batch normalization layer for normalization processing to obtain encoding features corresponding to the original image, and the encoding features are used to characterize the initial stereo model in the feature space.

[0088] In some feasible implementations, the convolutional decoding network includes at least a shape decoding network, the encoding features include at least three-dimensional vertices and shape codes, the attribute elements include at least object shape attribute elements, and the decoding module 605 is specifically used to:

[0089] The three-dimensional vertices and the shape codes are input into the shape decoding network for decoding to obtain corresponding object shape attribute elements, where the object shape attribute elements are used to characterize the vertex positions of the initial three-dimensional model.

[0090] In some feasible implementations, the convolutional decoding network further includes a material decoding network, the attribute elements at least include texture attribute elements, and the decoding module 605 is further specifically used for:

[0091] Determining a texture code for a shape attribute element of the object;

[0092] Input the texture code and the object shape attribute element into the material decoding network for decoding, generate a target image, and obtain the spherical coordinates corresponding to the target image;

[0093] Mapping the target image to a preset sphere through the spherical coordinates, associating the two-dimensional pixel points on the target image with each vertex in the sphere, and obtaining corresponding texture attribute elements;

[0094] The texture attribute elements are used to represent the texture information of the initial three-dimensional model.

[0095] In some feasible implementations, the convolutional decoding network at least further includes the affine transformation decoding network, the attribute elements at least further include affine transformation attribute elements, and the decoding module 605 is specifically further used for:

[0096] Obtaining affine transformation parameters for the texture attribute elements;

[0097] Inputting the affine transformation parameters and the texture attribute elements into the affine transformation decoding network to perform affine transformation, and obtaining corresponding affine transformation attribute elements;

[0098] The affine transformation attribute element is used to characterize the posture information of the initial three-dimensional model.

[0099] In some feasible implementations, the convolutional decoding network at least further includes a background decoding network, the attribute elements at least further include background attribute elements, and the decoding module 605 is specifically further used for:

[0100] Obtaining a background code for the affine transformation attribute element;

[0101] Inputting the affine transformation attribute element and the background code into the background decoding network, generating a background image corresponding to the background code through the background decoding network, and superimposing the affine transformation attribute element on the background image to obtain the corresponding background attribute element;

[0102] The background attribute elements are used to represent the background information of the initial three-dimensional model.

[0103] In some feasible implementations, the attribute elements at least include object shape attribute elements, texture attribute elements, affine transformation attribute elements and background attribute elements, the initial three-dimensional model is an ellipsoid model, and the reconstruction module 606 is specifically used to:

[0104] Performing shape processing on the ellipsoid model using the object shape attribute elements to obtain a first intermediate model corresponding to the object shape attribute elements;

[0105] Performing texture processing on the first intermediate model using the texture attribute element to obtain a second intermediate model corresponding to the texture attribute element;

[0106] Performing an affine transformation on the second intermediate model using the affine transformation attribute element to obtain a third intermediate model corresponding to the affine transformation attribute element;

[0107] The background attribute elements are used to perform background processing on the third intermediate model to obtain a target three-dimensional model corresponding to the initial three-dimensional model.

[0108] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0109] In addition, an embodiment of the present invention further provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, each process of the embodiment of the stereoscopic reconstruction method for the above-mentioned visual target is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0110] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned stereoscopic reconstruction method embodiment of the visual target is realized, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0111] Figure 7 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0112] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, a processor 710, and a power supply 711. Those skilled in the art will appreciate that the electronic device structure involved in the embodiments of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently. In the embodiments of the present invention, the electronic device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable device, and a pedometer.

[0113] It should be understood that in the embodiment of the present invention, the radio frequency unit 701 can be used for receiving and sending signals during information transmission or communication. Specifically, after receiving downlink data from the base station, it is sent to the processor 710 for processing; in addition, uplink data is sent to the base station. Generally, the radio frequency unit 701 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 701 can also communicate with the network and other devices through a wireless communication system.

[0114] The electronic device provides users with wireless broadband Internet access through the network module 702, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0115] The audio output unit 703 can convert the audio data received by the RF unit 701 or the network module 702 or stored in the memory 709 into an audio signal and output it as sound. Moreover, the audio output unit 703 can also provide audio output related to a specific function performed by the electronic device 700 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 703 includes a speaker, a buzzer, a receiver, etc.

[0116] The input unit 704 is used to receive audio or video signals. The input unit 704 may include a graphics processor (GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 706. The image frame processed by the graphics processor 7041 can be stored in the memory 709 (or other storage medium) or sent via the radio frequency unit 701 or the network module 702. The microphone 7042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 701 in the case of a telephone call mode.

[0117] The electronic device 700 also includes at least one sensor 705, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 7061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 7061 and / or the backlight when the electronic device 700 is moved to the ear. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used to identify the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; the sensor 705 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0118] The display unit 706 is used to display information input by the user or information provided to the user. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0119] The user input unit 707 can be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 7071 or near the touch panel 7071 using any suitable object or accessory such as a finger, stylus, etc.). The touch panel 7071 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the contact point coordinates, and then sends it to the processor 710, receives the command sent by the processor 710 and executes it. In addition, the touch panel 7071 can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic waves. In addition to the touch panel 7071, the user input unit 707 may also include other input devices 7072. Specifically, other input devices 7072 may include but are not limited to a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail here.

[0120] Further, the touch panel 7071 may be covered on the display panel 7061. When the touch panel 7071 detects a touch operation on or near it, it is transmitted to the processor 710 to determine the type of the touch event, and then the processor 710 provides a corresponding visual output on the display panel 7061 according to the type of the touch event. It can be understood that in one embodiment, the touch panel 7071 and the display panel 7061 are used as two independent components to implement the input and output functions of the electronic device, but in some embodiments, the touch panel 7071 and the display panel 7061 can be integrated to implement the input and output functions of the electronic device, which is not limited here.

[0121] The interface unit 708 is an interface for connecting an external device to the electronic device 700. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 708 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the electronic device 700 or may be used to transmit data between the electronic device 700 and an external device.

[0122] The memory 709 can be used to store software programs and various data. The memory 709 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 709 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0123] The processor 710 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 709, and calling data stored in the memory 709, so as to monitor the electronic device as a whole. The processor 710 may include one or more processing units; preferably, the processor 710 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 710.

[0124] The electronic device 700 may also include a power supply 711 (such as a battery) for supplying power to each component. Preferably, the power supply 711 may be logically connected to the processor 710 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0125] In addition, the electronic device 700 includes some functional modules not shown, which will not be described in detail here.

[0126] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0128] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0129] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0131] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0132] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0133] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0134] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0135] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for stereoscopic reconstruction of a visual target, characterized in that: include: Obtaining a video sequence corresponding to the initial stereo model; Sampling the video sequence to obtain an original image corresponding to the initial stereo model; Acquire a convolution processing network for the original image, wherein the convolution processing network at least includes a convolution encoding network and a plurality of convolution decoding networks; Extracting features of the original image according to the convolutional coding network to obtain coding features corresponding to the original image; Using each of the convolutional decoding networks to perform progressive decoding processing on the encoded features to obtain attribute elements corresponding to each of the convolutional decoding networks; The element sequence corresponding to each of the attribute elements is obtained, and the initial stereo model is progressively reconstructed in stereo mode according to the element sequence to obtain a target stereo model corresponding to the initial stereo model.

2. The method according to claim 1, characterized in that The convolutional coding network includes at least a convolutional layer, an activation layer, and a batch normalization layer, and the extracting features of the original image according to the convolutional coding network to obtain the coding features corresponding to the original image includes: Inputting the original image into the convolution layer for convolution to extract local features of the original image; Inputting the local features into the activation layer for mapping, introducing nonlinear features on the local features, and outputting corresponding mapping features; The mapping features are input into the batch normalization layer for normalization processing to obtain encoding features corresponding to the original image, and the encoding features are used to characterize the initial stereo model in the feature space.

3. The method according to claim 1, characterized in that The convolutional decoding network at least includes a shape decoding network, the coding features at least include three-dimensional vertices and shape codes, the attribute elements at least include object shape attribute elements, and the progressive decoding process of the coding features using each of the convolutional decoding networks to obtain the attribute elements corresponding to each of the convolutional decoding networks includes: The three-dimensional vertices and the shape codes are input into the shape decoding network for decoding to obtain corresponding object shape attribute elements, where the object shape attribute elements are used to characterize the vertex positions of the initial three-dimensional model.

4. The method according to claim 3, characterized in that The convolutional decoding network further includes a material decoding network, the attribute elements further include at least texture attribute elements, and the method of using each of the convolutional decoding networks to perform progressive decoding processing on the coded features to obtain attribute elements corresponding to each of the convolutional decoding networks further includes: Determining a texture code for a shape attribute element of the object; Input the texture code and the object shape attribute element into the material decoding network for decoding, generate a target image, and obtain the spherical coordinates corresponding to the target image; Mapping the target image to a preset sphere through the spherical coordinates, associating the two-dimensional pixel points on the target image with each vertex in the sphere, and obtaining corresponding texture attribute elements; The texture attribute elements are used to represent the texture information of the initial three-dimensional model.

5. The method according to claim 4, characterized in that The convolutional decoding network at least further includes the affine transformation decoding network, the attribute elements at least further include affine transformation attribute elements, and the step of using each of the convolutional decoding networks to perform progressive decoding processing on the coding features to obtain attribute elements corresponding to each of the convolutional decoding networks further includes: Obtaining affine transformation parameters for the texture attribute elements; Inputting the affine transformation parameters and the texture attribute elements into the affine transformation decoding network to perform affine transformation, and obtaining corresponding affine transformation attribute elements; The affine transformation attribute element is used to characterize the posture information of the initial three-dimensional model.

6. The method according to claim 5, characterized in that The convolutional decoding network at least further includes a background decoding network, the attribute elements at least further include background attribute elements, and the method of using each of the convolutional decoding networks to perform progressive decoding processing on the coding features to obtain attribute elements corresponding to each of the convolutional decoding networks further includes: Obtaining a background code for the affine transformation attribute element; Inputting the affine transformation attribute element and the background code into the background decoding network, generating a background image corresponding to the background code through the background decoding network, and superimposing the affine transformation attribute element on the background image to obtain the corresponding background attribute element; The background attribute elements are used to represent the background information of the initial three-dimensional model.

7. The method according to claim 1, characterized in that The attribute elements at least include object shape attribute elements, texture attribute elements, affine transformation attribute elements and background attribute elements, the initial stereo model is an ellipsoid model, and the initial stereo model is progressively reconstructed in accordance with the element sequence to obtain a target stereo model corresponding to the initial stereo model, including: Performing shape processing on the ellipsoid model using the object shape attribute elements to obtain a first intermediate model corresponding to the object shape attribute elements; Performing texture processing on the first intermediate model using the texture attribute element to obtain a second intermediate model corresponding to the texture attribute element; Performing an affine transformation on the second intermediate model using the affine transformation attribute element to obtain a third intermediate model corresponding to the affine transformation attribute element; The background attribute elements are used to perform background processing on the third intermediate model to obtain a target three-dimensional model corresponding to the initial three-dimensional model.

8. A stereoscopic reconstruction device for a visual target, characterized in that: include: A video acquisition module, used for acquiring a video sequence corresponding to the initial stereo model; A sampling module, used for sampling the video sequence to obtain an original image corresponding to the initial stereo model; A network acquisition module, used to acquire a convolutional processing network for the original image, wherein the convolutional processing network at least includes a convolutional encoding network and a plurality of convolutional decoding networks; A coding module, used for performing feature extraction on the original image according to the convolutional coding network to obtain coding features corresponding to the original image; A decoding module, used to use each of the convolutional decoding networks to perform progressive decoding processing on the encoded features to obtain attribute elements corresponding to each of the convolutional decoding networks; The reconstruction module is used to obtain the element sequence corresponding to each of the attribute elements, and to perform progressive stereoscopic reconstruction on the initial stereoscopic model according to the element sequence to obtain a target stereoscopic model corresponding to the initial stereoscopic model.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images