Image processing method, deep learning model training method and device

Abstract

The present disclosure provides an image processing method, a deep learning model training method and a device, which relate to the field of artificial intelligence technology, especially to the technical fields of computer vision, augmented reality, virtual reality, deep learning, etc., and can be applied to the scene of the metaverse. The specific implementation scheme of the image processing method is: based on a predetermined image, a key point alignment operation is performed on the image to be processed to obtain a target two-dimensional image; image features of the target two-dimensional image are extracted, wherein the image features represent target features that match the reconstruction parameters for three-dimensional reconstruction of the image to be processed; the image features are identified to obtain target parameters for three-dimensional reconstruction of the image to be processed; and based on the target parameters, the target two-dimensional image is three-dimensionally reconstructed to obtain a target three-dimensional image.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, augmented reality, virtual reality, and deep learning, and specifically to image processing methods, deep learning model training methods and devices. Background Art

[0002] Three-dimensional reconstruction based on monocular vision refers to the process of inferring the depth information of an image through the features of a single or multiple two-dimensional images, and reconstructing a three-dimensional image based on the depth information.

[0003] With the widespread application of 3D reconstruction technology in film and television special effects, virtual images, AR (Augmented Reality), VR (Virtual Reality) and other fields, the demand for the reconstruction accuracy of 3D images is also increasing. Summary of the invention

[0004] The present invention provides an image processing method, a deep learning model training method and a device.

[0005] According to one aspect of the present disclosure, there is provided an image processing method, comprising: based on a predetermined image, performing a key point alignment operation on an image to be processed to obtain a target two-dimensional image; extracting image features of the target two-dimensional image, wherein the image features represent target features that match reconstruction parameters for three-dimensional reconstruction of the image to be processed; identifying the image features to obtain target parameters for three-dimensional reconstruction of the image to be processed; and based on the target parameters, performing three-dimensional reconstruction on the target two-dimensional image to obtain a target three-dimensional image.

[0006] According to another aspect of the present disclosure, a training method for a deep learning model is provided, comprising: using an initial model to perform the following operations on a sample two-dimensional image: based on a predetermined image, performing a key point alignment operation on the sample two-dimensional image to obtain a target sample two-dimensional image; performing local masking on the target sample two-dimensional image to obtain a mask image; extracting mask image features of the mask image and sample image features of the target sample two-dimensional image; identifying the mask image features to obtain target sample parameters for three-dimensionally reconstructing the sample two-dimensional image; based on the target sample parameters, performing three-dimensional reconstruction on the target sample two-dimensional image to obtain a sample three-dimensional image corresponding to the sample two-dimensional image; based on a target loss function, obtaining a loss value according to the mask image features, the sample image features, the sample two-dimensional image and the sample three-dimensional image; and based on the loss value, adjusting the model parameters of the initial model to obtain a trained deep learning model.

[0007] According to another aspect of the present disclosure, there is provided an image processing device, including: a first alignment module, a first feature extraction module, a first feature recognition module and a first feature reconstruction module. The first alignment module is used to perform a key point alignment operation on the image to be processed based on a predetermined image to obtain a target two-dimensional image. The first feature extraction module is used to extract image features of the target two-dimensional image, wherein the image features represent target features that match reconstruction parameters for three-dimensional reconstruction of the image to be processed. The first feature recognition module is used to recognize image features to obtain target parameters for three-dimensional reconstruction of the image to be processed. The first feature reconstruction module is used to perform three-dimensional reconstruction of the target two-dimensional image based on the target parameters to obtain a target three-dimensional image.

[0008] According to another aspect of the present disclosure, a training device for a deep learning model is provided, comprising: a second alignment module, a mask module, a second feature extraction module, a second feature recognition module, a second feature reconstruction module, a loss calculation module and an adjustment module. The second alignment module is used to perform a key point alignment operation on a sample two-dimensional image based on a predetermined image to obtain a target sample two-dimensional image. The mask module is used to perform local mask processing on the target sample two-dimensional image to obtain a mask image. The second feature extraction module is used to extract mask image features of the mask image and sample image features of the target sample two-dimensional image. The second feature recognition module is used to recognize the mask image features to obtain target sample parameters for three-dimensional reconstruction of the sample two-dimensional image. The second feature reconstruction module is used to perform three-dimensional reconstruction of the target sample two-dimensional image based on the target sample parameters to obtain a sample three-dimensional image corresponding to the sample two-dimensional image. The loss calculation module is used to obtain a loss value based on the target loss function according to the mask image features, the sample image features, the sample two-dimensional image and the sample three-dimensional image. The adjustment module is used to adjust the model parameters of the initial model based on the loss value to obtain a trained deep learning model.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above method.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and the computer program implements the above method when executed by a processor.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 An exemplary system architecture of a training method and device to which an image processing method or a deep learning model can be applied according to an embodiment of the present disclosure is schematically shown;

[0015] Figure 2 The flowchart of the image processing method according to the embodiment of the present disclosure is schematically shown;

[0016] Figure 3 A schematic diagram schematically shows a key point alignment process for an image to be processed based on a predetermined image according to an embodiment of the present disclosure;

[0017] Figure 4 A schematic diagram schematically shows an image processing method according to an embodiment of the present disclosure;

[0018] Figure 5 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 6 A schematic diagram schematically shows a framework of a method for training a deep learning model according to an embodiment of the present disclosure;

[0020] Figure 7 A block diagram schematically shows an image processing device according to an embodiment of the present disclosure;

[0021] Figure 8 A block diagram schematically shows a training device for a deep learning model according to an embodiment of the present disclosure; and

[0022] Fig. 9 A block diagram of an electronic device suitable for implementing an image processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] The 3D reconstruction methods in related technologies mainly include the following two types: 1. 3D reconstruction based on virtual 3D data and 3D models; 2. Unsupervised learning and training models based on 2D image data, such as Detailed Expression Capture and Animation (DECA).

[0025] However, virtual three-dimensional data collection is difficult. The reconstruction parameters output by the unsupervised learning training model based on two-dimensional image data have large deviations from the three-dimensional parameters of the initial image, and there are also large differences in texture fitting, resulting in poor reconstruction effects.

[0026] In view of this, the embodiment of the present disclosure provides an image processing method, which extracts image features that match the reconstruction parameters for three-dimensional reconstruction of the image to be processed, and identifies the image features to obtain relatively accurate reconstruction parameters. Then, the two-dimensional image is reconstructed based on the reconstruction parameters, so that the reconstruction accuracy of the image can be improved.

[0027] Figure 1 An exemplary system architecture of a training method and apparatus for an image processing method or a deep learning model to which an embodiment of the present disclosure can be applied is schematically illustrated.

[0028] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the image processing method and apparatus can be applied may include a terminal device, but the terminal device may implement the image processing method and apparatus provided by the embodiments of the present disclosure without interacting with a server.

[0029] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0030] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0031] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0032] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0033] It should be noted that the image processing method or deep learning model training method provided in the embodiments of the present disclosure can generally be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Correspondingly, the image processing apparatus provided in the embodiments of the present disclosure can also be set in the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0034] Alternatively, the image processing method provided in the embodiment of the present disclosure may also be generally executed by the server 105. Accordingly, the image processing apparatus provided in the embodiment of the present disclosure may generally be arranged in the server 105. The image processing method provided in the embodiment of the present disclosure may also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the image processing apparatus provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0035] For example, the first terminal device 101, the second terminal device 102, and the third terminal device 103 can obtain the image to be processed, and then send the obtained image to be processed to the server 105, and the server 105 performs key point alignment operation on the image to be processed based on the predetermined image to obtain a target two-dimensional image; extracts image features of the target two-dimensional image, wherein the image features represent target features that match reconstruction parameters for three-dimensional reconstruction of the image to be processed; recognizes the image features to obtain target parameters for three-dimensional reconstruction of the image to be processed; and performs three-dimensional reconstruction on the target two-dimensional image based on the target parameters to obtain a target three-dimensional image. Alternatively, a server or server cluster that can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105 performs operations such as alignment, image feature extraction, feature recognition, and three-dimensional reconstruction on the image to be processed, and finally obtains a target three-dimensional image.

[0036] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0037] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0038] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0039] Figure 2 The flowchart of the image processing method according to the embodiment of the present disclosure is schematically shown.

[0040] like Figure 2 As shown, the method includes operations S210 to S240.

[0041] In operation S210, based on a predetermined image, a key point alignment operation is performed on the image to be processed to obtain a target two-dimensional image.

[0042] In operation S220, image features of the target two-dimensional image are extracted.

[0043] In operation S230, image features are identified to obtain target parameters for three-dimensional reconstruction of the image to be processed.

[0044] In operation S240 , the target two-dimensional image is three-dimensionally reconstructed based on the target parameters to obtain a target three-dimensional image.

[0045] According to an embodiment of the present disclosure, the image to be processed may be a two-dimensional image collected for a target part of a target object. The target part may be any part of the target object, for example, it may be a face. The predetermined image may be an image containing key points of the target part with a predetermined pixel size. For example, when the target part is a face, the key points of the target part may be the eyes or nose of the target object. The predetermined pixel size is 256x256 pixels.

[0046] According to an embodiment of the present disclosure, performing a key point alignment operation on the image to be processed may be to place the key point coordinates in the predetermined image and the key point coordinates in the image to be processed on the same horizontal line or to make the key point coordinates coincide with each other.

[0047] For example: in the image to be processed, the key point of the target object may be the pupil center point O1 (m, y1) of the target object. The predetermined image may include multiple key points of different parts of the target object. The standard point O2 (n, y2) corresponding to the pupil center point in the image to be processed can be determined from the multiple key points. The image to be processed can be adjusted so that the pupil center point O1 (m, y1) coincides with the standard point O2 (n, y2) to obtain the target two-dimensional image.

[0048] According to an embodiment of the present disclosure, a target two-dimensional image may be input into a trained vision transformer (ViT) to obtain image features. The image features represent target features that match reconstruction parameters for three-dimensional reconstruction of the image to be processed.

[0049] According to an embodiment of the present disclosure, there may be n reconstruction parameters for three-dimensionally reconstructing the image to be processed. Correspondingly, n linear layers may be used to identify image features, and each linear layer outputs a three-dimensional reconstruction parameter.

[0050] According to an embodiment of the present disclosure, a 3DMM model (3D Morphable Model, 3D deformation statistical model) can be used to perform three-dimensional reconstruction of a target two-dimensional image based on target parameters. The target parameters may include expression parameters, texture parameters, lighting parameters, camera parameters, and identity parameters.

[0051] According to the embodiments of the present disclosure, by extracting image features that match the reconstruction parameters of the image to be processed for three-dimensional reconstruction and identifying the image features, relatively accurate reconstruction parameters are obtained. Then, the two-dimensional image is reconstructed based on the reconstruction parameters, so that the reconstruction accuracy of the image can be improved.

[0052] According to an embodiment of the present disclosure, based on a predetermined image, a key point alignment operation is performed on the image to be processed to obtain a target two-dimensional image, which may include the following operations: performing key point detection on the image to be processed to obtain a first key point group; performing key point detection on the predetermined image to obtain a second key point group; and based on the correspondence between the key points in the first key point group and the second key point group, performing an alignment operation on the image to be processed to obtain a target two-dimensional image.

[0053] For example, a key point detection algorithm is used to detect key points of a processed image, and the first key point group obtained may include the first coordinates of points of key parts such as eyes, nose, and mouth. Similarly, a key point detection algorithm is used to detect key points of a predetermined image, and the second key point group obtained may include the second coordinates of points of key parts such as eyes, nose, and mouth.

[0054] According to an embodiment of the present disclosure, the predetermined image may be an image of a predetermined pixel size used when training a visual deformation network.

[0055] According to an embodiment of the present disclosure, the correspondence between the key points in the first key point group and the second key point group can characterize the correspondence between different types of key points, for example, the key points of the eye part of the target object in the image to be processed correspond to the key points of the eye part of the predetermined object in the predetermined image. The key points of the nose part of the target object in the image to be processed correspond to the key points of the nose part of the predetermined object in the predetermined image.

[0056] According to an embodiment of the present disclosure, a key point alignment algorithm, such as a Procrustes analysis algorithm, may be used to perform an alignment operation on the image to be processed to obtain a target two-dimensional image.

[0057] According to an embodiment of the present disclosure, an alignment operation is performed on the processed image based on the correspondence between the key points in the first key point group and the second key point group to obtain a target two-dimensional image, which may include the following operations: based on the correspondence between the key points in the first key point group and the second key point group, position offset information between key points with a corresponding relationship is obtained from the first key point group and the second key point group; and according to the position offset information, the processed image is scaled to obtain a target two-dimensional image.

[0058] For example, the first key point group may include three key points of the target object's eyes. The three key points may include two key points at the ends of the eyes and a key point at the pupil position. The second key point group may include multiple key points of the target object's eyes. The multiple key points may include multiple key points on the upper eyelid, multiple key points on the lower eyelid, and a key point at the pupil position.

[0059] Since the eye shapes of different target objects are different, the positions of the key points at the beginning and end of the eyes are also different. Therefore, the key point closest to the beginning or end of the eye in the first key point group among the multiple key points on the upper eyelid in the second key point group can be determined as the key point with a corresponding relationship with the first key point group. And based on the position offset information between the key point and the key point at the beginning or end of the eye in the first key point group, it is used as the position offset information between the first key point group and the second key point group.

[0060] According to an embodiment of the present disclosure, the position offset information between the first key point group and the second key point group may also be an average value or a weighted sum value of the position offsets between a plurality of key points having a corresponding relationship.

[0061] According to an embodiment of the present disclosure, based on the position offset information, the image to be processed may be scaled so that key points in the image to be processed coincide with key points in a predetermined image or are located at the same horizontal position.

[0062] Figure 3 A schematic diagram schematically shows a key point alignment process for an image to be processed based on a predetermined image according to an embodiment of the present disclosure.

[0063] like Figure 3 As shown, in the embodiment 300, the key point detection is performed on the image to be processed 311 to obtain a first key point group 313 including three key points P1, P2, and P3. The key point detection is performed on the predetermined image 312 to obtain a second key point group 314 including three key points m1, m2, and m3 corresponding to P1, P2, and P3. Based on the correspondence between the key points, that is, P1 corresponds to m1, P2 corresponds to m2, and P3 corresponds to m3, a key point position offset 315 is obtained. Based on the key point position offset 315, the image to be processed 311 is scaled to obtain a target two-dimensional image 316 aligned with the key points of the predetermined image 312.

[0064] According to an embodiment of the present disclosure, by performing a key point alignment operation on the image to be processed, the image features extracted from the target two-dimensional image are aligned with the image features of the sample two-dimensional image extracted during the visual deformation network training, and accurate three-dimensional reconstruction parameters are obtained by identifying the image features.

[0065] According to an embodiment of the present disclosure, extracting image features of a target two-dimensional image may include the following operations: segmenting the target two-dimensional image according to a predetermined pixel size to obtain a plurality of pixel blocks; and processing the plurality of pixel blocks based on a self-attention mechanism to obtain image features.

[0066] For example, the size of the target two-dimensional image may be 256x256 pixels, the predetermined pixel size may be 16x16 pixels, and 256 pixel blocks may be obtained by pixel segmentation of the target two-dimensional image according to the predetermined pixel size. Since the key points of the target two-dimensional image and the key points of the predetermined image are aligned in the foregoing description, when pixel segmentation is performed on the target two-dimensional image, semantic segmentation is also performed on the target two-dimensional image, and the obtained 256 pixel blocks are pixel blocks that include local image features of the target part of the target object.

[0067] According to an embodiment of the present disclosure, multiple pixel blocks can be input into a trained visual deformation network according to the input format of the visual deformation network, each pixel block corresponds to a local image feature, and an image feature vector composed of the local image features of the multiple pixel blocks can be obtained.

[0068] According to the embodiments of the present disclosure, in a visual deformation network, the local image features of each pixel block may correspond to a weight parameter. During the training process, the weight parameters of the visual deformation network are continuously learned and updated. Therefore, for a trained visual deformation network, image features matching the reconstruction parameters may be extracted from the target two-dimensional image, thereby achieving the technical effect of improving the reconstruction accuracy.

[0069] According to the embodiments of the present disclosure, by performing pixel segmentation on the target two-dimensional image, the visual deformation network can extract image features that can accurately predict three-dimensional reconstruction parameters based on the local image features of each pixel block, thereby improving the accuracy of three-dimensional reconstruction.

[0070] According to an embodiment of the present disclosure, identifying image features to obtain target parameters for three-dimensional reconstruction of the image to be processed may include the following operations: performing expression parameter identification on the image features to obtain target expression parameters; performing texture parameter identification on the image features to obtain target texture parameters; performing illumination parameter identification on the image features to obtain target illumination parameters; performing acquisition device parameter identification on the image features to obtain target acquisition device parameters; and performing identity parameter identification on the image features to obtain target identity parameters.

[0071] According to an embodiment of the present disclosure, for a 3DMM model, expression parameters can characterize facial deformation of a target object in a two-dimensional image. Texture parameters can characterize facial contours of a target object in a two-dimensional image. Lighting parameters can be used to compensate for the effect of light intensity on texture parameters in a two-dimensional image. Parameters of image acquisition devices can be used to compensate for acquisition errors when different image acquisition devices acquire images of a target object. Identity parameters can characterize the identity of a target object.

[0072] According to the embodiments of the present disclosure, the target parameter can be understood as the weighted value of identity, expression, texture, and illumination in the 3DMM model. Each dimension parameter of the 3DMM model controls the local changes of the face.

[0073] According to the embodiments of the present disclosure, a CNN (Convolutional Neural Network) can be constructed for each parameter to achieve feature recognition through training.

[0074] It should be noted that the local image features of multiple pixel blocks can be first concatenated to obtain image feature vectors, and then the image feature vectors are input into five CNN networks in parallel for feature recognition to obtain target parameters.

[0075] According to the embodiments of the present disclosure, the reconstruction parameters obtained by identifying image features have comprehensively learned the features in the entire image. Compared with the method of estimating reconstruction parameters based only on image features of key points in the related art, the information in the image features is retained complete and comprehensive, thereby improving the accuracy of the reconstruction parameters.

[0076] Figure 4 A schematic diagram of an image processing method according to an embodiment of the present disclosure is schematically shown.

[0077] like Figure 4 As shown, in embodiment 400, the target two-dimensional image 316 is pixel-segmented according to a predetermined pixel size to obtain a segmented image 411. The segmented image 411 includes a plurality of pixel blocks 421, and the plurality of pixel blocks 421 are input into a visual deformation network (Vision Transformer) to output image features 423. The image features are respectively input into an expression parameter recognition network 431, a texture parameter recognition network 432, an identity parameter recognition network 433, an image acquisition device parameter recognition network 434, and an illumination parameter recognition network 435, and expression parameters, texture parameters, identity parameters, image acquisition device parameters, and illumination parameters are output. Based on the expression parameters, texture parameters, identity parameters, image acquisition device parameters, and illumination parameters, a 3D Morphablemodels (3D deformation statistical model) 441 is used to perform three-dimensional reconstruction to obtain a target three-dimensional image 442.

[0078] According to an embodiment of the present disclosure, based on target parameters, three-dimensional reconstruction is performed on the image to be processed to obtain a target three-dimensional image, which may include the following operations: based on target texture parameters, the image to be processed is processed to obtain a target texture image; based on target acquisition device parameters, target expression parameters and target identity parameters, the image to be processed is processed to obtain target three-dimensional point cloud data; and based on target lighting parameters and the target texture image, the target three-dimensional point cloud data is rendered to obtain a target three-dimensional image.

[0079] For example, the 3DMM network can be used to process the image to be processed based on the target texture parameters, and a texture map including the color, brightness and other texture information of the target object in the image to be processed can be obtained. Based on the target expression parameters, target identity parameters and target acquisition device parameters, the three-dimensional point cloud data including the target object in the image to be processed can be obtained, which can also be understood as a 3D facial model of the target object composed of a three-dimensional point cloud.

[0080] According to an embodiment of the present disclosure, a differentiable renderer may be used to render starting from vertex data in target three-dimensional point cloud data based on lighting parameters and a target texture image to obtain a target three-dimensional image.

[0081] It should be noted that the technology of three-dimensional reconstruction based on target parameters using 3DMM network is a relatively mature technology and will not be elaborated here.

[0082] According to the embodiments of the present disclosure, since the target parameters are obtained based on image features extracted by a visual deformation network, which covers relatively complete image information in the target two-dimensional image, the target three-dimensional image can be reconstructed relatively accurately.

[0083] Figure 5 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown.

[0084] like Figure 5 As shown, the training method 500 may include using the initial model to perform the following operations S510 to S570 on the sample two-dimensional image.

[0085] In operation S510, a key point alignment operation is performed on a sample two-dimensional image based on a predetermined image to obtain a target sample two-dimensional image.

[0086] In operation S520, local mask processing is performed on the target sample two-dimensional image to obtain a mask image.

[0087] In operation S530, mask image features of the mask image and sample image features of the target sample two-dimensional image are extracted.

[0088] In operation S540, features of the mask image are identified to obtain target sample parameters for three-dimensionally reconstructing the sample two-dimensional image.

[0089] In operation S550, the target sample two-dimensional image is three-dimensionally reconstructed based on the target sample parameters to obtain a sample three-dimensional image corresponding to the sample two-dimensional image.

[0090] In operation S560, based on the target loss function, a loss value is obtained according to the mask image feature, the sample image feature, the sample two-dimensional image, and the sample three-dimensional image.

[0091] In operation S570, based on the loss value, the model parameters of the initial model are adjusted to obtain a trained deep learning model.

[0092] According to an embodiment of the present disclosure, a key point alignment operation is performed on a sample two-dimensional image based on a predetermined image to obtain a target sample two-dimensional image. This is the same as the process of performing a key point alignment operation on an image to be processed based on a predetermined image in the image processing method described above, and will not be repeated here.

[0093] According to the embodiments of the present disclosure, 50% of the target sample two-dimensional image may be randomly masked locally to obtain a masked image. For example, a special tag (token) is assigned to the masked area, and the tag of the masked area is a learnable parameter in the model training process.

[0094] According to an embodiment of the present disclosure, the initial model may include a feature extraction module, a feature recognition module and a feature reconstruction module. The feature extraction module may be a visual deformation network, the feature recognition module may include 5 parallel CNN networks, and the feature reconstruction module may be a 3DMM network. The output end of the feature extraction module is connected in parallel to the input end of the 5 CNN networks to input the extracted mask image features into the 5 CNN networks in parallel, and output 5 target sample parameters (expression parameters, texture parameters, identity parameters, image acquisition device parameters and illumination parameters). The 5 target sample parameters are input into the 3DMM network, and the target sample 2D image is reconstructed in three dimensions to obtain a sample 3D image.

[0095] According to an embodiment of the present disclosure, the target loss function may include any one of the following: a cross entropy loss function, an L1 loss function.

[0096] According to the embodiments of the present disclosure, the loss function based on the mask image features and the sample image features can be understood as a mask restoration loss function. During the training process, by continuously adjusting the model parameters of the visual deformation network, when the visual deformation network is used to extract image features, the local image features at the mask position are more similar to the sample image features at the same position corresponding to the mask position, and the local image features at the mask position are less similar to the sample image features at other positions that do not correspond to the mask position, so that the visual deformation network can extract more complete image information.

[0097] According to an embodiment of the present disclosure, the loss function based on the sample two-dimensional image and the sample three-dimensional image can be understood as a three-dimensional reconstruction loss function. The three-dimensional reconstruction loss function can adopt an L1 loss function.

[0098] According to an embodiment of the present disclosure, during the training process, the model parameters of the visual deformation network can be adjusted based on the loss value of the mask restoration loss function. When the loss value of the mask restoration loss function reaches the convergence condition, the model parameters of the visual deformation network can be fixed. Then, based on the three-dimensional reconstruction loss function, the model parameters of other modules in the initial model are adjusted. The convergence condition can be set according to the needs of the actual application scenario, for example: the loss value of the mask restoration loss function is less than a predetermined loss threshold, or it can be the maximum number of iterative training times.

[0099] According to the embodiments of the present disclosure, by performing a mask operation on the target sample two-dimensional image input into the visual deformation network and comparing and learning the sample image features with the mask image features, the deep learning model can learn more comprehensive image features during the training process, ensure the information integrity of the image features, and identify the mask image features to obtain three-dimensional reconstruction parameters, so that the mask image features extracted by the deep learning model match the three-dimensional reconstruction parameters, thereby improving the accuracy of the three-dimensional reconstruction parameters and the robustness and comprehensiveness of the model.

[0100] In the related art, unsupervised training methods are generally used to train ResNet (residual network) for 3D reconstruction of images. However, the 3D reconstruction parameters involve many dimensions, and the slight difference in the parameters of each dimension will cause a large difference in the accuracy of the 3D reconstructed image. Therefore, the model obtained by ResNet network training has low reconstruction accuracy.

[0101] According to an embodiment of the present disclosure, performing local mask processing on a target sample two-dimensional image to obtain a mask image may include the following operations: segmenting the target two-dimensional image according to a predetermined pixel size to obtain a plurality of sample pixel blocks; and randomly masking the plurality of sample pixel blocks according to a predetermined mask ratio to obtain a mask image.

[0102] According to an embodiment of the present disclosure, segmenting the target sample two-dimensional image according to a predetermined pixel size is the same as the operation of segmenting the target two-dimensional image according to a predetermined pixel size described above, and is not described again herein.

[0103] According to the embodiments of the present disclosure, the predetermined mask ratio can be set according to the needs of the actual application scenario. For example, the pixel size of the target two-dimensional image can be 256x256 pixels, and the predetermined pixel size can be 16x16 pixels. Then, the target sample two-dimensional image can be segmented according to the predetermined pixel size to obtain 256 pixel blocks. The predetermined mask ratio can be 50%, and 50% of the pixel blocks can be randomly selected from the 256 pixel blocks for masking. The obtained mask image can include 128 original pixel blocks and 128 masked pixel blocks.

[0104] According to an embodiment of the present disclosure, a visual deformation network (VisionTransformer) is introduced in the embodiment of the present disclosure, and mask features with high similarity to sample image features are obtained based on comparative learning between mask features and sample image features at corresponding positions, and feature recognition is performed based on the mask features with high similarity to obtain target parameters, so that the visual deformation network can extract image features that match the target parameters, thereby improving the reconstruction accuracy of the image.

[0105] According to an embodiment of the present disclosure, based on a predetermined image, a key point alignment operation is performed on a sample two-dimensional image to obtain a target two-dimensional image, which may include the following operations: performing key point detection on the sample two-dimensional image to obtain a first sample key point group; performing key point detection on the sample predetermined image to obtain a second sample key point group; and based on the correspondence between the key points in the first sample key point group and the second sample key point group, performing an alignment operation on the sample two-dimensional image to obtain a target sample two-dimensional image.

[0106] According to an embodiment of the present disclosure, based on a predetermined image, a key point alignment operation is performed on a sample two-dimensional image to obtain a target sample two-dimensional image, which is the same as the operation of performing a key point alignment operation on the image to be processed in the image processing method described above to obtain a target two-dimensional image. The sample two-dimensional image and the target sample two-dimensional image have the same definition range as the image to be processed and the target two-dimensional image, respectively, and are not described in detail here.

[0107] For example: based on the correspondence between the key points in the first sample key point group and the second sample key point group, sample position offset information between the key points with corresponding relationship is obtained from the first sample key point group and the second sample key point group; and according to the sample position offset information, the sample two-dimensional image is scaled to obtain the target sample two-dimensional image.

[0108] For example, the first sample key point group may be the key points of the nose, including the key point P2 at the nose tip, and the key points P1 and P3 on both sides of the nose. The second sample key point group may include the key point m1 at the nose tip, and the key points m1 and m3 on both sides of the nose at the corresponding positions.

[0109] According to the embodiments of the present disclosure, different weights can be configured for key points at different positions. Since the nose shapes of different objects are different, the distances between the key points on both sides of the nose wings are quite different. When performing the key point alignment operation, a higher weight can be configured for the key points at the nose tip; and a lower weight can be configured for the key points on both sides of the nose wings. The position offsets between the key points are weighted and summed based on the weights to obtain the final sample position offset information. Based on the final sample position offset information, the sample two-dimensional image is scaled so that the key points of the target sample two-dimensional image are aligned with the key points of the predetermined image.

[0110] According to an embodiment of the present disclosure, based on an alignment operation between a predetermined image and a sample two-dimensional image, after the sample two-dimensional image is segmented, the position of each pixel block corresponds to the position of a visual deformation network parameter. For the mask image and the sample image, comparative learning can be performed according to corresponding positions when the features are aligned.

[0111] According to an embodiment of the present disclosure, extracting mask image features of a mask image and sample image features of a target sample two-dimensional image may include the following operations: based on a self-attention mechanism, processing multiple sample pixel blocks before masking to obtain sample image features; and based on a self-attention mechanism, processing multiple sample pixel blocks after masking to obtain mask image features.

[0112] For example, multiple sample pixel blocks before the mask can be input into the visual deformation network, and the sample image features (f1, f2, ..., f n ). Input the masked sample pixel blocks into the visual deformation network and output the mask image features (F1, F2, ..., F n ). The feature of each pixel block in the sample image feature corresponds to the feature of each pixel block in the mask image feature, that is, the sample image feature f1 corresponds to the mask image feature F1.

[0113] According to an embodiment of the present disclosure, based on the target loss function, a loss value is obtained according to mask image features, sample image features, sample two-dimensional images and sample three-dimensional images, which may include the following operations: based on a first loss function, a mask feature loss value is obtained according to mask image features and sample image features; and based on a second loss function, a reconstruction loss value is obtained according to the sample two-dimensional image and the sample three-dimensional image.

[0114] For example, based on the similarity algorithm, the similarity value between the mask image feature and the sample image feature corresponding to each pixel block position can be obtained. For example, for the pixel block at the first position, the sample image feature f1 corresponds to the mask image feature F1, and the first loss function can be constructed based on the similarity between the mask image feature and the sample image feature as follows:

[0115]

[0116] Among them, sg[*] represents the iteration termination condition, S cos Indicates similarity, Represents the mask image feature of the i-th position on the mask image; Represents the sample image features at the i-th position on the target sample two-dimensional image.

[0117] According to an embodiment of the present disclosure, obtaining a reconstruction loss value based on a sample two-dimensional image and a sample three-dimensional image based on a second loss function may include the following operations: projecting the sample three-dimensional image to obtain a two-dimensional projection image corresponding to the sample three-dimensional image; and obtaining a reconstruction loss value based on the sample two-dimensional image and the two-dimensional projection image based on the second loss function.

[0118] According to an embodiment of the present disclosure, the second loss function may be an L1 loss function.

[0119] For example, a differentiable renderer can be used to project a sample 3D image in a 2D space to obtain a 2D projection image corresponding to the sample 3D image, and based on the L1 loss function, the reconstruction loss value of the 2D projection image and the sample 2D image is calculated.

[0120] According to the embodiments of the present disclosure, based on the first loss function, the mask image features output by the visual deformation network can be constrained so that the mask image features have a higher similarity with the sample image features at the corresponding position, so that the visual deformation network can extract image feature information as a whole. Based on the second loss function, the reconstruction data of the 3D deformation statistics network during three-dimensional reconstruction can be constrained so that the difference between the three-dimensional reconstruction data and the texture map is smaller, thereby improving the accuracy of the deep learning model.

[0121] When using a 3D deformable statistics network (3DMM), in order to further improve the accuracy of 3D reconstruction data, key point loss calculation can be performed on the 3D point cloud data before combining the 3D point cloud data with the texture map, thereby reducing the difference between the 3D point cloud data and the texture map.

[0122] According to an embodiment of the present disclosure, the above-mentioned training method may also include the following operations: performing key point detection on the sample two-dimensional image to obtain a first key point feature; based on the target sample parameters, processing the sample two-dimensional image to obtain three-dimensional point cloud data corresponding to the sample two-dimensional image; extracting a second key point feature from the three-dimensional point cloud data; and based on a third loss function, obtaining a key point feature loss value according to the first key point feature and the second key point feature.

[0123] According to an embodiment of the present disclosure, the third loss function may be a similarity loss function based on the first key point feature and the second key point feature.

[0124] According to an embodiment of the present disclosure, the first key point feature may include the key points described above when performing key point alignment on the sample two-dimensional image, such as: key points at the tip of the nose, key points on both sides of the nose. In order to improve the accuracy of key point reconstruction in three-dimensional point cloud data, the number of key points and feature dimensions in the first key point feature are greater than the number of key points and feature dimensions used when performing the key point alignment operation. For example: the first key point feature may include features of multiple key points of multiple parts such as eyes, nose, mouth, eyebrows, ears, etc.

[0125] According to an embodiment of the present disclosure, when calculating the third loss function, the more key points there are in the key point features and the more feature dimensions there are of the key points, the higher the three-dimensional reconstruction accuracy of the trained model.

[0126] According to an embodiment of the present disclosure, before the three-dimensional point cloud data is combined with the texture map, key point loss calculation is performed on the three-dimensional point cloud data, thereby reducing the difference between the three-dimensional point cloud data and the texture map and improving the reconstruction accuracy of the three-dimensional point cloud data.

[0127] In the related art, when training a deep learning model used for three-dimensional reconstruction of images, loss calculation is generally performed based on the output results of the final model and the sample labels, and the model parameters of the overall model are adjusted. This training method has a long training cycle and low training efficiency.

[0128] Therefore, in the embodiments of the present disclosure, the deep model can be trained using gradient training.

[0129] According to an embodiment of the present disclosure, adjusting the model parameters of the initial model based on the loss value to obtain a trained deep learning model may include the following operations: adjusting the parameters of the feature extraction module based on the mask feature loss value to obtain a first intermediate model; using the first intermediate model to perform the same operation on the sample two-dimensional image as the initial model performs on the sample two-dimensional image, to obtain the key point feature loss value of the first intermediate model; based on the key point feature loss value, adjusting the parameters of the feature recognition module of the first intermediate model to obtain a second intermediate model; using the second intermediate model to perform the same operation on the sample two-dimensional image as the initial model performs on the sample two-dimensional image, to obtain the reconstruction loss value of the second intermediate model; and adjusting the parameters of the feature reconstruction module of the second intermediate model based on the reconstruction loss value to obtain a trained deep learning model.

[0130] For example: First, during the first round of training, the parameters of the feature extraction module can be adjusted based only on the mask feature loss value, and the first intermediate model is obtained when the mask feature loss value converges to a predetermined threshold range.

[0131] Secondly, the sample two-dimensional image is input into the first intermediate model, and mask image features, sample image features, three-dimensional point cloud data, key point data of the sample two-dimensional image, and output sample three-dimensional image can also be obtained. In this round of training, the parameters of the feature recognition module of the first intermediate model can be adjusted mainly based on the key point feature loss value to improve the matching degree between the image features and the three-dimensional reconstruction parameters. When the key point feature loss value converges to a predetermined threshold range, the second intermediate model is obtained.

[0132] It should be noted that, in the process of adjusting the model parameters of the first intermediate model, the parameters of the feature extraction module may be fixed or fine-tuned to accelerate the convergence speed of the key point loss value.

[0133] Again, the sample two-dimensional image is input into the second intermediate model. After the previous training process, the feature extraction module and feature recognition module of the second intermediate model can already output image features that match the three-dimensional reconstruction parameters, and can accurately identify the three-dimensional reconstruction parameters. In this round of training, the feature reconstruction module can be adjusted mainly based on the reconstruction loss value. In the process of adjusting the parameters, weakly supervised training can also be performed with the mask feature loss value and the key point feature loss value at the same time, so that the final reconstruction loss value converges to a predetermined threshold range, and a trained deep learning model is obtained.

[0134] According to an embodiment of the present disclosure, based on the gradient training method, the parameters of a certain module are adjusted in a targeted manner so that the loss value corresponding to the module converges. When the next stage of training is carried out, the influence of the error of the output data of the module on the input data of the next module can be reduced, thereby improving the convergence speed of the model.

[0135] Figure 6 A schematic diagram of a framework of a deep learning model training method according to an embodiment of the present disclosure is schematically shown.

[0136] like Figure 6 As shown, pixel segmentation is performed on the target sample two-dimensional image 601 to obtain an unmasked image 602, which may include multiple unmasked pixel blocks 603. Pixel segmentation and masking are performed on the target sample two-dimensional image 601 to obtain a masked image 603, which may include multiple partially masked pixel blocks 607.

[0137] Multiple unmasked pixel blocks 603 are input into a visual deformation network (ViT) 604a to obtain sample image features 605. Multiple partially masked pixel blocks 607 are input into a visual deformation network (ViT) 604b to obtain mask image features 606. The parameters of the visual deformation network (ViT) 604b are adjusted based on the mask feature loss (Contrastive Loss) 614, so that the mask image features 606 have a higher similarity with the sample image features 605 at the corresponding position.

[0138] The mask image features 606 are identified to obtain the image acquisition device parameters (R, t), identity parameters (α id ), expression parameters (α exp ), texture parameters (α tex ) and illumination parameters (γ). And the image acquisition device parameters (R, t), identity parameters (α id ), expression parameters (α exp ), texture parameters (α tex ) and illumination parameters (γ) are input into 3DMM (3D deformation statistics network) 608 to obtain 3D point cloud data 609 and texture map 610. Differentiable renderer 611 is used to render the reconstructed image 612 based on the illumination parameters to obtain a 3D reconstructed image 613.

[0139] Photometric loss 615 can be calculated based on the 3D reconstructed image 613 and the initial sample 2D image 601 as the reconstruction loss value. Landmark loss 616 is calculated based on the point cloud features of the reconstructed image 612 and the key point features of the initial sample 2D image. The sum of the mask feature loss 614, the photometric loss 615 and the key point loss 616 is used as the target loss to adjust the parameters of the initial model and obtain a trained deep learning model.

[0140] Figure 7 The block diagram schematically shows an image processing apparatus according to an embodiment of the present disclosure.

[0141] like Figure 7 As shown, the image processing device 700 may include a first alignment module 710 , a first feature extraction module 720 , a first feature recognition module 730 and a first feature reconstruction module 740 .

[0142] The first alignment module 710 is used to perform a key point alignment operation on the image to be processed based on a predetermined image to obtain a target two-dimensional image.

[0143] The first feature extraction module 720 is used to extract image features of the target two-dimensional image, wherein the image features represent target features that match reconstruction parameters used for three-dimensional reconstruction of the image to be processed.

[0144] The first feature recognition module 730 is used to recognize image features and obtain target parameters for three-dimensional reconstruction of the image to be processed.

[0145] The first feature reconstruction module 740 is used to perform three-dimensional reconstruction on the target two-dimensional image based on the target parameters to obtain a target three-dimensional image.

[0146] According to an embodiment of the present disclosure, the first alignment module includes: a first detection submodule, a second detection submodule and a first alignment submodule. The first detection submodule is used to perform key point detection on the image to be processed to obtain a first key point group. The second detection submodule is used to perform key point detection on the predetermined image to obtain a second key point group. The first alignment submodule is used to perform an alignment operation on the image to be processed based on the correspondence between the key points in the first key point group and the second key point group to obtain a target two-dimensional image.

[0147] According to an embodiment of the present disclosure, the first alignment submodule includes: a first position offset unit and a first scaling unit. The first position offset unit is used to obtain position offset information between key points having a corresponding relationship from the first key point group and the second key point group based on the corresponding relationship between key points in the first key point group and the second key point group. The first scaling unit is used to scale the image to be processed according to the position offset information to obtain a target two-dimensional image.

[0148] According to an embodiment of the present disclosure, the first feature extraction module includes: a first segmentation submodule and a first attention submodule. The first segmentation submodule is used to segment the target two-dimensional image according to a predetermined pixel size to obtain a plurality of pixel blocks. The first attention submodule is used to process the plurality of pixel blocks based on a self-attention mechanism to obtain image features.

[0149] According to an embodiment of the present disclosure, the first feature recognition module includes: an expression parameter recognition submodule, a texture parameter recognition submodule, an illumination parameter recognition submodule, an acquisition device parameter recognition submodule and an identity parameter recognition submodule. The expression parameter recognition submodule is used to perform expression parameter recognition on image features to obtain target expression parameters. The texture parameter recognition submodule is used to perform texture parameter recognition on image features to obtain target texture parameters. The illumination parameter recognition submodule is used to perform illumination parameter recognition on image features to obtain target illumination parameters. The acquisition device parameter recognition submodule is used to perform acquisition device parameter recognition on image features to obtain target acquisition device parameters. The identity parameter recognition submodule is used to perform identity parameter recognition on image features to obtain target identity parameters.

[0150] According to an embodiment of the present disclosure, the first feature reconstruction module includes: a texture image generation submodule, a point cloud data generation submodule and a rendering submodule. The texture image generation submodule is used to process the image to be processed based on the target texture parameters to obtain the target texture image. The point cloud data generation submodule is used to process the image to be processed based on the target acquisition device parameters, the target expression parameters and the target identity parameters to obtain the target three-dimensional point cloud data. The rendering submodule is used to render the target three-dimensional point cloud data based on the target illumination parameters and the target texture image to obtain the target three-dimensional image.

[0151] Figure 8 A block diagram of a deep learning model training device according to an embodiment of the present disclosure is schematically shown.

[0152] like Figure 8 As shown, the training device 800 may include: a second alignment module 810, a mask module 820, a second feature extraction module 830, a second feature recognition module 840, a second feature reconstruction module 850, a loss calculation module 860 and an adjustment module 870.

[0153] The second alignment module 810 is used to perform a key point alignment operation on the sample two-dimensional image based on a predetermined image to obtain a target two-dimensional image.

[0154] The mask module 820 is used to perform local mask processing on the target sample two-dimensional image to obtain a mask image.

[0155] The second feature extraction module 830 is used to extract mask image features of the mask image and sample image features of the target sample two-dimensional image.

[0156] The second feature recognition module 840 is used to recognize features of the mask image to obtain target sample parameters for three-dimensional reconstruction of the sample two-dimensional image.

[0157] The second feature reconstruction module 850 is used to perform three-dimensional reconstruction on the target sample two-dimensional image based on the target sample parameters to obtain a sample three-dimensional image corresponding to the sample two-dimensional image.

[0158] The loss calculation module 860 is used to obtain a loss value based on a target loss function, according to mask image features, sample image features, sample two-dimensional images and sample three-dimensional images.

[0159] The adjustment module 870 is used to adjust the model parameters of the initial model based on the loss value to obtain a trained deep learning model.

[0160] According to an embodiment of the present disclosure, the loss calculation module includes: a mask feature loss calculation submodule and a reconstruction loss calculation submodule. The mask feature loss calculation submodule is used to obtain a mask feature loss value based on a first loss function, according to mask image features and sample image features. The reconstruction loss calculation submodule is used to obtain a reconstruction loss value based on a sample two-dimensional image and a sample three-dimensional image based on a second loss function.

[0161] According to an embodiment of the present disclosure, the reconstruction loss calculation submodule includes: a projection unit and a reconstruction loss calculation unit. The projection unit is used to project the sample three-dimensional image to obtain a two-dimensional projection image corresponding to the sample three-dimensional image. The reconstruction loss calculation unit is used to obtain a reconstruction loss value based on the sample two-dimensional image and the two-dimensional projection image based on a third loss function.

[0162] According to an embodiment of the present disclosure, the loss calculation module also includes: a key point detection submodule, a three-dimensional point cloud data generation submodule, a key point feature extraction submodule and a key point feature loss calculation module. The key point detection submodule is used to perform key point detection on the sample two-dimensional image to obtain the first key point feature. The three-dimensional point cloud data generation submodule is used to process the sample two-dimensional image based on the target sample parameters to obtain three-dimensional point cloud data corresponding to the sample two-dimensional image. The key point feature extraction submodule is used to extract the second key point feature from the three-dimensional point cloud data. The key point feature loss calculation module is used to obtain the key point feature loss value according to the first key point feature and the second key point feature based on the third loss function.

[0163] According to an embodiment of the present disclosure, the adjustment module includes: a first parameter adjustment submodule, a first training submodule, a second parameter adjustment submodule, a second training submodule and a third parameter adjustment submodule. The first parameter adjustment submodule is used to adjust the parameters of the feature extraction module based on the mask feature loss value to obtain the first intermediate model. The first training submodule is used to use the first intermediate model to perform the same operation on the sample two-dimensional image as the initial model performs on the sample two-dimensional image, and obtain the key point feature loss value of the first intermediate model. The second parameter adjustment submodule is used to adjust the parameters of the feature recognition module of the first intermediate model based on the key point feature loss value to obtain the second intermediate model. The second training submodule is used to use the second intermediate model to perform the same operation on the sample two-dimensional image as the initial model performs on the sample two-dimensional image, and obtain the reconstruction loss value of the second intermediate model. The third parameter adjustment submodule is used to adjust the parameters of the feature reconstruction module of the second intermediate model based on the reconstruction loss value to obtain a trained deep learning model.

[0164] According to an embodiment of the present disclosure, the mask module includes: a second segmentation submodule and a mask submodule. The second segmentation submodule is used to segment the target two-dimensional image according to a predetermined pixel size to obtain a plurality of sample pixel blocks. The mask submodule is used to randomly mask the plurality of sample pixel blocks according to a predetermined mask ratio to obtain a mask image.

[0165] According to an embodiment of the present disclosure, the second alignment module includes: a third detection submodule, a fourth detection submodule and a second alignment submodule. The third detection submodule is used to perform key point detection on the sample two-dimensional image to obtain a first sample key point group. The fourth detection submodule is used to perform key point detection on the sample predetermined image to obtain a second sample key point group. The second alignment submodule is used to perform an alignment operation on the sample two-dimensional image based on the correspondence between the key points in the first sample key point group and the second sample key point group to obtain a target sample two-dimensional image.

[0166] According to an embodiment of the present disclosure, the second alignment submodule includes: a second position offset unit and a second scaling unit. The second position offset unit is used to obtain sample position offset information between key points having a corresponding relationship from the first sample key point group and the second sample key point group based on the corresponding relationship between key points in the first sample key point group and the second sample key point group. The second scaling unit is used to scale the sample two-dimensional image according to the sample position offset information to obtain a target sample two-dimensional image.

[0167] According to an embodiment of the present disclosure, the second feature extraction module includes: a second attention submodule and a third attention submodule. The second attention submodule is used to process multiple sample pixel blocks before masking based on a self-attention mechanism to obtain sample image features. The third attention submodule is used to process multiple sample pixel blocks after masking based on a self-attention mechanism to obtain mask image features.

[0168] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0169] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0170] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0171] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.

[0172] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0173] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0174] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0175] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as image processing methods or training methods for deep learning models. For example, in some embodiments, the image processing method or the training method for a deep learning model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the image processing method or the training method for a deep learning model described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute an image processing method or a deep learning model training method in any other appropriate manner (e.g., by means of firmware).

[0176] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0177] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0178] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0179] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0180] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0181] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0182] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0183] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: Based on the predetermined image, a key point alignment operation is performed on the image to be processed to obtain a target two-dimensional image; Extracting image features of the target two-dimensional image using a visual deformation network, wherein the image features represent target features that match reconstruction parameters for three-dimensionally reconstructing the image to be processed; Identifying the image features to obtain target parameters for three-dimensional reconstruction of the image to be processed; and Using a 3D deformation statistics network to perform three-dimensional reconstruction on the target two-dimensional image based on the target parameters to obtain a target three-dimensional image; Among them, the visual deformation network is obtained by comparing and learning the features of the mask image with the features of the sample image at the corresponding position, so as to obtain the mask features with high similarity to the sample image features, and feature recognition is performed based on the mask features with high similarity to obtain the target sample parameters; the mask image is obtained by locally masking the two-dimensional image of the target sample; the mask features output by the visual deformation network are constrained based on the first loss function to improve the similarity between the mask features and the features of the sample image at the corresponding position; the reconstructed data of the 3D deformation statistics network during three-dimensional reconstruction is constrained based on the second loss function to reduce the difference between the three-dimensional reconstruction data and the texture map.

2. The method according to claim 1, wherein: The method of performing a key point alignment operation on the image to be processed based on the predetermined image to obtain a target two-dimensional image includes: Performing key point detection on the image to be processed to obtain a first key point group; Performing key point detection on the predetermined image to obtain a second key point group; and Based on the correspondence between the key points in the first key point group and the second key point group, an alignment operation is performed on the image to be processed to obtain a target two-dimensional image.

3. The method according to claim 2, wherein: The step of performing an alignment operation on the image to be processed based on the correspondence between the key points in the first key point group and the second key point group to obtain a target two-dimensional image includes: Based on the correspondence between the key points in the first key point group and the second key point group, obtaining position offset information between the key points having the corresponding relationship from the first key point group and the second key point group; and The image to be processed is scaled according to the position offset information to obtain the target two-dimensional image.

4. The method according to claim 1, wherein: The step of extracting the image features of the target two-dimensional image comprises: Segmenting the target two-dimensional image according to a predetermined pixel size to obtain a plurality of pixel blocks; and Based on the self-attention mechanism, the multiple pixel blocks are processed to obtain the image features.

5. The method according to claim 1, wherein: The identifying of the image features to obtain target parameters for three-dimensionally reconstructing the image to be processed includes: Performing expression parameter recognition on the image features to obtain target expression parameters; Performing texture parameter recognition on the image features to obtain target texture parameters; Performing illumination parameter recognition on the image features to obtain target illumination parameters; Performing acquisition device parameter identification on the image features to obtain target acquisition device parameters; and Perform identity parameter recognition on the image features to obtain target identity parameters.

6. The method according to claim 1, wherein: The target parameters include target expression parameters, target texture parameters, target illumination parameters, target acquisition device parameters and target identity parameters; The three-dimensional reconstruction of the image to be processed based on the target parameter to obtain a target three-dimensional image includes: Based on the target texture parameters, the image to be processed is processed to obtain a target texture image; Based on the target acquisition device parameters, the target expression parameters and the target identity parameters, the image to be processed is processed to obtain target three-dimensional point cloud data; and The target three-dimensional point cloud data is rendered based on the target illumination parameters and the target texture image to obtain the target three-dimensional image.

7. A method for training a deep learning model, comprising: Based on the predetermined image, a key point alignment operation is performed on the sample two-dimensional image to obtain a target sample two-dimensional image; Performing local mask processing on the target sample two-dimensional image to obtain a mask image; Extracting mask image features of the mask image and sample image features of the target sample two-dimensional image using a visual deformation network; Identifying the features of the mask image to obtain target sample parameters for three-dimensional reconstruction of the sample two-dimensional image; Using a 3D deformation statistics network, based on the target sample parameters, the target sample two-dimensional image is three-dimensionally reconstructed to obtain a sample three-dimensional image corresponding to the sample two-dimensional image; Based on the target loss function, a loss value is obtained according to the mask image feature, the sample image feature, the sample two-dimensional image and the sample three-dimensional image; as well as Based on the loss value, adjusting the model parameters of the initial model to obtain a trained deep learning model; the initial model includes a feature extraction module, a feature recognition module and a feature reconstruction module; The feature extraction module is the visual deformation network; The feature reconstruction module is the 3D deformation statistics network; Among them, the target loss function includes: a first loss function and a second loss function, the first loss function is used to constrain the mask image features output by the visual deformation network to improve the similarity between the mask image features and the sample image features at the corresponding position; the second loss function is used to constrain the reconstruction data of the 3D deformation statistics network when performing three-dimensional reconstruction to reduce the difference between the three-dimensional reconstruction data and the texture map.

8. The method according to claim 7, wherein: The method of obtaining a loss value based on the target loss function according to the mask image feature, the sample image feature, the sample two-dimensional image and the sample three-dimensional image includes: Based on the first loss function, obtaining a mask feature loss value according to the mask image feature and the sample image feature; and Based on the second loss function, a reconstruction loss value is obtained according to the sample two-dimensional image and the sample three-dimensional image.

9. The method according to claim 8, wherein: The step of obtaining a reconstruction loss value based on the sample two-dimensional image and the sample three-dimensional image based on the second loss function includes: Projecting the sample three-dimensional image to obtain a two-dimensional projection image corresponding to the sample three-dimensional image; and Based on the second loss function, the reconstruction loss value is obtained according to the sample two-dimensional image and the two-dimensional projection image.

10. The method according to claim 8 or 9, further comprising: Performing key point detection on the sample two-dimensional image to obtain a first key point feature; Based on the target sample parameters, the sample two-dimensional image is processed to obtain three-dimensional point cloud data corresponding to the sample two-dimensional image; Extracting a second key point feature from the three-dimensional point cloud data; as well as Based on the third loss function, a key point feature loss value is obtained according to the first key point feature and the second key point feature.

11. The method according to claim 7, wherein: The loss values ​​include mask feature loss values, reconstruction loss values ​​and key point feature loss values; The adjusting the model parameters of the initial model based on the loss value to obtain a trained deep learning model includes: Based on the mask feature loss value, adjusting the parameters of the feature extraction module to obtain a first intermediate model; Using the first intermediate model, perform the same operation on the sample two-dimensional image as that performed by the initial model on the sample two-dimensional image, to obtain a key point feature loss value of the first intermediate model; Based on the key point feature loss value, adjusting the parameters of the feature recognition module of the first intermediate model to obtain a second intermediate model; Using the second intermediate model, perform the same operation on the sample two-dimensional image as that performed by the initial model on the sample two-dimensional image, to obtain a reconstruction loss value of the second intermediate model; and Based on the reconstruction loss value, the parameters of the feature reconstruction module of the second intermediate model are adjusted to obtain the trained deep learning model.

12. The method according to claim 7, wherein: The performing local mask processing on the target sample two-dimensional image to obtain a mask image includes: Segmenting the target sample two-dimensional image according to a predetermined pixel size to obtain a plurality of sample pixel blocks; and The plurality of sample pixel blocks are randomly masked according to a predetermined mask ratio to obtain the mask image.

13. The method according to claim 7, wherein: The method of performing a key point alignment operation on the sample two-dimensional image based on the predetermined image to obtain a target sample two-dimensional image includes: Performing key point detection on the sample two-dimensional image to obtain a first sample key point group; Performing key point detection on the predetermined image to obtain a second sample key point group; and Based on the correspondence between the key points in the first sample key point group and the second sample key point group, an alignment operation is performed on the sample two-dimensional image to obtain a target sample two-dimensional image.

14. The method according to claim 13, wherein: The step of performing an alignment operation on the sample two-dimensional image based on the correspondence between the key points in the first sample key point group and the second sample key point group to obtain a target sample two-dimensional image includes: Based on the correspondence between the key points in the first sample key point group and the second sample key point group, obtaining sample position offset information between the key points having the corresponding relationship from the first sample key point group and the second sample key point group; and The sample two-dimensional image is scaled according to the sample position offset information to obtain the target sample two-dimensional image.

15. The method according to claim 7, wherein: The extracting of the mask image features of the mask image and the sample image features of the target sample two-dimensional image includes: Based on the self-attention mechanism, multiple sample pixel blocks before the mask are processed to obtain the sample image features; and Based on the self-attention mechanism, the masked multiple sample pixel blocks are processed to obtain the mask image features.

16. An image processing device, comprising: A first alignment module is used to perform a key point alignment operation on the image to be processed based on a predetermined image to obtain a target two-dimensional image; A first feature extraction module, configured to extract image features of the target two-dimensional image using a visual deformation network, wherein the image features represent target features that match reconstruction parameters used for three-dimensionally reconstructing the image to be processed; A first feature recognition module is used to recognize the image features and obtain target parameters for three-dimensional reconstruction of the image to be processed; and A first feature reconstruction module is used to perform three-dimensional reconstruction on the target two-dimensional image based on the target parameters by using a 3D deformation statistics network to obtain a target three-dimensional image; Among them, the visual deformation network is obtained by comparing and learning the features of the mask image with the features of the sample image at the corresponding position, so as to obtain the mask features with high similarity to the sample image features, and feature recognition is performed based on the mask features with high similarity to obtain the target sample parameters; the mask image is obtained by locally masking the two-dimensional image of the target sample; the mask features output by the visual deformation network are constrained based on the first loss function to improve the similarity between the mask features and the features of the sample image at the corresponding position; the reconstructed data of the 3D deformation statistics network during three-dimensional reconstruction is constrained based on the second loss function to reduce the difference between the three-dimensional reconstruction data and the texture map.

17. The device according to claim 16, wherein: The first alignment module comprises: A first detection submodule, configured to perform key point detection on the image to be processed to obtain a first key point group; A second detection submodule, configured to perform key point detection on the predetermined image to obtain a second key point group; and The first alignment submodule is used to perform an alignment operation on the image to be processed based on the correspondence between the key points in the first key point group and the second key point group to obtain a target two-dimensional image.

18. The device according to claim 17, wherein: The first alignment submodule includes: A first position offset unit, configured to obtain position offset information between key points having a corresponding relationship from the first key point group and the second key point group based on the corresponding relationship between key points in the first key point group and the second key point group; and The first scaling unit is used to scale the image to be processed according to the position offset information to obtain the target two-dimensional image.

19. The device according to claim 16, wherein: The first feature extraction module comprises: A first segmentation submodule, configured to segment the target two-dimensional image according to a predetermined pixel size to obtain a plurality of pixel blocks; and The first attention submodule is used to process the multiple pixel blocks based on a self-attention mechanism to obtain the image features.

20. The device according to claim 16, wherein: The first feature recognition module includes: An expression parameter recognition submodule is used to perform expression parameter recognition on the image features to obtain target expression parameters; A texture parameter recognition submodule is used to perform texture parameter recognition on the image features to obtain target texture parameters; An illumination parameter recognition submodule is used to perform illumination parameter recognition on the image features to obtain target illumination parameters; An acquisition device parameter identification submodule, used to identify acquisition device parameters for the image features to obtain target acquisition device parameters; and The identity parameter recognition submodule is used to perform identity parameter recognition on the image features to obtain target identity parameters.

21. The device according to claim 16, wherein: The target parameters include target expression parameters, target texture parameters, target illumination parameters, target acquisition device parameters and target identity parameters; the first feature reconstruction module includes: A texture image generation submodule, used for processing the image to be processed based on the target texture parameters to obtain a target texture image; a point cloud data generation submodule, configured to process the image to be processed based on the target acquisition device parameters, the target expression parameters and the target identity parameters to obtain target three-dimensional point cloud data; and The rendering submodule is used to render the target three-dimensional point cloud data based on the target illumination parameters and the target texture image to obtain the target three-dimensional image.

22. A training device for a deep learning model, comprising: A second alignment module is used to perform a key point alignment operation on the sample two-dimensional image based on a predetermined image to obtain a target sample two-dimensional image; A mask module, used for performing local mask processing on the target sample two-dimensional image to obtain a mask image; A second feature extraction module, configured to extract mask image features of the mask image and sample image features of the target sample two-dimensional image using a visual deformation network; A second feature recognition module is used to recognize the features of the mask image to obtain target sample parameters for three-dimensional reconstruction of the sample two-dimensional image; A second feature reconstruction module is used to perform three-dimensional reconstruction on the target sample two-dimensional image based on the target sample parameters by using a 3D deformation statistics network to obtain a sample three-dimensional image corresponding to the sample two-dimensional image; A loss calculation module, configured to obtain a loss value based on a target loss function, according to the mask image features, the sample image features, the sample two-dimensional image, and the sample three-dimensional image; as well as An adjustment module, used to adjust the model parameters of the initial model based on the loss value to obtain a trained deep learning model; the initial model includes a feature extraction module, a feature recognition module and a feature reconstruction module; The feature extraction module is the visual deformation network; The feature reconstruction module is the 3D deformation statistics network; Among them, the target loss function includes: a first loss function and a second loss function, the first loss function is used to constrain the mask image features output by the visual deformation network to improve the similarity between the mask image features and the sample image features at the corresponding position; the second loss function is used to constrain the reconstruction data of the 3D deformation statistics network when performing three-dimensional reconstruction to reduce the difference between the three-dimensional reconstruction data and the texture map.

23. The device according to claim 22, wherein: The loss calculation module includes: a mask feature loss calculation submodule, configured to obtain a mask feature loss value based on the mask image feature and the sample image feature based on a first loss function; and The reconstruction loss calculation submodule is used to obtain a reconstruction loss value according to the sample two-dimensional image and the sample three-dimensional image based on a second loss function.

24. The device according to claim 23, wherein: The reconstruction loss calculation submodule includes: a projection unit, configured to project the sample three-dimensional image to obtain a two-dimensional projection image corresponding to the sample three-dimensional image; and A reconstruction loss calculation unit is used to obtain the reconstruction loss value based on the second loss function according to the sample two-dimensional image and the two-dimensional projection image.

25. The device according to claim 23 or 24, wherein: The loss calculation module also includes: A key point detection submodule, used to perform key point detection on the sample two-dimensional image to obtain a first key point feature; A three-dimensional point cloud data generation submodule, used for processing the sample two-dimensional image based on the target sample parameters to obtain three-dimensional point cloud data corresponding to the sample two-dimensional image; A key point feature extraction submodule, used to extract a second key point feature from the three-dimensional point cloud data; and A key point feature loss calculation module is used to obtain a key point feature loss value based on the first key point feature and the second key point feature based on a third loss function.

26. The device according to claim 25, wherein The initial model includes a feature extraction module, a feature recognition module and a feature reconstruction module; the loss value includes a mask feature loss value, a reconstruction loss value and a key point feature loss value; The adjustment module comprises: A first parameter adjustment submodule, used to adjust the parameters of the feature extraction module based on the mask feature loss value to obtain a first intermediate model; A first training submodule is used to use a first intermediate model to perform the same operation on the sample two-dimensional image as that performed by the initial model on the sample two-dimensional image, so as to obtain a key point feature loss value of the first intermediate model; A second parameter adjustment submodule, used to adjust the parameters of the feature recognition module of the first intermediate model based on the key point feature loss value to obtain a second intermediate model; A second training submodule is configured to use the second intermediate model to perform the same operation on the sample two-dimensional image as that performed by the initial model on the sample two-dimensional image, to obtain a reconstruction loss value of the second intermediate model; and The third parameter adjustment submodule is used to adjust the parameters of the feature reconstruction module of the second intermediate model based on the reconstruction loss value to obtain the trained deep learning model.

27. The device according to claim 22, wherein: The mask module comprises: A second segmentation submodule is used to segment the target sample two-dimensional image according to a predetermined pixel size to obtain a plurality of sample pixel blocks; and The mask submodule is used to randomly mask the multiple sample pixel blocks according to a predetermined mask ratio to obtain the mask image.

28. The device according to claim 22, wherein: The second alignment module comprises: A third detection submodule is used to perform key point detection on the sample two-dimensional image to obtain a first sample key point group; a fourth detection submodule, configured to perform key point detection on the predetermined image to obtain a second sample key point group; and The second alignment submodule is used to perform an alignment operation on the sample two-dimensional image based on the correspondence between the key points in the first sample key point group and the second sample key point group to obtain a target sample two-dimensional image.

29. The device according to claim 28, wherein The second alignment submodule comprises: A second position offset unit, configured to obtain sample position offset information between key points having a corresponding relationship from the first sample key point group and the second sample key point group based on the corresponding relationship between key points in the first sample key point group and the second sample key point group; and The second scaling unit is used to scale the sample two-dimensional image according to the sample position offset information to obtain the target sample two-dimensional image.

30. The device according to claim 22, wherein: The second feature extraction module comprises: A second attention submodule is used to process the plurality of sample pixel blocks before the mask based on a self-attention mechanism to obtain the sample image features; and The third attention submodule is used to process the masked multiple sample pixel blocks based on the self-attention mechanism to obtain the mask image features.

31. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 15.

32. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-15.

33. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images