Model training method and device, model using method and device, equipment and storage medium

Abstract

The invention discloses a model training method and device, a model using method and device, equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a first limb image, wherein the first limb image is image information under a first visual angle; a feature coding network is called to carry out image coding on the first limb image, at least two fusion feature representations corresponding to at least two query points on the limb object in a one-to-one mode are extracted, and the nth fusion feature representation is used for indicating feature representations of image areas with symmetry on the physiological structure; calling a decoding network, performing feature decoding on the at least two fusion feature representations, and rendering to obtain a second limb image; the second limb image is image information under the second visual angle. Based on the symmetry of the limb object on the physiological structure, the feature representation on the second limb supplements the feature representation of the query point on the first limb, so that the dimensionality of feature extraction of the query point is expanded, and the effect of the limb image at the second view angle is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, a model use method, an apparatus, a device and a storage medium. Background Art

[0002] In multimedia technology, there is a need to observe complex actions from different angles; taking the hand placement gesture as an example, there is a need to observe the hands from different angles and check the relative position between the hands to facilitate imitation of the hand gesture.

[0003] In the related art, the camera-based image acquisition device can only acquire the observed images of the user's hands from a single direction according to the user's placement position; when there is a need to display the image of the user's hands in another direction, it is necessary to call an artificial neural network for image prediction, and predict and generate the images of the hands by extracting the global features in the image.

[0004] However, the above process cannot fully extract the feature information in the image, and the effect of generating two-handed images is not good. Summary of the invention

[0005] The present application provides a model training method, a model use method, an apparatus, a device and a storage medium, and the technical solution is as follows:

[0006] According to one aspect of the present application, a method for using a limb part image prediction model is provided, wherein the limb part image prediction model includes a feature encoding network and a decoding network; the method includes:

[0007] Acquire a first limb image, where the first limb image is image information of a limb object at a first viewing angle, the limb object includes a first limb and a second limb, and the first limb and the second limb are symmetrical in physiological structure;

[0008] Calling the feature encoding network to perform image encoding on the first limb image, extracting at least two fused feature representations corresponding to at least two query points on the limb object, wherein an nth fused feature representation corresponding to an nth query point among the at least two query points is used to indicate a feature representation of an image region that is symmetrical in physiological structure, and the nth fused feature representation is used to supplement the feature representation of the nth query point on the first limb based on the symmetry of the limb object through the feature representation on the second limb;

[0009] The decoding network is called to perform feature decoding on the at least two fused feature representations, and a second limb image is rendered; the generated area corresponding to the nth query point in the second limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the second limb image is image information of the limb object under a second perspective, and the second perspective is different from the first perspective.

[0010] According to another aspect of the present application, a method for training a limb part image prediction model is provided, wherein the limb part image prediction model includes a feature encoding network and a decoding network; the method includes:

[0011] Acquire a sample information pair, the sample information pair comprising a first sample image and a second sample image, the first sample image being image information of a sample object at a first viewing angle, the second sample image being image information of the sample object at a second viewing angle, the first viewing angle and the second viewing angle being different; the sample object comprising a first limb and a second limb, the first limb and the second limb being symmetrical in physiological structure;

[0012] Calling the feature encoding network to perform image encoding on the first sample image, extracting at least two predicted feature representations corresponding to at least two sample query points on the sample object, wherein an nth predicted feature representation corresponding to an nth sample query point among the at least two sample query points is used to indicate a feature representation of a sample region that is symmetrical in physiological structure, and the nth predicted feature representation is used to supplement the feature representation of the nth sample query point on the first limb through the feature representation on the second limb based on the symmetry of the sample object;

[0013] Calling the decoding network, performing feature decoding on the at least two predicted feature representations, and rendering to obtain a predicted limb image, wherein the generated region corresponding to the nth sample query point in the predicted limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the predicted limb image is the predicted image information of the sample object under the second viewing angle;

[0014] Based on the difference between the predicted limb image and the second sample image, the limb part image prediction model is trained to obtain a trained limb part image prediction model.

[0015] According to another aspect of the present application, a method for training a limb part image prediction model is provided, wherein the limb part image prediction model includes a feature encoding network, a decoding network, and a discriminant network; the method includes:

[0016] Acquire a first sample image, where the first sample image is image information of a sample object at a first viewing angle, where the sample object includes a first limb and a second limb, and the first limb and the second limb are symmetrical in physiological structure;

[0017] Calling the feature encoding network to perform image encoding on the first sample image, extracting at least two predicted feature representations corresponding to at least two sample query points on the sample object, wherein an nth predicted feature representation corresponding to an nth sample query point among the at least two sample query points is used to indicate a feature representation of a sample region that is symmetrical in physiological structure, and the nth predicted feature representation is used to supplement the feature representation of the nth sample query point on the first limb through the feature representation on the second limb based on the symmetry of the sample object;

[0018] Calling the decoding network, performing feature decoding on the at least two predicted feature representations, and rendering to obtain a predicted limb image, wherein the generated region corresponding to the nth sample query point in the predicted limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the predicted limb image is image information of the sample object obtained by prediction at a second viewing angle, and the first viewing angle and the second viewing angle are different;

[0019] Calling the discriminant network to predict and obtain first visibility information based on the predicted limb image, wherein the first visibility information is displayed under the second viewing angle, and is a visual condition of a sample object predicted in the predicted limb image observed under the first viewing angle;

[0020] Acquire second visibility information of the sample object, where the second visibility information is displayed at the second viewing angle, and is a visual condition of the sample object in the first sample image observed at the first viewing angle;

[0021] Based on the difference between the first visibility information and the second visibility information, adversarial training is performed on the discriminant network or the prediction sub-model of the limb part image prediction model to obtain a trained limb part image prediction model, and the prediction sub-model includes the feature encoding network and the decoding network.

[0022] According to another aspect of the present application, a device for using a limb part image prediction model is provided, wherein the limb part image prediction model includes a feature encoding network and a decoding network; the device includes:

[0023] A first acquisition module is used to acquire a first limb image, where the first limb image is image information of a limb object at a first viewing angle, where the limb object includes a first limb and a second limb, and the first limb and the second limb are symmetrical in physiological structure;

[0024] A first processing module is configured to call the feature encoding network to perform image encoding on the first limb image, extract at least two fused feature representations corresponding to at least two query points on the limb object, wherein an nth fused feature representation corresponding to an nth query point among the at least two query points is used to indicate a feature representation of an image region that is symmetrical in physiological structure, and the nth fused feature representation is used to supplement the feature representation of the nth query point on the first limb based on the symmetry of the limb object through the feature representation on the second limb;

[0025] The first rendering module is used to call the decoding network, perform feature decoding on the at least two fused feature representations, and render a second limb image; the generated area corresponding to the nth query point in the second limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the second limb image is image information of the limb object under a second perspective, and the second perspective is different from the first perspective.

[0026] According to another aspect of the present application, a training device for a limb part image prediction model is provided, wherein the limb part image prediction model includes a feature encoding network and a decoding network; the device includes:

[0027] a second acquisition module, configured to acquire a sample information pair, the sample information pair comprising a first sample image and a second sample image, the first sample image being image information of a sample object at a first viewing angle, the second sample image being image information of the sample object at a second viewing angle, the first viewing angle being different from the second viewing angle; the sample object comprising a first limb and a second limb, the first limb and the second limb being symmetrical in physiological structure;

[0028] a second processing module, configured to call the feature encoding network to perform image encoding on the first sample image, extract at least two predicted feature representations corresponding to at least two sample query points on the sample object, wherein an nth predicted feature representation corresponding to an nth sample query point among the at least two sample query points is used to indicate a feature representation of a sample region that is symmetrical in physiological structure, and the nth predicted feature representation is used to supplement the feature representation of the nth sample query point on the first limb through the feature representation on the second limb based on the symmetry of the sample object;

[0029] A second rendering module is used to call the decoding network, perform feature decoding on the at least two predicted feature representations, and render a predicted limb image, wherein the generated area corresponding to the nth sample query point in the predicted limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the predicted limb image is the predicted image information of the sample object under the second viewing angle;

[0030] The first training module is used to train the limb part image prediction model based on the difference between the predicted limb image and the second sample image to obtain a trained limb part image prediction model.

[0031] According to another aspect of the present application, a training device for a limb part image prediction model is provided, wherein the limb part image prediction model includes a feature encoding network, a decoding network, and a discriminant network; the device includes:

[0032] A third acquisition module is used to acquire a first sample image, where the first sample image is image information of a sample object at a first viewing angle, where the sample object includes a first limb and a second limb, and the first limb and the second limb are symmetrical in physiological structure;

[0033] a third processing module, configured to call the feature encoding network to perform image encoding on the first sample image, extract at least two predicted feature representations corresponding to at least two sample query points on the sample object, wherein an nth predicted feature representation corresponding to an nth sample query point among the at least two sample query points is used to indicate a feature representation of a sample region that is symmetrical in physiological structure, and the nth predicted feature representation is used to supplement the feature representation of the nth sample query point on the first limb through the feature representation on the second limb based on the symmetry of the sample object;

[0034] a third rendering module, configured to call the decoding network, perform feature decoding on the at least two predicted feature representations, and render a predicted limb image, wherein the generated region corresponding to the nth sample query point in the predicted limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the predicted limb image is image information of the sample object obtained by prediction at a second viewing angle, and the first viewing angle and the second viewing angle are different;

[0035] The third processing module is further used to call the discriminant network to predict and obtain first visibility information based on the predicted limb image, wherein the first visibility information is displayed under the second viewing angle, and is a visual condition of the sample object predicted in the predicted limb image observed under the first viewing angle;

[0036] The third acquisition module is further used to acquire second visibility information of the sample object, where the second visibility information is displayed at the second viewing angle, and is a visibility condition of the sample object in the first sample image observed at the first viewing angle;

[0037] The second training module is used to perform adversarial training on the discriminant network or prediction sub-model of the limb part image prediction model based on the difference between the first visibility information and the second visibility information to obtain a trained limb part image prediction model, wherein the prediction sub-model includes the feature encoding network and the decoding network.

[0038] According to another aspect of the present application, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the method for using the limb part image prediction model as described above, and / or the method for training the limb part image prediction model.

[0039] According to another aspect of the present application, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the method for using the limb part image prediction model as described above, and / or the method for training the limb part image prediction model.

[0040] According to another aspect of the present application, a computer program product is provided, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the method for using the limb part image prediction model described in the above aspects, and / or the method for training the limb part image prediction model.

[0041] The beneficial effects of the technical solution provided by this application include at least:

[0042] By performing image encoding on the first limb image, a fused feature representation of the query points on the limb object is obtained. Based on the symmetry of the limb object in the physiological structure, the feature representation of the query points on the first limb is supplemented by the feature representation of the second limb, thereby expanding the dimension of feature extraction of the query points. The feature representation of the query points is jointly described on two limb objects with symmetric physiological structures. The symmetry of the physiological structure is fully utilized, and even in the case of self-occlusion of the limb object, the feature information in the image can be effectively extracted, thereby improving the effect of rendering the limb image of the limb object at the second perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 is a schematic diagram of a computer system provided by an exemplary embodiment of the present application;

[0045] Figure 2 is a schematic diagram of a limb part image prediction model provided by an exemplary embodiment of the present application;

[0046] Figure 3 is a flow chart of a method for using a limb part image prediction model provided by an exemplary embodiment of the present application;

[0047] Figure 4 is a flow chart of a method for using a limb part image prediction model provided by an exemplary embodiment of the present application;

[0048] Figure 5 is a flow chart of a method for using a limb part image prediction model provided by an exemplary embodiment of the present application;

[0049] Figure 6 is a flow chart of a method for using a limb part image prediction model provided by an exemplary embodiment of the present application;

[0050] Figure 7 is a flow chart of a method for using a limb part image prediction model provided by an exemplary embodiment of the present application;

[0051] Figure 8 is a schematic diagram of a limb part image prediction model provided by an exemplary embodiment of the present application;

[0052] Fig. 9is a flow chart of a method for training a limb part image prediction model provided by an exemplary embodiment of the present application;

[0053] Fig.10 is a flow chart of a method for training a limb part image prediction model provided by an exemplary embodiment of the present application;

[0054] Fig.11 is a schematic diagram of a discriminant network provided by an exemplary embodiment of the present application;

[0055] Fig.12 is a flow chart of a method for training a limb part image prediction model provided by an exemplary embodiment of the present application;

[0056] Fig.13 is a schematic diagram of a two-handed image provided by an exemplary embodiment of the present application;

[0057] Fig.14 is a schematic diagram of a two-handed image provided by an exemplary embodiment of the present application;

[0058] Fig.15 is a schematic diagram of a two-handed image provided by an exemplary embodiment of the present application;

[0059] Fig.16 is a structural block diagram of a device for using a limb part image prediction model provided by an exemplary embodiment of the present application;

[0060] Fig.17 It is a structural block diagram of a training device for a limb part image prediction model provided by an exemplary embodiment of the present application;

[0061] Fig.18 It is a structural block diagram of a training device for a limb part image prediction model provided by an exemplary embodiment of the present application;

[0062] Fig.19 It is a structural block diagram of a server provided by an exemplary embodiment of the present application.

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. DETAILED DESCRIPTION

[0064] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0065] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0066] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0067] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the information such as limb images involved in this application is obtained with full authorization.

[0068] It should be understood that although the terms first, second, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first parameter may also be referred to as the second parameter, and similarly, the second parameter may also be referred to as the first parameter. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0069] Figure 1 A schematic diagram of a computer system provided by an embodiment of the present application is shown. The computer system can be implemented as a system architecture of a limb part image prediction model training method and / or use method. The computer system may include: a terminal 100 and a server 200.

[0070] The terminal 100 may be an electronic device such as a mobile phone, a tablet computer, a vehicle-mounted terminal (vehicle computer), a wearable device, a PC (Personal Computer), etc. The terminal 100 may be installed with a client that runs a target application, and the target application may be a limb part image prediction model training and / or use application, or may be other applications that provide limb part image prediction model training functions and / or use functions, which are not limited in this application. In addition, this application does not limit the form of the target application, including but not limited to App (Application, application), mini-programs, etc. installed in the terminal 100, and may also be in the form of a web page.

[0071] The server 200 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The server 200 may be a background server of the target application, used to provide background services for the client of the target application.

[0072] In the limb part image prediction model training method and / or use method provided in the embodiments of the present application, the execution subject of each step may be a computer device, which refers to an electronic device with data calculation, processing and storage capabilities. Figure 1 Taking the implementation environment of the scheme shown as an example, the limb part image prediction model training method and / or usage method can be executed by the terminal 100 (such as the client of the target application installed and running in the terminal 100 executes the limb part image prediction model training method and / or usage method), or the limb part image prediction model training method and / or usage method can be executed by the server 200, or the terminal 100 and the server 200 can interact and cooperate to execute, and this application is not limited to this.

[0073] In addition, the technical solution of the present application can be combined with blockchain technology. For example, the limb part image prediction model training method and / or use method disclosed in the present application, some of the data involved (limb image, distance transformation image, closed area and other data) can be stored on the blockchain. The terminal 100 and the server 200 can communicate through a network, such as a wired or wireless network.

[0074] Next, the limb part image prediction model in this application is introduced:

[0075] Figure 2 A schematic diagram of a limb part image prediction model provided by an embodiment of the present application is shown.

[0076] The limb part image prediction model includes a texture encoding network 402, a geometry encoding network 404, a structure reconstruction network 410, a fusion network 406 and a decoding network 415;

[0077] Among them, the texture coding network 402, the geometric coding network 404, and the structure reconstruction network 410 are networks arranged in parallel; the fusion network 406 in this embodiment is cascaded after the geometric coding network 404 and the structure reconstruction network 410, and is used to fuse the image geometric features 304. It can be seen that this implementation focuses on the technical solution of image prediction in the dimension of geometric features, but it should be noted that in different embodiments, the limb part image prediction model can also include another fusion network, cascaded after the texture coding network 402 and the structure reconstruction network 410, for fusing the image texture features 302. The above-mentioned image prediction for the geometric feature dimension and the texture feature dimension can be implemented separately or combined into a new technical solution implementation, and this application does not limit it.

[0078] The decoding network 415 in this embodiment is cascaded after the fusion network 406 , decodes the feature representation obtained by the fusion network 406 , and renders the second limb image 330 .

[0079] Next, each network in this embodiment will be further introduced.

[0080] A first limb image 300 is obtained, which is image information of a limb object under a first viewing angle. In this embodiment, the limb objects are both hands, and the first viewing angle is the oblique front direction of the right hand. Both hands include a left hand object and a right hand object, and the left hand object and the right hand object are symmetrical in physiological structure.

[0081] The texture coding network 402 in the limb image prediction model is used to extract texture features of both hands, and extracts texture features such as skin texture and horizontal lines at joints on both hands in the first limb image 300 to obtain image texture features 302. The geometric coding network 404 in the limb image prediction model is used to extract spatial geometric features of both hands, and extracts stereoscopic features of both hands in the first limb image 300 in three-dimensional space to obtain image geometric features 304.

[0082] The structure reconstruction network 410 is called to predict the first limb image 300 to obtain a three-dimensional structure grid 310 of both hands in the first limb image 300 . The three-dimensional structure grid 310 is a three-dimensional structure surrounded by at least two grid points.

[0083] The query points are key points on both hands. In a specific example, the key points are key points of both hands observed from the second perspective carried by the limb part image prediction model or the second perspective input into the limb part image prediction model. It is understandable that generating the second limb image 330 requires selecting multiple query points, such as 64 or 128. The above query points are randomly determined on both hands or selected in an equidistant manner.

[0084] In one example, the query point 311 is the middle point of the first knuckle on the right thumb, and the grid point adjacent to the query point 311 in the three-dimensional structure grid 310 is determined as the first grid point 312, and the first grid point 312 is the fingertip point of the right thumb; and the second grid point 314 corresponding to the first grid point 312 is determined, and the relative position of the first grid point 312 on the right hand is the same as the relative position of the second grid point 314 on the left hand; the second grid point 314 is the fingertip point of the left thumb.

[0085] The position information of the first grid point 312 and the second grid point 314 mapped on the first limb image 300 is determined; illustratively, if the grid point is not visible on the first limb image 300, the position information of the grid point is empty.

[0086] The fusion network 406 includes an interpolation layer 407 and a prediction layer 408; specifically:

[0087] Calling the interpolation layer 407 to interpolate the image geometric features 304 based on the position information, extracting the first sub-region features around the first grid point 312, the second sub-region features around the second grid point 314, and the third sub-region features around the query point 311;

[0088] The prediction layer 408 is called to predict the above three sub-region features to obtain three weights corresponding to the three sub-region features.

[0089] Based on the above three sub-region features and the three weights corresponding to the three sub-region features, the fused feature representation 320 corresponding to the query point 311 is obtained by concatenating the fused feature representations 320 corresponding to the query point 311. The fused feature representations corresponding to the multiple query points are determined one by one.

[0090] The decoding network 415 is called to perform feature decoding on the fused feature representation 320, and rendered to obtain a second limb image 330. The second limb image 330 is obtained by decoding the fused feature representation corresponding to multiple query points. The second limb image 330 is image information of the limb object at a second viewing angle. The second viewing angle is different from the first viewing angle, and the second viewing angle is the front direction of the right hand.

[0091] Next, the method of using the limb part image prediction model will be introduced through the following embodiments.

[0092] Figure 3 A flowchart of a method for using a limb part image prediction model provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. The method includes:

[0093] Step 510: Acquire a first limb image;

[0094] Exemplarily, the first limb image is image information of a limb object under a first perspective, and the limb object includes a first limb and a second limb, and the first limb and the second limb are symmetrical in physiological structure; Exemplarily, the limb object can be a limb part of a virtual object in a virtual environment, or a limb part of a physical object in the real world. Furthermore, the physical object in the real world can be a biological object, or an object with a physiological structure similar to that of a biological object (such as a physical model built with at least one material such as wood, stone, fur, etc.). Similarly, the virtual object in the virtual environment can be a virtual creature, or a virtual object with a physiological structure similar to that of a biological object.

[0095] For example, in various embodiments of the present application, the limb object is usually taken as an example of two hands, but it does not exclude the case where the limb object is other limb parts. For example, the limb object includes but is not limited to at least one of two hands, two arms, two feet, two legs, face, head, and body trunk. It should be noted that in the case where the limb object includes at least one of the face, the left and right sides of the face are symmetrical in physiological structure, and the head and body trunk are similar to the face, and are symmetrical in physiological structure on both sides of the central axis.

[0096] Step 520: calling a feature encoding network to perform image encoding on the first limb image, and extracting at least two fused feature representations corresponding to at least two query points on the limb object;

[0097] Exemplarily, the limb image prediction model in this embodiment includes a feature encoding network and a decoding network. In this embodiment, the limb image prediction model is used to predict image information of the limb object under the second viewing angle by encoding and decoding the first limb image.

[0098] Exemplarily, the query point is a point on a limb object, and the second limb image is obtained by encoding and decoding the query point. It should be noted that the query point can be a point that is not visible on the first limb image, that is, a point on the limb object that is blocked due to the observation angle under the first viewing angle; it can also be a point that is visible on the first limb image, and this application does not limit this. Exemplarily, the number of query points is multiple.

[0099] Exemplarily, the nth fused feature representation corresponding to the nth query point among the at least two query points is used to indicate the feature representation of the image region that is symmetrical in physiological structure. Exemplarily, the image region is the region that includes the nth query point. The image region can be a continuous image region or two or more image regions that are not connected to each other. Exemplarily, the image region is symmetrical in physiological structure, and one sub-portion in the image region belongs to the first limb, and the other sub-portion belongs to the second limb; for example, the image region is the region where the left thumb and the right thumb are located, that is, the image region includes a set of symmetrical positions on the limb object.

[0100] Exemplarily, the nth fused feature representation is used to indicate the feature representation of the image area on the first limb image in the latent space; since the image area includes image information on the first limb and the second limb, feature extraction of the nth query point on the first limb is achieved, and the feature representation of the nth query point is supplemented based on the image information on the second limb; exemplary, the nth fused feature representation supplements the feature representation of the nth query point on the first limb through the feature representation on the second limb based on the symmetry of the limb object.

[0101] Step 530: calling a decoding network to perform feature decoding on at least two fused feature representations, and rendering to obtain a second limb image;

[0102] Exemplarily, the decoding network is used to render a second limb image based on the input fused feature representation; exemplary, the decoding network performs feature encoding on feature information in the fused feature representation and decodes the feature representation in the latent space into image information.

[0103] Exemplarily, as the number of query points increases, the image details are more completely retained when predicting the second limb image. Exemplarily, when the number of dimensions represented by a single fusion feature is the same, as the number of query points increases, the second limb image carries more image detail information, such as the texture of the hand, the horizontal lines at the joints, etc.

[0104] Exemplarily, the generated area corresponding to the nth query point in the second limb image is obtained by decoding based on the nth fused feature representation, and the nth fused feature representation carries the feature representation on the second limb to supplement the feature of the nth query point on the first limb. Exemplarily, the generated area corresponding to the nth query point in the second limb image is obtained by decoding after supplementing the feature representation based on the feature representation on the second limb. The second limb image is the image information of the limb object at a second perspective, and the second perspective is different from the first perspective;

[0105] In summary, the method provided in this embodiment obtains a fused feature representation of the query points on the limb object by performing image encoding on the first limb image, and based on the symmetry of the limb object in the physiological structure, supplements the feature representation of the query points on the first limb with the feature representation on the second limb, thereby expanding the dimension of query point feature extraction; jointly describes the feature representation of the query points on two limb objects with physiological structure symmetry; fully utilizes the symmetry of the physiological structure, and even in the case of self-occlusion of the limb object, it can effectively extract the feature information in the image, thereby improving the effect of rendering the limb image of the limb object at the second perspective.

[0106] Figure 4 A flowchart showing a method for using a limb part image prediction model provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 3 In the illustrated embodiment, step 520 may be implemented as steps 522 and 524:

[0107] Step 522: calling a feature encoding network to extract a first region feature of a circumferential region of an nth query point on the first limb from the first limb image; and extract a second region feature of a symmetrical region on the second limb from the first limb image;

[0108] Exemplarily, the nth fusion feature representation corresponding to the nth query point includes the first region feature and the second region feature. The lateral region of the nth query point is a region including the nth query point, and this embodiment does not restrict the shape, area, etc. of the lateral region. Exemplarily, the symmetrical region and the lateral region of the nth query point are symmetrical in physiological structure; but at least one of the shapes, positions, and areas of the above two regions in the first limb image is not restricted, and the shapes, positions, and areas of the above two regions may be the same or different. In one example, due to the observation angle of the first perspective, the lateral region or the symmetrical region may be an empty set, that is, the observation under the first perspective is affected by the line of sight occlusion, and the lateral region or the symmetrical region does not exist in the first limb image.

[0109] Exemplarily, the symmetrical region corresponding to the lateral region may be obtained by marking, or may be determined in the first physiological image by prediction. This embodiment does not limit the method for determining the symmetrical region.

[0110] Step 524: calling the feature encoding network to concatenate the first region feature and the second region feature to obtain an nth fused feature representation corresponding to the nth query point, and obtaining at least two fused feature representations;

[0111] Exemplarily, the nth fused feature representation is obtained by splicing the first region feature and the second region feature; Exemplarily, the first region feature and the second region feature are connected end to end and spliced ​​to obtain the nth fused feature representation. Exemplarily, at least two fused feature representations corresponding to at least two query points are obtained by splicing feature information multiple times.

[0112] In an optional manner of the present application, there is corresponding weight information between the first regional feature and the second regional feature. Step 524 in this embodiment can be implemented as the following two sub-steps:

[0113] Sub-step 24a: calling the feature encoding network to predict the first region feature and the second region feature to obtain a first weight corresponding to the first region feature and a second weight corresponding to the second region feature;

[0114] Exemplarily, there is a multilayer perceptron (MLP) in the feature encoding network, which predicts the first region feature and the second region feature of the input to obtain weight information corresponding to the above features; further, the value of the weight information is greater than 0 and less than 1; exemplary, the weight information can be predicted by other artificial neural networks (ANN), which predict the weight information based on the hidden layer features of the feature representation.

[0115] Sub-step 24b: Based on the product of the first weight and the first regional feature, and the product of the second weight and the second regional feature, concatenate to obtain an nth fused feature representation corresponding to the nth query point, and obtain at least two fused feature representations.

[0116] Exemplarily, the nth fused feature representation includes two sub-parts spliced ​​end to end, the first sub-part is the product of the first weight and the first regional feature, and the second sub-part is the product of the second weight and the second regional feature.

[0117] In summary, the method provided in this embodiment obtains a fused feature representation of the query point on the limb object by performing image encoding on the first limb image, splices the first region feature and the second region feature based on the symmetry of the limb object in the physiological structure, and supplements the feature representation of the query point on the first limb with the second region feature on the second limb, thereby expanding the dimension of query point feature extraction; jointly describes the feature representation of the query point on two limb objects with physiological structure symmetry; fully utilizes the symmetry of the physiological structure, and can effectively extract feature information in the image even when the limb object is self-occluded, thereby improving the effect of rendering the limb image of the limb object at the second perspective.

[0118] Next, the encoding process of the first region feature and the second region feature is introduced.

[0119] Figure 5 A flowchart showing a method for using a limb part image prediction model provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 4 In the illustrated embodiment, step 522 may be implemented as step 522a and step 522b:

[0120] Step 522a: calling the encoding subnetwork to perform image encoding on the first limb image, and extracting image feature representation of the first limb image in the latent space;

[0121] In this embodiment, the feature encoding network includes an encoding subnetwork and a fusion subnetwork; specifically, the encoding subnetwork and the fusion subnetwork will be introduced in two steps of this embodiment.

[0122] Exemplarily, the first limb image is determined as an input parameter of the encoding subnetwork, and image encoding is performed to obtain an image feature representation of the first limb image in the hidden layer space, and the image feature representation is a global feature representation of the first limb image. Exemplarily, the encoding subnetwork obtains the image feature representation through two-dimensional convolutional encoding.

[0123] In one example, the image feature representation is used to indicate the texture feature in the first limb image, such as skin texture; accordingly, the encoding subnetwork is a residual connected convolutional neural network (CNN), also known as a texture encoder. In another example, the image feature representation is used to indicate the spatial geometric feature in the first limb image; accordingly, the encoding subnetwork is an hourglass network (Hourglass Network), also known as a geometric encoder. The above two examples can also be combined, and this application does not limit this.

[0124] Step 522b: Call the fusion sub-network to extract the first regional features of the surrounding area of ​​the nth query point on the first limb in the image feature representation; and extract the second regional features of the symmetrical area on the second limb;

[0125] Exemplarily, the image feature representation is determined as the input parameter of the fusion sub-network, and the surrounding area and the symmetric area are sampled one by one in the image feature representation in a bilinear interpolation manner to obtain a first area feature of the surrounding area and a second area feature of the symmetric area.

[0126] In one example, step 524 in this embodiment is implemented by calling a fusion sub-network, that is, calling the fusion sub-network to splice the first region feature and the second region feature to obtain the nth fused feature representation corresponding to the nth query point, and obtaining at least two fused feature representations.

[0127] In summary, the method provided in this embodiment performs image encoding on the first limb image to extract the global image feature representation of the first limb image, and then extracts the first region feature and the second region feature by interpolation; the feature representation of the query point on the first limb is supplemented by the feature representation on the second limb, thereby expanding the dimension of query point feature extraction; the feature representation of the query point is jointly described on two limb objects with physiological structure symmetry; the symmetry of the physiological structure is fully utilized, and the hidden layer features of the query point are described by the lateral area and the symmetrical area with physiological symmetry in the first limb image, thereby providing a fine-grained image feature extraction method, thereby avoiding interference with the generation of the limb image of the second perspective due to the different geometric structures and surface textures of the limb objects at various positions when using the global features of the image, thereby ensuring that the limb image of the second perspective can carry the local texture details of the limb object; and improving the effect of rendering the limb image of the limb object at the second perspective.

[0128] Next, the surrounding area and the symmetric area of ​​the nth query point are introduced.

[0129] Figure 6 A flowchart showing a method for using a limb part image prediction model provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 4 Based on the embodiment shown, step 515, step 516 and step 517 are also included:

[0130] Step 515: calling the structure reconstruction network to predict the first limb image to obtain a three-dimensional structure grid of the limb object in the first limb image;

[0131] Exemplarily, the first limb image is determined as an input parameter of a structure reconstruction network, and a three-dimensional structure grid of the limb object in the first limb image is predicted; the structure reconstruction network is a model with the ability to predict three-dimensional structure. In one example, the limb object is two hands, and the structure reconstruction network is a hand model with articulated and non-rigid defOrmations (MANO). The first limb image is input into the MANO model to obtain a three-dimensional structure grid of the two hands.

[0132] Step 516: calling the structure reconstruction network, determining the grid point adjacent to the nth query point in the three-dimensional structure grid as the first grid point, and determining the second grid point corresponding to the first grid point;

[0133] Exemplarily, the three-dimensional structure grid includes a plurality of grid points, and the three-dimensional structure of the limb object is surrounded by the grid points. Exemplarily, the grid points on the three-dimensional structure grid are vertices on the three-dimensional structure grid. As described above, the query point is a point randomly determined on both hands, or a point selected in an equidistant manner under a second viewing angle; further, the second viewing angle can be a viewing angle parameter carried by a limb part image prediction model, or a parameter input into a limb part image prediction model.

[0134] Exemplarily, since the grid points are vertices on the three-dimensional structure grid and are sparsely arranged on the three-dimensional structure grid, the grid point adjacent to the nth query point in the three-dimensional structure grid is determined as the first grid point, and the distance between the first grid point and the nth query point is less than or equal to any other grid point. Exemplarily, the relative position of the first grid point on the first limb is the same as the relative position of the second grid point on the second limb, so as to determine the second grid point corresponding to the first grid point.

[0135] Step 517: calling the structure reconstruction network, determining the surrounding area of ​​the nth query point based on the first grid point, and determining the symmetric area based on the second grid point;

[0136] Exemplarily, the surrounding area of ​​the nth query point includes the nth query point; for the surrounding area of ​​the nth query point, in one implementation, the surrounding area is centered on the first grid point and includes the nth query point, so as to determine the surrounding area of ​​the nth query point based on the first grid point. Exemplarily, corresponding to the surrounding area, there is a symmetrical query point in the symmetrical area, and the physiological structure position of the symmetrical query point on the second limb is the same as the physiological structure position of the nth query point on the first limb; similarly, the symmetrical area is centered on the second grid point and includes the symmetrical query point, so as to determine the symmetrical area based on the second grid point.

[0137] In an optional implementation, this step can be implemented as: calling a structural reconstruction network to determine first position information of a first grid point mapped on a first limb image; and determining a lateral area around the nth query point with the first position information as the center; calling a structural reconstruction network to determine second position information of a second grid point mapped on the first limb image; and determining a symmetrical area with the second position information as the center.

[0138] In summary, the method provided in this embodiment obtains a fused feature representation of the query point on the limb object by performing image encoding on the first limb image, determines the symmetric area and the lateral area on the limb object through a structure reconstruction network, and realizes the prediction of the three-dimensional structure of the limb object based on the two-dimensional image information, and then maps the area with physiological structure symmetry in the three-dimensional structure onto the two-dimensional image, and accurately determines the symmetric area and the lateral area that need to be extracted. Based on the symmetry of the limb object in the physiological structure, the first area feature and the second area feature are spliced, and the feature representation of the query point on the first limb is supplemented by the second area feature on the second limb, thereby expanding the dimension of query point feature extraction. On two limb objects with physiological structure symmetry, the feature representation of the query point is jointly described. The symmetry of the physiological structure is fully utilized, and even in the case of self-occlusion of the limb object, the feature information in the image can be effectively extracted, thereby improving the effect of rendering the limb image of the limb object at the second perspective.

[0139] Next, the spatial features are introduced. In an optional implementation of the present application, Figure 4 Based on the illustrated embodiment, the following steps are further included after step 524:

[0140] Obtain the interest position of the interest point on the three-dimensional structure grid, and the query position of the nth query point on the three-dimensional structure grid;

[0141] Exemplarily, the nth fused feature representation also includes a spatial feature (Spatial Feature), and the spatial feature is used to indicate the spatial depth of the nth query point relative to the interest point of the limb object.

[0142] Exemplarily, the interest point is a point on a limb object that is visually significant or associated with the limb object's activity; taking the limb objects as hands as an example, the interest point is a joint point on the hands. Exemplarily, the interest position is a position in a three-dimensional space, and the three-dimensional space is the three-dimensional space where the three-dimensional structure grid is located when predicting the three-dimensional structure of the limb object.

[0143] Calling the feature encoding network to construct spatial features based on the interest position and the query position; and adding spatial features to the fused feature representation.

[0144] Exemplarily, the spatial features include an interest position and a query position; taking the limb object as two hands as an example, the spatial features are obtained based on the query position of the nth query point in three-dimensional space and the interest position of the joints on the two hands.

[0145] In one example, the nth query point is denoted as q and the interest point is denoted as p k Here, k points of interest are taken as an example. For example, the plane where the first limb image is located is denoted as Pn ; The depth value of the nth query point to the plane where the first limb image is located is recorded as z(q|P n ), the depth value of the interest point to the plane where the first limb image is located is recorded as z(p k |P n ); compare the nth query point with all interest points p k The relative depth difference of the depth value is δ n (p k ,q)=z(p k |P n )-z(q|P n ). The spatial features of the nth query point are obtained by position encoding the relative depth difference and calculating the Gaussian exponential kernel:

[0146]

[0147] γL(x)=[sin(x),cos(x),…,sin(2 L-1 x),cos(2 L-1 x)]

[0148] in, is the spatial feature of the nth query point in the plane where the first limb image is located; is the Gaussian exponential kernel; l2 is the L2-norm calculation, α is the weight coefficient controlling the influence of the interest point, such as the weight coefficient of the influence of each hand joint; γ(·) is used to indicate the position encoding, specifically, L is a hyperparameter controlling the maximum encoding frequency, which is predetermined based on the granularity of the position encoding. k is the number of interest points.

[0149] In summary, the method provided in this embodiment obtains a fused feature representation of the query point on the limb object by performing image encoding on the first limb image, and based on the symmetry of the limb object in the physiological structure, supplements the feature representation of the query point on the first limb through the feature representation on the second limb; quantitatively describes the spatial depth of the query point relative to the interest point of the limb object through spatial features; expands the dimension of query point feature extraction; jointly describes the feature representation of the query point on two limb objects with physiological structure symmetry; makes full use of the symmetry of the physiological structure, and can effectively extract feature information in the image even when the limb object is self-occluded, thereby improving the effect of rendering the limb image of the limb object at the second perspective.

[0150] Figure 7 A flowchart showing a method for using a limb part image prediction model provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Figure 3 Based on the illustrated embodiment, step 525 is also included:

[0151] Step 525: calling a feature encoding network to perform image encoding on the first limb image, extracting global texture features; and adding the global texture features to the fused feature representation;

[0152] Exemplarily, in the case where the fused feature representation is used to indicate the texture features of the limb object, the nth fused feature representation also includes a global texture feature; Exemplarily, the global texture feature is used to indicate the global average texture information on the limb object, and the global texture information is added to the fused feature representation to supplement it with the global information; Even if occlusion occurs in the first perspective, and the nth query point and the corresponding symmetrical points on the second limb are not visible, since the texture information at each position on the limb object is similar, it can be effectively supplemented based on the global texture information, and the feature information of the surface texture of the nth query point is represented in the fused feature representation. Exemplarily, the global texture feature is a global feature of the limb object in the first limb image.

[0153] In one implementation of the present application, the fused feature represents a texture feature and / or a geometric structure feature used to indicate a limb object; as described above Figure 2 In the corresponding embodiment, when the fused feature represents the texture feature and the geometric structure feature for indicating the limb object, the feature information can be extracted through the parallel network structure. This application does not limit this. Figure 2 The corresponding embodiments show that the texture coding network and the geometry coding network perform image coding from two dimensions.

[0154] In one implementation of the present application, step 525 in this embodiment may be implemented as the following sub-steps:

[0155] Based on the position of the first limb, calling the feature encoding network to extract a first texture feature in the first limb image;

[0156] Based on the position of the second limb, calling the feature encoding network to extract a second texture feature in the first limb image;

[0157] The feature encoding network is called to predict the weight information corresponding to the first texture feature and the second texture feature, and the first texture feature and the second texture feature, as well as the one-to-one weight information of the first texture feature and the second texture feature, are added to the fused feature representation.

[0158] Exemplarily, the global texture feature includes a first texture feature corresponding to the first limb and a second texture feature corresponding to the second limb; by separating the global texture features on the first limb and the second limb, it is fully considered that when the limb object is at least one of the hands, arms, feet, and legs, the first limb and the second limb are two independent parts, and there are independent differences in texture features due to at least one of the factors such as the accessories worn and the distribution of hair. The feature information of the nth query point is described by extracting the first texture feature and the second texture feature respectively, and configuring independent weight information. Exemplarily, in the process of extracting the first texture feature, it is extracted based on the position of the first limb in the first limb image; similarly, the second texture feature is extracted based on the position of the second limb in the first limb image.

[0159] Exemplarily, the weight information is used to indicate the importance of the texture global feature in the fusion feature representation. Exemplarily, the weight information corresponding to different query points in the multiple query points is different, and the weight information is obtained based on prediction. Exemplarily, the feature encoding network has the ability to predict the weight information. It should be noted that this embodiment can be used together with the above Figure 4 The corresponding embodiments are implemented in combination to simultaneously predict the one-to-one weight information of the first region feature, the second region feature, the first texture feature, and the second texture feature. Similar to the above, there is a multilayer perceptron (MLP) in the feature encoding network to predict the input first texture feature and the second texture feature to obtain the one-to-one weight information of the first texture feature and the second texture feature. Exemplarily, the first texture feature and the second texture feature have independent weight information corresponding to each other in the fused feature representation, and the weight information corresponding to the first texture feature and the second texture feature in the fused feature representation may be the same or different.

[0160] In summary, the method provided in this embodiment obtains a fused feature representation of the query points on the limb object by performing image encoding on the first limb image, and based on the symmetry of the limb object in the physiological structure, supplements the feature representation of the query points on the first limb with the feature representation on the second limb; by adding global texture features, the similarity of the surface texture on the limb object is fully considered, and the dimension of query point feature extraction is expanded; on two limb objects with physiological structure symmetry, the feature representation of the query point is jointly described; the symmetry of the physiological structure is fully utilized, and even in the case of self-occlusion of the limb object, the feature information in the image can be effectively extracted, thereby improving the effect of rendering the limb image of the limb object at the second perspective.

[0161] Figure 8A schematic diagram of a limb part image prediction model provided by an embodiment of the present application is shown.

[0162] In this embodiment, feature extraction is performed on the geometric feature dimension and the texture feature dimension to predict the second limb image. The description of the first limb image 300, the structure reconstruction network 410, etc. in this embodiment can be referred to in the above Figure 2 The corresponding embodiments will not be described in detail in this embodiment.

[0163] Exemplarily, the texture coding network 402 includes a residual connected convolutional neural network (CNN), also known as a texture encoder. The geometric coding network 404 includes an hourglass network (Hourglass Network), also known as a geometric encoder. The texture coding network 402 extracts texture features of the first limb image 300; taking the hands as an example, the texture features such as the skin texture and the horizontal lines at the joints on the hands in the first limb image 300 are extracted to obtain the image texture features 302. The geometric coding network 404 extracts the spatial geometric features of the first limb image 300; taking the hands as an example, the three-dimensional features of how the knuckles of the hands in the first limb image 300 are bent in three-dimensional space and how the fingers are in contact with each other are extracted to obtain the image geometric features 304.

[0164] Exemplarily, the structure reconstruction subnetwork 410 is called to predict the first limb image 300, and obtain the three-dimensional structure grid 310 of both hands in the first limb image 300. Take the query point 311 as the middle point of the first knuckle on the right thumb as an example for explanation, the grid point adjacent to the query point 311 in the three-dimensional structure grid 310 is determined as the first grid point 312, and the first grid point 312 is the fingertip point of the right thumb; correspondingly, the second grid point 314 is the fingertip point of the left thumb.

[0165] For the image texture feature 302, the first interpolation layer 407a in the first fusion network 406a performs bilinear interpolation according to the positions of the query point 311, the first grid point 312, and the second grid point 314 mapped in the first limb image 300 to obtain the texture region feature 302a. In one example, the texture region feature 302a includes three sub-parts, the first sub-part is denoted as k(q), which is used to indicate the feature representation of the region with the query point 311 as the center point; the second sub-part is denoted as m(p), which is used to indicate the feature representation of the region with the first grid point 312 as the center point; the third sub-part is denoted as n(p′) which is used to indicate the feature representation of the region with the second grid point 314 as the center point. Exemplarily, the above describes the case where the texture region feature 302a only includes the second sub-part corresponding to the first grid point 312 and the third sub-part corresponding to the second grid point 314, and the present application does not limit this. The first sub-portion corresponds to a region with the query point 311 as the center point, and can accurately locate the feature information of the query point 311 , further avoiding the influence caused by the position offset between the first grid point 312 and the query point 311 .

[0166] It should be noted that the texture region feature 302a may also include a fourth sub-part, denoted as It is called spatial feature. For the method of obtaining spatial feature, please refer to the introduction of spatial feature above, which will not be repeated here.

[0167] For the texture region feature 302a, a fifth sub-part and a sixth sub-part may also be included; wherein the fifth sub-part is denoted as g l , used to indicate the first texture feature corresponding to the first limb, taking both hands as an example, it is used to indicate the texture feature of the left hand. The sixth sub-part is denoted as g r , used to indicate the first texture feature corresponding to the second limb, taking both hands as an example, it is used to indicate the texture feature of the right hand. It should be noted that, as described above, since the texture information of each position on the limb object has similarity, it can be effectively supplemented based on the global texture information, and the feature information of the surface texture of the nth query point is represented in the fusion feature representation. As for the image geometric feature 304, since the query point 311 is only an area in the limb object, there is no similarity between the local position in the limb object and the global geometric feature, and the geometric area feature 304a corresponding to the image geometric feature 304 usually does not have a global geometric feature.

[0168] Exemplarily, for the above six sub-parts in the texture region feature 302a, there is one-to-one corresponding weight information, the texture region feature 302a includes the product of the weight information and the corresponding sub-part, and the texture region feature 302a is obtained by splicing the six products front and back. In one example, the texture region feature 302a is recorded as t(q); specifically:

[0169]

[0170] Among them, a is the weight information corresponding to the above six sub-parts. Specifically, is the feature weight.

[0171] The first prediction layer 408a in the first fusion network 406a predicts the texture weight information 302b based on the above six sub-parts. In a specific example, the feature weight a is calculated by the function μ:

[0172] μ:R 6D+3 →[0,1]

[0173]

[0174] Wherein, v(q, d) is used to indicate the visibility of the query point 311 in the viewing direction d of the first limb image 300, if p is visible, then v(p, d) = 1, otherwise v(p, d) = 0. Similarly, v(p, d) is used to indicate the visibility of the first grid point 312, and v(p′, d) is used to indicate the visibility of the second grid point 314.

[0175] Exemplarily, when predicting the above six weight information, based on the visibility information, the prediction is obtained. Exemplarily, the first prediction layer 408a is a multilayer perceptron (MLP) that predicts the above weight information.

[0176] For the image geometric feature 304; the second interpolation layer 407b in the second fusion network 406b performs bilinear interpolation according to the positions of the query point 311, the first grid point 312, and the second grid point 314 mapped in the first limb image 300, and obtains the geometric region feature 304a. In one example, the geometric region feature 304a includes three sub-parts. For the specific three parts, please refer to the above description of the image texture feature 302, which will not be repeated here. It should be noted that the sub-parts in the geometric region feature 304a are used to indicate the feature information on the geometric structure, and the sub-parts in the texture region feature 302a are used to indicate the feature information on the surface texture. Exemplarily, similar to the first fusion network 406a above, the second prediction layer 408b in the second fusion network 406b predicts the geometric weight information 304b corresponding to each sub-part based on each sub-part in the geometric region feature 304a.

[0177] The decoding network 415 performs feature decoding on the texture fusion feature composed of the texture region feature 302a and the texture weight information 302b, and the geometric fusion feature composed of the geometric region feature 304a and the geometric weight information 304b, and renders the second limb image 330 .

[0178] In a specific example, the texture fusion feature composed of the texture region feature 302a and the texture weight information 302b is used to infer the color of the second limb image 330; for example, the texture fusion feature is predicted based on the multi-layer perceptron to obtain the color of the second limb image 330. The geometric fusion feature composed of the geometric region feature 304a and the geometric weight information 304b is used to infer the transparency (also called density) of the second limb image 330; for example:

[0179] σ(q)=w -1 sig(-(s(q)+δ(q)) / w)

[0180] Wherein, σ(q) is the transparency of the query point 311, w is the weight parameter obtained by model training; Sig(·) represents the sigmoid function. s(q)∈R is the signed distance field (SDF) of the query point 311 calculated using the grid surface as the zero horizontal plane, and δ(q)∈R is the deviation inferred by another multi-layer perceptron using the geometric features of q as input.

[0181] According to the color and transparency of the second limb image 330, the second limb image 330 is rendered. Exemplarily, feature decoding is performed by at least two fused feature representations corresponding to at least two query points, so that the image prediction task is decomposed into multiple image area prediction tasks around the query points. When the image area prediction around the query points is performed one by one, the image area around the query points is a local area of ​​the second limb image of the second perspective, which reduces the data processing amount of the decoding network; compared with the prior art, the decoding network in this embodiment does not need to perform an overall prediction of the second limb image globally, and can perform image prediction of local areas in sequence, which reduces the computing resources required when the computer device calls the decoding network, and reduces the data parallel processing capability requirements of the limb part image prediction model for the computer device hardware. Further, this embodiment provides independent image prediction from the two dimensions of color and transparency, and the generation task of the image area around the query point can be further split into the prediction task of color and transparency parameters, further reducing the data parallel processing capability requirements for the computer device hardware.

[0182] Next, the training method of the limb part image prediction model will be introduced through the following embodiments.

[0183] Fig. 9 A flowchart of a method for training a limb part image prediction model provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. The method includes:

[0184] Step 610: Obtain sample information pairs;

[0185] The sample information pair includes a first sample image and a second sample image, the first sample image is image information of the sample object at a first perspective, and the second sample image is image information of the sample object at a second perspective, and the first perspective and the second perspective are different; the sample object includes a first limb and a second limb, and the first limb and the second limb are symmetrical in physiological structure; exemplarily, the sample object includes but is not limited to at least one of hands, arms, feet, legs, face, head, and body trunk.

[0186] Step 620: Calling a feature encoding network to perform image encoding on the first sample image, and extracting at least two prediction feature representations corresponding to at least two sample query points on the sample object;

[0187] In this embodiment, the limb part image prediction model includes a feature encoding network and a decoding network. For the introduction of the above networks, please refer to the various embodiments above, which will not be repeated here.

[0188] Exemplarily, the sample query point is a point on the sample object, and the predicted limb image is obtained by encoding and decoding the sample query point. Exemplarily, the number of the sample query point is multiple.

[0189] Exemplarily, the nth predicted feature representation corresponding to the nth sample query point among the at least two sample query points is used to indicate a feature representation of a sample region that is symmetrical in physiological structure. The sample region is symmetrical in physiological structure, one sub-portion of the sample region belongs to a first limb, and another sub-portion of the sample region belongs to a second limb; the sample region includes a set of symmetrical positions on the sample object.

[0190] Exemplarily, the nth predicted feature representation is used to supplement the feature representation of the nth sample query point on the first limb by the feature representation on the second limb based on the symmetry of the sample object;

[0191] Step 630: calling a decoding network to perform feature decoding on at least two predicted feature representations, and rendering to obtain a predicted limb image;

[0192] The decoding network is used to render a predicted limb image based on the input predicted feature representation; illustratively, the decoding network performs feature encoding on the feature information in the predicted feature representation and decodes the feature representation in the latent space into image information.

[0193] Exemplarily, the generated region corresponding to the nth sample query point in the predicted limb image is obtained by decoding based on the nth predicted feature representation, and the nth predicted feature representation carries the feature representation on the second limb to perform feature supplementation on the nth sample query point on the first limb. The generated region corresponding to the nth sample query point in the predicted limb image is obtained by decoding based on the feature representation on the second limb after feature supplementation, and the predicted limb image is the predicted image information of the sample object at the second viewing angle;

[0194] Step 640: training the limb part image prediction model based on the difference between the predicted limb image and the second sample image to obtain a trained limb part image prediction model;

[0195] Exemplarily, based on the difference between the predicted limb image and the second sample image, the limb part image prediction model is trained by backward error propagation to reduce the difference between the predicted limb image and the second sample image, improve the prediction ability of the limb part image prediction model for the predicted limb image, and obtain a trained limb part image prediction model.

[0196] Exemplarily, the limb part image prediction model trained in this application is Figures 2 to 8 For the network model in the corresponding embodiment, during the training process, the introduction of the network structure of the limb part image prediction model can refer to the various embodiments above. In this embodiment, in order to avoid redundancy, they will not be repeated one by one.

[0197] In summary, the method provided in this embodiment obtains a predicted feature representation of a sample query point on a sample object by performing image encoding on a first sample image, and based on the symmetry of the sample object in the physiological structure, supplements the feature representation of the sample query point on the first limb with a feature representation on a second limb, thereby expanding the dimension of feature extraction of the sample query point; improves the ability to generate predicted limb images from a second perspective by training a limb part image prediction model; and improves the effect of rendering the limb image of a limb object from a second perspective.

[0198] exist Fig. 9 In an optional implementation of the illustrated embodiment, step 620 can be implemented as the following two sub-steps:

[0199] · Calling a feature encoding network to extract a first region feature of a surrounding region of an nth sample query point on a first limb from a first sample image; and extracting a second region feature of a symmetrical region on a second limb from the first sample image;

[0200] Calling the feature encoding network to concatenate the first region feature and the second region feature to obtain an nth predicted feature representation corresponding to the nth sample query point, thereby obtaining at least two predicted feature representations.

[0201] Exemplarily, the nth prediction feature corresponding to the nth sample query point includes the first region feature and the second region feature; the surrounding region of the nth sample query point is a region including the nth sample query point, and this embodiment does not restrictively stipulate the shape, area, etc. of the surrounding region. The symmetric region and the surrounding region of the nth sample query point are symmetrical in physiological structure.

[0202] For example, the symmetric region corresponding to the lateral region can be obtained by annotation or determined in the first physiological image by prediction. This embodiment does not limit the method for determining the symmetric region. In one example, the symmetric region corresponding to the lateral region is obtained based on the prediction of the structure reconstruction network. For details, please refer to the above Figure 6 The corresponding embodiments.

[0203] Furthermore, in an optional implementation, the first region feature and the second region feature have one-to-one corresponding weight information, specifically:

[0204] Calling a feature encoding network to predict the first region feature and the second region feature to obtain a first weight corresponding to the first region feature and a second weight corresponding to the second region feature;

[0205] Based on the product of the first weight and the first regional feature, and the product of the second weight and the second regional feature, an nth predicted feature representation corresponding to the nth query point is concatenated to obtain at least two predicted feature representations.

[0206] Exemplarily, a multilayer perceptron (MLP) exists in the feature encoding network, which predicts the input first region feature and the second region feature to obtain weight information corresponding to the above features; further, the value of the weight information is greater than 0 and less than 1; exemplary, the nth predicted feature representation includes two sub-parts spliced ​​together, the first sub-part is the product of the first weight and the first region feature, and the second sub-part is the product of the second weight and the second region feature.

[0207] In summary, the method provided in this embodiment obtains a predicted feature representation of a sample query point on a sample object by performing image encoding on a first sample image, and based on the symmetry of the sample object in the physiological structure, supplements the feature representation of the sample query point on the first limb with a feature representation on a second limb, thereby expanding the dimension of feature extraction of the sample query point; improves the ability to generate predicted limb images from a second perspective by training a limb part image prediction model; and improves the effect of rendering the limb image of a limb object from a second perspective.

[0208] Fig.10 A flowchart of a method for training a limb part image prediction model provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. Fig. 9 Based on the illustrated embodiment, step 652, step 654 and step 656 are also included:

[0209] Step 652: calling the discriminant network to obtain first visibility information based on the predicted limb image prediction;

[0210] Exemplarily, the first visibility information is displayed at the second viewing angle, predicting the visibility of the predicted sample object in the limb image at the first viewing angle; exemplary, the first visibility information is a binary image, and the pixel grayscale of 1 in the first visibility information is used to indicate that it is visible at the first viewing angle, and the pixel grayscale of 0 is used to indicate that it is not visible at the first viewing angle; in an example, the first visibility information is obtained based on the image quality of the predicted limb image; exemplary, the image quality is used to indicate the degree of detail of the texture; since the position that is not visible at the first viewing angle is a feature representation obtained based on the symmetry of the physiological structure or the global feature supplement, it cannot accurately reflect the image features, and the image quality is lower than the image area visible at the first viewing angle, based on the above-mentioned image quality characteristics, the first visibility information is predicted.

[0211] Exemplarily, the discriminant network can predict the first visibility information based on the predicted limb image and the image quality difference of different regions inside the predicted limb image; or the first sample image can be used as a reference for image quality to predict the first visibility information of the predicted limb image, which is not limited in this embodiment; Exemplarily, the discriminant network is also called a discriminator, denoted by Φ(I,I t ), where I is the first sample image providing image quality reference, I t It is a predicted limb image for which the first visibility information needs to be predicted. Specifically:

[0212] Φ(I,I t ):R H×W ×R t H×W →{0,1} H×W

[0213] Among them, I and I t The height and width of the first visibility information obtained by the discriminator are H×W, and the value is 0 or 1.

[0214] Fig.11 A schematic diagram of a discriminant network provided by an exemplary embodiment of the present application is shown. Exemplarily, the discriminant network 410 is called to predict the predicted limb image 350 to obtain the first visibility information 354; it should be noted that the predicted limb image 350 is an image of the sample object observed at the second viewing angle. Taking the sample object as two hands as an example, the second viewing angle is the front direction of the right hand; the predicted limb image 350 is predicted based on the first sample image 352, and the first sample image 352 is an image of the sample object observed at the first viewing angle, and the first viewing angle is the oblique front direction of the right hand. Exemplarily, the pixel grayscale of 1 in the first visibility information 354 is used to indicate that it is visible at the first viewing angle, and the pixel grayscale of 0 is used to indicate that it is not visible at the first viewing angle; the discriminant network 410 is used to predict the visibility of each pixel in the predicted limb image 350 at the first viewing angle. Exemplarily, the discriminant network 410 predicts the first visibility information 354 based on the image quality in the predicted limb image 350.

[0215] Step 654: Obtain second visibility information of the sample object;

[0216] Exemplarily, the second visibility information is displayed at a second viewing angle, and is the visual condition of the sample object in the first sample image observed at the first viewing angle; the first sample image is an image at the first viewing angle input into the limb part image prediction model, and the second visibility information is used to display the visual condition of the first sample image at the second viewing angle.

[0217] The first viewing angle and the second viewing angle of the exemplary first sample image are different, and the second visibility information is obtained based on the first sample image conversion based on the relative angle between the first viewing angle and the second viewing angle.

[0218] Step 656: Based on the difference between the first visibility information and the second visibility information, supplementary training is performed on the discriminant network or the prediction sub-model of the limb part image prediction model;

[0219] Exemplarily, the difference between the first visibility information and the second visibility information is used to indicate the prediction ability of the discriminant network for the first visibility information, that is, the ability of the discriminant network to accurately distinguish whether the predicted limb image is an image generated by the model. On the other hand, the difference between the first visibility information and the second visibility information indicates the prediction ability of the prediction sub-model for the predicted limb image, and the prediction sub-model includes a feature encoding network and a decoding network; that is, the ability of the prediction sub-model to accurately generate image information under the second perspective.

[0220] By supplementing the training of the discriminant network or the prediction sub-model of the limb part image prediction model, the discriminant network and the prediction sub-model are alternately trained through adversarial training to improve the prediction ability of the prediction sub-network for predicting limb images from the second perspective.

[0221] In a specific example, the difference between the first visibility information and the second visibility information is determined as:

[0222] L=λ rgb L rgb +λ VGG L VGG +λ adv L adv +λ vis L vis

[0223] where λ rgb , VGG , adv , vis is a preset loss weight. In one example, the loss weight is set to λ rgb =10.0,λ VGG =1.0,λ adv =0.1,λ vis =0.1.

[0224] L rgb is the L1 norm loss between the first visibility information and the second visibility information, L VGG is the perceptual loss between the first visibility information and the second visibility information, and is used to indicate the difference between the first visibility information and the second visibility information at the semantic level. adv is the non-saturated adversarial generation loss. L vis is the pixel cross entropy loss, which is used to supervise visibility learning and is used to indicate the predicted visibility V and V t The pixel-wise binary cross entropy between is shown below:

[0225] L vis (V,V t )=-(V t⊙logV+(1-V t )⊙log(1-V))

[0226] Where ⊙ represents the element-wise dot product. t represents the pixel point of the visibility map generated based on the real image in the second visibility information, and V is the pixel point of the visibility map predicted in the first visibility information.

[0227] In summary, the method provided in this embodiment obtains a predicted feature representation of a sample query point on a sample object by performing image encoding on a first sample image, and based on the symmetry of the sample object in the physiological structure, supplements the feature representation of the sample query point on the first limb with the feature representation on the second limb, thereby expanding the dimension of feature extraction of the sample query point; supplements the training of the discriminant network or prediction sub-model of the limb part image prediction model in an adversarial manner, thereby improving the ability to generate predicted limb images from a second perspective; and improving the effect of rendering the limb image of the limb object from a second perspective.

[0228] Fig.12 A flowchart of a method for training a limb part image prediction model provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. The method includes:

[0229] Step 710: Acquire a first sample image;

[0230] The first sample image is image information of the sample object at a first perspective, where the sample object includes a first limb and a second limb, and the first limb and the second limb are symmetrical in physiological structure; illustratively, the sample object includes but is not limited to at least one of two hands, two arms, two feet, two legs, a face, a head, and a body trunk.

[0231] It should be noted that the technical features of the feature encoding network, prediction feature representation, etc. in each step of this embodiment can refer to the above Fig. 9 , Fig.10 The corresponding embodiments will not be repeated here one by one.

[0232] Step 720: Calling a feature encoding network to perform image encoding on the first sample image, and extracting at least two prediction feature representations corresponding to at least two sample query points on the sample object;

[0233] The nth predicted feature representation corresponding to the nth sample query point of the at least two sample query points is used to indicate a feature representation of a sample region having symmetry in physiological structure, and the nth predicted feature representation is used to supplement the feature representation of the nth sample query point on the first limb through the feature representation on the second limb based on the symmetry of the sample object;

[0234] Step 730: calling a decoding network to perform feature decoding on at least two predicted feature representations, and rendering to obtain a predicted limb image;

[0235] The generated region corresponding to the nth sample query point in the predicted limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the predicted limb image is the image information of the sample object under the second viewing angle obtained by prediction, and the first viewing angle and the second viewing angle are different;

[0236] Step 740: calling the discriminant network to obtain first visibility information based on the predicted limb image prediction;

[0237] The first visibility information is displayed at the second viewing angle, and predicts the visibility of the sample object predicted in the limb image at the first viewing angle;

[0238] Step 750: Obtain second visibility information of the sample object;

[0239] The second visibility information is displayed at the second viewing angle, and is the visibility of the sample object in the first sample image observed at the first viewing angle;

[0240] Step 760: Based on the difference between the first visibility information and the second visibility information, perform adversarial training on the discriminant network or the prediction sub-model of the limb part image prediction model to obtain a trained limb part image prediction model;

[0241] The prediction sub-model includes a feature encoding network and a decoding network. Exemplarily, in the adversarial training process, the network parameters of the discriminant network or the prediction sub-model can be alternately trained to obtain a trained limb image prediction model; Exemplarily, the prediction sub-model in the trained limb image prediction model is the limb image prediction model in the method for using the limb image prediction model described above.

[0242] In summary, the method provided in this embodiment obtains the predicted feature representation of the sample query point on the sample object by performing image encoding on the first sample image, and based on the symmetry of the sample object in the physiological structure, supplements the feature representation of the sample query point on the first limb with the feature representation on the second limb, thereby expanding the dimension of feature extraction of the sample query point; trains the discriminant network or prediction sub-model of the limb part image prediction model in an adversarial manner, thereby improving the ability to generate predicted limb images from a second perspective; and improves the effect of rendering the limb image of the limb object from a second perspective.

[0243] In one implementation, the limb part image prediction model shown in this application is tested based on the public data set Interhand2.6M data set as an example for illustration. By comparing the limb part image prediction model provided in this application, the non-human primate (Non-Human Primates, NHP) model in the related technology, and the keypoint variational autoencoder-neural radiance field reconstruction network (KeypointNeRF) model, the following Table 1 is obtained.

[0244] Table 1

[0245] method PSNR SSIM LPIPS KeypointNeRF Model 23.49 0.82 0.27 NHP Model 23.63 0.83 0.33 Limb Part Image Prediction Model 24.62 0.85 0.21

[0246] It can be seen that the limb part image prediction model provided by the present application is superior to the KeypointNeRF model and NHP model in the related art in terms of three evaluation indicators: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS);

[0247] Specifically, the limb image prediction model improves the PSNR parameter from the best 23.63 in related technologies to 24.62, and the SSIM parameter from the best 0.83 in related technologies to 0.85. The LPIPS parameter is further reduced from the best 0.27 in related technologies to 0.21. The limb image prediction model provides better prediction capabilities for limb images. Taking the hands as an example, it better preserves the hand structure and details.

[0248] Fig.13 FIG. 1 shows a schematic diagram of a two-hand image provided by an exemplary embodiment of the present application. Fig.13 Including sub-image (a), sub-image (b) and sub-image (c); the prediction results of the limb part image prediction model provided by this application, the NHP model in the related art, and the KeypointNeRF model for the two-handed images of three postures are provided. In each sub-image, compared with the NHP model and the KeypointNeRF model, the two-handed images predicted by the limb part image prediction model provided by this application have better image quality and fewer artifacts.

[0249] In sub-figure (a), the texture clarity of the two-hand image predicted by the limb part image prediction model at the fingertips and palm lines is better than that of the NHP model and KeypointNeRF model in the related art. In sub-figure (b), the texture clarity of the two-hand image predicted by the limb part image prediction model at the thumb tip, the base of the finger, and the boundary of the non-hand area is better than that of the NHP model and KeypointNeRF model in the related art. In sub-figure (c), the texture clarity of the two-hand image predicted by the limb part image prediction model at the joints on the fingers, the base of the finger, and the boundary of the non-hand area is better than that of the NHP model and KeypointNeRF model in the related art.

[0250] Fig.14 FIG. 1 shows a schematic diagram of a two-hand image provided by an exemplary embodiment of the present application. Fig.14 including sub-graph (d), sub-graph (e), sub-graph (f) and sub-graph (g); Fig.14 The prediction results of the limb part image prediction model provided in this application, the NHP model in the related technology, and the KeypointNeRF model for images of both hands in three postures under large angle changes (for example, the rotation angle between the first perspective and the second perspective is greater than 30 degrees) are provided.

[0251] In sub-figures (d) to (g), the two-hand images predicted by the NHP model and KeypointNeRF model in the related technologies all have blurred fingers and distorted finger postures to varying degrees, which do not conform to the physiological structure. The limb image prediction model provided in the present application avoids the above problems and predicts clear two-hand images.

[0252] Fig.15 FIG. 1 shows a schematic diagram of a two-hand image provided by an exemplary embodiment of the present application. Fig.15 including sub-graph (h), sub-graph (i), sub-graph (j) and sub-graph (k); Fig.15 The prediction results of the limb part image prediction model provided by the present application, the NHP model in the related art, and the KeypointNeRF model for images of both hands in three postures in the presence of occlusion are provided.

[0253] In sub-images (h) to (k), the two-hand images predicted by the NHP model and KeypointNeRF model in the related technologies all have afterimages in the occluded parts and serious degradation of image quality to varying degrees. The limb image prediction model provided in the present application avoids the above problems and predicts clear two-hand images.

[0254] Those skilled in the art will appreciate that the above embodiments may be implemented independently, or the above embodiments may be freely combined to form new embodiments to implement the method for using the limb part image prediction model of the present application, and / or the method for training the limb part image prediction model.

[0255] Fig.16 The structural block diagram of a device for using a limb part image prediction model provided by an exemplary embodiment of the present application is shown. The limb part image prediction model includes a feature encoding network and a decoding network; the device includes:

[0256] A first acquisition module 910 is used to acquire a first limb image, where the first limb image is image information of a limb object at a first viewing angle, where the limb object includes a first limb and a second limb, and the first limb and the second limb are symmetrical in physiological structure;

[0257] A first processing module 920 is used to call the feature encoding network to perform image encoding on the first limb image, extract at least two fused feature representations corresponding to at least two query points on the limb object, wherein the nth fused feature representation corresponding to the nth query point of the at least two query points is used to indicate a feature representation of an image region that is symmetrical in physiological structure, and the nth fused feature representation is used to supplement the feature representation of the nth query point on the first limb based on the symmetry of the limb object through the feature representation on the second limb;

[0258] The first rendering module 930 is used to call the decoding network, perform feature decoding on the at least two fused feature representations, and render to obtain a second limb image; the generated area corresponding to the nth query point in the second limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the second limb image is the image information of the limb object under a second perspective, and the second perspective is different from the first perspective.

[0259] In an optional implementation of this embodiment, the nth fused feature representation corresponding to the nth query point includes a first region feature and a second region feature;

[0260] The first processing module 920 is further configured to:

[0261] Calling the feature encoding network to extract the first region feature of the surrounding region of the nth query point on the first limb from the first limb image; and extracting the second region feature of the symmetrical region on the second limb from the first limb image, wherein the symmetrical region and the surrounding region of the nth query point are symmetrical in physiological structure;

[0262] The feature encoding network is called to concatenate the first region feature and the second region feature to obtain an nth fused feature representation corresponding to the nth query point, thereby obtaining the at least two fused feature representations.

[0264] In an optional implementation of this embodiment, the first processing module 920 is further configured to:

[0265] Calling the feature encoding network to predict the first regional feature and the second regional feature to obtain a first weight corresponding to the first regional feature and a second weight corresponding to the second regional feature;

[0266] Based on the product of the first weight and the first regional feature and the product of the second weight and the second regional feature, an nth fused feature representation corresponding to the nth query point is concatenated to obtain the at least two fused feature representations.

[0267] In an optional implementation of this embodiment, the feature encoding network includes an encoding subnetwork and a fusion subnetwork;

[0268] The first processing module 920 is further configured to:

[0269] Calling the encoding subnetwork to perform image encoding on the first limb image, and extracting image feature representation of the first limb image in the hidden layer space;

[0270] The fusion sub-network is called to extract the first region features of the surrounding area of ​​the nth query point on the first limb in the image feature representation; and the second region features of the symmetrical area on the second limb are extracted.

[0271] In an optional implementation of this embodiment, the limb part image prediction model further includes a structure reconstruction network; and the first processing module 920 is further used for:

[0272] Calling the structure reconstruction network to predict the first limb image to obtain a three-dimensional structure grid of the limb object in the first limb image;

[0273] calling the structure reconstruction network, determining a grid point in the three-dimensional structure grid that is adjacent to the nth query point as a first grid point, and determining a second grid point corresponding to the first grid point, wherein a relative position of the first grid point on the first limb is the same as a relative position of the second grid point on the second limb;

[0274] The structure reconstruction network is called to determine the surrounding area of ​​the nth query point based on the first grid points, and to determine the symmetric area based on the second grid points.

[0275] In an optional implementation of this embodiment, the first processing module 920 is further configured to:

[0276] Calling the structure reconstruction network to determine first position information of the first grid point mapped on the first limb image; and determining a peripheral area of ​​the nth query point with the first position information as the center;

[0277] The structure reconstruction network is called to determine second position information of the second grid point mapped on the first limb image; and the symmetric area is determined with the second position information as the center.

[0278] In an optional implementation of this embodiment, the nth fused feature representation further includes a spatial feature, and the spatial feature is used to indicate the spatial depth of the nth query point relative to an interest point of the limb object, where the interest point is a point on the limb object that has visual significance or is associated with an activity of the limb object;

[0279] The first acquisition module 910 is further used to acquire the interest position of the interest point on the three-dimensional structure grid and the query position of the nth query point on the three-dimensional structure grid;

[0280] The first processing module 920 is further used to call the feature encoding network to construct the spatial feature according to the interest position and the query position; and to add the spatial feature to the fused feature representation.

[0281] In an optional implementation of this embodiment, the fused feature representation is used to indicate a texture feature and / or a geometric structure feature of the limb object.

[0282] In an optional implementation of this embodiment, when the fused feature representation is used to indicate the texture feature of the limb object, the nth fused feature representation also includes a global texture feature;

[0283] The first processing module 920 is further configured to:

[0284] The feature encoding network is called to perform image encoding on the first limb image to extract global texture features; and the global texture features are added to the fused feature representation, where the global texture features are global features of the limb object in the first limb image.

[0285] In an optional implementation of this embodiment, the global texture feature includes a first texture feature corresponding to the first limb and a second texture feature corresponding to the second limb; the first texture feature and the second texture feature have independent weight information corresponding to each other in the fused feature representation;

[0286] The first processing module 920 is further configured to:

[0287] Based on the position of the first limb, calling the feature encoding network to extract the first texture feature in the first limb image;

[0288] Based on the position of the second limb, calling the feature encoding network to extract the second texture feature in the first limb image;

[0289] The feature encoding network is called to predict weight information corresponding to the first texture feature and the second texture feature, and the first texture feature and the second texture feature, as well as the one-to-one weight information of the first texture feature and the second texture feature, are added to the fused feature representation.

[0290] In an optional implementation of this embodiment, the limb object includes but is not limited to at least one of hands, arms, feet, legs, face, head, and body trunk.

[0291] Fig.17 The structural block diagram of a training device for a limb part image prediction model provided by an exemplary embodiment of the present application is shown. The limb part image prediction model includes a feature encoding network and a decoding network; the device includes:

[0292] A second acquisition module 940 is used to acquire a sample information pair, wherein the sample information pair includes a first sample image and a second sample image, wherein the first sample image is image information of a sample object at a first viewing angle, and the second sample image is image information of the sample object at a second viewing angle, wherein the first viewing angle and the second viewing angle are different; the sample object includes a first limb and a second limb, and the first limb and the second limb are symmetrical in physiological structure;

[0293] A second processing module 950 is used to call the feature encoding network to perform image encoding on the first sample image, extract at least two predicted feature representations corresponding to at least two sample query points on the sample object, the nth predicted feature representation corresponding to the nth sample query point among the at least two sample query points is used to indicate a feature representation of a sample region that is symmetrical in physiological structure, and the nth predicted feature representation is used to supplement the feature representation of the nth sample query point on the first limb through the feature representation on the second limb based on the symmetry of the sample object;

[0294] A second rendering module 960 is used to call the decoding network, perform feature decoding on the at least two predicted feature representations, and render a predicted limb image, wherein the generated area corresponding to the nth sample query point in the predicted limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the predicted limb image is the image information of the sample object obtained by prediction at the second viewing angle;

[0295] The first training module 970 is used to train the limb part image prediction model based on the difference between the predicted limb image and the second sample image to obtain a trained limb part image prediction model.

[0296] In an optional implementation of this embodiment, the nth prediction feature representation corresponding to the nth sample query point includes a first region feature and a second region feature;

[0297] The second processing module 950 is further used for:

[0298] Calling the feature encoding network to extract the first region feature of the surrounding region of the nth sample query point on the first limb from the first sample image; and extracting the second region feature of the symmetrical region on the second limb from the first sample image, wherein the symmetrical region and the surrounding region of the nth sample query point are symmetrical in physiological structure;

[0299] The feature encoding network is called to concatenate the first region feature and the second region feature to obtain an nth predicted feature representation corresponding to the nth sample query point, thereby obtaining the at least two predicted feature representations.

[0300] In an optional implementation of this embodiment, the second processing module 950 is further configured to:

[0301] Calling the feature encoding network to predict the first regional feature and the second regional feature to obtain a first weight corresponding to the first regional feature and a second weight corresponding to the second regional feature;

[0302] Based on the product of the first weight and the first regional feature and the product of the second weight and the second regional feature, an nth predicted feature representation corresponding to the nth query point is concatenated to obtain the at least two predicted feature representations.

[0303] In an optional implementation of this embodiment, the limb part image prediction model further includes a discriminant network;

[0304] The second processing module 950 is further configured to call the discriminant network to predict and obtain first visibility information based on the predicted limb image, wherein the first visibility information is displayed under the second viewing angle, and is a visibility condition of the sample object predicted in the predicted limb image observed under the first viewing angle;

[0305] The second acquisition module 940 is further used to acquire second visibility information of the sample object, where the second visibility information is displayed at the second viewing angle, and is a visibility condition of the sample object in the first sample image observed at the first viewing angle;

[0306] The first training module 970 is also used to supplement the training of the discriminant network or prediction sub-model of the limb part image prediction model based on the difference between the first visibility information and the second visibility information, and the prediction sub-model includes the feature encoding network and the decoding network.

[0307] Fig.18 The structural block diagram of a training device for a limb part image prediction model provided by an exemplary embodiment of the present application is shown. The limb part image prediction model includes a feature encoding network, a decoding network and a discriminant network; the device includes:

[0308] A third acquisition module 1010 is used to acquire a first sample image, where the first sample image is image information of a sample object at a first viewing angle, where the sample object includes a first limb and a second limb, and the first limb and the second limb are symmetrical in physiological structure;

[0309] A third processing module 1020 is used to call the feature encoding network to perform image encoding on the first sample image, extract at least two predicted feature representations corresponding to at least two sample query points on the sample object, wherein the nth predicted feature representation corresponding to the nth sample query point among the at least two sample query points is used to indicate a feature representation of a sample region that is symmetrical in physiological structure, and the nth predicted feature representation is used to supplement the feature representation of the nth sample query point on the first limb through the feature representation on the second limb based on the symmetry of the sample object;

[0310] A third rendering module 1030 is used to call the decoding network, perform feature decoding on the at least two predicted feature representations, and render a predicted limb image, wherein the generated area corresponding to the nth sample query point in the predicted limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the predicted limb image is the image information of the sample object obtained by prediction at a second viewing angle, and the first viewing angle and the second viewing angle are different;

[0311] The third processing module 1020 is further configured to call the discriminant network to predict and obtain first visibility information based on the predicted limb image, wherein the first visibility information is displayed under the second viewing angle, and is a visual condition of the sample object predicted in the predicted limb image observed under the first viewing angle;

[0312] The third acquisition module 1010 is further used to acquire second visibility information of the sample object, where the second visibility information is displayed at the second viewing angle, and is a visibility condition of the sample object in the first sample image observed at the first viewing angle;

[0313] The second training module 1040 is used to perform adversarial training on the discriminant network or prediction sub-model of the limb part image prediction model based on the difference between the first visibility information and the second visibility information to obtain a trained limb part image prediction model, wherein the prediction sub-model includes the feature encoding network and the decoding network.

[0314] One point that needs to be explained is that the device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to implement its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0315] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method; the technical effects achieved by each module performing operations are the same as those in the embodiment of the method, and will not be elaborated here.

[0316] An embodiment of the present application also provides a computer device, which includes: a processor and a memory, wherein a computer program is stored in the memory; the processor is used to execute the computer program in the memory to implement the method of using the limb part image prediction model provided in the above-mentioned method embodiments, and / or the training method of the limb part image prediction model.

[0317] Optionally, the computer device is a server. Fig.19 It is a structural block diagram of a server provided by an exemplary embodiment of the present application.

[0318] Typically, the server 2300 includes: a processor 2301 and a memory 2302 .

[0319] The processor 2301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 2301 may be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 2301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 2301 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 2301 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0320] The memory 2302 may include one or more computer-readable storage media, which may be non-transitory. The memory 2302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 2302 is used to store at least one instruction, which is used to be executed by the processor 2301 to implement the method for using the limb part image prediction model provided in the method embodiment of the present application, and / or the training method of the limb part image prediction model.

[0321] In some embodiments, the server 2300 may also optionally include: an input interface 2303 and an output interface 2304. The processor 2301, the memory 2302, the input interface 2303, and the output interface 2304 may be connected via a bus or a signal line. Each peripheral device may be connected to the input interface 2303 and the output interface 2304 via a bus, a signal line, or a circuit board. The input interface 2303 and the output interface 2304 may be used to connect at least one peripheral device related to input / output (I / O) to the processor 2301 and the memory 2302. In some embodiments, the processor 2301, the memory 2302, the input interface 2303, and the output interface 2304 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 2301, the memory 2302, the input interface 2303, and the output interface 2304 may be implemented on a separate chip or circuit board, which is not limited in the embodiments of the present application.

[0322] Those skilled in the art will appreciate that the structure shown above does not constitute a limitation on the server 2300 , and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0323] In an exemplary embodiment, a chip is also provided, which includes a programmable logic circuit and / or program instructions. When the chip is run on a computer device, it is used to implement the method of using the limb part image prediction model described in the above aspects, and / or the training method of the limb part image prediction model.

[0324] In an exemplary embodiment, a computer program product is also provided, the computer program product includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor reads and executes the computer instructions from the computer-readable storage medium to implement the method for using the limb part image prediction model provided by the above-mentioned various method embodiments, and / or the method for training the limb part image prediction model.

[0325] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores a computer program, and the computer program is loaded and executed by a processor to implement the method of using the limb part image prediction model provided by the above-mentioned method embodiments, and / or the training method of the limb part image prediction model.

[0326] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0327] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0328] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for using a limb part image prediction model, characterized in that: The limb part image prediction model includes a feature encoding network and a decoding network; the method includes: Acquire a first limb image, where the first limb image is image information of a limb object at a first viewing angle, the limb object includes a first limb and a second limb, and the first limb and the second limb are symmetrical in physiological structure; Calling the feature encoding network to perform image encoding on the first limb image, extracting at least two fused feature representations corresponding to at least two query points on the limb object, wherein an nth fused feature representation corresponding to an nth query point among the at least two query points is used to indicate a feature representation of an image region that is symmetrical in physiological structure, and the nth fused feature representation is used to supplement the feature representation of the nth query point on the first limb based on the symmetry of the limb object through the feature representation on the second limb; The decoding network is called to perform feature decoding on the at least two fused feature representations, and a second limb image is rendered; the generated area corresponding to the nth query point in the second limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the second limb image is image information of the limb object under a second perspective, and the second perspective is different from the first perspective.

2. The method according to claim 1, characterized in that The nth fused feature representation corresponding to the nth query point includes a first region feature and a second region feature; The calling of the feature encoding network to perform image encoding on the first limb image, and extracting at least two fused feature representations corresponding to at least two query points on the limb object, comprises: Calling the feature encoding network to extract the first region feature of the surrounding region of the nth query point on the first limb from the first limb image; and extracting the second region feature of the symmetrical region on the second limb from the first limb image, wherein the symmetrical region and the surrounding region of the nth query point are symmetrical in physiological structure; The feature encoding network is called to concatenate the first region feature and the second region feature to obtain an nth fused feature representation corresponding to the nth query point, thereby obtaining the at least two fused feature representations.

3. The method according to claim 2, characterized in that The calling of the feature encoding network, splicing the first regional feature and the second regional feature to obtain the nth fused feature representation corresponding to the nth query point, and obtaining the at least two fused feature representations, includes: Calling the feature encoding network to predict the first regional feature and the second regional feature to obtain a first weight corresponding to the first regional feature and a second weight corresponding to the second regional feature; Based on the product of the first weight and the first regional feature and the product of the second weight and the second regional feature, an nth fused feature representation corresponding to the nth query point is concatenated to obtain the at least two fused feature representations.

4. The method according to claim 2, characterized in that: The feature encoding network includes an encoding subnetwork and a fusion subnetwork; The feature encoding network is called to extract the first region feature of the surrounding region of the nth query point on the first limb from the first limb image; And extracting the second region feature of the symmetrical region on the second limb from the first limb image comprises: Calling the encoding subnetwork to perform image encoding on the first limb image, and extracting image feature representation of the first limb image in the hidden layer space; The fusion sub-network is called to extract the first region features of the surrounding area of ​​the nth query point on the first limb in the image feature representation; and the second region features of the symmetrical area on the second limb are extracted.

5. The method according to claim 2, characterized in that: The limb part image prediction model also includes a structure reconstruction network; the method also includes: Calling the structure reconstruction network to predict the first limb image to obtain a three-dimensional structure grid of the limb object in the first limb image; calling the structure reconstruction network, determining a grid point in the three-dimensional structure grid that is adjacent to the nth query point as a first grid point, and determining a second grid point corresponding to the first grid point, wherein a relative position of the first grid point on the first limb is the same as a relative position of the second grid point on the second limb; The structure reconstruction network is called to determine the surrounding area of ​​the nth query point based on the first grid points, and to determine the symmetric area based on the second grid points.

6. The method according to claim 5, characterized in that The calling of the structure reconstruction network, determining the surrounding area of ​​the nth query point based on the first grid points, and determining the symmetric area based on the second grid points, comprises: Calling the structure reconstruction network to determine first position information of the first grid point mapped on the first limb image; and determining a peripheral area of ​​the nth query point with the first position information as the center; The structure reconstruction network is called to determine second position information of the second grid point mapped on the first limb image; and the symmetric area is determined with the second position information as the center.

7. The method according to claim 5, characterized in that The nth fused feature representation also includes a spatial feature, wherein the spatial feature is used to indicate the spatial depth of the nth query point relative to an interest point of the limb object, wherein the interest point is a point on the limb object that is visually significant or associated with an activity of the limb object; The method further comprises: Acquire an interest position of the interest point on the three-dimensional structure grid and a query position of the nth query point on the three-dimensional structure grid; The feature encoding network is called to construct the spatial feature according to the interest position and the query position; and the spatial feature is added to the fused feature representation.

8. The method according to any one of claims 1 to 7, characterized in that: The fused feature representation is used to indicate a texture feature and / or a geometric structure feature of the limb object.

9. The method according to any one of claims 1 to 7, characterized in that: In the case where the fused feature representation is used to indicate a texture feature of the limb object, the nth fused feature representation also includes a global texture feature; The method further comprises: The feature encoding network is called to perform image encoding on the first limb image to extract global texture features; and the global texture features are added to the fused feature representation, where the global texture features are global features of the limb object in the first limb image.

10. The method according to claim 9, characterized in that The global texture feature includes a first texture feature corresponding to the first limb and a second texture feature corresponding to the second limb; the first texture feature and the second texture feature have independent weight information corresponding to each other in the fused feature representation; calling the feature encoding network to perform image encoding on the first limb image to extract global texture features; And adding the global texture feature to the fused feature representation comprises: Based on the position of the first limb, calling the feature encoding network to extract the first texture feature in the first limb image; Based on the position of the second limb, calling the feature encoding network to extract the second texture feature in the first limb image; The feature encoding network is called to predict weight information corresponding to the first texture feature and the second texture feature, and the first texture feature and the second texture feature, as well as the one-to-one weight information of the first texture feature and the second texture feature, are added to the fused feature representation.

11. A training method for a limb part image prediction model, characterized in that: The limb part image prediction model includes a feature encoding network and a decoding network; the method includes: Acquire a sample information pair, the sample information pair comprising a first sample image and a second sample image, the first sample image being image information of a sample object at a first viewing angle, the second sample image being image information of the sample object at a second viewing angle, the first viewing angle and the second viewing angle being different; the sample object comprising a first limb and a second limb, the first limb and the second limb being symmetrical in physiological structure; Calling the feature encoding network to perform image encoding on the first sample image, extracting at least two predicted feature representations corresponding to at least two sample query points on the sample object, wherein an nth predicted feature representation corresponding to an nth sample query point among the at least two sample query points is used to indicate a feature representation of a sample region that is symmetrical in physiological structure, and the nth predicted feature representation is used to supplement the feature representation of the nth sample query point on the first limb through the feature representation on the second limb based on the symmetry of the sample object; Calling the decoding network, performing feature decoding on the at least two predicted feature representations, and rendering to obtain a predicted limb image, wherein the generated region corresponding to the nth sample query point in the predicted limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the predicted limb image is the predicted image information of the sample object under the second viewing angle; Based on the difference between the predicted limb image and the second sample image, the limb part image prediction model is trained to obtain a trained limb part image prediction model.

12. The method according to claim 11, characterized in that The nth prediction feature representation corresponding to the nth sample query point includes a first region feature and a second region feature; The calling the feature encoding network to perform image encoding on the first sample image, and extracting at least two prediction feature representations corresponding to at least two sample query points on the sample object, comprises: Calling the feature encoding network to extract the first region feature of the surrounding region of the nth sample query point on the first limb from the first sample image; and extracting the second region feature of the symmetrical region on the second limb from the first sample image, wherein the symmetrical region and the surrounding region of the nth sample query point are symmetrical in physiological structure; The feature encoding network is called to concatenate the first region feature and the second region feature to obtain an nth predicted feature representation corresponding to the nth sample query point, thereby obtaining the at least two predicted feature representations.

13. The method according to claim 11 or 12, characterized in that: The limb part image prediction model also includes a discriminant network; The method further comprises: Calling the discriminant network to predict and obtain first visibility information based on the predicted limb image, wherein the first visibility information is displayed under the second viewing angle, and is a visual condition of a sample object predicted in the predicted limb image observed under the first viewing angle; Acquire second visibility information of the sample object, where the second visibility information is displayed at the second viewing angle, and is a visual condition of the sample object in the first sample image observed at the first viewing angle; Based on the difference between the first visibility information and the second visibility information, the discriminant network or the prediction sub-model of the limb part image prediction model is supplementarily trained, and the prediction sub-model includes the feature encoding network and the decoding network.

14. A training method for a limb part image prediction model, characterized in that: The limb part image prediction model includes a feature encoding network, a decoding network and a discriminant network; the method includes: Acquire a first sample image, where the first sample image is image information of a sample object at a first viewing angle, where the sample object includes a first limb and a second limb, and the first limb and the second limb are symmetrical in physiological structure; Calling the feature encoding network to perform image encoding on the first sample image, extracting at least two predicted feature representations corresponding to at least two sample query points on the sample object, wherein an nth predicted feature representation corresponding to an nth sample query point among the at least two sample query points is used to indicate a feature representation of a sample region that is symmetrical in physiological structure, and the nth predicted feature representation is used to supplement the feature representation of the nth sample query point on the first limb through the feature representation on the second limb based on the symmetry of the sample object; Calling the decoding network, performing feature decoding on the at least two predicted feature representations, and rendering to obtain a predicted limb image, wherein the generated region corresponding to the nth sample query point in the predicted limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the predicted limb image is image information of the sample object obtained by prediction at a second viewing angle, and the first viewing angle and the second viewing angle are different; Calling the discriminant network to predict and obtain first visibility information based on the predicted limb image, wherein the first visibility information is displayed under the second viewing angle, and is a visual condition of a sample object predicted in the predicted limb image observed under the first viewing angle; Acquire second visibility information of the sample object, where the second visibility information is displayed at the second viewing angle, and is a visual condition of the sample object in the first sample image observed at the first viewing angle; Based on the difference between the first visibility information and the second visibility information, adversarial training is performed on the discriminant network or the prediction sub-model of the limb part image prediction model to obtain a trained limb part image prediction model, and the prediction sub-model includes the feature encoding network and the decoding network.

15. A device for using a limb part image prediction model, characterized in that: The limb part image prediction model includes a feature encoding network and a decoding network; the device includes: A first acquisition module is used to acquire a first limb image, where the first limb image is image information of a limb object at a first viewing angle, where the limb object includes a first limb and a second limb, and the first limb and the second limb are symmetrical in physiological structure; A first processing module is configured to call the feature encoding network to perform image encoding on the first limb image, extract at least two fused feature representations corresponding to at least two query points on the limb object, wherein an nth fused feature representation corresponding to an nth query point among the at least two query points is used to indicate a feature representation of an image region that is symmetrical in physiological structure, and the nth fused feature representation is used to supplement the feature representation of the nth query point on the first limb based on the symmetry of the limb object through the feature representation on the second limb; The first rendering module is used to call the decoding network, perform feature decoding on the at least two fused feature representations, and render a second limb image; the generated area corresponding to the nth query point in the second limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the second limb image is image information of the limb object under a second perspective, and the second perspective is different from the first perspective.

16. A training device for a limb part image prediction model, characterized in that: The limb part image prediction model includes a feature encoding network and a decoding network; the device includes: a second acquisition module, configured to acquire a sample information pair, the sample information pair comprising a first sample image and a second sample image, the first sample image being image information of a sample object at a first viewing angle, the second sample image being image information of the sample object at a second viewing angle, the first viewing angle being different from the second viewing angle; the sample object comprising a first limb and a second limb, the first limb and the second limb being symmetrical in physiological structure; a second processing module, configured to call the feature encoding network to perform image encoding on the first sample image, extract at least two predicted feature representations corresponding to at least two sample query points on the sample object, wherein an nth predicted feature representation corresponding to an nth sample query point among the at least two sample query points is used to indicate a feature representation of a sample region that is symmetrical in physiological structure, and the nth predicted feature representation is used to supplement the feature representation of the nth sample query point on the first limb through the feature representation on the second limb based on the symmetry of the sample object; A second rendering module is used to call the decoding network, perform feature decoding on the at least two predicted feature representations, and render a predicted limb image, wherein the generated area corresponding to the nth sample query point in the predicted limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the predicted limb image is the predicted image information of the sample object under the second viewing angle; The first training module is used to train the limb part image prediction model based on the difference between the predicted limb image and the second sample image to obtain a trained limb part image prediction model.

17. A training device for a limb part image prediction model, characterized in that: The limb part image prediction model includes a feature encoding network, a decoding network and a discriminant network; the device includes: A third acquisition module is used to acquire a first sample image, where the first sample image is image information of a sample object at a first viewing angle, where the sample object includes a first limb and a second limb, and the first limb and the second limb are symmetrical in physiological structure; a third processing module, configured to call the feature encoding network to perform image encoding on the first sample image, extract at least two predicted feature representations corresponding to at least two sample query points on the sample object, wherein an nth predicted feature representation corresponding to an nth sample query point among the at least two sample query points is used to indicate a feature representation of a sample region that is symmetrical in physiological structure, and the nth predicted feature representation is used to supplement the feature representation of the nth sample query point on the first limb through the feature representation on the second limb based on the symmetry of the sample object; a third rendering module, configured to call the decoding network, perform feature decoding on the at least two predicted feature representations, and render a predicted limb image, wherein the generated region corresponding to the nth sample query point in the predicted limb image is obtained by decoding after feature supplementation based on the feature representation on the second limb, and the predicted limb image is image information of the sample object obtained by prediction at a second viewing angle, and the first viewing angle and the second viewing angle are different; The third processing module is further used to call the discriminant network to predict and obtain first visibility information based on the predicted limb image, wherein the first visibility information is displayed under the second viewing angle, and is a visual condition of the sample object predicted in the predicted limb image observed under the first viewing angle; The third acquisition module is further used to acquire second visibility information of the sample object, where the second visibility information is displayed at the second viewing angle, and is a visibility condition of the sample object in the first sample image observed at the first viewing angle; The second training module is used to perform adversarial training on the discriminant network or prediction sub-model of the limb part image prediction model based on the difference between the first visibility information and the second visibility information to obtain a trained limb part image prediction model, wherein the prediction sub-model includes the feature encoding network and the decoding network.

18. A computer device, characterized in that: The computer device includes: a processor and a memory, wherein at least one program is stored in the memory; the processor is used to execute the at least one program in the memory to implement the method for using the limb part image prediction model as described in any one of claims 1 to 10, and / or the method for training the limb part image prediction model as described in any one of claims 11 to 14.

19. A computer-readable storage medium, characterized in that: The readable storage medium stores executable instructions, which are loaded and executed by the processor to implement the method for using the limb part image prediction model as described in any one of claims 1 to 10, and / or the method for training the limb part image prediction model as described in any one of claims 11 to 14.

20. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor reads and executes the computer instructions from the computer-readable storage medium to implement the method for using the limb part image prediction model as described in any one of claims 1 to 10, and / or the method for training the limb part image prediction model as described in any one of claims 11 to 14.

Images