Network training and object attitude estimation method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a network training and object attitude estimation method and device, equipment and a storage medium, and the method comprises the steps: obtaining a first sample image containing a preset object and a position mark of a key point of each object in the first sample image; based on the position label of each object key point, shielding at least one image sub-block in the first sample image to obtain a shielded image; utilizing a to-be-trained visual converter network to perform feature extraction on the occlusion image to obtain a first visual mark sequence; using a first decoder network to restore the occluded image sub-blocks in the occluded image based on the first visual mark sequence to obtain a restored image; and updating network parameters of the visual converter network at least once based on a first loss value between the first sample image and the restored image to obtain the trained visual converter network.

Description

Technical Field

[0001] The present disclosure relates to, but is not limited to, the field of computer vision technology, and in particular to a network training and object posture estimation method, device, equipment, and storage medium. Background Art

[0002] As a very important and challenging problem in the field of computer vision, object pose estimation has been widely used in many fields. In the related art, the Vision Transformer (ViT) network can be used to implement the object pose estimation task. However, in the object pose estimation scheme of the related art, on the one hand, a large amount of additional pre-training sample data is required to pre-train the vision transformer network; on the other hand, there is a large deviation between the upstream pre-training task and the downstream object pose estimation task, resulting in low accuracy of the object pose estimation task implemented by the trained vision transformer network. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure at least provide a network training and object posture estimation method, device, equipment, and storage medium.

[0004] The technical solution of the embodiment of the present disclosure is implemented as follows:

[0005] The present disclosure provides a network training method, the method comprising:

[0006] Acquire a first sample image containing a preset object and a position annotation of a key point of each object in the first sample image;

[0007] Based on the position annotation of each object key point, performing occlusion processing on at least one image sub-block in the first sample image to obtain an occluded image;

[0008] Using the visual converter network to be trained, extracting features from the occluded image to obtain a first visual marker sequence;

[0009] Using a first decoder network, based on the first visual marker sequence, restore the occluded image sub-block in the occluded image to obtain a restored image;

[0010] Based on a first loss value between the first sample image and the restored image, the network parameters of the visual converter network are updated at least once to obtain the trained visual converter network.

[0011] The present disclosure provides a method for estimating an object posture, the method comprising:

[0012] Acquire a first image including a preset object and a set fourth key point mark sequence, wherein each key point mark in the fourth key point mark sequence corresponds to an object key point of the preset object;

[0013] Using a trained visual converter network, extracting features from the first image to obtain a third visual tag sequence; wherein the visual converter network is trained using the above network training method;

[0014] Using the trained second prediction network, based on the fourth key point label sequence and the third visual label sequence, the positions of the key points of the object in the first image are predicted to obtain the predicted position of each key point of the object in the first image.

[0015] The present disclosure provides a network training device, the device comprising:

[0016] A first acquisition module, used to acquire a first sample image containing a preset object and a position annotation of a key point of each object in the first sample image;

[0017] an occlusion module, configured to perform occlusion processing on at least one image sub-block in the first sample image based on the position annotation of each key point of the object, so as to obtain an occlusion image;

[0018] A first extraction module, configured to extract features from the occluded image using a visual converter network to be trained, to obtain a first visual marker sequence;

[0019] a restoration module, configured to perform restoration processing on the occluded image sub-blocks in the occluded image based on the first visual marker sequence by using the first decoder network to obtain a restored image;

[0020] The first updating module is used to update the network parameters of the visual converter network at least once based on a first loss value between the first sample image and the restored image to obtain the trained visual converter network.

[0021] The present disclosure provides an object posture estimation device, the device comprising:

[0022] A third acquisition module is used to acquire a first image containing a preset object and a set fourth key point mark sequence, wherein each key point mark in the fourth key point mark sequence corresponds to an object key point of the preset object;

[0023] A third extraction module, configured to extract features from the first image using a trained visual converter network to obtain a third visual tag sequence; wherein the visual converter network is trained using the above-mentioned network training method;

[0024] The second prediction module is used to use the trained second prediction network to predict the position of the key points of the object in the first image based on the fourth key point label sequence and the third visual label sequence, so as to obtain the predicted position of each key point of the object in the first image.

[0025] An embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0026] An embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, some or all of the steps in the above method are implemented.

[0027] An embodiment of the present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented.

[0028] In an embodiment of the present disclosure, a first sample image including a preset object and a position annotation of each key point of the object in the first sample image are obtained; based on the position annotation of each key point of the object, at least one image sub-block in the first sample image is occluded to obtain an occluded image; using a visual converter network to be trained, features are extracted from the occluded image to obtain a first visual marker sequence; using a first decoder network, occluded image sub-blocks in the occluded image are restored based on the first visual marker sequence to obtain a restored image; based on a first loss value between the first sample image and the restored image, network parameters of the visual converter network are updated at least once to obtain a trained visual converter network. In this way, on the one hand, since the occluded image is obtained by occluding at least one image sub-block in the first sample image based on the position annotation of each object key point, the network parameters of the visual converter network are updated at least once based on the first loss value between the first sample image and the restored image, which can guide the visual converter network to learn the ability to understand the semantics related to the object key points, so that the visual converter network can better extract features related to the object key points, so as to improve the accuracy of object pose estimation when the trained visual converter network is used to implement the object pose estimation task; on the other hand, the first sample image used for training the visual converter network is an image containing a preset object, and the first sample image has the position annotation of each object key point, and the sample image used for network training in the object pose estimation task is usually also an image containing a preset object and having the position annotation of each object key point. In this way, the same sample image can be used in the pre-training of the visual converter network and in the network training process after migrating the pre-trained visual converter network to the object pose estimation task, thereby reducing the demand for training sample data.

[0029] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0031] Figure 1 A schematic diagram of an implementation flow of a network training method provided in an embodiment of the present disclosure;

[0032] Figure 2 A schematic diagram of an implementation flow of a network training method provided in an embodiment of the present disclosure;

[0033] Figure 3A schematic diagram of an implementation flow of a network training method provided in an embodiment of the present disclosure;

[0034] Figure 4 A schematic diagram of an implementation flow of an object posture estimation method provided in an embodiment of the present disclosure;

[0035] Figure 5A A schematic diagram of an implementation flow of pre-training a visual converter network in a human pose estimation task using a first sample image provided by an embodiment of the present disclosure;

[0036] Figure 5B A schematic diagram of the effect of performing occlusion processing on at least one image sub-block in a first sample image in a network training method provided by an embodiment of the present disclosure;

[0037] Figure 5C A schematic diagram of the structure of a human body posture estimation model provided in an embodiment of the present disclosure;

[0038] Figure 6 A schematic diagram of the structure of a network training device provided in an embodiment of the present disclosure;

[0039] Figure 7 A schematic diagram of the structure of a human body posture estimation device provided by an embodiment of the present disclosure;

[0040] Figure 8 A hardware entity schematic diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present disclosure. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0042] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0043] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the present disclosure and are not intended to limit the present disclosure.

[0045] In order to better understand the network training method and the object posture recognition method provided by the embodiments of the present disclosure, the object posture estimation scheme used in the related art is first described below.

[0046] Object pose estimation refers to estimating the position of each key point of the target object in a given image (such as the nose / eyes / knees in the human body). Object pose estimation in related technologies is usually implemented using a solution based on a convolutional neural network. Due to the explosive application and broad prospects of visual converter networks in various fields of computer vision, some related technologies also consider applying visual converter networks to object pose estimation tasks. However, it is difficult to directly apply visual converter networks. There are two significant problems to consider: 1) The demand for training sample data is large: in addition to the training sample data required for network training in the object pose estimation task, a large amount of additional pre-training sample data is required to pre-train the visual converter network; 2) The problem of customized utilization of features: in the object pose estimation task, it is necessary to consider both the capture of visual features and the capture of the intrinsic correlation between the key points of the object. In response to these problems, in the object pose estimation solution in the related technology, specialized designs such as hierarchical features (inductive bias) are introduced in the converter structure of the visual converter network. However, the visual converter network with the introduction of inductive bias is not universal and cannot be applied to object pose estimation tasks in different business scenarios.

[0047] The embodiment of the present disclosure provides a network training method, which can be executed by a processor of a computer device. The computer device may refer to a server, a laptop, a tablet computer, a desktop computer, a smart TV, a set-top box, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device) and other devices with data processing capabilities. Figure 1 A schematic diagram of the implementation flow of a network training method provided in an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the method includes the following steps S101 to S105:

[0048] Step S101: obtaining a first sample image containing a preset object and a position annotation of a key point of each object in the first sample image.

[0049] Here, the preset object may include but is not limited to at least one of a human body, a cat, a dog, a robot, a human hand, a face, etc. The first sample image may be any suitable image containing at least one preset object. In implementation, the first sample image may be pre-captured using an image acquisition device, or may be generated using an image generation model, which is not limited in the embodiments of the present disclosure.

[0050] The object key points may be any suitable key points in the preset object for posture estimation. In implementation, the object key points may be predetermined according to the actual preset object. For example, in the case where the preset object is a human body, the object key points may include human body key points, such as at least one of facial features, neck, shoulder, elbow, hand, hip, knee, foot, etc.; in the case where the preset object is a human hand, the object key points may include hand key points, such as wrists, joints of each finger, etc.

[0051] The position annotation of each object key point in the first sample image can be used to annotate the position of each object key point in the first sample image. The position annotation of each object key point in the first sample image can be manually pre-annotated, or it can be automatically annotated after the object key point is recognized in the first sample image using a target detection algorithm. For example, in a human posture estimation task, the first sample image can be an image containing a human object, and the position annotation of each object key point in the first sample image can be the position of each human key point in the pre-annotated first sample image; in a hand gesture estimation task, the first sample image can be an image containing a hand object, and the position annotation of each object key point in the first sample image can be the position of each hand key point in the pre-annotated first sample image.

[0052] In some embodiments, a third sample image containing a preset object can be first obtained, and the preset object in the third sample image can be detected by a set object detector to obtain at least one sub-image containing the preset object, and each sub-image can be scaled to the same size. Each scaled sub-image can be used as the first sample image.

[0053] Step S102: Based on the position annotation of each object key point, perform occlusion processing on at least one image sub-block in the first sample image to obtain an occlusion image.

[0054] Here, at least one image sub-block to be occluded can be determined from the first sample image based on the position annotation of each object key point, and the at least one image sub-block to be occluded can be subjected to occlusion processing to obtain an occluded image. During implementation, any suitable method can be used to determine at least one image sub-block to be occluded from the first sample image based on the position annotation of each object key point according to actual conditions, and the embodiments of the present disclosure are not limited to this.

[0055] In some embodiments, based on the position annotation of each object key point, at least one image sub-block containing the object key point can be determined from the first sample image, and occlusion processing can be performed on the at least one image sub-block containing the object key point to obtain an occlusion image.

[0056] In some embodiments, the first sample image can be divided into multiple image sub-blocks; based on the position annotation of each object key point, the target image sub-block containing the object key point in the first sample image can be determined; based on the distance between each image sub-block and each target image sub-block, the occlusion probability of each image sub-block can be determined; based on the occlusion probability of each image sub-block, occlusion processing can be performed on at least one image sub-block in the first sample image to obtain an occluded image.

[0057] The occlusion processing of the image sub-block may include but is not limited to at least one of deleting the image sub-block in the first sample image, filling the image sub-block with a set color, filling the image sub-block with a set pattern, etc., and the embodiments of the present disclosure are not limited to this.

[0058] Step S103: using the visual converter network to be trained, extracting features from the occluded image to obtain a first visual marker sequence.

[0059] Here, the visual transformer network to be trained can be any suitable transformer-based feature extraction network. In implementation, the visual transformer network to be trained can be untrained or pre-trained, which is not limited in the embodiments of the present disclosure.

[0060] The first visual marker sequence may include at least one first visual marker, and each first visual marker may represent an image sub-block in the occluded image. In implementation, using the visual transformer network to be trained, the occluded image may be evenly divided into a plurality of image sub-blocks, and each image sub-block may be mapped to an embedding vector with a fixed dimension to generate a corresponding first visual marker.

[0061] Step S104: using a first decoder network, based on the first visual marker sequence, restore the occluded image sub-block in the occluded image to obtain a restored image.

[0062] Here, the first decoder network can be any suitable deep learning network, which can be used to predict the pixel values ​​of the occluded image sub-blocks in the occluded image based on the first visual marker sequence to obtain a restored image after occlusion restoration, which is not limited in the embodiments of the present disclosure. In implementation, the first decoder network can include but is not limited to at least one of a decoder network based on a converter, a prediction network based on a convolutional neural network, and the like.

[0063] Step S105: Based on a first loss value between the first sample image and the restored image, update the network parameters of the visual converter network at least once to obtain the trained visual converter network.

[0064] Here, any suitable loss function may be used to determine the first loss value between the first sample image and the restored image. The loss function may include but is not limited to at least one of an absolute value loss function, a minimum square error loss function, a cosine loss function, a BYOL (Bootstrap Your Own Latent) algorithm, and the like.

[0065] The method for updating the network parameters of the visual converter network can be determined according to actual conditions, and may include but is not limited to at least one parameter updating algorithm of the gradient descent method, the momentum update method, the Newton momentum method, etc., which is not limited here.

[0066] In some embodiments, it can be determined whether the network parameters of the visual converter network need to be updated based on the first loss value between the first sample image and the restored image. If the network parameters of the visual converter network need to be updated, a suitable parameter update algorithm is used to update the network parameters of the visual converter network, and the restored image is re-determined after the update to determine whether the network parameters of the visual converter network need to be further updated based on the first loss value between the first sample image and the re-determined restored image. If it is determined that the network parameters of the visual converter network do not need to be further updated, the finally updated visual converter network is determined as the trained visual converter network.

[0067] For example, when the first loss value between the first sample image and the restored image does not meet the first preset condition, the network parameters of the visual converter network can be updated. When the first loss value meets the first preset condition or the number of updates to the network parameters of the visual converter network reaches a first set threshold, the network parameters of the visual converter network are stopped from being updated, and the final updated visual converter network is determined as the trained visual converter network. The first preset condition may include, but is not limited to, the first loss value being less than a set first loss value threshold, the first loss value change converging, etc. During implementation, the first preset condition can be set according to actual conditions and is not limited here.

[0068] In an embodiment of the present disclosure, a first sample image including a preset object and a position annotation of each key point of the object in the first sample image are obtained; based on the position annotation of each key point of the object, at least one image sub-block in the first sample image is occluded to obtain an occluded image; using a visual converter network to be trained, features are extracted from the occluded image to obtain a first visual marker sequence; using a first decoder network, occluded image sub-blocks in the occluded image are restored based on the first visual marker sequence to obtain a restored image; based on a first loss value between the first sample image and the restored image, network parameters of the visual converter network are updated at least once to obtain a trained visual converter network. In this way, on the one hand, since the occluded image is obtained by occluding at least one image sub-block in the first sample image based on the position annotation of each object key point, the network parameters of the visual converter network are updated at least once based on the first loss value between the first sample image and the restored image, which can guide the visual converter network to learn the ability to understand the semantics related to the object key points, so that the visual converter network can better extract features related to the object key points, so as to improve the accuracy of object pose estimation when the trained visual converter network is used to implement the object pose estimation task; on the other hand, the first sample image used for training the visual converter network is an image containing a preset object, and the first sample image has the position annotation of each object key point, and the sample image used for network training in the object pose estimation task is usually also an image containing a preset object and having the position annotation of each object key point. In this way, the same sample image can be used in the pre-training of the visual converter network and in the network training process after migrating the pre-trained visual converter network to the object pose estimation task, thereby reducing the demand for training sample data.

[0069] The present disclosure provides a network training method, which can be executed by a processor of a computer device. Figure 2 As shown, the method includes the following steps S201 to S208:

[0070] Step S201: obtaining a first sample image containing a preset object and a position annotation of a key point of each object in the first sample image.

[0071] Here, the above step S201 corresponds to the above step S101, and the specific implementation of the above step S101 may be referred to during implementation.

[0072] Step S202: dividing the first sample image into grids according to a set grid size to obtain an image sub-block set.

[0073] Here, the grid size refers to the size of each grid when the first sample image is grid-divided, that is, the size of each image sub-block after the division. In implementation, the grid size can be set according to actual conditions, and the embodiments of the present disclosure do not limit this. For example, when the size of the first sample image is 224*224 pixels, the first sample image can be grid-divided with a grid size of 16*16 pixels to obtain an image sub-block set including 14*14 image sub-blocks.

[0074] Step S203 : based on the position annotation of each object key point, determine the target image sub-block where each object key point is located from the image sub-block set.

[0075] Here, each image sub-block may correspond to a position area in the first sample image. Based on the position annotation of each object key point, the position of each object key point in the first sample image may be determined. Then, according to the position of each object key point in the first sample image, the image sub-block where each object key point is located, i.e., the target image sub-block, may be determined.

[0076] Step S204, for each of the image sub-blocks in the image sub-block set, based on the distance between the image sub-block and each of the target image sub-blocks, determining the occlusion probability corresponding to the image sub-block; wherein, the smaller the minimum distance between the image sub-block and each target image sub-block, the greater the occlusion probability corresponding to the image sub-block.

[0077] Here, in some embodiments, the minimum distance between each image sub-block and each target image sub-block can be determined based on the distance between each image sub-block and each target image sub-block, so as to determine the occlusion probability corresponding to each image sub-block based on the minimum distance corresponding to each image sub-block, and for each image sub-block, the smaller the minimum distance between the image sub-block and each target image sub-block, the greater the occlusion probability corresponding to the image sub-block; the greater the minimum distance between the image sub-block and each target image sub-block, the smaller the occlusion probability corresponding to the image sub-block. In implementation, the occlusion probability corresponding to the image sub-block can be determined based on the minimum distance between the image sub-block and each target image sub-block in any suitable manner, and the embodiments of the present disclosure are not limited to this. For example, the minimum distance corresponding to the image sub-block can be multiplied by a set inverse coefficient to obtain an occlusion parameter, and the occlusion parameter can be normalized to the interval [0,1] to obtain the occlusion probability corresponding to the image sub-block. For another example, the inverse of the minimum distance corresponding to the image sub-block can be multiplied by a set proportional coefficient to obtain an occlusion parameter, and the occlusion parameter is normalized to the interval [0,1] to obtain the occlusion probability corresponding to the image sub-block. For another example, the minimum distance corresponding to the image sub-block can be multiplied by a set inverse coefficient to obtain an occlusion parameter, and an exponential operation is performed using the occlusion parameter as an exponent and the natural constant e as a base to obtain the occlusion probability corresponding to the image sub-block.

[0078] In some embodiments, for each target image sub-block, a candidate probability of the image sub-block can be determined based on the distance between the image sub-block and the target image sub-block; the maximum value among the candidate probabilities corresponding to the image sub-block is determined as the occlusion probability of the image sub-block. The smaller the distance between the image sub-block and the target image sub-block, the larger the candidate probability corresponding to the image sub-block; the larger the distance between the image sub-block and the target image sub-block, the smaller the candidate probability corresponding to the image sub-block. In implementation, any suitable method can be used to determine the candidate probability corresponding to the image sub-block based on the distance between the image sub-block and the target image sub-block, and the embodiments of the present disclosure are not limited to this. For example, the distance between the image sub-block and the target image sub-block can be multiplied by a set inverse coefficient to obtain an occlusion parameter, and the occlusion parameter can be normalized to the interval [0,1] to obtain the candidate probability corresponding to the image sub-block. For another example, the inverse of the distance between the image sub-block and the target image sub-block may be multiplied by a set proportional coefficient to obtain an occlusion parameter, and the occlusion parameter may be normalized to the interval [0,1] to obtain the candidate probability corresponding to the image sub-block. For another example, the distance between the image sub-block and the target image sub-block may be multiplied by a set inverse coefficient to obtain an occlusion parameter, and an exponential operation may be performed using the occlusion parameter as an exponent and the natural constant e as a base to obtain the candidate probability corresponding to the image sub-block.

[0079] Step S205 : performing occlusion processing on at least one image sub-block in the first sample image based on the occlusion probability corresponding to each image sub-block to obtain an occlusion image.

[0080] Here, for each image sub-block, it can be determined whether the image sub-block currently needs to be occluded based on the occlusion probability corresponding to the image sub-block. If it is determined that the image sub-block currently needs to be occluded, the image sub-block is subjected to occlusion processing. If it is determined that the image sub-block currently does not need to be occluded, the image sub-block is not subjected to occlusion processing. In this way, based on the occlusion probability corresponding to each image sub-block, occlusion processing can be performed on at least one image sub-block in the first sample image to obtain an occluded image.

[0081] Step S206: Using the visual converter network to be trained, extract features from the occluded image to obtain a first visual marker sequence.

[0082] Step S207: using the first decoder network, based on the first visual marker sequence, restore the occluded image sub-block in the occluded image to obtain a restored image.

[0083] Step S208: Based on a first loss value between the first sample image and the restored image, update the network parameters of the visual converter network at least once to obtain the trained visual converter network.

[0084] Here, the above steps S206 to S208 correspond to the above steps S103 to S105 respectively, and the specific implementation methods of the above steps S103 to S105 may be referred to during implementation.

[0085] In the disclosed embodiment, the first sample image is grid-divided according to the set grid size to obtain an image sub-block set; based on the position annotation of each object key point, the target image sub-block where each object key point is located is determined from the image sub-block set; for each image sub-block in the image sub-block set, the occlusion probability corresponding to the image sub-block is determined based on the distance between the image sub-block and each target image sub-block; based on the occlusion probability corresponding to each image sub-block, at least one image sub-block in the first sample image is occluded to obtain an occluded image. In this way, since the smaller the minimum distance between the image sub-block in the first sample image and each target image sub-block is, the greater the occlusion probability corresponding to the image sub-block is, the closer the image sub-block is to the target image sub-block where the object key point is located, the greater the probability of being occluded, thereby further improving the ability of the visual converter network to extract features related to the object key point, so as to further improve the accuracy of the object pose estimation when the trained visual converter network is used to implement the object pose estimation task.

[0086] In some embodiments, the determining of the occlusion probability corresponding to the image sub-block based on the distance between the image sub-block and each of the target image sub-blocks in the above step S204 may include the following steps S211 to S213:

[0087] Step S211, determining the distance between the image sub-block and each target image sub-block.

[0088] Here, for each target image sub-block, the distance between the image sub-block and the target image sub-block can be determined based on the position of the image sub-block and the position of the target image sub-block. In implementation, the distance between the image sub-block and the target image sub-block can include but is not limited to at least one of the Euclidean distance, the Mahalanobis distance, etc.

[0089] In some implementations, the Mahalanobis distance between the image sub-block and the target image sub-block may be determined as the distance between the image sub-block and the target image sub-block. For example, the Mahalanobis distance d1 between the image sub-block A and the i-th target image sub-block may be determined as shown in Formula 1:

[0090] d1=(gm i ) T Σ -1 (gm i ) (1);

[0091] Among them, i is a positive integer, g is the position coordinate of the image sub-block A, and m i is the position coordinate of the i-th target image sub-block, Σ is the set diagonal covariance matrix, and when Σ is the unit matrix, the Mahalanobis distance d1 is also the Euclidean distance between the image sub-block A and the i-th target image sub-block.

[0092] Step S212: for each of the target image sub-blocks, based on the distance between the image sub-block and the target image sub-block, determine a candidate probability corresponding to the image sub-block; the smaller the distance between the image sub-block and the target image sub-block, the greater the candidate probability corresponding to the image sub-block.

[0093] Here, the smaller the distance between the image sub-block and the target image sub-block, the greater the candidate probability corresponding to the image sub-block; the larger the distance between the image sub-block and the target image sub-block, the smaller the candidate probability corresponding to the image sub-block. During implementation, any suitable method can be used to determine the candidate probability corresponding to the image sub-block based on the distance between the image sub-block and the target image sub-block, and the embodiments of the present disclosure are not limited to this.

[0094] In some embodiments, the distance between the image sub-block and the target image sub-block may be multiplied by a set inverse coefficient to obtain an occlusion parameter, and the occlusion parameter may be normalized to the interval [0, 1] to obtain the candidate probability corresponding to the image sub-block.

[0095] In some embodiments, the inverse of the distance between the image sub-block and the target image sub-block may be multiplied by a set proportional coefficient to obtain an occlusion parameter, and the occlusion parameter may be normalized to the interval [0,1] to obtain the candidate probability corresponding to the image sub-block.

[0096] In some embodiments, the distance between the image sub-block and the target image sub-block may be multiplied by a set inverse coefficient to obtain an occlusion parameter, and an exponential operation may be performed using the occlusion parameter as an exponent and the natural constant e as a base to obtain a candidate probability corresponding to the image sub-block. For example, the candidate probability p corresponding to the image sub-block may be determined based on the distance d2 between the image sub-block and the target image sub-block in the manner shown in the following formula 2:

[0097]

[0098] Wherein, exp() is an exponential operation with the natural constant e as the base. It should be noted that the distance d2 in Formula 2 may be the Mahalanobis distance d1 in Formula 1 above, or may be a distance calculated in other ways, which is not limited in the embodiments of the present disclosure.

[0099] Step S213: determining the occlusion probability corresponding to the image sub-block based on the maximum value of each candidate probability corresponding to the image sub-block.

[0100] Here, in some implementations, the maximum value of each candidate probability corresponding to the image sub-block may be used to determine the occlusion probability corresponding to the image sub-block.

[0101] In some embodiments, a minimum threshold can be set for the occlusion probability. When the maximum value of each candidate probability corresponding to the image sub-block is less than the minimum threshold, the minimum threshold is determined as the occlusion probability corresponding to the image sub-block; when the maximum value of each candidate probability corresponding to the image sub-block is not less than the minimum threshold, the maximum value is determined as the occlusion probability corresponding to the image sub-block.

[0102] In some embodiments, a maximum threshold can be set for the occlusion probability. When the maximum value of each candidate probability corresponding to the image sub-block is greater than the maximum threshold, the maximum threshold is determined as the occlusion probability corresponding to the image sub-block; when the maximum value of each candidate probability corresponding to the image sub-block is not greater than the maximum threshold, the maximum value is determined as the occlusion probability corresponding to the image sub-block.

[0103] In the above embodiment, the distance between the image sub-block and each target image sub-block is determined; for each target image sub-block, a candidate probability corresponding to the image sub-block is determined based on the distance between the image sub-block and the target image sub-block, and the smaller the distance between the image sub-block and the target image sub-block, the greater the candidate probability corresponding to the image sub-block; based on the maximum value of each candidate probability corresponding to the image sub-block, the occlusion probability corresponding to the image sub-block is determined. In this way, the occlusion probability corresponding to each image sub-block can be determined quickly and accurately.

[0104] In some embodiments, the above step S213 may include the following step S221 or step S222:

[0105] Step S221, when the maximum value of each candidate probability corresponding to the image sub-block is less than a set probability threshold, determine the probability threshold as the occlusion probability corresponding to the image sub-block.

[0106] Here, the probability threshold may be pre-set by the user according to an actual business scenario, or may be a default value, and the embodiments of the present disclosure are not limited to this.

[0107] Step S222: when the maximum value among each candidate probability corresponding to the image sub-block is not less than the probability threshold, determine the maximum value as the occlusion probability corresponding to the image sub-block.

[0108] In the above embodiment, when the maximum value of each candidate probability corresponding to the image sub-block is less than the set probability threshold, the probability threshold is determined as the occlusion probability corresponding to the image sub-block, or when the maximum value of each candidate probability corresponding to the image sub-block is not less than the probability threshold, the maximum value is determined as the occlusion probability corresponding to the image sub-block. In this way, the occlusion probability corresponding to each image sub-block can be made not less than the set probability threshold, thereby improving the occlusion rate of the image sub-block in the occluded image, and further promoting the ability of the trained visual converter network to learn to understand the semantics related to the key points of the object.

[0109] The present disclosure provides a network training method, which can be executed by a processor of a computer device. Figure 3 As shown, the method includes the following steps S301 to S309:

[0110] Step S301: obtaining a first sample image containing a preset object and a position annotation of a key point of each object in the first sample image.

[0111] Step S302: Based on the position annotation of each object key point, perform occlusion processing on at least one image sub-block in the first sample image to obtain an occlusion image.

[0112] Step S303: Using the visual converter network to be trained, extract features from the occluded image to obtain a first visual marker sequence.

[0113] Step S304: using a first decoder network, based on the first visual marker sequence, restore the occluded image sub-block in the occluded image to obtain a restored image.

[0114] Step S305: Based on a first loss value between the first sample image and the restored image, update the network parameters of the visual converter network at least once to obtain the trained visual converter network.

[0115] Here, the above steps S301 to S305 correspond to the above steps S101 to S105 respectively, and the specific implementation methods of the above steps S101 to S105 may be referred to during implementation.

[0116] Step S306, obtaining a second sample image containing a preset object and a position mark of each object key point in the second sample image, and a first key point mark sequence, wherein each key point mark in the first key point mark sequence corresponds to an object key point of the preset object.

[0117] Here, the second sample image may be any suitable image containing at least one preset object. In implementation, the second sample image may be pre-captured using an image acquisition device, or may be generated using an image generation model, which is not limited in the embodiments of the present disclosure. The second sample image may be the same as or different from the first sample image. The second sample image may also be from the same sample data set as the first sample image.

[0118] The position annotation of each object key point in the second sample image can be used to annotate the position of each object key point in the second sample image. The position annotation of each object key point in the second sample image can be manually pre-annotated, or can be automatically annotated after the object key point is identified in the second sample image using a target detection algorithm.

[0119] The first key point marker sequence may include at least one first key point marker. Each first key point marker corresponds to a type of object key point (e.g., left knee, left ankle, right eye, etc.). Each first key point marker may be an embedded vector with a fixed dimension, and the dimension of the first key point marker may be the same as the dimension of the first visual marker. In implementation, each first key point marker in the first key point marker sequence may be learnable after being initialized in any suitable manner.

[0120] Step S307: Use the visual converter network to extract features from the second sample image to obtain a second visual marker sequence.

[0121] Here, step S307 corresponds to step S103 in the aforementioned embodiment, and reference may be made to the specific implementation of the aforementioned step S103 during implementation.

[0122] Step S308: using the first prediction network, based on the first key point label sequence and the second visual label sequence, predict the position of the object key points in the second sample image to obtain the predicted position of each of the object key points in the second sample image.

[0123] Here, the first prediction network may be any suitable prediction head, which is not limited in the embodiments of the present disclosure. In implementation, the first prediction network may include at least one of a transformer-based decoder network, a transformer-based encoder network, a classification network, and the like.

[0124] In some embodiments, the first prediction network can be used to learn the association between each first key point mark in the first key point mark sequence and each second visual mark in the second visual mark sequence to obtain the key point representation corresponding to each first key point mark, and for each first key point mark, the position of the object key point corresponding to the first key point mark is predicted based on the key point representation corresponding to the first key point mark to obtain the predicted position of the object key point in the second sample image.

[0125] In some embodiments, the first prediction network can be used to learn the association between each first key point mark in the first key point mark sequence and each second visual mark in the second visual mark sequence, as well as the association between the first key point marks, to obtain the key point representation corresponding to each first key point mark, and for each first key point mark, the position of the object key point corresponding to the first key point mark is predicted based on the key point representation corresponding to the first key point mark, to obtain the predicted position of the object key point in the second sample image.

[0126] Step S309, based on the second loss value between the position annotation of each object key point in the second sample image and the predicted position of each object key point in the second sample image, update at least one of the network parameters of the visual converter network, the network parameters of the first prediction network and the first key point labeling sequence to obtain the trained visual converter network, the trained first prediction network and the trained first key point labeling sequence.

[0127] Here, any suitable loss function may be used to determine the second loss value between the position annotation of each object key point in the second sample image and the predicted position of each object key point in the second sample image. For example, the loss function may include but is not limited to at least one of an absolute value loss function, a minimum square error loss function, a cosine loss function, a BYOL (Bootstrap Your Own Latent) algorithm, and the like.

[0128] The method for updating the network parameters of the visual converter network, the network parameters of the first prediction network, and the first key point labeling sequence can be determined according to actual conditions, and can include but is not limited to at least one parameter updating algorithm of the gradient descent method, the momentum update method, the Newton momentum method, etc., which is not limited here. In implementation, the method for updating the network parameters of the visual converter network, the network parameters of the first prediction network, and the first key point labeling sequence can be the same or different, which is not limited here.

[0129] In some embodiments, it can be determined whether it is necessary to update the network parameters of the visual converter network, the network parameters of the first prediction network, and / or the first key point labeling sequence based on the second loss value between the position annotation of each object key point in the second sample image and the predicted position of each object key point in the second sample image. If it is necessary to update the network parameters of the visual converter network, the network parameters of the first prediction network, and / or the first key point labeling sequence, a suitable parameter update algorithm is used to update them, and the predicted position of each object key point in the second sample image is re-determined after the update, so as to determine whether it is necessary to continue to update the network parameters of the visual converter network, the network parameters of the first prediction network, and / or the first key point labeling sequence based on the second loss value between the position annotation of each object key point in the second sample image and the re-determined predicted position of each object key point in the second sample image. If it is determined that it is not necessary to continue to update the network parameters of the visual converter network, the network parameters of the first prediction network, and the first key point labeling sequence, the finally updated visual converter network, the first prediction network, and the first key point labeling sequence are respectively determined as the trained visual converter network, the trained first prediction network, and the trained first key point labeling sequence.

[0130] For example, when the second loss value between the position mark of each object key point in the second sample image and the predicted position of each object key point in the second sample image does not meet the second preset condition, the network parameters of the visual converter network, the network parameters of the first prediction network and / or the first key point labeling sequence are updated. When the second loss value meets the second preset condition or the number of updates to the network parameters of the visual converter network, the network parameters of the first prediction network and / or the first key point labeling sequence reaches the second set threshold, the network parameters of the visual converter network, the network parameters of the first prediction network and the first key point labeling sequence are stopped from being updated, and the visual converter network, the first prediction network and the first key point labeling sequence after the final update are respectively determined as the trained visual converter network, the trained first prediction network and the trained first key point labeling sequence. The second preset condition may include but is not limited to the second loss value being less than the set second loss value threshold, the second loss value change converges, etc. During implementation, the second preset condition can be set according to actual conditions and is not limited here.

[0131] In the disclosed embodiment, after the network parameters of the visual converter network are updated at least once based on the first loss value between the first sample image and the restored image to obtain the trained visual converter network, a second sample image containing a preset object and the position annotation of each object key point in the second sample image, as well as a first key point labeling sequence are obtained; the second sample image is feature extracted using the visual converter network to obtain a second visual labeling sequence; the positions of the object key points in the second sample image are predicted using the first prediction network based on the first key point labeling sequence and the second visual labeling sequence to obtain the predicted position of each object key point in the second sample image; based on the second loss value between the position annotation of each object key point in the second sample image and the predicted position of each object key point in the second sample image, at least one of the network parameters of the visual converter network, the network parameters of the first prediction network and the first key point labeling sequence is updated to obtain the trained visual converter network, the trained first prediction network and the trained first key point labeling sequence. In this way, the visual converter network pre-trained on the first sample image can be transferred to the object pose estimation task for training, so as to promote the visual converter network, the first prediction network and the first key point labeling sequence to learn the information related to the object pose estimation, thereby improving the accuracy of the object pose estimation using the trained visual converter network, the trained first prediction network and the first key point labeling sequence. In addition, when the second sample image used to train the visual converter network, the first prediction network and the first key point labeling sequence in the object pose estimation task is the same as the first sample image, or the second sample image and the first sample image are from the same sample data set, the demand for training sample data can be reduced.

[0132] In some embodiments, the first prediction network includes a second decoder network based on a converter, and a position classification network; the above step S308 may include the following steps S311 to S312:

[0133] Step S311, using a second decoder network based on the converter, adopting a self-attention mechanism to update the first key point marker sequence to obtain a second key point marker sequence, and adopting a cross-attention mechanism to update the second key point marker sequence based on the second visual marker sequence to obtain a first key point representation of each key point of the object.

[0134] Here, the second decoder network can use a self-attention mechanism to perform self-attention calculation on each first key point marker in the first key point marker sequence to capture the association relationship between the first key point markers, and update the first key point marker sequence based on the association relationship to obtain a second key point marker sequence.

[0135] The second decoder network can use a cross-attention mechanism to perform cross-attention calculation on the second key point mark sequence and the second visual mark sequence to capture the association relationship between each first key point mark in the second key point mark sequence and each second visual mark in the second visual mark sequence, and update the second key point mark sequence based on the association relationship to obtain the first key point representation of each object key point.

[0136] Step S312: using a position classification network, based on each of the first key point representations, predict the position of the object key point in the second sample image to obtain the predicted position of each of the object key points in the second sample image.

[0137] Here, the position classification network can adopt any suitable network structure, which is not limited here. In implementation, the position classification network can include but is not limited to at least one of a fully connected layer, a Softmax layer, a linear mapping layer, etc.

[0138] In the above embodiment, the first key point labeling sequence is updated by using the second decoder network based on the converter and the self-attention mechanism is adopted to obtain the second key point labeling sequence, and the second key point labeling sequence is updated by using the cross-attention mechanism based on the second visual labeling sequence to obtain the first key point representation of each object key point; the position of the object key point in the second sample image is predicted based on each first key point representation by using the position classification network to obtain the predicted position of each object key point in the second sample image. In this way, the constraint relationship between each object key point and the visual clues of each object key point in the visual features of the second sample image can be learned from the first key point representation of each object key point determined by the second decoder network based on the converter, so that the position of the object key point in the second sample image can be predicted more accurately based on the first key point representation of each object key point, thereby further improving the accuracy of object posture estimation using the trained visual converter network, the trained first prediction network and the first key point labeling sequence.

[0139] In some embodiments, the first prediction network may include a plurality of cascaded transformer-based second decoder networks. Thus, by using the cascaded plurality of transformer-based second decoder networks, a more accurate first key point representation may be learned for each object key point, thereby further improving the accuracy of object pose estimation using the trained visual transformer network, the trained first prediction network, and the first key point labeling sequence.

[0140] In some embodiments, the second decoder network includes a self-attention layer, a cross-attention layer, and a feed-forward network layer; the above step S311 may include the following steps S321 to S323:

[0141] Step S321, using the self-attention layer and the self-attention mechanism, to update the first key point labeling sequence to obtain a second key point labeling sequence.

[0142] Here, the self-attention layer can be the self-attention module in the transformer-based decoder network.

[0143] Step S322, using the cross-attention layer and the cross-attention mechanism, based on the second visual marker sequence, the second key point marker sequence is updated to obtain a third key point marker sequence.

[0144] Here, the criss-cross attention layer can be a criss-cross attention module in a transformer-based decoder network.

[0145] Step S323: using the feed-forward network layer, convert each key point tag in the third key point tag sequence into a key point representation of a corresponding object key point.

[0146] Here, the feed-forward network layer may be a feed-forward network module in a transformer-based decoder network.

[0147] In some embodiments, the self-attention mechanism described in step S321 is used to update the first key point label sequence to obtain the second key point label sequence, which may include the following steps S331 to S332:

[0148] Step S331: adopt a self-attention mechanism to extract the association relationship between the key point tags in the first key point tag sequence to obtain first association information.

[0149] Here, the first association information may represent the association relationship between the key point tags in the first key point tag sequence, that is, the association relationship between the object key points corresponding to the key point tags.

[0150] Step S332: Based on the first association information, the first key point labeling sequence is updated to obtain a second key point labeling sequence.

[0151] Here, in some embodiments, the first association information may be a feature matrix of the same dimension as the first key point labeling sequence, and the first association information may be fused with the first key point labeling sequence to obtain an updated first key point labeling sequence, that is, a second key point labeling sequence. For example, the first association information may be added to the first key point labeling sequence to obtain the second key point labeling sequence.

[0152] In some embodiments, the cross-attention mechanism described in step S322 above is used to update the second key point mark sequence based on the second visual mark sequence to obtain a third key point mark sequence, which may include the following steps S341 to S342:

[0153] Step S341: Use a cross-attention mechanism to extract the association relationship between the second key point mark sequence and the second visual mark sequence to obtain second association information.

[0154] Here, the second association information may represent the association relationship between the second key point mark sequence and the second visual mark sequence, that is, the association relationship between the object key points corresponding to each key point mark and the visual features of the second sample image.

[0155] Step S342: Based on the second association information, the second key point labeling sequence is updated to obtain a third key point labeling sequence.

[0156] Here, in some embodiments, the second association information may be a feature matrix of the same dimension as the second key point labeling sequence, and the second association information may be fused with the second key point labeling sequence to obtain an updated second key point labeling sequence, that is, a third key point labeling sequence. For example, the second association information may be added to the second key point labeling sequence to obtain the third key point labeling sequence.

[0157] In some embodiments, the position classification network includes a first classification network and a second classification network, and the predicted position includes first position information in the horizontal direction and second position information in the vertical direction; the above step S312 may include the following steps S351 to S352:

[0158] Step S351, using the first classification network, based on each of the key point representations, predicting the position of the object key point in the second sample image in the horizontal direction, and obtaining first position information of each of the object key point in the second sample image.

[0159] Here, during implementation, the first classification network can be used to perform linear mapping on each key point representation in the horizontal direction of the second sample image to obtain the mapped key point representation corresponding to each key point representation, and based on each mapped key point representation, the position of the object key point in the second sample image in the horizontal direction is predicted to obtain the first position information of each object key point in the second sample image.

[0160] Step S352: using the second classification network, based on each of the key point representations, predict the position of the object key point in the second sample image in the vertical direction, and obtain second position information of each of the object key point in the second sample image.

[0161] Here, during implementation, a second classification network can be used to linearly map each key point representation in the vertical direction of the second sample image to obtain a mapped key point representation corresponding to each key point representation, and based on each mapped key point representation, the position of the object key point in the second sample image in the vertical direction is predicted to obtain the second position information of each object key point in the second sample image.

[0162] In the above embodiment, the first position information of each object key point in the horizontal direction of the second sample image and the second position information in the vertical direction of the second sample image can be predicted respectively, thereby improving the accuracy of the predicted position of each object key point, that is, improving the accuracy of object posture estimation.

[0163] The present disclosure provides a method for estimating an object posture, which can be executed by a processor of a computer device. Figure 4 As shown, the method includes the following steps S401 to S403:

[0164] Step S401, acquiring a first image including a preset object and a set fourth key point mark sequence, wherein each key point mark in the fourth key point mark sequence corresponds to an object key point of the preset object;

[0165] Here, the first image may be any suitable image for which object posture estimation is to be performed. During implementation, those skilled in the art may select a suitable first image according to the actual application scenario, and the embodiments of the present disclosure are not limited to this.

[0166] The fourth key point label sequence may include at least one fourth key point label. Each fourth key point label corresponds to a type of object key point (e.g., left knee, left ankle, right eye, etc.). Each fourth key point label may be an embedding vector with a fixed dimension. In implementation, each fourth key point label in the fourth key point label sequence may be learnable after being initialized in any suitable manner.

[0167] Step S402, using a trained visual converter network to extract features from the first image to obtain a third visual marker sequence; wherein the visual converter network is trained using the network training method described in the above embodiment;

[0168] Step S403, using the trained second prediction network, based on the fourth key point mark sequence and the third visual mark sequence, predict the position of the key points of the object in the first image to obtain the predicted position of each key point of the object in the first image.

[0169] Here, the second prediction network may be any suitable prediction head, which is not limited in the embodiment of the present disclosure. In implementation, the second prediction network may include at least one of a transformer-based decoder network, a transformer-based encoder network, a classification network, and the like.

[0170] The above steps S401 to S403 correspond to steps S306 to S308 in the above embodiments respectively, and the specific implementation of the above steps S306 to S308 may be referred to during implementation.

[0171] In the embodiments of the present disclosure, since the network training method described in the above embodiments can guide the visual converter network to learn the ability to understand the semantics related to the key points of the object, so that the visual converter network can better extract the features related to the key points of the object, the accuracy of the object pose estimation can be improved by using the trained visual converter network to implement object pose estimation.

[0172] In some embodiments, the fourth key point labeling sequence includes the first key point labeling sequence obtained by using the network training method described in the above embodiments, and the second prediction network includes the first prediction network obtained by using the network training method described in the above embodiments.

[0173] In this way, since the network training method described in the above embodiments can promote the visual converter network, the first prediction network and the first key point labeling sequence to learn information related to object pose estimation, the trained visual converter network, the trained first prediction network and the first key point labeling sequence can be used to perform object pose estimation, the accuracy of the predicted position of each object key point in the first image can be further improved.

[0174] The following describes the application of the network training method and object posture estimation method provided by the embodiments of the present disclosure in actual scenarios.

[0175] Taking the human pose estimation scenario as an example, the human pose estimation method based on the visual transformer network in the related art usually adopts the following scheme: supervised pre-training of the visual transformer network on the image classification dataset ImageNet, and after migrating the pre-trained visual transformer network to the object pose estimation task, fine-tuning the network parameters using the Common Objects in Context (COCO) dataset or the human pose dataset MPII. This scheme has the following problems: 1) In addition to the COCO dataset or MPII dataset used for fine-tuning the network parameters, it relies on an additional pre-training dataset ImageNet; 2) there is a significant deviation between the pre-training task and the downstream key point localization problem; (3) the ImageNet dataset used for pre-training itself is not a dataset with the human body as the main body, so it cannot provide more benefits for the model to understand the semantics related to the key points of the human body.

[0176] The disclosed embodiments provide a network training method centered on human key points, which can be applied to human posture estimation tasks. The method does not need to rely on additional data such as ImageNet, and guides the visual converter network to learn more information related to human key points to promote the semantic understanding of human key points during the pre-training process, thereby strengthening the correlation between the pre-training task and the downstream key point positioning problem.

[0177] Different from the method of performing supervised pre-training on the ImageNet dataset in the related art, the network training method provided in the embodiment of the present disclosure only uses the COCO dataset to pre-train the visual converter network. For each sample image (corresponding to the third sample image in the aforementioned embodiment), the human body in the sample image is detected by a set human body detector to obtain at least one image area containing a human body, and each image area containing a human body is cut out to obtain a sub-image, and then each sub-image is scaled to a size of 224*224 pixels to obtain each scaled sub-image, and each scaled sub-image is used as a first sample image. The following describes the process of pre-training the visual converter network in the human posture estimation task using each first sample image in the network training method provided in the embodiment of the present disclosure, such as Figure 5A As shown, the process includes the following steps S501 to S506:

[0178] Step S501 , grid-dividing the first sample image with a grid size of 16×16 pixels to obtain an image sub-block set including 14*14 image sub-blocks.

[0179] Step S502 : based on the position annotation of each object key point in the first sample image, determine the target image sub-block where each object key point is located from the image sub-block set.

[0180] Step S503, for each image sub-block in the image sub-block set, based on the distance between the image sub-block and each target image sub-block, determine the occlusion probability corresponding to the image sub-block; wherein, the smaller the minimum distance between the image sub-block and each target image sub-block, the greater the occlusion probability corresponding to the image sub-block.

[0181] In implementation, a heuristic Gaussian-like distribution can be used to determine the occlusion probability corresponding to each image sub-block. For example, the occlusion probability P(g; m, Σ, t) corresponding to the image sub-block A can be determined in the manner shown in Formula 3:

[0182]

[0183] Among them, m is the set of position coordinates of the target image sub-block, m i is the position coordinate of the i-th target image sub-block, i is a positive integer, g is the position coordinate of the image sub-block A, Σ is the set diagonal covariance matrix, Σ can be the unit matrix, and t is the set probability threshold.

[0184] Step S504: Based on the occlusion probability corresponding to each image sub-block, perform occlusion processing on at least one image sub-block in the first sample image to obtain an occlusion image.

[0185] here, Figure 5B A schematic diagram of the effect of performing occlusion processing on at least one image sub-block in a first sample image in a network training method provided by an embodiment of the present disclosure, such as Figure 5B As shown, the first sample image 10 includes a human body 11. Based on the occlusion probability 21 corresponding to each image sub-block in the first sample image 10, at least one image sub-block in the first sample image 10 is subjected to occlusion processing to obtain an occluded image 22. It can be seen that in the occluded image 22, the closer the image sub-block is to the target image sub-block where the key point of the human body is located, the greater the probability of being occluded.

[0186] Step S505, using the visual converter network to be trained and the first decoder network, restore the occluded image sub-block in the occluded image to obtain a restored image.

[0187] Here, the feature representations learned from the visual markers visible in the occluded image by the visual converter network to be trained and the first decoder network can be used to restore the original pixel values ​​of the visual markers corresponding to the masked image sub-blocks, thereby obtaining a restored image.

[0188] Step S506: Based on the first loss value between the first sample image and the restored image, update the network parameters of the visual converter network at least once to obtain a trained visual converter network.

[0189] Here, an L2 norm loss function may be used to determine a first loss value between the first sample image and the restored image.

[0190] After pre-training the visual converter network, the trained visual converter network can be applied to the human pose estimation model, and the human pose estimation model can be fine-tuned to obtain a fine-tuned human pose estimation model. In the fine-tuning training stage, the visual converter network and the prediction network in the human pose estimation model are optimized together, and the loss function can use the cross entropy loss function.

[0191] The following describes a human body posture estimation model provided by an embodiment of the present disclosure. Figure 5C A schematic diagram of the structure of a human body posture estimation model provided in an embodiment of the present disclosure is shown in FIG. Figure 5CAs shown, the human body posture estimation model 300 includes a visual transformer network 310 and a prediction network 320 (corresponding to the first prediction network or the second prediction network in the aforementioned embodiment), wherein the visual transformer network 310 serves as the backbone network of the human body posture estimation model 300, the visual transformer network 310 uses the original visual transformer (ViT) structure, and the visual transformer network 310 is initialized by the network training method provided by the above-mentioned embodiment of the present disclosure; the prediction network 320 includes M transformer-based second decoder networks 321 and a position classification network 322, where M is a positive integer; the second decoder network 321 may include a self-attention layer 3211, a cross-attention layer 3212 and a feedforward network layer 3213, and the position classification network 322 may include a first classification network 3221 and a second classification network 3222. In the task of human posture estimation, the visual converter network 310 can be used to extract features from the first image 40 to obtain a third visual marker sequence 51, and the self-attention layer 3211 in the decoder network 321 is used to update the fourth key point marker sequence 52 using the self-attention mechanism, and the cross-attention layer 3212 is used to update the fifth key point marker sequence output by the self-attention layer 3211 based on the third visual marker sequence 51 using the cross-attention mechanism, and the feedforward network layer 3213 is used to update the sixth key point marker sequence 52 output by the cross-attention layer 3212. Each key point mark in the image is converted into a key point representation of the corresponding human key point; the first classification network 3221 can be used to predict the horizontal position of the human key point in the first image based on each key point representation output by the feedforward network layer 3213, and obtain the first position information 61 of each human key point in the first image; the second classification network 3222 can be used to predict the vertical position of the human key point in the first image based on each key point representation output by the feedforward network layer 3213, and obtain the second position information 62 of each human key point in the first image.

[0192] In some implementations, the prediction network 320 may include three transformer-based decoder networks 321 .

[0193] In some embodiments, the first classification network 3221 may include: a horizontal linear mapping layer, used to linearly map each key point representation in the horizontal direction of the first image to obtain a mapped key point representation corresponding to each key point representation; a first Softmax layer, used to predict the position of the human body key points in the first image in the horizontal direction based on each mapped key point representation output by the horizontal linear mapping layer, and obtain the first position information of each human body key point in the first image.

[0194] In some embodiments, the second classification network 3222 may include: a vertical linear mapping layer, which is used to linearly map each key point representation in the vertical direction of the first image to obtain a mapped key point representation corresponding to each key point representation; a second Softmax layer, which is used to predict the position of the human body key points in the first image in the vertical direction based on each mapped key point representation output by the vertical linear mapping layer, to obtain the second position information of each human body key point in the first image.

[0195] In the human posture estimation model provided by the above-mentioned embodiment of the present disclosure, the prediction network can extract the key point features corresponding to each human key point in the general visual features (i.e., the third visual mark sequence) extracted by the visual converter network in a customized manner based on the fourth key point mark sequence, and predict the position of the human key point based on the key point features corresponding to each human key point, and obtain the predicted position of each human key point in the first image. In the problem of human key point positioning, on the one hand, it depends on the visual information provided by the image, and on the other hand, it also depends on the inherent connection between human key points (such as: adjacency constraints and symmetry constraints, etc.). Therefore, in the human posture estimation model provided by the embodiment of the present disclosure, the prediction network can use the good features of the second decoder network based on the converter to represent each human key point type through a set of embedding vector parameters (i.e., the fourth key point mark sequence) that can be learned and optimized, and each key point mark in the fourth key point mark sequence is used as a query vector to obtain the key point representation corresponding to each key point type from the third visual mark sequence extracted by the visual converter network. In this process, the self-attention layer in the second decoder network based on the converter corresponds to the modeling process of the constraint information between the key points, which can capture the correlation information between the key points of the human body, while the cross-attention layer corresponds to the process of visual information modeling, which can capture the correlation between the key points of the human body and the visual features of the first image.

[0196] The human body posture estimation model provided by the embodiments of the present disclosure can, on the one hand, capture the intrinsic correlation between the key points of the human body, thereby improving the accuracy of human body posture estimation; on the other hand, it can improve the performance of the original straight-tube visual converter network in the human body posture estimation task, thereby improving the versatility of the original visual converter network in the human body posture estimation task.

[0197] Based on the foregoing embodiments, the embodiments of the present disclosure provide a network training device, which includes the units included and the modules included in the units, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0198] Figure 6 A schematic diagram of the structure of a network training device provided in an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the network training device 600 includes: a first acquisition module 610, an occlusion module 620, a first extraction module 630, a restoration module 640 and a first update module 650, wherein:

[0199] A first acquisition module 610 is used to acquire a first sample image containing a preset object and a position annotation of a key point of each object in the first sample image;

[0200] An occlusion module 620, configured to perform occlusion processing on at least one image sub-block in the first sample image based on the position annotation of each object key point to obtain an occlusion image;

[0201] A first extraction module 630, configured to extract features from the occluded image using a visual converter network to be trained, to obtain a first visual marker sequence;

[0202] A restoration module 640 is configured to perform restoration processing on the occluded image sub-block in the occluded image based on the first visual marker sequence using a first decoder network to obtain a restored image;

[0203] The first updating module 650 is used to update the network parameters of the visual converter network at least once based on the first loss value between the first sample image and the restored image to obtain the trained visual converter network.

[0204] In some embodiments, the occlusion module is also used to: grid the first sample image according to a set grid size to obtain an image sub-block set; determine the target image sub-block where each object key point is located from the image sub-block set based on the position annotation of each object key point; for each image sub-block in the image sub-block set, determine the occlusion probability corresponding to the image sub-block based on the distance between the image sub-block and each target image sub-block; wherein, the smaller the minimum distance between the image sub-block and each target image sub-block, the greater the occlusion probability corresponding to the image sub-block; based on the occlusion probability corresponding to each image sub-block, perform occlusion processing on at least one image sub-block in the first sample image to obtain an occluded image.

[0205] In some embodiments, the occlusion module is further used to: determine the distance between the image sub-block and each target image sub-block; for each target image sub-block, based on the distance between the image sub-block and the target image sub-block, determine a candidate probability corresponding to the image sub-block, the smaller the distance between the image sub-block and the target image sub-block, the larger the candidate probability corresponding to the image sub-block; based on the maximum value of each candidate probability corresponding to the image sub-block, determine the occlusion probability corresponding to the image sub-block.

[0206] In some embodiments, the occlusion module is also used for one of the following: when the maximum value of each candidate probability corresponding to the image sub-block is less than a set probability threshold, the probability threshold is determined as the occlusion probability corresponding to the image sub-block; when the maximum value of each candidate probability corresponding to the image sub-block is not less than the probability threshold, the maximum value is determined as the occlusion probability corresponding to the image sub-block.

[0207] In some embodiments, the device further includes: a second acquisition module, used to acquire a second sample image containing a preset object and a position annotation of each object key point in the second sample image, and a first key point tag sequence, wherein each key point tag in the first key point tag sequence corresponds to an object key point of the preset object; a second extraction module, used to use the visual converter network to perform feature extraction on the second sample image to obtain a second visual tag sequence; a first prediction module, used to use the first prediction network to predict the position of the object key points in the second sample image based on the first key point tag sequence and the second visual tag sequence to obtain the predicted position of each object key point in the second sample image; a second update module, used to update at least one of the network parameters of the visual converter network, the network parameters of the first prediction network and the first key point tag sequence based on a second loss value between the position annotation of each object key point in the second sample image and the predicted position of each object key point in the second sample image, to obtain a trained visual converter network, a trained first prediction network and a trained first key point tag sequence.

[0208] In some embodiments, the first prediction network includes a second decoder network based on a converter, and a position classification network; the first prediction module is also used to: use the second decoder network based on the converter, adopt a self-attention mechanism to update the first key point label sequence to obtain a second key point label sequence, and adopt a cross-attention mechanism to update the second key point label sequence based on the second visual label sequence to obtain a first key point representation of each of the object key points; use the position classification network to predict the position of the object key point in the second sample image based on each of the first key point representations to obtain the predicted position of each of the object key points in the second sample image.

[0209] In some embodiments, the second decoder network includes a self-attention layer, a cross-attention layer and a feedforward network layer; the first prediction module is also used to: use the self-attention layer and the self-attention mechanism to update the first key point mark sequence to obtain a second key point mark sequence; use the cross-attention layer and the cross-attention mechanism to update the second key point mark sequence based on the second visual mark sequence to obtain a third key point mark sequence; use the feedforward network layer to convert each key point mark in the third key point mark sequence into a key point representation of the corresponding object key point.

[0210] In some embodiments, the first prediction module is also used to: use a self-attention mechanism to extract the correlation relationship between each key point mark in the first key point mark sequence to obtain first correlation information; based on the first correlation information, update the first key point mark sequence to obtain a second key point mark sequence.

[0211] In some embodiments, the first prediction module is also used to: use a cross-attention mechanism to extract the association relationship between the second key point marker sequence and the second visual marker sequence to obtain second association information; based on the second association information, update the second key point marker sequence to obtain a third key point marker sequence.

[0212] In some embodiments, the position classification network includes a first classification network and a second classification network, and the predicted position includes first position information in the horizontal direction and second position information in the vertical direction; the first prediction module is also used to: use the first classification network to predict the position of the object key point in the second sample image in the horizontal direction based on each key point representation, and obtain the first position information of each object key point in the second sample image; use the second classification network to predict the position of the object key point in the vertical direction in the second sample image based on each key point representation, and obtain the second position information of each object key point in the second sample image.

[0213] Figure 7 A schematic diagram of the structure of an object posture estimation device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the object posture estimation device 700 includes: a third acquisition module 710, a third extraction module 720 and a second prediction module 730, wherein:

[0214] A third acquisition module 710 is used to acquire a first image including a preset object and a set fourth key point mark sequence, wherein each key point mark in the fourth key point mark sequence corresponds to an object key point of the preset object;

[0215] A third extraction module 720 is used to extract features from the first image using a trained visual converter network to obtain a third visual tag sequence; wherein the visual converter network is trained using the network training method described in the above embodiment;

[0216] The second prediction module 730 is used to use the trained second prediction network to predict the position of the object key points in the first image based on the fourth key point mark sequence and the third visual mark sequence, and obtain the predicted position of each object key point in the first image.

[0217] In some embodiments, the fourth key point labeling sequence includes the first key point labeling sequence obtained by using the network training method described in the above embodiments, and the second prediction network includes the first prediction network obtained by using the network training method described in the above embodiments.

[0218] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present disclosure, please refer to the description of the method embodiment of the present disclosure for understanding.

[0219] It should be noted that in the embodiments of the present disclosure, if the above-mentioned network training method or object posture estimation method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiments of the present disclosure are not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0220] An embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0221] The embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0222] An embodiment of the present disclosure provides a computer program, including a computer-readable code. When the computer-readable code is executed in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0223] An embodiment of the present disclosure provides a computer program product, including a computer program or instructions, which implements some or all of the steps in the above method when executed by a processor.

[0224] The present disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0225] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product disclosed in the present invention, please refer to the description of the method embodiment disclosed in the present invention for understanding.

[0226] It should be noted that Figure 8 is a schematic diagram of a hardware entity of a computer device in an embodiment of the present disclosure, such as Figure 8 As shown, the hardware entity of the computer device 800 includes: a processor 801, a communication interface 802 and a memory 803, wherein:

[0227] Processor 801 generally controls the overall operation of computer device 800 .

[0228] The communication interface 802 enables the computer device to communicate with other terminals or servers through a network.

[0229] The memory 803 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or processed by the processor 801 and each module in the computer device 800 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be performed between the processor 801, the communication interface 802, and the memory 803 through the bus 804.

[0230] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial numbers of the above-mentioned steps / processes does not mean the order of execution. The execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the embodiments of the present disclosure are for description only and do not represent the advantages and disadvantages of the embodiments.

[0231] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0232] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0233] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0234] In addition, all functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0235] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0236] Alternatively, if the above-mentioned integrated unit of the present disclosure is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0237] The above description is only an implementation mode of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure.

Claims

1. A network training method, characterized in that: The method comprises: Acquire a first sample image containing a preset object and a position annotation of a key point of each object in the first sample image; Based on the position annotation of each object key point, performing occlusion processing on at least one image sub-block in the first sample image to obtain an occluded image; Using the visual converter network to be trained, extracting features from the occluded image to obtain a first visual marker sequence; Using a first decoder network, based on the first visual marker sequence, restore the occluded image sub-block in the occluded image to obtain a restored image; Based on a first loss value between the first sample image and the restored image, the network parameters of the visual converter network are updated at least once to obtain the trained visual converter network.

2. The method according to claim 1, characterized in that The step of performing occlusion processing on at least one image sub-block in the first sample image based on the position annotation of each object key point to obtain an occluded image includes: Dividing the first sample image into grids according to a set grid size to obtain an image sub-block set; Based on the position annotation of each object key point, determining the target image sub-block where each object key point is located from the image sub-block set; For each of the image sub-blocks in the image sub-block set, based on the distance between the image sub-block and each of the target image sub-blocks, determining the occlusion probability corresponding to the image sub-block; wherein the smaller the minimum distance between the image sub-block and each target image sub-block, the greater the occlusion probability corresponding to the image sub-block; Based on the occlusion probability corresponding to each of the image sub-blocks, occlusion processing is performed on at least one image sub-block in the first sample image to obtain an occlusion image.

3. The method according to claim 2, characterized in that The determining, based on the distances between the image sub-block and each of the target image sub-blocks, the occlusion probability corresponding to the image sub-block comprises: determining a distance between the image sub-block and each target image sub-block; For each of the target image sub-blocks, based on the distance between the image sub-block and the target image sub-block, determining a candidate probability corresponding to the image sub-block, wherein the smaller the distance between the image sub-block and the target image sub-block, the greater the candidate probability corresponding to the image sub-block; Based on the maximum value of each candidate probability corresponding to the image sub-block, the occlusion probability corresponding to the image sub-block is determined.

4. The method according to any one of claims 1 to 3, characterized in that After updating the network parameters of the visual converter network at least once based on the first loss value between the first sample image and the restored image to obtain the trained visual converter network, the method further includes: Acquire a second sample image containing a preset object and a position mark of each object key point in the second sample image, and a first key point mark sequence, wherein each key point mark in the first key point mark sequence corresponds to an object key point of the preset object; Using the visual transformer network, extracting features from the second sample image to obtain a second visual marker sequence; Using a first prediction network, based on the first key point label sequence and the second visual label sequence, predict the position of the key points of the object in the second sample image to obtain the predicted position of each key point of the object in the second sample image; Based on a second loss value between the position annotation of each object key point in the second sample image and the predicted position of each object key point in the second sample image, at least one of the network parameters of the visual converter network, the network parameters of the first prediction network and the first key point labeling sequence is updated to obtain a trained visual converter network, a trained first prediction network and a trained first key point labeling sequence.

5. The method according to claim 4, characterized in that The first prediction network includes a second decoder network based on a converter, and a position classification network; the using the first prediction network to predict the position of the object key point in the second sample image based on the first key point label sequence and the second visual label sequence to obtain the predicted position of each of the object key points in the second sample image includes: Using a second decoder network based on the converter, the first key point label sequence is updated by a self-attention mechanism to obtain a second key point label sequence, and the second key point label sequence is updated by a cross-attention mechanism based on the second visual label sequence to obtain a first key point representation of each key point of the object; Using a position classification network, based on each of the first key point representations, the position of the object key point in the second sample image is predicted to obtain the predicted position of each of the object key point in the second sample image.

6. The method according to claim 5, characterized in that The second decoder network includes a self-attention layer, a cross-attention layer, and a feed-forward network layer; The method uses a second decoder network based on the converter to update the first key point label sequence using a self-attention mechanism to obtain a second key point label sequence, and uses a cross-attention mechanism to update the second key point label sequence based on the second visual label sequence to obtain a first key point representation of each key point of the object, including: Using the self-attention layer and the self-attention mechanism, the first key point labeling sequence is updated to obtain a second key point labeling sequence; Using the cross attention layer and adopting the cross attention mechanism, based on the second visual mark sequence, the second key point mark sequence is updated to obtain a third key point mark sequence; Using the feed-forward network layer, each key point tag in the third key point tag sequence is converted into a key point representation of a corresponding object key point.

7. The method according to claim 6, characterized in that The self-attention mechanism is used to update the first key point label sequence to obtain a second key point label sequence, including: Using a self-attention mechanism, extracting the association relationship between the key point tags in the first key point tag sequence to obtain first association information; Based on the first association information, the first key point labeling sequence is updated to obtain a second key point labeling sequence.

8. The method according to claim 6 or 7, characterized in that: The method adopts the cross attention mechanism and updates the second key point mark sequence based on the second visual mark sequence to obtain a third key point mark sequence, including: Using a cross attention mechanism, extracting the association relationship between the second key point mark sequence and the second visual mark sequence to obtain second association information; Based on the second association information, the second key point labeling sequence is updated to obtain a third key point labeling sequence.

9. The method according to any one of claims 5 to 8, characterized in that The position classification network includes a first classification network and a second classification network, and the predicted position includes first position information in a horizontal direction and second position information in a vertical direction; The using the position classification network to predict the position of the object key point in the second sample image based on each of the first key point representations to obtain the predicted position of each of the object key point in the second sample image includes: Using the first classification network, based on each of the key point representations, predicting the position of the object key point in the second sample image in the horizontal direction, to obtain first position information of each of the object key point in the second sample image; The second classification network is used to predict the position of the object key point in the second sample image in the vertical direction based on each of the key point representations, so as to obtain second position information of each of the object key point in the second sample image.

10. A method for estimating an object posture, characterized in that: The method comprises: Acquire a first image including a preset object and a set fourth key point mark sequence, wherein each key point mark in the fourth key point mark sequence corresponds to an object key point of the preset object; Using a trained visual converter network, extracting features from the first image to obtain a third visual tag sequence; wherein the visual converter network is trained using the network training method according to any one of claims 1 to 9; Using the trained second prediction network, based on the fourth key point label sequence and the third visual label sequence, the positions of the key points of the object in the first image are predicted to obtain the predicted position of each key point of the object in the first image.

11. The method according to claim 10, characterized in that The fourth key point labeling sequence includes the first key point labeling sequence obtained by using the network training method as described in any one of claims 4 to 9, and the second prediction network includes the first prediction network obtained by using the network training method as described in any one of claims 4 to 9.

12. A network training device, characterized in that: include: A first acquisition module, used to acquire a first sample image containing a preset object and a position annotation of a key point of each object in the first sample image; an occlusion module, configured to perform occlusion processing on at least one image sub-block in the first sample image based on the position annotation of each key point of the object, so as to obtain an occlusion image; A first extraction module, configured to extract features from the occluded image using a visual converter network to be trained, to obtain a first visual marker sequence; a restoration module, configured to perform restoration processing on the occluded image sub-blocks in the occluded image based on the first visual marker sequence by using the first decoder network to obtain a restored image; The first updating module is used to update the network parameters of the visual converter network at least once based on a first loss value between the first sample image and the restored image to obtain the trained visual converter network.

13. An object posture estimation device, characterized in that: include: A third acquisition module is used to acquire a first image containing a preset object and a set fourth key point mark sequence, wherein each key point mark in the fourth key point mark sequence corresponds to an object key point of the preset object; a third extraction module, configured to extract features from the first image using a trained visual converter network to obtain a third visual tag sequence; wherein the visual converter network is trained using the network training method according to any one of claims 1 to 9; The second prediction module is used to use the trained second prediction network to predict the position of the key points of the object in the first image based on the fourth key point label sequence and the third visual label sequence, so as to obtain the predicted position of each key point of the object in the first image.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 9 or the method according to any one of claims 10 to 11 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 9 or the method according to any one of claims 10 to 11.

Images