Method and apparatus for extracting skeleton of quadruped animal, and storage medium

Abstract

The application discloses a four-limbed animal skeleton extraction method, comprising the following steps: (1) obtaining pictures of a plurality of four-limbed animals as samples, and marking all key nodes and numbers of the four-limbed animals in the pictures as corresponding labels; (2) constructing a skeleton extraction network, wherein the skeleton extraction network comprises a backbone extraction network, a Deformable DETR network, a feature replication module, a key point regression submodule and a skeleton label regression submodule; (3) inputting all samples and corresponding labels into the skeleton extraction network for training; (4) inputting a four-limbed animal picture to be extracted into the trained skeleton extraction network to obtain coordinates and numbers of all key nodes of the four-limbed animals in the picture; (5) repairing key points with wrong positions in the key nodes; and (6) mapping the repaired key nodes to a 3D space and connecting the key nodes in sequence according to the numbers to obtain a four-limbed animal skeleton. The application has higher extraction accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to image processing technology, and in particular to a quadruped skeleton extraction method, device and storage medium. BACKGROUND

[0002] Quadrupeds include amphibians, reptiles and mammals. At present, there are many studies on human key point and skeleton estimation, but the skeleton estimation of human body is difficult to apply to quadrupeds, the main reasons including: (1) the quadruped body has high similarity, the movement is complex, and higher requirements are put forward for accurate distinction of the front and rear four limbs; (2) the fur of part of the quadruped has similar color, which has a negative impact on the positioning of the key point; (3) the complex natural environment occlusion, color interference and self-occlusion of the quadruped are serious; (4) individual differences, shooting angles, distances and other factors cause the diversity of the same subject in scale. Therefore, it is necessary to study the skeleton extraction technology of quadrupeds. SUMMARY

[0003] The present application provides a quadruped skeleton extraction method, device and storage medium with higher recognition accuracy.

[0004] Technical scheme: the quadruped skeleton extraction method provided by the present application comprises:

[0005] (1) obtaining pictures of a plurality of quadrupeds as samples, and marking all key nodes and numbers of the quadrupeds in the pictures as corresponding labels;

[0006] (2) constructing a skeleton extraction network, the skeleton extraction network comprising:

[0007] a main extraction network for extracting coarse-grained subject features from the samples;

[0008] a Deformable DETR network for extracting fine-grained features from the coarse-grained subject features;

[0009] a feature replication module for replicating the fine-grained features;

[0010] a key point regression submodule for extracting key node coordinates and numbers from the replicated fine-grained features;

[0011] a skeleton label regression submodule for extracting the ID number of the quadruped to which the fine-grained features belong;

[0012] (3) inputting all samples and corresponding labels into the skeleton extraction network for training;

[0013] (4) inputting the picture of the four-limbed animal to be extracted into the trained skeleton extraction network to obtain the key node coordinates and numbers of all four-limbed animals in the picture;

[0014] (5) repairing the key nodes with position errors in the key nodes;

[0015] (6) mapping the repaired key nodes to a 3D space and connecting the key nodes in the order of the numbers to obtain the skeleton of the four-limbed animal.

[0016] Further, the key nodes of the four-limbed animal include a mouth, a neck, a front left hip joint, a front left knee joint, a front left hoof, a front spine, a front right hip joint, a front right knee joint, a front right hoof, a back left hip joint, a back left knee joint, a back left hoof, a back spine, a tail, a back right hip joint, a back right knee joint, a back right hoof, and the corresponding numbers are 0-16 in sequence.

[0017] Further, the main stem extraction network is specifically a swin-base network.

[0018] Further, the key point regression sub-module includes a first branch and a second branch, the first branch is a one-layer softmax layer, and is used for extracting key node numbers from fine-grained features, and the second branch includes three full connection layers connected in sequence, and is used for extracting key node coordinates from fine-grained features.

[0019] Further, the skeleton label regression sub-module is specifically a one-layer softmax layer.

[0020] Further, the repairing of the key nodes with position errors specifically includes:

[0021] obtaining two or three key nodes adjacent to the key node with the position error;

[0022] calculating a maximum probability position point of the key node with the position error according to the positions of the adjacent key nodes and prior information;

[0023] modifying the position coordinates of the key node with the position error to the coordinates of the maximum probability position point.

[0024] The four-limbed animal skeleton extraction device provided by the application includes:

[0025] one or more processors;

[0026] a memory for storing one or more programs;

[0027] When the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0028] The storage medium comprising computer executable instructions for performing the above method when executed by a computer processor.

[0029] Beneficial effects: Compared with the prior art, the present application has the following advantages: the present application provides a general four-limbed animal 3D skeleton model and a fast extraction method, adopts a skeleton extraction network based on self-attention to extract each key node defined by the user, and fully utilizes multi-scale, multi-channel and context information enhancement methods in feature extraction to improve the accuracy and stability of the output key points. The present application can be applied to various four-limbed animals, effectively improving the accuracy and practicality of animal skeleton extraction. The skeleton extracted by the present application can be applied to animal behavior recognition (walking, running, foraging, hunting, playing, lying, mating, fighting, etc.), animation synthesis, games and animal breeding, and has high practical value. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a flowchart of the four-limbed animal skeleton extraction method provided by the present application;

[0031] Figure 2 is a schematic diagram of the key nodes of the skeleton in the present application;

[0032] Figure 3 is a flowchart of the skeleton extraction in the present application;

[0033] Figure 4 is a structural schematic diagram of the four-limbed animal skeleton extraction device provided by the present application. DETAILED DESCRIPTION

[0034] Embodiment one

[0035] The present embodiment provides a four-limbed animal skeleton extraction method, as shown in Figure 1 , comprising the following steps:

[0036] (1) Obtain pictures of several four-limbed animals as samples, and mark all the key nodes and numbers of the four-limbed animals in the pictures as corresponding labels.

[0037] The pictures of the four-limbed animals can be photos obtained by taking pictures, or picture frames obtained by processing videos. Considering that the motion information of the four-limbed animals is mainly concentrated in the four-limb joints, head information and tail area, the present application defines 17 key nodes, numbered 0-16, as shown in Figure 2As shown, respectively: mouth (0), neck (1), left front hip joint (2), left front knee joint (5), left front hoof (7), spine front (3), right front hip joint (4), right front knee joint (6), right front hoof (8), left rear hip joint (9), left rear knee joint (12), left rear hoof (15), spine rear (10), tail (13), right rear hip joint (11), right rear knee joint (14), right rear hoof (16). The above key points basically cover most quadruped animals, including relatively special targets such as kangaroos with asymmetric front and rear limbs.

[0038] (2) Construct a skeleton extraction network.

[0039] The skeleton extraction network includes a backbone extraction network, a Deformable DETR network, a feature replication module, a key point regression submodule, and a skeleton label regression submodule. The backbone extraction network is specifically a swin-base network, which is used to extract coarse-grained main features from samples; the Deformable DETR network is used to extract fine-grained features A from the coarse-grained main features; the feature replication module is used to replicate the fine-grained features to obtain features B; the key point regression submodule is used to extract key node coordinates and numbers from the replicated fine-grained features, including a first branch C and a second branch D. The first branch C is a one-layer softmax layer, which is used to extract key node numbers (Types) from the fine-grained features B. The second branch D includes three fully connected layers connected in turn, which are used to extract key node coordinates (Points) from the fine-grained features. The skeleton label regression submodule is specifically a one-layer softmax layer E, which is used to extract the ID number of the quadruped animal from the fine-grained features A; for example Figure 2 There are two target objects in the middle, and the Skeleton ID is 0 and 1 respectively. The main purpose of this is to distinguish multiple objects in a picture, so that their skeletons can also be distinguished.

[0040] (3) Input all samples and corresponding labels into the skeleton extraction network for training.

[0041] (4) Input the quadruped animal picture to be extracted into the trained skeleton extraction network to obtain the key node coordinates and numbers of all quadruped animals in the picture.

[0042] (5) Repair the key points with position errors in the key points.

[0043] The repairing method specifically comprises: acquiring two or three key nodes adjacent to the key node with a position error; calculating a maximum probability position point of the key node with the position error according to positions of the adjacent key nodes and prior information; wherein the prior information is mainly a position relationship between the key nodes, for example, a front left hoof (7) is generally within a preset angle range below a front left knee joint (5), a rear left knee joint (12) is generally within a preset angle range below a rear left hip joint (9), and the maximum probability position point of the key node can be obtained according to the prior information; and modifying a position coordinate of the key node with the position error to a coordinate of the maximum probability position point.

[0044] (6) mapping the repaired key nodes into a 3D space, and connecting the key nodes in a numbering order to obtain a skeleton of a quadruped animal, as shown in Figure 3 .

[0045] Figure 3 is a structural schematic diagram of a device provided by the third embodiment of the present application, and the third embodiment of the present application provides services for implementation of the calculation method of the carbon cost considering the user-side carbon responsibility of the first embodiment of the present application, and can configure the calculation device of the carbon cost considering the user-side carbon responsibility in the first embodiment. Figure 3 A block diagram of an exemplary device 12 suitable for use in implementing embodiments of the present application is shown. Figure 3 The device 12 shown is merely one example and should not be taken as limiting the functionality or use of embodiments of the present application.

[0046] After the animal skeleton of the present application is generated, there are mainly three applications:

[0047] (1) used for animal behavior analysis: through an existing skeleton sequence analysis model, specific behaviors of an animal, such as walking, running, foraging, hunting, playing, lying, mating and fighting, can be identified.

[0048] (2) used for animation synthesis: real animal movements (i.e. skeleton movements) are used to drive animal behaviors in animation, so that the animal behaviors are more realistic.

[0049] (3) game: through skeleton movement analysis, an electronic dog can be driven to perform corresponding movements, increasing the entertainment.

[0050] (4) breeding: through the animal skeleton, the position, behavior and body shape of a bred animal can be accurately and quickly identified, and used for feed delivery, data analysis and the like.

[0051] Embodiment Two

[0052] As shown in Figure 4As shown, the device 12 is in the form of a general-purpose computing device. The components of the device 12 can include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 to the processing unit 16.

[0053] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics bus (e.g., AGP, PCI-Express bus), and a local bus using any of a variety of bus architectures. By way of example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0054] The device 12 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by the device 12 and includes both volatile and non- volatile media, removable and non-removable media.

[0055] The system memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The device 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 can be provided for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"). Figure 3 Not shown, a removable / non-removable interface can also be provided and can include at least one drive ultra / low density (e.g., a floppy disk drive) for reading from and / or writing to a removable, non- volatile magnetic medium (e.g., a "floppy disk"), and an optical disk drive for reading from and / or writing to a removable, non-volatile optical medium (e.g., a CD-ROM, DVD-ROM, or other optical media). In such instances, each drive can be connected to the bus 18 by one or more data media interfaces. The storage system 28 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application. Figure 4 The program / utility 40, having a set (at least one) of program modules 42, can be stored in, for example, the memory 28 by way of example, such program modules 42 include an operating system, one or more application programs, other program modules, and program data, each of which or a combination can include implementation of a network environment. The program modules 42 generally carry out the functions and / or methodologies of embodiments of the application as described herein.

[0056]

[0057] ​Device 12 can also communicate with one or more external devices 14 such as a keyboard, a pointing device, a display 24, etc.; one or more devices that enable a user to interact with device 12; and / or any devices (e.g., network card, modem, etc.) that enable device 12 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface(s) 22. Still yet, device 12 can communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via network adapter 20. As depicted, network adapter 20 communicates with the other components of device 12 via bus 18. It should be appreciated that although not shown, other hardware and / or software modules could be used in conjunction with device 12. Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc. Figure 4

[0058] Processing unit(s) 16 can execute instructions for a variety of functions applications and data processing, such as implementing the method for providing a carbon cost-considered offer in accordance with embodiments of the present application, by running programs stored in system memory 28.

[0059] Embodiment Three

[0060] Embodiments of the present application provide a storage medium containing computer-executable instructions for performing the method of embodiment one when executed by a computer processor,

[0061] The computer storage media of embodiments of the present application can be any media that can be read by a computer. Instead of having one or more of the computer-readable media, a combination of one or more computer-readable media can be used. A computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0062] ​A computer readable signal medium can include a propagated data signal with computer executable instructions. Such a propagated signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that can be involved in

[0063] The computer readable medium can include any medium that can store or transfer information for use by or in connection with an instruction execution system, apparatus, or device.

[0064] The computer program product or computer readable medium of the application can be a transmission medium (for example, wired or wireless) that software is downloaded and then executed (like, for example, a carrier wave of a signal).

[0065] Of course, the storage medium provided by the embodiments of the application includes computer executable instructions, and the computer executable instructions are not limited to the method operations described above, but can also perform related operations in the skeleton extraction method provided by any embodiment of the application.

Claims

1. A method for extracting the skeleton of a four-limbed animal, characterized in that... The method includes: (1) Obtain images of several four-limbed animals as samples, and mark the key nodes and numbers of all four-limbed animals in the images as corresponding labels; (2) Constructing a skeleton extraction network, the skeleton extraction network comprising: The backbone extraction network is used to extract coarse-grained main features from the sample. Deformable DETR network is used to extract fine-grained features from coarse-grained main features; The feature copying module is used to copy fine-grained features; The keypoint regression submodule is used to extract key node coordinates and numbers from the replicated fine-grained features; The skeleton marker regression submodule is used to extract the ID number of the corresponding limb animal from fine-grained features; (3) Input all samples and their corresponding labels into the skeleton extraction network for training; (4) Input the images of the four-limbed animals to be extracted into the trained skeleton extraction network to obtain the coordinates and numbers of the key nodes of all four-limbed animals in the images; (5) Repair the critical nodes that are in the wrong position; (6) Map the repaired key nodes into 3D space and connect the key nodes in the order of their numbers to obtain the skeleton of the four-limbed animal.

2. The method for extracting the skeleton of a four-limbed animal according to claim 1, characterized in that: The key nodes of the limbed animal include the mouth, neck, front left hip joint, front left knee joint, front left hoof, front of the spine, front right hip joint, front right knee joint, front right hoof, rear left hip joint, rear left knee joint, rear left hoof, rear of the spine, tail, rear right hip joint, rear right knee joint, and rear right hoof, which are numbered sequentially from 0 to 16.

3. The method for extracting the skeleton of a four-limbed animal according to claim 1, characterized in that: The backbone extraction network is specifically a swin-base network.

4. The method for extracting the skeleton of a tetrapod according to claim 1, characterized in that: The keypoint regression submodule includes a first branch and a second branch. The first branch is a softmax layer used to extract key node numbers from fine-grained features. The second branch includes three fully connected layers connected in sequence used to extract key node coordinates from fine-grained features.

5. The method for extracting the skeleton of a four-limbed animal according to claim 1, characterized in that: The skeleton marker regression submodule is specifically a softmax layer.

6. The method for extracting the skeleton of a four-limbed animal according to claim 1, characterized in that: The process of repairing critical nodes with incorrect positions specifically includes: Find two or three critical nodes adjacent to the critical node with the incorrect position; Based on the positions of adjacent critical nodes and prior information, calculate the maximum probability location of the critical node with the incorrect position. Modify the coordinates of the critical node with the incorrect location to the coordinates of the point with the highest probability.

7. A device for extracting the skeleton of a four-limbed animal, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

8. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the method as described in any one of claims 1-6.

Images