Three-dimensional skeleton estimation method, device, equipment and storage medium
By using feature extraction and logistic regression value multiplication operations in three-dimensional skeleton estimation, the problem of poor generalization of the model in the prior art is solved, and higher estimation accuracy and robustness are achieved, and the computational complexity is reduced.
Patent Information
- Application Number
- CN202111204155.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-15
- Publication Date
- 2025-03-25
- Estimated Expiration
- 2041-10-15
AI Technical Summary
The existing 3D skeleton estimation methods have poor generalization and are prone to poor estimation effects.
By inputting the monocular color image to be processed to be processed to extract the model, obtaining two-dimensional coordinates and depth values, and calculating the three-dimensional coordinates of the target skeleton through the preset estimation model, multiplication of logistic regression values and depth values is used to improve the estimation accuracy.
It improves the accuracy and robustness of three-dimensional skeleton estimation, reduces the computational complexity and cost, achieves a balance between speed and accuracy, and enhances the generalization ability of the model.
Smart Images
Figure CN113850892B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a three-dimensional skeleton estimation method, device, equipment and storage medium. Background Art
[0002] Compared with the 2D skeleton method, the 3D skeleton estimation method of monocular color images can achieve a more natural gesture interaction in three-dimensional space, and can realize an immersive interactive experience in the AR and MR fields.
[0003] Most existing 3D skeleton estimation schemes are based on machine learning and deep learning regression model methods based on depth image data, such as directly learning coordinate values or learning the offset of coordinates relative to a reference point to perform 3D skeleton estimation.
[0004] However, this estimation method is strongly dependent on the distribution of training data, and its model has poor generalization and is prone to poor estimation results. Summary of the invention
[0005] The purpose of the present application is to provide a three-dimensional skeleton estimation method, device, equipment and storage medium to address the shortcomings of the above-mentioned prior art, so as to solve the problem that the model in the prior art has poor generalization and is prone to poor estimation effect.
[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:
[0007] In a first aspect, an embodiment of the present application provides a three-dimensional skeleton estimation method, the method comprising:
[0008] Inputting the monocular color image to be processed into a preset feature extraction model to obtain the two-dimensional coordinates of the monocular color image to be processed and the depth value corresponding to each of the two-dimensional coordinates;
[0009] The target skeleton three-dimensional coordinates of the monocular color image to be processed are calculated through a preset estimation model according to each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates.
[0010] Optionally, the target skeleton of the monocular color image to be processed includes M key points, where M represents a preset number of channels and is an integer greater than 0;
[0011] The step of inputting the monocular color image to be processed into a preset feature extraction model, and obtaining the two-dimensional coordinates of the monocular color image to be processed and the depth values corresponding to the two-dimensional coordinates includes:
[0012] The monocular color image to be processed is input into a preset feature extraction model to obtain two-dimensional coordinates of M channels and M depth values corresponding to each of the two-dimensional coordinates.
[0013] Optionally, the calculating the skeleton three-dimensional coordinates of the monocular color image to be processed according to each of the two-dimensional coordinates and a depth value corresponding to each of the two-dimensional coordinates by using a preset estimation model includes:
[0014] By using a preset estimation model, according to each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates, a logistic regression value corresponding to each of the two-dimensional coordinates is calculated;
[0015] Multiply each two-dimensional coordinate by the corresponding logistic regression value and then add them together to obtain the UV coordinate corresponding to each two-dimensional coordinate;
[0016] The logistic regression value is multiplied by the depth value corresponding to each of the two-dimensional coordinates to obtain a depth value of each of the two-dimensional coordinates relative to a preset reference point.
[0017] Optionally, the calculating and obtaining the logistic regression value corresponding to each of the two-dimensional coordinates according to each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates by using a preset estimation model includes:
[0018] Stretching each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates to one dimension;
[0019] By means of a preset estimation model, the logistic regression value corresponding to each of the two-dimensional coordinates is calculated based on the stretched two-dimensional coordinates and the depth value corresponding to each of the stretched two-dimensional coordinates.
[0020] Optionally, the two-dimensional coordinates include: the resolution of the two-dimensional coordinates of each channel; the depth value includes: the number of key points of the target skeleton, the resolution and the thickness parameter;
[0021] The resolution of the two-dimensional coordinates is equal to the thickness parameter of the corresponding depth value.
[0022] Optionally, the method further comprises:
[0023] Calculating a first loss function value according to the target skeleton three-dimensional coordinates and the monocular color image data to be processed;
[0024] Calculate a second loss function value according to the two-dimensional coordinates and the logistic regression value corresponding to each of the two-dimensional coordinates;
[0025] The two-dimensional coordinates and the depth values corresponding to the two-dimensional coordinates are balanced according to the first loss function value and the second loss function value, and the two-dimensional coordinates that meet the conditions and the depth values corresponding to the two-dimensional coordinates are selected.
[0026] Optionally, calculating the second loss function value according to the two-dimensional coordinates and the logistic regression value corresponding to each of the two-dimensional coordinates includes:
[0027] Performing regression of the F-norm loss function on the two-dimensional coordinates corresponding to the monocular color image to be processed;
[0028] The second loss function value is calculated based on the regressed two-dimensional coordinates and the logistic regression values corresponding to each of the two-dimensional coordinates.
[0029] In a second aspect, another embodiment of the present application provides a three-dimensional skeleton estimation device, the device comprising: an acquisition module and a calculation module, wherein:
[0030] The acquisition module is used to input the monocular color image to be processed into a preset feature extraction model to obtain the two-dimensional coordinates of the monocular color image to be processed and the depth value corresponding to each of the two-dimensional coordinates;
[0031] The calculation module is used to calculate the target skeleton three-dimensional coordinates of the monocular color image to be processed according to each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates through a preset estimation model.
[0032] Optionally, the target skeleton of the monocular color image to be processed includes M key points, where M represents a preset number of channels and is an integer greater than 0; the acquisition module is specifically used to input the monocular color image to be processed into a preset feature extraction model to obtain the two-dimensional coordinates of the M channels and M depth values corresponding to each of the two-dimensional coordinates.
[0033] Optionally, the calculation module is specifically used to calculate the logistic regression value corresponding to each of the two-dimensional coordinates through a preset estimation model, according to each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates; multiply each two-dimensional coordinate by the corresponding logistic regression value and then add them to obtain the UV coordinate corresponding to each of the two-dimensional coordinates; multiply the logistic regression value by the depth value corresponding to each of the two-dimensional coordinates to obtain the depth value of each of the two-dimensional coordinates relative to a preset reference point.
[0034] Optionally, the device further comprises: a stretching module, configured to stretch each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates into one dimension;
[0035] The calculation module is specifically used to calculate and obtain the logistic regression value corresponding to each of the two-dimensional coordinates through a preset estimation model according to the stretched two-dimensional coordinates and the depth value corresponding to each of the stretched two-dimensional coordinates.
[0036] Optionally, the two-dimensional coordinates include: the resolution of the two-dimensional coordinates of each channel; the depth value includes: the number of key points of the target skeleton, the resolution and the thickness parameter;
[0037] The resolution of the two-dimensional coordinates is equal to the thickness parameter of the corresponding depth value.
[0038] Optionally, the device further comprises: a selection module, wherein:
[0039] The calculation module is specifically used to calculate a first loss function value according to the three-dimensional coordinates of the target skeleton and the monocular color image data to be processed; and calculate a second loss function value according to the two-dimensional coordinates and the logistic regression values corresponding to each of the two-dimensional coordinates;
[0040] The selection module is specifically used to balance each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates according to the first loss function value and the second loss function value, and select each of the two-dimensional coordinates that meet the conditions and the depth value corresponding to each of the two-dimensional coordinates.
[0041] Optionally, the calculation module is specifically used to regress the F-norm loss function on the two-dimensional coordinates corresponding to the monocular color image to be processed; and calculate the second loss function value based on the regressed two-dimensional coordinates and the logistic regression values corresponding to each of the two-dimensional coordinates.
[0042] In the third aspect, another embodiment of the present application provides a three-dimensional skeleton estimation device, comprising: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, when the three-dimensional skeleton estimation device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of any method described in the first aspect above.
[0043] In a fourth aspect, another embodiment of the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any method described in the first aspect are executed.
[0044] The beneficial effect of the present application is that the three-dimensional skeleton estimation method provided by the present application calculates the three-dimensional coordinates of the target skeleton of the monocular color image to be processed by extracting the two-dimensional coordinates of the monocular color image to be processed and the depth value of the two-dimensional coordinates through a preset estimation model. Such a method of determining the three-dimensional coordinates of the target skeleton is no longer to directly regress the two-dimensional coordinates, but to determine the three-dimensional coordinates of the target skeleton according to the depth coordinates of each two-dimensional coordinate. Such a determination method is more accurate and more robust, so that the calculation method of the three-dimensional coordinates of the target skeleton is simple, low in computational complexity, and fast in calculation speed, with strong generalization, thereby achieving a balance between calculation speed and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0046] Figure 1 A schematic diagram of a flow chart of a three-dimensional skeleton estimation method provided in one embodiment of the present application;
[0047] Figure 2 A schematic diagram of a flow chart of a three-dimensional skeleton estimation method provided in another embodiment of the present application;
[0048] Figure 3 A schematic diagram of a flow chart of a three-dimensional skeleton estimation method provided in another embodiment of the present application;
[0049] Figure 4 A schematic diagram of the structure of a three-dimensional skeleton estimation device provided in one embodiment of the present application;
[0050] Figure 5 A schematic diagram of the structure of a three-dimensional skeleton estimation device provided in another embodiment of the present application;
[0051] Figure 6 A schematic diagram of the structure of a three-dimensional skeleton estimation device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.
[0053] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0054] In addition, the flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of this application, and can also remove one or more operations from the flowchart.
[0055] The following is an explanation of a three-dimensional skeleton estimation method provided in an embodiment of the present application in combination with a number of specific application examples. Figure 1 A schematic diagram of a flow chart of a three-dimensional skeleton estimation method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:
[0056] S101: Inputting a monocular color image to be processed into a preset feature extraction model to obtain two-dimensional coordinates of the monocular color image to be processed and depth values corresponding to each two-dimensional coordinate.
[0057] In one embodiment of the present application, the monocular color image to be processed is a color system (RGB) image, the preset feature extraction model may be, for example, a lightweight feature extraction module (backbone), the monocular color image to be processed may be, for example, uploaded by a user through a terminal device, or crawled from the network, or photographed, etc. The specific method of acquiring the monocular color image to be processed can be flexibly adjusted according to user needs, and is not limited to the above embodiments.
[0058] Among them, the content included in the monocular color image to be processed needs to be determined according to the application scenario. For example, if the application scenario is the estimation of the finger skeleton, the monocular color image to be processed needs to include at least one hand image; if the application scenario is the estimation of the human body posture, the monocular color image to be processed needs to include at least one human body image. The specific content included in the monocular color image to be processed needs to be flexibly adjusted according to the application scenario, and is not limited to the above embodiments.
[0059] In some possible embodiments, the target skeleton of the monocular color image to be processed includes M key points, where M represents the preset number of channels, which is an integer greater than 0. That is, the monocular color image to be processed can be input into a preset feature extraction model to obtain the two-dimensional coordinates of the M channels (uv-heatmap) and M depth values (depth-fc) corresponding to each two-dimensional coordinate.
[0060] In one embodiment of the present application, if the application scenario is the estimation of the finger skeleton, M can be 21, for example, that is, the default number of hand skeleton coordinate points is 21, so the number of output channels of the two-dimensional coordinates of the monocular color image to be processed is also 21, wherein each hand skeleton coordinate point has its own corresponding depth value, so the number of channels corresponding to the depth value is also 21. The specific setting of M is flexibly adjusted according to the application scenario and is not limited to the above embodiment.
[0061] S102: Calculate the target skeleton three-dimensional coordinates of the monocular color image to be processed according to each two-dimensional coordinate and the depth value corresponding to each two-dimensional coordinate by using a preset estimation model.
[0062] Among them, the target skeleton is determined according to the application scenario. If the application is the estimation of the finger skeleton, the target skeleton is the hand skeleton in the monocular color image to be processed; if the application scenario is the estimation of the human body posture, the target skeleton is the human body posture skeleton in the monocular color image to be processed; the type of specific target skeleton and the correspondence with the application scenario can be flexibly adjusted according to user needs, and is not limited to the above embodiments.
[0063] In some possible embodiments, if the target skeleton needs to be displayed in a virtual scene and the parameters of the virtual camera in the virtual scene are known, the three-dimensional coordinates of the target skeleton of the monocular color image to be processed can be converted into spatial coordinate values xyz under the coordinates of the virtual camera, where xy is the coordinate value of the target skeleton in the virtual camera, and z is the depth value of the target skeleton relative to the virtual camera.
[0064] The three-dimensional skeleton estimation method provided in the present application is to calculate the three-dimensional coordinates of the target skeleton of the monocular color image to be processed by a preset estimation model, based on the extracted two-dimensional coordinates of the monocular color image to be processed and the depth value of the two-dimensional coordinates. This method of determining the three-dimensional coordinates of the target skeleton is no longer to directly regress the two-dimensional coordinates, but to determine the three-dimensional coordinates of the target skeleton based on the depth coordinates of each two-dimensional coordinate. This determination method is more accurate and more robust, so that the calculation method of the three-dimensional coordinates of the target skeleton is simple, low in computational complexity, and fast in calculation speed, with strong generalization, thereby achieving a balance between calculation speed and accuracy.
[0065] Optionally, based on the above embodiment, the embodiment of the present application can also provide a three-dimensional skeleton estimation method. The following is an example of the implementation process of calculating the skeleton three-dimensional coordinates of the monocular color image to be processed in the above method in combination with the accompanying drawings. Figure 2 A flowchart of a three-dimensional skeleton estimation method provided by another embodiment of the present application is shown as follows: Figure 2 As shown, S102 may include:
[0066] S103: Calculate and obtain the logistic regression value corresponding to each two-dimensional coordinate according to each two-dimensional coordinate and the depth value corresponding to each two-dimensional coordinate by using a preset estimation model.
[0067] In one embodiment of the present application, the calculation of the logistic regression value of the two-dimensional coordinate can be, for example, by dividing the two-dimensional coordinate by the width and height (w and h) of the monocular color image to be processed to obtain the logistic regression value in [0, 1] corresponding to the two-dimensional coordinate.
[0068] For example, in some possible embodiments, the two-dimensional coordinates include: the resolution of the two-dimensional coordinates of each channel, for example, it may include two resolutions W and H, where the resolution can be an integer value, for example; that is, the two-dimensional coordinates include: the number of key points of the target skeleton, and the output resolution W, H; the depth value includes: the number of key points of the target skeleton, the resolution and the thickness parameter; where the resolution can be, for example, 1*1, and the thickness parameter can be, for example, D, where D is an integer value; that is, the depth value includes: the number of target key points, the resolution 1*1 and the thickness parameter D.
[0069] In one embodiment of the present application, in order to facilitate calculation and reduce the amount of calculation, the resolution of the preset two-dimensional coordinates and the thickness parameter of the corresponding depth value can be set to be equal in advance, that is, W=H=D. At this time, since the preset u-coords, v-coords, and d-coords are all unit indexes of the preset coordinate positions, where u-coords is the horizontal coordinate system, v-coords is the vertical coordinate system, and d-coords is the depth coordinate system; its range is [-1,1]. For u-coords, the U coordinate of the two-dimensional coordinate in the three-dimensional coordinate can be calculated in the following way. The specific calculation method is shown as follows:
[0070]
[0071] For v-coords, the V coordinate of the two-dimensional coordinate in the three-dimensional coordinate can be calculated as follows:
[0072]
[0073] For d-coords, the D coordinate of the two-dimensional coordinate in the three-dimensional coordinate, that is, the depth coordinate, can be calculated in the following way. The specific calculation method is as follows:
[0074] [-1,…,-0.75,…,-0.5,…,0.5,…,0.75,…1].1*D.
[0075] In some possible embodiments, the calculation method of the logistic regression value corresponding to each two-dimensional coordinate can be, for example: stretching each two-dimensional coordinate and the depth value corresponding to each two-dimensional coordinate to one dimension, and arranging the two-dimensional data into one-dimensional data in this way, thereby obtaining the stretched uv-softmax. For example, for a 2*2 two-dimensional data series, it can be converted into a 1*4 one-dimensional data series by stretching; through a preset estimation model, according to the stretched two-dimensional coordinates and the depth value corresponding to each stretched two-dimensional coordinate, the logistic regression value (softmax) corresponding to each two-dimensional coordinate is calculated.
[0076] The logistic regression value is a value vector of [0, 1], which is equivalent to the probability value of the sequence position corresponding to each two-dimensional coordinate or the depth value corresponding to the two-dimensional coordinate.
[0077] S104: Multiply each two-dimensional coordinate by the corresponding logistic regression value and then add them together to obtain a UV coordinate corresponding to each two-dimensional coordinate.
[0078] By multiplying each two-dimensional coordinate by the corresponding probability value point by point and then summing them up, we can get the UV coordinates corresponding to the coordinates.
[0079] S105: Multiplying the logistic regression value by the depth value corresponding to each two-dimensional coordinate to obtain the depth value of each two-dimensional coordinate relative to a preset reference point.
[0080] By multiplying the depth value corresponding to each two-dimensional coordinate with the corresponding probability value point by point and then summing them up, the depth value of each two-dimensional coordinate relative to the preset reference point can be obtained, wherein the preset reference point is pre-set, and generally the coordinate origin (0,0,0) in the three-dimensional coordinate system is selected as the preset reference point; at this point, the three-dimensional coordinates (uvd coordinates) of the total target skeleton of the monocular color image data to be processed are obtained.
[0081] Optionally, based on the above embodiment, the embodiment of the present application may further provide a three-dimensional skeleton estimation method. The implementation process of the above method is illustrated below in conjunction with the accompanying drawings. Figure 3 A flowchart of a three-dimensional skeleton estimation method provided by another embodiment of the present application is shown as follows: Figure 3 As shown, the method also includes:
[0082] S106: Calculate a first loss function value according to the three-dimensional coordinates of the target skeleton and preset target skeleton data.
[0083] Among them, the preset training data is preset target skeleton data with preset labels. The first loss function value can, for example, be directly based on the three-dimensional coordinates of the target skeleton and the monocular color image data to be processed for regression supervision, and the mean square error MSE or smoothL1, wingLoss is used to calculate the first loss function value. It should be understood that the method for determining the first loss function value can be flexibly adjusted according to user needs, and is not limited to the above embodiments.
[0084] S107: Calculate a second loss function value according to the two-dimensional coordinates and the logistic regression value corresponding to each two-dimensional coordinate.
[0085] In an embodiment of the present application, the F-norm loss function can be regressed on the two-dimensional coordinates corresponding to the monocular color image to be processed; the second loss function value is calculated based on the regressed two-dimensional coordinates and the logistic regression values corresponding to each two-dimensional coordinate.
[0086] For example, in order to add more effective constraints, a second loss function value is added as auxiliary supervision after obtaining the two-dimensional coordinates. Specifically, the label data UV in the preset target skeleton data can be converted into a two-dimensional coordinate data format of a Gaussian kernel and then regressed with the F-norm loss function. At the same time, the probability distribution after logistic regression corresponding to the monocular color image data to be processed and the KL divergence loss value after uv-softmax can be calculated according to the similarity of the learning data distribution as the second loss function value. At the same time, in order to balance the depth branch and uv-heatmap loss, if the ratio of the two exceeds one order of magnitude, one of them is multiplied by a corresponding multiple so that the difference between the two does not exceed one order of magnitude.
[0087] In some possible embodiments, for example, if the ratio between the loss function of the depth branch and the uv-heatmap loss is greater than 10 times, the smaller loss function can be amplified and adjusted so that the ratio between the loss function of the depth branch and the uv-heatmap loss is within 10.
[0088] S108: Balance each two-dimensional coordinate and the depth value corresponding to each two-dimensional coordinate according to the first loss function value and the second loss function value, and select each two-dimensional coordinate that meets the conditions and the depth value corresponding to each two-dimensional coordinate.
[0089] By adjusting the first loss function and the second loss function, overall supervision can be achieved, which can balance the regression gradients of coordinates in all directions.
[0090] By adopting the three-dimensional skeleton estimation method provided in this application, the three-dimensional coordinates of the target skeleton can be estimated using only a monocular color image. Compared with other solutions, it saves costs and reduces the amount of calculation, is easier to implement in scenarios where mobile hardware devices are used, and has strong generalization capabilities, achieving a balance between speed and accuracy, which can meet users' daily entertainment and interactive usage scenarios.
[0091] The following is an explanation of the three-dimensional skeleton estimation device provided by the present application in conjunction with the accompanying drawings. The three-dimensional skeleton estimation device can perform the above Figure 1-Figure 3 The specific implementation and beneficial effects of any three-dimensional skeleton estimation method are as mentioned above and will not be repeated below.
[0092] Figure 4 A schematic diagram of the structure of a three-dimensional skeleton estimation device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the device includes: an acquisition module 201 and a calculation module 202, wherein:
[0093] An acquisition module 201 is used to input the monocular color image to be processed into a preset feature extraction model to obtain the two-dimensional coordinates of the monocular color image to be processed and the depth value corresponding to each two-dimensional coordinate;
[0094] The calculation module 202 is used to calculate the target skeleton three-dimensional coordinates of the monocular color image to be processed according to each two-dimensional coordinate and the depth value corresponding to each two-dimensional coordinate through a preset estimation model.
[0095] Optionally, the target skeleton of the monocular color image to be processed includes M key points, where M represents a preset number of channels and is an integer greater than 0; the acquisition module is specifically used to input the monocular color image to be processed into a preset feature extraction model to obtain the two-dimensional coordinates of the M channels and M depth values corresponding to each two-dimensional coordinate.
[0096] Optionally, the calculation module 202 is specifically used to calculate the logistic regression value corresponding to each two-dimensional coordinate through a preset estimation model, according to each two-dimensional coordinate and the depth value corresponding to each two-dimensional coordinate; multiply each two-dimensional coordinate by the corresponding logistic regression value and then add them to obtain the UV coordinate corresponding to each two-dimensional coordinate; multiply the logistic regression value by the depth value corresponding to each two-dimensional coordinate to obtain the depth value of each two-dimensional coordinate relative to a preset reference point.
[0097] Optionally, based on the above embodiment, the present application embodiment may also provide a three-dimensional skeleton estimation device, as shown below in conjunction with the accompanying drawings. Figure 4 The implementation process of the given device is described by way of example. Figure 5 A schematic diagram of the structure of a three-dimensional skeleton estimation device provided in another embodiment of the present application is shown in FIG. Figure 5As shown, the device further includes: a stretching module 203, for stretching each two-dimensional coordinate and the depth value corresponding to each two-dimensional coordinate to one dimension;
[0098] The calculation module 202 is specifically used to calculate and obtain the logistic regression value corresponding to each two-dimensional coordinate according to the stretched two-dimensional coordinates and the depth value corresponding to each stretched two-dimensional coordinate through a preset estimation model.
[0099] Optionally, the two-dimensional coordinates include: the resolution of the two-dimensional coordinates of each channel; the depth value includes: the number of key points of the target skeleton, the resolution and the thickness parameter;
[0100] The resolution of the two-dimensional coordinates is equal to the thickness parameter of the corresponding depth value.
[0101] like Figure 5 As shown, the device further includes: a selection module 204, wherein:
[0102] The calculation module 202 is specifically used to calculate the first loss function value according to the three-dimensional coordinates of the target skeleton and the monocular color image data to be processed; and calculate the second loss function value according to the two-dimensional coordinates and the logistic regression value corresponding to each two-dimensional coordinate;
[0103] The selection module 204 is specifically used to balance each two-dimensional coordinate and the depth value corresponding to each two-dimensional coordinate according to the first loss function value and the second loss function value, and select each two-dimensional coordinate that meets the conditions and the depth value corresponding to each two-dimensional coordinate.
[0104] Optionally, the calculation module 202 is specifically used to regress the F-norm loss function on the two-dimensional coordinates corresponding to the monocular color image to be processed; and calculate the second loss function value based on the regressed two-dimensional coordinates and the logistic regression values corresponding to each two-dimensional coordinate.
[0105] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.
[0106] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
[0107] Figure 6 This is a schematic diagram of the structure of a three-dimensional skeleton estimation device provided in one embodiment of the present application. The three-dimensional skeleton estimation device can be integrated into a terminal device or a chip of the terminal device.
[0108] like Figure 6 As shown, the 3D skeleton estimation device includes: a processor 501 , a storage medium 502 and a bus 503 .
[0109] The processor 501 is used to store programs. The processor 501 calls the program stored in the storage medium 502 to execute the above Figure 1-Figure 3 The corresponding method embodiment has similar specific implementation methods and technical effects, which will not be described here.
[0110] Optionally, the present application also provides a program product, such as a storage medium, on which a computer program is stored, including a program that, when executed by a processor, executes an embodiment corresponding to the above method.
[0111] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0113] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0114] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.
Claims
1. A three-dimensional skeleton estimation method, characterized in that: The method comprises: Inputting the monocular color image to be processed into a preset feature extraction model to obtain the two-dimensional coordinates of the monocular color image to be processed and the depth value corresponding to each of the two-dimensional coordinates; Calculate the target skeleton three-dimensional coordinates of the monocular color image to be processed according to each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates by using a preset estimation model; The method further comprises: Calculating a first loss function value according to the target skeleton three-dimensional coordinates and the monocular color image data to be processed; Calculate a second loss function value according to the two-dimensional coordinates and the logistic regression value corresponding to each of the two-dimensional coordinates; The two-dimensional coordinates and the depth values corresponding to the two-dimensional coordinates are balanced according to the first loss function value and the second loss function value, and the two-dimensional coordinates that meet the conditions and the depth values corresponding to the two-dimensional coordinates are selected.
2. The method according to claim 1, characterized in that The target skeleton of the monocular color image to be processed includes M key points, where M represents the preset number of channels and is an integer greater than 0; The step of inputting the monocular color image to be processed into a preset feature extraction model, and obtaining the two-dimensional coordinates of the monocular color image to be processed and the depth values corresponding to the two-dimensional coordinates includes: The monocular color image to be processed is input into a preset feature extraction model to obtain two-dimensional coordinates of M channels and M depth values corresponding to each of the two-dimensional coordinates.
3. The method according to claim 2, characterized in that The method of calculating the skeleton three-dimensional coordinates of the monocular color image to be processed according to each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates by using a preset estimation model includes: By using a preset estimation model, according to each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates, a logistic regression value corresponding to each of the two-dimensional coordinates is calculated; wherein the logistic regression value is a probability value of a sequence position corresponding to the depth value corresponding to each of the two-dimensional coordinates; Multiply each two-dimensional coordinate by the corresponding logistic regression value and then add them together to obtain the UV coordinate corresponding to each two-dimensional coordinate; The logistic regression value is multiplied by the depth value corresponding to each of the two-dimensional coordinates to obtain a depth value of each of the two-dimensional coordinates relative to a preset reference point.
4. The method according to claim 3, characterized in that The method of calculating and obtaining the logistic regression value corresponding to each of the two-dimensional coordinates according to each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates by using a preset estimation model includes: Stretching each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates to one dimension; By means of a preset estimation model, the logistic regression value corresponding to each of the two-dimensional coordinates is calculated based on the stretched two-dimensional coordinates and the depth value corresponding to each of the stretched two-dimensional coordinates.
5. The method according to any one of claims 2 to 4, characterized in that: The two-dimensional coordinates include: the resolution of the two-dimensional coordinates of each channel; the depth value includes: the number of key points of the target skeleton, the resolution and the thickness parameter; The resolution of the two-dimensional coordinates is equal to the thickness parameter of the corresponding depth value.
6. The method according to claim 1, characterized in that The calculating the second loss function value according to the two-dimensional coordinates and the logistic regression value corresponding to each of the two-dimensional coordinates includes: Performing regression of the F-norm loss function on the two-dimensional coordinates corresponding to the monocular color image to be processed; The second loss function value is calculated based on the regressed two-dimensional coordinates and the logistic regression values corresponding to each of the two-dimensional coordinates.
7. A three-dimensional skeleton estimation device, characterized in that: The device comprises: an acquisition module and a calculation module, wherein: The acquisition module is used to input the monocular color image to be processed into a preset feature extraction model to obtain the two-dimensional coordinates of the monocular color image to be processed and the depth value corresponding to each of the two-dimensional coordinates; The calculation module is used to calculate the target skeleton three-dimensional coordinates of the monocular color image to be processed according to each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates through a preset estimation model; The device also includes: a selection module, wherein: The calculation module is specifically used to calculate a first loss function value according to the three-dimensional coordinates of the target skeleton and the monocular color image data to be processed; and calculate a second loss function value according to the two-dimensional coordinates and the logistic regression values corresponding to each of the two-dimensional coordinates; The selection module is specifically used to balance each of the two-dimensional coordinates and the depth value corresponding to each of the two-dimensional coordinates according to the first loss function value and the second loss function value, and select each of the two-dimensional coordinates that meet the conditions and the depth value corresponding to each of the two-dimensional coordinates.
8. A three-dimensional skeleton estimation device, characterized in that: The device includes: a processor, a storage medium and a bus, the storage medium stores machine-readable instructions executable by the processor, when the three-dimensional skeleton estimation device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the method described in any one of claims 1-6 above.
9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is executed.