Information acquisition method and device, equipment and medium

Abstract

The embodiment of the invention relates to an information acquisition method and device, equipment and a medium. The method comprises the steps that a to-be-processed target image is acquired; wherein the target image comprises a target object; based on the target image, obtaining position information of a key point of a target part in the target object and part size information corresponding to the key point through a preset information prediction model; wherein the key point is a point located in the target part, and the part size information corresponding to the key point comprises distance information between the key point and the edge contour of the target part; the position information of the key points and the part size information corresponding to the key points are at least used for determining edge contour information of the target part. The information acquisition mode provided by the embodiment of the invention is higher in reliability, and the effect of executing the processing operation on the body part based on the acquired related information of the body part is further guaranteed.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a method, apparatus, device, and medium for obtaining information. Background Art

[0002] In some image processing scenarios, it is necessary to perform processing operations such as part annotation and part deformation on body parts of a target object such as a person. All of the above operations require prior acquisition of relevant information about the body parts to be processed. Taking the leg beautification operation in the part deformation operation as an example, it is necessary to first detect the leg edge contour, and then perform a leg slimming operation based on the leg edge contour. However, in the related art, the method of obtaining relevant information about body parts is not good, which easily leads to poor effects of subsequent processing operations on body parts based on the obtained relevant information about body parts. Summary of the Invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a method, apparatus, device, and medium for obtaining information.

[0004] In a first aspect, an embodiment of the present disclosure provides an information obtaining method, the method including: obtaining a target image to be processed, where the target image includes a target object; based on the target image, obtaining position information of key points of a target part in the target object and part size information corresponding to the key points through a preset information prediction model, where the key points are points located in the target part, and the part size information corresponding to the key points includes distance information between the key points and an edge contour of the target part; the position information of the key points and the part size information corresponding to the key points are at least used to determine edge contour information of the target part.

[0005] In a second aspect, an embodiment of the present disclosure further provides an information obtaining apparatus, including: an image obtaining module, configured to obtain a target image to be processed, where the target image includes a target object; an information obtaining module, configured to obtain position information of key points of a target part in the target object and part size information corresponding to the key points through a preset information prediction model based on the target image, where the key points are points located in the target part, and the part size information corresponding to the key points includes distance information between the key points and an edge contour of the target part; the position information of the key points and the part size information corresponding to the key points are at least used to determine the edge contour of the target part.

[0006] In a third aspect, an embodiment of the present disclosure further provides an electronic device, including: a storage device storing a computer program thereon; and a processing device configured to execute the computer program in the storage device to implement the steps of the information acquisition method provided by the embodiment of the present disclosure.

[0007] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program, and the computer program is used to execute the information acquisition method provided by the embodiment of the present disclosure.

[0008] The above technical solution provided by the embodiment of the present disclosure directly uses a preset information prediction model to obtain the position information of key points of a target part in a target object and the part size information corresponding to the key points. Herein, the key points are points located in the target part, and the part size information corresponding to the key points includes the distance information between the key points and the edge contour of the target part. In this way, the edge contour information of the target part can be determined more accurately and reliably. Compared with directly detecting the points on the edge contour in the related art to obtain the edge contour, the reliability of the above information acquisition method provided by the embodiment of the present disclosure is stronger, and it also helps to further ensure the effect of performing a processing operation on the body part based on the relevant information of the acquired body part.

[0009] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understandable through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present disclosure and used together with the specification to explain the principles of the present disclosure.

[0011] To more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the accompanying drawings required for describing the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0012] Figure 1 It is a schematic flowchart of an information acquisition method provided by an embodiment of the present disclosure;

[0013] Figure 2 It is a schematic diagram of key points provided by an embodiment of the present disclosure;

[0014] Figure 3 It is a schematic structural diagram of an information prediction model provided by an embodiment of the present disclosure;

[0015] Figure 4 Structural schematic diagram of an information prediction model provided by an embodiment of the present disclosure;

[0016] Figure 5 Specific structural schematic diagram of an information prediction model provided by an embodiment of the present disclosure;

[0017] Figure 6 Schematic diagram of key points of an arm provided by an embodiment of the present disclosure;

[0018] Figure 7 Structural schematic diagram of an information acquisition device provided by an embodiment of the present disclosure;

[0019] Figure 8 Structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0020] In order to more clearly understand the above objects, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments may be combined with each other.

[0021] Many specific details are set forth in the following description in order to provide a thorough understanding of the present disclosure, but the present disclosure may be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all of the embodiments.

[0022] In the related art, the method of obtaining relevant information of a body part is not good. Specifically, in order to perform processing operations such as beautifying and deforming a body part, the related art usually needs to first detect the edge contour of the body part. Some techniques will directly detect the body part area by using a network model with high computational cost and complex structure to obtain the edge contour of the body part, but it is usually restricted by the device hardware and not applicable to mobile devices; some techniques will directly detect several key points on the edge contour of the body part, and then determine the edge contour based on the key points detected on the edge contour. However, this method of obtaining points on the edge contour has poor effect and it is difficult to accurately determine the key points on the contour edge. Specifically, the samples used in training the model that can directly detect the key points on the edge contour usually require manual key point annotation on the edge contour, but the manual annotation accuracy is not high, and only the approximate positions can be marked on the edge contour, and even the marked key points will deviate from the edge contour. In addition, the above annotation method is also prone to ambiguity, such as the points on the outer edge contour of the arm and the points on the inner edge contour of the arm are not easy to correspond or are prone to correspondence errors. In summary, the above methods will all lead to inaccurate positions of the key points on the edge contour detected by the model trained based on the manually annotated samples, thereby affecting the accuracy of the edge contour of the body part. Therefore, the relevant information of the body part obtained in the related art, such as the key points on the edge contour and the edge contour, is not reasonable, resulting in poor effects of the related art in performing processing operations on the body part based on the obtained relevant information of the body part.

[0023] It should be noted that the above-mentioned defects existing in the related art are the results obtained by the applicant after practice and careful research. Therefore, the discovery process of the above-mentioned defects and the solutions proposed by the embodiments of the present disclosure below for the above-mentioned defects should both be regarded as the contributions made by the applicant to this application.

[0024] To improve at least one of the above problems, the embodiments of the present disclosure provide an information acquisition method, device, equipment and medium, which will be elaborated in detail below.

[0025] The embodiments of the present disclosure first provide an information acquisition method. Figure 1 As a schematic flowchart of the information acquisition method provided by the embodiments of the present disclosure, this method can be executed by an information acquisition device, where the device can be implemented by software and / or hardware and is generally integrated in an electronic device. As Figure 1 shown, this method mainly includes the following steps S102 to step S104:

[0026] Step S102: Obtain a target image to be processed; wherein, the target image contains a target object. The target object can be a person, an animal, etc., and there is no limitation here. In addition, the present disclosure embodiment does not limit the acquisition method of the target image either. For example, the target image can be an image collected by the user, or a user image captured with the user's authorization (in this case, the user is the target object), or an image selected by the user from an image library, etc.

[0027] Step S104: Based on the target image, obtain the position information of the key points of the target part in the target object and the part size information corresponding to the key points through a preset information prediction model; wherein, the key points are points located in the target part, and the part size information corresponding to the key points includes the distance information between the key points and the edge contour of the target part; the position information of the key points and the part size information corresponding to the key points are at least used to determine the edge contour information of the target part.

[0028] In some implementation examples, the target part includes but is not limited to the arm and / or the leg. Further, the arm can be further divided into the upper arm and the lower arm, and the leg can be further divided into the thigh and the calf, etc. Any part of the target object that needs to be processed subsequently can be used as the target part, and there is no limitation here. In the present disclosure embodiment, the key points are points located in the target part, that is, the key points are located inside the target part, rather than on the edge contour of the target part. In some specific implementation examples, the key points include the points on the bone midline of the target part, and the number of key points is multiple (it can be understood as at least two), and can be flexibly set according to specific requirements. For example, it can be set to 28. The edge contour of the target part includes the edge contour formed by the target part along the bone midline, and can also be understood as the longitudinal edge contour of the target part, such as the longitudinal edge contour of the arm and the longitudinal edge contour of the leg. For easy understanding, reference can be made to Figure 2 a schematic diagram of key points shown in Figure 2 which simply shows some of the key points required in the present disclosure embodiment. Taking the target part as the arm as an example for a simple illustration, and considering the symmetry of the target part, the part size information is simply indicated by the circles corresponding to the key points. Specifically, it can be simply considered that the distances from the key points on the bone midline to the two side edge contours (such as the inner edge contour of the arm and the outer edge contour of the arm) formed along the bone midline of the target part are the same. At this time, the key point is the center of the circle, and the distance from the key point to the edge contour is the radius. The transverse width of the target part can be determined based on twice the radius (that is, the diameter of the circle). Therefore, the part size information corresponding to the key points can also be characterized by the cross-sectional width of the target part corresponding to the key points.

[0029] Compared with directly detecting points on the edge contour in the related art to obtain the edge contour, the above information acquisition method provided by the embodiments of the present disclosure has stronger reliability, the method of directly detecting points in the target part is also simpler, and the accuracy of the detected key points is also higher, which also helps to further ensure the effect of performing processing operations on the body part based on the relevant information of the acquired body part.

[0030] In some embodiments, the information prediction model includes a feature extraction network, a key point prediction network, and a size prediction network. Exemplarily, referring to Figure 3 the structural schematic diagram of an information prediction model shown, the input of the feature extraction network is the target image, and the outputs of the feature extraction network are respectively used as the inputs of the key point prediction network and the size prediction network. The key point prediction network and the size prediction network respectively predict the position information of the key points and the part size information corresponding to the key points based on the outputs of the feature extraction network. Specifically, the above-mentioned three networks included in the information prediction model are further elaborated and described respectively in the following (1) to (3):

[0031] (1) The feature extraction network is used to perform multi-scale feature extraction on the target image to obtain features of multiple scales. Exemplarily, the feature extraction network includes a plurality of downsampling layers, which can perform layer-by-layer downsampling on the target image to obtain multiple features with scales from large to small. For example, perform 2-fold downsampling, 4-fold downsampling, 8-fold downsampling, and 16-fold downsampling on the target image to obtain the corresponding scale features respectively.

[0032] (2) The key point prediction network is used to predict the position information of the key points of the target part in the target object based on one or more features among the features of multiple scales.

[0033] In practical applications, one or more features among the features of multiple scales can be selected according to requirements. It can be understood that the receptive fields of features of different scales are different, and the position information of key points can be predicted more accurately and reliably through the features of multiple scales.

[0034] Exemplarily, when the key point prediction network predicts the position information of the key points of the target part in the target object based on one or more features among the features of multiple scales, it can mainly perform the following steps A to C:

[0035] Step A: Perform a first fusion process on the features of at least two target scales among multiple scales to obtain a first fusion feature. Exemplarily, the target scale may include the feature of the smallest scale among the features of multiple scales, and one or more features of the intermediate scale among the features of multiple scales. For example, the features obtained by 16-fold downsampling and the features obtained by 8-fold downsampling can be selected for the first fusion process. When specifically fusing, the features of different scales can be unified into the same scale. For example, the features obtained by 16-fold downsampling are upsampled to obtain the same size as the features obtained by 8-fold downsampling, and then fusion operations such as splicing, dot multiplication, addition, and convolution can be performed. The embodiments of the present disclosure do not limit the specific fusion method adopted for the first fusion process. The above-mentioned first fusion feature finally obtained fuses the feature information of different scales and carries more abundant and comprehensive information.

[0036] Step B: Obtain a heat map corresponding to the key points of the target part in the target object according to the first fusion feature. Exemplarily, a convolution operation can be performed on the first fusion feature to obtain a heat map corresponding to the key points of the target part in the target object. In practical applications, each key point can correspond to a heat map. Assuming there are a total of 28 key points, 28 heat maps can be obtained. Using the first fusion feature with more abundant and comprehensive information, the heat map corresponding to each key point can be obtained more accurately and reliably, which helps to further ensure the accuracy of the position prediction of the key points.

[0037] Step C: Predict the position information of the key points of the target part in the target object based on the heat map. In practical applications, a key point detection algorithm based on the heat map can be used to determine the position information of the key points of the target part in the target object. For details, reference can be made to the related technology and will not be elaborated here.

[0038] (3) The size prediction network is used to predict the part size information corresponding to the key points based on the feature of the smallest scale among the features of multiple scales. Exemplarily, the feature of the smallest scale can be the feature obtained by the above-mentioned 16-fold downsampling. Based on the feature of the smallest scale, it helps the size prediction network to predict the part size information corresponding to the key points as a whole.

[0039] In practical applications, in order to further improve the accuracy of the part size information corresponding to the key points output by the size prediction network, the size prediction network can also combine the intermediate information generated by the key point prediction network on the basis of the feature of the smallest scale among the features of multiple scales. This intermediate information is the information related to the position of the key points obtained during the process of the key point prediction network generating the position information of the key points. Exemplarily, this intermediate information can be the heat map mentioned in the foregoing (2), or the first fusion feature in the foregoing (2). On this basis, reference can be made to Figure 4Schematic structural diagram of an information prediction model shown Figure 4 Based on Figure 3 it is also shown that the size prediction network is also associated with the key point prediction network. The following are two association methods:

[0040] Method 1: If the intermediate information of the key point prediction network obtained by the size prediction network is a heat map, based on this, when the size prediction network predicts the part size information corresponding to the key point based on the feature of the smallest scale among multiple scales of features, it can predict the part size information corresponding to the key point based on the heat map corresponding to the key point and the feature of the smallest scale among multiple scales of features. Specifically, the following steps a1 and a2 can be executed:

[0041] Step a1, perform a second fusion process based on the heat map corresponding to the key point and the feature of the smallest scale among multiple scales of features to obtain a second fusion feature. In some specific implementation examples, since the heat map corresponding to the above key point is obtained from the first fusion feature, and the scales of the first fusion feature and the feature of the smallest scale are different, the heat map can be downsampled to obtain a downsampled heat map, and then fusion operations such as splicing and convolution are performed on the downsampled heat map and the feature of the smallest scale to obtain a second fusion feature.

[0042] Step a2, predict the part size information corresponding to the key point according to the second fusion feature. Since the second fusion feature carries both heat map information and the feature information of the smallest scale, the part size information predicted based on the second fusion feature is more reliable. Specifically, the size prediction network can obtain information related to the key point position by combining the heat map information. Based on this, it is more convenient and effective to correspond the key point position with the part size information, effectively improving the prediction accuracy of the part size information.

[0043] For ease of understanding, the embodiments of the present disclosure further provide a method such as Figure 5Specific structural schematic diagram of the information prediction model shown. It shows that the feature extraction network (which can also be called the backbone network) can output 16-fold downsampled features and 8-fold downsampled features. Among them, the 16-fold downsampled features, after passing through the upsampling layer, are jointly input into the first fusion layer with the 8-fold downsampled features. The first fusion layer can output the first fusion feature, and the first fusion feature is input into the heatmap processing layer. The heatmap processing layer is used to obtain the heatmap corresponding to the key points based on the first fusion feature and output the position information of the key points based on the heatmap. The heatmap output by the heatmap processing layer can be input into the downsampling layer to obtain the downsampled heatmap. The downsampled heatmap and the 16-fold downsampled features output by the feature extraction network are jointly input into the second fusion layer. The second fusion layer can output the second fusion feature. By processing the second fusion feature through the size prediction layer, the part size information corresponding to the key points can be obtained. Among them, the first fusion layer and the heatmap processing layer belong to the key point prediction network, and the downsampling layer and the size prediction layer belong to the size prediction network. In some specific implementation examples, the first fusion layer may specifically include a feature superposition layer and a convolutional layer, the downsampling layer may specifically include a 1*1 convolutional layer and a downsampling unit, the second fusion layer may specifically include a splicing layer and a 3*3 convolutional layer, and the size prediction layer may specifically include a GAP (Global average pooling) layer and an FC (Full Connection) layer. The above is only an exemplary description. In actual applications, there may be more or fewer networks, and the network layers inside the network can be flexibly adjusted according to requirements, which will not be limited here.

[0044] Method 2: If the intermediate information of the key point prediction network obtained by the size prediction network is the first fusion feature, on this basis, when the size prediction network predicts the part size information corresponding to the key points based on the features of the smallest scale among multiple scales of features, the following steps b1 to b3 can be executed:

[0045] Step b1, obtain the first prediction result of the part size information corresponding to the key points according to the first fusion feature.

[0046] Step b2, obtain the second prediction result of the part size information corresponding to the key points according to the features of the smallest scale among multiple scales of features.

[0047] Step b3, determine the part size information corresponding to the key points according to the first prediction result and the second prediction result. For example, fusion operations such as averaging the first prediction result and the second prediction result can be performed to comprehensively determine the part size information corresponding to the key points.

[0048] Exemplarily, the size prediction network includes a first size prediction unit, a second size prediction unit, and a result fusion unit. The first size prediction unit is used to obtain a first prediction result of the part size information corresponding to the key points according to the first fusion feature. The second size prediction unit is used to obtain a second prediction result of the part size information corresponding to the key points according to the feature of the smallest scale among the features of multiple scales. The result fusion unit is used to determine the part size information corresponding to the key points according to the first prediction result and the second prediction result. The structures of the first size prediction unit and the second size prediction unit may be the same or different, which is not limited herein.

[0049] In practical applications, the above-mentioned method 1 or method 2 can be flexibly selected according to needs, which is not limited herein. Through the above methods, the size prediction network can predict the size information by combining the intermediate information generated by the key point prediction network based on the feature of the smallest scale among the features of multiple scales, effectively improving the accuracy of the part size information corresponding to the key points output by the size prediction network.

[0050] The embodiment of the present disclosure also provides a method for obtaining an information prediction model. Exemplarily, the information prediction model is obtained according to the following steps 1 to 3:

[0051] Step 1: Obtain a sample image carrying label information; wherein, the sample image includes a target object, and the label information includes the position labels of the key points of the target part in the target object and the part size labels corresponding to the key points. For the sake of understanding, in some specific implementation examples, step 1 can be executed with reference to the following steps 1.1 to 1.4:

[0052] Step 1.1: Obtain a sample image including a target object.

[0053] Step 1.2: Detect the region of the target part of the target object in the sample image through pose recognition technology. The embodiment of the present disclosure does not limit the pose recognition technology, and any pose recognition technology capable of detecting the region of the target part can be used. Exemplarily, the pose recognition technology can be DensePose technology, which can establish a mapping between a 2D image and a 3D model of the target object (such as a 3D human model), and finally realize real-time pose recognition of a dense crowd. Based on DensePose technology, the reliability of the detected region of the target part in the sample image can be better ensured.

[0054] Step 1.3: Determine the key points in the target part based on the region of the target part in the sample image, obtain the position information corresponding to the key points in the sample image, and obtain the part size information corresponding to the key points in the sample image.

[0055] In order to more accurately and quickly determine key points and reduce the acquisition cost of training data, the embodiments of the present disclosure may not adopt the manual annotation method. Instead, a specific implementation example of determining key points in the target part based on the region of the target part in the sample image is further provided, which can be executed according to the following steps (1) to (2):

[0056] Step (1): Based on the region of the target part in the sample image, determine the first type of key points located in the region. Among them, the first type of key points is determined based on the main joints corresponding to the bone midline of the target part. The main joints corresponding to the bone midline can be one or more of the shoulder joint, elbow joint, wrist joint, hip joint, knee joint, etc. Correspondingly, the first type of key points can be shoulder points, elbow points, wrist points, hip points, knee points, ankle points, etc. The above method of determining the first type of key points based on the main joints corresponding to the bone midline is more convenient and accurate. Exemplarily, the above step (1) can be executed according to the following steps (1.1) and (1.2):

[0057] Step (1.1): Obtain the texture map of the 3D model corresponding to the target object in the sample image to obtain the position information of the points in the texture map corresponding to the surface of the 3D model. Among them, the 3D model is obtained by performing mapping processing on the sample image. In the case where the foregoing pose recognition technology is the DensePose technology, the DensePose technology can be used to directly establish a mapping between the sample image and the 3D model of the target object. Specifically, the DensePose technology can use deep learning to map 2D image coordinates to the surface of the 3D model of the target object, segment the target object into many UV maps (that is, the foregoing texture maps), and then process the dense coordinates to achieve accurate positioning and pose estimation of the target object. The embodiments of the present disclosure can directly obtain the position information (that is, UV values) of the points in the texture map corresponding to the surface of the 3D model based on the UV map.

[0058] Step (1.2): Based on the position information of the points in the texture map corresponding to the surface of the 3D model and the region of the target part in the sample image, determine the first type of key points located in the region. On the basis of knowing the UV values and the region of the target part in the sample image, the first type of key points located at the main joints corresponding to the bone midline of the target part can be directly determined.

[0059] Step (2): form key point pairs by two adjacent key points of the first type, and obtain the corresponding key points of the second type for each key point pair; wherein, the key points of the second type corresponding to the key point pair are located on the midline of the bone between the two key points of the first type in the key point pair. In other words, the key points of the second type are directly determined based on the key points of the first type. The key point pairs may include, for example: shoulder point - elbow point, elbow point - wrist point, hip point - knee point, knee point - ankle point. In some specific implementation examples, for each key point pair, at least one equal division point is determined based on the midline of the bone between the two key points of the first type in the key point pair, so as to determine the key points of the second type corresponding to the key point pair according to the at least one equal division point. The type of the equal division point is not limited in the embodiments of the present disclosure, and may be, for example, a bisecting point, a trisection point, a quartering point, etc., and can be flexibly set according to requirements. For ease of understanding, reference may be made to Figure 6 a schematic diagram of key points of an arm shown in

[0060] Through the above steps (1) and (2), the key points in the target part can be conveniently and reliably determined by using the above algorithm without manual annotation.

[0061] Step 1.4: attach label information to the sample image based on the position information and part size information corresponding to the key points in the sample image.

[0062] Based on the known position information and part size information corresponding to the key points in the sample image, the label information of the sample image can be determined.

[0063] Through the foregoing steps 1.1 to 1.4, a sample image with attached label information can be obtained without manual annotation. Therefore, the cost of obtaining the sample image is relatively low, and a large number of sample images can be obtained for model training according to requirements. Moreover, the label information carried by the above sample image is more reliable, which helps to train a more reliable information prediction model in terms of both the reliability of the sample image and the quantity of the sample images.

[0064] Step 2: obtain the information prediction result output by the preset neural network model for the sample image; the information prediction result includes the position prediction information and part size prediction information corresponding to the key points of the target part in the sample image.

[0065] The structure of the neural network model is the same as that of the aforementioned information prediction model, and the processing method for images is also the same. By adjusting the parameters of the neural network model, an information prediction model that can accurately output the position information of the key points of the target part and the part size information corresponding to the key points is finally obtained.

[0066] Step 3: Based on the label information and the information prediction result, train the neural network model to obtain an information prediction model based on the trained neural network model. Specifically, the parameters of the neural network model can be adjusted in the direction of reducing the difference between the label information and the information prediction result until the information prediction result of the neural network model meets the requirements, and an information prediction model is obtained.

[0067] In some specific implementation examples, Step 3 can be executed with reference to the following Steps 3.1 to 3.3:

[0068] Step 3.1: Determine the first loss based on the difference between the position label corresponding to the key point in the sample image and the position prediction information. Exemplarily, based on the difference between the position label corresponding to the key point in the sample image and the position prediction information, a preset first loss function can be used to determine the first loss. The embodiments of the present disclosure do not limit the first loss function. For example, it can be an MSE (Mean-Square Error) loss function.

[0069] Step 3.2: Determine the second loss based on the difference between the part size label corresponding to the key point in the sample image and the part size prediction information. Exemplarily, based on the difference between the part size label corresponding to the key point in the sample image and the part size prediction information, a preset second loss function can be used to determine the second loss. The embodiments of the present disclosure do not limit the second loss function. For example, it can be an L2 loss function.

[0070] Step 3.3: Based on the first loss and the second loss, train the neural network model to obtain an information prediction model based on the trained neural network model.

[0071] In practical applications, the neural network model includes a first initial network, a second initial network, and a third initial network; the input of the first initial network is a sample image, and the output of the first initial network is features of multiple scales corresponding to the sample image; the input of the second initial network is the output of the first initial network, and the output of the second initial network is position prediction information corresponding to key points in the sample image; the input of the third initial network is the output of the first initial network and the intermediate information of the second initial network, and the output of the third initial network is part size prediction information corresponding to key points in the sample image; wherein, the intermediate information is information related to the positions of the key points obtained during the process of the second initial network generating the position prediction information, such as the aforementioned heat map or the first fusion feature, etc., which will not be elaborated here.

[0072] When the structure of the neural network model is Figure 3 the structure shown, the total loss can be determined based on the first loss and the second loss, and the first initial network, the second initial network, and the third initial network can be trained simultaneously. When the structure of the neural network model is Figure 4 the structure shown, in order to improve the model training efficiency, different networks in the neural network model can be trained in stages. Specifically, step 3.3 above can be executed with reference to the following steps 3.3.1 to 3.3.4:

[0073] Step 3.3.1, based on the first loss, adjust the parameters of the first initial network and the second initial network during the first-stage training process until a preset first training stop condition is reached, and obtain the first initial network after the first-stage training and the second initial network after the first-stage training. The first training stop condition can be, for example, that the first loss converges within the range of the first preset threshold.

[0074] Step 3.3.2, based on the second loss, adjust the parameters of the third initial network during the second-stage training process until a preset second training stop condition is reached, and obtain the third initial network after the second-stage training. The second training stop condition can be, for example, that the second loss converges within the range of the second preset threshold. During the second-stage training process, the parameters of the first initial network and the second initial network after the first-stage training can be fixed. It can be understood that through the above-mentioned staged training, the first initial network and the second initial network obtained from the first-stage training can already output relatively accurate results. On this basis, the second-stage training is carried out, so that the third initial network can predict the size information based on the relatively accurate output of the first initial network and the relatively accurate intermediate information of the second initial network during the second-stage training process, which can effectively improve the second-stage training efficiency.

[0075] Step 3.3.3: Obtain a feature extraction network based on the first initial network after the first-stage training, obtain a key point prediction network based on the second initial network after the first-stage training, and obtain a size prediction network based on the third initial network after the second-stage training.

[0076] Step 3.3.4: Based on the feature extraction network, the key point prediction network, and the size prediction network, construct an information prediction model.

[0077] Through the above Steps 1 to 3, an information prediction model can be efficiently and reliably trained.

[0078] After obtaining the position information of the key points of the target part and the part size information corresponding to the key points through the information prediction model, the method provided by the embodiments of the present disclosure further includes: determining the edge points of the target part according to the position information of the key points of the target part and the part size information corresponding to the key points; based on the edge points of the target part, determining the edge contour information of the target part. It can be understood that for each key point, given the position of the key point and the part size information corresponding to the key point (including the distance information between the key point and the edge contour of the target part), the edge point corresponding to the key point can be obtained. The edge point is also the point located on the edge contour. Furthermore, based on the edge points corresponding to each key point, the edge contour of the target part can be determined.

[0079] In summary, compared with directly detecting the points located on the edge contour in the related art to obtain the edge contour, the above information acquisition method provided by the embodiments of the present disclosure has stronger reliability, and also helps to further ensure the effect of performing processing operations on the body part based on the relevant information of the acquired body part. Moreover, the training data of the information prediction model provided by the embodiments of the present disclosure does not require manual annotation, which fully guarantees the reliability of the information prediction model in terms of both the quality and quantity of the training samples.

[0080] Corresponding to the foregoing information acquisition method, the embodiments of the present disclosure provide an information acquisition device. Figure 7 As shown in the structural schematic diagram of an information acquisition device provided by the embodiments of the present disclosure, the device can be implemented by software and / or hardware, and is generally integrated in an electronic device. It can execute the information acquisition method, such as Figure 7 As shown, the information acquisition device includes:

[0081] An image acquisition module 702, configured to acquire a target image to be processed; wherein, the target image includes a target object.

[0082] An information acquisition module 704, configured to obtain the position information of the key points of the target part in the target object and the part size information corresponding to the key points based on the target image through a preset information prediction model.

[0083] Among them, the key points are the points located in the target part, and the part size information corresponding to the key points includes the distance information between the key points and the edge contour of the target part; the position information of the key points and the part size information corresponding to the key points are at least used to determine the edge contour of the target part.

[0084] Compared with directly detecting the points on the edge contour in the related art to obtain the edge contour, the reliability of the above information acquisition device provided by the embodiments of the present disclosure is stronger, and it also helps to further ensure the effect of performing processing operations on the body part based on the relevant information of the acquired body part.

[0085] In some embodiments, the key points include the points on the bone midline of the target part, and the number of the key points is multiple; the edge contour of the target part includes the edge contour formed by the target part along the bone midline.

[0086] In some embodiments, the information prediction model includes a feature extraction network, a key point prediction network, and a size prediction network;

[0087] The feature extraction network is used to perform multi-scale feature extraction on the target image to obtain features of multiple scales;

[0088] The key point prediction network is used to predict the position information of the key points of the target part in the target object based on one or more features among the multiple scales of features;

[0089] The size prediction network is used to predict the part size information corresponding to the key points based on the feature of the smallest scale among the multiple scales of features.

[0090] In some embodiments, the key point prediction network is specifically used for: performing first fusion processing on the features of at least two target scales among the multiple scales to obtain a first fusion feature; obtaining a heat map corresponding to the key points of the target part in the target object according to the first fusion feature; predicting the position information of the key points of the target part in the target object based on the heat map.

[0091] In some embodiments, the size prediction network is specifically used for: predicting the part size information corresponding to the key points based on the heat map corresponding to the key points and the feature of the smallest scale among the multiple scales of features.

[0092] In some embodiments, the size prediction network is specifically used for: performing second fusion processing on the heat map corresponding to the key points and the feature of the smallest scale among the multiple scales of features to obtain a second fusion feature; predicting the part size information corresponding to the key points according to the second fusion feature.

[0093] In some embodiments, the size prediction network is specifically configured to: obtain a first prediction result of the part size information corresponding to the key point according to the first fusion feature; obtain a second prediction result of the part size information corresponding to the key point according to the feature of the smallest scale among the multiple-scale features; and determine the part size information corresponding to the key point according to the first prediction result and the second prediction result.

[0094] In some embodiments, the apparatus further includes a model acquisition module, which is configured to obtain the information prediction model according to the following steps: obtain a sample image carrying label information, where the sample image includes a target object, and the label information includes the position label of the key point of the target part in the target object and the part size label corresponding to the key point; obtain the information prediction result output by a preset neural network model for the sample image, where the information prediction result includes the position prediction information and the part size prediction information corresponding to the key point of the target part in the sample image; and train the neural network model based on the label information and the information prediction result, so as to obtain the information prediction model based on the trained neural network model.

[0095] In some embodiments, the model acquisition module is specifically configured to: obtain a sample image including a target object; detect the region of the target part of the target object in the sample image through pose recognition technology; determine the key points in the target part based on the region of the target part in the sample image, and obtain the position information corresponding to the key points in the sample image, and obtain the part size information corresponding to the key points in the sample image; and attach label information to the sample image based on the position information and the part size information corresponding to the key points in the sample image.

[0096] In some embodiments, the model acquisition module is specifically configured to: determine, based on the region of the target part in the sample image, the first type of key points located in the region, where the first type of key points is determined based on the main joints corresponding to the bone midline of the target part; form key point pairs by combining two adjacent first type of key points, and obtain the second type of key points corresponding to each key point pair, where the second type of key points corresponding to the key point pair is located on the bone midline between the two first type of key points in the key point pair.

[0097] In some embodiments, the model acquisition module is specifically configured to: obtain a texture map of the 3D model corresponding to the target object in the sample image to obtain the position information of the points in the texture map corresponding to the surface of the 3D model; wherein the 3D model is obtained by performing a mapping process based on the sample image; based on the position information of the points in the texture map corresponding to the surface of the 3D model and the region of the target part in the sample image, determine the first type of key points located in the region.

[0098] In some embodiments, the model acquisition module is specifically configured to: for each of the key point pairs, determine at least one equal division point based on the bone midline between the two first type of key points in the key point pair, so as to determine the second type of key points corresponding to the key point pair according to the at least one equal division point.

[0099] In some embodiments, the model acquisition module is specifically configured to: determine a first loss based on the difference between the position label and the position prediction information corresponding to the key points in the sample image; determine a second loss based on the difference between the part size label and the part size prediction information corresponding to the key points in the sample image; based on the first loss and the second loss, train the neural network model, so as to obtain an information prediction model based on the trained neural network model.

[0100] In some embodiments, the neural network model includes a first initial network, a second initial network, and a third initial network; the input of the first initial network is the sample image, and the output of the first initial network is the features of multiple scales corresponding to the sample image; the input of the second initial network is the output of the first initial network, and the output of the second initial network is the position prediction information corresponding to the key points in the sample image; the input of the third initial network is the output of the first initial network and the intermediate information of the second initial network, and the output of the third initial network is the part size prediction information corresponding to the key points in the sample image; wherein the intermediate information is the information related to the positions of the key points obtained by the second initial network in the process of generating the position prediction information.

[0101] In some embodiments, the model acquisition module is specifically configured to: based on the first loss, adjust the parameters of the first initial network and the second initial network during the first-stage training process until a preset first training stop condition is reached, to obtain the first initial network after the first-stage training and the second initial network after the first-stage training; based on the second loss, adjust the parameters of the third initial network during the second-stage training process until a preset second training stop condition is reached, to obtain the third initial network after the second-stage training; obtain a feature extraction network according to the first initial network after the first-stage training, obtain a key point prediction network according to the second initial network after the first-stage training, and obtain a size prediction network according to the third initial network after the second-stage training; and construct an information prediction model based on the feature extraction network, the key point prediction network, and the size prediction network.

[0102] In some embodiments, the device further includes a contour determination module, configured to determine edge points of the target part according to the position information of the key points of the target part and the part size information corresponding to the key points; and determine edge contour information of the target part based on the edge points of the target part.

[0103] In some embodiments, the target part includes the arm and / or leg of the target object.

[0104] The information acquisition device provided by the embodiments of the present disclosure can execute the information acquisition method provided by any embodiment of the present disclosure, and has corresponding functional modules and beneficial effects for executing the method.

[0105] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the device embodiments described above can refer to the corresponding process in the method embodiments, which will not be elaborated here.

[0106] The embodiments of the present disclosure provide an electronic device, which includes: a storage device storing a computer program thereon; and a processing device configured to execute the computer program in the storage device to implement the steps of any one of the methods in the present disclosure.

[0107] Reference is made below to Figure 8 , which shows a schematic structural diagram of an electronic device 800 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8The electronic device shown is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0108] As Figure 8 shown, the electronic device 800 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage device 808 into the random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. The input / output (I / O) interface 805 is also connected to the bus 804.

[0109] Generally, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 can allow the electronic device 800 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 8 the electronic device 800 with various devices is shown, it should be understood that it is not required to implement or include all the shown devices. More or fewer devices may be implemented or included alternatively.

[0110] Specifically, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments of the present disclosure include a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above functions defined in the methods of the embodiments of the present disclosure are performed.

[0111] In addition to the above methods and devices, embodiments of the present disclosure may also be computer program products, which include computer program instructions that, when run on a processor, cause the processor to execute the image processing method provided by the embodiments of the present disclosure. The computer program products may be written in any combination of one or more programming languages for programming code to perform the operations of the embodiments of the present disclosure. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, executed as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0112] In addition, embodiments of the present disclosure may also be computer-readable storage media, on which computer program instructions are stored, and the computer program instructions, when run on a processor, cause the processor to execute the information acquisition method provided by the embodiments of the present disclosure.

[0113] The computer-readable storage media may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0114] Embodiments of the present disclosure also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the information acquisition method in the embodiments of the present disclosure.

[0115] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the users and the users' authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0116] For example, when responding to an active request received from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that performs the operations of the present disclosure's technical solution based on the prompt message.

[0117] As an optional but non-limiting implementation manner, the manner of sending a prompt message to the user in response to receiving an active request from the user may be, for example, in the form of a pop-up window. The prompt message may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0118] It can be understood that the above process of notifying and obtaining user authorization is merely illustrative and does not constitute a limitation on the implementation manner of the present disclosure. Other manners that comply with relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0119] It should be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0120] The above are only specific implementation manners of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for obtaining information, characterized in that: include: Acquire a target image to be processed; wherein the target image contains a target object; Based on the target image, obtaining position information of key points of a target part of the target object and part size information corresponding to the key points through a preset information prediction model; Among them, the key point is a point located in the target part, and the part size information corresponding to the key point includes the distance information between the key point and the edge contour of the target part; the position information of the key point and the part size information corresponding to the key point are at least used to determine the edge contour information of the target part.

2. The method according to claim 1, characterized in that The key points include points on the bone midline of the target part, and the number of the key points is multiple; the edge contour of the target part includes the edge contour of the target part formed along the bone midline.

3. The method according to claim 1, characterized in that The information prediction model includes a feature extraction network, a key point prediction network and a size prediction network; The feature extraction network is used to perform multi-scale feature extraction on the target image to obtain features of multiple scales; The key point prediction network is used to predict the position information of the key points of the target part in the target object based on one or more features of the features of the multiple scales; The size prediction network is used to predict the part size information corresponding to the key point based on the feature of the smallest scale among the features of the multiple scales.

4. The method according to claim 3, characterized in that The predicting the position information of the key points of the target part in the target object based on one or more features of the features of the multiple scales includes: Performing a first fusion process on features of at least two target scales among the multiple scales to obtain a first fused feature; Acquire a heat map corresponding to a key point of a target part of the target object according to the first fusion feature; The position information of the key points of the target part in the target object is predicted based on the heat map.

5. The method according to claim 4, characterized in that The predicting the part size information corresponding to the key point based on the feature with the smallest scale among the features with multiple scales includes: Based on the heat map corresponding to the key point and the feature of the smallest scale among the features of the multiple scales, the part size information corresponding to the key point is predicted.

6. The method according to claim 5, characterized in that The predicting the part size information corresponding to the key point based on the heat map corresponding to the key point and the feature of the smallest scale among the features of the multiple scales includes: Performing a second fusion process based on the heat map corresponding to the key point and the feature of the smallest scale among the features of the multiple scales to obtain a second fusion feature; The part size information corresponding to the key point is predicted according to the second fusion feature.

7. The method according to claim 4, characterized in that The predicting the part size information corresponding to the key point based on the feature with the smallest scale among the features with multiple scales includes: Acquire a first prediction result of the part size information corresponding to the key point according to the first fusion feature; Acquire a second prediction result of the part size information corresponding to the key point according to the feature of the smallest scale among the features of the multiple scales; The part size information corresponding to the key point is determined according to the first prediction result and the second prediction result.

8. The method according to claim 1, characterized in that The information prediction model is obtained according to the following steps: Acquire a sample image carrying label information; wherein the sample image contains a target object, and the label information includes a position label of a key point of a target part of the target object and a part size label corresponding to the key point; Obtaining information prediction results output by a preset neural network model for the sample image; the information prediction results include position prediction information and part size prediction information corresponding to key points of the target part in the sample image; Based on the label information and the information prediction result, the neural network model is trained to obtain an information prediction model based on the trained neural network model.

9. The method according to claim 8, characterized in that The obtaining of a sample image carrying label information includes: Acquire a sample image containing a target object; Detecting the area of ​​the target part of the target object in the sample image by using gesture recognition technology; Determine key points in the target part based on the area of ​​the target part in the sample image, obtain position information corresponding to the key points in the sample image, and obtain part size information corresponding to the key points in the sample image; Based on the position information and part size information corresponding to the key points in the sample image, label information is attached to the sample image.

10. The method according to claim 9, characterized in that The determining of the key points in the target part based on the area of ​​the target part in the sample image comprises: Based on the area of ​​the target part in the sample image, determining a first type of key point located in the area; wherein the first type of key point is determined based on a main joint corresponding to a skeletal midline of the target part; Two adjacent first-type key points are combined into a key point pair, and the second-type key points corresponding to each of the key point pairs are obtained; wherein the second-type key points corresponding to the key point pair are located on the bone midline between the two first-type key points in the key point pair.

11. The method according to claim 10, characterized in that The determining, based on the region of the target part in the sample image, a first type of key point located in the region comprises: Acquire a texture map of a 3D model corresponding to the target object in the sample image to obtain position information of points in the texture map corresponding to the surface of the 3D model; wherein the 3D model is obtained by mapping based on the sample image; Based on the position information of the points in the texture map corresponding to the surface of the 3D model and the area of ​​the target part in the sample image, a first type of key point located in the area is determined.

12. The method according to claim 10, characterized in that The obtaining of the second type of key points corresponding to each of the key point pairs comprises: For each of the key point pairs, at least one equally divided point is determined based on a skeleton midline between two first type key points in the key point pair, so as to determine the second type key point corresponding to the key point pair according to the at least one equally divided point.

13. The method according to claim 8, characterized in that The step of training the neural network model based on the label information and the information prediction result to obtain an information prediction model based on the trained neural network model includes: Determining a first loss based on a difference between a position label and position prediction information corresponding to a key point in the sample image; Determining a second loss based on a difference between a part size label corresponding to a key point in the sample image and the part size prediction information; Based on the first loss and the second loss, the neural network model is trained to obtain an information prediction model based on the trained neural network model.

14. The method according to claim 13, characterized in that The neural network model includes a first initial network, a second initial network and a third initial network; The input of the first initial network is the sample image, and the output of the first initial network is features of multiple scales corresponding to the sample image; The input of the second initial network is the output of the first initial grid, and the output of the second initial grid is the position prediction information corresponding to the key point in the sample image; The input of the third initial network is the output of the first initial grid and the intermediate information of the second initial grid, and the output of the third initial grid is the part size prediction information corresponding to the key point in the sample image; wherein the intermediate information is the information related to the position of the key point obtained by the second initial grid in the process of generating the position prediction information.

15. The method according to claim 14, characterized in that The step of training the neural network model based on the first loss and the second loss to obtain an information prediction model based on the trained neural network model includes: Based on the first loss, adjusting the parameters of the first initial network and the second initial network during a one-stage training process until a preset first training stop condition is reached, thereby obtaining the first initial network after the one-stage training and the second initial network after the one-stage training; Based on the second loss, adjusting the parameters of the third initial network during the two-stage training process until a preset second training stop condition is reached, thereby obtaining a third initial network after the two-stage training; Obtaining a feature extraction network according to the first initial network after the one-stage training, obtaining a key point prediction network according to the second initial network after the one-stage training, and obtaining a size prediction network according to the third initial network after the two-stage training; Based on the feature extraction network, the key point prediction network and the size prediction network, an information prediction model is constructed.

16. The method according to claim 1, characterized in that The method further comprises: Determine edge points of the target part according to position information of key points of the target part and part size information corresponding to the key points; Based on the edge points of the target part, edge contour information of the target part is determined.

17. The method according to claim 1, characterized in that The target area includes an arm and / or a leg of the target subject.

18. An information acquisition device, characterized in that: include: An image acquisition module, used to acquire a target image to be processed; wherein the target image contains a target object; An information acquisition module, used to acquire, based on the target image, position information of key points of a target part of the target object and part size information corresponding to the key points through a preset information prediction model; Among them, the key point is a point located in the target part, and the part size information corresponding to the key point includes the distance information between the key point and the edge contour of the target part; the position information of the key point and the part size information corresponding to the key point are at least used to determine the edge contour of the target part.

19. An electronic device, characterized in that: The electronic device comprises: a storage device having a computer program stored thereon; A processing device, used to execute the computer program in the storage device to implement the steps of the information acquisition method according to any one of claims 1 to 17.

20. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the information acquisition method described in any one of claims 1 to 17.

Images