Attitude parameter determination method and device and intelligent equipment

By processing images with intelligent devices and using mask loss value constraints of the pose detection model, the problem of low accuracy of pose parameters is solved, the fit between the virtual hand model and the real hand is improved, and the hand shape of different users is adapted to the hand shapes of different users, and the improvement of User interaction experience.

CN119091500APending Publication Date: 2024-12-06VIVO MOBILE COMM CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411096096.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In the prior art, in action recognition, the accuracy of posture parameters is low, and the real posture cannot be accurately characterized, resulting in the low fit between the virtual hand model and the real hand, and it is impossible to adapt to the hand shape of different users.

Method used

Images are collected by intelligent devices, and the images of interest and masks are obtained, and the trained pose detection model is input. The matching between the virtual model and the real model is constrained by the mask loss value, thereby improving the accuracy of determining pose parameters.

Benefits of technology

It realizes accurate representation of posture parameters, improves the fit between the virtual hand model and the real hand, adapts to the hand shapes of different users, and improves the user interaction experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091500A_ABST
    Figure CN119091500A_ABST
Patent Text Reader

Abstract

The invention discloses an attitude parameter determination method and device and intelligent equipment, and belongs to the technical field of intelligent equipment. The method comprises the following steps: processing a first image collected by the intelligent equipment at a first moment to obtain a first interested image and a first mask image of a target object; inputting the first interested image and the first mask image into a trained attitude detection model to obtain a first attitude parameter of the target shooting object at the first moment; wherein in the model training process, the loss value of the attitude detection model comprises a mask loss value of the target object; the mask loss value is used for representing a matching degree between the first virtual model and a second mask image of the target object in the training sample, and the mask loss value is in negative correlation with the matching degree; the first virtual model is a virtual model, corresponding to the second attitude parameter, of the target shooting object; the second attitude parameter is the attitude parameter of the target shooting object predicted by the attitude detection model based on the training sample.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the technical field of intelligent devices, and specifically relates to a method and device for determining posture parameters and an intelligent device. Background Art

[0002] With the continuous development of human-computer interaction methods, motion recognition has become one of the mainstream interaction methods in extended reality (XR) devices.

[0003] In the related art, before performing the action recognition of part 1, a model of part 1 is prefabricated. If part 1 of user a is currently being recognized, the posture parameters of part 1 of user a are predicted based on the image of part 1 of user a, and the posture parameters include a global size parameter corresponding to part 1 of user a. Afterwards, the action of part 1 of user a is recognized using the posture parameters of part 1 of user a. In specific implementation, the product of the prefabricated model of part 1 and the predicted global size parameter corresponding to part 1 of user a is first taken as the shape of part 1 of user a, and then subsequent action recognition is performed. In the part 1 model generated by the related art, the shape of part 1 of the user is controlled by the global size parameter, without taking into account that the shapes of the same part of different users are different. Therefore, the posture parameters determined by the existing method cannot accurately represent the real posture, and the accuracy of the posture parameters is low. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, apparatus and intelligent device for determining posture parameters, which can solve the problem of low accuracy of posture parameters.

[0005] In a first aspect, an embodiment of the present application provides a method for determining a posture parameter, which is executed by a smart device and includes:

[0006] Processing a first image collected by the smart device at a first moment to obtain a first image of interest and a first mask image of a target object; wherein the target object is an image of a target photographed object;

[0007] Inputting the first image of interest and the first mask image into a trained posture detection model to obtain a first posture parameter of the target object at the first moment;

[0008] Wherein, during the training process of the posture detection model, the loss value of the posture detection model includes the mask loss value of the target object; the mask loss value is used to characterize the matching degree between the first virtual model and the second mask image of the target object in the training sample, and the mask loss value is negatively correlated with the matching degree; the first virtual model is a virtual model of the target photographed object corresponding to the second posture parameter; the second posture parameter is the posture parameter of the target photographed object predicted by the posture detection model based on the training sample.

[0009] In a second aspect, an embodiment of the present application provides a posture parameter determination device, which is applied to a smart device, including:

[0010] A first acquisition module is used to process the first image collected by the smart device at the first moment to obtain a first image of interest and a first mask image of a target object; wherein the target object is an image of a target photographed object;

[0011] A second acquisition module is used to input the first image of interest and the first mask image into a trained posture detection model to obtain a first posture parameter of the target object at the first moment;

[0012] Wherein, during the training process of the posture detection model, the loss value of the posture detection model includes the mask loss value of the target object; the mask loss value is used to characterize the matching degree between the first virtual model and the second mask image of the target object in the training sample, and the mask loss value is negatively correlated with the matching degree; the first virtual model is a virtual model of the target photographed object corresponding to the second posture parameter; the second posture parameter is the posture parameter of the target photographed object predicted by the posture detection model based on the training sample.

[0013] In a third aspect, an embodiment of the present application provides an intelligent device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be executed on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0016] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0017] In an embodiment of the present application, the first image collected by the smart device at the first moment can be processed to obtain the first image of interest and the first mask image of the target object; then the first image of interest and the first mask image are input into the trained posture detection model to obtain the posture parameters of the target object at the first moment. Wherein, in the training process of the posture detection model, the loss value of the posture detection model includes the mask loss value of the target object; the mask loss value is used to characterize the matching degree between the first virtual model and the second mask image of the target object in the training sample, and the mask loss value is negatively correlated with the matching degree; the first virtual model is the virtual model corresponding to the second posture parameter of the target object; the second posture parameter is the posture parameter of the target object predicted by the posture detection model based on the training sample. It can be seen that in the training process of the posture detection model of the embodiment of the present application, the matching degree between the target object and its corresponding virtual model is constrained by the mask image of the target object in the training sample, so that the matching degree between the target object and its corresponding virtual model is high, so that the posture parameter of the target object predicted by the trained posture detection model can accurately characterize the real posture of the target object, thereby improving the accuracy of determining the posture parameter. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a diagram of the architecture for obtaining a virtual hand model provided by the related technology;

[0019] Figure 2 It is a comparison diagram between a virtual hand model and a real hand provided by the relevant technology;

[0020] Figure 3 is a flow chart of a method for determining posture parameters provided in an embodiment of the present application;

[0021] Figure 4 This is a diagram of an architecture for obtaining a virtual hand model provided in an embodiment of the present application;

[0022] Figure 5 It is a result diagram of each step provided in the embodiment of the present application;

[0023] Figure 6 This is the third diagram of the architecture for obtaining a virtual hand model provided in the embodiment of the present application;

[0024] Figure 7 is a structural diagram of a posture parameter determination device provided in an embodiment of the present application;

[0025] Figure 8It is one of the structural diagrams of the electronic device provided in the embodiment of the present application;

[0026] Fig. 9 This is the second structural diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the target object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0029] For ease of understanding, some contents involved in the embodiments of the present application are described below.

[0030] With the continuous development of human-computer interaction, action recognition, such as gesture recognition, has become one of the mainstream interaction methods in XR devices. For ease of understanding, the following uses gesture recognition as an example.

[0031] XR gesture recognition solutions in related technologies include Figure 1 As shown. First, the hand frame in the fisheye grayscale image is selected through the hand frame detection network to obtain the hand's region of interest (ROI). Then, the hand ROI is input into the hand skeleton key point detection network to predict the two-dimensional (2D) coordinates of the 21 skeleton key points of the hand, and then the hand three-dimensional (3D) posture solution module is used to calculate the hand posture parameters. The hand posture parameters include: wrist 6 degrees of freedom information (3D rotation, 3D translation); finger joint angles, 4 degrees of freedom for each finger, a total of 20; global size parameter φ hand .

[0032] Finally, these hand posture parameters are used to drive the movement and rendering of the virtual hand model. When driving the virtual hand model, the prefabricated hand model is first multiplied by the global size parameter φ handThen, the positions of each vertex of the virtual hand model are calculated by combining the 6-DOF information of the wrist, the 20 joint rotation angles of the fingers and the mesh skin matrix.

[0033] The virtual hand mask driven by the above solution cannot well reflect the shape of the user's real hand. For example, in the video see-through (VST) scene, the user can clearly feel that the virtual hand model cannot fit the real hand well, which brings a strong sense of unreality. For easier understanding, please refer to Figure 2 ,exist Figure 2 In the example, the virtual hand model 101 and the real hand 102 do not fit well. The fit can also be called matching degree.

[0034] The above solution cannot be well adapted to the hand shapes of users with various distributions, which is reflected in the following two points:

[0035] First, the shape of the virtual hand model does not conform to the actual hand shape of the user. When driving the virtual hand model, the above method uses a global size parameter to control the size, which does not change the shape of the virtual hand model. For example, if the virtual hand model is originally thin and long, but the user's hand is fat, no matter how the overall size of the hand is changed, the two hands cannot fit together well.

[0036] Secondly, the user's hand shape is not taken into account when predicting and calculating the finger joint angles and the 6-DOF information of the wrist. In other words, no special constraints are imposed on fitting the user's actual hand shape, resulting in large errors in the final presentation.

[0037] In summary, the above-mentioned gesture recognition schemes have major defects in terms of both the fitting of the hand shape and the accuracy of the hand posture.

[0038] Based on this, an embodiment of the present application provides a new method for determining posture parameters, which can improve the accuracy of posture parameter determination.

[0039] The following is a detailed description of the posture parameter determination method provided in the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0040] The posture determination method of the embodiment of the present application can be executed by a smart device, which can be any device integrated with a camera, such as an electronic device or a vehicle, etc. The electronic device can further be an XR device or a mobile phone, etc. Figure 3 As shown, the posture parameter determination method of the embodiment of the present application may include:

[0041] Step 301: Process a first image acquired by the smart device at a first moment to obtain a first image of interest and a first mask image of a target object; wherein the target object is an image of a target photographed object.

[0042] It is worth noting that, in practical applications, the smart device may include at least one camera. When the number of cameras is greater than 1, further, the types of different cameras may be the same or different. In some examples, some cameras may be cameras for capturing color images (such as RGB images), and such cameras may be referred to as VST cameras, and color images may also be referred to as VST images; some cameras may be cameras for capturing grayscale images, such as fisheye cameras, and the images captured by fisheye cameras are fisheye grayscale images. The smart device may capture images through some or all of its own cameras, and it is understood that the smart device may capture at least one first image at the first moment.

[0043] The purpose of the posture parameter determination method of the embodiment of the present application is to determine the posture parameters of the target photographed object. Therefore, the imaging of the target photographed object, that is, the target object is an object of particular concern in the embodiment of the present application, that is, an object of interest. It can be understood that the first image includes the target object.

[0044] In the embodiment of the present application, the target photographing object may be the whole body or a part of the body of an entity, and the entity may be: a living being, such as a human or an animal, etc.; or an artificial body, such as a robot, etc. The target photographing object may be predetermined, and the embodiment of the present application does not limit the specific performance of the target photographing object. In other words, the posture parameter method of the embodiment of the present application can be used to determine the posture parameters of a part or the whole of any entity.

[0045] As can be seen from the above content, the target object is an object of particular interest in the embodiment of the present application. In order to reduce the interference of other objects in the first image on the determination of the posture parameters of the target photographed object, in step 201, on the one hand, the area where the target object is located in the first image can be used as the first region of interest of the target object. It is worth noting that the area of ​​the first region of interest is larger than the area of ​​the target object, and the outline of the first region of interest can be the same as or different from the outline of the target object, such as the outline of the first region of interest can be a rectangle or a circle. On the other hand, the target object in the first image can be used as the first mask image of the target object. It is worth noting that the area of ​​the first mask image is equal to the area of ​​the target object, and the outline of the first mask image is the same as the outline of the target object. It can be understood that obtaining the first mask image of the target object in the first image is equivalent to obtaining the outline of the target photographed object at the first moment. In the embodiment of the present application, the mask image can also be called a mask image or a mask image.

[0046] In some embodiments, when the number of first images is greater than 1 and the first images are all grayscale images, for each grayscale image, a first image of interest and a first mask image can be processed respectively. It can be understood that in these embodiments, the number of first images of interest and the number of first mask images are equal to the number of first images.

[0047] In other embodiments, when the number of first images is greater than 1, and the first images include color images and grayscale images, since the shape of the target object presented in the color image is more consistent with the shape of the target object, the grayscale image can be used to obtain the first region of interest, and the color image can be used to obtain the first mask image, so that the accuracy of determining the posture parameters of the target object can be further improved. It can be understood that in these embodiments, the number of first images of interest is equal to the number of grayscale images in the first image, and the number of first mask images is equal to the number of color images in the first image.

[0048] Step 302: Input the first image of interest and the first mask image into a trained posture detection model to obtain a first posture parameter of the target object at the first moment.

[0049] Wherein, during the training process of the posture detection model, the loss value of the posture detection model includes the mask loss value of the target object; the mask loss value is used to characterize the matching degree between the first virtual model and the second mask image of the target object in the training sample, and the mask loss value is negatively correlated with the matching degree; the first virtual model is a virtual model of the target photographed object corresponding to the second posture parameter; the second posture parameter is the posture parameter of the target photographed object predicted by the posture detection model based on the training sample.

[0050] In the embodiment of the present application, a virtual model of the target object can be rendered based on the posture parameters of the target object. Therefore, there is a corresponding relationship between the posture parameters of the target object and the virtual model.

[0051] The matching degree between the first virtual model and the second mask image can be determined based on the positional relationship between the mesh vertices of the first virtual model and the second mask image. Specifically, the matching degree between the first virtual model and the second mask image is positively correlated with the number of mesh vertices in the first virtual model whose projection positions are located in the second mask image, that is, the more mesh vertices in the first virtual model whose projection positions are located in the second mask image, the greater the matching degree between the first virtual model and the second mask image of the target object in the training sample, and vice versa. The mesh vertices of the virtual model of the target object are the mesh vertices corresponding to the target object.

[0052] Based on this, in the training process of the posture detection model, after the posture detection model predicts the posture parameters of the target object, the 3D position of each mesh vertex corresponding to the target object can be inferred by the predicted posture parameters, and then the positional relationship between each mesh vertex and the second mask image can be determined by using the 2D projection position of the 3D position of each mesh vertex inferred. Then, the positional relationship between each mesh vertex and the second mask image can be used to determine the mask loss value of the target object and the loss value of the posture detection model. Specifically, the more the number of mesh vertices whose projection positions fall into the second mask image, the higher the matching degree between the first virtual model and the second mask image, and the smaller the mask loss value of the target object and the loss value of the posture detection model; the fewer the number of mesh vertices whose projection positions fall into the second mask image, the lower the matching degree between the first virtual model and the second mask image, and the larger the mask loss value of the target object and the loss value of the posture detection model. Then, based on whether the loss value of the posture detection model is less than the loss threshold, it is determined whether the posture detection model is well trained. Specifically, if the loss value of the posture detection model is less than the loss threshold, it is determined that the posture detection model is well trained; if the loss value of the posture detection model is greater than or equal to the loss threshold, it is determined that the posture detection model is not well trained.

[0053] Since the contour of the mask image of the target object is the same as the contour of the target photographed object, the matching degree between the first virtual model and the second mask image can represent the matching degree between the first virtual model and the target photographed object.

[0054] It can be seen that in the training process of the posture detection model in the embodiment of the present application, the matching degree between the target photographed object and its corresponding virtual model is constrained by the mask image of the target object in the training sample, so that the matching degree between the target photographed object and its corresponding virtual model is relatively high. In this way, the posture parameters of the target photographed object predicted by the trained posture detection model can accurately represent the true posture of the target photographed object, thereby improving the accuracy of determining the posture parameters.

[0055] It should be noted that, since the matching degree between the first virtual model and the second mask image can be determined based on the positional relationship between the mesh vertices of the first virtual model and the second mask image, it can be understood that in the training process of the posture detection model, the positions of each mesh vertex corresponding to the target object can be inferred by the predicted posture parameters, and the rendering operation of the virtual model of the target object does not need to be performed, that is, the first virtual model does not need to be rendered.

[0056] In the embodiment of the present application, the input of the posture detection model includes at least an image of interest and a mask image of the target object.

[0057] In some embodiments, when the input image of interest and mask image of the target object are determined based on different cameras of the smart device, the input of the posture detection model may also include external parameters of these cameras to eliminate posture parameter determination errors caused by differences in camera external parameters.

[0058] In some embodiments, the input of the posture detection model may also include the posture parameters of the target photographed object at the previous moment before the first moment predicted by the trained posture detection model. In this way, on the one hand, the prediction effect of the posture detection model when the target object in the first image is occluded can be improved; on the other hand, the smoothness of the posture parameters output by the posture detection network at adjacent moments can be improved.

[0059] The embodiment of the present application does not limit the format of the input information of the posture detection model. In some embodiments, the above information can be directly input into the posture detection model. In other embodiments, the above information can be first converted into vectors respectively, and then the vectors are spliced ​​and input into the posture detection model.

[0060] The embodiments of the present application do not limit the format of the posture parameters output by the posture detection model. Any posture parameters that can accurately represent the true posture of the target photographed object can fall within the protection scope of the embodiments of the present application.

[0061] In some embodiments, the posture parameters of the target object may include: a shape parameter (β) of the target object, a rotation parameter (θ) of the target object, and a motion parameter of the target object relative to a main camera of the smart device.

[0062] The shape parameters of the target object may be used to determine the original shape of the target object.

[0063] The rotation parameters of the target object may include the rotation parameters of each skeleton key point of the target object. The rotation parameters of each skeleton key point may be composed of 3 degrees of freedom axis angle (θ x ,θ y ,θ z The rotation parameters of the target object can be used to determine the predicted positions of the skeleton key points of the target object in the coordinate system of the target object.

[0064] The motion parameters of the target object relative to the main camera of the smart device may include parameters such as translation R and rotation t of the target object relative to the main camera. The motion parameters of the target object relative to the main camera may be used to determine the predicted position of the target object in the main camera coordinate system.

[0065] The posture determination method of the embodiment of the present application can first process the first image collected by the smart device at the first moment to obtain the first image of interest and the first mask image of the target object; then input the first image of interest and the first mask image into the trained posture detection model to obtain the posture parameters of the target shooting object at the first moment. Wherein, in the training process of the posture detection model, the loss value of the posture detection model includes the mask loss value of the target object; the mask loss value is used to characterize the matching degree between the first virtual model and the second mask image of the target object in the training sample, and the mask loss value is negatively correlated with the matching degree; the first virtual model is the virtual model corresponding to the second posture parameter of the target shooting object; the second posture parameter is the posture parameter of the target shooting object predicted by the posture detection model based on the training sample. It can be seen that in the training process of the posture detection model of the embodiment of the present application, the matching degree between the target shooting object and its corresponding virtual model is constrained by the mask image of the target object in the training sample, so that the matching degree between the target shooting object and its corresponding virtual model is high, so that the posture parameter of the target shooting object predicted by the trained posture detection model can accurately characterize the real posture of the target shooting object, thereby improving the accuracy of the determination of the posture parameter.

[0066] In some embodiments of the present application, after inputting the first image of interest and the first mask image into a trained posture detection model to obtain a first posture parameter of the target photographed object at the first moment, the method may further include:

[0067] Determine, using the first posture parameter, predicted positions of N mesh vertices corresponding to the target photographed object at the first moment in a first coordinate system; wherein the first coordinate system is a coordinate system of a main camera of the smart device, and N is an integer greater than 1;

[0068] Rendering and displaying a second virtual model of the target photographed object corresponding to the first posture parameter using the predicted positions of the N mesh vertices in the first coordinate system at the first moment;

[0069] The outline of the second virtual model matches the outline of the target photographed object at the first moment.

[0070] In these embodiments, after the smart device predicts the first posture parameters at the first moment using the trained posture detection model, it can further use the first posture parameters to render and display a second virtual model of the target object corresponding to the first posture parameters.

[0071] In a specific implementation, the first posture parameter may be used to first determine the predicted position of each mesh vertex corresponding to the target object at the first moment in the main camera coordinate system. Then, the predicted position of each mesh vertex corresponding to the target object at the first moment in the main camera coordinate system may be used to render a second virtual model of the target object corresponding to the first posture parameter on the screen of the smart device.

[0072] Since the first posture parameter can accurately represent the real posture of the target object at the first moment, it can be understood that the second virtual model is close to the real state of the target object at the first moment, and the outline of the second virtual model is the same or basically the same as the outline of the target object at the first moment, which can be regarded as a match.

[0073] In this way, the virtual model of the target object rendered and displayed by the smart device can accurately represent the real posture of the target object, thereby improving the accuracy of the virtual model of the target object.

[0074] In practical applications, for posture parameters of different formats, the predicted positions of each mesh vertex corresponding to the target object in the main camera coordinate system may be different. The specific implementation can be set based on actual needs, and the embodiments of the present application are not limited to this.

[0075] In a case where the posture parameters of the target object include: a shape parameter of the target object, a rotation parameter of the target object, and a motion parameter of the target object relative to a main camera of the smart device, optionally, using the first posture parameters to determine predicted positions of N mesh vertices corresponding to the target object at the first moment in the first coordinate system may include:

[0076] Using the first shape parameter in the first posture parameter, converting the default positions of N mesh vertices corresponding to the target object in a second coordinate system to obtain original positions of the N mesh vertices in the second coordinate system, wherein the second coordinate system is the coordinate system of the target object;

[0077] Determine the original positions of M skeleton key points corresponding to the target photographed object in the second coordinate system by using the original positions of the N mesh vertices in the second coordinate system, wherein the original position of each skeleton key point in the M skeleton key points is determined based on the original position of at least two mesh vertices in the N mesh vertices; M is an integer greater than 1;

[0078] Using the first rotation parameter in the first posture parameter, the original positions of the M skeleton key points in the second coordinate system are transformed to obtain the predicted positions of the M skeleton key points in the second coordinate system at the first moment;

[0079] Determine the predicted positions of the N mesh vertices in the second coordinate system at the first moment by using the predicted positions of the M skeleton key points in the second coordinate system at the first moment;

[0080] The predicted positions of the N mesh vertices in the second coordinate system at the first moment are transformed using the first motion parameter in the first posture parameter to obtain the predicted positions of the N mesh vertices in the first coordinate system at the first moment.

[0081] In this embodiment, for the target body part corresponding to the target photographed object, on the one hand, a default shape of the target body part can be preset. On the other hand, the K default shapes of the target body parts can be pre-set by prior knowledge. The offset shape k In this case, the number of shape parameters β in the model parameters is K, and K is a positive integer.

[0082] It can be understood that the target body parts of different life forms are unique, and the target shooting object in the embodiment of the present application is the target body part of a certain life form. In order to make the virtual model of the target shooting object rendered and displayed match the actual shape of the target shooting object, the first shape parameter included in the first posture parameter can be used to first obtain the original shape T of the virtual model of the target shooting object.

[0083] In specific implementation, the original shape T of the virtual model of the target object can be calculated by formula (1):

[0084]

[0085] From formula (1), we can see that the shape parameter β k It indicates the target object's k The weight of k ranges from 1 to K. The original shape T of the virtual model of the target object calculated by formula (1) matches the actual shape of the target object.

[0086] Default shape and the original shape T of the virtual model of the target object both include N mesh vertices. The default positions of the N mesh vertices corresponding to the target object in the second coordinate system are the positions of the N mesh vertices included in the default shape; the original positions of the N mesh vertices corresponding to the target object in the second coordinate system are the positions of the N mesh vertices included in the original shape T of the virtual model of the target object. Based on this, the original positions of the mesh vertices corresponding to the target object can be obtained based on the original shape T of the virtual model of the target object, and the original position of the nth mesh vertex is recorded as T n .

[0087] After obtaining the original positions of the mesh vertices corresponding to the target object, the original positions of the skeleton key points corresponding to the target object in the second coordinate system can be calculated by formula (2) based on the correspondence between the mesh vertices corresponding to the target object and the skeleton key points:

[0088] L original =∑ i JW i ×T i (2)

[0089] In formula (2), L original Indicates the original position of a certain skeleton key point corresponding to the target object in the second coordinate system. Each skeleton key point corresponding to the target object corresponds to a mesh vertex set, and each mesh vertex set includes at least one mesh vertex. i Indicates the original position of the i-th mesh vertex in the mesh vertex set corresponding to the skeleton key point in the second coordinate system; i It represents the preset weight of the ith mesh vertex in the mesh vertex set corresponding to the bone key point, and represents the degree of association between the ith mesh vertex in the mesh vertex set corresponding to the bone key point and the bone key point.

[0090] After calculating the original position of the skeleton key point corresponding to the target object in the second coordinate system, the predicted position of the skeleton key point corresponding to the target object in the second coordinate system at the first moment can be calculated by using the first rotation parameter in the first posture parameter through the forward dynamics method. In specific implementation, the predicted position of the skeleton key point corresponding to the target object in the second coordinate system can be calculated by formula (3):

[0091] L′=L parent +A2R(θ x ,θ y ,θ z )×L original (3)

[0092] In formula (3), L′ represents the predicted position of a certain skeleton key point corresponding to the target object in the second coordinate system. parent Indicates the parent joint point of the bone key point. (θ x ,θ y ,θ z ) represents the 3-DOF axis angle of the key point of the skeleton. A2R is used to convert the axis angle into a rotation matrix. original Indicates the original position of the key point in the second coordinate system. In the second coordinate system, the highest parent joint point of the skeleton key point corresponding to the target object can be used as the origin.

[0093] After calculating the predicted position of the skeleton key points corresponding to the target object in the second coordinate system at the first moment, the linear skinning algorithm LBS can be used to obtain the predicted position of the N mesh vertices in the second coordinate system at the first moment. In specific implementation, the predicted position T′ of the N mesh vertices in the second coordinate system at the first moment can be calculated by formula (4): local .

[0094] T′ local = LBS(L′ local ) (4)

[0095] In formula (4), L′ local Represents the predicted positions of the M skeleton key points corresponding to the target object in the second coordinate system.

[0096] The predicted positions T′ of the N mesh vertices in the second coordinate system at the first moment are calculated. local Afterwards, the first motion parameter R in the first posture parameter can be used l2c , calculate the predicted position T of the N mesh vertices in the main camera coordinate system of the smart device at the first moment _main_camera In specific implementation, T can be calculated by formula (5): _main_camera :

[0097] T _main_camera =R l2c T′ local (5)

[0098] The predicted positions of the N mesh vertices in the main camera coordinate system of the smart device at the first moment calculated in the above manner can make the shape of the rendered virtual model of the target object match the actual posture of the target object, thereby improving the determination accuracy of the virtual model of the target object.

[0099] The following is a detailed description of the training of the posture detection network in the embodiment of the present application.

[0100] In some embodiments, before inputting the first image of interest and the first mask image into a trained posture detection model to obtain a first posture parameter of the target photographed object at the first moment, the method further includes:

[0101] Acquire a training sample; wherein the training sample includes a second image of interest and the second mask image of the target object; the second image of interest and the second mask image are obtained by processing a second image collected by the smart device at a second moment; the second moment is before the first moment;

[0102] Inputting the training sample into a posture detection model to be trained to obtain the second posture parameter;

[0103] Determine predicted positions of N mesh vertices of the first virtual model in a first coordinate system using the second posture parameter, wherein the first coordinate system is a coordinate system of a main camera of the smart device, and N is an integer greater than 1;

[0104] In a case where the smart device includes at least two cameras, using the predicted positions of the N mesh vertices in the first coordinate system, obtaining the projection positions of the N mesh vertices in the coordinate systems of each camera in the smart device;

[0105] Determining a mask loss value of the target object according to a positional relationship between the projection position and the second mask image;

[0106] Determining a loss value of the posture detection model according to a mask loss value of the target object;

[0107] When the loss value is greater than or equal to the loss threshold, the model parameters and training samples of the posture detection model are updated, and the step of inputting the training samples into the posture detection model to be trained is returned to obtain the second posture parameters, until the loss value of the posture detection model is less than the loss threshold, thereby obtaining a trained posture detection model.

[0108] In the training phase of the posture detection model, the posture detection model can be trained using training samples. For each training sample, the above steps can be performed until the loss value of the posture detection model is less than the loss threshold, thereby obtaining a trained posture detection model.

[0109] It should be noted that the specific implementation of "using the second posture parameters to determine the predicted positions of the N mesh vertices of the first virtual model in the first coordinate system" is similar to the specific implementation of "using the first posture parameters to determine the predicted positions of the N mesh vertices corresponding to the target shooting object at the first moment in the first coordinate system". For details, please refer to the aforementioned related description, which will not be repeated here.

[0110] When the smart device includes at least two cameras, each camera has its own coordinate system. In order to ensure the reliability of determining the positional relationship between the mesh vertices and the mask image of the target object, when determining the positional relationship between the mesh vertices and the mask image of the target object, it is necessary to ensure that the projection position of the mesh vertices used and the mask image are in the same camera coordinate system, as described below.

[0111] 1. The number of second images is greater than 1, and all second images are grayscale images.

[0112] We can first use the external parameters of each camera to transform the predicted positions of N mesh vertices in the first coordinate system to the coordinate systems of each camera using formula (6):

[0113]

[0114] In formula (6), Represents the predicted positions of N mesh vertices in the coordinate system of the jth camera. j Represents the transformation matrix from the main camera to the j-th camera, which is determined based on the external parameters of the main camera and the j-th camera.

[0115] Afterwards, the predicted positions of the N mesh vertices in each camera coordinate system can be projected onto the 2D image of each camera perspective using formula (7) to obtain the projected positions of the N mesh vertices in the camera coordinate system:

[0116]

[0117] In formula (7), and represents the projection position of N mesh vertices in the coordinate system of the jth camera, P j represents the projection equation of the jth camera. Combining the above content, we can know that in, Identifies the projected position of the nth mesh vertex in the jth camera's coordinate system.

[0118] Then, a second mask image of the target object obtained by processing the grayscale image captured by the jth camera is obtained. The second mask image can be called the mask image corresponding to the jth camera. are all in the coordinate system of the jth camera, so we can judge Whether it falls in the second mask image is used to determine the mask loss value of the target object corresponding to the jth camera.

[0119] In specific implementation, the mask loss value of the target object It can be calculated by formulas (8) and (9):

[0120]

[0121] From (8) and (9), we can see that when the projection position of a mesh vertex in the coordinate system of a camera is not in the second mask image corresponding to the camera, the base value of the mask loss value of the target object is will be added by 1; in the second mask image corresponding to the camera at the projection position of a certain mesh vertex in the coordinate system of a certain camera, the base value of the mask loss value of the target object will be added by 0. Afterwards, the calculated base value and λ can be used to mask Multiply them together to get the mask loss value of the target object. mask Can be preset.

[0122] It should be noted that 0 and 1 in formula (9) are only examples. In practical applications, 0 can be replaced by a, 1 can be replaced by b, and a and b satisfy that a is less than b. The values ​​of a and b are set based on specific requirements and are not limited in this embodiment of the present application.

[0123] 2. When the number of second images is greater than 1 and the second images include color images and grayscale images, since the second mask image is obtained based on the color image, the predicted positions of the N mesh vertices in the main camera coordinate system can be used to obtain the projection positions of the N mesh vertices in the VST camera coordinate system; then the positional relationship between these projection positions and the second mask image is determined to determine the mask loss value of the target object. The specific implementation is similar to the above and will not be repeated here.

[0124] In addition, when the smart device only includes a main camera, the predicted positions of the N mesh vertices in the main camera coordinate system can be used to obtain the projection positions of the N mesh vertices in the main camera coordinate system; then the positional relationship between these projection positions and the second mask image is determined to determine the mask loss value of the target object. The specific implementation is similar to the above and will not be repeated here.

[0125] The mask loss value of the target object may also be referred to as the mask loss item of the posture detection model. After the mask loss item of the posture detection model is obtained, the loss value of the posture detection model may be determined using the mask loss item of the posture detection model.

[0126] In some implementations, the value of the mask loss term may be directly used as the loss value of the pose detection model.

[0127] In other embodiments, the mask loss term may be combined with other loss terms to determine the loss value of the gesture detection model. The loss value of the gesture detection model may be the sum of the values ​​of all loss terms.

[0128] In some implementations, the loss value of the pose detection model may also be determined based on a value of at least one of the following: a 3D loss term, a 2D loss term, a multi-view geometry parameter, and a regularization term.

[0129] 3D loss term It can be calculated by formula (10):

[0130]

[0131] In formula (10), J(θ p , β p , R p , t p ) n is the predicted position of the nth skeleton key point in the main camera coordinate system, is the 3D point true value of the nth skeleton key point in the jth camera coordinate system. This constraint can make the prediction result of the pose detection model as close as possible to the 3D data true value of the training sample. In this case, the training sample also includes the 3D true value of the skeleton key point corresponding to the target object.

[0132] 2D loss term It can be calculated by formula (11):

[0133]

[0134] Similar to the 3D loss term, constraints can also be placed on the 2D projection points. is the 2D point true value of the nth skeleton key point in the jth camera coordinate system. This makes it possible to make the projection position of the prediction result of the posture detection model as close as possible to the 2D data true value of the training sample. In this case, the training sample also includes the 2D true value of the skeleton key point corresponding to the target object.

[0135] Multi-view geometry parameters It can be calculated by formula (12):

[0136]

[0137] R, t represent the rotation and translation of the target object in the main camera coordinate system, R p ,t p Represents the prediction result of the posture detection model, R g ,t g In this case, the training sample also includes R g ,t g .

[0138] Regularization term It can be calculated by formula (13):

[0139]

[0140] Through the regularization term, the pose parameters of the target object can be regularized and constrained to prevent overfitting.

[0141] In some embodiments, the loss value of the posture detection model can be calculated by formula (14):

[0142]

[0143] The posture detection model trained in the above manner can make the posture parameters of the target object output by the trained posture detection model accurately represent the actual posture of the target object, thereby improving the prediction reliability of the posture detection model.

[0144] The embodiments of the present application do not limit the method of acquiring the image of interest and the mask image of the target object.

[0145] The image of interest of the target object can be obtained by cropping the image collected by the camera. In some embodiments, the cropping process can be implemented by a cropping process algorithm. In other embodiments, the cropping process can be implemented by a cropping model of the target object.

[0146] The mask image of the target object can be obtained by performing a cutout process on the image collected by the camera. In some embodiments, the cutout process can be implemented by a cutout process algorithm. In other embodiments, the cutout process can be implemented by a cutout model of the target photographed object.

[0147] In some embodiments, the smart device includes: a first camera for capturing color images, and a second camera for capturing grayscale images;

[0148] The first image includes: a first color image acquired by the first camera at the first moment, and a first grayscale image acquired by the second camera at the first moment;

[0149] The step of processing the first image acquired by the smart device at the first moment to obtain a first image of interest and a first mask image of the target object includes:

[0150] Performing cropping processing on the first grayscale image to obtain a first image of interest of the target object;

[0151] The first color image is subjected to a matting process to obtain a first mask image of the target object.

[0152] Since the shape of the target object presented in the color image is more consistent with the shape of the target object, the first grayscale image can be used to obtain the first region of interest, and the first color image can be used to obtain the first mask image, so that the accuracy of determining the posture parameters of the target object can be further improved. It can be understood that in these embodiments, the number of first images of interest is equal to the number of grayscale images in the first image, and the number of first mask images is equal to the number of color images in the first image.

[0153] Furthermore, performing a matting process on the first color image to obtain a first mask image of the target object includes:

[0154] In the case where the image frame rates of the first camera and the second camera are the same and synchronized, the first image of interest is converted using a conversion relationship from a camera extrinsic parameter matrix of the second camera to a camera extrinsic parameter matrix of the first camera to obtain a third image of interest of the target object in the first color image;

[0155] The third image of interest is subjected to a matting process to obtain a first mask image of the target object.

[0156] It can be understood that before obtaining the mask image of the target object, it is necessary to first locate the target object in the image captured by the camera. In this embodiment, when the image frame rates of the first camera and the second camera are the same and synchronized, after the image of interest of the target object is acquired based on the grayscale image, the conversion relationship from the camera extrinsic parameter matrix of the second camera to the camera extrinsic parameter matrix of the first camera can be used to locate the image of interest of the target object in the first color image, and then the mask image of the target object can be directly obtained from the first color image by performing cutout processing on the image of interest.

[0157] In this way, by acquiring the image of interest of the target object based on the grayscale image and locating the image of interest of the target object in the color image, the cropping process of the color image can be saved, thereby improving the acquisition efficiency of the mask image of the target object.

[0158] It should be noted that the various optional embodiments introduced in the embodiments of the present application can be implemented in combination with each other or can be implemented separately if they do not conflict with each other, and the embodiments of the present application are not limited to this.

[0159] For ease of understanding, the following example is taken as an example of a hand as the target object. In the following example, the virtual model of the target object is a virtual hand model.

[0160] This example implements a complete set of gesture tracking algorithms that can run in real time on XR devices. Compared with the current mainstream XR gesture tracking algorithm, a new hand cutout module is added to constrain the hand shape, and it can be extended to use VST cameras as input. It will perform better in the fit between virtual hand touch and real hand gestures, thereby greatly improving the user interaction experience.

[0161] Embodiment 1

[0162] Considering that the hand posture parameters of the related art cannot fit different types of hand shapes well, a new set of hand posture parameters can be used to simulate a richer distribution of hand shapes. In addition, a hand cutout module is added as an additional constraint to better help the neural network regress accurate parameters. In the first embodiment, the process of obtaining the virtual hand model is as follows: Figure 4 shown.

[0163] In the first embodiment, a parameterized model similar to 3DMM is used to represent the hand posture parameters: the hand posture parameters can be defined as three parts: shape (β), posture (θ) and hand geometry information (ie, the aforementioned motion information).

[0164] The shape part may include 10 parameters. The original shape of the final virtual hand model is mainly controlled by these 10 parameters. The original shape of the virtual hand model can be calculated by formula (1).

[0165] The posture part contains 16 groups of parameters, representing the rotation of the wrist and the rotation of the 15 joints of the fingers. Each group of parameters is represented by a 3-DOF axis angle, so there are 48 parameters in total. When calculating the 3D key points of the hand skeleton, forward dynamics can be used to calculate through formula (3).

[0166] The hand geometry information mainly includes the translation and rotation parameters of the hand relative to the main camera, which is used to calculate the absolute position information of the virtual hand model.

[0167] The advantage of using this set of hand parameters is that the virtual hand model output by the algorithm has richer shapes and conforms to the user's actual hand shape, which is the prerequisite for improving the fit between the virtual hand model and the real hand.

[0168] like Figure 4As shown, the first embodiment introduces a hand cutout module, and adds constraints when training a neural network (i.e., a posture detection model, which may also be referred to as a posture detection module or a 3D posture detection module), thereby constraining the fit between the virtual and real hands. By applying the hand cutout algorithm, a mask image of the current user's hand on the hand ROI image can be obtained, which is equivalent to obtaining information about the contour shape of the current user's hand. When the downstream hand 3D posture detection module performs posture reasoning, an additional mask constraint can be added, so that the contour of the final predicted virtual hand model projected onto the 2D image is as close as possible to the actual contour of the user.

[0169] Specifically, the steps to complete the construction of this entire gesture tracking system are as follows:

[0170] Step 1: Train the hand box detection module separately: You can train a YOLOx single-stage target detection network model to complete the detection task. This module accepts a whole fisheye grayscale image as input and outputs a cropped hand ROI image.

[0171] Step 2: Train the gesture matting module separately: The function of this module is to receive the hand ROI image and output its corresponding gesture mask image. The model can adopt a CNN encoder-decoder architecture neural network similar to that used in BackgroundMattingV2. The main loss term constraint can be expressed as:

[0172]

[0173] where α p ,α g , respectively represent the predicted image transparency value and the real transparency value, i represents the pixel sequence number, and e is generally set to a minimum value to improve the stability of model training. The reason why the cutout module is trained separately in step 2 instead of end-to-end training (training the cutout module and the hand 3D module together) is that the output result of the hand cutout module is required to be high, and end-to-end training is difficult to ensure the accuracy of a single module.

[0174] Step 3: Train the hand 3D posture detection module: This module accepts the hand ROI image, hand mask image, multi-camera geometry information (camera external parameters, i.e., the spatial position relationship between several cameras) and the hand posture parameters of the previous frame as input (to improve the network prediction effect when the hand is blocked), and outputs new hand posture parameters. The following loss functions can be used as constraints during the training of this network:

[0175] 1. Mask alignment term, which can be calculated by formula (8).

[0176] The calculation can be done through the following steps. The calculation results of each step can be found in Figure 5 .

[0177] 201: The shape parameter β predicted by the network first p The original mesh vertex position of the current hand can be calculated using formula (1):

[0178] Then, based on this new hand mesh vertex, we can obtain the original position of the hand bone key points through weighted average calculation, which can be calculated by formula (2).

[0179] 202: The 3D position of the current hand skeleton key point is calculated by the forward dynamics method, which can be calculated by formula (3).

[0180] 203: By applying the linear skinning algorithm, the current mesh vertex position of the virtual hand model can be obtained, which can be calculated by formula (4).

[0181] And transform it to the main camera coordinate system, which can be calculated by formula (5)

[0182] 204: To project these mesh vertices onto the 2D images of each camera perspective, we first convert them to the camera coordinate system using the external parameters of each camera, which can be calculated using formula (6)

[0183] Then the 2D projection point is calculated, which can be obtained by formula (7).

[0184] 205: Determine whether these projected 2D points fall within the gesture mask area. Formula (9) can be used to determine whether the 2D point is within the predicted image mask area.

[0185] Through this constraint, the virtual hand model and the cutout mask can be linked, so that the prediction results of the hand parameter posture detection network will try to ensure that the hand contour projected onto the 2D image by the virtual hand model is consistent with the hand contour obtained by the cutout.

[0186] In addition, 3D loss terms, 2D loss terms, multi-view geometric parameters and regularization terms can also be considered. Then, the loss value of the hand parameter posture detection network is calculated through each term.

[0187] The network training is completed through the above steps. In actual use (network inference), the ROI image of the hand is first obtained through the hand frame detection model, and then the gesture cutout module is applied to the ROI image to obtain the gesture mask image. The hand ROI image, gesture mask image, hand posture parameters of the previous frame, and geometric external parameter information of the camera are then spliced ​​together and input into the 3D posture detection module to obtain new hand posture parameters. Finally, the mesh vertex information of the virtual hand model is calculated through the same method as the first step in the "Mask Alignment Item", which can be provided to the rendering side to render the posture of the virtual hand model on the screen.

[0188] In the scheme of embodiment 1, while maintaining the fisheye grayscale image input, by adding a gesture cutout module and the corresponding loss term constraints, the hand posture parameters predicted by the gesture algorithm and the virtual hand model calculated by this set of parameters are closer to the user's real hand shape, thereby improving the accuracy of the gesture tracking algorithm and the user experience.

[0189] Embodiment 2:

[0190] If the VST camera is turned on, the input of the gesture cutout module can be changed to the VST image input. In this way, the VST mask information can be used to constrain the hand posture and shape predicted by the network, making the output of the gesture tracking algorithm more consistent with the hand shape on the VST image. Figure 6 shown.

[0191] The hand posture parameters and the hand 3D posture detection module are basically the same as those described in Example 1, except that for the posture detection network, only the hand cutout mask image of the VST is input, and the hand cutout mask image of the multi-eye grayscale fisheye image is no longer input. In this way, it can be ensured that the output virtual hand model shape is as close as possible to the hand shape presented in the VST image.

[0192] For the hand cutout module, assuming that the frame rate of the VST image and the fisheye grayscale image is the same and synchronized, the ROI area of ​​the hand on the VST image can be calculated through the multi-eye fisheye hand frame detection and the camera external parameter information. The hand mask image can be obtained by applying the hand cutout algorithm in this ROI area, and then input into the 3D posture detection network as a constraint. The advantage of reusing the hand frame detection results of the multi-eye fisheye image is that the hand frame detection network for the VST image can be omitted, improving the efficiency and performance of the algorithm.

[0193] Compared with the first embodiment, the second embodiment combines the VST image with two advantages:

[0194] First, VST is a three-channel RGB image, which has more texture feature information than fisheye images. It can make the prediction of hand mask, especially the edge of hand contour, more accurate and stable.

[0195] Secondly, if a virtual hand model is not needed, and the hand cutout method is directly used for front-end display (in a pure VR scenario, the user's hand is cut out and displayed), Example 2 can also directly obtain the cutout result, which is equivalent to combining the gesture tracking algorithm and the cutout algorithm. It not only reuses the hand frame detection module and reduces network redundancy, but also uses the result of the cutout algorithm as the input of gesture tracking to constrain its result, thereby improving the overall tracking accuracy and ensuring a better experience.

[0196] It should be noted that the posture parameter determination method of the embodiment of the present application can be extended to any device with multiple cameras, such as a vehicle-mounted system or a high-performance mobile phone. If there is a demand for a gesture tracking algorithm on these devices, the architecture of the embodiment of the present application can be considered.

[0197] The posture parameter determination method provided in the embodiment of the present application can be executed by a posture parameter determination device. In the embodiment of the present application, the posture parameter determination device executing the posture parameter determination method is taken as an example to illustrate the posture parameter determination device provided in the embodiment of the present application.

[0198] like Figure 7 As shown, the posture parameter determination device includes the following:

[0199] A first acquisition module 701 is used to process a first image acquired by the smart device at a first moment to obtain a first image of interest and a first mask image of a target object; wherein the target object is an image of a target photographed object;

[0200] A second acquisition module 702 is used to input the first image of interest and the first mask image into a trained posture detection model to obtain a first posture parameter of the target object at the first moment;

[0201] Wherein, during the training process of the posture detection model, the loss value of the posture detection model includes the mask loss value of the target object; the mask loss value is used to characterize the matching degree between the first virtual model and the second mask image of the target object in the training sample, and the mask loss value is negatively correlated with the matching degree; the first virtual model is a virtual model of the target photographed object corresponding to the second posture parameter; the second posture parameter is the posture parameter of the target photographed object predicted by the posture detection model based on the training sample.

[0202] In some embodiments, the apparatus further comprises:

[0203] A first determination module, configured to determine, by using the first posture parameter, predicted positions of N mesh vertices corresponding to the target photographed object at the first moment in a first coordinate system; wherein the first coordinate system is a coordinate system of a main camera of the smart device, and N is an integer greater than 1;

[0204] a rendering module, configured to render and display a second virtual model of the target photographed object corresponding to the first posture parameter by using the predicted positions of the N mesh vertices in the first coordinate system at the first moment;

[0205] The outline of the second virtual model matches the outline of the target photographed object at the first moment.

[0206] In some embodiments, the posture parameters of the target object include: shape parameters of the target object, rotation parameters of the target object, and motion parameters of the target object relative to the main camera of the smart device;

[0207] The first determining module includes:

[0208] a first acquisition unit, configured to transform default positions of N mesh vertices corresponding to the target object in a second coordinate system by using a first shape parameter in the first posture parameter, so as to obtain original positions of the N mesh vertices in the second coordinate system, wherein the second coordinate system is a coordinate system of the target object;

[0209] A first determining unit is used to determine the original positions of M skeleton key points corresponding to the target photographed object in the second coordinate system by using the original positions of the N mesh vertices in the second coordinate system, wherein the original position of each skeleton key point in the M skeleton key points is determined based on the original position of at least two mesh vertices in the N mesh vertices; M is an integer greater than 1;

[0210] A second acquisition unit is used to transform the original positions of the M skeleton key points in the second coordinate system by using the first rotation parameter in the first posture parameter to obtain the predicted positions of the M skeleton key points in the second coordinate system at the first moment;

[0211] A second determining unit, configured to determine the predicted positions of the N mesh vertices in the second coordinate system at the first moment by using the predicted positions of the M skeleton key points in the second coordinate system at the first moment;

[0212] The third acquisition unit is used to transform the predicted positions of the N mesh vertices in the second coordinate system at the first moment by using the first motion parameter in the first posture parameter to obtain the predicted positions of the N mesh vertices in the first coordinate system at the first moment.

[0213] In some embodiments, the apparatus further comprises:

[0214] A third acquisition module is used to acquire training samples; wherein the training samples include a second image of interest and the second mask image of the target object; the second image of interest and the second mask image are obtained by processing a second image collected by the smart device at a second moment; the second moment is before the first moment;

[0215] A fourth acquisition module, used for inputting the training sample into the posture detection model to be trained to obtain the second posture parameter;

[0216] a second determination module, configured to determine predicted positions of N mesh vertices of the first virtual model in a first coordinate system using the second posture parameter, wherein the first coordinate system is a coordinate system of a main camera of the smart device, and N is an integer greater than 1;

[0217] A fifth acquisition module, configured to acquire, when the smart device includes at least two cameras, projection positions of the N mesh vertices in the coordinate system of each camera in the smart device by using the predicted positions of the N mesh vertices in the first coordinate system;

[0218] a third determining module, configured to determine a mask loss value of the target object according to a positional relationship between the projection position and the second mask image;

[0219] A fourth determination module, used to determine the loss value of the posture detection model according to the mask loss value of the target object;

[0220] A training module is used to update the model parameters and training samples of the posture detection model when the loss value is greater than or equal to the loss threshold, and return to execute the step of inputting the training samples into the posture detection model to be trained to obtain the second posture parameters, until the loss value of the posture detection model is less than the loss threshold, thereby obtaining a trained posture detection model.

[0221] In some embodiments, the smart device includes: a first camera for capturing color images, and a second camera for capturing grayscale images;

[0222] The first image includes: a first color image acquired by the first camera at the first moment, and a first grayscale image acquired by the second camera at the first moment;

[0223] The first acquisition module includes:

[0224] A first processing unit, configured to perform cropping processing on the first image to obtain a first image of interest of a target object;

[0225] The second processing unit is used to perform a matting process on the first color image to obtain a first mask image of the target object.

[0226] In some embodiments, the second processing unit is specifically configured to:

[0227] In a case where the image frame rates of the first camera and the second camera are the same and synchronized, the first image of interest is converted using a conversion matrix from the second camera to the first camera to obtain a third image of interest of the target object in the first color image;

[0228] The third image of interest is subjected to a matting process to obtain a first mask image of the target object.

[0229] The attitude parameter determination device in the embodiment of the present application can be an intelligent device, or a component in the intelligent device, such as an integrated circuit or a chip. The intelligent device can be a terminal, or other devices other than a terminal. Exemplarily, the intelligent device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car-mounted intelligent device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0230] The posture parameter determination device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0231] The posture parameter determination device provided in the embodiment of the present application can implement each process of the method embodiment, and will not be described again here to avoid repetition.

[0232] Alternatively, if Figure 8 As shown, an embodiment of the present application also provides an intelligent device 800, including a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions that can be executed on the processor 801, and when the program or instructions are executed by the processor 801, the various steps of the above-mentioned posture parameter determination method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0233] It should be noted that the smart devices in the embodiments of the present application include the mobile smart devices and non-mobile smart devices mentioned above.

[0234] Fig. 9 A schematic diagram of the hardware structure of a smart device for implementing an embodiment of the present application.

[0235] The smart device 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910.

[0236] Those skilled in the art will appreciate that the smart device 900 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 910 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The structure of the smart device shown in the figure does not constitute a limitation on the smart device. The smart device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0237] The processor 910 is configured to:

[0238] Processing a first image acquired by the smart device at a first moment to obtain a first image of interest and a first mask image of a target object; wherein the target object is an image of a target photographed object;

[0239] Inputting the first image of interest and the first mask image into a trained posture detection model to obtain a first posture parameter of the target object at the first moment;

[0240] Wherein, during the training process of the posture detection model, the loss value of the posture detection model includes the mask loss value of the target object; the mask loss value is used to characterize the matching degree between the first virtual model and the second mask image of the target object in the training sample, and the mask loss value is negatively correlated with the matching degree; the first virtual model is a virtual model of the target photographed object corresponding to the second posture parameter; the second posture parameter is the posture parameter of the target photographed object predicted by the posture detection model based on the training sample.

[0241] In some embodiments, the processor 910 is further configured to:

[0242] Determine, using the first posture parameter, predicted positions of N mesh vertices corresponding to the target photographed object at the first moment in a first coordinate system; wherein the first coordinate system is a coordinate system of a main camera of the smart device, and N is an integer greater than 1;

[0243] Rendering and displaying a second virtual model of the target photographed object corresponding to the first posture parameter using the predicted positions of the N mesh vertices in the first coordinate system at the first moment;

[0244] The outline of the second virtual model matches the outline of the target photographed object at the first moment.

[0245] In some embodiments, the posture parameters of the target object include: shape parameters of the target object, rotation parameters of the target object, and motion parameters of the target object relative to the main camera of the smart device;

[0246] The processor 910 is further configured to:

[0247] Using the first shape parameter in the first posture parameter, converting the default positions of N mesh vertices corresponding to the target object in a second coordinate system to obtain original positions of the N mesh vertices in the second coordinate system, wherein the second coordinate system is the coordinate system of the target object;

[0248] Determine the original positions of M skeleton key points corresponding to the target photographed object in the second coordinate system by using the original positions of the N mesh vertices in the second coordinate system, wherein the original position of each skeleton key point in the M skeleton key points is determined based on the original position of at least two mesh vertices in the N mesh vertices; M is an integer greater than 1;

[0249] Using the first rotation parameter in the first posture parameter, the original positions of the M skeleton key points in the second coordinate system are transformed to obtain the predicted positions of the M skeleton key points in the second coordinate system at the first moment;

[0250] Determine the predicted positions of the N mesh vertices in the second coordinate system at the first moment by using the predicted positions of the M skeleton key points in the second coordinate system at the first moment;

[0251] The predicted positions of the N mesh vertices in the second coordinate system at the first moment are transformed using the first motion parameter in the first posture parameter to obtain the predicted positions of the N mesh vertices in the first coordinate system at the first moment.

[0252] In some embodiments, the processor 910 is further configured to:

[0253] Acquire a training sample; wherein the training sample includes a second image of interest and the second mask image of the target object; the second image of interest and the second mask image are obtained by processing a second image collected by the smart device at a second moment; the second moment is before the first moment;

[0254] Inputting the training sample into a posture detection model to be trained to obtain the second posture parameter;

[0255] Determine predicted positions of N mesh vertices of the first virtual model in a first coordinate system using the second posture parameter, wherein the first coordinate system is a coordinate system of a main camera of the smart device, and N is an integer greater than 1;

[0256] In a case where the smart device includes at least two cameras, using the predicted positions of the N mesh vertices in the first coordinate system, obtaining the projection positions of the N mesh vertices in the coordinate systems of each camera in the smart device;

[0257] Determining a mask loss value of the target object according to a positional relationship between the projection position and the second mask image;

[0258] Determining a loss value of the posture detection model according to a mask loss value of the target object;

[0259] When the loss value is greater than or equal to the loss threshold, the model parameters and training samples of the posture detection model are updated, and the step of inputting the training samples into the posture detection model to be trained is returned to obtain the second posture parameters, until the loss value of the posture detection model is less than the loss threshold, thereby obtaining a trained posture detection model.

[0260] In some embodiments, the smart device includes: a first camera for capturing color images, and a second camera for capturing grayscale images;

[0261] The first image includes: a first color image acquired by the first camera at the first moment, and a first grayscale image acquired by the second camera at the first moment;

[0262] The processor 910 is further configured to:

[0263] Performing cropping processing on the first grayscale image to obtain a first image of interest of the target object;

[0264] The first color image is subjected to a matting process to obtain a first mask image of the target object.

[0265] In some embodiments, the processor 910 is further configured to:

[0266] When the image frame rates of the first camera and the second camera are the same and synchronized, the first image of interest is converted by using a conversion relationship from a camera extrinsic parameter matrix of the second camera to a camera extrinsic parameter matrix of the first camera to obtain a third image of interest of the target object in the first color image;

[0267] The third image of interest is subjected to a matting process to obtain a first mask image of the target object. The smart device 900 provided in the embodiment of the present application can implement each process of the method embodiment, which will not be described here in detail to avoid repetition.

[0268] It should be understood that in the embodiment of the present application, the input unit 904 may include a graphics processor (Graphics Processing Unit, GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0269] The memory 909 can be used to store software programs and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 909 may include a volatile memory or a non-volatile memory, or the memory 909 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0270] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 910.

[0271] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned posture parameter determination method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0272] The processor is the processor in the smart device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0273] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned posture parameter determination method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0274] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0275] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned posture parameter determination method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0276] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0277] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0278] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for determining attitude parameters, It is characterized in that Executed by smart devices, including: Processing a first image acquired by the smart device at a first moment to obtain a first image of interest and a first mask image of a target object; wherein the target object is an image of a target photographed object; Inputting the first image of interest and the first mask image into a trained posture detection model to obtain a first posture parameter of the target object at the first moment; Wherein, during the training process of the posture detection model, the loss value of the posture detection model includes the mask loss value of the target object; the mask loss value is used to characterize the matching degree between the first virtual model and the second mask image of the target object in the training sample, and the mask loss value is negatively correlated with the matching degree; the first virtual model is a virtual model of the target photographed object corresponding to the second posture parameter; the second posture parameter is the posture parameter of the target photographed object predicted by the posture detection model based on the training sample.

2. The method according to claim 1, It is characterized in that After inputting the first image of interest and the first mask image into the trained posture detection model to obtain the first posture parameter of the target photographed object at the first moment, the method further includes: Determine, using the first posture parameter, predicted positions of N mesh vertices corresponding to the target photographed object at the first moment in a first coordinate system; wherein the first coordinate system is a coordinate system of a main camera of the smart device, and N is an integer greater than 1; Rendering and displaying a second virtual model of the target photographed object corresponding to the first posture parameter using the predicted positions of the N mesh vertices in the first coordinate system at the first moment; The outline of the second virtual model matches the outline of the target photographed object at the first moment.

3. The method according to claim 2, It is characterized in that The posture parameters of the target object include: shape parameters of the target object, rotation parameters of the target object, and motion parameters of the target object relative to the main camera of the smart device; The step of using the first posture parameter to determine predicted positions of N mesh vertices corresponding to the target photographed object in the first coordinate system at the first moment includes: Using the first shape parameter in the first posture parameter, converting the default positions of N mesh vertices corresponding to the target object in a second coordinate system to obtain original positions of the N mesh vertices in the second coordinate system, wherein the second coordinate system is the coordinate system of the target object; Determine the original positions of M skeleton key points corresponding to the target photographed object in the second coordinate system by using the original positions of the N mesh vertices in the second coordinate system, wherein the original position of each skeleton key point in the M skeleton key points is determined based on the original position of at least two mesh vertices in the N mesh vertices; M is an integer greater than 1; Using the first rotation parameter in the first posture parameter, the original positions of the M skeleton key points in the second coordinate system are transformed to obtain the predicted positions of the M skeleton key points in the second coordinate system at the first moment; Determine the predicted positions of the N mesh vertices in the second coordinate system at the first moment by using the predicted positions of the M skeleton key points in the second coordinate system at the first moment; The predicted positions of the N mesh vertices in the second coordinate system at the first moment are transformed using the first motion parameter in the first posture parameter to obtain the predicted positions of the N mesh vertices in the first coordinate system at the first moment.

4. The method according to claim 1, It is characterized in that Before inputting the first image of interest and the first mask image into the trained posture detection model to obtain the first posture parameter of the target photographed object at the first moment, the method further includes: Acquire a training sample; wherein the training sample includes a second image of interest and the second mask image of the target object; the second image of interest and the second mask image are obtained by processing a second image collected by the smart device at a second moment; the second moment is before the first moment; Inputting the training sample into a posture detection model to be trained to obtain the second posture parameter; Determine predicted positions of N mesh vertices of the first virtual model in a first coordinate system using the second posture parameter, wherein the first coordinate system is a coordinate system of a main camera of the smart device, and N is an integer greater than 1; In a case where the smart device includes at least two cameras, using the predicted positions of the N mesh vertices in the first coordinate system, obtaining the projection positions of the N mesh vertices in the coordinate systems of each camera in the smart device; Determining a mask loss value of the target object according to a positional relationship between the projection position and the second mask image; Determining a loss value of the posture detection model according to a mask loss value of the target object; When the loss value is greater than or equal to the loss threshold, the model parameters and training samples of the posture detection model are updated, and the step of inputting the training samples into the posture detection model to be trained is returned to obtain the second posture parameters, until the loss value of the posture detection model is less than the loss threshold, thereby obtaining a trained posture detection model.

5. The method according to claim 1, It is characterized in that The smart device comprises: a first camera for acquiring color images, and a second camera for acquiring grayscale images; The first image includes: a first color image acquired by the first camera at the first moment, and a first grayscale image acquired by the second camera at the first moment; The step of processing the first image acquired by the smart device at the first moment to obtain a first image of interest and a first mask image of the target object includes: Performing cropping processing on the first grayscale image to obtain a first image of interest of the target object; The first color image is subjected to a matting process to obtain a first mask image of the target object.

6. The method according to claim 5, It is characterized in that The step of performing a matting process on the first color image to obtain a first mask image of the target object includes: When the image frame rates of the first camera and the second camera are the same and synchronized, the first image of interest is converted by using a conversion relationship from a camera extrinsic parameter matrix of the second camera to a camera extrinsic parameter matrix of the first camera to obtain a third image of interest of the target object in the first color image; The third image of interest is subjected to a matting process to obtain a first mask image of the target object.

7. A posture parameter determination device, applied to smart devices, It is characterized in that include: A first acquisition module is used to process the first image collected by the smart device at the first moment to obtain a first image of interest and a first mask image of a target object; wherein the target object is an image of a target photographed object; A second acquisition module is used to input the first image of interest and the first mask image into a trained posture detection model to obtain a first posture parameter of the target object at the first moment; Wherein, during the training process of the posture detection model, the loss value of the posture detection model includes the mask loss value of the target object; the mask loss value is used to characterize the matching degree between the first virtual model and the second mask image of the target object in the training sample, and the mask loss value is negatively correlated with the matching degree; the first virtual model is a virtual model of the target photographed object corresponding to the second posture parameter; the second posture parameter is the posture parameter of the target photographed object predicted by the posture detection model based on the training sample.

8. The device according to claim 7, It is characterized in that The device also includes: A first determination module, configured to determine, by using the first posture parameter, predicted positions of N mesh vertices corresponding to the target photographed object at the first moment in a first coordinate system; wherein the first coordinate system is a coordinate system of a main camera of the smart device, and N is an integer greater than 1; a rendering module, configured to render and display a second virtual model of the target photographed object corresponding to the first posture parameter by using the predicted positions of the N mesh vertices in the first coordinate system at the first moment; The outline of the second virtual model matches the outline of the target photographed object at the first moment.

9. The device according to claim 8, It is characterized in that The posture parameters of the target object include: shape parameters of the target object, rotation parameters of the target object, and motion parameters of the target object relative to the main camera of the smart device; The first determining module includes: a first acquisition unit, configured to transform default positions of N mesh vertices corresponding to the target object in a second coordinate system by using a first shape parameter in the first posture parameter, so as to obtain original positions of the N mesh vertices in the second coordinate system, wherein the second coordinate system is a coordinate system of the target object; A first determining unit is used to determine the original positions of M skeleton key points corresponding to the target photographed object in the second coordinate system by using the original positions of the N mesh vertices in the second coordinate system, wherein the original position of each skeleton key point in the M skeleton key points is determined based on the original position of at least two mesh vertices in the N mesh vertices; M is an integer greater than 1; A second acquisition unit is used to transform the original positions of the M skeleton key points in the second coordinate system by using the first rotation parameter in the first posture parameter to obtain the predicted positions of the M skeleton key points in the second coordinate system at the first moment; A second determining unit, configured to determine the predicted positions of the N mesh vertices in the second coordinate system at the first moment by using the predicted positions of the M skeleton key points in the second coordinate system at the first moment; The third acquisition unit is used to transform the predicted positions of the N mesh vertices in the second coordinate system at the first moment by using the first motion parameter in the first posture parameter to obtain the predicted positions of the N mesh vertices in the first coordinate system at the first moment.

10. The device according to claim 7, It is characterized in that The device also includes: A third acquisition module is used to acquire training samples; wherein the training samples include a second image of interest and the second mask image of the target object; the second image of interest and the second mask image are obtained by processing a second image collected by the smart device at a second moment; the second moment is before the first moment; A fourth acquisition module, used for inputting the training sample into the posture detection model to be trained to obtain the second posture parameter; a second determination module, configured to determine predicted positions of N mesh vertices of the first virtual model in a first coordinate system using the second posture parameter, wherein the first coordinate system is a coordinate system of a main camera of the smart device, and N is an integer greater than 1; A fifth acquisition module, configured to acquire, when the smart device includes at least two cameras, projection positions of the N mesh vertices in the coordinate system of each camera in the smart device by using the predicted positions of the N mesh vertices in the first coordinate system; a third determining module, configured to determine a mask loss value of the target object according to a positional relationship between the projection position and the second mask image; A fourth determination module, used to determine the loss value of the posture detection model according to the mask loss value of the target object; A training module is used to update the model parameters and training samples of the posture detection model when the loss value is greater than or equal to the loss threshold, and return to execute the step of inputting the training samples into the posture detection model to be trained to obtain the second posture parameters, until the loss value of the posture detection model is less than the loss threshold, thereby obtaining a trained posture detection model.

11. The device according to claim 7, It is characterized in that The smart device comprises: a first camera for acquiring color images, and a second camera for acquiring grayscale images; The first image includes: a first color image acquired by the first camera at the first moment, and a first grayscale image acquired by the second camera at the first moment; The first acquisition module includes: A first processing unit, configured to perform cropping processing on the first image to obtain a first image of interest of a target object; The second processing unit is used to perform a matting process on the first color image to obtain a first mask image of the target object.

12. The device according to claim 11, It is characterized in that The second processing unit is specifically configured to: In a case where the image frame rates of the first camera and the second camera are the same and synchronized, the first image of interest is converted using a conversion matrix from the second camera to the first camera to obtain a third image of interest of the target object in the first color image; The third image of interest is subjected to a matting process to obtain a first mask image of the target object.

13. A smart device, It is characterized in that The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the posture parameter determination method as described in any one of claims 1 to 6 are implemented.