Data set generation method, electronic equipment and storage medium

Abstract

The invention provides a data set generation method, electronic equipment and a storage medium. The method comprises the following steps: determining a reference object belonging to the same category as a to-be-matched object in a virtual scene; under the condition that the to-be-matched object and the reference object are registered, obtaining a transformation matrix based on the object similarity between the to-be-matched object and the reference object; on the basis of the transformation matrix, updating the local coordinate system of the object to be matched into the local coordinate system of the reference object; based on the updated local coordinate system of the to-be-configured object, determining a posture of the to-be-configured object on a target object in the virtual scene; and obtaining a target picture corresponding to each to-be-matched object on the target object under the posture, and constructing a data set based on each target picture. According to the method, the construction efficiency of the data set and the precision of training the model by using the data set can be improved.

Description

Technical Field

[0001] The present application belongs to the field of computers and relates to virtual scene technology, and in particular to a method for generating a data set, an electronic device and a storage medium. Background Art

[0002] When generating data, it is usually necessary to create virtual objects through 3D modeling technology and configure some handheld objects for the virtual objects. Such objects often have a unified local coordinate system as a data set for training the model. Since the definition of the local coordinate system depends on semantics, objects are usually adjusted one by one to achieve a unified definition, thereby achieving a unified local coordinate system. However, for a large number of object materials from different sources, the method of adjusting one by one is inefficient, and there may also be inconsistent definitions due to different sources, resulting in such objects showing different postures in the virtual scene. As a result, when the images corresponding to objects in different postures are used as data sets to train the model, the training accuracy of the model is affected. Summary of the invention

[0003] The embodiments of the present application disclose a method for generating a data set, an electronic device, and a storage medium, which solve the technical problems that low configuration efficiency leads to low efficiency in constructing a data set when configuring handheld objects for a large number of virtual objects, and that the data set affects the training accuracy of the model.

[0004] The present application provides a method for generating a data set, the method comprising: determining a reference object that belongs to the same category as an object to be matched in a virtual scene, the object to be matched and the reference object being virtual objects in the virtual scene; determining a transformation matrix based on the object similarity between the object to be matched and the reference object when the object to be matched is matched; updating the local coordinate system of the object to be matched to the local coordinate system of the reference object based on the transformation matrix; determining the posture of the object to be matched on a target object in the virtual scene based on the updated local coordinate system of the object to be matched; obtaining a target image corresponding to each object to be matched on the target object in the posture, and constructing a data set based on each target image, wherein each object to be matched and the reference object have the same category.

[0005] In some optional embodiments, obtaining the transformation matrix based on the object similarity between the object to be matched and the reference object includes: determining the object similarity according to the point cloud distribution of the object to be matched and the reference object; and obtaining the transformation matrix based on the object similarity, a preset angle and a preset displacement.

[0006] In some optional embodiments, determining the object similarity based on the point cloud distribution of the object to be matched and the reference object includes: dividing the point clouds of the object to be matched and the reference object based on the center point of the local coordinate system of the reference object to obtain multiple regions; determining a first similarity based on the point cloud distribution of the multiple regions; determining a second similarity based on the point cloud distribution of a first group of contour images of the object to be matched at different viewing angles and the point cloud distribution of a second group of contour images of the reference object; and determining the object similarity based on the first similarity and the second similarity.

[0007] In some optional embodiments, obtaining the first similarity based on the point cloud distribution of the multiple regions includes: obtaining a first shape feature based on the point cloud distribution of the multiple regions of the object to be matched; obtaining a second shape feature based on the point cloud distribution of the multiple regions of the reference object; and determining the first similarity based on the first shape feature and the second shape feature.

[0008] In some optional embodiments, determining the second similarity based on the point cloud distribution of the first group of contour images of the object to be matched at different viewing angles and the point cloud distribution of the second group of contour images of the reference object includes: obtaining a first contour feature according to the point cloud distribution of the first group of contour images of the object to be matched; obtaining a second contour feature according to the point cloud distribution of the second group of contour images of the reference object; and determining the second similarity according to the first contour feature and the second contour feature.

[0009] In some optional embodiments, determining a reference object in a virtual scene that belongs to the same category as the object to be matched includes: obtaining hand posture parameters of each virtual object in the virtual scene; determining the hand movement type of each virtual object based on the hand posture parameters; determining a reference object corresponding to a hand movement type that matches a preset hand movement type; and using the object held by the reference object as the reference object.

[0010] In some optional implementations, the method further includes: when registering the object to be matched with the reference object, inputting point clouds of the object to be matched and the reference object into a pre-trained prediction model to obtain the transformation matrix.

[0011] In some optional embodiments, the training method of the prediction model includes: obtaining training samples and sample labels corresponding to the training samples; inputting the training samples into the prediction model to obtain prediction results; based on the prediction results and the sample labels, calculating the loss value of the prediction model; if the loss value is less than a preset loss threshold, determining that the training of the prediction model is completed.

[0012] The present application also provides an electronic device, which includes a processor and a memory, and the processor is used to implement the method for generating a data set when executing a computer program stored in the memory.

[0013] The present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for generating a data set is implemented.

[0014] In the method for generating a data set provided in the present application, a reference object belonging to the same category as the object to be matched is determined in a virtual scene, so as to achieve the purpose of adjusting the objects to be matched in batches based on the reference object. In order to calculate the object similarity later, the object to be matched is roughly registered with the reference object. Then, when the object to be matched is registered with the reference object, based on the object similarity between the object to be matched and the reference object, a transformation matrix is ​​determined so as to update the local coordinate system of the object to be matched to the local coordinate system of the reference object, so as to achieve the purpose of unifying the local coordinate systems of objects of the same category. Based on the updated local coordinate system of the object to be matched, the posture of the object to be matched on the target object in the virtual scene is determined, which improves the efficiency of configuring the object to be matched for the target object, and avoids the problem of inconsistent postures of objects of the same category due to different sources of object materials. In addition, through the posture of the object to be matched on the target object in the virtual scene, the postures of multiple objects to be matched of the same category on the target object can be determined, so that multiple target images can be obtained and a data set can be constructed. The construction efficiency and accuracy of the data set are improved, and the training accuracy of the model is improved to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a posture diagram of a virtual object applied in a virtual scene provided by an embodiment of the present application.

[0016] Figure 2 It is a schematic diagram of a virtual object holding a virtual object in a virtual scene provided by an embodiment of the present application.

[0017] Figure 3 It is a flowchart of a method for generating a data set provided in an embodiment of the present application.

[0018] Figure 4 This is a flow chart for calculating object similarity provided in an embodiment of the present application.

[0019] Figure 5 It is a schematic diagram of the area division of a two-dimensional object provided in an embodiment of the present application.

[0020] Figure 6 It is a schematic diagram of the area division of a three-dimensional object provided in an embodiment of the present application.

[0021] Figure 7 These are the first set of contour images and the second set of contour images provided in the embodiments of the present application.

[0022] Figure 8 This is a flowchart of a method for generating a data set provided in yet another embodiment of the present application.

[0023] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] To facilitate understanding, some illustrations of concepts related to the embodiments of the present application are given by way of example for reference.

[0025] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0026] When generating data, it is usually necessary to create virtual objects through 3D modeling technology and configure some handheld objects for the virtual objects. Such objects often have a unified local coordinate system as training data for the model. In one example, when a virtual object in a virtual scene performs the task of grasping a periodic object, a unified coordinate system is required for the grasped object so that it can be accurately grasped by the virtual object. Since the definition of the local coordinate system depends on semantics, objects are usually adjusted one by one to achieve a unified definition, thereby achieving a unified local coordinate system. However, for a large number of object materials from different sources, the method of adjusting one by one is inefficient, and there may also be inconsistent definitions due to different sources, resulting in such objects showing different postures in the virtual scene. As a result, when the images corresponding to objects in different postures are used as training data to train the model, the training accuracy of the model is affected. The following is combined with Figure 1 and Figure 2 The related technology is described.

[0027] Figure 1 It is a posture diagram of a virtual object applied in a virtual scene provided by an embodiment of the present application. Figure 2 It is a schematic diagram of a virtual object holding a virtual object in a virtual scene provided by an embodiment of the present application.

[0028] like Figure 1In the virtual scene shown, there are three objects from different sources. Due to the different sources, the local coordinate systems corresponding to each object are different. Due to the inconsistency of the local coordinate systems, the orientations of the objects are different. Suppose the virtual object in the virtual scene needs to grab an object, such as Figure 2 As shown in the figure, when the virtual object grasps three objects from different sources, the orientations of the objects are different due to the different local coordinate systems, which leads to improper postures of the grasped objects and the virtual object after the virtual object grasps the objects. Figure 2 The examples shown show the postures of virtual objects holding different objects, which are obviously not the correct grasping postures. This will inevitably affect the construction effect of characters and objects in the virtual scene and affect the user experience.

[0029] In order to solve the technical problems that the low configuration efficiency leads to low efficiency in constructing data sets when configuring handheld objects for a large number of virtual objects, and the data set affects the training accuracy of the model, the present application proposes a data set generation method, electronic device and storage medium, which updates the local coordinate system of the object to be configured to the local coordinate system of the reference object through a transformation matrix, and can improve the accuracy and efficiency of configuring handheld objects for virtual objects, thereby avoiding the situation where the postures of objects of the same category are not uniform due to different sources of object materials, so that the visual effects of the object to be configured and the reference object when being held are consistent, thereby ensuring the accuracy of the constructed data set. The following is an explanation in conjunction with a specific embodiment.

[0030] Figure 3 This is a flow chart of a method for generating a data set provided in an embodiment of the present application, which is applied in an electronic device. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted.

[0031] Step S301: determining a reference object that belongs to the same category as the object to be matched in a virtual scene.

[0032] In some embodiments of the present application, the virtual scene can be a humanistic environment created by three-dimensional modeling technology, or a digital scene outlined by digital communication technology. In the virtual scene, virtual objects can be created, such as modeled people, animals, custom animated characters, etc.; virtual objects can also be created, such as tools, infrastructure, and natural scenery. The object to be matched and the reference object can be tools, and the tool can be a tool belonging to the same category. In addition, the category of the tool is divided according to the way the virtual object holds the tool (hand posture parameters). For example, if applied to a virtual scene in a game application, the object to be matched corresponding to the game player can be a weapon, a mount, or other game props.

[0033] In multiple embodiments of the present application, the object represents a virtual object in a virtual scene. For example, taking a weapon as an example of a virtual object, if the way of holding a stick is the same as the way of holding a spear, the stick and the spear are classified into the same category.

[0034] In some embodiments of the present application, in order to speed up the configuration efficiency of the object to be matched, a reference object belonging to the same category as the object to be matched can be searched in the virtual scene. Specifically, in the virtual scene, the hand posture parameters of each virtual object are obtained. According to the hand posture parameters, the hand movement type of each virtual object can be determined, and the hand movement type can include a lifting movement type, a clamping movement type, a holding movement type, a carrying movement type, etc. The preset hand movement type of the object to be matched is obtained, the hand movement type of each virtual object is traversed, and the hand movement type of the virtual object that matches the preset hand movement type is found, so as to determine the virtual object corresponding to the hand movement type that matches the preset hand movement type as the reference object. The object held by the reference object is used as a reference object, for example, the tool held by the reference object is used as a reference object.

[0035] In one example, the virtual scene includes a virtual object A and a virtual object B. The hand motion type of the virtual object A is determined to be a holding motion type according to the hand posture parameters of the virtual object A. The hand motion type of the virtual object B is determined to be a lifting motion type according to the hand posture parameters of the virtual object B. If the preset hand motion type of the object to be matched is a holding motion type, the virtual object A is used as a reference object, and the object held by the virtual object A is used as a reference object.

[0036] In other embodiments of the present application, if no reference object is found in the virtual scene, it means that the object type corresponding to the object to be registered appears in the virtual scene for the first time. The posture of the object to be registered can be adjusted according to actual needs.

[0037] Step S302 : when the object to be registered is registered with the reference object, a transformation matrix is ​​determined based on the object similarity between the object to be registered and the reference object.

[0038] In some embodiments of the present application, in order to improve the accuracy of the subsequent calculation of the similarity between objects so as to accurately identify objects of the same category, the object to be matched can be first registered with the reference object. Specifically, the point cloud distribution of the object to be matched and the reference object is obtained respectively, and the point cloud can be pre-processed by denoising, filtering, downsampling, etc. to improve the accuracy of the registration. The iterative closest point (ICP) algorithm is used to align the point cloud of the object to be matched with the point cloud of the reference object, including: extracting multiple first key point clouds corresponding to the object to be matched. In the point cloud of the reference object, the point cloud closest to each first key point cloud is found as the second key point cloud. According to the first key point cloud and the second key point cloud, a point pair set is formed. According to the point pair set, the rotation amount and the translation amount are calculated. The first key point cloud is transformed based on the rotation amount and the translation amount, so that the first key point cloud is close to the second key point cloud, so as to achieve the purpose of registering the object to be registered with the reference object.

[0039] In some embodiments of the present application, when the object to be matched is aligned with a reference object, the object similarity is calculated. Specifically, Shape Context can be used to calculate the shape features of the object to be matched and the reference object to obtain a first similarity, and Light Field Descriptor can be used to calculate the contour features of the object to be matched and the reference object to obtain a second similarity. Among them, Shape Context is a shape description method, which constructs a feature matrix by calculating the angle and distance from the surrounding points of each point in the figure to the point, thereby describing the image features. Light Field Descriptor is a technology for obtaining model space and contour information, and realizes the projection of depth image and binary image through a specific projection method. According to the first similarity and the second similarity, the object similarity is obtained. For a specific description, please refer to the following Figure 4 The embodiment shown.

[0040] The above method of calculating object similarity is only an example, and other methods may be used to calculate the object similarity between the object to be matched and the reference object, for example, a method based on a neural network algorithm, a method based on distance, etc., which is not limited in the present application.

[0041] In some embodiments of the present application, after determining the object similarity of the object to be matched at the current position, multiple preset angles and preset displacements can be selected within the surrounding preset area range with the current position as the reference point, and the similarity between the object to be matched and multiple objects located at the corresponding positions of each preset angle and / or each preset displacement is calculated. The maximum object similarity is selected from the multiple object similarities, and a transformation matrix is ​​obtained according to the preset angle and preset displacement corresponding to the maximum object similarity. If the maximum object similarity is less than or equal to the preset similarity threshold, the position of the object to be matched is adjusted based on the transformation matrix, for example, the object to be matched is transformed to a new position. Based on the new position, multiple preset angles and preset displacements are selected again within the surrounding preset area range to obtain a new transformation matrix. Until the maximum object similarity corresponding to the new position is greater than or equal to the preset similarity, or the number of steps moved reaches the preset number of steps, the transformation matrix is ​​stopped from being updated. Wherein, the number of steps represents the shortest path length from one discrete point to another discrete point in three-dimensional space, where each step can only move one unit along the coordinate axis direction. The discrete point refers to the point where the position change of the object to be matched is marked in each time period by dividing the time into several small segments during the continuous movement of the object to be matched (or the process of the object to be matched moving from the current position to the new position).

[0042] In some embodiments of the present application, the method of obtaining the transformation matrix also includes: when the object to be matched is registered with the reference object, the point cloud of the object to be matched and the reference object can be input into the pre-trained prediction model to obtain the transformation matrix. For a specific description, please refer to the following Figure 8 The prediction model may be a model composed of one or more networks such as a deep neural network (DNN), a convolutional neural network (CNN), or a recurrent neural network (RNN), or may be a geometric transformation model, a point cloud registration model, etc. The application does not limit the type of prediction model.

[0043] Step S303: Based on the transformation matrix, the local coordinate system of the object to be matched is updated to the local coordinate system of the reference object.

[0044] In some embodiments of the present application, a preset angle and a preset displacement corresponding to the transformation matrix are obtained. The product of the point cloud of the object to be matched and the transformation matrix is ​​calculated, and the coordinates of the point cloud of the object to be matched in the local coordinate system of the reference object are determined based on the product. Based on the coordinates, the preset angle and the preset displacement, the point cloud of the object to be matched is subjected to rotation transformation and translation transformation, thereby obtaining an updated local coordinate system of the object to be matched.

[0045] Step S304: determining the posture of the object to be matched on the target object in the virtual scene based on the updated local coordinate system of the object to be matched.

[0046] In some embodiments of the present application, the posture of the object to be matched is adjusted based on the updated local coordinate system of the object to be matched, a target position of the object to be matched is obtained, and the object to be matched after the posture is adjusted is moved to the target position.

[0047] Step S305 , obtaining a target image corresponding to each object to be matched on the target object in a posture, and constructing a data set based on each target image.

[0048] In some embodiments of the present application, each object to be matched has the same category. After determining the posture of the object to be matched on the target object in the virtual scene in step S304, the postures of multiple objects to be matched of the same category on the target object can be determined in batches, so that a target image can be generated according to the posture of each object to be matched on the target object. A data set is constructed based on multiple target images.

[0049] In one example, it is assumed that the object to be matched determined in step S304 is a stick, and the objects to be matched in step S305 include a stick and a spear. After the posture of the stick is determined, a first target image of the stick on the target object is obtained. Based on the posture of the stick, a second target image of the stick on the target object and a third target image of the spear on the target object can be obtained, and a data set is constructed based on the first target image, the second target image, and the third target image. The above is just an example, and multiple target images can be obtained through the above embodiments to construct a data set.

[0050] In addition, after the data set is constructed, the training accuracy of the model can be improved when the data set is used to train the model. This application does not limit the type of model.

[0051] In the above embodiment, a reference object belonging to the same category as the object to be matched is determined in the virtual scene, so as to achieve the purpose of batch adjustment of the object to be matched based on the reference object. In order to calculate the object similarity later, the object to be matched is roughly registered with the reference object. Then, when the object to be matched is registered with the reference object, a transformation matrix is ​​obtained based on the object similarity between the object to be matched and the reference object, so as to update the local coordinate system of the object to be matched to the local coordinate system of the reference object, so as to achieve the purpose of unifying the local coordinate systems of objects of the same category. Based on the updated local coordinate system of the object to be matched, the posture of the object to be matched on the target object in the virtual scene is determined, which improves the efficiency of configuring the object to be matched for the target object, and avoids the problem of inconsistent postures of objects of the same category due to different sources of object materials. In addition, through the posture of the object to be matched on the target object in the virtual scene, the postures of multiple objects to be matched of the same category on the target object can be determined, so that multiple target images can be obtained and a data set can be constructed. The construction efficiency and accuracy of the data set are improved, and the training accuracy of the model is improved to a certain extent.

[0052] Figure 4 is a flow chart of object similarity calculation provided by the embodiment of the present application, such as Figure 3 The object similarity shown can be measured using Figure 4 The calculation in the embodiment shown includes: using Shape Context to calculate the shape features of the object to be matched and the reference object to obtain a first similarity, and using Light Field Descriptor to calculate the contour features of the object to be matched and the reference object to obtain a second similarity. The object similarity is obtained according to the first similarity and the second similarity. Specifically, the following steps are included.

[0053] Step S401 : dividing the point clouds of the object to be matched and the reference object based on the center point of the local coordinate system of the reference object to obtain a plurality of regions.

[0054] In some embodiments of the present application, the object to be matched and the reference object can be two-dimensional objects or three-dimensional objects. Figure 5 Describes the region division when the object to be matched and the reference object are two-dimensional objects. Specifically, the contour position of the reference object is determined by the edge detection operator, and a set of discrete point sets is obtained based on the contour position sampling, such as Figure 5 The letter A shown in the figure. Similarly, the contour position of the object to be matched is determined by the edge detection operator, and another set of discrete points is obtained based on the contour position sampling. In the local coordinate system of the reference object, N concentric circles are established with the center point as the center, and the concentric circles are divided into M equal parts along the circumferential direction to form multiple areas (such as Figure 5Similarly, for the object to be matched, N concentric circles can be established with the center point as the center, and the concentric circles can be divided into M equal parts along the circumferential direction to form multiple areas.

[0055] Combination Figure 6 Describes the area division when the object to be matched and the reference object are three-dimensional objects. Specifically, in the local coordinate system of the reference object, a local space is established with the center point as the center (such as Figure 6 Based on the local space, different regions are divided, and the point cloud distribution in these regions is recorded as a spatial region bin.

[0056] Step S402: determining a first similarity based on point cloud distributions of multiple regions.

[0057] In some embodiments of the present application, when the object to be matched and the reference object are two-dimensional objects, calculating the first similarity includes: counting the number of point clouds of the reference object and the object to be matched in the corresponding fan-shaped area, and using a shape histogram to compare the distribution of point cloud data of the reference object and the object to be matched in the same area, thereby obtaining the first similarity.

[0058] In some embodiments of the present application, when the object to be matched and the reference object are three-dimensional objects, calculating the first similarity includes: obtaining the shape distribution corresponding to the reference object and the object to be matched by counting the number of point clouds of the reference object and the object to be matched in the corresponding space region bin, and the relationship between each space region bin and the adjacent space region bin. The Chi-square metric is used to evaluate the difference between the two shape distributions, thereby obtaining the first similarity.

[0059] In order to avoid the impact of non-uniform object sizes, the points to be counted can be normalized.

[0060] Step S403 : determining a second similarity based on the point cloud distribution of the first group of contour images of the object to be matched at different viewing angles and the point cloud distribution of the second group of contour images of the reference object.

[0061] In some embodiments of the present application, a first group of contour images of the object to be configured under different viewing angles and a second group of contour images of the reference object under different viewing angles are obtained. A first contour feature is obtained based on the point cloud distribution of the first group of contour images, and a second contour feature is obtained based on the point cloud distribution of the second group of contour images. A second similarity is calculated based on the first contour feature and the second contour feature. Figure 7 Provide explanation.

[0062] like Figure 7As shown, the first contour feature under the first viewing angle in the first group of contour images is compared with the second contour feature under the first viewing angle in the second group of contour images, and the similarity under the first viewing angle is obtained. The first contour feature under the second viewing angle in the first group of contour images is compared with the second contour feature under the second viewing angle in the second group of contour images, and the similarity under the second viewing angle is obtained. Similarly, the similarities under multiple viewing angles are obtained, for example, the similarities under the first viewing angle to the fifth viewing angle are obtained. According to the similarities under multiple viewing angles, the second similarity is obtained.

[0063] Step S404: determining the object similarity according to the first similarity and the second similarity.

[0064] In some embodiments of the present application, the first similarity and the second similarity are weighted and summed to calculate the object similarity. The formula is as follows: ; In the formula, x represents the object to be configured, y represents the reference object, Indicates the similarity of objects. represents the first similarity, represents the second similarity, Represents weight.

[0065] In the above embodiment, the first similarity is obtained by the first shape feature and the second shape feature, and the second similarity is obtained by the first contour feature and the second contour feature, so that the object similarity determined according to the first similarity and the second similarity can cover the attributes of the shape feature and the contour feature. It is possible to more comprehensively and accurately evaluate the object similarity between the object to be matched and the reference object, make up for the shortcomings of the single feature similarity, and improve the accuracy of calculating the object similarity. In addition, since the object similarity is obtained based on the weighted sum of the first similarity and the second similarity, the final contribution ratio of the first similarity corresponding to the shape feature and the second similarity corresponding to the contour feature can be flexibly controlled according to the needs of the specific application scenario, so as to better adapt to different application needs and improve the robustness of the object similarity.

[0066] Figure 8 is a flow chart of a method for generating a data set provided by another embodiment of the present application. When the object to be registered is registered with the reference object, a prediction model can also be used to obtain a transformation matrix, such as Figure 8 As shown, the following steps are included.

[0067] Step S801: determining a reference object that belongs to the same category as the object to be matched in a virtual scene.

[0068] In some embodiments of the present application, the specific description of step S801 can be found in Figure 3Step S301 in the illustrated embodiment will not be described again herein.

[0069] Step S802 : when the object to be matched is matched with the reference object, the point clouds of the object to be matched and the reference object are input into the pre-trained prediction model to obtain a transformation matrix.

[0070] In some embodiments of the present application, the specific description of "registering the object to be registered with the reference object" in step S802 can be referred to in Figure 3 Step S302 in the illustrated embodiment will not be described again herein.

[0071] In some embodiments of the present application, training the prediction model includes: obtaining training samples and sample labels corresponding to the training samples. Specifically, a standard object and multiple target objects are selected in a virtual scene, and the point clouds corresponding to the standard object and the multiple target objects are used as training samples. The positions of the multiple target objects are randomly transformed, and multiple positions corresponding to each target object are obtained. The transformation matrix from the multiple positions to the position of the standard object is calculated, which is recorded as the training matrix. The training matrix is ​​used as the sample label.

[0072] Use the training samples and sample labels to train the prediction model and obtain the prediction results of each forward propagation output. Calculate the loss value of the prediction model based on the prediction results and sample labels output each time. If the loss value is less than the preset loss threshold, it means that the prediction model training is complete. If the loss value is greater than or equal to the loss threshold, it means that the prediction model training is not complete. The parameters of the prediction model can be adjusted through back propagation until the loss value is less than the preset loss threshold and the training is stopped.

[0073] In some embodiments of the present application, after the prediction model is trained, the point clouds of the object to be matched and the reference object are input into the prediction model to obtain a transformation matrix from the object to be matched to the reference object.

[0074] Step S803: Based on the transformation matrix, the local coordinate system of the object to be matched is updated to the local coordinate system of the reference object.

[0075] Step S804: determining the posture of the object to be matched on the target object in the virtual scene based on the updated local coordinate system of the object to be matched.

[0076] In some embodiments of the present application, the specific description of steps S803 to S804 can be found in Figure 3 Steps S303 to S304 in the illustrated embodiment will not be described again herein.

[0077] In the above embodiment, by using the prediction model to obtain the transformation matrix, the computational complexity can be reduced, the efficiency of obtaining the transformation matrix can be improved, and the fully trained prediction model has good generalization ability. In addition, for objects with more complex structures, predicting the transformation matrix through the prediction model can effectively deal with the problem of nonlinear transformation.

[0078] Fig. 9 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 10 can be a mobile phone, a tablet computer, a smart wearable device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, a netbook, an energy storage device, a power distribution device, a vehicle-mounted device, a self-mobile device, and other electronic devices with a display screen. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device.

[0079] like Fig. 9 As shown, the electronic device 10 may include a display screen 100, a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is coupled to the display screen 100, the communication interface 101, the memory 102, and the I / O interface 104 through the bus 105.

[0080] The display screen 100 may be a touch screen, which is an inductive touch-sensitive liquid crystal display device. Alternatively, the display screen 100 may also be a non-touch screen.

[0081] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as Universal Serial Bus (USB), Controller Area Network (CAN), etc. The wireless communication module may provide one or more wireless communication solutions such as Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication network, Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR), etc.

[0082] The memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 103, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0083] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0084] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110. The non-volatile memory may include a disk storage device and a flash memory.

[0085] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions, and when the multiple instructions are executed by the processor 103, the method for generating a data set executed on the electronic device 10 can be implemented.

[0086] In other embodiments, the electronic device 10 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10 .

[0087] The processor 103 may include one or more processing units, for example, the processor 103 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0088] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute a computer program stored in the memory 102 to implement the above-mentioned method for generating a data set.

[0089] The I / O interface 104 is used to provide a channel for user input or output. For example, the I / O interface 104 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0090] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the I / O interface 104 in the electronic device 10 .

[0091] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0092] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0093] The computer-readable storage medium may be an internal memory of the electronic device of the above-mentioned embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.

[0094] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0095] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0096] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0097] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0098] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0099] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for generating a data set, characterized in that: The method comprises: Determining a reference object that belongs to the same category as the object to be matched in the virtual scene, wherein the object to be matched and the reference object are virtual objects in the virtual scene; In the case of registering the object to be registered with the reference object, determining a transformation matrix based on the object similarity between the object to be registered and the reference object; Based on the transformation matrix, the local coordinate system of the object to be matched is updated to the local coordinate system of the reference object; Determining the posture of the object to be matched on the target object in the virtual scene based on the updated local coordinate system of the object to be matched; A target image corresponding to each object to be matched on the target object in the posture is obtained, and a data set is constructed based on each target image, wherein each object to be matched has the same category as the reference object.

2. The method for generating a data set according to claim 1, characterized in that: The obtaining of a transformation matrix based on the object similarity between the object to be matched and the reference object comprises: Determining the object similarity according to the point cloud distribution of the object to be matched and the reference object; The transformation matrix is ​​obtained based on the object similarity, the preset angle and the preset displacement.

3. The method for generating a data set according to claim 2, characterized in that: The determining the object similarity according to the point cloud distribution of the object to be matched and the reference object includes: Dividing the point clouds of the object to be matched and the reference object based on the center point of the local coordinate system of the reference object to obtain a plurality of regions; Determining a first similarity based on the point cloud distribution of the plurality of regions; Determining a second similarity based on the point cloud distribution of the first group of contour images of the object to be matched at different viewing angles and the point cloud distribution of the second group of contour images of the reference object; The object similarity is determined according to the first similarity and the second similarity.

4. The method for generating a data set according to claim 3, characterized in that: The obtaining a first similarity based on the point cloud distribution of the plurality of regions includes: Based on the point cloud distribution of multiple regions of the object to be matched, a first shape feature is obtained; Obtaining a second shape feature based on point cloud distribution of multiple regions of the reference object; The first similarity is determined according to the first shape feature and the second shape feature.

5. The method for generating a data set according to claim 3, characterized in that: The determining of the second similarity based on the point cloud distribution of the first group of contour images of the object to be matched at different viewing angles and the point cloud distribution of the second group of contour images of the reference object comprises: Obtaining a first contour feature according to the point cloud distribution of the first group of contour images of the object to be matched; Obtaining a second contour feature according to the point cloud distribution of the second group of contour images of the reference object; The second similarity is determined according to the first contour feature and the second contour feature.

6. The method for generating a data set according to claim 1, characterized in that: The step of determining in the virtual scene a reference object that belongs to the same category as the object to be matched includes: Obtaining hand gesture parameters of each virtual object in the virtual scene; Based on the hand gesture parameters, determining a hand motion type of each virtual object; Determining a reference object corresponding to a hand motion type that matches a preset hand motion type; The object held by the reference subject is used as the reference object.

7. The method for generating a data set according to claim 1, characterized in that: The method further comprises: When the object to be matched is registered with the reference object, the point clouds of the object to be matched and the reference object are input into a pre-trained prediction model to obtain the transformation matrix.

8. The method for generating a data set according to claim 7, characterized in that: The training method of the prediction model includes: Obtaining training samples and sample labels corresponding to the training samples; Inputting the training samples into the prediction model to obtain prediction results; Based on the prediction result and the sample label, calculating the loss value of the prediction model; If the loss value is less than a preset loss threshold, it is determined that the training of the prediction model is completed.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the method for generating a data set as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the method for generating a data set according to any one of claims 1 to 8 is implemented.

Images