Position determination model training method, position determination method, device, and medium

Abstract

The present disclosure relates to the field of computer technology, and specifically to a method for training a position determination model, a method for determining a position, a training device for determining a position determination model, a device for determining a position determination, a computer-readable storage medium, and an electronic device, including: obtaining sample data of a detection object; inputting 2D key position information and 3D key position information into a first model to be trained to obtain predicted 3D position information of the detection object; updating the neural network parameters of the first model to be trained according to the predicted 3D position information and the 3D position information label of the detection object to obtain a position determination model. The technical solution of the embodiment of the present disclosure can solve the problem of inaccurate position when capturing actions through a single camera module in the prior art.

Description

Background Art

[0002] With the rapid development of the Internet, the application areas of virtual scenes are becoming more and more extensive. For example, virtual live broadcast, virtual wear, game interaction, content creation and other scenes can realize corresponding business needs through virtual objects in virtual scenes.

[0003] In the related art, the cost of the motion supplement scheme of a single camera module is relatively low. The motion of a person can be captured by a single camera module, and the captured motion can be mapped to a virtual scene to meet business needs. However, in the prior art, when a single camera module is used to capture motion, the motion of a person can be captured, but it is difficult to determine the position of the person in the virtual scene. In the virtual scene, it is easy for a virtual object to perform a corresponding action, but the position of the movement is wrong. For example, when a person walks, when a single camera module is used to capture the action, the bending change of the leg can be captured, but the position of the virtual object in the virtual scene remains unchanged.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a position determination model training method, a position determination model training device, a computer-readable storage medium and an electronic device, which can solve the problem of inaccurate position when capturing actions through a single camera module in the prior art.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, a position determination model training method is provided, comprising: acquiring sample data of a detection object; wherein the sample data of the detection object includes 2D key position information, 3D key position information and a 3D position information label; wherein the 3D position information label is used to indicate the position of the detection object in a 3D virtual environment; inputting the 2D key position information and the 3D key position information into a first model to be trained to obtain predicted 3D position information of the detection object; and updating the neural network parameters of the first model to be trained according to the predicted 3D position information and the 3D position information label of the detection object to obtain a position determination model.

[0008] Optionally, based on the above-mentioned scheme, the first model to be trained includes a first sub-model to be trained and a second sub-model to be trained, and the 2D key position information and the 3D key position information are input into the first model to be trained to obtain the predicted 3D position information of the detection object, including: inputting the 2D key position information into the first sub-model to be trained to obtain a first prediction result, and inputting the 3D key position information into the second sub-model to be trained to obtain a second prediction result; determining the predicted 3D position information of the detection object based on the first prediction result and the second prediction result.

[0009] Optionally, based on the aforementioned scheme, the inputting the 2D key position information and the 3D key position information into the first model to be trained to obtain the predicted 3D position information of the detection object includes: inputting a first number of the 2D key position information and a second number of the 3D key position information into the first model to be trained to obtain the predicted 3D position information of the second number of detection objects; wherein the detection object sample data corresponding to the first number of the 2D key position information is the data acquired within a first time period, and the second number of the 3D key position information is the data acquired within a second time period; the second time period is within the time interval of the first time period, and the first number is greater than the second number.

[0010] Optionally, based on the aforementioned scheme, the neural network parameters of the first model to be trained are updated according to the predicted 3D position information and the 3D position information label of the detection object to obtain a position determination model, including: updating the neural network parameters of the first model to be trained according to the predicted 3D position information and the 3D position information label of the detection object, and when the first model to be trained meets the convergence conditions, determining the first model to be trained as the position determination model.

[0011] Optionally, based on the aforementioned scheme, when the first model to be trained meets the convergence condition, the first model to be trained is determined as the position determination model, including: obtaining predicted 3D position information of multiple detection objects; wherein the predicted 3D position information of the multiple detection objects is obtained by predicting sample data of multiple detection objects within a third time period; determining the movement speed of the detection object according to the 3D position information labels in the sample data of the multiple detection objects; determining the predicted movement speed of the detection object according to the predicted 3D position information of the multiple detection objects; when the speed error between the movement speed of the detection object and the predicted movement speed of the detection object is less than a first threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as the position determination model.

[0012] Optionally, based on the aforementioned scheme, when the first model to be trained meets the convergence condition, the first model to be trained is determined as the position determination model, including: obtaining predicted 3D position information of multiple detection objects; wherein the predicted 3D position information of the multiple detection objects is obtained by predicting the sample data of multiple detection objects within a fourth time period; determining the motion acceleration of the detection object according to the 3D position information labels in the sample data of the multiple detection objects; determining the predicted motion acceleration of the detection object according to the predicted 3D position information of the multiple detection objects; when the acceleration error between the motion acceleration of the detection object and the predicted motion acceleration of the detection object is less than a second threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as the position determination model.

[0013] Optionally, based on the above-mentioned scheme, when the first model to be trained meets the convergence condition, the first model to be trained is determined as the position determination model, including: determining the predicted 3D position information difference based on the predicted 3D position information of the detection object and the 3D position information label; when the predicted 3D position information difference is less than a third threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as the position determination model.

[0014] Optionally, based on the aforementioned scheme, before obtaining the sample data of the detection object, the method also includes: obtaining an image corresponding to the detection object, inputting the image corresponding to the detection object into a 2D key position detection model to obtain 2D key position information corresponding to the detection object; using a multi-perspective reconstruction method to obtain a 3D position information label corresponding to the detection object, and obtaining 3D key position information corresponding to the detection object; determining the sample data of the detection object based on the 2D key position information corresponding to the detection object, the 3D key position information corresponding to the detection object, and the 3D position information label corresponding to the detection object.

[0015] Optionally, based on the above-mentioned scheme, obtaining 3D key position information corresponding to the detection object includes: inputting 2D key position information corresponding to the detection object into a 3D key position estimation model to obtain 3D key position information corresponding to the detection object.

[0016] Optionally, based on the aforementioned scheme, the 3D key position information corresponding to the detection object is determined according to the 3D position information label corresponding to the detection object; the 2D key position information corresponding to the detection object is input into the second model to be trained to obtain predicted 3D key position information; the neural network parameters of the second model to be trained are updated according to the predicted 3D key position information and the 3D key position information to obtain a 3D key position estimation model.

[0017] According to a second aspect of the present disclosure, a position determination method is provided, the method comprising: acquiring an image corresponding to a detection object; inputting the image corresponding to the detection object into a 2D key position detection model to obtain 2D key position information corresponding to the detection object; inputting the 2D key position information corresponding to the detection object into a 3D key position estimation model to obtain 3D key position information corresponding to the detection object; inputting the 2D key position information corresponding to the detection object and the 3D key position information corresponding to the detection object into a position determination model to obtain 3D position information of the detection object; wherein the 3D position information is used to indicate the position of the detection object in a 3D virtual environment, and the position determination model is obtained by a position determination model training method such as any of the above.

[0018] According to a third aspect of the present disclosure, a position determination model training device is provided, the device comprising: a sample data acquisition unit, configured to execute acquisition of detection object sample data; wherein the detection object sample data comprises 2D key position information, 3D key position information and 3D position information label; wherein the 3D position information label is used to indicate the position of the detection object in the 3D virtual environment; a prediction information acquisition unit, configured to execute input of the 2D key position information and the 3D key position information into a first model to be trained to obtain predicted 3D position information of the detection object; a position determination model training unit, configured to execute updating of the neural network parameters of the first model to be trained according to the predicted 3D position information of the detection object and the 3D position information label to obtain a position determination model.

[0019] According to a fourth aspect of the present disclosure, a position determination device is provided, the device comprising: a detection image acquisition unit, configured to execute acquisition of an image corresponding to a detection object; a 2D key position acquisition unit, configured to execute inputting the image corresponding to the detection object into a 2D key position detection model to obtain 2D key position information corresponding to the detection object; a 3D key position acquisition unit, configured to execute inputting the 2D key position information corresponding to the detection object into a 3D key position estimation model to obtain 3D key position information corresponding to the detection object; a position determination unit, configured to execute inputting the 2D key position information corresponding to the detection object and the 3D key position information corresponding to the detection object into a position determination model to obtain 3D position information of the detection object; wherein the 3D position information is used to indicate the position of the detection object in a 3D virtual environment, and the position determination model is obtained by the position determination model training method as described in the first aspect above.

[0020] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the position determination model training method of the first aspect and the position determination method of the second aspect in the above-mentioned embodiments are implemented.

[0021] According to a sixth aspect of the present disclosure, there is provided an electronic device, including:

[0022] Processor; and

[0023] The memory is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the first aspect of the position determination model training method and the second aspect of the position determination method in the above-mentioned embodiments.

[0024] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0025] In a position determination model training method provided by an embodiment of the present disclosure, sample data of the detection object can be obtained; wherein the sample data of the detection object includes 2D key position information, 3D key position information and 3D position information labels, the 2D key position information and the 3D key position information are input into a first model to be trained to obtain predicted 3D position information of the detection object, and the neural network parameters of the first model to be trained are updated according to the predicted 3D position information and the 3D position information labels of the detection object to obtain a position determination model.

[0026] In the embodiments of the present disclosure, the 2D key position information and the 3D key position information can be input into the model to be trained to obtain the predicted 3D position information, and the model to be trained can be trained according to the predicted 3D position information and the label. The 3D key position information of the detection object can be added to the position determination model, and the spatial information of the detection object can be integrated into the position determination model, thereby improving the accuracy of the position of the detection object in the virtual scene.

[0027] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0029] Figure 1 A schematic diagram schematically shows an exemplary system architecture of a location determination model training method in an exemplary embodiment of the present disclosure;

[0030] Figure 2 A flowchart of a location determination model training method in an exemplary embodiment of the present disclosure is schematically shown;

[0031] Figure 3 A flowchart schematically illustrates a method of determining predicted 3D position information of a detection object according to a first prediction result and a second prediction result in an exemplary embodiment of the present disclosure;

[0032] Figure 4 A schematic diagram schematically illustrates a position determination model in an exemplary embodiment of the present disclosure;

[0033] Figure 5 A flowchart schematically illustrates that when a speed error between a detected object's motion speed and a predicted motion speed of the detected object is less than a first threshold value, the first model to be trained satisfies a convergence condition, and the first model to be trained is determined as a position determination model in an exemplary embodiment of the present disclosure;

[0034] Figure 6 A flowchart schematically illustrates an exemplary embodiment of the present disclosure in which, when an acceleration error between a motion acceleration of a detection object and a predicted motion acceleration of the detection object is less than a second threshold, the first model to be trained satisfies a convergence condition and the first model to be trained is determined as a position determination model;

[0035] Figure 7 A flowchart schematically illustrates a method of determining sample data of a detection object according to 2D key position information corresponding to the detection object, 3D key position information corresponding to the detection object, and 3D position information labels corresponding to the detection object in an exemplary embodiment of the present disclosure;

[0036] Figure 8 A flowchart of updating the neural network parameters of the second model to be trained according to the predicted 3D key position information and the 3D key position label to obtain the 3D key position estimation model in an exemplary embodiment of the present disclosure is schematically shown;

[0037] Fig. 9 A flow chart schematically illustrates a method for determining a position in an exemplary embodiment of the present disclosure;

[0038] Fig.10 A schematic diagram schematically shows the composition of a position determination model training device in an exemplary embodiment of the present disclosure;

[0039] Fig.11 A schematic diagram schematically shows the composition of a position determination device in an exemplary embodiment of the present disclosure;

[0040] Fig.12 The structure diagram of a computer system suitable for implementing the electronic device of the exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0041] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations, materials, or operations are not shown or described in detail to avoid blurring various aspects of the present disclosure.

[0042] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0043] Figure 1 A schematic diagram of an exemplary system architecture of a location determination model training method or a location determination method to which an embodiment of the present disclosure can be applied is shown.

[0044] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 1001, 1002, 1003, a network 1004, and a server 1005. The network 1004 is used to provide a medium for communication links between the terminal devices 1001, 1002, 1003 and the server 1005. The network 1004 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0045] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 1005 may be a server cluster composed of multiple servers.

[0046] Users can use terminal devices 1001, 1002, 1003 to interact with server 1005 through network 1004 to receive or send messages, etc. Terminal devices 1001, 1002, 1003 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, etc. In addition, server 1005 can be a server that provides various services.

[0047] In one embodiment, the execution subject of the position determination model training method disclosed in the present invention may be server 1005, which may obtain the sample data of the detection object sent by terminal devices 1001, 1002, and 1003, and input the 2D key position information and the 3D key position information into the first model to be trained to obtain the predicted 3D position information of the detection object, and update the neural network parameters of the first model to be trained according to the predicted 3D position information and the 3D position information label of the detection object to obtain the position determination model. In addition, the position determination model training method disclosed in the present invention may also be executed by terminal devices 1001, 1002, 1003, etc., to realize the process of updating the neural network parameters of the first model to be trained according to the predicted 3D position information and the 3D position information label of the detection object to obtain the position determination model.

[0048] In addition, the implementation process of the location determination model training method disclosed in the present invention can also be implemented by the terminal devices 1001, 1002, 1003 and the server 1005. For example, the terminal devices 1001, 1002, 1003 can obtain the sample data of the detection object, and then send the sample data of the detection object to the server 1005, so that the server 1005 can input the 2D key position information and the 3D key position information into the first model to be trained to obtain the predicted 3D position information of the detection object, and update the neural network parameters of the first model to be trained according to the predicted 3D position information and the 3D position information label of the detection object to obtain the location determination model.

[0049] With the rapid development of the Internet, the application areas of virtual scenes are becoming more and more extensive. For example, virtual live broadcast, virtual wear, game interaction, content creation and other scenes can realize corresponding business needs through virtual objects in virtual scenes.

[0050] In the prior art, the cost of the motion supplement scheme of a single camera module is low. The motion of a person can be captured by a single camera module, and the captured motion can be mapped to a virtual scene to meet business needs. However, when the prior art uses a single camera module to capture motion, the motion of the person can be captured, but it is difficult to determine the position of the person in the virtual scene. In the virtual scene, it is easy for the virtual object to perform the corresponding action, but the position of the movement is wrong. For example, when a person walks, when the action is captured by a single camera module, the bending change of the leg can be captured, but the position of the virtual object in the virtual scene remains unchanged.

[0051] According to the location determination model training method provided in this exemplary embodiment, sample data of the detection object can be obtained, 2D key location information and 3D key location information can be input into the first model to be trained to obtain the predicted 3D location information of the detection object, and the neural network parameters of the first model to be trained can be updated according to the predicted 3D location information and 3D location information label of the detection object to obtain the location determination model. Figure 2 As shown, the location determination model training method may include the following steps S210 to S230:

[0052] Step S210, acquiring detection object sample data; wherein the detection object sample data includes 2D key position information, 3D key position information and 3D position information tags; wherein the 3D position information tags are used to indicate the position of the detection object in the 3D virtual environment;

[0053] Step S220, inputting the 2D key position information and the 3D key position information into the first model to be trained to obtain the predicted 3D position information of the detection object;

[0054] Step S230, updating the neural network parameters of the first model to be trained according to the predicted 3D position information and the 3D position information label of the detected object to obtain a position determination model.

[0055] In the embodiments of the present disclosure, 2D key position information and 3D key position information can be input into the model to be trained to obtain predicted 3D position information, and the model to be trained can be trained based on the predicted 3D position information and labels. The 3D key position information of the detection object can be added to the position determination model, and the spatial information of the detection object can be integrated into the position determination model, thereby improving the accuracy of the position of the detection object in the virtual scene.

[0056] Next, steps S210 to S230 of the position determination model in this exemplary embodiment will be described in more detail with reference to the accompanying drawings and embodiments.

[0057] Step S210, obtaining sample data of the detection object;

[0058] In an example embodiment of the present disclosure, sample data of the detection object may be obtained. The sample data of the detection object includes 2D key position information, 3D key position information, and a 3D position information tag, and the 3D position information tag is used to indicate the position of the detection object in the 3D virtual environment. For example, the solution of the present disclosure may be applied to a virtual reality scene. It should be noted that the present disclosure does not specifically limit the type of the 3D virtual environment.

[0059] Specifically, 2D key position information is used to indicate the key position of the detection object on the plane. For example, if the detection object is a person, the 2D key position information may include key positions such as the person's head and limbs. 3D key position information is used to indicate the key position of the detection object in space. For example, if the detection object is a person, the 3D key position information may include key positions such as the person's head and limbs. The 3D position information tag is used to indicate the position of the detection object in a 3D virtual environment. For example, if the detection object is a person, the 3D position information tag can be used to indicate the specific position of the person in the virtual 3D environment.

[0060] Furthermore, a plane coordinate system can be established to represent 2D key position information; a spatial coordinate system can be established to represent 3D key position information. For example, if the detection object is a person, a spatial coordinate system can be established with the center of gravity of the human body as the coordinate origin, and the coordinates of key positions such as the limbs and head of the human body can be represented by this coordinate system; a spatial coordinate system can be established to represent 3D position information tags. For example, a spatial coordinate system can be established with the position of the virtual lens in the virtual environment as the coordinate origin, and the position of the detection object can be used as the 3D position information tag.

[0061] It should be noted that the present disclosure does not specifically limit the specific forms of 2D key position information, 3D key position information and 3D position information tags.

[0062] Further, detecting the position of the object in the 3D virtual environment may include detecting the position of the center of gravity of the object, or detecting the position of the object in the 3D virtual environment may include detecting the position of the vertices of the object, or detecting the position of the object in the 3D virtual environment may include detecting the position of the lowest point of the object.

[0063] It should be noted that the present disclosure does not specifically limit the reference standard for detecting the position of the object in the 3D virtual environment.

[0064] Step S220, inputting the 2D key position information and the 3D key position information into the first model to be trained to obtain the predicted 3D position information of the detection object;

[0065] In an exemplary embodiment of the present disclosure, after obtaining the sample data of the detection object through the above steps, the 2D key position information and the 3D key position information can be input into the first model to be trained to obtain the predicted 3D position information of the detection object. Specifically, the first model to be trained can be used to predict the predicted 3D position information of the detection object according to the 2D key position information corresponding to the detection object and the 3D key position information corresponding to the detection object, and the predicted 3D position information of the detection object is used to indicate the predicted position of the detection object in the 3D virtual environment.

[0066] Specifically, the first model to be trained may be a model having a neural network, and the first model to be trained may include a convolutional layer, a pooling layer, an excitation layer, a fully connected layer, etc. For example, the first model to be trained may be a model based on a atrous convolutional neural network, or the first model to be trained may be a model based on a dense convolutional neural network.

[0067] Furthermore, after the 2D key position information and the 3D key position information are input into the first model to be trained, the 2D key position information and the 3D key position information can be converted into 2D key position feature vectors and 3D key position feature vectors, and then the 2D key position feature vectors and the 3D key position feature vectors are input into the first model to be trained to obtain the predicted 3D position information of the detection object.

[0068] For example, a vector conversion model based on graph neural networks (GNN) can be used to convert 2D key position information into 2D key position feature vectors, and 3D key position information into 3D key position feature vectors, and then the 2D key position feature vectors and 3D key position feature vectors are input into the first model to be trained to obtain the predicted 3D position information of the detected object.

[0069] It should be noted that the present disclosure does not specifically limit the specific method of converting the 2D key position information and the 3D key position information into the 2D key position feature vector and the 3D key position feature vector.

[0070] In an exemplary embodiment of the present disclosure, the first model to be trained may include multiple hidden layers, and the hidden layers may include convolution layers, normalization layers, excitation layers, etc. The 2D key position information and the 3D key position information may be sequentially input into the multiple hidden layers of the first model to be trained to obtain hidden layer calculation results, and the predicted 3D position information of the detection object may be obtained through the hidden layer calculation results.

[0071] In an exemplary embodiment of the present disclosure, the 2D key position information can be input into the first sub-model to be trained to obtain a first prediction result, and the 3D key position information can be input into the second sub-model to be trained to obtain a second prediction result, and the predicted 3D position information of the detection object can be determined according to the first prediction result and the second prediction result. Figure 3 As shown, determining the predicted 3D position information of the detection object according to the first prediction result and the second prediction result may include the following steps S310-S320:

[0072] Step S310, inputting the 2D key position information into the first sub-model to be trained to obtain a first prediction result, and inputting the 3D key position information into the second sub-model to be trained to obtain a second prediction result;

[0073] In an exemplary embodiment of the present disclosure, after obtaining the 2D key position information and the 3D key position information corresponding to the detection object through the above steps, the 2D key position information can be input into the first sub-model to be trained to obtain a first prediction result, and the 3D key position information can be input into the second sub-model to be trained to obtain a second prediction result. Specifically, the first sub-model to be trained can include the first sub-model to be trained and the second sub-model to be trained. The first sub-model to be trained is used to process the 2D key position information corresponding to the detection object, and the second sub-model to be trained is used to process the 3D key position information corresponding to the detection object.

[0074] Specifically, the first sub-model to be trained may include at least one hidden layer, which may include a convolutional layer, a normalization layer, an excitation layer, etc. The first sub-model to be trained may include at least one hidden layer, which may include a convolutional layer, a normalization layer, an excitation layer, etc.

[0075] For example, the 2D key position information corresponding to the detection object can be input into the first hidden layer corresponding to the first sub-model to be trained to obtain the first hidden layer result corresponding to the first sub-model to be trained, and then the first hidden layer result corresponding to the first sub-model to be trained is input into the second hidden layer corresponding to the first sub-model to be trained to obtain the second hidden layer result corresponding to the first sub-model to be trained, and then the second hidden layer result corresponding to the first sub-model to be trained is input into the third hidden layer corresponding to the first sub-model to be trained to obtain the first prediction result; the 3D key position information corresponding to the detection object can be input into the first hidden layer corresponding to the second sub-model to be trained to obtain the first hidden layer result corresponding to the second sub-model to be trained, and then the first hidden layer result corresponding to the second sub-model to be trained is input into the second hidden layer corresponding to the second sub-model to be trained to obtain the second prediction result.

[0076] Step S320: determining predicted 3D position information of the detection object according to the first prediction result and the second prediction result.

[0077] In an exemplary embodiment of the present disclosure, after the first prediction result and the second prediction result are obtained through the above steps, the predicted 3D position information of the detection object can be determined according to the first prediction result and the second prediction result. Specifically, the first prediction result and the second prediction result can be input into a fully connected layer, and then processed through the hidden layer of the first model to be trained to obtain the predicted 3D position information of the detection object.

[0078] For example, Figure 4 As shown, the 2D key position information can be input into the first sub-model to be trained, which includes 6 hidden layers, each of which includes a convolution layer, a normalization layer, and an activation layer, to obtain a first prediction result, and the 3D key position information can be input into the second sub-model to be trained, which includes 1 hidden layer, each of which includes a convolution layer, a normalization layer, and an activation layer, to obtain a second prediction result, and the first prediction result and the second prediction result can be input into the fully connected layer, and then input into the hidden layer of the first model to be trained to obtain the predicted 3D position information of the detected object. Among them, the convolution layer can include multiple convolution parameters, such as the convolution kernel size, the number of input channels, and the number of output channels, such as Figure 4 The three numbers in (5,51,1024) represent the convolution kernel size, the number of input channels, and the number of output channels of the convolution layer respectively.

[0079] Through the above steps S310 to S320, the 2D key position information can be input into the first sub-model to be trained to obtain a first prediction result, and the 3D key position information can be input into the second sub-model to be trained to obtain a second prediction result. The predicted 3D position information of the detection object is determined according to the first prediction result and the second prediction result. Through the embodiments of the present disclosure, the 2D key position information and the 3D key position information can be input into different sub-models to be trained, respectively, and different neural network structures are used for key information of different dimensions, which can improve the learning efficiency of key position information of different dimensions and improve the training efficiency of the first model to be trained.

[0080] In an exemplary embodiment of the present disclosure, the 2D key position information and 3D key position information corresponding to a first number of detection object sample data may be input into a first model to be trained to obtain predicted 3D position information of a second number of detection objects. The detection object sample data corresponding to the first number of 2D key position information are data acquired within a first time period, the detection object sample data corresponding to the second number of 3D key position information and the second number of predicted 3D position information are data acquired within a second time period, the second time period is within the time interval of the first time period, and the first number is greater than the second number. Specifically, 3D position information of a shorter time period may be predicted using 2D key position information of a longer time period. That is, when predicting the tth time period, i Frame to t jWhen predicting the 3D position information of the detected object in the frame, the tth i -Δt frame to tth frame i +Δt frame’s 2D key position information and the tth i Frame to t j Using longer 2D key position information to predict shorter 3D position information can improve the accuracy of the predicted 3D position information of the detected object.

[0081] Step S230, updating the neural network parameters of the first model to be trained according to the predicted 3D position information and the 3D position information label of the detected object to obtain a position determination model.

[0082] In an exemplary embodiment of the present disclosure, after the predicted 3D position information of the detection object is obtained through the above steps, the 3D position information label in the detection object sample data can be obtained, and the neural network parameters of the first model to be trained are updated according to the predicted 3D position information and the 3D position information label of the detection object to obtain a position determination model. Specifically, the predicted 3D position information of the detection object obtained through the above steps can be used to indicate the predicted position of the detection object in the 3D virtual environment, and the predicted 3D position information of the detection object is a predicted value. At this time, the true value corresponding to the detection object can be obtained, that is, the 3D position information label in the detection object sample data, and the 3D position information label can be used to indicate the position of the detection object in the 3D virtual environment. At this time, the predicted 3D position information (predicted value) of the detection object can be compared with the 3D position information label (true value) of the detection object to obtain the predicted difference between the predicted 3D position information (predicted value) of the detection object and the 3D position information label (true value) of the detection object, and the neural network parameters of the first model to be trained are updated according to the predicted difference to obtain a position determination model.

[0083] Specifically, the neural network parameters of the first model to be trained may include the number of model layers, the number of feature channels, the learning rate, etc. When the neural network parameters of the first model to be trained are updated according to the prediction difference to obtain the position determination model, the number of model layers, the number of feature channels, and the learning rate of the first model to be trained may be updated to obtain the position determination model.

[0084] It should be noted that the present disclosure does not specifically limit the specific type of the neural network parameters of the first model to be trained.

[0085] In an exemplary embodiment of the present disclosure, the neural network parameters of the first model to be trained may be updated by a back-propagation algorithm, and after the training is completed, a position determination model is obtained.

[0086] It should be noted that the present disclosure does not specifically limit the specific method of updating the neural network parameters of the first model to be trained according to the predicted 3D position information and the 3D position information label of the detection object.

[0087] In an exemplary embodiment of the present disclosure, the neural network parameters of the first model to be trained can be updated according to the predicted 3D position information and the 3D position information label of the detected object, and when the first model to be trained meets the convergence condition, the first model to be trained is determined as the position determination model. Specifically, the first model to be trained meets the convergence condition, which means that the prediction accuracy of the first model to be trained is high and can be applied. For example, the convergence condition may include the number of training times, such as when the first model to be trained is trained N times, the training is terminated; for another example, the convergence condition may include the training duration, such as when the first model to be trained is trained for T duration, the training is terminated.

[0088] It should be noted that the present disclosure does not make any special restrictions on the specific content of the convergence conditions. By applying convergence conditions to the model, the training process of the first model to be trained can be better controlled to avoid the problem of over-training of the neural network, thereby improving the training efficiency of the first model to be trained.

[0089] In an exemplary embodiment of the present disclosure, the predicted 3D position information of multiple detection objects can be obtained, the movement speed of the detection object can be determined according to the 3D position information labels in the sample data of the multiple detection objects, and the predicted movement speed of the detection object can be determined according to the predicted 3D position information of the multiple detection objects. When the speed error between the movement speed of the detection object and the predicted movement speed of the detection object is less than a first threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as the position determination model. Figure 5 As shown, when the speed error between the movement speed of the detection object and the predicted movement speed of the detection object is less than the first threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as the position determination model, which may include the following steps S510 to S540:

[0090] Step S510, obtaining predicted 3D position information of multiple detection objects; wherein the predicted 3D position information of the multiple detection objects is obtained by predicting sample data of multiple detection objects within a third time period;

[0091] In an exemplary embodiment of the present disclosure, the above steps can be used to predict multiple detection object sample data to obtain predicted 3D position information of multiple detection objects. The predicted 3D position information of multiple detection objects is obtained by predicting multiple detection object sample data within a third time period. Specifically, the predicted 3D position information of the multiple detection objects obtained is 3D position information of multiple consecutive frames within the third time period.

[0092] It should be noted that the present disclosure does not specifically limit the specific method of obtaining the predicted 3D position information of multiple detection objects.

[0093] Step S520, determining the movement speed of the detection object according to the 3D position information tags in the sample data of the detection object;

[0094] In an exemplary embodiment of the present disclosure, after the predicted 3D position information of multiple detection objects is obtained through the above steps, the motion speed of the detection object can be determined according to the 3D position information tags in the multiple detection object sample data. Specifically, the detection object sample data corresponding to the multiple predicted 3D position information can be obtained, and the 3D position information tags corresponding to the detection object sample data can be obtained, and the motion trajectories of the 3D position information tags corresponding to the object sample data can be recorded. At this time, the motion speed of the detection object can be calculated according to the motion trajectories of the 3D position information tags corresponding to the detection object sample data.

[0095] Specifically, the movement speed of the detection object may include an average speed of the detection object in the third time period, and may also include an instantaneous speed of the detection object at a certain moment in the third time period.

[0096] It should be noted that the present disclosure does not specifically limit the specific form of the motion speed of the detection object and the specific method of determining the motion speed of the detection object based on the 3D position information tags in the sample data of multiple detection objects.

[0097] Step S530, determining a predicted motion speed of the detection object according to the predicted 3D position information of the multiple detection objects;

[0098] In an exemplary embodiment of the present disclosure, after the predicted 3D position information of the multiple detection objects is obtained through the above steps, the predicted motion speed of the detection object can be determined according to the predicted 3D position information of the multiple detection objects. Specifically, the motion trajectory of the predicted 3D position information corresponding to the object sample data can be recorded, and the motion speed of the detection object can be calculated according to the motion trajectory of the predicted 3D position information corresponding to the detection object sample data.

[0099] Specifically, the predicted movement speed of the detection object may include an average predicted speed of the detection object in the third time period, and may also include an instantaneous predicted speed of the detection object at a certain moment in the third time period.

[0100] It should be noted that the present disclosure does not specifically limit the specific form of the predicted motion speed of the detection object. If the motion speed of the detection object determined by the above steps is an average speed, it is necessary to determine the average predicted speed of the detection object based on the predicted 3D position information of multiple detection objects; if the motion speed of the detection object determined by the above steps is an instantaneous speed, it is necessary to determine the instantaneous predicted speed of the detection object based on the predicted 3D position information of multiple detection objects.

[0101] In an exemplary embodiment of the present disclosure, after obtaining the motion trajectory of the detection object and the motion trajectory of the detection object, the motion speed of the detection object can be determined by the motion trajectory of the detection object, and the predicted motion speed of the detection object can be determined by the predicted motion trajectory of the detection object. Specifically, the motion speed of the detection object can be obtained by calculating the first-order derivative of the motion trajectory of the detection object, and the predicted motion speed of the detection object can be obtained by calculating the first-order derivative of the predicted motion trajectory of the detection object.

[0102] It should be noted that the present disclosure does not specifically limit the specific method of determining the movement speed of the detection object based on the 3D position information tags in the multiple detection object sample data and determining the predicted movement speed of the detection object based on the predicted 3D position information of the multiple detection objects.

[0103] Step S540: When the speed error between the movement speed of the detection object and the predicted movement speed of the detection object is less than a first threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as a position determination model.

[0104] In an exemplary embodiment of the present disclosure, after obtaining the movement speed of the detection object and the predicted movement speed of the detection object through the above steps, the movement speed of the detection object and the predicted movement speed of the detection object can be compared. When the speed error between the movement speed of the detection object and the predicted movement speed of the detection object is less than a first threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as a position determination model. Specifically, when performing model training, a convergence condition can be configured for model training, that is, the convergence condition is that when the speed error between the movement speed of the detection object and the predicted movement speed of the detection object is less than a first threshold, the first model to be trained meets the convergence condition, that is, at this time, the movement speed of the detection object and the predicted movement speed of the detection object can be subtracted to obtain a speed error. When the speed error is less than the first threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as a position determination model.

[0105] It should be noted that the present disclosure does not specifically limit the specific value of the first threshold.

[0106] In an exemplary embodiment of the present disclosure, the speed error between the motion speed of the detected object and the predicted motion speed of the detected object may include a direct error. For example, the difference between the motion speed of the detected object and the predicted motion speed of the detected object may be directly calculated, and the difference is the direct error. Alternatively, the difference between the motion speed of the detected object and the predicted motion speed of the detected object may be calculated, and the squared result of the difference may be obtained.

[0107] It should be noted that the present disclosure does not specifically limit the specific type of speed error.

[0108] Through the above steps S510 to S540, the predicted 3D position information of multiple detection objects can be obtained, the movement speed of the detection object can be determined according to the 3D position information labels in the sample data of the multiple detection objects, and the predicted movement speed of the detection object can be determined according to the predicted 3D position information of the multiple detection objects. When the speed error between the movement speed of the detection object and the predicted movement speed of the detection object is less than a first threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as the position determination model. Through the embodiments of the present disclosure, on the basis of satisfying the accuracy of the predicted 3D position information, the accuracy of the movement speed of the predicted 3D position information of the detection object is also improved, so that the movement speed of the mapping object of the detection object in the virtual environment is more consistent with the movement speed of the detection object.

[0109] In an exemplary embodiment of the present disclosure, the predicted 3D position information of multiple detection objects can be obtained, the motion acceleration of the detection object can be determined according to the 3D position information labels in the sample data of the multiple detection objects, and the predicted motion acceleration of the detection object can be determined according to the predicted 3D position information of the multiple detection objects. When the acceleration error between the motion acceleration of the detection object and the predicted motion acceleration of the detection object is less than a second threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as the position determination model. Figure 6 As shown, when the acceleration error between the motion acceleration of the detection object and the predicted motion acceleration of the detection object is less than the second threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as the position determination model, which may include the following steps S610-S640:

[0110] Step S610, obtaining predicted 3D position information of multiple detection objects; wherein the predicted 3D position information of the multiple detection objects is obtained by predicting sample data of multiple detection objects within a fourth time period;

[0111] In an exemplary embodiment of the present disclosure, the above steps can be used to predict the sample data of multiple detection objects to obtain the predicted 3D position information of the multiple detection objects. The predicted 3D position information of the multiple detection objects is obtained by predicting the sample data of multiple detection objects in the fourth time period. Specifically, the predicted 3D position information of the multiple detection objects is the 3D position information of multiple consecutive frames in the fourth time period.

[0112] It should be noted that the present disclosure does not specifically limit the specific method of obtaining the predicted 3D position information of multiple detection objects.

[0113] Step S620, determining the motion acceleration of the detection object according to the 3D position information tags in the sample data of the multiple detection objects;

[0114] In an exemplary embodiment of the present disclosure, after the predicted 3D position information of multiple detection objects is obtained through the above steps, the motion acceleration of the detection object can be determined according to the 3D position information tags in the multiple detection object sample data. Specifically, the detection object sample data corresponding to the multiple predicted 3D position information can be obtained, and the 3D position information tags corresponding to the detection object sample data can be obtained, and the motion trajectories of the 3D position information tags corresponding to the object sample data can be recorded. At this time, the motion acceleration of the detection object can be calculated according to the motion trajectories of the 3D position information tags corresponding to the detection object sample data.

[0115] Specifically, the movement speed of the detection object may include the average acceleration of the detection object in the fourth time period, and may also include the instantaneous acceleration of the detection object at a certain moment in the fourth time period.

[0116] It should be noted that the present disclosure does not specifically limit the specific form of the motion acceleration of the detection object and the specific method of determining the motion acceleration of the detection object based on the 3D position information tags in the sample data of multiple detection objects.

[0117] Step S630, determining a predicted motion acceleration of the detection object according to the predicted 3D position information of the multiple detection objects;

[0118] In an exemplary embodiment of the present disclosure, after the predicted 3D position information of the multiple detection objects is obtained through the above steps, the predicted motion acceleration of the detection object can be determined according to the predicted 3D position information of the multiple detection objects. Specifically, the motion trajectory of the predicted 3D position information corresponding to the object sample data can be recorded, and the motion acceleration of the detection object can be calculated according to the motion trajectory of the predicted 3D position information corresponding to the detection object sample data.

[0119] Specifically, the predicted motion acceleration of the detection object may include an average predicted acceleration of the detection object in the fourth time period, and may also include an instantaneous predicted acceleration of the detection object at a certain moment in the fourth time period.

[0120] It should be noted that the present disclosure does not specifically limit the specific form of the predicted motion acceleration of the detection object. If the motion acceleration of the detection object determined by the above steps is an average acceleration, it is necessary to determine the average predicted acceleration of the detection object based on the predicted 3D position information of multiple detection objects; if the motion speed of the detection object determined by the above steps is an instantaneous acceleration, it is necessary to determine the instantaneous predicted acceleration of the detection object based on the predicted 3D position information of multiple detection objects.

[0121] In an exemplary embodiment of the present disclosure, after obtaining the motion trajectory of the detection object and the motion trajectory of the detection object, the motion acceleration of the detection object can be determined by the motion trajectory of the detection object, and the predicted motion acceleration of the detection object can be determined by the predicted motion trajectory of the detection object. Specifically, the motion acceleration of the detection object can be obtained by calculating the second-order derivative of the motion trajectory of the detection object, and the predicted motion acceleration of the detection object can be obtained by calculating the second-order derivative of the predicted motion trajectory of the detection object.

[0122] It should be noted that the present disclosure does not specifically limit the specific method of determining the motion acceleration of the detection object based on the 3D position information tags in the multiple detection object sample data and determining the predicted motion acceleration of the detection object based on the predicted 3D position information of the multiple detection objects.

[0123] Step S640: When the acceleration error between the motion acceleration of the detection object and the predicted motion acceleration of the detection object is less than a second threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as the position determination model.

[0124] In an exemplary embodiment of the present disclosure, after obtaining the motion acceleration of the detection object and the predicted motion acceleration of the detection object through the above steps, the motion acceleration of the detection object and the predicted motion acceleration of the detection object can be compared. When the speed error between the motion acceleration of the detection object and the predicted motion acceleration of the detection object is less than a second threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as a position determination model. Specifically, when performing model training, a convergence condition can be configured for the model training, that is, the convergence condition is that when the acceleration error between the motion acceleration of the detection object and the predicted motion acceleration of the detection object is less than a second threshold, the first model to be trained meets the convergence condition, that is, at this time, the motion acceleration of the detection object and the predicted motion acceleration of the detection object can be subtracted to obtain the acceleration error. When the speed error is less than the second threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as a position determination model.

[0125] It should be noted that the present disclosure does not specifically limit the specific value of the second threshold.

[0126] In an exemplary embodiment of the present disclosure, the acceleration error between the motion acceleration of the detected object and the predicted motion acceleration of the detected object may include a direct error. For example, the difference between the motion acceleration of the detected object and the predicted motion acceleration of the detected object may be directly calculated, and the difference is the direct error. Alternatively, the difference between the motion acceleration of the detected object and the predicted motion acceleration of the detected object may be calculated, and the squared difference may be calculated to obtain the squared error.

[0127] It should be noted that the present disclosure does not specifically limit the specific type of acceleration error.

[0128] Through the above steps S610 to S640, the predicted 3D position information of multiple detection objects can be obtained, the motion acceleration of the detection object can be determined according to the 3D position information labels in the sample data of the multiple detection objects, and the predicted motion acceleration of the detection object can be determined according to the predicted 3D position information of the multiple detection objects. When the acceleration error between the motion acceleration of the detection object and the predicted motion acceleration of the detection object is less than a second threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as the position determination model. The acceleration error between the motion acceleration of the detection object and the predicted motion acceleration of the detection object is less than the second threshold as the convergence condition, which can improve the accuracy of the motion acceleration of the mapping object of the detection object in the virtual environment, so that the movement of the mapping object in the virtual environment can match the movement of the detection object, with lower latency and better motion smoothness.

[0129] In an exemplary embodiment of the present disclosure, a predicted 3D position information difference can be determined based on the predicted 3D position information of the detection object and the 3D position information label. When the predicted 3D position information difference is less than a third threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as a position determination model. Specifically, the predicted 3D position information of the detection object and the 3D position information label can be compared to determine the predicted 3D position information difference. For example, the Euclidean distance between the predicted 3D position information of the detection object and the 3D position information label can be calculated, and the Euclidean distance is used as the predicted 3D position information difference. When the predicted 3D position information difference is less than the third threshold, it indicates that the first model to be trained meets the convergence condition, and the first model to be trained can be determined as a position determination model.

[0130] It should be noted that the present disclosure does not specifically limit the specific method of determining the difference between the predicted 3D position information and the 3D position information tag of the detection object and the specific value of the third threshold.

[0131] In an exemplary embodiment of the present disclosure, an image corresponding to the detection object can be obtained, the image corresponding to the detection object can be input into a 2D key position detection model to obtain 2D key position information corresponding to the detection object, a multi-view reconstruction method can be used to obtain a 3D position information label corresponding to the detection object, and 3D key position information corresponding to the detection object can be obtained, and the detection object sample data can be determined based on the 2D key position information corresponding to the detection object, the 3D key position information corresponding to the detection object, and the 3D position information label corresponding to the detection object. Figure 7 As shown, determining the detection object sample data according to the 2D key position information corresponding to the detection object, the 3D key position information corresponding to the detection object, and the 3D position information label corresponding to the detection object may include the following steps S710 to S730:

[0132] Step S710, obtaining an image corresponding to the detection object, and inputting the image corresponding to the detection object into a 2D key position detection model to obtain 2D key position information corresponding to the detection object;

[0133] In an exemplary embodiment of the present disclosure, an image corresponding to the detection object can be obtained. Specifically, the image corresponding to the detection object can be obtained by a camera module of a terminal device. It should be noted that the present disclosure does not specifically limit the specific method for obtaining the image corresponding to the detection object.

[0134] After obtaining the image corresponding to the detection object through the above steps, the image corresponding to the detection object can be input into the 2D key position detection model to obtain the 2D key position information corresponding to the detection object. Specifically, the 2D key position detection model can be used to obtain the 2D key position corresponding to the detection object in the image corresponding to the detection object, and the 2D key position detection model can be a pre-trained model based on a neural network.

[0135] It should be noted that the present disclosure does not specifically limit the specific form of the 2D key position detection model.

[0136] Step S720, using a multi-view reconstruction method to obtain a 3D position information label corresponding to the detection object, and obtaining 3D key position information corresponding to the detection object;

[0137] In an exemplary embodiment of the present disclosure, a multi-perspective reconstruction method may be used to obtain a 3D position information label corresponding to a detection object, and obtain 3D key position information corresponding to the detection object. Specifically, the multi-perspective reconstruction method refers to a method of estimating a corresponding three-dimensional structure from a two-dimensional image sequence that may be combined with a local motion signal, that is, a three-dimensional structure may be reconstructed from multiple two-dimensional images. For example, if the detection object is a human body, multiple camera modules may be used to obtain images corresponding to multiple detection objects at multiple shooting angles, and the 3D position information label corresponding to the detection object may be determined based on the images corresponding to the multiple detection objects, and the 3D key position information corresponding to the detection object may be obtained.

[0138] Specifically, after obtaining the 3D position information label corresponding to the detection object through the multi-view reconstruction method, the 3D key position information corresponding to the detection object can be determined according to the 3D position information label corresponding to the detection object.

[0139] For example, the 3D position information label corresponding to the detection object includes multiple mapping points of the detection object mapping the object in the virtual environment. Mapping points of key parts of the human body can be selected from the multiple mapping points as the 3D key position information corresponding to the detection object, such as mapping points of the human head, limbs, etc. as the 3D key position information corresponding to the detection object.

[0140] It should be noted that the present disclosure does not specifically limit the specific method of using the multi-view reconstruction method to obtain the 3D position information label corresponding to the detection object and the specific method of obtaining the 3D key position information corresponding to the detection object.

[0141] Step S730 , determining the detection object sample data according to the 2D key position information corresponding to the detection object, the 3D key position information corresponding to the detection object, and the 3D position information label corresponding to the detection object.

[0142] In an example embodiment of the present disclosure, after obtaining the 2D key position information corresponding to the detection object, the 3D key position information corresponding to the detection object and the 3D position information label corresponding to the detection object through the above steps, the 2D key position information corresponding to the detection object, the 3D key position information corresponding to the detection object and the 3D position information label corresponding to the detection object can be synthesized into detection object sample data, and the detection object sample data can be used to train the first model to be trained.

[0143] Through the above steps S710 to S730, an image corresponding to the detection object can be obtained, and the image corresponding to the detection object can be input into the 2D key position detection model to obtain the 2D key position information corresponding to the detection object, and a multi-perspective reconstruction method is used to obtain the 3D position information label corresponding to the detection object, and the 3D key position information corresponding to the detection object is obtained. According to the 2D key position information corresponding to the detection object, the 3D key position information corresponding to the detection object and the 3D position information label corresponding to the detection object, the detection object sample data is determined. Through the embodiments of the present disclosure, training data can be produced to solve the problem of insufficient training data when training the first model to be trained.

[0144] In an exemplary embodiment of the present disclosure, the 2D key position information corresponding to the detection object can be input into the 3D key position estimation model to obtain the 3D key position information corresponding to the detection object. Specifically, the 3D key position estimation model can be used to estimate the 3D key position information corresponding to the detection object through the 2D key position information corresponding to the detection object. For example, the detection object is a human body, and the 2D key position information corresponding to the detection object is the 2D head position, 2D limb position, etc. of the human body. At this time, the 2D key position information corresponding to the detection object can be input into the 3D key position estimation model to obtain the 3D key position information corresponding to the detection object, that is, the 3D head position, 3D limb position, etc. of the human body. It should be noted that the present disclosure does not make any special limitation on the neural network structure of the 3D key position estimation model. Through the embodiments of the present disclosure, the 3D key position information corresponding to the detection object can be obtained on the basis of obtaining the 2D key position information corresponding to the detection object, thereby improving the efficiency of obtaining the 3D position information of the detection object.

[0145] In an exemplary embodiment of the present disclosure, the 3D key position information corresponding to the detection object can be determined according to the 3D position information label corresponding to the detection object, the 2D key position information corresponding to the detection object is input into the second model to be trained to obtain the predicted 3D key position information, and the neural network parameters of the second model to be trained are updated according to the predicted 3D key position information and the 3D key position information to obtain the 3D key position estimation model. Figure 8As shown, updating the neural network parameters of the second to-be-trained model according to the predicted 3D key position information and the 3D key position label to obtain the 3D key position estimation model may include the following steps S810 to S830:

[0146] Step S810, determining 3D key position information corresponding to the detection object according to the 3D position information tag corresponding to the detection object;

[0147] In an exemplary embodiment of the present disclosure, after obtaining the 3D position information tag corresponding to the detection object through the above steps, the 3D key position information corresponding to the detection object can be determined based on the 3D position information tag corresponding to the detection object. Specifically, the 3D position information tag corresponding to the detection object may include the 3D position information of the detection object mapped in the virtual environment, and the 3D key position information corresponding to the detection object can be determined through the 3D position information. For example, if the detection object is a human body, the position information of the head and limbs in the 3D position information can be used as the 3D key position information corresponding to the detection object.

[0148] It should be noted that the present disclosure does not specifically limit the specific method of determining the 3D key position information corresponding to the detection object according to the 3D position information tag corresponding to the detection object.

[0149] Step S820, inputting the 2D key position information corresponding to the detected object into the second to-be-trained model to obtain predicted 3D key position information;

[0150] In an exemplary embodiment of the present disclosure, the 2D key position information corresponding to the detection object can be input into the second model to be trained to obtain the predicted 3D key position information. Specifically, the second model to be trained can be used to estimate the predicted 3D key position information corresponding to the detection object through the 2D key position information corresponding to the detection object.

[0151] Step S830, updating the neural network parameters of the second model to be trained according to the predicted 3D key position information and the 3D key position information to obtain a 3D key position estimation model.

[0152] In an example embodiment of the present disclosure, after obtaining the 3D key position information corresponding to the detected object and the predicted 3D key position information through the above steps, the neural network parameters of the second model to be trained can be updated according to the predicted 3D key position information and the 3D key position information to obtain a 3D key position estimation model. Specifically, the predicted 3D key position information and the 3D key position information can be compared to obtain a predicted difference, and the neural network parameters of the second model to be trained can be updated according to the predicted difference to obtain a 3D key position estimation model. For example, the back propagation algorithm can be used to update the neural network parameters of the second model to be trained.

[0153] It should be noted that the present disclosure does not specifically limit the specific method of updating the neural network parameters of the second model to be trained according to the predicted 3D key position information and the 3D key position information to obtain the 3D key position estimation model.

[0154] Through the above steps S810 to S830, the 3D key position information corresponding to the detection object can be determined according to the 3D position information label corresponding to the detection object, the 2D key position information corresponding to the detection object is input into the second model to be trained to obtain predicted 3D key position information, and the neural network parameters of the second model to be trained are updated according to the predicted 3D key position information and the 3D key position information to obtain a 3D key position estimation model. Through the embodiments of the present disclosure, the 3D key position estimation model can be trained. When using the position determination model of the present disclosure, the 3D key position estimation model can be used to estimate the 3D key position information corresponding to the detection object, and the 3D key position information corresponding to the detection object and the 2D key position information corresponding to the detection object are input into the position determination model to obtain 3D position information, thereby improving the efficiency of 3D position information prediction.

[0155] In an exemplary embodiment of the present disclosure, an image corresponding to a detection object can be obtained through a camera module, and the image corresponding to the detection object can be input into a 2D key position detection model to obtain 2D key position information corresponding to the detection object, and the 2D key position information corresponding to the detection object can be input into a 3D key position estimation model to obtain 3D key position information corresponding to the detection object, and the 2D key position information corresponding to the detection object and the 3D key position information corresponding to the detection object can be input into a position determination model to obtain 3D position information of the detection object. Fig. 9 As shown, inputting the 2D key position information corresponding to the detection object and the 3D key position information corresponding to the detection object into the position determination model to obtain the 3D position information of the detection object may include the following steps S910 to S940:

[0156] Step S910, acquiring an image corresponding to the detection object through a camera module;

[0157] In an exemplary embodiment of the present disclosure, an image corresponding to the detected object can be obtained. The solution of the present disclosure can be applied to scenarios such as virtual live broadcast, virtual wear, game interaction, content creation, etc.

[0158] For example, in a virtual live broadcast scenario, the host (detection object) can obtain the live broadcast screen through a single camera module and display it in a virtual live broadcast room. The use of a single camera module can determine the position of the virtual object in the virtual scene, improving the motion capture efficiency of the single camera module, thereby avoiding the problem of cost increase caused by using multiple camera modules in the same scene.

[0159] It should be noted that the present disclosure does not specifically limit the specific method of acquiring the image corresponding to the detection object through the camera module.

[0160] Step S920, inputting the image corresponding to the detection object into the 2D key position detection model to obtain the 2D key position information corresponding to the detection object;

[0161] In an exemplary embodiment of the present disclosure, after obtaining an image corresponding to the detection object through the above steps, the image corresponding to the detection object can be input into a 2D key position detection model to obtain 2D key position information corresponding to the detection object. Specifically, the 2D key position detection model may include a pre-trained neural network model, and the 2D key position detection model can be used to obtain 2D key position information corresponding to the detection object according to the image corresponding to the detection object.

[0162] In an exemplary embodiment of the present disclosure, the image corresponding to the detection object is an RGB image.

[0163] Step S930, inputting the 2D key position information corresponding to the detection object into the 3D key position estimation model to obtain the 3D key position information corresponding to the detection object;

[0164] In an exemplary embodiment of the present disclosure, after obtaining the 2D key position information corresponding to the detection object through the above steps, the 2D key position information corresponding to the detection object can be input into a 3D key position estimation model to obtain the 3D key position information corresponding to the detection object. Specifically, the 3D key position estimation model may include a neural network model trained in the present disclosure, and the 3D key position estimation model can be used to obtain the 3D key position information corresponding to the detection object based on the 2D key position information corresponding to the detection object.

[0165] Step S940, inputting the 2D key position information corresponding to the detection object and the 3D key position information corresponding to the detection object into the position determination model to obtain the 3D position information of the detection object;

[0166] In an exemplary embodiment of the present disclosure, after obtaining the 2D key position information corresponding to the detection object and the 3D key position information corresponding to the detection object through the above steps, the 2D key position information corresponding to the detection object and the 3D key position information corresponding to the detection object can be input into the position determination model to obtain the 3D position information of the detection object. The 3D position information is used to indicate the position of the detection object in the 3D virtual environment, and the position determination model is obtained by the position determination model training method as described above. Specifically, the 3D position information of the detection object can be used to indicate the 3D position of the mapping object of the detection object in the virtual environment.

[0167] It should be noted that the present disclosure does not specifically limit the specific forms of the 2D key position information, the 3D key position information corresponding to the detection object, and the 3D position information of the detection object.

[0168] Through the above steps S910 to S940, an image corresponding to the detection object can be obtained, and the image corresponding to the detection object can be input into the 2D key position detection model to obtain the 2D key position information corresponding to the detection object, and the 2D key position information corresponding to the detection object can be input into the 3D key position estimation model to obtain the 3D key position information corresponding to the detection object, and the 2D key position information corresponding to the detection object and the 3D key position information corresponding to the detection object can be input into the position determination model to obtain the 3D position information of the detection object.

[0169] In a position determination model training method provided by an embodiment of the present disclosure, sample data of the detection object can be obtained; wherein the sample data of the detection object includes 2D key position information, 3D key position information and 3D position information labels, the 2D key position information and the 3D key position information are input into a first model to be trained to obtain predicted 3D position information of the detection object, and the neural network parameters of the first model to be trained are updated according to the predicted 3D position information and the 3D position information labels of the detection object to obtain a position determination model.

[0170] In the embodiments of the present disclosure, 2D key position information and 3D key position information can be input into the model to be trained to obtain predicted 3D position information, and the model to be trained can be trained based on the predicted 3D position information and labels. The 3D key position information of the detection object can be added to the position determination model, and the spatial information of the detection object can be integrated into the position determination model, thereby improving the accuracy of the position of the detection object in the virtual scene.

[0171] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0172] In addition, in an exemplary embodiment of the present disclosure, a location determination model training device is also provided. Fig.10 As shown, a location determination model training device 1000 includes: a sample data acquisition unit 1010, a prediction information acquisition unit 1020 and a location determination model training unit 1030.

[0173] Among them, the sample data acquisition unit is configured to execute acquisition of sample data of the detection object; wherein the sample data of the detection object includes 2D key position information, 3D key position information and 3D position information label; wherein the 3D position information label is used to indicate the position of the detection object in the 3D virtual environment; the prediction information acquisition unit is configured to execute input of the 2D key position information and the 3D key position information into the first model to be trained to obtain the predicted 3D position information of the detection object; the position determination model training unit is configured to execute updating of the neural network parameters of the first model to be trained according to the predicted 3D position information and the 3D position information label of the detection object to obtain the position determination model.

[0174] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, the first model to be trained includes a first sub-model to be trained and a second sub-model to be trained, and the 2D key position information and the 3D key position information are input into the first model to be trained to obtain the predicted 3D position information of the detection object, and the position determination model training device also includes: a prediction result acquisition unit, configured to execute the input of the 2D key position information into the first sub-model to be trained to obtain a first prediction result, and the input of the 3D key position information into the second sub-model to be trained to obtain a second prediction result; a prediction result fusion unit, configured to execute the determination of the predicted 3D position information of the detection object according to the first prediction result and the second prediction result.

[0175] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, 2D key position information and 3D key position information are input into the first model to be trained to obtain predicted 3D position information of the detection object, and the position determination model training device also includes: a 3D position information prediction unit, which is configured to execute inputting a first number of 2D key position information and a second number of 3D key position information into the first model to be trained to obtain predicted 3D position information of the detection object of the second number; wherein the detection object sample data corresponding to the first number of 2D key position information is data acquired within a first time period, and the second number of 3D key position information is data acquired within a second time period; the second time period is within the time interval of the first time period, and the first number is greater than the second number.

[0176] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, the neural network parameters of the first model to be trained are updated according to the predicted 3D position information and the 3D position information label of the detection object to obtain a position determination model, and the position determination model training device also includes: a first convergence condition judgment unit, which is configured to execute the update of the neural network parameters of the first model to be trained according to the predicted 3D position information and the 3D position information label of the detection object, and when the first model to be trained meets the convergence condition, the first model to be trained is determined as the position determination model.

[0177] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, when the first model to be trained meets the convergence condition, the first model to be trained is determined as a position determination model, and the position determination model training device also includes: a first position acquisition unit, configured to execute acquisition of predicted 3D position information of multiple detection objects; wherein the predicted 3D position information of the multiple detection objects is obtained by predicting the sample data of the multiple detection objects within a third time period; a motion speed determination unit, configured to execute determination of the motion speed of the detection object according to the 3D position information label in the sample data of the multiple detection objects; a predicted motion speed determination unit, configured to execute determination of the predicted motion speed of the detection object according to the predicted 3D position information of the multiple detection objects; a second convergence condition judgment unit, configured to execute when the speed error between the motion speed of the detection object and the predicted motion speed of the detection object is less than a first threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as the position determination model.

[0178] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, when the first model to be trained meets the convergence condition, the first model to be trained is determined as a position determination model, and the position determination model training device also includes: a second position acquisition unit, configured to execute acquisition of predicted 3D position information of multiple detection objects; wherein the predicted 3D position information of the multiple detection objects is obtained by predicting the sample data of the multiple detection objects within a fourth time period; a motion acceleration determination unit, configured to execute determination of the motion acceleration of the detection object according to the 3D position information labels in the sample data of the multiple detection objects; a predicted motion acceleration determination unit, configured to execute determination of the predicted motion acceleration of the detection object according to the predicted 3D position information of the multiple detection objects; a third convergence condition judgment unit, configured to execute when the acceleration error between the motion acceleration of the detection object and the predicted motion acceleration of the detection object is less than a second threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as the position determination model.

[0179] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, when the first model to be trained meets the convergence condition, the first model to be trained is determined as a position determination model, and the position determination model training device also includes: a prediction difference determination unit, configured to determine the predicted 3D position information difference based on the predicted 3D position information of the detection object and the 3D position information label; a fourth convergence condition judgment unit, configured to execute when the predicted 3D position information difference is less than a third threshold, the first model to be trained meets the convergence condition, and determines the first model to be trained as a position determination model.

[0180] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, before acquiring the sample data of the detection object, the position determination model training device also includes: an image acquisition unit, configured to acquire an image corresponding to the detection object, and input the image corresponding to the detection object into the 2D key position detection model to obtain the 2D key position information corresponding to the detection object; a multi-perspective reconstruction unit, configured to acquire the 3D position information label corresponding to the detection object by using a multi-perspective reconstruction method, and acquire the 3D key position information corresponding to the detection object; a sample data determination unit, configured to determine the sample data of the detection object according to the 2D key position information corresponding to the detection object, the 3D key position information corresponding to the detection object and the 3D position information label corresponding to the detection object.

[0181] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, 3D key position information corresponding to the detection object is obtained, and the position determination model training device also includes: a 3D key position information estimation unit, which is configured to execute the input of the 2D key position information corresponding to the detection object into the 3D key position estimation model to obtain the 3D key position information corresponding to the detection object.

[0182] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, the position determination model training device also includes: a key position determination unit, configured to determine the 3D key position information corresponding to the detection object according to the 3D position information label corresponding to the detection object; a predicted key position determination unit, configured to execute a fifth intermediate calculation result acquisition unit, configured to input the 2D key position information corresponding to the detection object into the second model to be trained to obtain predicted 3D key position information; a key position estimation model training unit, configured to update the neural network parameters of the second model to be trained according to the predicted 3D key position information and the 3D key position information to obtain a 3D key position estimation model.

[0183] Since the various functional modules of the position determination model training device of the example embodiment of the present disclosure correspond to the steps of the example embodiment of the position determination model training method described above, for details not disclosed in the embodiment of the device of the present disclosure, please refer to the embodiment of the position determination model training method described above in the present disclosure.

[0184] In addition, in an exemplary embodiment of the present disclosure, a location determination model training device is also provided. Fig.11 As shown, a position determination device 1100 includes: a detection image acquisition unit 1110 , a 2D key position acquisition unit 1120 , a 3D key position acquisition unit 1130 and a position determination unit 1140 .

[0185] Among them, the detection image acquisition unit is configured to execute the acquisition of the image corresponding to the detection object; the 2D key position acquisition unit is configured to execute the input of the image corresponding to the detection object into the 2D key position detection model to obtain the 2D key position information corresponding to the detection object; the 3D key position acquisition unit is configured to execute the input of the 2D key position information corresponding to the detection object into the 3D key position estimation model to obtain the 3D key position information corresponding to the detection object; the position determination unit is configured to execute the input of the 2D key position information corresponding to the detection object and the 3D key position information corresponding to the detection object into the position determination model to obtain the 3D position information of the detection object; wherein the 3D position information is used to indicate the position of the detection object in the 3D virtual environment, and the position determination model is obtained by any of the position determination model training methods as mentioned above.

[0186] Since the various functional modules of the position determination device in the exemplary embodiment of the present disclosure correspond to the steps of the exemplary embodiment of the position determination method described above, for details not disclosed in the embodiment of the device of the present disclosure, please refer to the embodiment of the position determination method described above in the present disclosure.

[0187] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0188] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above-mentioned position determination model training method or position determination method is also provided.

[0189] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0190] Refer to the following Fig.12 1200 according to this embodiment of the present disclosure is described. Fig.12 The electronic device 1200 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0191] like Fig.12As shown, the electronic device 1200 is in the form of a general computing device. The components of the electronic device 1200 may include, but are not limited to: the at least one processing unit 1210, the at least one storage unit 1220, a bus 1230 connecting different system components (including the storage unit 1220 and the processing unit 1210), and a display unit 1240.

[0192] The storage unit stores program codes, which can be executed by the processing unit 1210, so that the processing unit 1210 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1210 can perform the following steps: Figure 2 In step S210, the sample data of the detection object is obtained; wherein the sample data of the detection object includes 2D key position information, 3D key position information and 3D position information label; wherein the 3D position information label is used to indicate the position of the detection object in the 3D virtual environment; step S220, the 2D key position information and the 3D key position information are input into the first model to be trained to obtain the predicted 3D position information of the detection object; step S230, the neural network parameters of the first model to be trained are updated according to the predicted 3D position information of the detection object and the 3D position information label to obtain the position determination model. Alternatively, the following can also be performed: Fig. 9 In step S910, an image corresponding to the detection object is obtained; in step S920, the image corresponding to the detection object is input into a 2D key position detection model to obtain 2D key position information corresponding to the detection object; in step S930, the 2D key position information corresponding to the detection object is input into a 3D key position estimation model to obtain 3D key position information corresponding to the detection object; in step S940, the 2D key position information corresponding to the detection object and the 3D key position information corresponding to the detection object are input into a position determination model to obtain 3D position information of the detection object;

[0193] For example, electronic devices can achieve Figure 2 and Fig. 9 The steps shown.

[0194] The storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 1221 and / or a cache memory unit 1222 , and may further include a read-only memory unit (ROM) 1223 .

[0195] The storage unit 1220 may also include a program / utility 1224 having a set (at least one) of program modules 1225, such program modules 1225 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0196] Bus 1230 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0197] The electronic device 1200 may also communicate with one or more external devices 1270 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1200, and / or communicate with any device that enables the electronic device 1200 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1250. In addition, the electronic device 1200 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1260. As shown, the network adapter 1260 communicates with other modules of the electronic device 1200 via a bus 1230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0198] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0199] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of the device to complete the above method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0200] In an exemplary embodiment, a computer program product is also provided, including a computer program / instruction, which implements the position determination model training method or the position determination method in the above-mentioned embodiment when the computer program / instruction is executed by a processor.

[0201] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0202] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training a location determination model, characterized in that: The method comprises: Acquire detection object sample data; wherein the detection object sample data includes 2D key position information, 3D key position information and 3D position information tag; wherein the 3D position information tag is used to indicate the position of the detection object in the 3D virtual environment; Inputting the 2D key position information and the 3D key position information into a first model to be trained to obtain predicted 3D position information of the detection object, wherein the first model to be trained includes a first sub-model to be trained and a second sub-model to be trained; According to the predicted 3D position information of the detected object and the 3D position information label, updating the neural network parameters of the first to-be-trained model to obtain a position determination model; The step of inputting the 2D key position information and the 3D key position information into a first model to be trained to obtain the predicted 3D position information of the detection object includes: Input the 2D key position information into the first sub-model to be trained to obtain a first prediction result, and input the 3D key position information into the second sub-model to be trained to obtain a second prediction result; The predicted 3D position information of the detection object is determined according to the first prediction result and the second prediction result.

2. The method according to claim 1, characterized in that The step of inputting the 2D key position information and the 3D key position information into a first model to be trained to obtain the predicted 3D position information of the detection object includes: Inputting the first quantity of 2D key position information and the second quantity of 3D key position information into the first to-be-trained model to obtain predicted 3D position information of the second quantity of detection objects; Among them, the detection object sample data corresponding to the first number of 2D key position information is data obtained within a first time period, and the second number of 3D key position information is data obtained within a second time period; the second time period is within the time interval of the first time period, and the first number is greater than the second number.

3. The method according to claim 1, characterized in that The updating of the neural network parameters of the first to-be-trained model according to the predicted 3D position information of the detected object and the 3D position information label to obtain a position determination model includes: The neural network parameters of the first model to be trained are updated according to the predicted 3D position information of the detection object and the 3D position information label, and when the first model to be trained meets the convergence condition, the first model to be trained is determined as a position determination model.

4. The method according to claim 3, characterized in that: When the first model to be trained meets the convergence condition, determining the first model to be trained as the position determination model includes: Acquire predicted 3D position information of multiple detection objects; wherein the predicted 3D position information of the multiple detection objects is obtained by predicting sample data of multiple detection objects within a third time period; Determine the movement speed of the detection object according to the 3D position information tags in the plurality of detection object sample data; Determining a predicted movement speed of the detection object according to the predicted 3D position information of the multiple detection objects; When a speed error between the movement speed of the detection object and the predicted movement speed of the detection object is less than a first threshold, the first model to be trained meets a convergence condition, and the first model to be trained is determined as a position determination model.

5. The method according to claim 3, characterized in that: The step of determining the first model to be trained as a position determination model when the first model to be trained meets a convergence condition includes: Acquire predicted 3D position information of multiple detection objects; wherein the predicted 3D position information of the multiple detection objects is obtained by predicting sample data of multiple detection objects within a fourth time period; Determine the motion acceleration of the detection object according to the 3D position information tags in the plurality of detection object sample data; Determining a predicted motion acceleration of the detection object according to the predicted 3D position information of the multiple detection objects; When the acceleration error between the motion acceleration of the detection object and the predicted motion acceleration of the detection object is less than a second threshold, the first model to be trained meets the convergence condition, and the first model to be trained is determined as the position determination model.

6. The method according to claim 3, characterized in that When the first model to be trained meets the convergence condition, determining the first model to be trained as the position determination model includes: Determining a predicted 3D position information difference according to the predicted 3D position information of the detection object and the 3D position information label; When the predicted 3D position information difference is less than a third threshold, the first model to be trained meets a convergence condition, and the first model to be trained is determined as a position determination model.

7. The method according to claim 1, characterized in that Before acquiring the sample data of the detection object, the method further includes: Acquire an image corresponding to the detection object, and input the image corresponding to the detection object into a 2D key position detection model to obtain 2D key position information corresponding to the detection object; Acquire the 3D position information label corresponding to the detection object and acquire the 3D key position information corresponding to the detection object by using a multi-view reconstruction method; The detection object sample data is determined according to the 2D key position information corresponding to the detection object, the 3D key position information corresponding to the detection object, and the 3D position information label corresponding to the detection object.

8. The method according to claim 7, characterized in that The obtaining of 3D key position information corresponding to the detected object includes: The 2D key position information corresponding to the detection object is input into a 3D key position estimation model to obtain the 3D key position information corresponding to the detection object.

9. The method according to claim 8, characterized in that The method further comprises: Determine the 3D key position information corresponding to the detection object according to the 3D position information tag corresponding to the detection object; Inputting the 2D key position information corresponding to the detected object into a second to-be-trained model to obtain predicted 3D key position information; The neural network parameters of the second model to be trained are updated according to the predicted 3D key position information and the 3D key position information to obtain the 3D key position estimation model.

10. A method for determining a position, characterized in that: The method comprises: Obtain the image corresponding to the detected object; Inputting the image corresponding to the detection object into a 2D key position detection model to obtain 2D key position information corresponding to the detection object; Inputting the 2D key position information corresponding to the detection object into a 3D key position estimation model to obtain the 3D key position information corresponding to the detection object; The 2D key position information corresponding to the detection object and the 3D key position information corresponding to the detection object are input into a position determination model to obtain the 3D position information of the detection object; wherein the 3D position information is used to indicate the position of the detection object in a 3D virtual environment, and the position determination model is obtained by the position determination model training method as described in any one of claims 1 to 9.

11. A location determination model training device, characterized in that: include: A sample data acquisition unit is configured to acquire sample data of the detection object; wherein the sample data of the detection object includes 2D key position information, 3D key position information and a 3D position information tag; wherein the 3D position information tag is used to indicate the position of the detection object in the 3D virtual environment; A prediction information acquisition unit is configured to input the 2D key position information and the 3D key position information into a first model to be trained to obtain predicted 3D position information of the detection object, wherein the first model to be trained includes a first sub-model to be trained and a second sub-model to be trained; a position determination model training unit, configured to update the neural network parameters of the first to-be-trained model according to the predicted 3D position information of the detected object and the 3D position information label, so as to obtain a position determination model; The prediction information acquisition unit is also configured to input the 2D key position information into the first sub-model to be trained to obtain a first prediction result, and input the 3D key position information into the second sub-model to be trained to obtain a second prediction result; and determine the predicted 3D position information of the detection object based on the first prediction result and the second prediction result.

12. A position determination device, characterized in that: include: A detection image acquisition unit is configured to acquire an image corresponding to the detection object; A 2D key position acquisition unit is configured to input the image corresponding to the detection object into a 2D key position detection model to obtain 2D key position information corresponding to the detection object; A 3D key position acquisition unit is configured to input the 2D key position information corresponding to the detection object into a 3D key position estimation model to obtain the 3D key position information corresponding to the detection object; A position determination unit is configured to input the 2D key position information corresponding to the detection object and the 3D key position information corresponding to the detection object into a position determination model to obtain the 3D position information of the detection object; wherein the 3D position information is used to indicate the position of the detection object in the 3D virtual environment, and the position determination model is obtained by the position determination model training method as described in any one of claims 1 to 9.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the executable instructions to implement the position determination model training method as described in any one of claims 1 to 9 or the position determination method as described in claim 10.

14. A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the position determination model training method described in any one of claims 1 to 9 or the position determination method described in claim 10.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the position determination model training method according to any one of claims 1 to 9 or the position determination method according to claim 10 is implemented.

Images