Dynamic updating method for model parameters in VR equipment and VR equipment
By taking user hand images in VR devices and updating the quantitative parameters of gesture detection model, the problem of low efficiency in model parameter update in the prior art is solved, and the model is quickly adapted and efficiently updated in different environments is achieved.
Patent Information
- Application Number
- CN202311022339.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-14
- Publication Date
- 2025-02-28
AI Technical Summary
In the prior art, the gesture detection model parameter update efficiency in VR devices is inefficient and cannot update the already started model in real time, resulting in poor performance in a specific environment.
By taking a user's hand image, traversing and inputting it into the gesture detection model for detection, counting the target input value of the model detection layer, and calculating the deviation value of the quantization parameter. If the deviation value is greater than the specified threshold, update the model quantization parameters and dynamically update the gesture detection model.
The gesture detection model is realized to quickly adapt to new environments and scenarios without manual retraining of the model, improving the efficiency of model parameters update.
Smart Images
Figure CN119536500A_ABST
Abstract
Description
Background Art
[0002] VR (Virtual Reality) provides users with an immersive gaming experience. The translation and rotation of the user's head in three-dimensional space is an important part of interaction in the VR field. With the introduction of the concept of the metaverse and the improvement of mobile computing capabilities in recent years, the market's technical requirements for users to fully interact in the field of AR (Augmented Reality) / VR technology are gradually increasing.
[0003] Traditional VR interaction methods mainly rely on external devices such as controllers, handles, and keyboards for input. However, this method may destroy the sense of immersion and limit the user's freedom and body expression. Through 3D (Three Dimensional) gesture technology, users can use natural gestures, such as hand movements, finger movements, etc. to interact with the virtual environment, making the interaction more intuitive and natural.
[0004] At present, many all-in-one VR devices have deployed models containing gesture algorithms to mobile devices to complete edge computing. Although gesture detection models generally try to increase the generalization ability of the model as much as possible during the training phase so as to cover as many application scenarios as possible, due to factors such as the parameter size limit of the model itself and the difficulty of producing training data, the performance in some specific environments is often not up to expectations. Therefore, in the face of such problems, the existing technology mainly uses manual intervention to collect some on-site pictures, conduct multiple iterative training of the model, and then upgrade the model through remote OTA (Over-the-Air Technology) and other methods to achieve the purpose of updating the model parameters. However, this method cannot update the model that has already been started, resulting in low efficiency in updating the model parameters. Summary of the invention
[0005] The present application provides a method for dynamically updating model parameters in a VR device and a VR device, which are used to improve the updating efficiency of the model parameters.
[0006] In a first aspect, an embodiment of the present application provides a method for dynamically updating model parameters in a VR device, the method comprising:
[0007] When it is determined that the VR device is operated without a handle and both hands of the user are detected, multiple first target hand images in the multiple groups of hand images of the user captured are traversed, and for any traversed first target hand image, the first target hand image is input into the gesture detection model for gesture detection to determine whether the user's hand is detected in the first target hand image;
[0008] If so, after counting the target input values of each model detection layer of the gesture detection model inputted by the first target hand image, return to the step of traversing multiple first target hand images in the multiple groups of hand images of the user captured, until the number of first target hand images of the user's hand detected is greater than the specified number, then the traversal ends;
[0009] For any model detection layer, based on each target input value of the model detection layer and a preset model quantization parameter value corresponding to the model detection layer, a quantization parameter deviation value of the first target hand image in the model detection layer is obtained;
[0010] If the number of model detection layers whose quantization parameter deviation values are greater than a first specified threshold value satisfies a preset condition, then for any model detection layer whose number of deviations is greater than a specified number, the model quantization parameter value of the model detection layer is updated using each target input value whose model quantization parameter in the model detection layer is greater than the first specified threshold value to obtain an updated model quantization parameter;
[0011] The gesture detection model is updated using the updated model quantization parameters to obtain an updated gesture detection model.
[0012] A second aspect of the present application provides a VR device, including a processor and a memory, wherein the processor and the memory are connected via a bus;
[0013] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:
[0014] When it is determined that the VR device is operated without a handle and both hands of the user are detected, multiple first target hand images in the multiple groups of hand images of the user captured are traversed, and for any traversed first target hand image, the first target hand image is input into the gesture detection model for gesture detection to determine whether the user's hand is detected in the first target hand image;
[0015] If so, after counting the target input values of each model detection layer of the gesture detection model inputted by the first target hand image, return to the step of traversing multiple first target hand images in the multiple groups of hand images of the user captured, until the number of first target hand images of the user's hand detected is greater than the specified number, then the traversal ends;
[0016] For any model detection layer, based on each target input value of the model detection layer and a preset model quantization parameter value corresponding to the model detection layer, a quantization parameter deviation value of the first target hand image in the model detection layer is obtained;
[0017] If the number of model detection layers whose quantization parameter deviation values are greater than a first specified threshold value satisfies a preset condition, then for any model detection layer whose number of deviations is greater than a specified number, the model quantization parameter value of the model detection layer is updated using each target input value whose model quantization parameter in the model detection layer is greater than the first specified threshold value to obtain an updated model quantization parameter;
[0018] The gesture detection model is updated using the updated model quantization parameters to obtain an updated gesture detection model.
[0019] According to a third aspect provided by an embodiment of the present invention, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect.
[0020] In the above-mentioned embodiment of the present application, the quantization parameter deviation value of each first target hand image in the model detection layer is determined by the target input value of each model detection layer in the gesture detection model and the preset model quantization parameter value corresponding to the model detection layer. If the number of model detection layers whose quantization parameter deviation value is greater than the first specified threshold value meets the preset condition, then for any model detection layer whose quantization parameter deviation value is greater than the first specified threshold value, the quantization parameter value of the model detection layer is updated using each target input value whose model quantization parameter in the model detection layer is greater than the first specified threshold value to obtain an updated model quantization parameter. The gesture detection model is updated by the updated model quantization parameter to obtain an updated gesture detection model. Therefore, in the embodiment of the present application, the gesture detection model is quickly adapted to new environments and scenarios by dynamically updating the model parameters, and the model parameters do not need to be updated manually by retraining the model, so the updating efficiency of the model parameters is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0022] Figure 1 A schematic diagram of a process for dynamically updating model parameters in a VR device provided in an embodiment of the present application is exemplified;
[0023] Figure 2 The terminal interface diagram provided by the embodiment of the present application is exemplarily shown;
[0024] Figure 3The schematic diagram of the structure of the convolutional neural network provided in the embodiment of the present application is exemplified;
[0025] Figure 4 The schematic diagram of the process of determining the quantization parameter deviation value of the model detection layer provided by the embodiment of the present application is exemplarily shown;
[0026] Figure 5 A schematic diagram exemplarily shows a flow chart of updating weight parameters of a gesture detection model provided in an embodiment of the present application;
[0027] Figure 6 The following is a schematic diagram of a process for determining the image depth of a hand image provided by an embodiment of the present application;
[0028] Figure 7 The schematic diagram of the predicted area and the actual area provided in the embodiment of the present application is exemplarily shown;
[0029] Figure 8 The schematic diagram for updating weight parameters provided by the embodiment of the present application is exemplarily shown;
[0030] Fig. 9 The following is a schematic diagram of a process for updating the model quantization parameters provided by an embodiment of the present application;
[0031] Fig.10 The flowchart of updating the weight parameters of the gesture detection model provided by the embodiment of the present application is exemplarily shown;
[0032] Fig.11 A schematic diagram of a device for dynamically updating model parameters in a VR device provided in an embodiment of the present application is exemplarily shown;
[0033] Fig.12 The hardware structure diagram of the VR device provided in the embodiment of the present application is exemplified. DETAILED DESCRIPTION
[0034] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.
[0035] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.
[0036] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.
[0037] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a series of components is not necessarily limited to those components clearly listed, but may include other components not clearly listed or inherent to these products or devices.
[0038] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.
[0039] The idea of the embodiments of the present application is summarized below.
[0040] In the prior art, manual intervention is mainly used to collect some on-site pictures, perform multiple iterations of model training, and then upgrade the model through remote OTA (Over-the-Air Technology) and other methods to achieve the purpose of updating model parameters. However, this method cannot update the model that has already been started, resulting in low efficiency in updating model parameters.
[0041] Based on the problem of low updating efficiency of model parameters in the prior art, the embodiment of the present application provides a method for dynamically updating model parameters in a VR device, and determines the quantization parameter deviation value of each first target hand image in the model detection layer through the target input value of each model detection layer in the gesture detection model and the preset model quantization parameter value corresponding to the model detection layer. If the number of model detection layers whose quantization parameter deviation value is greater than the first specified threshold satisfies the preset condition, then for any model detection layer whose quantization parameter deviation value is greater than the first specified threshold, the quantization parameter value of the model detection layer is updated using each target input value whose model quantization parameter in the model detection layer is greater than the first specified threshold, and an updated model quantization parameter is obtained. The gesture detection model is updated by the updated model quantization parameter to obtain an updated gesture detection model. Therefore, in the embodiment of the present application, the gesture detection model is quickly adapted to the new environment and scene by dynamically updating the model parameters, and the model parameters do not need to be updated by manually retraining the model, so the updating efficiency of the model parameters is improved. Below, the dynamic updating method of model parameters in the VR device of the present application is described in detail in conjunction with the accompanying drawings.
[0042] like Figure 1 FIG. 1 is a flow chart of a method for dynamically updating model parameters in a VR device, which may include the following steps:
[0043] Step 101: when it is determined that the VR device is operated without a handle and both hands of the user are detected, multiple first target hand images in the multiple groups of hand images of the user captured are traversed, and for any traversed first target hand image, the first target hand image is input into a gesture detection model for gesture detection, and it is determined whether the user's hand is detected in the first target hand image;
[0044] In the embodiment of the present application, when the user starts the gesture calibration program, the user is prompted to place both hands in a fixed area so as to take a picture of the user's hands for subsequent updating of the model parameters. Figure 2 As shown, the terminal interface will prompt the user to place both hands at the corresponding positions.
[0045] In the embodiment of the present application, multiple groups of hand images will be captured, and any group of hand images includes a left image and a right image, and the left image is a hand image captured by the left camera in the VR device, and the right image is a hand image captured by the right camera in the VR device; any group of hand images corresponds to a first target hand image, and the first target hand image is the left image or the right image in the any group of hand images.
[0046] In one embodiment, whether the first target hand image detects the user's hand is determined in the following manner: if the gesture detection model detects the user's hand in the first target hand image, then it is determined that the first target hand image detects the user's hand; otherwise, it is determined that the first target hand image does not detect the user's hand.
[0047] Step 102: If yes, then counting the target input values of each model detection layer inputted into the gesture detection model by the first target hand image;
[0048] The input value may be an image brightness value, a grayscale value, a feature value, etc., and may be specifically set according to actual conditions. The type of input value of each model detection layer may be the same or different, and the embodiment of the present application does not limit this.
[0049] The gesture detection model in the embodiment of the present application is a neural network model, but the specific structure of the gesture detection model is not limited in the embodiment of the present application. The specific structure of the gesture detection model in the embodiment of the present application can be set according to actual conditions. Taking the gesture detection model in the embodiment of the present application as a convolutional neural network model as an example, Figure 3 As shown, it is a schematic diagram of the structure of a convolutional neural network, and the model detection layer of the convolutional neural network includes convolutional layer 1, pooling layer 1, convolutional layer 2, pooling layer 2, fully connected layer 1 and fully connected layer 2.
[0050] Step 103: determining whether the number of the first target hand images detected of the user's hand is greater than a specified number, if so, executing step 104, if not, returning to executing step 101;
[0051] The specified number in the embodiment of the present application is 200, but the embodiment of the present application does not limit the specified number. The specified number in the embodiment of the present application can be set according to actual conditions.
[0052] Step 104: for any model detection layer, based on each target input value of the model detection layer and a preset model quantization parameter value corresponding to the model detection layer, obtain a quantization parameter deviation value of the first target hand image in the model detection layer;
[0053] Next, the specific method of determining the quantization parameter deviation value of any model detection layer is described, such as Figure 4 As shown, a flow chart of determining the quantization parameter deviation value of any model detection layer may include the following steps:
[0054] Step 401: Obtain a first intermediate value according to the target input value with the smallest value among the target input values of the model detection layer and the minimum value of the model quantization parameter of the model detection layer;
[0055] In one embodiment, step 401 may be specifically implemented as follows: determining the absolute value of the difference between the minimum target input value and the minimum value of the model quantization parameter as the first intermediate value.
[0056] Step 402: Obtain a second intermediate value according to a maximum value of a model quantization parameter of the model detection layer and a minimum value of a model quantization parameter of the model detection layer;
[0057] In one embodiment, step 402 may be specifically implemented as: determining the difference between the maximum value of the model quantization parameter and the minimum value of the model quantization parameter as the second intermediate value.
[0058] Step 403: Utilize the first intermediate value and the second intermediate value to obtain a quantization parameter deviation value of the first target hand image in the model detection layer.
[0059] In one embodiment, step 403 may be specifically implemented as follows: determining the ratio of the first intermediate value to the second intermediate value as the quantization parameter difference of the first target hand image at the model detection layer. The quantization parameter deviation value may be obtained by formula (1):
[0060]
[0061] Among them, α i is the quantization parameter deviation value of the first target hand image in the i-th model detection layer in the gesture detection model, p min is the target input value with the smallest value among the target input values of the i-th model detection layer, q min is the minimum value of the model quantization parameter, q max The maximum value of the quantization parameter for the model.
[0062] Step 105: If the number of model detection layers whose quantization parameter deviation values are greater than the first specified threshold value meets a preset condition, then for any model detection layer whose quantization parameter deviation value is greater than the first specified threshold value, the model quantization parameter value of the model detection layer is updated by using each target input value whose model quantization parameter in the model detection layer is greater than the first specified threshold value, to obtain an updated model quantization parameter;
[0063] Among them, the model quantization parameter value includes a model quantization parameter maximum value and a model quantization parameter minimum value, and the target input value includes an input maximum value and an input minimum value. The preset condition in the embodiment of the present application is that the number of model detection layers whose quantization parameter deviation value is greater than the first specified threshold is greater than the target number. The target number in the embodiment of the present application is based on 20% of the specified number, but the specific ratio of the target number to the specified number can be set according to actual conditions, and the embodiment of the present application does not limit the target number here.
[0064] In one embodiment, the updated model quantization parameter is obtained by:
[0065] For any model detection layer, the average value of the input minimum value among the target input values of each first target hand image whose quantization parameter deviation value in the model detection layer is greater than the first specified threshold value is determined as the updated model quantization parameter minimum value of the model detection layer. And the average value of the input maximum value among the target input values of each first target hand image whose quantization parameter deviation value in the model detection layer is greater than the first specified threshold value is determined as the updated model quantization parameter maximum value of the model detection layer; the updated model quantization parameter is obtained according to the updated model quantization parameter minimum value and the updated model quantization parameter maximum value.
[0066] In the embodiment of the present application, the updated model quantization parameter includes an updated model quantization parameter maximum value and a model quantization parameter minimum value.
[0067] Step 106: updating the gesture detection model using the updated model quantization parameters to obtain an updated gesture detection model.
[0068] In one embodiment, step 106 may be specifically implemented as follows: setting the values of each model quantization parameter in the gesture detection model to be the same as the values of the updated model quantization parameters to obtain an updated gesture detection model.
[0069] The following is a detailed description of the method for updating the weight parameters of the gesture detection model in the embodiment of the present application. Figure 5 FIG. 1 is a flow chart of a method for updating weight parameters of a gesture detection model in an embodiment of the present application, comprising the following steps:
[0070] Step 501: If the current usage environment obtained based on the multiple groups of hand images of the user does not meet the specified condition, the multiple groups of hand images are filtered using the image depths of the multiple groups of hand images to obtain the filtered multiple groups of hand images, wherein any second target hand image includes the user's hand and the actual area where the hand is located;
[0071] In one embodiment, the current usage environment is determined by:
[0072] An average value of the image brightness of the multiple groups of hand images is determined as the current usage environment.
[0073] In one embodiment, whether the current usage environment meets the specified condition is determined by:
[0074] The difference between the average value of the image brightness of the multiple groups of hand images and the preset standard brightness is compared with a preset threshold; if the difference is greater than the preset threshold, it is determined that the current usage environment does not meet the specified condition; if the difference is not greater than the preset threshold, it is determined that the current usage environment meets the specified condition.
[0075] It should be noted that the standard brightness and the preset threshold in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application does not limit the standard brightness and the preset threshold.
[0076] Below, a method for filtering multiple groups of hand images is introduced. In one embodiment, the multiple groups of hand images are filtered in the following manner: for any hand image in the multiple groups of hand images, if the image depth of the hand image is not within a preset depth range, the hand image is deleted.
[0077] It should be noted that the preset depth range in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application does not limit the preset depth range.
[0078] The following is a detailed description of the method for determining the image depth of the hand image in the embodiment of the present application. Figure 6 As shown, it is a schematic diagram of a process of determining the image depth of a hand image, comprising the following steps:
[0079] Step 601: for any group of hand images, perform hand detection on the left image in the hand images and the right image in the hand images respectively to obtain the hand position in the left image and the hand position in the right image;
[0080] The hand detection method in the embodiment of the present application uses the gesture detection model in the embodiment of the present application for detection, but the embodiment of the present application does not limit the hand detection method. And the hand position in the left image in the embodiment of the present application is the position coordinate of the center point of the hand. And the hand position in the right image is the position coordinate of the center point of the hand.
[0081] Step 602: According to the hand position in the left image and the hand position in the right image, obtain the image depth corresponding to the hand image. The image depth of the hand image can be obtained by formula (2):
[0082]
[0083] Among them, D is the image depth of the hand image, f is the focal length of the camera, b is the distance between the optical center of the left camera and the optical center of the right camera in the VR device, d1 is the horizontal coordinate of the hand position in the left image, and d2 is the horizontal position of the hand position in the right image.
[0084] Step 502: for a second target hand image in any group of filtered hand images, input the second target hand image into the gesture detection model to obtain a predicted area of the hand in the second target hand image;
[0085] Step 503: Based on the predicted area of the hand in the second target hand image and the actual area of the hand, a loss value is obtained; and gradient backpropagation is performed on the gesture detection model through the loss value to obtain gradients corresponding to weight parameters in the gesture detection model respectively; wherein the loss value can be obtained by formula (3):
[0086]
[0087] Among them, L is the loss value, IoU is the intersection over union ratio of the predicted area and the actual area, r is the distance between the center point of the predicted area and the center point of the actual area, and r is the length of the diagonal of the minimum enclosing rectangular box between the predicted area and the actual area.
[0088] The method for determining the intersection over union (IoU) of the predicted area and the actual area in the embodiment of the present application is: the area of the union area of the predicted area and the actual area is divided by the area of the intersection of the predicted area and the actual area. And the distance between the center point of the predicted area and the center point of the actual area in the embodiment of the present application can be obtained by formula (4):
[0089]
[0090] Among them, r is the distance between the center point of the predicted area and the center point of the actual area, x1 is the horizontal coordinate of the center point of the predicted area, x2 is the horizontal coordinate of the center point of the actual area, y1 is the vertical coordinate of the center point of the predicted area, and y2 is the vertical coordinate of the center point of the actual area.
[0091] In one embodiment, the length of the diagonal of the minimum enclosing rectangular box of the predicted area and the actual area is determined in the following manner: the first target vertex and the second target vertex are obtained by comparing the position coordinates of the vertices of each area in the predicted area and the actual area, and the distance between the first target vertex and the second target vertex is determined as the length of the diagonal of the minimum enclosing rectangular box of the predicted area and the actual area. The first target vertex is the vertex with the smallest horizontal coordinate value and the smallest vertical coordinate value among the vertices of the area, and the second target vertex is the vertex with the largest horizontal coordinate value and the largest vertical coordinate value among the vertices of the area. Figure 7 As shown in the figure, it is a schematic diagram of the predicted area and the actual area. Figure 7 It can be seen from the figure that the first target vertex is vertex A and the second target vertex is vertex F. Then the length of the diagonal line of the minimum enclosing rectangular box of the predicted area and the actual area is the distance between vertex A and vertex F.
[0092] Step 504: classifying the gradients according to the weight categories of the weight parameters to obtain gradient vectors corresponding to the weight categories, and updating the weight parameters using a specified number of gradient values in the gradient vectors corresponding to the weight categories to obtain updated weight parameters corresponding to the second target hand image, wherein the weight categories of the weight parameters are obtained by clustering the weight parameters;
[0093] In one embodiment, each extracted category is determined in the following manner: for any gradient, a weight category of a weight parameter corresponding to the gradient is determined as the category of the gradient.
[0094] In the embodiment of the present application, the weight categories of the weight parameters of the gesture detection model are obtained by K-means (k-means clustering algorithm), which belongs to the clustering method in the prior art and will not be described in detail in the embodiment of the present application.
[0095] In one embodiment, the updated weight parameters corresponding to the second target hand image are obtained in the following manner: for any weight category, the gradient average value corresponding to the weight category is obtained based on the first specified number of gradient values in the gradient vector corresponding to the weight category; and the gradient average value corresponding to the weight category is added to the weight parameters corresponding to the weight category respectively to obtain the updated weight parameters.
[0096] like Figure 8 As shown, this is a schematic diagram of updating weight parameters. Figure 8The colors of each gradient and each weight parameter in represent a corresponding weight category. First, each voxel is classified to obtain each gradient corresponding to each weight category. In the embodiment of the present application, the first 50% of the specified number is used as an example to illustrate, and the average gradient value corresponding to each weight category is obtained. Then, for any weight category, each weight parameter corresponding to the weight category is added to the gradient average value to obtain the updated weight parameter.
[0097] Step 505: updating the weight parameters of the gesture detection model using the updated weight parameters corresponding to the multiple second target hand images to obtain an updated gesture detection model.
[0098] In one embodiment, step 505 may be specifically implemented as follows: setting the values of each weight parameter of the gesture detection model to be equal to the values of each updated weight parameter to obtain the updated gesture detection model.
[0099] In order to further connect the method in this application, the dynamic update method of the model parameters in the VR device in this application is introduced in combination with the use scenario. The dynamic update method of the model parameters in the embodiment of this application includes two parts, one is to update the model quantization parameters, and the other is to update the model weight parameters. Fig. 9 FIG. 1 is a flow chart of updating the model quantization parameters in an embodiment of the present application, which includes the following steps:
[0100] Step 901: when it is determined that the VR device is operated without a handle and both hands of the user are detected, a plurality of first target hand images in a plurality of groups of hand images of the user captured are traversed;
[0101] Step 902: for any traversed first target hand image, input the first target hand image into the gesture detection model for gesture detection, and determine whether the user's hand is detected in the first target hand image. If so, execute step 903; if not, return to execute step 901;
[0102] Step 903: Counting target input values of each model detection layer in the gesture detection model inputted from the first target hand image;
[0103] Step 904: determine whether the number of the first target hand images detected of the user's hand is greater than the specified number, if so, execute step 905, if not, return to execute step 901;
[0104] Step 905: for any model detection layer, a first intermediate value is obtained according to the target input value with the smallest value among the target input values of the model detection layer and the minimum value of the model quantization parameter of the model detection layer;
[0105] Step 906: Obtain a second intermediate value according to the maximum value of the model quantization parameter of the model detection layer and the minimum value of the model quantization parameter of the model detection layer;
[0106] Step 907: using the first intermediate value and the second intermediate value, obtaining a quantization parameter deviation value of the first target hand image in the model detection layer;
[0107] Step 908: determine whether the number of model detection layers whose quantization parameter deviation value is greater than the first specified threshold meets the preset condition, if so, execute step 909, if not, return to execute step 901;
[0108] Step 909: for any model detection layer whose quantization parameter deviation value is greater than the first specified threshold, determine the average of the minimum input values among the target input values of each first target hand image whose quantization parameter deviation value in the model detection layer is greater than the first specified threshold as the updated model quantization parameter minimum value of the model detection layer;
[0109] Step 910: Determine the average of the maximum input values of the target input values of each first target hand image whose quantization parameter deviation value in the model detection layer is greater than the first specified threshold as the updated maximum model quantization parameter value of the model detection layer;
[0110] Step 911: Obtain the updated model quantization parameter according to the updated model quantization parameter minimum value and the updated model quantization parameter maximum value;
[0111] Step 912: updating the gesture detection model using the updated model quantization parameters to obtain an updated gesture detection model.
[0112] like Fig.10 FIG. 1 is a flow chart of updating the weight parameters of the gesture detection model in an embodiment of the present application, which includes the following steps:
[0113] Step 1001: after determining that the VR device is operated without a handle and detecting the user's hands, determining whether the current usage environment obtained based on the multiple sets of hand images of the user meets the specified conditions, if so, ending, if not, executing step 1002;
[0114] Step 1002: for any group of hand images, perform hand detection on the left image in the hand images and the right image in the hand images respectively to obtain the hand position in the left image and the hand position in the right image;
[0115] Step 1003: Obtaining an image depth corresponding to the hand image according to the hand position in the left image and the hand position in the right image;
[0116] Step 1004: Filtering the multiple groups of hand images by using the image depths of the multiple groups of hand images to obtain multiple groups of filtered hand images, wherein any second target hand image includes the user's hand and the actual area where the hand is located;
[0117] Step 1005: for a second target hand image in any group of filtered hand images, input the second target hand image into the gesture detection model to obtain a predicted area of the hand in the second target hand image;
[0118] Step 1006: Obtain a loss value based on the predicted area of the hand in the second target hand image and the actual area of the hand;
[0119] Step 1007: performing gradient back propagation on the gesture detection model through the loss value to obtain gradients corresponding to weight parameters in the gesture detection model respectively;
[0120] Step 1008: Classify the gradients according to the weight categories of the weight parameters to obtain gradient vectors corresponding to the weight categories.
[0121] Step 1009: for any weight category, according to the first specified number of gradient values in the gradient vector corresponding to the weight category, obtain the gradient average value corresponding to the weight category;
[0122] Step 1010: Adding the gradient average value corresponding to the weight category to each weight parameter corresponding to the weight category to obtain the updated weight parameter;
[0123] Step 1011: updating the weight parameters of the gesture detection model using the updated weight parameters corresponding to the multiple second target hand images to obtain an updated gesture detection model.
[0124] Based on the same inventive concept, the method for dynamically updating model parameters in a VR device disclosed above can also be implemented by a device for dynamically updating model parameters in a VR device. The effect of the device for dynamically updating model parameters in the VR device is similar to that of the aforementioned method, and will not be described in detail here.
[0125] Fig.11 The figure is a schematic diagram of the structure of a device for dynamically updating model parameters in a VR device according to an embodiment of the present disclosure.
[0126] like Fig.11As shown, the dynamic updating device 1100 of model parameters in the VR device of the present disclosure may include a gesture detection module 1110, a statistics module 1120, a quantization parameter deviation value determination module 1130, a model quantization parameter update module 1140 and a model update module 1150.
[0127] The gesture detection module 1110 is used to traverse a plurality of first target hand images in a plurality of groups of hand images of the user captured when it is determined that the VR device is operated without a handle and both hands of the user are detected, and for any traversed first target hand image, input the first target hand image into a gesture detection model for gesture detection, and determine whether the user's hand is detected in the first target hand image;
[0128] The statistical module 1120 is used for, if yes, counting the target input values of each model detection layer of the gesture detection model inputted by the first target hand image, and then returning to the step of traversing the multiple first target hand images in the multiple groups of hand images of the user captured, until the number of the first target hand images of the user's hand detected is greater than the specified number, and then terminating the traversal;
[0129] A quantization parameter deviation value determination module 1130 is used to obtain, for any model detection layer, a quantization parameter deviation value of the first target hand image in the model detection layer based on each target input value of the model detection layer and a preset model quantization parameter value corresponding to the model detection layer;
[0130] A model quantization parameter updating module 1140 is configured to update the model quantization parameter value of any model detection layer whose quantization parameter deviation value is greater than the first specified threshold value by using each target input value whose model quantization parameter in the model detection layer is greater than the first specified threshold value, if the number of model detection layers whose quantization parameter deviation value is greater than the first specified threshold value satisfies a preset condition, to obtain an updated model quantization parameter;
[0131] The model updating module 1150 is used to update the gesture detection model by using the updated model quantization parameters to obtain an updated gesture detection model.
[0132] In one embodiment, the model quantization parameter value includes a model quantization parameter maximum value and a model quantization parameter minimum value;
[0133] The quantization parameter deviation value determination module 1130 is specifically used to:
[0134] Obtaining a first intermediate value according to the target input value with the smallest value among the target input values of the model detection layer and the minimum value of the model quantization parameter of the model detection layer; and obtaining a second intermediate value according to the maximum value of the model quantization parameter of the model detection layer and the minimum value of the model quantization parameter of the model detection layer;
[0135] The first intermediate value and the second intermediate value are used to obtain a quantization parameter deviation value of the first target hand image in the model detection layer.
[0136] In one embodiment, the quantization parameter deviation value determination module 1130 is specifically configured to:
[0137] The quantization parameter deviation value of the first target hand image in the model detection layer is obtained by the following formula:
[0138]
[0139] Among them, α i is the quantization parameter deviation value of the first target hand image in the i-th model detection layer in the gesture detection model, p min is the target input value with the smallest value among the target input values of the i-th model detection layer, q min is the minimum value of the model quantization parameter, q max The maximum value of the quantization parameter for the model.
[0140] In one embodiment, the model quantization parameter value includes a model quantization parameter maximum value and a model quantization parameter minimum value, and the target input value includes an input maximum value and an input minimum value; the model quantization parameter updating module 1140 is specifically used to:
[0141] For any model detection layer, determining the average of the minimum input values among the target input values of each first target hand image whose quantization parameter deviation value in the model detection layer is greater than the first specified threshold as the updated model quantization parameter minimum value of the model detection layer; and
[0142] Determine the average of the maximum input values among the target input values of the first target hand images whose quantization parameter deviation values in the model detection layer are greater than the first specified threshold as the updated maximum model quantization parameter value of the model detection layer;
[0143] The updated model quantization parameter is obtained according to the updated model quantization parameter minimum value and the updated model quantization parameter maximum value.
[0144] In one embodiment, the apparatus further comprises:
[0145] The weight parameter updating module 1160 is used for, when it is determined that the VR device is operated without a handle and both hands of the user are detected, if the current usage environment obtained based on the multiple groups of hand images of the user does not meet the specified conditions, then the multiple groups of hand images are filtered using the image depth of the multiple groups of hand images to obtain the filtered multiple groups of hand images, wherein any second target hand image includes the user's hand and the actual area where the hand is located;
[0146] For a second target hand image in any group of filtered hand images, input the second target hand image into the gesture detection model to obtain a predicted area of the hand in the second target hand image; and
[0147] Based on the predicted area of the hand in the second target hand image and the actual area of the hand, a loss value is obtained; and gradient backpropagation is performed on the gesture detection model through the loss value to obtain gradients corresponding to weight parameters in the gesture detection model respectively; and
[0148] Classifying the gradients according to the weight categories of the weight parameters to obtain gradient vectors corresponding to the weight categories, and updating the weight parameters using a specified number of gradient values in the gradient vectors corresponding to the weight categories to obtain updated weight parameters corresponding to the second target hand image, wherein the weight categories of the weight parameters are obtained by clustering the weight parameters;
[0149] The weight parameters of the gesture detection model are updated using the updated weight parameters corresponding to the multiple second target hand images to obtain an updated gesture detection model.
[0150] In one embodiment, any set of hand images includes a left image and a right image, and the left image is a hand image captured by a left camera in the VR device, and the right image is a hand image captured by a right camera in the VR device; the apparatus further includes:
[0151] The image depth determination module 1170 is used to obtain the image depth of the hand image by:
[0152] For any group of hand images, perform hand detection on a left image in the hand images and a right image in the hand images respectively to obtain a hand position in the left image and a hand position in the right image;
[0153] According to the hand position in the left image and the hand position in the right image, an image depth corresponding to the hand image is obtained.
[0154] In one embodiment, the image depth determination module 1170 is specifically configured to:
[0155] The depth of the hand image is obtained by the following formula:
[0156]
[0157] Among them, D is the image depth of the hand image, f is the focal length of the camera, b is the distance between the optical center of the left camera and the optical center of the right camera in the VR device, d1 is the horizontal coordinate of the hand position in the left image, and d2 is the horizontal position of the hand position in the right image.
[0158] In one embodiment, the weight parameter updating module 1160 is specifically used to:
[0159] For any weight category, according to the first specified number of gradient values in the gradient vector corresponding to the weight category, the gradient average value corresponding to the weight category is obtained; and,
[0160] The gradient average corresponding to the weight category is added to each weight parameter corresponding to the weight category to obtain the updated weight parameter.
[0161] In one embodiment, the arbitrary group of hand images includes a left image and a right image, and the left image is a hand image captured by a left camera in the VR device, and the right image is a hand image captured by a right camera in the VR device;
[0162] Any group of hand images corresponds to a first target hand image, and the first target hand image is a left image or a right image in the any group of hand images.
[0163] After introducing a method and apparatus for dynamically updating model parameters in a VR device according to an exemplary embodiment of the present invention, a VR device according to another exemplary embodiment of the present invention is introduced next.
[0164] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".
[0165] In some possible implementations, the VR device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the method for dynamically updating model parameters in the VR device according to various exemplary implementations of the present invention described above in this specification. For example, the processor may execute the following steps: Figure 1 Steps 101-105 shown in FIG.
[0166] Refer to the following Fig.12 hereinafter, a VR device 1200 according to this embodiment of the present invention is described. Fig.12 The VR device 1200 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.
[0167] like Fig.12 As shown, the VR device 1200 is presented in the form of a general VR device. The components of the VR device 1200 may include, but are not limited to: the at least one processor 1201, the at least one computer storage medium 1202, and a bus 1203 connecting different system components (including the computer storage medium 1202 and the processor 1201).
[0168] Bus 1203 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.
[0169] Computer storage media 1202 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1221 and / or cache storage media 1222 , and may further include read-only computer storage media (ROM) 1223 .
[0170] The computer storage medium 1202 may also include a program / utility 1225 having a set (at least one) of program modules 1224, such program modules 1224 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0171] The VR device 1200 may also communicate with one or more external devices 1204 (e.g., keyboards, pointing devices, etc.), and may also communicate with one or more devices that enable a user to interact with the VR device 1200 and / or any device that enables the VR device 1200 to communicate with one or more other VR devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1205. Furthermore, the VR device 1200 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1206. As shown, the network adapter 1206 communicates with other modules for the VR device 1200 via a bus 1203. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the VR device 1200, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0172] In some possible implementations, various aspects of the method for dynamically updating model parameters in a VR device provided by the present invention may also be implemented in the form of a program product, which includes program code. When the program product runs on a computer device, the program code is used to enable the computer device to execute the steps of the method for dynamically updating model parameters in a VR device according to various exemplary embodiments of the present invention described above in this specification.
[0173] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A method for dynamically updating model parameters in a VR device, characterized in that: The method comprises: When it is determined that the VR device is operated without a handle and both hands of the user are detected, multiple first target hand images in the multiple groups of hand images of the user captured are traversed, and for any traversed first target hand image, the first target hand image is input into the gesture detection model for gesture detection to determine whether the user's hand is detected in the first target hand image; If so, after counting the target input values of each model detection layer of the gesture detection model inputted by the first target hand image, return to the step of traversing multiple first target hand images in the multiple groups of hand images of the user captured, until the number of first target hand images of the user's hand detected is greater than the specified number, then the traversal ends; For any model detection layer, based on each target input value of the model detection layer and a preset model quantization parameter value corresponding to the model detection layer, a quantization parameter deviation value of the first target hand image in the model detection layer is obtained; If the number of model detection layers whose quantization parameter deviation values are greater than a first specified threshold value satisfies a preset condition, then for any model detection layer whose quantization parameter deviation value is greater than the first specified threshold value, the model quantization parameter value of the model detection layer is updated using each target input value whose model quantization parameter in the model detection layer is greater than the first specified threshold value to obtain an updated model quantization parameter; The gesture detection model is updated using the updated model quantization parameters to obtain an updated gesture detection model.
2. The method according to claim 1, characterized in that The model quantization parameter value includes a model quantization parameter maximum value and a model quantization parameter minimum value; The obtaining, based on each target input value of the model detection layer and a preset model quantization parameter value corresponding to the model detection layer, a quantization parameter deviation value of the first target hand image in the model detection layer includes: Obtaining a first intermediate value according to the target input value with the smallest value among the target input values of the model detection layer and the minimum value of the model quantization parameter of the model detection layer; and obtaining a second intermediate value according to the maximum value of the model quantization parameter of the model detection layer and the minimum value of the model quantization parameter of the model detection layer; The first intermediate value and the second intermediate value are used to obtain a quantization parameter deviation value of the first target hand image in the model detection layer.
3. The method according to claim 1 or 2, characterized in that: The quantization parameter deviation value of the first target hand image in the model detection layer is obtained by the following formula: Among them, α i is the quantization parameter deviation value of the first target hand image in the i-th model detection layer in the gesture detection model, p min is the target input value with the smallest value among the target input values of the i-th model detection layer, q min is the minimum value of the model quantization parameter, q max The maximum value of the quantization parameter for the model.
4. The method according to claim 1, characterized in that: The model quantization parameter value includes a model quantization parameter maximum value and a model quantization parameter minimum value, and the target input value includes an input maximum value and an input minimum value; The method of updating the model quantization parameter value of the model detection layer by using each target input value whose model quantization parameter in the model detection layer is greater than the first specified threshold value to obtain the updated model quantization parameter includes: For any one model detection layer, determining the average of the minimum input values among the target input values of each first target hand image whose quantization parameter deviation value in the model detection layer is greater than the first specified threshold as the updated model quantization parameter minimum value of the model detection layer; and Determine the average of the maximum input values among the target input values of the first target hand images whose quantization parameter deviation values in the model detection layer are greater than the first specified threshold as the updated maximum model quantization parameter value of the model detection layer; The updated model quantization parameter is obtained according to the updated model quantization parameter minimum value and the updated model quantization parameter maximum value.
5. The method according to claim 1, characterized in that After determining that the VR device is operated without a handle and detecting both hands of the user, the method further includes: If the current usage environment obtained based on the multiple groups of hand images of the user does not meet the specified condition, the multiple groups of hand images are filtered using the image depths of the multiple groups of hand images to obtain the filtered multiple groups of hand images, wherein any one of the second target hand images includes the user's hand and the actual area where the hand is located; For a second target hand image in any group of filtered hand images, input the second target hand image into the gesture detection model to obtain a predicted area of the hand in the second target hand image; and Based on the predicted area of the hand in the second target hand image and the actual area of the hand, a loss value is obtained; and gradient backpropagation is performed on the gesture detection model through the loss value to obtain gradients corresponding to weight parameters in the gesture detection model respectively; and Classifying the gradients according to the weight categories of the weight parameters to obtain gradient vectors corresponding to the weight categories, and updating the weight parameters using a specified number of gradient values in the gradient vectors corresponding to the weight categories to obtain updated weight parameters corresponding to the second target hand image, wherein the weight categories of the weight parameters are obtained by clustering the weight parameters; The weight parameters of the gesture detection model are updated using the updated weight parameters corresponding to the multiple second target hand images to obtain an updated gesture detection model.
6. The method according to claim 5, characterized in that Any set of hand images includes a left image and a right image, and the left image is a hand image captured by a left camera in the VR device, and the right image is a hand image captured by a right camera in the VR device; the method further includes: The image depth of the hand image is obtained by: For any group of hand images, perform hand detection on a left image in the hand images and a right image in the hand images respectively to obtain a hand position in the left image and a hand position in the right image; According to the hand position in the left image and the hand position in the right image, an image depth corresponding to the hand image is obtained.
7. The method according to claim 5 or 6, characterized in that: The depth of the hand image is obtained by the following formula: Among them, D is the image depth of the hand image, f is the focal length of the camera, b is the distance between the optical center of the left camera and the optical center of the right camera in the VR device, d1 is the horizontal coordinate of the hand position in the left image, and d2 is the horizontal position of the hand position in the right image.
8. The method according to claim 5, characterized in that The updating of each weight parameter by using a specified number of gradient values in the gradient vector corresponding to each weight category to obtain updated weight parameters corresponding to the second target hand image includes: For any weight category, according to the first specified number of gradient values in the gradient vector corresponding to the weight category, the gradient average value corresponding to the weight category is obtained; and, The gradient average corresponding to the weight category is added to each weight parameter corresponding to the weight category to obtain the updated weight parameter.
9. The method according to claim 1, characterized in that: The arbitrary group of hand images includes a left image and a right image, and the left image is a hand image captured by a left camera in the VR device, and the right image is a hand image captured by a right camera in the VR device; Any group of hand images corresponds to a first target hand image, and the first target hand image is a left image or a right image in the any group of hand images.
10. A VR device, characterized in that: comprising a processor and a memory, wherein the processor and the memory are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: When it is determined that the VR device is operated without a handle and both hands of the user are detected, multiple first target hand images in the multiple groups of hand images of the user captured are traversed, and for any traversed first target hand image, the first target hand image is input into the gesture detection model for gesture detection to determine whether the user's hand is detected in the first target hand image; If so, after counting the target input values of each model detection layer of the gesture detection model inputted by the first target hand image, return to the step of traversing multiple first target hand images in the multiple groups of hand images of the user captured, until the number of first target hand images of the user's hand detected is greater than the specified number, then the traversal ends; For any model detection layer, based on each target input value of the model detection layer and a preset model quantization parameter value corresponding to the model detection layer, a quantization parameter deviation value of the first target hand image in the model detection layer is obtained; If the number of model detection layers whose quantization parameter deviation values are greater than a first specified threshold value meets a preset condition, then for any model detection layer whose deviation times are greater than a specified number, the model quantization parameter value of the model detection layer is updated using each target input value whose model quantization parameter in the model detection layer is greater than the first specified threshold value to obtain an updated model quantization parameter; The gesture detection model is updated using the updated model quantization parameters to obtain an updated gesture detection model.