Training method of neural network model and electronic device

Abstract

Embodiments of the present application are applicable to the field of artificial intelligence, and provide a neural network model training method and an electronic device. A first image set including a plurality of first sample images is obtained, wherein the first sample images include a target object and a first bounding box obtained by detecting the target object. The first bounding box of the first sample image in the first image set is moved to obtain a second image set. Then, an initial first key point recognition model is trained based on the first image set and the second image set to obtain a trained first key point recognition model. The first bounding box in the first sample image is moved to simulate a case where a hand shakes and the position of the hand in the image greatly differs, and a case where the hand partially moves out of the image. The types of sample images used to train the initial first key point recognition model are enriched, and the robustness of the trained first key point model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and more particularly, to a neural network model training method and an electronic device. BACKGROUND

[0002] At present, gesture recognition, as a very important human-computer interaction method, is widely used in smart phones, smart wear, car interaction, augmented reality (AR) and virtual reality (VR) products, etc. Among them, gesture recognition can refer to collecting a video stream about gesture information through a camera device, recognizing gestures in the video stream through a preset neural network model, and then performing corresponding operations based on the recognized gestures. In this way, the user does not need to contact the electronic device, and does not need to wear additional sensors, and the electronic device can perform corresponding operations according to the user's gestures.

[0003] In a possible case, due to the small field of view angle of the camera device, the gestures in the video stream collected by the camera device are prone to jitter, or the hand moves out of the field of view range of the camera device, thereby causing the accuracy of the gestures recognized by the preset neural network model to be low, that is, the robustness of the neural network model is low.

[0004] Therefore, how to improve the robustness of the neural network model for gesture recognition has become a problem to be solved. SUMMARY

[0005] The present application provides a neural network model training method, which can improve the robustness of the neural network model for gesture recognition.

[0006] In a first aspect, a neural network model training method is provided, comprising:

[0007] obtaining a first image set, the first image set comprising a plurality of first sample images, each first sample image comprising a target object and a first bounding box obtained by detecting the target object;

[0008] moving the first bounding box of each first sample image in the first image set to obtain a second image set;

[0009] training an initial first key point recognition model based on the first image set and the second image set to obtain a trained first key point recognition model, the first key point recognition model being used for recognizing key points in the target object.

[0010] The neural network model training method provided in the embodiments of the present application comprises the following steps: first, a first image set comprising a plurality of first sample images is acquired, wherein the first sample images comprise a target object and a first bounding box obtained by detecting the target object; then, the first bounding box of the first sample image in the first image set is moved to obtain a second image set; then, an initial first key point recognition model is trained based on the first image set and the second image set to obtain a trained first key point recognition model; and the first key point recognition model is used to recognize key points in the target object. By moving the first bounding box in the first sample image, the case that the position of the hand in the image greatly differs due to hand shaking during image acquisition and the case that the hand partially moves out of the image due to a small field of view angle of the camera are simulated, so that the sample images for training the initial first key point recognition model not only include the initial first sample images, but also include sample images adjusted for the small field of view angle of the camera and the hand shaking problem, thereby enriching the types of sample images for training the initial first key point recognition model, and improving the robustness of the trained first key point model.

[0011] In combination with the first aspect, in some embodiments of the first aspect, the second image set comprises a second sample image and a third sample image, and the moving of the bounding box of the first sample image in the first image set to obtain the second image set comprises: randomly moving the first bounding box in the first sample image to obtain the second sample image, the second sample image comprising the target object and a second bounding box obtained by randomly moving the first bounding box; and moving the bounding box in the first sample image in parallel to obtain the third sample image, the third sample image comprising the target object and a third bounding box obtained by moving the first bounding box in parallel, the third bounding box being obtained by moving the first bounding box in parallel along the X axis, or the third bounding box being obtained by moving the first bounding box in parallel along the Y axis.

[0012] The second sample image obtained by randomly moving the first bounding box can simulate an image obtained by hand shaking. The third sample image obtained by moving the first bounding box in parallel can simulate an image in which the hand moves out of the field of view angle of the camera.

[0013] The neural network model training method provided in the embodiments of the present application comprises the following steps: obtaining a first image set, the first image set comprising a plurality of first sample images, the first sample images comprising a target object and a first bounding box obtained by detecting the target object; randomly moving the bounding box in the first sample image to obtain a second sample image, the second sample image comprising the target object and a second bounding box obtained by randomly moving the first bounding box; moving the first bounding box in the first sample image in parallel to obtain a third sample image, the third sample image comprising the target object and a third bounding box obtained by moving the first bounding box in parallel, the third bounding box being obtained by moving the first bounding box in the X-axis direction or the third bounding box being obtained by moving the first bounding box in the Y-axis direction; and training an initial first key point recognition model based on the first image set, the second sample image and the third sample image to obtain a trained first key point recognition model, the first key point recognition model being used to recognize key points in the target object. The first bounding box in the first sample image is randomly moved to simulate the case that the position of the hand in the image greatly differs due to hand shaking during image acquisition, and the first bounding box is moved in parallel to simulate the case that part of the hand moves out of the image due to a small field of view angle of the camera. Therefore, the sample images for training the initial first key point recognition model not only comprise the initial first sample images, but also comprise the second sample images and the third sample images adjusted for hand shaking and the hand moving out of the field of view angle of the camera. The types of the sample images for training the initial first key point recognition model are enriched, thereby improving the robustness of the trained first key point model.

[0014] With reference to the first aspect, in some embodiments of the first aspect, the parallel moving the bounding box in the first sample image to obtain the third sample image comprises: parallel moving a first bounding box in the first sample image according to a first parameter to obtain the third sample image, the first parameter comprising a first sub-parameter and a second sub-parameter, the first sub-parameter being used to indicate a ratio of a number of third sample images obtained by parallel moving to a number of first sample images in the first image set, and the second sub-parameter being used to indicate a ratio of the first distance to a width or a height of the first bounding box, the first distance being a distance of the first bounding box moving to a third bounding box; and the training the initial first key point recognition model based on the first image set and the second image set to obtain the trained first key point recognition model comprises: pre-training the initial first key point recognition model based on the first image set, the second sample image and the third sample image to obtain a pre-trained first key point recognition model; parallel moving the bounding box in the first sample image according to a second parameter to obtain a fourth sample image, the fourth sample image comprising a target object and a fourth bounding box, the second parameter comprising a third sub-parameter and a fourth sub-parameter, the third sub-parameter being used to indicate a ratio of a number of fourth sample images obtained by parallel moving to a number of first sample images in the first image set, and the fourth sub-parameter being used to indicate a ratio of a second distance to a width or a height of the first bounding box, the second distance being a distance of the first bounding box moving to the fourth bounding box; and training the pre-trained first key point recognition model based on the first image set, the second sample image and the fourth sample image to obtain the trained first key point recognition model.

[0015] The method for training a neural network model provided in the embodiments of the present application comprises the following steps: obtaining a first image set, the first image set comprising a plurality of first sample images, each first sample image comprising a target object and a first bounding box obtained by detecting the target object; randomly moving the first bounding box in the first sample image to obtain a second sample image, the second sample image comprising the target object and a second bounding box obtained by randomly moving the first bounding box; moving the first bounding box in the first sample image in parallel by using a first parameter to obtain a third sample image, the third sample image comprising the target object and a third bounding box obtained by moving the first bounding box in parallel, wherein the third bounding box is obtained by moving the first bounding box in the X-axis direction, or the third bounding box is obtained by moving the first bounding box in the Y-axis direction; pre-training an initial first key point recognition model based on the first image set, the second sample image and the third sample image to obtain a pre-trained first key point recognition model; moving the first bounding box in the first sample image in parallel by using a second parameter to obtain a fourth sample image; and further training the pre-trained first key point recognition model based on the first sample image, the second sample image and the fourth sample image to obtain a trained first key point recognition model. This is equivalent to training the initial first key point recognition model in two steps, i.e., pre-training and training. Since the hand part may move out of the field of view of the camera, the hand key points may not be within the bounding box, and thus, if the initial first key point recognition model is trained in one step, the model may not converge due to the hand key points not being within the bounding box. However, the model is trained in two steps, i.e., pre-training and training, which can effectively avoid the situation that the model does not converge due to the hand key points not being within the bounding box.

[0016] In combination with the first aspect, in some embodiments of the first aspect, the first sub-parameter is greater than the third sub-parameter, and the second sub-parameter is greater than the fourth sub-parameter.

[0017] In the method for training a neural network model provided in the embodiments of the present application, the data augmentation rate (the first sub-parameter) used when pre-training the initial first key point recognition model is greater than the data augmentation rate (the third sub-parameter) used when further training the pre-trained first key point recognition model, and the movement distance of the third bounding box in the third sample image used when pre-training the initial first key point recognition model is greater than the movement distance of the fourth bounding box in the fourth sample image used when further training the pre-trained first key point recognition model. This is equivalent to pre-training the initial first key point recognition model by using a third sample image with a larger change, so that the pre-trained first key point recognition model is further trained by using a fourth sample image with a smaller change, which is easier to converge, thereby improving the accuracy of the trained first key point recognition model.

[0018] With reference to the first aspect, in some embodiments of the first aspect, the pre-training of the initial first key point recognition model based on the first image set, the second sample image and the third sample image comprises: removing an image region outside the third bounding box in the third sample image to obtain an updated third sample image; and pre-training the initial first key point recognition model based on the first image set, the second sample image and the updated third sample image to obtain the pre-trained first key point recognition model.

[0019] The method for training a neural network model provided in the embodiments of the present application can remove the image region outside the third bounding box in the third sample image to obtain an updated third sample image in the process of pre-training the initial first key point recognition model, and then pre-train the initial first key point recognition model based on the first image set, the second sample image and the updated third sample image to obtain the pre-trained first key point recognition model, so that the points outside the third bounding box can be prevented from participating in the model training, the data volume of the pre-trained neural network model is reduced, and the difficulty of pre-training the initial first key point recognition model is reduced.

[0020] With reference to the first aspect, in some embodiments of the first aspect, the method further comprises: obtaining a third image set, the third image set comprising a fifth sample image, the fifth sample image being an image collected by the camera device; and obtaining the first sample image based on the fifth sample image, depth information corresponding to the fifth sample image and a target detection model, the target detection model being configured to label the first bounding box of the target object.

[0021] With reference to the first aspect, in some embodiments of the first aspect, the obtaining of the first sample image based on the fifth sample image, depth information corresponding to the fifth sample image and the target detection model comprises: obtaining a sixth sample image based on the fifth sample image and the target detection model, the sixth sample image comprising the first bounding box; and performing three-dimensional conversion on the sixth sample image based on the depth information corresponding to the fifth sample image to obtain the first sample image.

[0022] It can be understood that some gestures are gestures perpendicular to the screen of the electronic device, such as the gesture shown in FIG. 8A. In this case, if the coordinates of the hand key points in the image are usually two-dimensional coordinates, gesture recognition based only on the coordinates of the hand key points in the image may result in an inaccurate gesture recognition result. Figure 12

[0023] It can be understood that, since the coordinates of each pixel point in the first sample image are three-dimensional coordinates, and the second sample image is obtained by randomly moving the first bounding box in the first sample image, the coordinates of each pixel point in the second sample image are also three-dimensional coordinates, including the depth information of each pixel point.​

[0024] It can be understood that, since the coordinates of each pixel point in the first sample image are three-dimensional coordinates, the third sample image is obtained by parallelly moving the first boundary in the first sample image according to the first parameter, and therefore the coordinates of each pixel point in the third sample image are also three-dimensional coordinates, including depth information of each pixel point.

[0025] With reference to the first aspect, in some embodiments of the first aspect, the above-mentioned three-dimensional conversion of the sixth sample image based on the depth information corresponding to the fifth sample image to obtain the first sample image comprises:

[0026] The first formula comprises:

[0027] x=X / W;

[0028] y=Y / H;

[0029] z=(Z-Z0) / W;

[0030] wherein x is the coordinate of the pixel point on the x-axis in the first sample image, y is the coordinate of the pixel point on the y-axis in the first sample image, z is the coordinate of the pixel point on the z-axis in the first sample image, X is the coordinate of the pixel point corresponding to x on the x-axis in the sixth sample image, Y is the coordinate of the pixel point corresponding to y on the y-axis in the sixth sample image, Z is the coordinate of the pixel point corresponding to z on the z-axis in the sixth sample image, Z0 represents the coordinate of the target key point on the z-axis in the sixth sample image, W represents the width of the first boundary box, and H represents the height of the first boundary box.

[0031] The method for training a neural network model provided in the embodiments of the present application comprises the following steps: obtaining a third image set comprising a fifth sample image collected by a camera; obtaining a sixth sample image based on the fifth sample image and a target detection model, the sixth sample image comprising a first bounding box; performing three-dimensional conversion on the sixth sample image based on depth information corresponding to the fifth sample image to obtain a first sample image, so that the first sample image is an image of three-dimensional coordinates, i.e., the first sample image comprises depth information of each pixel point; obtaining a second sample image, a third sample image and a fourth sample image based on the first sample image; and training an initial first key point model based on the first sample image, the second sample image, the third sample image and the fourth sample image. As a result, the first sample image, the second sample image, the third sample image and the fourth sample image used in the process of training the initial first key point model are all three-dimensional images and all comprise depth information, so that the coordinates of the hand key points obtained based on the trained first key point recognition model are three-dimensional coordinates comprising depth information, and the gesture recognition based on the three-dimensional coordinates of the hand key points is more accurate, especially when recognizing gestures perpendicular to the screen of the electronic device, the gesture recognition result obtained is more accurate.

[0032] In combination with the first aspect, in some embodiments of the first aspect, the target object comprises a hand in the image.

[0033] In combination with the first aspect, in some embodiments of the first aspect, the first key point recognition model is used to identify hand key points in the image.

[0034] In combination with the first aspect, in some embodiments of the first aspect, after the initial first key point recognition model is trained based on the first image set and the second image set to obtain the trained first key point recognition model, the method further comprises the following steps: obtaining a fourth image set, the fourth image set comprising a seventh sample image, the seventh sample image being an image obtained for a preset gesture of the target object; training an initial second key point recognition model based on the fourth image set to obtain a trained second key point recognition model, the second key point recognition model being used to identify key points corresponding to the preset gesture; and correcting parameters in the trained first key point recognition model based on parameters in the trained second key point recognition model to obtain an updated first key point recognition model.

[0035] The method for training a neural network model provided in the embodiments of the present application comprises the following steps: obtaining a third image set comprising a fifth sample image collected by a camera; obtaining a sixth sample image based on the fifth sample image and a target detection model, the sixth sample image comprising a first bounding box; performing three-dimensional conversion on the sixth sample image based on depth information corresponding to the fifth sample image to obtain a first sample image, so that the first sample image is an image of three-dimensional coordinates, i.e., the first sample image comprises depth information of each pixel point; obtaining a second sample image, a third sample image and a fourth sample image based on the first sample image; and training an initial first key point model based on the first sample image, the second sample image, the third sample image and the fourth sample image. As a result, the first sample image, the second sample image, the third sample image and the fourth sample image used in the process of training the initial first key point model are all three-dimensional images and all comprise depth information, so that the coordinates of the hand key points obtained based on the trained first key point recognition model are three-dimensional coordinates comprising depth information, and the gesture recognition based on the three-dimensional coordinates of the hand key points is more accurate, especially when recognizing gestures perpendicular to the screen of the electronic device, the gesture recognition result obtained is more accurate.

[0036] With reference to the first aspect, in some embodiments of the first aspect, the first key point recognition model comprises a first network and a first regression layer, and the second key point recognition model comprises the first network and a second regression layer, wherein a dimension of the first regression layer is higher than a dimension of the second regression layer.

[0037] The training method of the neural network model provided in the embodiments of the present application, the first key point recognition model comprises a first network and a first regression layer, and the second key point recognition model comprises the first network and a second regression layer, wherein a dimension of the first regression layer is higher than a dimension of the second regression layer. The second regression layer with a lower dimension is used to train the hand key points corresponding to the preset gesture. This is only to train the small number of hand key points corresponding to the preset gesture, so as to strengthen the training of the hand key points corresponding to the preset gesture and improve the recognition ability of the hand key points corresponding to the preset gesture while avoiding the influence on other hand key points.

[0038] In a second aspect, a training method of a neural network model is provided, comprising:

[0039] training an initial first key point recognition model based on the first image set and the second image set to obtain a trained first key point recognition model;

[0040] obtaining a fourth image set, the fourth image set comprising a seventh sample image, the seventh sample image being an image obtained for a preset gesture of the target object;

[0041] training an initial second key point recognition model based on the fourth image set to obtain a trained second key point recognition model, the second key point recognition model being used to recognize key points corresponding to the preset gesture;

[0042] correcting parameters in the trained first key point recognition model based on parameters in the trained second key point recognition model to obtain an updated first key point recognition model.

[0043] In the first image set, a plurality of first sample images are included, the first sample images include the target object, and a first bounding box obtained by detecting the target object; the sample images in the second image set are obtained by moving the first bounding box of the first sample images in the first image set.

[0044] The method for training the neural network model provided in the embodiments of the present application trains the second key point recognition model corresponding to the special gesture, and then corrects the parameters of the first key point recognition model using the parameters of the trained second key point recognition model, to obtain the updated first key point recognition model, so that the updated first key point recognition model can also accurately recognize the hand key points corresponding to the special gesture, thereby improving the robustness of the updated first key point recognition model.

[0045] In combination with the second aspect, in some embodiments of the second aspect, the first key point recognition model includes a first network and a first regression layer, the second key point recognition model includes the first network and a second regression layer, and the dimension of the first regression layer is higher than the dimension of the second regression layer.

[0046] The method for training the neural network model provided in the embodiments of the present application trains the second key point recognition model corresponding to the special gesture, and then corrects the parameters of the first key point recognition model using the parameters of the trained second key point recognition model, to obtain the updated first key point recognition model, so that the updated first key point recognition model can also accurately recognize the hand key points corresponding to the special gesture, thereby improving the robustness of the updated first key point recognition model.

[0047] In combination with the second aspect, in some embodiments of the second aspect, before the first image set and the second image set are used to train the initial first key point recognition model to obtain the trained first key point recognition model, the method further includes: obtaining the first image set, the first image set including a plurality of first sample images, the first sample images including a target object and a first bounding box obtained by detecting the target object; and moving the first bounding box of the first sample images in the first image set to obtain the second image set.

[0048] In combination with the second aspect, in some embodiments of the second aspect, the second image set includes a second sample image and a third sample image, and moving the bounding box of the first sample image in the first image set to obtain the second image set includes: randomly moving the first bounding box in the first sample image to obtain the second sample image, the second sample image including the target object and a second bounding box obtained by randomly moving the first bounding box; and moving the bounding box in the first sample image in parallel to obtain the third sample image, the third sample image including the target object and a third bounding box obtained by moving the first bounding box in parallel, the third bounding box being obtained by moving the first bounding box in parallel along the X axis, or the third bounding box being obtained by moving the first bounding box in parallel along the Y axis.

[0049] With reference to the second aspect, in some embodiments of the second aspect, the parallel moving the bounding box in the first sample image to obtain the third sample image comprises: parallel moving a first bounding box in the first sample image according to a first parameter to obtain the third sample image, the first parameter comprising a first sub-parameter and a second sub-parameter, the first sub-parameter being used to indicate a ratio of a number of third sample images obtained by parallel moving to a number of first sample images in the first image set, and the second sub-parameter being used to indicate a ratio of the first distance to a width or a height of the first bounding box, the first distance being a distance from the first bounding box to a third bounding box; and the training the initial first key point recognition model based on the first image set and the second image set to obtain the trained first key point recognition model comprises: pre-training the initial first key point recognition model based on the first image set, the second sample image and the third sample image to obtain a pre-trained first key point recognition model; parallel moving the bounding box in the first sample image according to a second parameter to obtain a fourth sample image, the fourth sample image comprising a target object and a fourth bounding box, the second parameter comprising a third sub-parameter and a fourth sub-parameter, the third sub-parameter being used to indicate a ratio of a number of fourth sample images obtained by parallel moving to a number of first sample images in the first image set, and the fourth sub-parameter being used to indicate a ratio of a second distance to a width or a height of the first bounding box, the second distance being a distance from the first bounding box to the fourth bounding box; and training the pre-trained first key point recognition model based on the first image set, the second sample image and the fourth sample image to obtain the trained first key point recognition model.

[0050] With reference to the second aspect, in some embodiments of the second aspect, the first sub-parameter is greater than the third sub-parameter, and the second sub-parameter is greater than the fourth sub-parameter.

[0051] With reference to the second aspect, in some embodiments of the second aspect, the pre-training the initial first key point recognition model based on the first image set, the second sample image and the third sample image to obtain the pre-trained first key point recognition model comprises: removing an image region outside the third bounding box in the third sample image to obtain an updated third sample image; and pre-training the initial first key point recognition model based on the first image set, the second sample image and the updated third sample image to obtain the pre-trained first key point recognition model.

[0052] With reference to the second aspect, in some embodiments of the second aspect, the method further comprises: obtaining a third image set, the third image set comprising a fifth sample image, the fifth sample image being an image collected by a camera device; and obtaining the first sample image based on the fifth sample image, depth information corresponding to the fifth sample image and a target detection model, the target detection model being used to label the first bounding box of the target object.

[0053] With reference to the second aspect, in some embodiments of the second aspect, the obtaining the first sample image based on the fifth sample image, the depth information corresponding to the fifth sample image, and the target detection model comprises: obtaining a sixth sample image based on the fifth sample image and the target detection model, the sixth sample image comprising the first bounding box; and performing three-dimensional conversion on the sixth sample image based on the depth information corresponding to the fifth sample image to obtain the first sample image.

[0054] With reference to the second aspect, in some embodiments of the second aspect, the obtaining the first sample image based on the fifth sample image, the depth information corresponding to the fifth sample image, and the target detection model comprises: obtaining a sixth sample image based on the fifth sample image and the target detection model, the sixth sample image comprising the first bounding box; and performing three-dimensional conversion on the sixth sample image based on the depth information corresponding to the fifth sample image to obtain the first sample image.

[0055] x = X / W;

[0056] y = Y / H;

[0057] z = (Z-Z0) / W;

[0058] wherein x is a coordinate of a pixel point on an x-axis in the first sample image, y is a coordinate of the pixel point on a y-axis in the first sample image, z is a coordinate of the pixel point on a z-axis in the first sample image, X is a coordinate of a pixel corresponding to x on an x-axis in the sixth sample image, Y is a coordinate of a pixel corresponding to y on a y-axis in the sixth sample image, Z is a coordinate of a pixel corresponding to z on a z-axis in the sixth sample image, Z0 represents a coordinate of the target key point on the z-axis in the sixth sample image, and W represents a width of the first bounding box and H represents a height of the first bounding box.

[0059] With reference to the second aspect, in some embodiments of the second aspect, the target object comprises a hand in the image.

[0060] With reference to the second aspect, in some embodiments of the second aspect, the first key point identification model is configured to identify a hand key point in the image.

[0061] In a third aspect, a device for training a neural network model is provided, which comprises a unit for executing any method in the first aspect or the second aspect. The device can be a server, a terminal device, or a chip in a terminal device. The device can comprise an acquisition unit and a processing unit.

[0062] When the device is a terminal device, the processing unit can be a processor, and the acquisition unit can be a communication interface. The terminal device can further comprise a memory configured to store computer program code, and when the processor executes the computer program code stored in the memory, the terminal device is caused to execute any method in the first aspect or the second aspect.

[0063] When the apparatus is a chip in a terminal device, the processing unit can be a processing unit inside the chip, and the acquisition unit can be an output interface, a pin, a circuit, or the like. The chip can further include a memory, which can be a memory (for example, a register, a cache, or the like) inside the chip or a memory (for example, a read-only memory, a random access memory, or the like) outside the chip. The memory is configured to store computer program code. When the processor executes the computer program code stored in the memory, the chip is caused to perform any one of the methods in the first aspect or the second aspect.

[0064] In a possible implementation, the memory is configured to store computer program code; and the processor is configured to execute the computer program code stored in the memory, and when the computer program code stored in the memory is executed, the processor is configured to perform: acquiring a first image set, the first image set including a plurality of first sample images, each of the first sample images including a target object, and a first bounding box obtained by detecting the target object; moving the first bounding box of each of the first sample images in the first image set to obtain a second image set; and training an initial first key point recognition model based on the first image set and the second image set to obtain a trained first key point recognition model, the first key point recognition model being configured to recognize a key point in the target object.

[0065] In a possible implementation, the memory is configured to store computer program code; and the processor is configured to execute the computer program code stored in the memory, and when the computer program code stored in the memory is executed, the processor is configured to perform: training an initial first key point recognition model based on a first image set and a second image set to obtain a trained first key point recognition model; acquiring a fourth image set, the fourth image set including a seventh sample image, the seventh sample image being an image obtained by a preset gesture of the target object; training an initial second key point recognition model based on the fourth image set to obtain a trained second key point recognition model, the second key point recognition model being configured to recognize a key point corresponding to the preset gesture; and correcting parameters in the trained first key point recognition model based on parameters in the trained second key point recognition model to obtain an updated first key point recognition model.

[0066] In a fourth aspect, a computer readable storage medium is provided, which stores computer program code. When the computer program code is run by a neural network model training apparatus, the neural network model training apparatus performs any one of the neural network model training methods in the first aspect or the second aspect.

[0067] In a fifth aspect, a computer program product is provided, which includes computer program code, which, when executed by a training apparatus of a neural network model, causes the training apparatus of the neural network model to perform any of the apparatus methods of the first aspect or the second aspect.

[0068] The neural network model training method and the electronic device provided in the embodiments of the present application first acquire a first image set including a plurality of first sample images, wherein the first sample images include a target object and a first bounding box obtained by detecting the target object, then move the first bounding box of the first sample image in the first image set to obtain a second image set, and then train an initial first key point recognition model based on the first image set and the second image set to obtain a trained first key point recognition model. The first key point recognition model is used to recognize key points in the target object. By moving the first bounding box in the first sample image, the case that the position of the hand in the image greatly differs due to hand shaking in the image acquisition process and the case that the hand partially moves out of the image due to the small field of view angle of the camera are simulated. Therefore, the sample images for training the initial first key point recognition model not only include the initial first sample images, but also include sample images adjusted for the small field of view angle of the camera and the hand shaking problem. The types of the sample images for training the initial first key point recognition model are enriched, and thus the robustness of the trained first key point model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 FIG. 1 is a flow diagram of a gesture recognition process;

[0070] Figure 2 FIG. 2 is a schematic diagram of a hand bounding box;

[0071] Figure 3 FIG. 3 is a schematic diagram of a hand key point;

[0072] Figure 4 FIG. 4 is a structural diagram of a convolutional neural network model;

[0073] Figure 5 FIG. 5 is a structural diagram of a depthwise separable convolution layer;

[0074] Figure 6 FIG. 6 is a structural diagram of a system architecture provided in the embodiments of the present application;

[0075] Figure 7 FIG. 7 is a flow diagram of a neural network model training method provided in the embodiments of the present application;

[0076] Figure 8 FIG. 8 is a flow diagram of another neural network model training method provided in the embodiments of the present application;

[0077] Figure 9 is a schematic diagram of the first bounding box and the second bounding box provided by an embodiment of the present application;

[0078] Figure 10 is a schematic diagram of the first bounding box and the third bounding box provided by an embodiment of the present application;

[0079] Figure 11 is a flowchart of another method for training a neural network model provided by an embodiment of the present application;

[0080] Figure 12 is a schematic diagram of a preset gesture provided by an embodiment of the present application;

[0081] Figure 13 is a flowchart of another method for training a neural network model provided by an embodiment of the present application;

[0082] Figure 14 is a schematic diagram of the first bounding box and the third bounding box provided by an embodiment of the present application;

[0083] Figure 15 is a flowchart of another method for training a neural network model provided by an embodiment of the present application;

[0084] Figure 16 is a structural schematic diagram of the first key point recognition model and the second key point recognition model provided by an embodiment of the present application;

[0085] Figure 17 is a flowchart of another method for training a neural network model provided by an embodiment of the present application;

[0086] Figure 18 is a schematic diagram of a training device of a neural network model provided by the present application;

[0087] Figure 19 is a schematic diagram of an electronic device for training a neural network model provided by the present application. DETAILED DESCRIPTION

[0088] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; in this document, "and / or" only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0089] Hereinafter, the terms "first", "second", "third" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", "third" can explicitly or implicitly include one or more of the features.

[0090] For the convenience of understanding, the examples are given for reference in the description of the concepts related to the embodiments of the present application.

[0091] (1) Neural network.

[0092] The neural network can be composed of neural units, and the neural unit can refer to an operation unit taking x s and intercept 1 as input, and the output of the operation unit can be as shown in formula (1):

[0093]

[0094] wherein s = 1, 2, … n, n is a natural number greater than 1, W s is the weight of x s , and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to perform nonlinear transformation on the features in the neural network, so as to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next layer of convolution layer, and the activation function can be sigmoid function. The neural network is a network formed by connecting a plurality of the above single neural units, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected with the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of a plurality of neural units.

[0095] (2) Convolutional neural network (CNN).

[0096] The CNN is a deep neural network with a convolutional structure, and includes a feature extractor composed of a convolutional layer and a subsampling layer, which can be regarded as a filter. The convolutional layer refers to a neuron layer that performs convolution processing on an input signal in the CNN. In the convolutional layer of the CNN, a neuron is connected to only part of the adjacent layer neurons. In a convolutional layer, a plurality of feature planes are usually included, and each feature plane can be composed of a plurality of rectangularly arranged neural units. The neural units of the same feature plane share weights, and the shared weights are the convolution kernel. The shared weights can be understood as being irrelevant to the manner and position of extracting image information. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning in the training process of the CNN. In addition, the shared weights directly reduce the connections between the layers of the CNN and reduce the risk of overfitting.

[0097] In the specific feature extraction, a convolution operation can be used, which can be composed of a convolution operator with the same size of the convolution kernel, a rectified linear unit (ReLU), and batch normalization (BN). The ReLU is used to enhance the nonlinear relationship between the input and output of the model, and the expression is RelU(x)=max(0,x). In a possible case, the ReLU can also be referred to as an activation function. The BN is used to transform the input data into a normal distribution with a mean value of 0 and a unit variance by using a normalization transformation function.

[0098] (3) Back propagation algorithm.

[0099] The neural network can use a back propagation (BP) algorithm to correct the values of the parameters in the initial neural network model in the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward propagation of the input signal until the output generates an error loss, and the initial neural network model is updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by the error loss, and aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0100] Gesture recognition is a very important human-computer interaction method, which is widely used in smart phones, smart wear, car-machine interaction, augmented reality (AR) and virtual reality (VR) and other electronic devices. Among them, gesture recognition can refer to collecting a video stream about gesture information through a camera, identifying the gesture in the video stream through a preset neural network model, and then performing a corresponding operation based on the identified gesture. In this way, the user does not need to contact the electronic device, and does not need to wear additional sensors, and the electronic device can perform corresponding operations according to the user's gestures.

[0101] In a possible case, the camera is a front camera of a smart phone, and the field of view angle of the front camera is usually small. When the user performs gesture recognition, the distance between the user's hand and the camera is usually small, and the slight shaking of the user's hand will also cause the gesture in the video stream collected by the camera to have large jitter, thereby causing the gesture recognition result to be inaccurate due to the large displacement of the hand; or the user slightly moves the hand, and the hand moves out of the field of view range of the camera, thereby causing the gesture recognition result based on the incomplete hand to be inaccurate; that is, in the case where the field of view angle of the camera is small, the accuracy of gesture recognition is prone to be low.

[0102] Therefore, the embodiment of the present application provides a neural network model training method and an electronic device, and the neural network model training method provided by the embodiment of the present application first acquires a first image set including a plurality of first sample images, wherein the first sample image includes a target object and a first bounding box obtained by detecting the target object, then moves the first bounding box of the first sample image in the first image set to obtain a second image set, and then trains an initial first key point recognition model based on the first image set and the second image set to obtain a trained first key point recognition model. The first key point recognition model is used to identify key points in the target object. By moving the first bounding box in the first sample image, the case that the position of the hand in the image is greatly different due to hand shaking in the image acquisition process and the case that part of the hand moves out of the image due to the small field of view angle of the camera are simulated. The sample images for training the initial first key point recognition model not only include the initial first sample images, but also include sample images adjusted for the small field of view angle of the camera and the hand shaking problem, which enriches the types of sample images for training the initial first key point recognition model, thereby improving the robustness of the trained first key point model.

[0103] It can be understood that the process of gesture recognition on the video stream collected by the camera can be divided into multiple steps. Among them, the camera can be a camera (for example, a color camera, a grayscale camera, and a depth camera, etc.), and the image containing the hand image stream (for example, it can be a continuous frame image containing the hand image stream) can be obtained through the camera; Next, the image containing the hand image stream is processed, and the hand movement therein is recognized into a predefined gesture category; Finally, the gesture category recognized by the dynamic gesture recognition system is responded (for example, taking a photo, playing music, etc.), so as to realize the interaction of the gesture.

[0104] For example, as shown in Figure 1 , in the process of gesture recognition on the video stream collected by the camera, the following steps can be taken:

[0105] Step 1, target detection is performed on the image in the video stream to identify the hand region in the image.

[0106] Target detection is performed on the image in the video stream to identify the hand region in the image, and a bounding box is set in the hand region in the image. For example, the bounding box 11 in image 1 in Figure 2 . Among them, the target detection on the image in the video stream can be realized by a target detection model.

[0107] Step 2, key point recognition is performed on the image region in the bounding box to obtain hand key points.

[0108] Since the human body is usually driven by the movement of the joints when moving, the movement of the human body can be determined based on the movement of the joints. The hand key points can refer to the joints of the hand. For example, as shown in Figure 3 , the hand key points can include 21 hand key points, such as wrist (0), wrist metacarpal joint (1), thumb metacarpophalangeal joint (2), thumb interphalangeal joint (3), thumb tip (4), index finger metacarpophalangeal joint (5), index finger proximal interphalangeal joint (6), index finger distal interphalangeal joint (7), index finger tip (8), middle finger metacarpophalangeal joint (9), middle finger proximal interphalangeal joint (10), middle finger distal interphalangeal joint (11), middle finger tip (12), ring finger metacarpophalangeal joint (13), ring finger proximal interphalangeal joint (14), ring finger distal interphalangeal joint (15), ring finger tip (16), little finger metacarpophalangeal joint (17), little finger proximal interphalangeal joint (18), little finger distal interphalangeal joint (19), little finger tip (20).

[0109] Among them, the key point recognition on the image region in the bounding box to obtain the hand key points can be realized by a key point recognition model, and the key point recognition model can be a neural network model, such as a convolutional neural network model of MobileNet v1.

[0110] The MobileNet v1 convolutional neural network model provided in this application embodiment, wherein, as Figure 4 As shown, MobileNetv1 may include an input layer 210, a convolutional layer 220, a deep sequential convolutional network (DSConv) 230, an adaptive pooling layer 240, a linear regression layer 250, and an output layer 260.

[0111] The image with the hand region marked is used as input to the input layer 210. After passing through the convolutional layer 220, the depthwise classifiable convolutional layer 230, the adaptive pooling layer 240, and the linear regression layer 250, the hand key points are determined. The image with the hand key points marked is output through the output layer 260.

[0112] The following section will use a convolutional layer as an example to introduce the internal working principle of a convolutional layer.

[0113] A convolutional layer can contain multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator is a weight matrix, typically predefined. During the convolution operation, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, extracting specific features. The size of this weight matrix should be related to the image size. Importantly, the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during the convolution operation, the weight matrix extends to the entire depth of the input image.

[0114] Therefore, convolution with a single weight matrix produces a convolutional output with a single dimension of depth. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied, i.e., multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. This dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix is ​​used to extract edge information, another to extract specific colors, and yet another to blur unwanted noise. Since these multiple weight matrices are of the same size (rows × columns), the resulting convolutional feature maps are also of the same size. These multiple convolutional feature maps of the same size are then merged to form the output of the convolution operation.

[0115] The weight values in these weight matrices need to be obtained through a large amount of training in practical applications, and each weight matrix formed by the weight values obtained through training can be used to extract information from an input image, so that the convolutional neural network makes correct predictions.

[0116] When the convolutional neural network has multiple convolutional layers, the initial convolutional layer often extracts more general features, which can also be referred to as low-level features; as the depth of the convolutional neural network deepens, the features extracted by the later convolutional layers are more and more complex, such as high-level semantic features and the like, and the higher the semantic features, the more suitable they are for the problem to be solved.

[0117] Adaptive pooling layer (referred to as pooling layer):

[0118] Because it is often necessary to reduce the number of training parameters, a pooling layer is often periodically introduced after a convolutional layer, which can be a convolutional layer followed by a pooling layer, or a plurality of convolutional layers followed by one or more pooling layers. In image processing, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain a smaller size image. The average pooling operator can calculate the average value of the pixel values in the image within a certain range to obtain the average pooling result. The maximum pooling operator can take the maximum pixel value in the range as the maximum pooling result. In addition, just as the size of the weight matrix in the convolutional layer should be related to the size of the image, the operator in the pooling layer should also be related to the size of the image. The size of the image output by the pooling layer can be smaller than the size of the image input to the pooling layer, and each pixel point in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.

[0119] Linear regression layer (or full connection layer):

[0120] After processing by the convolutional layer / pooling layer, the convolutional neural network is not sufficient to output the required output information. As described above, the convolutional layer / pooling layer only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network needs to use the full connection layer to generate an output of one or a group of the required number of classes. Therefore, the full connection layer can include multiple hidden layers and an output layer, and the parameters contained in the multiple hidden layers can be pre-trained according to the related training data of a specific task type, for example, the task type can include image recognition, image classification, image super-resolution reconstruction, and the like.

[0121] After the multi-layer hidden layer in the full connection layer, that is, the last layer of the entire convolutional neural network is the output layer 260, which has a loss function similar to classification cross-entropy, which is specifically used to calculate the prediction error, and once the forward propagation of the entire convolutional neural network is completed, the reverse propagation starts to update the weight values and biases of the layers mentioned above to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0122] Step 3, based on the hand key points, the gesture recognition result is obtained.

[0123] After obtaining the hand key points, the gesture recognition result can be determined based on the displacement between the hand key points of N frames of images. The gesture recognition result can include upturn, downturn, left turn, right turn, screen capture, pinch, etc.

[0124] For example, the coordinate origin of the mobile phone display screen is at the top left corner. In the first frame, the coordinates of the distal interphalangeal joint (7) of the index finger are (X1, Y1), in the second frame, the coordinates of the distal interphalangeal joint (7) of the index finger are (X2, Y1), in the third frame, the coordinates of the distal interphalangeal joint (7) of the index finger are (X3, Y1)……, in the Nth frame, the coordinates of the distal interphalangeal joint (7) of the index finger are (XN, Y1), wherein X1>X2>X3……>XN, then the gesture recognition result can be determined as an upturn gesture based on this.

[0125] It should be noted that the neural network model as shown in Figure 4 is only an example of a convolutional neural network, and in specific applications, the convolutional neural network can also exist in the form of other network models.

[0126] As shown in Figure 5 , the DSConv 230 can include a 3*3 depth separable convolution layer 231, a normalization layer BN 232, an activation function ReLU 233, a 1*1 convolution layer 234, a normalization layer BN 235, and an activation function ReLU 236.

[0127] The DSConv is composed of a multi-layer convolutional neural network, and each layer is composed of a convolution layer, a pooling layer (normalization layer BN), and an activation function.

[0128] The convolution layer is used to extract local features of the input sequence, and a series of feature maps are obtained by sliding a learnable convolution kernel (filter) on the input sequence. Each feature map corresponds to a convolution kernel, and different convolution kernels can extract different features.

[0129] The pooling layer is used to reduce the size of the feature map and retain important features. Common pooling operations include max pooling and average pooling.

[0130] The activation function is a non-linear transformation in DSConv, used to introduce non-linear capabilities. Commonly used activation functions include ReLU, Sigmoid, and Tanh. Activation functions can enhance the expressive power of the model, enabling it to learn more complex features and patterns. In the embodiments of this application... Figure 5 The activation function in the DSConv shown is ReLU.

[0131] The following is through Figure 6 The system architecture provided in the embodiments of this application is described below. See appendix. Figure 6 , Figure 6 This is a schematic diagram of the system architecture according to an embodiment of this application. For example... Figure 6 As shown, the system architecture 100 includes an execution device, a training device 120, a database 130, a client device 140, a data storage system 150, and a data acquisition system 160. Additionally, the execution device 110 includes a computing module 111, an I / O interface 112, a preprocessing module 113, and a preprocessing module 114. The computing module 111 may include a target model / rule 101, while the preprocessing modules 113 and 114 are optional.

[0132] The data acquisition device 160 is used to collect training data. After acquiring the training data, the target model / rule 101 can be trained based on the training data. Then, the trained target model / rule 101 can be used to execute the gesture recognition process of this application embodiment.

[0133] The target model / rule 101 mentioned above may include multiple neural network sub-models, each of which is used to perform the corresponding recognition process.

[0134] Specifically, the target model / rule 101 mentioned above can be composed of a first neural network model, a second neural network model, a third neural network model, and a fourth neural network model. The functions of these three neural network models are described below.

[0135] The first neural network model (e.g., object detection model) is used to detect the image stream to determine the hand bounding box of each frame in the image stream, that is, the bounding box of the hand.

[0136] The second neural network model (e.g., the key point detection model) is used to determine the key points of the hand based on the hand bounding box of each frame image, that is, the bounding box of the hand.

[0137] The third neural network model (gesture recognition model) is used to perform gesture recognition on image streams based on hand key points in order to determine the user's gesture actions;

[0138] The fourth neural network model is used for recognizing a frame of image to determine a hand gesture type of a user.

[0139] The four neural network sub-models can be obtained through separate training.

[0140] The first neural network sub-model can be trained by first type of training data, the first type of training data including multiple frames of hand images and label data of the multiple frames of hand images, wherein the label data of the multiple frames of hand images includes a bounding box in which a hand in each frame of hand image is located, i.e., a bounding box of the hand.

[0141] The second neural network sub-model can be trained by second type of training data, the second type of training data including multiple image streams and label data of the multiple image streams, wherein each image stream in the multiple image streams is composed of multiple frames of continuous hand images, and the label data of the multiple image streams includes hand key points corresponding to each image stream.

[0142] The third neural network sub-model can be trained by third type of training data, the third type of training data including multiple image streams and label data of the multiple image streams, wherein each image stream in the multiple image streams is composed of multiple frames of continuous hand images, and the label data of the multiple image streams includes gesture actions corresponding to each image stream.

[0143] The fourth neural network sub-model can be trained by fourth type of training data, the fourth type of training data including multiple frames of hand images and label data of the multiple frames of images, wherein the label data of the multiple frames of images includes gesture poses corresponding to each frame of image.

[0144] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as being applied to Figure 6 The execution device 110 shown in the figure can be a terminal, such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) / virtual reality (VR), a vehicle-mounted terminal, etc., and can also be a server or a cloud, etc. Figure 6 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device, and a user can input data to the I / O interface 112 through a client device 140, wherein the input data in the embodiment can include an image to be processed input by the client device. The client device 140 can be a terminal device.

[0145] The preprocessing modules 113 and 114 are configured to perform preprocessing on input data (e.g., images to be processed) received by the I / O interface 112. In some embodiments, there can be no preprocessing modules 113 and 114 or only one preprocessing module. When there are no preprocessing modules 113 and 114, the input data can be directly processed by the computing module 111.

[0146] During the preprocessing of the input data by the execution device 110 or the processing of the computing module 111 of the execution device 110, the execution device 110 can call data, codes, etc. in the data storage system 150 for the corresponding processing, and can store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0147] It is worth noting that, Figure 6 The above is only an example of a system architecture provided by the embodiments of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 6 In the above example, the data storage system 150 is an external memory relative to the execution device 110, and in other cases, the data storage system 150 can be placed in the execution device 110.

[0148] As shown in Figure 6 The target model / rule 101 trained by the training device 120 can be a neural network in the embodiments of the present application. Specifically, the neural network provided by the embodiments of the present application can be a CNN and a deep convolutional neural network (DCNN), etc.

[0149] The application scenarios of the neural network model training method provided by the embodiments of the present application will be briefly introduced below.

[0150] The neural network model training method provided by the embodiments of the present application can be applied in the scenario of gesture recognition. For example, in the gesture interaction scenario of a smart phone, through gesture recognition, simple, natural and convenient operation of the smart phone can be realized. For example, the smart phone can use a camera or other peripheral camera as an image sensor to obtain image information containing a hand image stream, and then process the image information containing the hand image stream to obtain gesture recognition information, and then report the gesture recognition information to the operating system for response. Through gesture recognition, functions such as page up and down, audio and video playing, volume control, reading and browsing, etc. can be realized, greatly improving the technological sense and interaction convenience of the smart phone.

[0151] It should be understood that the above is only an example of an application scenario, and does not limit the application scenario of the present application.

[0152] The following describes the method for training the neural network model in detail. Figure 7 to Figure 17 The method for training the neural network model provided in the embodiments of the present application is described in detail.

[0153] Figure 7 A flowchart of the method for training the neural network model provided in the embodiments of the present application is shown in FIG. 1, which comprises the following steps. Figure 7

[0154] S101, obtaining a first image set, the first image set comprising a plurality of first sample images, the first sample images comprising a target object and a first bounding box obtained by detecting the target object.

[0155] For example, the target detection model can be used to detect the target in the image captured by the camera to obtain the first sample image, i.e., the image comprising the target object and the first bounding box annotated for the target object. The first image set can comprise a plurality of first sample images.

[0156] Optionally, the target object can be the hand of a user, and the first bounding box annotated for the target object can be the bounding box annotated for the hand.

[0157] For example, as shown in FIG. 2, the first bounding box can be the bounding box 11 for the hand. Figure 2

[0158] S102, moving the first bounding box of the first sample image in the first image set to obtain a second image set.

[0159] The first bounding box of all the first sample images in the first image set can be moved to obtain the second image set, or the first bounding box of part of the first sample images in the first image set can be moved to obtain the second image set, which is not limited in the embodiments of the present application.

[0160] For example, the first image set comprises 1000 first sample images, the first bounding box of 30% of the first sample images in the first image set is moved, i.e., the first bounding box of 300 first sample images is moved, and 300 moved sample images are obtained as the second image set.

[0161] The first bounding box in the first sample image can be moved randomly or in a preset direction, which is not limited in the embodiments of the present application.

[0162] S103, training an initial first key point recognition model based on the first image set and the second image set to obtain a trained first key point recognition model.

[0163] ​​The first key point recognition model is used to identify key points in a target object. The target object can be a hand of a user, and thus the first key point model can be used to identify hand key points. The initial first key point recognition model can refer to a neural network with initialized parameters. The initial first key point recognition model cannot accurately identify hand key points from an image. By training the initialized first key point model and adjusting the initialized parameters, a trained first key point recognition model is obtained, which can accurately identify hand key points in an image.

[0164] During the training of the initial first key point model, the first sample images in the first image set are used, and sample images obtained by moving the first bounding boxes in the first sample images are also used. That is, sample images simulating a case where a hand partially moves out of the field of view of a camera due to a small field of view angle and sample images simulating a case where a hand shakes and thus the position of the hand in a sample image greatly changes. Thus, the trained first key point recognition model obtained by training the initial first key point recognition model is trained for the case where a hand partially moves out of the field of view of a camera due to a small field of view angle and the case where a hand shakes and thus the position of the hand in a sample image greatly changes.

[0165] The neural network model training method provided in the embodiments of the present application first obtains a first image set including a plurality of first sample images, wherein the first sample images include a target object and a first bounding box obtained by detecting the target object. Then, the first bounding box of the first sample image in the first image set is moved to obtain a second image set. Then, an initial first key point recognition model is trained based on the first image set and the second image set to obtain a trained first key point recognition model. The first key point recognition model is used to identify key points in a target object. The first bounding box in the first sample image is moved to simulate a case where a hand shakes and thus the position of the hand in an image greatly changes during image acquisition and a case where a hand partially moves out of an image due to a small field of view angle of a camera. Thus, the sample images used to train the initialized first key point recognition model include not only the initial first sample images but also sample images adjusted for the small field of view angle of the camera and the hand shaking problem. The types of sample images used to train the initial first key point recognition model are enriched, and thus the robustness of the trained first key point model is improved.

[0166] Since the gesture recognition result may be inaccurate due to hand shaking or hand moving out of the field of view of the camera, the first bounding box can be randomly moved to simulate the scenario of hand shaking, and the first bounding box can be moved in parallel to simulate the scenario of hand moving out of the field of view of the camera, so that the adjusted sample image is more consistent with the real situation, and the accuracy of the gesture recognition result is improved. The following will be described in detail with reference to the embodiments shown in the drawings. Figure 8

[0167] Figure 8 Another flowchart of a training method of a neural network model provided by the embodiments of the present application is shown in FIG. 8, and the method comprises the following steps. Figure 8

[0168] S201, a first image set is obtained, the first image set comprises a plurality of first sample images, the first sample image comprises a target object, and a first bounding box obtained by detecting the target object.

[0169] S202, the first bounding box in the first sample image is randomly moved to obtain a second sample image, the second sample image comprises the target object and a second bounding box obtained by randomly moving the first bounding box.

[0170] The following describes the target object as a hand. The first bounding box in the first sample image can be randomly moved as shown in (a) of FIG. 4 and (b) of FIG. 5, wherein the first sample image can be as shown in (a) of FIG. 4, comprising a hand 11 and a first bounding box 12 of the hand, the first bounding box 12 is randomly moved to obtain a second sample image as shown in (b) of FIG. 5, wherein the second sample image also comprises the hand 11 and a second bounding box 22. It can be seen that the position of the hand 11 in the second sample image is unchanged, which is the same as the position of the hand 11 in the first sample image, however, the positions of the first bounding box 12 and the second bounding box 21 are different. That is to say, the second sample image is a sample image obtained by moving the first bounding box 12 in the first sample image. Figure 9 Figure 9 Figure 9 Figure 9

[0171] That is to say, the second sample image is a sample image obtained by randomly moving the first bounding box in the first sample image. The second sample image obtained by randomly moving the first bounding box can simulate an image obtained by hand shaking.

[0172] The second sample image obtained by randomly moving the first bounding box can simulate an image obtained by hand shaking.

[0173] ​​​​​​It should be understood that the number of sample images when training the neural network model is usually large, and therefore, when the first bounding box is randomly moved, it can be moved according to preset parameters. For example, the preset parameters include setting a data enhancement probability parameter prob and a ratio move_ratio of moving the height and width of the first bounding box. Wherein, prob represents the proportion of the first sample image in the first image set that needs to move the first bounding box; move_ratio represents the ratio of the first distance to the width of the first bounding box, and the ratio of the first distance to the height of the first bounding box; wherein the first distance refers to the maximum moving distance of the first bounding box to the second bounding box. When moving the first bounding box according to move_ratio, a value is randomly selected between 0-move_ratio, and the first bounding box is moved according to the randomly selected value. For example, if move_ratio is 0.3, then according to move_ratio to move the first bounding box, a value is randomly selected from 0-0.3 multiple times to move the first bounding box to obtain multiple second sample images, which can enrich the obtained second sample images.

[0174] Figure 9 (c) is a schematic view of placing the first bounding box 12 and the second bounding box 22 in the same image, as shown in (c) of FIG. 1. Figure 9 As shown in (c) of FIG. 1, each edge of the first bounding box 12 and the second bounding box 22 coincides. The first distance can refer to the distance between the A1 point in the first bounding box 12 and the A2 point in the second bounding box 22.

[0175] S203, parallelly moving the first bounding box in the first sample image to obtain a third sample image, the third sample image comprising the target object and a third bounding box obtained by parallelly moving the first bounding box, the third bounding box being obtained by moving the first bounding box along the X-axis direction, or the third bounding box being obtained by moving the first bounding box along the Y-axis direction.

[0176] Unlike randomly moving the first bounding box in the first sample image, parallelly moving the first bounding box in the first sample image refers to that when the first bounding box is moved to obtain the third bounding box, the first bounding box and the third bounding box have the same coordinate in one direction. For example, the coordinate of the first bounding box on the X-axis is the same as the coordinate of the third bounding box on the X-axis, that is, the third bounding box is obtained by translating the first bounding box along the Y-axis direction; or the coordinate of the first bounding box on the Y-axis is the same as the coordinate of the third bounding box on the Y-axis, that is, the third bounding box is obtained by translating the first bounding box along the X-axis direction.

[0177] For example, as shown in (a) to (c) of FIG. 2, wherein the first sample image can be as shown in (a) of FIG. 2. Figure 10 Figure 10 (c) of FIG. 2, wherein the first sample image can be as shown in (a) of FIG. 2. Figure 10 ​As shown in (a), a first bounding box 12, including a hand 11 and a hand, is moved parallel to the X-axis to obtain the following: Figure 10 The third sample image shown in (b) also includes hand 11 and third bounding box 32. It can be seen that the position of hand 11 in the third sample image remains unchanged, the same as the position of hand 11 in the first sample image. However, the positions of the first bounding box 12 and the third bounding box 32 are different, but the Y-axis coordinate of the horizontal side of the third bounding box 32 is the same as the Y-axis coordinate of the horizontal side of the first bounding box 12. Figure 10 (c) in the image is a schematic diagram showing the first bounding box 12 and the third bounding box 32 placed in the same image, as shown below. Figure 10 As shown in (c), the horizontal edge of the first bounding box 12 and the horizontal edge of the third bounding box 32 completely coincide. However, the vertical edge on the left side of the third bounding box 32 does not coincide with the vertical edge on the left side of the first bounding box 12. The vertical edge on the left side of the third bounding box 32 is obtained by moving the vertical edge on the left side of the first bounding box 12 to the right along the X-axis. It is understandable that the bounding boxes are used to annotate the hand and are usually set along the edge of the hand. Therefore, although the vertical edge on the left side of the third bounding box 32 does not coincide with the vertical edge on the left side of the first bounding box 12, the vertical edge on the right side of the third bounding box 32 coincides with the vertical edge on the right side of the first bounding box 12, both being vertical edges set along the right edge of the hand. Simultaneously, it can be seen that in the third sample image, the key points of the little finger tip and the ring finger tip in the hand key points are outside the third bounding box 32.

[0178] For example, such as Figure 10 (a) Figure 10 (d) and Figure 10 As shown in (e) in the figure, the first sample image can be as follows: Figure 10 As shown in (a), a first bounding box 12, including a hand 11 and a hand, is moved parallel to the Y-axis to obtain the following: Figure 10 The third sample image shown in (d) also includes hand 11 and third bounding box 42. It can be seen that the position of hand 11 in the third sample image remains unchanged, the same as its position in the first sample image. However, the positions of the first bounding box 12 and the third bounding box 42 are different, but the X-axis coordinates of the vertical side of the third bounding box 42 are the same as those of the vertical side of the first bounding box 12. Figure 10 (e) in the image is a schematic diagram showing the first bounding box 12 and the third bounding box 42 placed in the same image, as shown below. Figure 10As shown in (e) of FIG. 1, the vertical edge of the first bounding box 12 and the horizontal edge of the third bounding box 42 completely coincide, but the horizontal edge below the third bounding box 42 is not coincided with the horizontal edge below the first bounding box 12, which is obtained by moving upward along the Y-axis direction. It can be understood that the bounding box is labeled for the hand, and is usually arranged along the edge of the hand, so that although the horizontal edge below the third bounding box 42 is the horizontal edge below the first bounding box 12 which is moved upward along the Y-axis, the horizontal edge above the third bounding box 42 is coincided with the horizontal edge above the first bounding box 12, which are both the horizontal edges arranged along the upper edge of the hand.

[0179] That is, the third sample image is a sample image obtained by parallel moving the first bounding box 12 in the first sample image. The third sample image obtained by parallel moving the first bounding box can simulate the image in which the hand moves out of the field of view of the camera.

[0180] It should be understood that the number of sample images when training the neural network model is usually large, so when parallel moving the first bounding box, the first bounding box can also be moved according to the preset parameters. For example, the preset parameters include setting a data enhancement probability parameter prob and a shift ratio of parallel moving the height and width of the first bounding box. Wherein, prob represents the proportion of the first sample image in the first image set which needs to move the first bounding box; shift_ratio represents the ratio of the first distance to the width of the first bounding box, and the ratio of the first distance to the height of the first bounding box; wherein the first distance refers to the maximum moving distance of the first bounding box to the third bounding box. When moving the first bounding box according to shift_ratio, a value is randomly selected between 0-shift_ratio, and the first bounding box is moved according to the randomly selected value. For example, shift_ratio is 0.2, then according to shift_ratio to move the first bounding box, a value is randomly selected from 0-0.2 multiple times to move the first bounding box to obtain multiple third sample images, which can enrich the obtained third sample images. Since the parallel movement is along the X-axis direction or the Y-axis direction, the first distance can refer to the distance between A1 point in the first bounding box 12 and A3 point in the third bounding box 32 as shown in (c) of FIG. 1; or the first distance can refer to the distance between A1 point in the first bounding box 12 and A4 point in the third bounding box 43 as shown in (e) of FIG. 1. Figure 10 Figure 11

[0181] S204, training the initial first key point recognition model based on the first image set, the second sample image and the third sample image to obtain a trained first key point recognition model, and the first key point recognition model is used to identify the key point in the target object.

[0182] ​​The neural network model training method provided in the embodiments of the present application comprises the following steps: obtaining a first image set, the first image set comprising a plurality of first sample images, the first sample images comprising a target object and a first bounding box obtained by detecting the target object; randomly moving the bounding box in the first sample image to obtain a second sample image, the second sample image comprising the target object and a second bounding box obtained by randomly moving the first bounding box; moving the first bounding box in the first sample image in parallel to obtain a third sample image, the third sample image comprising the target object and a third bounding box obtained by moving the first bounding box in parallel, the third bounding box being obtained by moving the first bounding box in the X-axis direction or being obtained by moving the first bounding box in the Y-axis direction; and training an initial first key point recognition model based on the first image set, the second sample image and the third sample image to obtain a trained first key point recognition model, the first key point recognition model being used to recognize key points in the target object. The first bounding box in the first sample image is randomly moved to simulate the case that the position of the hand in the image greatly differs due to hand shaking in the image acquisition process, and the first bounding box is moved in parallel to simulate the case that the hand partially moves out of the image due to the small field of view angle of the camera, so that the sample images for training the initial first key point recognition model not only comprise the initial first sample images, but also comprise the second sample images and the third sample images adjusted for hand shaking and the hand moving out of the field of view angle of the camera, the types of the sample images for training the initial first key point recognition model are enriched, and the robustness of the trained first key point model is improved.

[0183] Due to the small field of view angle of the camera, the distance of the hand moving out of the field of view angle of the camera is large, and when the third sample image is obtained by moving the first bounding box in the first sample image in parallel to simulate the above case, the distance of moving the first bounding box is usually large, so that the difference between the first sample image and the third sample image is large, and if the initial first key point recognition model is directly trained by using the third sample image, the model may not converge. Therefore, the initial first key point recognition model can be pre-trained by using a sample image with a small first bounding box moving distance to obtain a pre-trained first key point recognition model, and then the pre-trained first key point recognition model is trained by using a sample image with a large first bounding box moving distance to obtain a trained first key point recognition model. The embodiments are described in detail below. Figure 11

[0184] Figure 11 The flowchart of another neural network model training method provided in the embodiments of the present application is shown in FIG. 6, and the method comprises the following steps: Figure 12

[0185] ​​S301, acquire a first image set, the first image set comprising a plurality of first sample images, the first sample images comprising a target object, and a first bounding box obtained by detecting the target object.

[0186] S302, randomly move the bounding box in the first sample image to obtain a second sample image, the second sample image comprising the target object and a second bounding box obtained by randomly moving the first bounding box.

[0187] S303, move the first bounding box in the first sample image in parallel according to a first parameter to obtain a third sample image.

[0188] The first parameter comprises a first sub-parameter and a second sub-parameter, the first sub-parameter being used to indicate a ratio of a number of third sample images obtained by parallel movement to a number of first sample images in the first image set, and the second sub-parameter being used to indicate a ratio of a first distance to a width or height of the first bounding box, the first distance being a distance of the first bounding box moving to a third bounding box.

[0189] For example, the first sub-parameter can be a data enhancement probability parameter prob, and the second sub-parameter can be a ratio shift_ratio of parallel movement relative to the height and width of the first bounding box.

[0190] S304, pre-train an initial first key point recognition model based on the first image set, the second sample image and the third sample image to obtain a pre-trained first key point recognition model.

[0191] It can be understood that the third sample image is an image simulating a hand part moving out of a range of a field of view angle of a camera, in which some hand key points can be missing, and therefore, if a smaller data enhancement probability parameter and a smaller ratio of parallel movement relative to the height and width of the first bounding box are directly used to obtain the third sample image, the parameters of the neural network model can not converge in the training process, and the pre-trained first key point recognition model cannot be obtained. Therefore, a larger data enhancement probability parameter and a larger ratio of parallel movement relative to the height and width of the first bounding box can be used to obtain the third sample image to reduce the training difficulty.

[0192] For example, a data enhancement probability parameter prob=0.3 and a ratio shift_ratio=0.4 of parallel movement relative to the height and width of the first bounding box can be used to obtain the third sample image, and then the initial first key point recognition model is pre-trained based on the first image set, the second sample image and the third sample image to obtain the pre-trained first key point recognition model.

[0193] In order to further reduce the difficulty of training the neural network model, the points outside the bounding box can be shielded first, and in the process of training the neural network model, the points outside the bounding box do not participate in the model training when the loss function is returned, so that the training difficulty of the neural network model can be further reduced.

[0194] Optionally, an image region outside the third bounding box in the third sample image is removed to obtain an updated third sample image; and the initial first key point recognition model is pre-trained based on the first image set, the second sample image and the updated third sample image to obtain a pre-trained first key point recognition model.

[0195] The training method of the neural network model provided in the embodiments of the present application removes the image region outside the third bounding box in the third sample image to obtain an updated third sample image, and then pre-trains the initial first key point recognition model based on the first image set, the second sample image and the updated third sample image to obtain a pre-trained first key point recognition model, so that the points outside the third bounding box can be avoided to participate in the model training, the data volume of the pre-trained neural network model is reduced, and the difficulty of pre-training the initial first key point recognition model is reduced.

[0196] S305, moving the bounding box in the first sample image in parallel according to the second parameter to obtain a fourth sample image, the fourth sample image comprising the target object and a fourth bounding box.

[0197] The second parameter comprises a third sub-parameter and a fourth sub-parameter, the third sub-parameter being used to indicate a ratio of a number of the fourth sample images obtained by parallel moving to a number of the first sample images in the first image set, and the fourth sub-parameter being used to indicate a ratio of the second distance to a width or a height of the first bounding box, the second distance being a distance that the first bounding box is moved to the fourth bounding box.

[0198] For example, the third sub-parameter can be a data enhancement probability parameter, and the fourth sub-parameter can be a ratio of parallel moving to the height and the width of the first bounding box.

[0199] Optionally, the first sub-parameter is greater than the third sub-parameter, and the second sub-parameter is greater than the fourth sub-parameter.

[0200] For example, the third sub-parameter prob=0.1, and the fourth sub-parameter shift_ratio=0.3.

[0201] If the second sub-parameter is smaller, the fourth sample image has smaller difference with the first sample image, and thus the neural network model can be trained based on the first sample image and the fourth sample image with smaller difference with the first sample image, so as to improve the accuracy of the trained neural network model, that is, the accuracy of the trained first key point recognition model.

[0202] In S306, the pre-trained first key point recognition model is trained based on the first image set, the second sample image and the fourth sample image, to obtain a trained first key point recognition model.

[0203] The method for training a neural network model provided in the embodiments of the present application comprises the following steps: obtaining a first image set, the first image set comprising a plurality of first sample images, the first sample image comprising a target object and a first bounding box obtained by detecting the target object; randomly moving the bounding box in the first sample image to obtain a second sample image, the second sample image comprising the target object and a second bounding box obtained by randomly moving the first bounding box; moving the first bounding box in the first sample image in parallel by using a first parameter to obtain a third sample image, the third sample image comprising the target object and a third bounding box obtained by moving the first bounding box in parallel, the third bounding box being obtained by moving the first bounding box in the X-axis direction, or the third bounding box being obtained by moving the first bounding box in the Y-axis direction; pre-training an initial first key point recognition model based on the first image set, the second sample image and the third sample image to obtain a pre-trained first key point recognition model; moving the first bounding box in the first sample image in parallel by using a second parameter to obtain a fourth sample image; and further training the pre-trained first key point recognition model based on the first sample image, the second sample image and the fourth sample image to obtain a trained first key point recognition model. In this way, the training of the initial first key point recognition model is divided into two steps, pre-training and training, so as to effectively avoid the situation that the model training does not converge due to the hand key points not being in the bounding box.

[0204] It can be understood that some gestures are gestures perpendicular to the screen of the electronic device, for example, as shown in FIG. 6A, the gesture is perpendicular to the screen of the electronic device, and the gesture is not in the bounding box of the first sample image. Figure 13the gesture. In this case, if the coordinates of the hand key points in the image are usually two-dimensional coordinates, gesture recognition based only on the coordinates of the hand key points in the image can lead to inaccurate gesture recognition results. Therefore, the electronic device converts the hand key points into three-dimensional coordinates based on the depth information of the image to improve the accuracy of gesture recognition based on the hand key points. The following will be described in detail. Figure 13

[0205] Figure 13 Another flowchart of a training method of a neural network model provided by an embodiment of the present application is shown in FIG. 5, and the method includes the following steps. Figure 3

[0206] S401, a third image set is obtained, and the third image set includes a fifth sample image. The fifth sample image is an image collected by a camera device.

[0207] It can be understood that the camera device of the electronic device usually includes a color camera and a depth camera. The fifth sample image can be an image collected by the color camera and the depth camera, which includes not only color information of each pixel point in the image but also depth information of each pixel point.

[0208] S402, a sixth sample image is obtained based on the fifth sample image and a target detection model, and the sixth sample image includes a first bounding box.

[0209] The target detection model can be used to label a target object in an image to obtain a bounding box of the target object. By labeling the target object in the fifth sample image through the target detection model, a sixth sample image including the fifth sample image and a bounding box (i.e., the first bounding box) of the target object in the fifth sample image can be obtained. That is, the sixth sample includes the fifth sample image and the first bounding box.

[0210] S403, the sixth sample image is three-dimensionally converted based on the depth information corresponding to the fifth sample image to obtain a first sample image.

[0211] The electronic device can three-dimensionally convert the sixth sample image including the first bounding box based on the depth information in the fifth sample image to obtain a first sample image represented by three-dimensional coordinates, that is, the coordinates of each pixel point in the first sample image also include depth information.

[0212] Optionally, the sixth sample image is three-dimensionally converted based on the first formula and the depth information corresponding to the fifth sample image to obtain the first sample image, and the first formula includes:

[0213] x = X / W;

[0214] y = Y / H; ​​

[0215] z = (Z - Z0) / W;

[0216] wherein x is a coordinate of a pixel point in the first sample image on an x-axis, y is a coordinate of the pixel point in the first sample image on a y-axis, z is a coordinate of the pixel point in the first sample image on a z-axis, X is a coordinate of a pixel point corresponding to x on an x-axis in the sixth sample image, Y is a coordinate of a pixel point corresponding to y on a y-axis in the sixth sample image, Z is a coordinate of a pixel point corresponding to z on a z-axis in the sixth sample image, Z0 represents a coordinate of the target key point on the z-axis in the sixth sample image, and W represents a width of the first bounding box and H represents a height of the first bounding box.

[0217] The target key point can be a hand key point such as a wrist (0) as shown in FIG. 1. Figure 14 The coordinates of the pixel points on the z-axis can be represented by the depth information of the pixel points.

[0218] S404, obtaining a first image set, the first image set including a plurality of first sample images, the first sample images including a target object and a first bounding box obtained by detecting the target object.

[0219] S405, randomly moving the bounding box in the first sample image to obtain a second sample image, the second sample image including the target object and a second bounding box obtained by randomly moving the first bounding box.

[0220] It can be understood that, since the coordinates of the pixel points in the first sample image are three-dimensional coordinates, and the second sample image is obtained by randomly moving the first bounding box in the first sample image, the coordinates of the pixel points in the second sample image are also three-dimensional coordinates, including the depth information of the pixel points.

[0221] S406, moving the bounding box in the first sample image in parallel according to a first parameter to obtain a third sample image.

[0222] The first parameter includes a first sub-parameter and a second sub-parameter, the first sub-parameter is used to indicate a ratio of a number of the third sample images obtained by moving in parallel to a number of the first sample images in the first image set, and the second sub-parameter is used to indicate a ratio of a first distance to a width or a height of the first bounding box, the first distance being a distance of moving the first bounding box to a third bounding box.

[0223] It can be understood that, since the coordinates of the pixel points in the first sample image are three-dimensional coordinates, and the third sample image is obtained by moving the first bounding box in the first sample image in parallel according to the first parameter, the coordinates of the pixel points in the third sample image are also three-dimensional coordinates, including the depth information of the pixel points.

[0224] S407, pre-training the initial first key point recognition model based on the first image set, the second sample image and the third sample image to obtain a pre-trained first key point recognition model.

[0225] When pre-training the initial first key point recognition model, the image region outside the third bounding box in the third sample image can be removed, which can further reduce the amount of data for pre-training the initial first key point recognition model, thereby reducing the training difficulty of the initial first key point recognition model.

[0226] Optionally, the image region outside the third bounding box in the third sample image is removed to obtain an updated third sample image, and the initial first key point recognition model is pre-trained based on the first image set, the second sample image and the updated third sample image to obtain a pre-trained first key point recognition model.

[0227] The training method of the neural network model provided by the embodiments of the present application can remove the image region outside the third bounding box in the third sample image when pre-training the initial first key point recognition model to obtain an updated third sample image, and then pre-train the initial first key point recognition model based on the first image set, the second sample image and the updated third sample image to obtain a pre-trained first key point recognition model, which can reduce the amount of data for pre-training the initial first key point recognition model, thereby reducing the training difficulty of the initial first key point recognition model.

[0228] S408, moving the bounding box in the first sample image in parallel according to the second parameter to obtain a fourth sample image.

[0229] The fourth sample image includes a target object and a fourth bounding box, and the second parameter includes a third sub-parameter and a fourth sub-parameter. The third sub-parameter is used to indicate the ratio of the number of fourth sample images obtained by parallel movement to the number of first sample images in the first image set, and the fourth sub-parameter is used to indicate the ratio of the second distance to the width or height of the first bounding box. The second distance refers to the distance of moving the first bounding box to the fourth bounding box.

[0230] Optionally, the first sub-parameter is greater than the third sub-parameter, and the second sub-parameter is greater than the fourth sub-parameter.

[0231] It should be understood that the first sub-parameter indicates the ratio of the number of third sample images obtained through parallel movement to the number of first sample images in the first image set, and the third sub-parameter indicates the ratio of the number of fourth sample images obtained through parallel movement to the number of first sample images in the first image set. Since the third sample images obtained by moving the first bounding box are used to simulate a hand moving out of the camera's field of view, the first sub-parameter can also be considered as the data augmentation rate, that is, the proportion of images simulating a hand moving out of the camera's field of view. Similarly, the third sub-parameter, indicating the ratio of the number of fourth sample images obtained through parallel movement to the number of first sample images in the first image set, can also be considered as the data augmentation rate, that is, the proportion of images simulating a hand moving out of the camera's field of view. The first sub-parameter being greater than the third sub-parameter can mean that the ratio of the number of third sample images to the number of first sample images is greater than the ratio of the number of fourth sample images to the number of first sample images.

[0232] The second sub-parameter indicates the ratio of the first distance to the width or height of the first bounding box, where the first distance refers to the distance the first bounding box moves to the third bounding box. The fourth sub-parameter indicates the ratio of the second distance to the width or height of the first bounding box, where the second distance refers to the distance the first bounding box moves to the fourth bounding box. The third sample image is obtained by moving the first bounding box in the first sample image parallel to the X-axis or parallel to the Y-axis. Similarly, the fourth sample image is obtained by moving the first bounding box in the first sample image parallel to the X-axis or parallel to the Y-axis. Therefore, the second sub-parameter being greater than the fourth sub-parameter can mean that the moving distance of the first bounding box corresponding to the third sample image is greater than the moving distance of the first bounding box corresponding to the fourth sample image. In other words, the coordinate difference between the third and first bounding boxes in the third sample image is greater than the coordinate difference between the fourth and first bounding boxes in the fourth sample image.

[0233] For example, Figure 14 (a) is a schematic diagram showing the first bounding box 22 and the third bounding box 32 placed in the same image. Figure 15 (b) is a schematic diagram showing the first bounding box 22 and the fourth bounding box 42 placed in the same image. The second sub-parameter being greater than the fourth sub-parameter can mean that the distance between the left vertex A3 of the third bounding box and the left vertex A1 of the first bounding box is greater than the distance between the left vertex A4 of the fourth bounding box and the left vertex A1 of the first bounding box. This is equivalent to the third bounding box in the third sample image moving a greater distance than the fourth bounding box in the fourth sample image.

[0234] The method for training a neural network model provided in the embodiments of the present application is characterized in that, when pre-training the initial first key point recognition model, the data enhancement rate (the first sub-parameter) used is greater than the data enhancement rate (the third sub-parameter) used when continuing to train the pre-trained first key point recognition model, and the moving distance of the third bounding box in the third sample image used when pre-training the initial first key point recognition model is greater than the moving distance of the fourth bounding box in the fourth sample image used when continuing to train the pre-trained first key point recognition model. This is equivalent to first pre-training the initial first key point recognition model using the third sample image in which the image region outside the bounding box is shielded until the model converges, so that the next step of continuing to train the pre-trained first key point recognition model using the fourth sample image including the image region outside the bounding box, so that the trained first key point recognition model has improved prediction ability for the image region outside the bounding box and improved stability.

[0235] S409, training the pre-trained first key point recognition model based on the first image set, the second sample image and the fourth sample image to obtain a trained first key point recognition model.

[0236] It should be noted that the first sample image is a three-dimensional coordinate image, that is, the first sample image includes depth information, and correspondingly, the fourth sample image is also a three-dimensional coordinate image and includes depth information.

[0237] The method for training a neural network model provided in the embodiments of the present application is characterized in that, by obtaining a third image set including a fifth sample image collected by a camera device, and then obtaining a sixth sample image based on the fifth sample image and a target detection model, the sixth sample image includes a first bounding box, and then performing three-dimensional conversion on the sixth sample image based on the depth information corresponding to the fifth sample image to obtain a first sample image, which is equivalent to that the first sample image is a three-dimensional coordinate image, that is, the first sample image includes depth information of each pixel point, and then the second sample image, the third sample image and the fourth sample image obtained based on the first sample image also include depth information of each pixel point. Therefore, in the process of training the initial first key point model, the first sample image, the second sample image, the third sample image and the fourth sample image used are all three-dimensional images and include depth information, so that the coordinates of the hand key points obtained based on the trained first key point recognition model are three-dimensional coordinates including depth information, and further, the gesture recognition based on the three-dimensional hand key point coordinates is more accurate, especially when recognizing the gesture perpendicular to the screen of the electronic device, the obtained gesture recognition result is more accurate.

[0238] It can be understood that in the recognition process of some special gestures, the coordinate positions of part of the hand key points change greatly, and the positions of the rest of the hand key points change less. For example, the gesture of pinching the thumb and the index finger changes greatly for the two hand key points of the thumb tip (4) and the index finger tip (8). If the neural network model is trained based on all hand key points, it will occupy a large amount of resources of the electronic device, causing unnecessary waste of resources. Therefore, the key point recognition model can be trained for part of the hand key points. The following will be described in detail. Figure 15

[0239] Figure 15 Another flowchart of a training method of a neural network model provided by an embodiment of the present application is shown in Figure 16 The method comprises the following steps.

[0240] S501, a first image set is obtained, the first image set comprising a plurality of first sample images, the first sample images comprising a target object, and a first bounding box obtained by detecting the target object.

[0241] S502, the bounding box in the first sample image is randomly moved to obtain a second sample image, the second sample image comprising the target object and a second bounding box obtained by randomly moving the first bounding box.

[0242] S503, the first bounding box in the first sample image is moved in parallel to obtain a third sample image.

[0243] The third sample image comprises the target object and a third bounding box obtained by moving the first bounding box in parallel, the third bounding box having the same coordinates as the first bounding box on the X axis and different coordinates on the Y axis, or the third bounding box having the same coordinates as the first bounding box on the Y axis and different coordinates on the X axis.

[0244] S504, an initial first key point recognition model is trained based on the first image set, the second sample image and the third sample image to obtain a trained first key point recognition model, the first key point recognition model being used to recognize key points in the target object.

[0245] S505, a fourth image set is obtained, the fourth image set comprising a seventh sample image, the seventh sample image being an image obtained for a preset gesture of the target object.

[0246] Taking the recognition of the gesture of pinching the thumb and the index finger as an example, the seventh sample image can be a sample image annotated for the two hand key points of the thumb tip (4) and the index finger tip (8). Compared with the first sample image, the number of hand key points in the seventh sample image is less than the number of hand key points in the first sample image.

[0247] ​S506. The initial second key point recognition model is trained based on the fourth image set to obtain the trained second key point recognition model. The second key point recognition model is used to recognize the key points corresponding to the preset gesture.

[0248] The second keypoint recognition model can be a neural network model with the same structure as the first keypoint recognition model. The electronic device can use a fourth image set, namely multiple seventh sample images labeled for a preset gesture, to train the initial second keypoint recognition model, resulting in a trained second keypoint recognition model. Since the second keypoint recognition model can be a neural network model with the same structure as the first keypoint recognition model, the trained second keypoint recognition model has the same structure as the trained first keypoint recognition model, but different parameters.

[0249] In one possible scenario, the first keypoint recognition model and the second keypoint recognition model reuse part of the network structure.

[0250] Optionally, the first keypoint recognition model includes a first network and a first regression layer, and the second keypoint recognition model includes a first network and a second regression layer, wherein the dimension of the first regression layer is higher than that of the second regression layer.

[0251] For example, such as Figure 17 As shown, the first keypoint recognition model and the second keypoint recognition model reuse the first network. Since the number of typical hand keypoints is 21, to recognize these 21 hand keypoints, the first regression layer in the first keypoint recognition model can be a regression layer with an output dimension of 3*21, which is a regression layer with an output dimension of 63. When recognizing the pinch gesture, the focus is on recognizing the two hand keypoints: the tip of the thumb (4) and the tip of the index finger (8). Therefore, a regression layer with an input dimension of n*3 can be used, where n is the number of hand keypoints that need to be reinforced. That is to say, the second regression layer can be a regression layer with an output dimension of 2*3, which is a regression layer with an output dimension of 6.

[0252] After training the initial first keypoint recognition model using the first and second image sets to obtain the trained first keypoint recognition model, the electronic device can freeze the parameters of the first network and use the fourth image set to train the second keypoint recognition model composed of the first network and the second regression layer until the model converges.

[0253] The training method of the neural network model provided in the embodiments of the present application, the first key point recognition model comprises a first network and a first regression layer, the second key point recognition model comprises a first network and a second regression layer, the dimension of the first regression layer is higher than the dimension of the second regression layer, and the second regression layer with a lower dimension is used to identify the hand key points corresponding to the preset gesture. This is only to train the small number of hand key points corresponding to the preset gesture, so as to strengthen the training of the hand key points corresponding to the preset gesture and improve the identification ability of the hand key points corresponding to the preset gesture under the premise of avoiding the influence on other hand key points.

[0254] S507, based on the parameters in the trained second key point recognition model, the parameters in the trained first key point recognition model are corrected to obtain an updated first key point recognition model.

[0255] As described above for the neural network, the output of the operation unit of the neural unit in the neural network can be as shown in formula (1), W s is the weight of the neural unit input x s , and b is the bias of the neural unit. The weight W' of the neural unit input x in the trained second key point recognition model can be represented by W' = W1*W2*…*W i , and the bias b' of the neural unit can be represented by b' = f(W i , b i-1 , b i ), wherein f(W i , b i-1 , b i ) = W i *b i-1 +b i .

[0256] The weight W0 of the neural unit input x and the bias b0 of the neural unit in the trained first key point recognition model. The weight W' of the neural unit input x and the bias b' of the neural unit in the trained second key point recognition model are used to correct the trained first key point recognition model, and an updated first key point recognition model can be obtained. The weight W of the neural unit input x in the updated first key point recognition model is W0+W', and the bias b of the neural unit is b0+b'.

[0257] The method for training the neural network model provided in the embodiments of the present application trains the trained second key point recognition model corresponding to the special gesture, and then corrects the parameters of the trained first key point recognition model by using the parameters of the trained second key point recognition model, to obtain the updated first key point recognition model, so that the updated first key point recognition model can also accurately recognize the hand key points corresponding to the special gesture, thereby improving the robustness of the updated first key point recognition model.

[0258] Figure 17 The flowchart of another method for training a neural network model provided in the embodiments of the present application is shown in FIG. 6, and the method comprises the following steps. Figure 7 to Figure 16

[0259] S601, a third image set is obtained, and the third image set comprises a fifth sample image, the fifth sample image being an image collected by a camera device.

[0260] S602, a sixth sample image is obtained based on the fifth sample image and a target detection model, and the sixth sample image comprises a first bounding box.

[0261] S603, the sixth sample image is converted into three dimensions based on the depth information corresponding to the fifth sample image, to obtain a first sample image.

[0262] Optionally, the sixth sample image is converted into three-dimensional coordinates based on the first formula and the depth information corresponding to the fifth sample image, to obtain the first sample image, and the first formula comprises:

[0263] x = X / W;

[0264] y = Y / H;

[0265] z = (Z-Z0) / W;

[0266] wherein x is the coordinate of a pixel point on the x-axis in the first sample image, y is the coordinate of the pixel point on the y-axis in the first sample image, z is the coordinate of the pixel point on the z-axis in the first sample image, X is the coordinate of the pixel point corresponding to x on the x-axis in the sixth sample image, Y is the coordinate of the pixel point corresponding to y on the y-axis in the sixth sample image, Z is the coordinate of the pixel point corresponding to z on the z-axis in the sixth sample image, Z0 represents the coordinate of the target key point on the z-axis in the sixth sample image, W represents the width of the first bounding box, and H represents the height of the first bounding box.

[0267] S604, a first image set is obtained, and the first image set comprises a plurality of first sample images, the first sample image comprises a target object and a first bounding box obtained by detecting the target object.

[0268] ​S605, randomly move the bounding box in the first sample image to obtain a second sample image.

[0269] The second sample image includes the target object and a second bounding box obtained by randomly moving the first bounding box.

[0270] S606, move the bounding box in the first sample image in parallel according to a first parameter to obtain a third sample image.

[0271] The first parameter includes a first sub-parameter and a second sub-parameter. The first sub-parameter is used to indicate a ratio of a number of the third sample images obtained by parallel movement to a number of the first sample images in the first image set. The second sub-parameter is used to indicate a ratio of the first distance to a width or a height of the first bounding box. The first distance refers to a distance of the first bounding box moving to a third bounding box.

[0272] S607, pre-train the initial first key point recognition model based on the first image set, the second sample image and the third sample image to obtain a pre-trained first key point recognition model.

[0273] S608, move the bounding box in the first sample image in parallel according to a second parameter to obtain a fourth sample image.

[0274] The fourth sample image includes the target object and a fourth bounding box. The second parameter includes a third sub-parameter and a fourth sub-parameter. The third sub-parameter is used to indicate a ratio of a number of the fourth sample images obtained by parallel movement to a number of the first sample images in the first image set. The fourth sub-parameter is used to indicate a ratio of the second distance to a width or a height of the first bounding box. The second distance refers to a distance of the first bounding box moving to the fourth bounding box.

[0275] S609, train the pre-trained first key point recognition model based on the first image set, the second sample image and the fourth sample image to obtain a trained first key point recognition model.

[0276] S610, obtain a fourth image set. The fourth image set includes a seventh sample image. The seventh sample image is an image obtained for a preset gesture of the target object.

[0277] S611, train an initial second key point recognition model based on the fourth image set to obtain a trained second key point recognition model. The second key point recognition model is used to recognize key points corresponding to the preset gesture.

[0278] S612, correct parameters in the trained first key point recognition model based on parameters in the trained second key point recognition model to obtain an updated first key point recognition model.

[0279] The implementation principle and beneficial effects of the method for training a neural network model provided in the embodiments of the present application are similar to those of the above Figure 18 The implementation principle and beneficial effects of the method for training a neural network model provided in the embodiments of the present application are similar to those of the above

[0280] It should be understood that, although each step in the flowchart in the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0281] It can be understood that, in order to implement the above functions, the electronic device contains hardware and / or software modules corresponding to each function. The algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of the present application.

[0282] The embodiments of the present application can divide the functional modules of the electronic device according to the above method examples, for example, each functional module can be divided according to each function, or two or more functions can be integrated into one module. It should be noted that the division of modules in the embodiments of the present application is illustrative, and is only a logical functional division. In actual implementation, there can be another division method. It should be noted that the names of the modules in the embodiments of the present application are illustrative, and the names of the modules are not limited in actual implementation.

[0283] Figure 7 to Figure 17 A structural schematic diagram of the neural network model training apparatus provided in the embodiments of the present application.

[0284] It should be understood that the neural network model training apparatus 600 can execute Figure 19 the neural network model training method shown in the above embodiments; the neural network model training apparatus 600 comprises an acquisition unit 610 and a processing unit 620.

[0285] The acquisition unit 610 is configured to acquire a first image set, the first image set including a plurality of first sample images, the first sample images including a target object, and a first bounding box obtained by detecting the target object;

[0286] The processing unit 620 is configured to move the first bounding box of the first sample image in the first image set to obtain a second image set.

[0287] The processing unit 620 is configured to train an initial first key point recognition model based on the first image set and the second image set to obtain a trained first key point recognition model, the first key point recognition model being used to recognize a key point in the target object.

[0288] In one embodiment, the second image set includes a second sample image and a third sample image, and the processing unit 620 is specifically configured to randomly move the first bounding box in the first sample image to obtain the second sample image, the second sample image including the target object and a second bounding box obtained by randomly moving the first bounding box; and move the first bounding box in the first sample image in parallel to obtain the third sample image, the third sample image including the target object and a third bounding box obtained by moving the first bounding box in parallel, the third bounding box being obtained by moving the first bounding box in parallel along an X axis, or the third bounding box being obtained by moving the first bounding box in parallel along a Y axis.

[0289] In one embodiment, the processing unit 620 is specifically configured to move the first bounding box in the first sample image in parallel according to a first parameter to obtain a third sample image, the first parameter including a first sub-parameter and a second sub-parameter, the first sub-parameter being used to indicate a ratio of a number of the third sample images obtained by moving in parallel to a number of the first sample images in the first image set, and the second sub-parameter being used to indicate a ratio of a first distance to a width or a height of the first bounding box, the first distance being a distance of the first bounding box moving to the third bounding box; pre-train the initial first key point recognition model based on the first image set, the second sample image and the third sample image to obtain a pre-trained first key point recognition model; move the bounding box in the first sample image in parallel according to a second parameter to obtain a fourth sample image, the fourth sample image including the target object and a fourth bounding box, the second parameter including a third sub-parameter and a fourth sub-parameter, the third sub-parameter being used to indicate a ratio of a number of the fourth sample images obtained by moving in parallel to a number of the first sample images in the first image set, and the fourth sub-parameter being used to indicate a ratio of a second distance to a width or a height of the first bounding box, the second distance being a distance of the first bounding box moving to the fourth bounding box; and train the pre-trained first key point recognition model based on the first image set, the second sample image and the fourth sample image to obtain the trained first key point recognition model.

[0290] In one embodiment, the first sub-parameter is greater than the third sub-parameter, and the second sub-parameter is greater than the fourth sub-parameter.

[0291] In one embodiment, the processing unit 620 is specifically configured to remove an image region outside the third bounding box in the third sample image to obtain an updated third sample image; and pre-train the initial first key point recognition model based on the first image set, the second sample image, and the updated third sample image to obtain the pre-trained first key point recognition model.

[0292] In one embodiment, the acquisition unit 610 is further configured to acquire a third image set, the third image set including a fifth sample image, the fifth sample image being an image collected by the camera device; and the processing unit 620 is further configured to obtain the first sample image based on the fifth sample image, depth information corresponding to the fifth sample image, and a target detection model, the target detection model being used to label the first bounding box of the target object.

[0293] In one embodiment, the processing unit 620 is specifically configured to obtain a sixth sample image based on the fifth sample image and the target detection model, the sixth sample image including the first bounding box; and perform three-dimensional conversion on the sixth sample image based on the depth information corresponding to the fifth sample image to obtain the first sample image.

[0294] In one embodiment, the processing unit 620 is specifically configured to perform three-dimensional coordinate conversion on the sixth sample image based on the first formula and the depth information corresponding to the fifth sample image to obtain the first sample image, the first formula including:

[0295] x = X / W;

[0296] y = Y / H;

[0297] z = (Z-Z0) / W;

[0298] wherein x is a coordinate of a pixel point in the first sample image on an x-axis, y is a coordinate of the pixel point in the first sample image on a y-axis, z is a coordinate of the pixel point in the first sample image on a z-axis, X is a coordinate of a pixel corresponding to x on an x-axis in the sixth sample image, Y is a coordinate of a pixel corresponding to y on a y-axis in the sixth sample image, Z is a coordinate of a pixel corresponding to z on a z-axis in the sixth sample image, Z0 represents a coordinate of the target key point on the z-axis in the sixth sample image, W represents a width of the first bounding box, and H represents a height of the first bounding box.

[0299] In one embodiment, the target object includes a hand in the image.

[0300] In one embodiment, the first key point recognition model is used to identify a hand key point in the image.

[0301] In an embodiment, the acquisition unit 610 is further configured to acquire a fourth image set, the fourth image set comprising a seventh sample image, the seventh sample image being an image obtained for a preset gesture of the target object; the processing unit 620 is further configured to train the initial second key point recognition model based on the fourth image set to obtain a trained second key point recognition model, the second key point recognition model being used to identify a key point corresponding to the preset gesture; and correct parameters in the trained first key point recognition model based on parameters in the trained second key point recognition model to obtain an updated first key point recognition model.

[0302] In an embodiment, the first key point recognition model comprises a first network and a first regression layer, and the second key point recognition model comprises the first network and a second regression layer, a dimension of the first regression layer being higher than a dimension of the second regression layer.

[0303] The training apparatus of the neural network model provided in this embodiment is used to perform the training method of the neural network model in the above-described embodiments, and has similar technical principles and technical effects, which are not described here again.

[0304] It should be noted that the training apparatus 600 of the neural network model is embodied in the form of functional units. The term "unit" herein can be implemented in the form of software and / or hardware, which is not specifically limited.

[0305] For example, the "unit" can be a software program, a hardware circuit, or a combination of both, which implements the above-described functions. The hardware circuit can include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, and a combination of logic circuit and / or other suitable components supporting the described functions.

[0306] Therefore, the units of each example described in the embodiments of the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0307] Figure 19 A structural schematic diagram of an electronic device provided by the present application is shown. ​ The dashed line in the electronic device 700 indicates that the unit or the module is optional. The electronic device 700 can be used to implement the training method of the neural network model described in the above method embodiments.

[0308] The electronic device 700 includes one or more processors 701 that can support the electronic device 700 to implement the training method of the neural network model in the method embodiments. The processor 701 can be a general-purpose processor or a special-purpose processor. For example, the processor 701 can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices such as discrete gates or transistor logic devices, or discrete hardware components.

[0309] The processor 701 can be configured to control the electronic device 700, execute a software program, and process data of the software program. The electronic device 700 can further include a communication unit 705 configured to implement input (reception) and output (transmission) of signals.

[0310] For example, the electronic device 700 can be a chip, the communication unit 705 can be an input and / or output circuit of the chip, or the communication unit 705 can be a communication interface of the chip, and the chip can be a component of a terminal device or other electronic device.

[0311] For another example, the electronic device 700 can be a terminal device, and the communication unit 705 can be a transceiver of the terminal device, or the communication unit 705 can be a transceiving circuit of the terminal device.

[0312] The electronic device 700 can include one or more memories 702 having programs 704 stored thereon, and the programs 704 can be executed by the processor 701 to generate instructions 703, so that the processor 701 performs the impedance matching method described in the above method embodiments according to the instructions 703.

[0313] Optionally, the memory 702 can further store data. Optionally, the processor 701 can further read the data stored in the memory 702, and the data can be stored in the same storage address as the program 704, or the data can be stored in a different storage address from the program 704.

[0314] The processor 701 and the memory 702 can be separately arranged or integrated together, for example, integrated on a system on chip (SOC) of the terminal device.

[0315] Exemplarily, the memory 702 can be configured to store a program 704 related to the method for training a neural network model provided in the embodiments of the present application, and the processor 701 can be configured to invoke the program 704 related to the method for training a neural network model stored in the memory 702 when training the neural network model, and execute the method for training a neural network model in the embodiments of the present application, which comprises: obtaining a first image set, the first image set comprising a plurality of first sample images, the first sample images comprising a target object and a first bounding box obtained by detecting the target object; moving the first bounding box of the first sample images in the first image set to obtain a second image set; training an initial first key point recognition model based on the first image set and the second image set to obtain a trained first key point recognition model, the first key point recognition model being configured to recognize key points in the target object.

[0316] Exemplarily, the memory 702 can be configured to store a program 704 related to the method for training a neural network model provided in the embodiments of the present application, and the processor 701 can be configured to invoke the program 704 related to the method for training a neural network model stored in the memory 702 when training the neural network model, and execute the method for training a neural network model in the embodiments of the present application, which comprises: training an initial first key point recognition model based on the first image set and the second image set to obtain a trained first key point recognition model; obtaining a fourth image set, the fourth image set comprising a seventh sample image, the seventh sample image being an image obtained by a preset gesture of the target object; training an initial second key point recognition model based on the fourth image set to obtain a trained second key point recognition model, the second key point recognition model being configured to recognize key points corresponding to the preset gesture; correcting parameters in the trained first key point recognition model based on parameters in the trained second key point recognition model to obtain an updated first key point recognition model, wherein the first image set comprises a plurality of first sample images, the first sample images comprising the target object and first bounding boxes obtained by detecting the target object; and the sample images in the second image set are obtained by moving the first bounding boxes of the first sample images in the first image set.

[0317] The present application also provides a computer program product, which, when executed by the processor 701, implements the method for training a neural network model described in any of the method embodiments of the present application.

[0318] The computer program product can be stored in the memory 702, for example, the program 704, which is finally converted into an executable object file capable of being executed by the processor 701 through preprocessing, compiling, assembling and linking and other processing processes.

[0319] The application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a computer to implement the training method of the neural network model described in any method embodiment of the application. The computer program can be a high-level language program or an executable target program.

[0320] The computer readable storage medium is, for example, the memory 702. The memory 702 can be a volatile memory or a non-volatile memory, or the memory 702 can include both volatile memory and non-volatile memory. The non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM can be used, such as a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous dynamic RAM (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a synchlink DRAM (SLDRAM), and a direct rambus RAM (DR RAM).

[0321] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of the items, including any combination of single item or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0322] It should be understood that the size of the sequence number of the above-mentioned processes in various embodiments of the present application does not mean the order of execution, and the execution order of the processes should be determined according to their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0323] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0324] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0325] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic; for example, the division of the units is only a logical function division, and actual implementation can have another division manner; for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0326] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0327] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0328] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for training a neural network model, characterized in that, The method includes: Acquire a third image set, the third image set including a fifth sample image, the fifth sample image being an image captured by a camera device; A first sample image is obtained based on a fifth sample image, the depth information corresponding to the fifth sample image, and a target detection model. The first sample image includes a target object and a first bounding box obtained by detecting the target object. The target detection model is used to annotate the first bounding box of the target object. Obtain a first image set, which includes multiple first sample images; Move the first bounding box of the first sample image in the first image set to obtain the second image set; The initial first key point recognition model is trained based on the first image set and the second image set to obtain the trained first key point recognition model. The first key point recognition model is used to recognize hand key points in the image. The hand key points are obtained through the trained first key point recognition model; Based on the key hand points, a gesture recognition result is obtained, which includes at least one of the following: swipe up, swipe down, swipe left, swipe right, grab screen, and pinch. The second image set includes a second sample image, a third sample image, and a fourth sample image. The second sample image is a sample image obtained by randomly moving the first bounding box in the first sample image. The second sample image includes the target object and a second bounding box obtained by randomly moving the first bounding box. Randomly moving the first bounding box is used to simulate a hand tremor scenario. The third sample image is a sample image obtained by parallel moving the first bounding box in the first sample image according to a first parameter. The third sample image includes the target object and a third bounding box obtained by parallel moving the first bounding box according to the first parameter. Parallel moving the first bounding box is used to simulate a hand moving out of the camera's field of view. The fourth sample image is a sample image obtained by parallel moving the first bounding box in the first sample image according to a second parameter. The fourth sample image includes the target object and a fourth bounding box obtained by parallel moving the first bounding box according to the second parameter. The first parameter is greater than the second parameter. The step of training the initial first keypoint recognition model based on the first image set and the second image set to obtain the trained first keypoint recognition model includes: First, the initial first keypoint recognition model is pre-trained based on the first image set, the second sample image, and the third sample image to obtain the pre-trained first keypoint recognition model. Then, the pre-trained first keypoint recognition model is trained based on the first image set, the second sample image, and the fourth sample image to obtain the trained first keypoint recognition model.

2. The method according to claim 1, characterized in that, The third bounding box is obtained by moving the first bounding box parallel to the X-axis, or the third bounding box is obtained by moving the first bounding box parallel to the Y-axis.

3. The method according to claim 2, characterized in that, The first parameter includes a first sub-parameter and a second sub-parameter. The first sub-parameter indicates the ratio of the number of third sample images obtained by the parallel movement to the number of first sample images in the first image set. The second sub-parameter indicates the ratio of a first distance to the width or height of the first bounding box. The first distance refers to the distance the first bounding box moves to the third bounding box. The second parameter includes a third sub-parameter and a fourth sub-parameter. The third sub-parameter indicates the ratio of the number of fourth sample images obtained by the parallel movement to the number of first sample images in the first image set. The fourth sub-parameter indicates the ratio of a second distance to the width or height of the first bounding box. The second distance refers to the distance the first bounding box moves to the fourth bounding box.

4. The method according to claim 3, characterized in that, The first sub-parameter is greater than the third sub-parameter, and the second sub-parameter is greater than the fourth sub-parameter.

5. The method according to any one of claims 1 to 4, characterized in that, The initial first keypoint recognition model is pre-trained based on the first image set, the second sample image, and the third sample image to obtain a pre-trained first keypoint recognition model, including: Remove the image region outside the third bounding box in the third sample image to obtain the updated third sample image; The initial first keypoint recognition model is pre-trained based on the first image set, the second sample image, and the updated third sample image to obtain the pre-trained first keypoint recognition model.

6. The method according to any one of claims 1 to 5, characterized in that, The first sample image is obtained based on the fifth sample image, the depth information corresponding to the fifth sample image, and the target detection model, including: A sixth sample image is obtained based on the fifth sample image and the target detection model, wherein the sixth sample image includes the first bounding box; The sixth sample image is transformed into three dimensions based on the depth information corresponding to the fifth sample image to obtain the first sample image.

7. The method according to claim 6, characterized in that, The step of performing a 3D transformation on the sixth sample image based on the depth information corresponding to the fifth sample image to obtain the first sample image includes: The first sample image is obtained by performing a three-dimensional coordinate transformation on the sixth sample image using the first formula and the depth information corresponding to the fifth sample image. The first formula includes: x = X / W; y = Y / H; z = (Z - Z0) / W; Where x is the x-coordinate of the pixel in the first sample image, y is the y-coordinate of the pixel in the first sample image, z is the z-coordinate of the pixel in the first sample image, X is the x-coordinate of the pixel corresponding to x in the sixth sample image, Y is the y-coordinate of the pixel corresponding to y in the sixth sample image, Z is the z-coordinate of the pixel corresponding to z in the sixth sample image, Z0 represents the z-coordinate of the target key point in the sixth sample image, W represents the width of the first bounding box, and H represents the height of the first bounding box.

8. The method according to any one of claims 1 to 7, characterized in that, The target object includes the hand in the image.

9. The method according to any one of claims 1 to 8, characterized in that, After training the initial first keypoint recognition model based on the first image set and the second image set to obtain the trained first keypoint recognition model, the method further includes: A fourth image set is obtained, the fourth image set including a seventh sample image, the seventh sample image being an image obtained for a preset gesture of the target object; The initial second key point recognition model is trained based on the fourth image set to obtain the trained second key point recognition model. The second key point recognition model is used to recognize the key points corresponding to the preset gesture. Based on the parameters in the trained second keypoint recognition model, the parameters in the trained first keypoint recognition model are corrected to obtain the updated first keypoint recognition model.

10. The method according to claim 9, characterized in that, The first keypoint recognition model includes a first network and a first regression layer, and the second keypoint recognition model includes the first network and a second regression layer, wherein the dimension of the first regression layer is higher than that of the second regression layer.

11. An electronic device, characterized in that, The electronic device includes a module for performing the method as described in any one of claims 1 to 10.

12. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 10.

13. A chip system, characterized in that, The chip system includes a processor for calling and running a computer program from memory, causing an electronic device on which the chip system is installed to perform the method as described in any one of claims 1 to 10.

14. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1 to 10.

Images