Training method and training device

Abstract

The present application provides a training method and a training device. The method includes: obtaining a training sample, wherein the training sample is an image sequence recording the movement of a rodent; inputting the training sample into a neural network model to obtain a recognition result of the posture of the rodent, wherein the recognition result includes key points in the posture of the rodent; and according to the recognition result of the posture of the rodent, using a loss function, the neural network model is trained using a gradient descent method. The loss function includes a time constraint term, and the time constraint term is used to constrain the position of the key points in the posture of the rodent between adjacent image frames in the image sequence. The neural network model trained according to the above method can effectively avoid the jitter phenomenon of the recognition result in the time domain when performing posture recognition of rodents such as mice.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method and a training device. Background Art

[0002] Posture recognition refers to the use of neural network models to identify and / or extract the movements and / or key points of organisms in images or videos.

[0003] In the prior art, when identifying key points in an image sequence of a moving rodent, key points are usually extracted from a single image or a single image frame, and the key points may be, for example, joints of the rodent, etc. The recognition result obtained in this way is prone to jitter, making the movement trajectory of the key points not smooth enough. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a training method and a training device to improve the accuracy and recognition efficiency of a neural network model when performing rodent posture recognition, and ensure the continuity of the recognition results in the time domain.

[0005] In a first aspect, a training method is provided, the method comprising: obtaining a training sample, the training sample being an image sequence recording the movement of a rodent; inputting the training sample into a neural network model to obtain a recognition result of the posture of the rodent, the recognition result comprising key points in the posture of the rodent; and training the neural network model using a gradient descent method using a loss function according to the recognition result of the posture of the rodent. The loss function comprises a time constraint term, the time constraint term being used to constrain the position of the key points in the posture of the rodent between adjacent image frames in the image sequence.

[0006] Optionally, the neural network model includes a HRNet network.

[0007] Optionally, before training the neural network model, the training method further comprises: determining the time constraint item according to an error between a position of a key point obtained by a tracking method and a position of the key point in the recognition result.

[0008] Optionally, the tracking method includes a Lucas-Kanade optical flow method.

[0009] Optionally, the determining the time constraint item according to the error between the position of the key point obtained by the tracking method and the position of the key point in the recognition result includes: selecting m samples from the training samples as tracking samples, where m is a positive integer greater than or equal to 2; The first frame in As the initial frame, forward tracking is performed to obtain the forward tracking result Determine the difference between the forward tracking result and the recognition result of the mth image frame in the tracking sample as: Where ω is the number of key points on each image frame; the last frame in the recognition result of the tracking sample As the termination frame, perform backward tracking and obtain the backward tracking result Determine that the difference between the backward tracking result and the recognition result of the first image frame in the tracking sample is The first frame in the recognition result of the tracking sample As the initial frame Perform forward tracking and obtain forward tracking results Based on the forward tracking results As the initial frame, perform backward tracking and obtain the second backward tracking result The second backward tracking result and the initial frame The difference is Determine the time constraints as: Where E1 is the threshold.

[0010] Optionally, the loss function further includes a spatial constraint term, which is used to limit the positions of key points in the posture of the rodent in the same frame image.

[0011] Optionally, before training the neural network model, the training method further comprises: determining the spatial constraint item according to the difference between the positions of multiple key points in the recognition result.

[0012] Optionally, determining the spatial constraint item according to the difference between the positions of the multiple key points in the recognition result includes: selecting p samples from the training samples, where p is a positive integer greater than or equal to 2; determining the recognition results of the p samples The distance between two key points in The distance According to the Gaussian distribution, the distance The mean μ and variance σ of 2 , where ω is the number of key points on each image frame; the spatial constraint term is determined as:

[0013] Optionally, the loss function further includes an error constraint term, and the error constraint term is used to constrain the error of key points in the posture of the rodent in the recognition result and the labeling result.

[0014] Optionally, the error constraint term is a mean square error loss term.

[0015] In a second aspect, a training device is provided, comprising: an acquisition module for acquiring training samples, wherein the training samples are image sequences recording the movement of rodents; an input module for inputting the training samples into a neural network model to obtain a recognition result of the posture of the rodent, wherein the recognition result includes key points in the posture of the rodent; and a training module for training the neural network model using a gradient descent method using a loss function according to the recognition result of the posture of the rodent. The loss function includes a time constraint term, and the time constraint term is used to constrain the position of the key points in the posture of the rodent between adjacent image frames in the image sequence.

[0016] Optionally, the neural network model includes a HRNet network.

[0017] Optionally, before training the neural network model, the training device further includes: a first determination module, configured to determine the time constraint item based on an error between a position of a key point obtained using a tracking method and a position of the key point in the recognition result.

[0018] Optionally, the tracking method includes a Lucas-Kanade optical flow method.

[0019] Optionally, the first determination module is used to: select m samples from the training samples as tracking samples, where m is a positive integer greater than or equal to 2; The first frame in As the initial frame, forward tracking is performed to obtain the forward tracking result Determine the difference between the forward tracking result and the recognition result of the mth image frame in the tracking sample as: Where ω is the number of key points on each image frame; the last frame in the recognition result of the tracking sample As the termination frame, perform backward tracking and obtain the backward tracking result Determine that the difference between the backward tracking result and the recognition result of the first image frame in the tracking sample is The first frame in the recognition result of the tracking sample As the initial frame Perform forward tracking and obtain forward tracking results Based on the forward tracking results As the initial frame, perform backward tracking and obtain the second backward tracking result The second backward tracking result and the initial frame The difference is Determine the time constraints as: Where E1 is the threshold.

[0020] Optionally, the loss function further includes a spatial constraint term, which is used to limit the positions of key points in the posture of the rodent in the same frame image.

[0021] Optionally, the training device further includes: a second determination module, configured to determine the spatial constraint item according to a difference between positions of a plurality of key points in the recognition result.

[0022] Optionally, the second determination module is used to: select p samples from the training samples, where p is a positive integer greater than or equal to 2; and determine the recognition results of the p samples. The distance between two key points in The distance According to the Gaussian distribution, the distance The mean μ and variance σ of 2 , where ω is the number of key points on each image frame; the spatial constraint term is determined as:

[0023] Optionally, the loss function further includes an error constraint term, and the error constraint term is used to constrain the error of key points in the posture of the rodent in the recognition result and the labeling result.

[0024] Optionally, the error constraint term is a mean square error loss term.

[0025] The present application introduces time constraints in the training process of the neural network model, so that the neural network model can have a higher accuracy when processing occluded and blurred images, and can effectively suppress the jitter phenomenon of the recognition results of the neural network model in the time domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic flowchart of a training method provided in one embodiment of the present application.

[0027] Figure 2 A schematic flowchart of a method for determining a time constraint item provided in an embodiment of the present application.

[0028] Figure 3 A schematic flowchart of a method for determining a spatial constraint item provided in one embodiment of the present application.

[0029] Figure 4 A schematic flowchart of a method for determining an error constraint item provided in an embodiment of the present application.

[0030] Figure 5 A schematic block diagram of a training device provided in one embodiment of the present application.

[0031] Figure 6 A schematic block diagram of a training device provided in another embodiment of the present application.

[0032] Figure 7 A schematic block diagram of an application scenario of an embodiment of the present application. DETAILED DESCRIPTION

[0033] The method and apparatus in the embodiments of the present application can be applied to various scenarios of posture recognition of rodents based on image sequences. The image sequence can be multiple image frames in a video. The multiple image frames can be multiple continuous image frames in a video. The image sequence can also be multiple images of animals captured by an image acquisition device such as a camera. The rodent can be, for example, a mouse.

[0034] In order to facilitate understanding of the embodiments of the present application, the background of the present application is firstly described in detail with examples.

[0035] The behavior of biological neurons is closely related to the activities of animals. Changes in the posture of animals usually cause corresponding changes in neurons. Therefore, the exploration of the connection and interaction of complex networks composed of neurons under specific behaviors is very important in the fields of neuroscience and medicine. This field generally adopts quantitative analysis methods, that is, by obtaining the posture information of animals and the behavior of neurons, the corresponding relationship is determined.

[0036] The behavior of animal neurons can be acquired using methods such as X-ray scanning and miniaturized multiphoton microscopy.

[0037] There are many ways to obtain the posture information of animals. For example, the posture information of animals can be obtained by manually marking the key points in the image sequence. However, in the face of massive amounts of data, manual processing is inefficient and prone to errors, and the accuracy of the posture information obtained cannot be guaranteed.

[0038] For another example, markers (such as displacement or acceleration sensors) can be set at key points on the animal's body to determine the animal's posture changes based on changes in information such as the marker's position. However, for rodents, due to their small size, setting markers will interfere with their natural behavior, resulting in reduced accuracy of the collected data.

[0039] For another example, a depth camera can be used to locate an animal in space to obtain its posture information. However, this method is sensitive to imaging conditions and scene changes and is not suitable for all occasions.

[0040] With the development of artificial intelligence, animal posture recognition methods based on neural networks are gradually replacing traditional technologies. Current neural network models usually do not consider the movement of key points of rodents in image sequences over time during training. These neural network models have the following problems in the posture recognition process:

[0041] When identifying the posture of an animal in an image sequence, a neural network model usually performs posture recognition based on each frame of the image itself. For example, the image sequence to be identified includes a first frame of image and a second frame of image in chronological order. The neural network model identifies the posture of the animal in the first frame of image based on the image of the first frame, and obtains a first posture recognition result corresponding to the first frame of image. The posture of the animal in the second frame of image is identified based on the image of the second frame, and obtains a second posture recognition result corresponding to the second frame of image. The above method of directly using the current frame image to identify the posture of the animal has a low accuracy rate and is not smooth enough in time. In addition, when the image frames in the acquired image sequence are blurred or blocked, for example, when the tail of a rodent is curled or blocked, the accuracy of the key point position information output by the neural network model is low.

[0042] In addition, existing neural network models usually construct a loss function based on the error between the recognition result and the manual annotation result, and use the back propagation algorithm for training. This kind of neural network model does not consider the continuous change of key points in the time domain during training, which leads to low accuracy when performing rodent posture recognition. On the other hand, using the error between the above recognition result and the manual annotation result to construct a loss function to train the neural network model usually makes the initial training process slower.

[0043] In view of the above problems, the embodiments of the present application provide a training method and a training device. The method provided in the embodiments of the present application effectively suppresses the jitter phenomenon of the recognition result of the neural network model in the time domain by introducing time constraints in the training process of the neural network model.

[0044] Combine the following Figure 1-Figure 4 , the training method provided in the embodiment of the present application is introduced in detail. Figure 1 It is a schematic flow chart of the training method provided in the embodiment of the present application. Figure 1 The training method shown may include steps S11 - S13 .

[0045] Step S11, obtaining training samples.

[0046] In one embodiment of the present application, the training sample may include an image sequence recording the movement of a rodent and a labeling result. It is understood that the labeling result may include position information of a preset number of key points of the rodent's body. For example, the key points may be joints and key parts of the body, such as joints on the limbs of a mouse, as well as a tail, eyes, nose, ears, etc. The position information may be coordinate information of the key points.

[0047] The embodiment of the present application does not limit the method for obtaining the pre-annotated results. For example, a manual annotation method can be used to annotate the image frames in the image sequence frame by frame. As a possible implementation method, other methods with higher confidence can also be used for annotation.

[0048] There are many ways to obtain training samples, and the embodiments of the present application do not limit this. For example, as an implementation method, an image sequence can be directly obtained by, for example, an image acquisition device (such as a camera, a camera, a medical imaging device, a lidar, etc.), and the image sequence may include multiple images of rodents arranged in chronological order. For another example, training samples can be obtained from a server (such as a local server or a cloud server, etc.). Alternatively, training samples can also be obtained on the Internet or other content platforms, for example, open source training data sets such as the MSCOCO data set, the MPII data set, and the POSETTRACK data set can be used; or, it can also be an image sequence pre-stored locally.

[0049] Step S12, inputting the training samples obtained in the aforementioned step S11 into the neural network model to obtain the recognition result of the posture of the rodent.

[0050] The embodiments of the present application do not specifically limit the neural network model, and any neural network model that can realize the posture recognition described in the present application can be used. For example, the neural network model can be a 2D convolutional neural network such as VGG, ResNet, HRNet, etc.

[0051] Optionally, HRNet (High Resolution Network) can maintain high resolution throughout the feature extraction process and can perform differential fusion of features of different resolutions during the feature extraction process. It is particularly suitable for applications in scenarios such as semantic segmentation, human posture, image classification, facial landmark detection, and general object recognition.

[0052] The recognition result may include the position information of a preset number of key points of the rodent's body recognized by the neural network model (also referred to as recognition position for short).

[0053] Step S13, according to the recognition result in step S12, using the loss function and the gradient descent method to train the neural network model.

[0054] Among them, the loss function may include the time constraint term L temporal .

[0055] Combine the following Figure 2 For the time constraint L temporal The method of determining is described in detail. Figure 2 , Figure 2 A method for determining a time constraint item is shown.

[0056] Time constraint L temporal Key points in the posture of the rodent may be used to constrain the positions between adjacent image frames in the image sequence.

[0057] In some embodiments, the time constraint L temporal It can be determined based on the error between the position information of the key points obtained by the tracking method and the position information of the key nodes in the recognition result.

[0058] In the training method provided in the embodiment of the present application, the tracking method may be an unsupervised tracking method, for example, the Lucas-Kanade optical flow method.

[0059] Figure 2 The method shown may include steps S1311 - S1315 .

[0060] Step S1311, selecting m images from the training samples as tracking samples.

[0061] The m images are any m images in the training sample. The m images can be m consecutive images in the training sample. It can be understood that the m images can also be all the images in the training sample.

[0062] Step S1312, taking the first image frame of the m images in the training sample as the initial frame, performing forward tracking using the recognition result of the initial frame to obtain a first forward tracking result, and determining a first difference between the first forward tracking result and the recognition result of the m-th image frame, wherein the first forward tracking result includes the tracking position of the key point in the m-th image frame. Wherein, m is a positive integer greater than or equal to 2. In other words, the first difference may be the difference between the tracking position and the recognition position of the same key point in the m-th image frame.

[0063] For the convenience of description, the set of m images will be denoted as I 1,i (i=1,2,…,m), the set I 1,iThe recognition result is recorded as Among them, ω is the number of key points in each image frame.

[0064] The first frame of the m images is used as the initial frame, and the recognition result of the initial frame is used Perform forward tracking and obtain the first forward tracking result Determine the first forward tracking result and set I 1,i The recognition result of the mth frame in The difference F1 between them is:

[0065]

[0066] Step S1313, taking the mth image frame of the m images as the termination frame, and performing backward tracking using the recognition result of the termination frame to obtain a first backward tracking result, wherein the first backward tracking result includes the tracking position of the key point in the first image frame. It can be understood that the mth image can also be called the last image frame of the m images. Determine the second difference between the first backward tracking result and the recognition result of the first image frame. In other words, the second difference can be the difference between the tracking position and the recognition position of the same key point in the first image frame.

[0067] The last frame of the m images is taken as the termination frame, and the recognition result of the termination frame is used Perform backward tracking and obtain the first backward tracking result Determine the first backward tracking result and set I 1,i The recognition result of the first frame in The difference F2 between them is:

[0068]

[0069] In step S1314, the first frame of the m images is used as the initial frame. Using the recognition result of the initial frame Perform forward tracking and obtain the first forward tracking result Then use the first forward tracking result As the termination frame, perform backward tracking and determine the second backward tracking result Determine the second backward tracking result With the initial frame The difference F3 is:

[0070]

[0071] In step S1315, time constraint items are determined.

[0072] When the first difference and the second difference are both less than or equal to a preset threshold, the time constraint item is determined to be 0; when the first difference and / or the second difference is greater than the preset threshold, the time constraint item is determined to be a second backward tracking result. With the initial frame The time constraint is

[0073] Wherein E1 is a preset threshold value, which is related to the movement characteristics of the organism. It should be noted that, compared with the prediction result of the neural network model, the tracking result obtained by the tracking method can ensure that the tracking position of the same key point changes smoothly in the time domain. Therefore, when the difference (such as the first difference or the second difference) is less than the preset threshold value, it means that the recognition result is close to the tracking result, and the recognition result of the neural network model is relatively smooth in the time domain. At this time, the time constraint item may not be set. When the difference is greater than the preset threshold value, it means that the recognition result is quite different from the tracking result. In other words, the recognition result is relatively jittery in the time domain. At this time, the neural network model can be trained by setting the time constraint item to make the recognition result output by the neural network model smoother.

[0074] The embodiment of the present application does not specifically limit the manner of determining the time constraint. For example, the first difference may be used as the time constraint. For another example, the second difference may be used as the time constraint. For another example, the first forward tracking result may be backward tracked to obtain a second backward tracking result; and the time constraint may be determined based on the difference between the second backward tracking result and the recognition result of the first image frame. For another example, the first backward tracking result may be forward tracked to obtain a second forward tracking result; and the time constraint may be determined based on the difference between the second forward tracking result and the recognition result of the first image frame.

[0075] In some embodiments, the loss function further includes a spatial constraint term, which is used to limit the position of key points in the posture of the rodent in the same frame image.

[0076] See also Figure 3 , Figure 3 A method for determining a spatial constraint item is shown.

[0077] Spatial constraint L spatical It can be used to define the position of the key points in the posture of the creature in the same image frame. In some embodiments, the spatial constraint term L spatical It can be determined based on the difference between the positions of multiple key points in the recognition result.

[0078] An embodiment of the present application provides a method for determining a spatial constraint item L spatical The method may include steps S1321-S1322.

[0079] Step S1321, selecting p samples from the training samples, where p is a positive integer greater than or equal to 2;

[0080] The p images are any p images in the training sample. The p images can be p consecutive images in the training sample. It is understandable that the p images can also be all images in the training sample. Wherein, p is a positive integer greater than or equal to 2.

[0081] For the convenience of description, the set consisting of p images will be denoted as I 2,j (j=1,2,…,p), the set I 2,j The recognition result is recorded as Among them, ω is the number of key points in each image frame.

[0082] Step S1322, determining the distance between two key points in the same image in the training sample.

[0083] Determine the recognition results of the p samples The distance between two key points in

[0084] Step S1323, determine the space constraint items.

[0085] The above distance According to the Gaussian distribution, the distance The mean μ and variance σ of 2 , where ω is the number of key points in each image frame;

[0086] Determine the spatial constraints as:

[0087] It should be noted that the spatial constraint item L provided in the above steps S1321-S1323 is spatical The method is only an example, and can also be determined by other methods. For example, the spatial constraint item can also be determined based on the error between the distance between two key points in the recognition result and the distance between two key points in the corresponding annotation result, which is not limited in this application.

[0088] In some embodiments, the loss function may also include an error constraint term L MSE In some embodiments, the error constraint term L MSE It can be determined based on the error between the position information of the same key point in the recognition result of the training sample and the labeling result. Figure 4 Taking mean square error as an example, determining the error constraint item may include steps S1331-S1333.

[0089] Step S1331, select n images from the training samples obtained in step S11 to form a sample set I 3,k (k=1,2,…,n), where n is a positive integer greater than or equal to 1.

[0090] The n images are any n images in the training sample. The n images can be n consecutive images in the training sample. It can be understood that the n images can also be all images in the training sample.

[0091] Step S1332, determine sample set I 3,k The recognition result And the marking results

[0092] Step S1333, calculate the above recognition result And the annotation results The mean square error of , the error constraint term is determined as:

[0093]

[0094] For the error constraint term, in addition to the mean square error loss, cross entropy loss, 0-1 loss, absolute value loss, etc. commonly used in the art may also be used. The method shown in the above steps S1331-S1333 is only an example and does not limit the scope of protection of this application.

[0095] In some embodiments, the error constraint term L can also be MSE , time constraint L temporal And the spatial constraint L spatical The weighted sum is used to determine the loss function. That is, the loss function L = L MSE +a L temporal +bL spatical , where a and b are hyperparameters, and their values ​​are greater than or equal to 0.

[0096] Combine the following Figure 5 The embodiments of the training device provided by the present application are described in detail. It should be understood that the device embodiments correspond to the description of the aforementioned method embodiments. Therefore, the parts not described in detail can refer to the aforementioned method embodiments.

[0097] Figure 5 is a schematic block diagram of a training device 50 provided in one embodiment of the present application. It should be understood that Figure 5 The device 50 shown is only an example, and the device 50 of the embodiment of the present invention may also include other modules or units.

[0098] It should be understood that the device 50 is capable of performing Figure 1-Figure 4 To avoid repetition, the steps in the method are not described here.

[0099] As a possible implementation, the device includes:

[0100] The acquisition module 51 is used to acquire training samples.

[0101] The training samples and the method for obtaining the training samples may be consistent with step S11 of the aforementioned method, and will not be described in detail here.

[0102] The input module 52 is used to input the training sample into the neural network model to obtain the recognition result of the posture of the rodent, and the recognition result includes the key points in the posture of the rodent.

[0103] The training module 53 is used to train the neural network model using a gradient descent method using a loss function according to the recognition result of the rodent's posture.

[0104] Optionally, the neural network model includes a HRNet network.

[0105] Optionally, before training the neural network model, the training device further includes: a first determination module, configured to determine the time constraint item based on an error between a position of a key point obtained using a tracking method and a position of the key point in the recognition result.

[0106] Optionally, the tracking method includes a Lucas-Kanade optical flow method.

[0107] Optionally, the first determination module is used to: select m samples from the training samples as tracking samples, where m is a positive integer greater than or equal to 2; The first frame in As the initial frame, forward tracking is performed to obtain the forward tracking result Determine the difference between the forward tracking result and the recognition result of the mth image frame in the tracking sample as: Where ω is the number of key points on each image frame; the last frame in the recognition result of the tracking sample As the termination frame, perform backward tracking and obtain the backward tracking result Determine that the difference between the backward tracking result and the recognition result of the first image frame in the tracking sample is The first frame in the recognition result of the tracking sample As the initial frame Perform forward tracking and obtain forward tracking results Based on the forward tracking results As the initial frame, perform backward tracking and obtain the second backward tracking result The second backward tracking result and the initial frame The difference is Determine the time constraints as: Where E1 is the threshold.

[0108] Optionally, the loss function further includes a spatial constraint term, which is used to limit the positions of key points in the posture of the rodent in the same frame image.

[0109] Optionally, the training device further includes: a second determination module, configured to determine the spatial constraint item according to a difference between positions of a plurality of key points in the recognition result.

[0110] Optionally, the second determination module is used to: select p samples from the training samples, where p is a positive integer greater than or equal to 2; and determine the recognition results of the p samples. The distance between two key points in The distance According to the Gaussian distribution, the distance The mean μ and variance σ of 2 , where ω is the number of key points on each image frame; the spatial constraint term is determined as:

[0111] Optionally, the loss function further includes an error constraint term, and the error constraint term is used to constrain the error of key points in the posture of the organism in the recognition result and the labeling result.

[0112] Optionally, the error constraint term is a mean square error loss term.

[0113] It should be understood that the apparatus 50 for training the neural network model here is embodied in the form of a functional module. The term "module" here can be implemented in the form of software and / or hardware, and is not specifically limited to this. For example, a "module" can be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a merged logic circuit, and / or other suitable components that support the described functions.

[0114] As an example, the apparatus 50 for training a neural network model provided in the embodiment of the present invention may be a processor or a chip for executing the method described in the embodiment of the present invention.

[0115] Figure 6 It is a schematic block diagram of a training device 60 provided in another embodiment of the present application. Figure 6 The device 60 shown includes a memory 61, a processor 62, a communication interface 63 and a bus 64. The memory 61, the processor 62 and the communication interface 63 are connected to each other through the bus 64.

[0116] The memory 61 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 61 may store a program. When the program stored in the memory 61 is executed by the processor 62, the processor 62 is used to execute the various steps of the training method provided in the embodiment of the present invention. For example, the processor 62 may execute Figure 1-Figure 4 The various steps of the illustrated embodiment.

[0117] The processor 62 can adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs to implement the training method of the embodiment of the method of the present invention.

[0118] The processor 62 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the training method provided by the embodiment of the present invention may be completed by an integrated logic circuit of hardware in the processor 62 or by instructions in the form of software.

[0119] The processor 62 may also be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0120] The steps of the method disclosed in the embodiment of the present invention can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 61, and the processor 62 reads the information in the memory 61, and combines its hardware to complete the functions required to be executed by the units included in the gesture recognition device in the embodiment of the present invention, or execute the training method of the method embodiment of the present invention. For example, it can be executed Figure 1-Figure 4 The various steps / functions of the illustrated embodiment.

[0121] The communication interface 63 may use, but is not limited to, a transceiver or other transceiver device to implement communication between the device 60 and other devices or a communication network.

[0122] The bus 64 may include a path for transmitting information between the various components of the device 60 (eg, the memory 61 , the processor 62 , and the communication interface 63 ).

[0123] It should be understood that the device 60 shown in the embodiment of the present invention may be a processor or a chip, so as to execute the method described in the embodiment of the present invention.

[0124] It should be understood that the processor in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0125] Combine the following Figure 7 The specific application of the embodiment of the present application is introduced in the following application scenarios. Figure 7 The description is only an example and not a limitation. The method in the embodiment of the present application is not limited to this and can also be applied to other gesture recognition scenarios.

[0126] Figure 7 The application scenario may include an image acquisition device 71 and an image processing device 72.

[0127] Among them, the image acquisition device 71 can be used to acquire an image sequence of rodents. The image processing device 72 can be integrated in an electronic device, which can be a server or a terminal and other devices, and the embodiments of the present application are not limited to this. For example, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud computing, cloud storage, cloud communications, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a computer, and an intelligent Internet of Things device. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and the present application does not limit this.

[0128] The image processing device 72 may be equipped with a neural network model, which can be used to identify the image sequence acquired by the image acquisition device 71 using the neural network model to obtain the position information of the key points in the image to be processed. The position information of the key points may include, for example, the position coordinate information of the body joints, trunk or facial features of the rodent.

[0129] The electronic device can also use the image acquisition device 71 to acquire training samples, and train the neural network model using the loss function according to the recognition results of the training samples and the results of manual annotation. The image processing device 72 can also recognize the image to be processed through the trained neural network model, thereby achieving the purpose of accurately recognizing the image.

[0130] The embodiments described above are only a part of the embodiments of the present application, rather than all the embodiments. The description order of the above embodiments is not used as a limitation on the preferred order of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0131] It should be understood that in the embodiment of the present application, "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0132] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0133] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0134] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that can be read by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0135] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A training method, characterized in that: include: Acquire a training sample, wherein the training sample is an image sequence recording the movement of a rodent; Inputting the training sample into a neural network model to obtain a recognition result of the posture of the rodent, wherein the recognition result includes key points in the posture of the rodent, and the neural network model includes a HRNet network; Determining a time constraint item according to an error between a position of a key point obtained by a tracking method and a position of a key point in the recognition result; According to the recognition result of the posture of the rodent, the neural network model is trained using a gradient descent method using a loss function; The loss function includes the time constraint term, and the time constraint term is used to constrain the position of the key point in the posture of the rodent between adjacent image frames in the image sequence. The time constraint term is determined according to the error between the position of the key point obtained by the tracking method and the position of the key point in the recognition result, including: Select m samples from the training samples as tracking samples, where m is a positive integer greater than or equal to 2; The identification result of the tracking sample The first frame in As the initial frame, forward tracking is performed to obtain the forward tracking result Determine the difference between the forward tracking result and the recognition result of the mth image frame in the tracking sample as: Where ω is the number of key points in each image frame; The last frame in the recognition result of the tracking sample As the termination frame, perform backward tracking and obtain the backward tracking result Determine that the difference between the backward tracking result and the recognition result of the first image frame in the tracking sample is The first frame in the recognition result of the tracking sample As the initial frame Perform forward tracking and obtain forward tracking results Based on the forward tracking results As the initial frame, perform backward tracking and obtain the second backward tracking result The second backward tracking result and the initial frame The difference is The time constraint is determined as: Where E1 is the threshold.

2. The training method according to claim 1, characterized in that: The tracking method includes the Lucas-Kanade optical flow method.

3. The training method according to claim 1, characterized in that: The loss function also includes a spatial constraint term, which is used to limit the position of key points in the posture of the rodent in the same frame image.

4. The training method according to claim 3, characterized in that: Before training the neural network model, the training method further includes: The spatial constraint item is determined according to the difference between the positions of the multiple key points in the recognition result.

5. The training method according to claim 4, characterized in that: The determining the spatial constraint item according to the difference between the positions of the plurality of key points in the recognition result comprises: Select p samples from the training samples, where p is a positive integer greater than or equal to 2; Determine the recognition results of the p samples The distance between two key points in The distance According to the Gaussian distribution, the distance The mean μ and variance σ of 2 , where ω is the number of key points in each image frame; Determine the spatial constraints as:

6. The training method according to claim 1, characterized in that: The loss function also includes an error constraint term, which is used to constrain the errors of key points in the posture of the rodent in the recognition results and the labeling results.

7. The training method according to claim 6, characterized in that: The error constraint term is a mean square error loss term.

8. A training device, characterized in that: include: An acquisition module, used for acquiring training samples, wherein the training samples are image sequences recording the movement of rodents; An input module, used for inputting the training sample into a neural network model to obtain a recognition result of the posture of the rodent, wherein the recognition result includes key points in the posture of the rodent, and the neural network model includes a HRNet network; A first determination module, configured to determine a time constraint item according to an error between a position of a key point obtained by a tracking method and a position of a key point in the recognition result; A training module, for training the neural network model using a gradient descent method using a loss function according to the recognition result of the posture of the rodent; The loss function includes the time constraint term, and the time constraint term is used to constrain the position of the key point in the posture of the rodent between adjacent image frames in the image sequence. The time constraint term is determined according to the error between the position of the key point obtained by the tracking method and the position of the key point in the recognition result, including: Select m samples from the training samples as tracking samples, where m is a positive integer greater than or equal to 2; The identification result of the tracking sample The first frame in As the initial frame, forward tracking is performed to obtain the forward tracking result Determine the difference between the forward tracking result and the recognition result of the mth image frame in the tracking sample as: Where ω is the number of key points in each image frame; The last frame in the recognition result of the tracking sample As the termination frame, perform backward tracking and obtain the backward tracking result Determine that the difference between the backward tracking result and the recognition result of the first image frame in the tracking sample is The first frame in the recognition result of the tracking sample As the initial frame Perform forward tracking and obtain forward tracking results Based on the forward tracking results As the initial frame, perform backward tracking and obtain the second backward tracking result The second backward tracking result and the initial frame The difference is The time constraint is determined as: Where E1 is the threshold.

9. The training device according to claim 8, characterized in that The tracking method includes the Lucas-Kanade optical flow method.

10. The training device according to claim 8, characterized in that The loss function also includes a spatial constraint term, which is used to limit the position of key points in the posture of the rodent in the same frame image.

11. The training device according to claim 10, characterized in that The training device also includes: The second determination module is used to determine the spatial constraint item according to the difference between the positions of multiple key points in the recognition result.

12. The training device according to claim 11, characterized in that The second determining module is used for: Select p samples from the training samples, where p is a positive integer greater than or equal to 2; Determine the recognition results of the p samples The distance between two key points in The distance According to the Gaussian distribution, the distance The mean μ and variance σ of 2 , where ω is the number of key points in each image frame; Determine the spatial constraints as:

13. The training device according to claim 8, characterized in that The loss function also includes an error constraint term, which is used to constrain the errors of key points in the posture of the rodent in the recognition results and the labeling results.

14. The training device according to claim 13, characterized in that The error constraint term is a mean square error loss term.

Images