A method and device for motion recognition

Abstract

The present application provides a method and device for motion recognition, which relates to the field of detection and monitoring technology. The method includes obtaining a two-dimensional image captured by a camera and azimuth data provided by a gravity sensor mounted on the camera; transforming the two-dimensional image according to the camera azimuth data; and using the transformed multiple frames of two-dimensional images as input to a trained motion recognition neural network model to obtain motion recognition results output by the motion recognition neural network model. The present application corrects the image through the data collected by the gravity sensor, which weakens the problem of insufficient concentration of intra-class features caused by different shooting angles, greatly reduces the difficulty of classification, reduces the complexity requirements of the neural network, and significantly improves the accuracy of the classification results.

Description

Technical Field

[0001] The present application relates to the field of detection and monitoring technology, and in particular to a method and device for motion recognition. Background Art

[0002] In the field of computer vision, action recognition has always been the focus of cutting-edge applications. At present, several mainstream action recognition models (such as 2D+1D, 3D convolution kernels, human key point recognition, etc.) are all based on two-dimensional images and frame sequence data for model training. In the process of directly performing action recognition through two-dimensional images, considering the different actual application scenarios, the cameras used will have different placements and shooting angles, and the images obtained for the same action in different directions will be quite different, which brings great difficulties to the training and recognition of the model. On the one hand, if the actual placement angle of the camera is different from the angle during model training, it will cause obvious classification errors. On the other hand, in actual applications, it is necessary to place cameras under different conditions in various occasions, and it is difficult to achieve direct action recognition using a single model.

[0003] Although using 3D images to train neural networks can alleviate this problem to a certain extent, it also places higher requirements on input data. The acquisition of 3D images requires more complex hardware functions, which cannot be directly adapted to most current hardware products and will bring higher hardware facility costs. At the same time, the data analysis and processing of 3D images also requires higher neural network complexity, which brings additional overhead. Therefore, it is difficult to meet the needs of a wider range of users.

[0004] Therefore, how to improve the accuracy of 2D image action recognition under various shooting angles is still a critical issue. Currently, more methods focus on designing deep learning models with higher efficiency and accuracy, but lack the analysis and processing of image shooting angles. Summary of the invention

[0005] In order to solve at least one of the above-mentioned technical problems, the present application provides a method and device for motion recognition, so as to weaken the influence of the image shooting angle through appropriate data processing, and thereby build a complete set of methods and devices for data collection and processing, model training, and motion recognition.

[0006] The first aspect of the present application is a method for motion recognition, comprising: acquiring a two-dimensional image captured by a camera and azimuth data provided by a gravity sensor mounted on the camera; transforming the two-dimensional image according to the camera azimuth data; and using the transformed multiple frames of two-dimensional images as input to a trained motion recognition neural network model to obtain a motion recognition result output by the motion recognition neural network model.

[0007] Preferably, transforming the two-dimensional image includes: acquiring each frame of the two-dimensional image captured by the camera and the azimuth data of the gravity sensor corresponding to the two-dimensional image, and forming a data pair; using the data pair as the input of the trained image transformation neural network model, and obtaining the transformed two-dimensional image output by the image transformation neural network model.

[0008] Preferably, transforming the two-dimensional image includes: calculating a camera angle based on azimuth data provided by a gravity sensor; and transforming the two-dimensional image according to the camera angle to obtain a transformed two-dimensional image.

[0009] Preferably, the transformation of the two-dimensional image includes: y=Ax+b; wherein x is the coordinates of each pixel point of the two-dimensional image before the transformation, y is the coordinates of each pixel point of the two-dimensional image after the transformation, A is the transformation matrix formed by the camera angle transformation θ, and b is the offset vector of the two-dimensional image in the plane caused by the camera angle transformation θ.

[0010] In a second aspect of the present application, a motion recognition device is provided, which mainly includes: a data acquisition module, used to acquire a two-dimensional image captured by a camera and azimuth data provided by a gravity sensor mounted on the camera; an image transformation module, used to transform the two-dimensional image according to the camera azimuth data; and a motion recognition module, used to use the transformed multi-frame two-dimensional image as input to a trained motion recognition neural network model to obtain the motion recognition result output by the motion recognition neural network model.

[0011] Preferably, the image transformation module includes: a data binding unit, used to obtain each frame of two-dimensional image captured by the camera and the azimuth data of the gravity sensor corresponding to the two-dimensional image, and form a data pair; an image transformation neural network processing unit, used to use the data pair as the input of the trained image transformation neural network model to obtain the transformed two-dimensional image output by the image transformation neural network model.

[0012] Preferably, the image transformation module includes: a camera angle calculation unit, used to calculate the camera angle based on the azimuth data provided by the gravity sensor; and a coordinate transformation unit, used to transform the two-dimensional image according to the camera angle to obtain the transformed two-dimensional image.

[0013] Preferably, the coordinate transformation unit includes: y=Ax+b; wherein x is the coordinate of each pixel point of the two-dimensional image before transformation, y is the coordinate of each pixel point of the two-dimensional image after transformation, A is the transformation matrix formed by the camera angle transformation θ, and b is the offset vector of the two-dimensional image in the plane caused by the camera angle transformation θ.

[0014] According to a third aspect of the present application, a computer device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the action recognition method as described in any one of the above items.

[0015] A fourth aspect of the present application is a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the action recognition method as described above.

[0016] After the image is corrected by the data collected by the gravity sensor, the input data features of the same category in the processed image data are more significant than the original RGB data, which weakens the problem of insufficient concentration of intra-class features caused by different shooting angles, greatly reduces the difficulty of classification, reduces the complexity requirements of the neural network, and significantly improves the accuracy of the classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a preferred embodiment of the action recognition method of the present application.

[0018] Figure 2 It is a schematic diagram of the structure of a computer device of a terminal or server suitable for implementing the implementation mode of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the implementation of this application clearer, the technical scheme in the implementation of this application will be described in more detail in combination with the drawings in the implementation of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described implementation is a part of the implementation of this application, not all of the implementations. The implementation described below with reference to the drawings is exemplary and is intended to be used to explain this application, and cannot be understood as a limitation on this application. Based on the implementation in this application, all other implementations obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The implementation of this application is described in detail below in combination with the drawings.

[0020] According to the first aspect of this application, if Figure 1 As shown, a method for action recognition includes:

[0021] Step S1, obtaining a two-dimensional image captured by a camera and azimuth angle data provided by a gravity sensor mounted on the camera.

[0022] This application uses a camera equipped with a G-sensor to collect raw data. In actual use, the camera will capture the RGB data of each frame in the video, and bind the image data of the frame with the azimuth data collected by the G-sensor at that time to form all the data of a frame. For each frame, perform the above operations to obtain the raw data corresponding to the video captured by the camera.

[0023] Step S2: transform the two-dimensional image according to the camera azimuth angle data.

[0024] In step S2, based on the data collected by the G-sensor, the original two-dimensional image RGB data is processed to obtain distorted image data, and the processed image data is still RGB data.

[0025] With the help of G-sensor, this application can obtain the angle of the camera relative to the ground when taking an image, adjust the captured image accordingly, process the images obtained at different shooting angles, and obtain images at a more ideal shooting angle, which is convenient for the neural network to process in the subsequent data analysis module, thereby solving the problems caused by different shooting angles.

[0026] Step S3: using the transformed multiple frames of two-dimensional images as input to the trained action recognition neural network model to obtain the action recognition results output by the action recognition neural network model.

[0027] As the processed data, the processed data of the continuous frames is the video after orientation correction. After that, each frame of processed data is continuously input into the action recognition neural network model. The action recognition neural network model is a model trained by sampling data. For different action recognition tasks, the neural network may have different choices, such as 3D CNN or ST-GCN (Spatial Temporal Graph Convolutional Networks). The neural network will analyze the image data in one frame and the information brought by the continuous frames at the spatial and temporal levels, and output the final action recognition result after the neural network.

[0028] Compared with the existing technology, the introduction of G-sensor makes reasonable angle correction for the image, which greatly facilitates the correct analysis and recognition of the image by the neural network, reduces the complexity requirement of the neural network, and improves the accuracy and analysis efficiency of action recognition. After angle correction, this action recognition system can handle action recognition tasks under various application conditions, reduces the requirements for camera placement in actual applications, and greatly improves the generalization performance. At the same time, the hardware requirements of G-sensor are easy to implement, and it is easy to install on different types of cameras. The additional cost introduced is low, which is conducive to the wide application of the system.

[0029] In some optional embodiments, transforming the two-dimensional image includes: acquiring each frame of the two-dimensional image captured by the camera and the azimuth data of the gravity sensor corresponding to the two-dimensional image, and forming a data pair; using the data pair as input to the trained image transformation neural network model, and acquiring the transformed two-dimensional image output by the image transformation neural network model.

[0030] This embodiment processes the collected raw data through CNN (Convolutionnal Neural Networks). When training the image transformation neural network model, the camera is first used to collect RGB expected data at a preset angle, and then deflected by a certain angle relative to the preset value, and an input value is obtained. The input value consists of a two-dimensional image and a deflection angle. These input values ​​and the expected data form a training sample, and the image transformation neural network model is trained to obtain a forming model that can calibrate image data deflected by a certain angle, which is used for image transformation in this embodiment.

[0031] In an alternative implementation, before using the image transformation neural network model to transform the two-dimensional image, the collected data needs to be further preprocessed.

[0032] In this embodiment, when training the image transformation neural network model, a two-dimensional image deflected by a certain angle is usually bound to the deflection angle as an input value for training. The deflection angle here is usually selected using a fixed step size, for example, a deflection of 1°, 3°, 5°, etc. The image transformation neural network model trained by these data usually only has good processing results for these deflection angles. There may be deviations in the processing results for two-dimensional images taken at other angles. For this reason, the collected data needs to be preprocessed.

[0033] The preprocessing includes transforming the collected two-dimensional image according to the deviation between the angle of the collected gravity sensor and the training angle. For example, during the training process, there are deflection angles given by the gravity sensor at fixed intervals of 1°, 3°, 5°, etc., and during the actual processing process, a deflection angle given by the gravity sensor of 2.5° is collected. Then, the two-dimensional image is transformed according to +0.5° to form a two-dimensional image corresponding to 3°. The transformation formula is:

[0034] y = Ax, x is the coordinates of each pixel point in the two-dimensional image before transformation, y is the coordinates of each pixel point in the two-dimensional image after transformation, and A is the transformation matrix composed of deviations, for example:

[0035]

[0036] In this embodiment,

[0037] The transformed two-dimensional image y corresponds to 3°, so it is bound and input into the image transformation neural network model for calculation.

[0038] In some optional implementations, transforming the two-dimensional image includes: calculating a camera angle based on azimuth data provided by a gravity sensor; and transforming the two-dimensional image according to the camera angle to obtain a transformed two-dimensional image.

[0039] In some optional embodiments, transforming the two-dimensional image includes: y=Ax+b; wherein x is the coordinates of each pixel point in the two-dimensional image before the transformation, y is the coordinates of each pixel point in the two-dimensional image after the transformation, A is the transformation matrix formed by the camera angle transformation θ, and b is the offset vector of the two-dimensional image in the plane caused by the camera angle transformation θ.

[0040] As in the above embodiment, the present application may also directly transform the two-dimensional image by using a transformation matrix, where: θ is the deflection azimuth of the camera, which can be obtained through the deflection azimuth of the gravity sensor and the coordinate system transformation of the gravity sensor and the camera. After the image is matrix transformed, displacement is also required. The displacement vector b usually refers to the position change of the character in the two-dimensional image. b can be set to b=s*sinθ, where s is the distance between the camera and the character.

[0041] In some optional implementations, the two-dimensional images obtained by the above two processing methods may be subjected to weighted processing.

[0042] For example, the first two-dimensional image is obtained through the image transformation neural network model; the second two-dimensional image is obtained through coordinate transformation, and the coordinate points in the two images are arithmetic averaged to obtain the final two-dimensional image, or weighted average is performed, and the weight coefficient a of the image transformation neural network model and the weight coefficient b of the coordinate transformation satisfy:

[0043] a+b=1;

[0044] a=n / m;

[0045] Among them, n is the set value, which represents the optimal step size. It is used to indicate that when the sample data collected under the optimal step size n is trained, the constructed image transformation neural network model basically meets the processing requirements and can obtain better processing results. Usually n is 1, and m is the actual step size of the training angle transformation during the sample collection process when training the image transformation neural network model, and m>n.

[0046] It can be understood that the above formula shows that when the training samples are insufficient, the actual step size m of the azimuth transformation of the gravity sensor collected is too large, resulting in inaccurate image transformation neural network model. For this reason, the weighted value a of the image transformation neural network model is relatively small. On the contrary, if the training samples are sufficient, the step size of the azimuth transformation of the gravity sensor is very small, and the image transformation neural network model will be more accurate. The corresponding weighted value a of the image transformation neural network model is relatively large. If samples are collected every 1°, the image transformation neural network model obtained thereby is particularly accurate. In this way, the weighted value a of the image transformation neural network model can be set to 1, and there is no need to use coordinate transformation for correction.

[0047] The second aspect of the present application provides a motion recognition device corresponding to the above method, which mainly includes: a data acquisition module, used to acquire a two-dimensional image captured by a camera and azimuth data provided by a gravity sensor mounted on the camera; an image transformation module, used to transform the two-dimensional image according to the camera azimuth data; and a motion recognition module, used to use the transformed multi-frame two-dimensional image as input to a trained motion recognition neural network model to obtain the motion recognition result output by the motion recognition neural network model.

[0048] In some optional embodiments, the image transformation module includes: a data binding unit, used to obtain each frame of two-dimensional image captured by the camera and the azimuth data of the gravity sensor corresponding to the two-dimensional image, and form a data pair; an image transformation neural network processing unit, used to use the data pair as the input of the trained image transformation neural network model to obtain the transformed two-dimensional image output by the image transformation neural network model.

[0049] In some optional embodiments, the image transformation module includes: a camera angle calculation unit, used to calculate the camera angle based on the azimuth data provided by the gravity sensor; and a coordinate transformation unit, used to transform the two-dimensional image according to the camera angle to obtain a transformed two-dimensional image.

[0050] In some optional embodiments, the coordinate transformation unit includes: y=Ax+b; wherein x is the coordinate of each pixel point of the two-dimensional image before the transformation, y is the coordinate of each pixel point of the two-dimensional image after the transformation, A is the transformation matrix formed by the camera angle transformation θ, and b is the offset vector of the two-dimensional image in the plane caused by the camera angle transformation θ.

[0051] In a third aspect of the present application, a computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a motion recognition method.

[0052] In a fourth aspect of the present application, a readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the above-mentioned action recognition method. The computer-readable storage medium may be included in the device described in the above-mentioned embodiment; or it may exist alone without being assembled into the device. The above-mentioned computer-readable storage medium carries one or more programs, and when the above-mentioned one or more programs are executed by the device, the data is processed according to the above-mentioned method.

[0053] Reference below Figure 2 , which shows a schematic structural diagram of a computer device 400 suitable for implementing the embodiments of the present application. Figure 2 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0054] like Figure 2 As shown, the computer device 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 are also stored. The CPU 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0055] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.

[0056] In particular, according to the embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the above functions defined in the method of the present application are executed. It should be noted that the computer storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus or a device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0057] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0058] The modules or units involved in the embodiments described in this application may be implemented by software or hardware. The modules or units described may also be set in a processor, and the names of these modules or units do not constitute limitations on the modules or units themselves in some cases.

[0059] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for motion recognition, characterized in that: include: Acquire a two-dimensional image captured by a camera and azimuth data given by a gravity sensor mounted on the camera; transforming the two-dimensional image according to the camera azimuth data; as well as Using the transformed multiple frames of two-dimensional images as input to a trained action recognition neural network model, and obtaining an action recognition result output by the action recognition neural network model; Wherein, transforming the two-dimensional image comprises: Obtain each frame of a two-dimensional image captured by the camera and the azimuth angle data of the gravity sensor corresponding to the two-dimensional image, and form a data pair; Using the data pair as input of a trained image transformation neural network model, and obtaining a transformed first two-dimensional image output by the image transformation neural network model; Calculate the camera angle based on the azimuth data provided by the gravity sensor; Transform the two-dimensional image according to the camera angle to obtain a transformed second two-dimensional image; The first two-dimensional image and the second two-dimensional image are weighted, and the weight coefficient a of the first two-dimensional image and the weight coefficient c of the second two-dimensional image satisfy: a+c=1; a=n / m; Among them, n is a set value, indicating the optimal step size, and m is the actual step size of the training angle transformation during the sample collection process when training the image transformation neural network model, and m>n.

2. The motion recognition method according to claim 1, characterized in that: Transforming the two-dimensional image includes: y = Ax + b; Among them, x is the coordinate of each pixel point of the two-dimensional image before transformation, y is the coordinate of each pixel point of the two-dimensional image after transformation, A is the transformation matrix formed by the camera angle transformation θ, and b is the offset vector of the two-dimensional image in the plane caused by the camera angle transformation θ.

3. A motion recognition device, characterized in that: include: A data acquisition module, used to acquire a two-dimensional image captured by a camera and azimuth data given by a gravity sensor mounted on the camera; An image transformation module, used for transforming the two-dimensional image according to the camera azimuth data; as well as An action recognition module, used to use the transformed multiple frames of two-dimensional images as input to a trained action recognition neural network model, and obtain an action recognition result output by the action recognition neural network model; Wherein, the image transformation module comprises: A data binding unit, used to obtain each frame of a two-dimensional image collected by the camera and the azimuth angle data of the gravity sensor corresponding to the two-dimensional image, and form a data pair; An image transformation neural network processing unit, used for taking the data pair as input of a trained image transformation neural network model, and obtaining a transformed first two-dimensional image output by the image transformation neural network model; A camera angle calculation unit, used to calculate the camera angle based on the azimuth data provided by the gravity sensor; A coordinate transformation unit is used to transform the two-dimensional image according to the camera angle, obtain the transformed second two-dimensional image, and perform weighted processing on the first two-dimensional image and the second two-dimensional image, wherein the weight coefficient a of the first two-dimensional image and the weight coefficient c of the second two-dimensional image satisfy: a+c=1; a=n / m; Among them, n is a set value, indicating the optimal step size, and m is the actual step size of the training angle transformation during the sample collection process when training the image transformation neural network model, and m>n.

4. The motion recognition device according to claim 3, characterized in that: The coordinate transformation unit comprises: y = Ax + b; Among them, x is the coordinate of each pixel point of the two-dimensional image before transformation, y is the coordinate of each pixel point of the two-dimensional image after transformation, A is the transformation matrix formed by the camera angle transformation θ, and b is the offset vector of the two-dimensional image in the plane caused by the camera angle transformation θ.

5. A computer device, characterized in that it comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the action recognition method according to any one of claims 1 to 2.

6. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the action recognition method as described in any one of claims 1-2.

Images