Human body behavior tracking method, medium, equipment and system based on multi-sensor fusion

Abstract

The invention relates to a human body behavior tracking method based on multi-sensor fusion, a medium, equipment and a system, and the method comprises the steps: constructing a classification neural network model based on generative adversarial improvement, configuring a plurality of sensors based on a human body, obtaining a data sample, training the network model, positioning the human body based on a preset initialization action, and carrying out the tracking of the human body behavior. Acquiring position signals of a plurality of sensor ranges, inputting the position signals into the trained network model, and outputting a current human body behavior tracking result; implementing hardware by using the method; the system comprises at least two sensors and a controller for executing the method, and is connected to the VR wearable device and the central processing unit in cooperation with the controller. The method constructs the classification neural network model based on generative adversarial improvement, can expand the data volume on the premise of obtaining a small amount of effective screening data, considers the application to human body behavior tracking, can quickly achieve the convergence of the classification network along with the generation of training data by using a context attention module, and improves the accuracy of human body behavior tracking. The robustness is improved, and the training efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a human behavior tracking method, medium, device and system based on multi-sensor fusion. Background Art

[0002] Multi-sensor Information Fusion (MSIF) is an information processing process that uses computer technology to carry out multi-level and multi-space information complementarity and optimal combination processing under certain criteria, and automatically analyzes and synthesizes information and data from multiple sensors or multiple sources to complete the required decision-making and estimation, which will ultimately produce a consistent interpretation of the observed environment.

[0003] In fact, multi-sensor information fusion is already a conventional technology in the field of sensing. Considering the deepening of applications, the increase in data volume, and the increase in data dimensions, when faced with how to partition different types of data, how to sort and combine sensor data, and how to make decisions based on multi-sensor sensor data, the conventional processing method of the existing technology is to clean the raw data, fuse the objects, and transmit them to the control end, and classify the sensor data with a trained network.

[0004] However, in the actual process, due to the differences between different sensors, the data acquisition and processing methods are different, so data fusion is difficult. At the same time, the existing artificial intelligence technology is not perfect. Improving the accuracy and reliability of the fusion algorithm and reducing the complexity of the algorithm has always been the direction of research and development. Multiple sensors may provide repeated or contradictory information. The transmission of raw data will bring a large amount of data and bandwidth requirements, increasing the cost and complexity of the system. Therefore, how to obtain the information needed for training and remove a large amount of invalid information is the concern of researchers. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a human behavior tracking method, medium, device and system based on multi-sensor fusion.

[0006] The technical solution adopted by the present invention is a human behavior tracking method based on multi-sensor fusion, and the method comprises the following steps: S1 builds a classification neural network model based on generative adversarial improvement; S2 configures a number of sensors based on the human body, obtains data samples, and trains the network model; S3 locates the human body based on the preset initialization action. If the human body is located, proceed to the next step, otherwise repeat S3; S4 collects position signals from several sensor ranges, inputs them into the trained network model, and outputs the current human behavior tracking results.

[0007] Preferably, the classification neural network model based on generative adversarial improvement includes two VGG networks arranged in parallel; One of the VGG networks has an improved generative adversarial network configured at the input; The improved generative adversarial network and two VGG networks are trained.

[0008] Preferably, the improved generative adversarial network includes a generator, a judger and a context attention module; The original data sample is processed by the context attention module to obtain a processed data sample, the processed data sample is input into the generator, and the generated new sample data is input into the judger.

[0009] Preferably, the classification neural network model based on generative adversarial improvement includes a neural network corresponding to the half body; The neural network corresponding to any half of the body outputs the corresponding label, and the neural network corresponding to the half of the body is also configured with a fully connected layer for identifying and combining the labels.

[0010] Preferably, in S2, sensors are arranged on the head, shoulders, elbows, hands, knees and feet of the human body.

[0011] Preferably, the data samples comprise image-label pairs; the images being skeleton images of the behavior connected by sensor sites.

[0012] Preferably, in S3, the initialization actions include standing, extending arms and squatting, and based on the initialization actions, the human head, the combination of shoulders-elbow joints-hands, and the combination of knee joints-feet are respectively positioned.

[0013] A computer-readable storage medium stores a human behavior tracking program based on multi-sensor fusion, and the program implements the human behavior tracking method based on multi-sensor fusion when executed by a processor.

[0014] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the human behavior tracking method based on multi-sensor fusion is implemented.

[0015] A human behavior tracking system based on multi-sensor fusion, the system comprising: At least two sensors are placed at different key positions on the human body to locate and sense the posture of the corresponding positions; A controller, used to realize the positioning and behavior tracking of a human body by using the human body behavior tracking method based on multi-sensor fusion; The controller is connected to the VR wearable device and the central processing unit.

[0016] The present invention relates to a human behavior tracking method, medium, device and system based on multi-sensor fusion, which constructs a classification neural network model based on generative adversarial improvement, configures a number of sensors based on the human body, obtains data samples, trains the network model, locates the human body based on a preset initialization action, collects position signals within the range of a number of sensors, inputs the trained network model, and outputs the current human behavior tracking result; the method is used to implement hardware; the system includes at least two sensors and a controller for executing the method, which cooperates with the controller to be connected to a VR wearable device and a central processing unit.

[0017] The beneficial effect of the present invention is that the present invention constructs a classification neural network model based on generative adversarial improvement, which can expand the data volume under the premise of obtaining a small amount of effective screening data, and considering the application to human behavior tracking, the contextual attention module is further used to accompany the generation of training data, which can quickly achieve the convergence of the classification network, improve robustness and improve training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a structural schematic diagram of the improved classification neural network model of the present invention; Figure 3 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described in detail below in conjunction with embodiments, but the protection scope of the present invention is not limited thereto.

[0020] The present invention relates to a human behavior tracking method based on multi-sensor fusion, the method comprising the following steps: (1) Construct a classification neural network model based on generative adversarial improvement; (2) configuring a number of sensors based on the human body, obtaining data samples, and training the network model; (3) Locate the human body based on the preset initialization action. If the human body is located, proceed to the next step; otherwise, repeat (3); (4) Collect position signals within the range of several sensors, input them into the trained network model, and output the current human behavior tracking results.

[0021] The following is a detailed description based on the steps of the method.

[0022] (1) Construct a classification neural network model based on generative adversarial improvement; First, a basic neural network is built. The improved classification neural network model based on generative adversarial network of the present invention sets two parallel VGG networks, here VGG-16, including 5 groups of convolution blocks, each group of convolution blocks contains multiple convolution layers and a maximum pooling layer. The VGG network has a small number of parameters and a large network depth, thereby improving the generalization ability of the model and being able to learn complex features; Here, one branch is the conventional VGG, which is used for conventional classification processing, and the other branch is used to expand the input data with an improved generative adversarial network; For this improved generative adversarial network, it is set as a connected generator and a judge, and the generator is modified by a context attention module; specifically, the original data sample is processed by the context attention module to obtain a processed data sample, the processed data sample is input into the generator, and the generated new sample data is input into the judge, while the data volume is increased, the noise is limited by the context attention mechanism, the training time is reduced, and the training efficiency is improved; Finally, the improved classification neural network model based on generative adversarial network of the present invention synchronously trains the generative adversarial network and two VGG networks, so that the generator of the generative adversarial network generates less erroneous data and the classification error between the two VGG networks is smaller.

[0023] Furthermore, in order to reduce the difficulty of training, the user behavior can be modeled and trained with half of the body. The neural network corresponding to either half of the body outputs the corresponding label, and the labels of the two halves are merged with a fully connected layer to achieve classification based on the combination of labels.

[0024] (2) configuring a number of sensors based on the human body, obtaining data samples, and training the network model; In S2, sensors are configured on the head, shoulders, elbows, hands, knees, and feet of the human body.

[0025] The data samples consist of image-label pairs; images are skeleton images of behaviors connected by sensor sites.

[0026] In the present invention, in order to better analyze human behavior and optimize the tracking effect, sensors are configured for the main joints of human behavior; Sensors on the head, shoulders and feet can roughly determine the human body configuration; Sensors based on the elbow joint and hand can determine hand movements; Sensors based on the knee joints and feet can determine the footsteps.

[0027] Specifically, (3) locate the human body based on the preset initialization action, if the human body is located, proceed to the next step, otherwise repeat (3); The initialization actions include standing, extending arms and squatting, and based on the initialization actions, the human head, the combination of shoulder-elbow joint-hand, and the combination of knee joint-foot are respectively positioned; That is, in standing position, the main focus is on locating the human head; With the arms extended, locate the shoulder-elbow-hand combination; In the case of squatting, the knee-foot combination can be positioned; Completing positioning means that the sensor signal has been found and subsequent collection and tracking can be carried out.

[0028] (4) Collect position signals within the range of several sensors, input them into the trained network model, and output the current human behavior tracking results.

[0029] The present invention also relates to a computer-readable storage medium, on which a human behavior tracking program based on multi-sensor fusion is stored. When the program is executed by a processor, the human behavior tracking method based on multi-sensor fusion is implemented.

[0030] The present invention also relates to a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the human behavior tracking method based on multi-sensor fusion is implemented.

[0031] The present invention also relates to a human behavior tracking system based on multi-sensor fusion, the system comprising: At least two sensors are placed at different key positions on the human body to locate and sense the posture of the corresponding positions; A controller, used to realize the positioning and behavior tracking of a human body by using the human body behavior tracking method based on multi-sensor fusion; The controller is connected to the VR wearable device and the central processing unit.

[0032] In the present invention, this system is configured with a VR device, and after obtaining sensor data and tracking human behavior based on the method, it can be transmitted to a VR wearable device through a central processor for synchronization. The VR wearable device here mainly includes a VR headset.

[0033] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0034] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0035] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0036] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0037] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0038] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A human behavior tracking method based on multi-sensor fusion, characterized by: The method comprises the following steps: S1 builds a classification neural network model based on generative adversarial improvement; S2 configures a number of sensors based on the human body, obtains data samples, and trains the network model; S3 locates the human body based on the preset initialization action. If the human body is located, proceed to the next step, otherwise repeat S3; S4 collects position signals from several sensor ranges, inputs them into the trained network model, and outputs the current human behavior tracking results.

2. The method for tracking human behavior based on multi-sensor fusion according to claim 1, characterized in that: The classification neural network model based on generative adversarial improvement includes two VGG networks set in parallel; One of the VGG networks has an improved generative adversarial network configured at the input; The improved generative adversarial network and two VGG networks are trained.

3. The method for tracking human behavior based on multi-sensor fusion according to claim 2, characterized in that: The improved generative adversarial network includes a generator, a judger and a context attention module; The original data sample is processed by the context attention module to obtain a processed data sample, the processed data sample is input into the generator, and the generated new sample data is input into the judger.

4. The method for tracking human behavior based on multi-sensor fusion according to claim 1, characterized in that: The classification neural network model based on generative adversarial improvement includes a neural network corresponding to the half body; The neural network corresponding to any half of the body outputs the corresponding label, and the neural network corresponding to the half of the body is also configured with a fully connected layer for identifying and combining the labels.

5. The method for tracking human behavior based on multi-sensor fusion according to claim 1, characterized in that: In S2, sensors are configured on the head, shoulders, elbows, hands, knees, and feet of the human body.

6. The method for tracking human behavior based on multi-sensor fusion according to claim 5, characterized in that: The data samples consist of image-label pairs; images are skeleton images of behaviors connected by sensor sites.

7. The method for tracking human behavior based on multi-sensor fusion according to claim 1, characterized in that: In S3, the initialization actions include standing, extending arms and squatting, and the human head, the combination of shoulders-elbow joints-hands, and the combination of knee joints-feet are positioned based on the initialization actions.

8. A computer-readable storage medium, characterized in that: A human behavior tracking program based on multi-sensor fusion is stored thereon, and when the program is executed by the processor, the human behavior tracking method based on multi-sensor fusion as described in one of claims 1 to 7 is implemented.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the human behavior tracking method based on multi-sensor fusion as described in any one of claims 1 to 7 is implemented.

10. A human behavior tracking system based on multi-sensor fusion, characterized by: The system comprises: At least two sensors are placed at different key positions on the human body to locate and sense the posture of the corresponding positions; A controller, used to realize human body positioning and behavior tracking by using the human body behavior tracking method based on multi-sensor fusion as claimed in any one of claims 1 to 7; The controller is connected to the VR wearable device and the central processing unit.

Images