A method for detecting standing up and sitting down actions based on face position information

Abstract

A method for detecting standing up and sitting down actions based on face position information, which relates to the field of image processing technology, includes: constructing a feature space for outlier detection; constructing pseudo-depth information; constructing pseudo-three-dimensional coordinates as face features; using a linear regression algorithm to perform boundary learning tasks and learn the best plane; for situations where multiple people stand, matching and tracking the detection objects of the previous and next frames are performed through the optimal transmission theory, thereby achieving the function of maintaining stable tracking of objects while multiple people stand. The present invention improves the recognition effect of action detection by strengthening key spatial features, improves the accuracy and universality of computer vision recognition, and improves the real-time efficiency of detection.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for detecting standing up and sitting down actions based on face position information. Background Art

[0002] At present, most standing detection algorithms are implemented based on deep learning posture recognition or target tracking algorithms. Among them, posture recognition algorithms based on deep learning usually have high requirements for GPU and CPU computing power, and the number of model parameters is also large. Therefore, when deployed in scenes such as classroom teaching or lecture halls, the real-time standing detection of all objects in the crowd has high requirements for the computing resources of the equipment. Standing detection based on tracking algorithms requires continuous video streams as input, which has a great influence on the previous and next frames, and cannot achieve independent single-frame detection. It has poor recognition ability for high-speed moving targets, and the input based on video streams will make the size of the input data larger than the single-frame input, and the related pre-processing operations have a large amount of calculation. Summary of the invention

[0003] In view of the above-mentioned defects or deficiencies in the prior art, the purpose of the present invention is to provide a method for detecting standing up and sitting down actions based on facial position information. This method and this algorithm combine face detection with a linear regression model through the idea of ​​outlier detection and cross-border detection, and realize single-frame standing detection with low computational complexity, high sensitivity, and smooth operation even on the CPU. This solution has lower configuration requirements for deployed equipment and is more cost-effective. It improves the recognition effect of action detection, improves the accuracy and universality of computer vision recognition, and improves the real-time efficiency of detection.

[0004] In order to solve the above problems, in a first aspect, the present invention provides a method for detecting standing up and sitting down actions based on face position information, the method comprising:

[0005] Use face detection and alignment algorithms to detect and locate the current scene space, use the positioning results as raw data, and construct a feature space for outlier detection;

[0006] Construct pseudo depth information z, which can be used as a reference to determine the distance of the face from the camera. Among them, e1 and e2 are the coordinates of the two eyes of the face, e1, e2∈R2, m1 and m2 are the coordinates of the two mouth corners,

[0007] Construct pseudo 3D coordinates as facial features Among them, concat(·) is the cascade operation of vectors. It is easy to know The first and second dimensions are the x- and y-coordinates of the nose tip, the third dimension is the pseudo-depth z-coordinate, and n is the nose tip position coordinate.

[0008] Use linear regression algorithm for boundary learning task to transform face position features The first and third dimensions, i.e., the x-coordinate of the nose tip and the pseudo-depth z-coordinate, are used as input parameters of the linear regression algorithm. The second dimension, i.e., the y-coordinate of the nose tip is used as the regression label. Through the linear regression algorithm, we only need to input the corresponding coordinates of the detected people when they are all sitting down, and we can learn an optimal plane y=k1x+k2z+k3, which is used to predict the y-coordinate of people at different x-coordinates and z-coordinates when they are sitting down, where k1, k2, and k3 are learning parameters.

[0009] Use the optimal transmission theory to establish a matching relationship between the detection objects in the previous and next frames, and use the Sinkhorn method to solve the optimal transmission problem T = arg min T ∑ ij T ij C ij st where C∈R N×M is the distance matrix of the objects in the previous and next frames, T∈R N×M is the transmission matrix.

[0010] At the same time, in order to meet the needs of more accurate face matching and face recognition, this method also provides an interface for specific face recognition algorithms. By combining a given face similarity metric with the optimal transmission theory, more accurate face matching is achieved. Through the K-nearest neighbor algorithm (KNN), the detected face is matched with the faces in the database, and the function of outputting the personal information of the standing person while detecting the standing person is realized.

[0011] According to the matching relationship, a record of the standing duration of the detected object is established, and only the target standing for more than a certain time is tracked to eliminate false detection.

[0012] In a second aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the embodiment of the present application when executing the computer program.

[0013] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to:

[0014] When the computer program is executed by a processor, the method described in the embodiment of the present application is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The embodiments of the present invention will be described below with reference to the accompanying drawings, wherein

[0016] Figure 1 A schematic diagram of the process of detecting standing up and sitting down actions based on face position information of the present application is shown;

[0017] Figure 2 A schematic diagram of linear plane learning in an actual application scenario of the present application is shown. DETAILED DESCRIPTION

[0018] In order to make the invention purpose, technical process and technical innovation of the present invention more clearly described, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0019] In this embodiment, the detected population is regarded as a stable cluster in the feature space. Based on the premise that the standing population is a minority in the detected population, the standing population can be understood as outliers in the cluster and detected. Therefore, the problem can be transformed into how to learn an optimal boundary through a machine learning algorithm and detect outliers through a cross-boundary detection method.

[0020] To achieve the above objectives, the present invention provides a method for detecting standing up and sitting down actions based on face position information. The main process is as follows: Figure 1 As shown, the method includes:

[0021] Use the face detection and alignment algorithm to detect and locate the current scene space, use the positioning results as raw data, and construct a feature space for outlier detection. Establish a rectangular coordinate system with the lower left corner of the image as the origin of the coordinate axis. Suppose the coordinates of the known eye positions of the face are The position coordinates of the two mouth corners are The coordinates of the nose tip are

[0022] Construct pseudo depth information z, which can be used as a reference to determine the distance of the face from the camera. Among them, e1 and e2 are the coordinates of the two eyes of the face, e1, e2∈R2, m1 and m2 are the coordinates of the two mouth corners, The distance between the center points of the two eyes and the center points of the two mouth corners is used as the pseudo depth information of the face. Due to the phenomenon that objects appear larger when near and smaller when far during shooting, the pseudo depth information z can be used as a reference for judging the distance of the face from the camera.

[0023] Construct pseudo 3D coordinates as facial features Among them, concat(·) is the cascade operation of vectors. It is easy to know The first and second dimensions are the x- and y-coordinates of the nose tip, the third dimension is the pseudo-depth z-coordinate, and n is the nose tip position coordinate.

[0024] Use linear regression algorithm for boundary learning task to transform face position features The first and third dimensions, i.e., the x-coordinate of the nose tip and the pseudo-depth z-coordinate, are used as input parameters of the linear regression algorithm. The second dimension, i.e., the y-coordinate of the nose tip is used as the regression label. Through the linear regression algorithm, we only need to input the corresponding coordinates of the detected people when they are all sitting down, and we can learn an optimal plane y=k1x+k2z+k3, which is used to predict the y-coordinate of people at different x-coordinates and z-coordinates when they are sitting down, where k1, k2, and k3 are learning parameters. Figure 2 This is an example of linear plane learning in a practical application scenario. The points are the positions of the detected faces in the pseudo three-dimensional coordinate space. It can be seen that the algorithm can well distinguish between people sitting in the front row and those sitting in the back row.

[0025] For situations where multiple people are standing, in order to ensure that the detected object has a good matching relationship between the previous and next frames, the optimal transmission theory is used to calculate a matching relationship between the detected objects in the previous and next frames. N×M is the distance matrix of the objects in the previous and next frames, where N and M represent the number of detected objects in the previous frame and the number of detected objects in the next frame, respectively. N×M is the transmission matrix, that is, the correspondence matrix of the objects in the previous and next frames. Then solve the following problem: T = arg min T ∑ ij T ij C ij , while satisfying the conditions and Where T ij , C ij Represents the values ​​of the i-th row and j-th column of the matrix T and C, respectively. N ∈R N , 1 M ∈R M Respectively represent N-dimensional vectors and M-dimensional vectors whose values ​​are all 1. In order to solve the above problem, it is necessary to define the distance matrix C. Here we use a metric that combines location information and scale information in are the nose coordinates of the two comparison objects, z1 and z2 are the pseudo depth coordinates of the two comparison objects, and based on this metric, the distance matrix C can be constructed. At the same time, we use the Sinkhorn method to solve the above formula and get the optimal solution T ★ ∈RN×M . Then for the i-th detected object in the previous frame, It is the serial number of the target corresponding to the detected object in the subsequent frame.

[0026] At the same time, in order to meet the needs of more accurate face matching and face recognition, this method also provides an interface for specific face recognition algorithms. If the face similarity metric f is given, the mixed metric d′=αd+βf will be used in the above optimal transmission problem, where α and β are emphasis coefficients. At the same time, according to the metric f, combined with the K-nearest neighbor algorithm (KNN), the detected face is matched with the face in the database, realizing the function of detecting the standing person and outputting the personal information of the standing person.

[0027] According to the correspondence between the detected object in the previous and next frames, we maintain a duration record for each newly detected target. If and only if the newly detected target remains standing for a certain period of time (such as 1 second), we determine that the target is a stable target and track it. This approach can effectively avoid misjudgments, such as misjudgments caused by the detected target standing up briefly.

[0028] Further face recognition is performed on the object detected to be in a standing state, the face information of the standing object is matched with the stored face data, and the detailed information of the object with successful matching is displayed.

[0029] Existing solutions generally extract features based on the frame difference before and after the video stream for learning and prediction, while this patented algorithm only uses a single frame for training and prediction; existing solutions are generally based on human skeleton alignment and use continuous video sequences as input convolutional neural networks for learning and prediction, which have high requirements for GPU and other configurations, while this algorithm is based on a linear regression algorithm and has lower configuration requirements; existing algorithms generally learn video streams based on convolutional neural networks, with complex data structures, more noisy data, and high coupling, while this algorithm extracts features based on face position and simple data processing methods, has a simple feature structure, less noise, and can be easily connected to different face detection and alignment modules, with high flexibility.

[0030] In order to better understand the present invention, the above is described in detail in conjunction with specific examples of the present invention, but it is not intended to limit the present invention. Any simple modification made to the above embodiments based on the technical essence of the present invention still falls within the scope of the technical solution of the present invention.

[0031] As another aspect, the present application further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the embodiment of the present application when executing the computer program.

[0032] As another aspect, the present application further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the aforementioned device in the above-mentioned embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the aforementioned programs are used by one or more processors to execute the method described in the embodiment of the present application.

[0033] Any reference to the storage medium used in the embodiments of the present application may include non-volatile, volatile memory. Suitable non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), optical disk (including CD-ROM read-only memory (CD-ROM) and digital versatile disk (DVD)), or flash memory. Volatile memory may include random access memory (RAM), which is used as an external high-speed cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0034] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (Programmable Gate Array; hereinafter referred to as: PGA), a field programmable gate array (Field Programmable Gate Array; hereinafter referred to as: FPGA), etc.

[0035] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0036] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software and firmware. If the integrated module is implemented in the form of software and firmware and sold or used as an independent product, it can be transferred from a storage medium or a network to a computer with a dedicated hardware structure for functional implementation.

[0037] It should also be noted that the steps of performing the above series of processes can naturally be performed in chronological order according to the order of description, but do not necessarily need to be performed in chronological order. Some steps can be performed in parallel or independently of each other.

[0038] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present application. Any person skilled in the art can change, modify, replace and modify the above embodiments without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention is not limited thereto, and should be based on the scope defined by the claims.

Claims

1. A method for detecting standing up and sitting down actions based on face position information, the method comprising: Use face detection and alignment algorithms to detect and locate the current scene space, use the positioning results as raw data, and construct a feature space for outlier detection; Construct pseudo depth information z, which can be used as a reference to determine the distance of the face from the camera. Among them, e1 and e2 are the coordinates of the two eyes of the face, e1, e2∈R2, m1 and m2 are the coordinates of the two mouth corners, Construct pseudo 3D coordinates as facial features Among them, concat(·) is the cascade operation of vectors. It is easy to know The first and second dimensions are the x- and y-coordinates of the nose tip, the third dimension is the pseudo-depth z-coordinate, and n is the nose tip position coordinate. Use linear regression algorithm for boundary learning task to transform face position features The first and third dimensions, i.e., the x-coordinate of the nose tip and the pseudo-depth z-coordinate, are used as input parameters of the linear regression algorithm, and the second dimension, i.e., the y-coordinate of the nose tip is used as the regression label. Through the linear regression algorithm, only the corresponding coordinates of the detected people when they are all sitting down need to be input, and an optimal plane y=k1x+k2z+k3 can be learned to predict the y-coordinates of people at different x-coordinates and z-coordinates when they are sitting down, where k1, k2, k3 are learning parameters; Use the optimal transmission theory to establish a matching relationship between the detection objects in the previous and next frames, and use the Sinkhorn method to solve the optimal transmission problem where C∈R N×M is the distance matrix of the objects in the previous and next frames, T∈R N × M is the transmission matrix, T ij , C ij Represents the values ​​of the i-th row and j-th column of the matrix T and C, respectively. N ∈R N , 1 M ∈R M Respectively represent N-dimensional vectors and M-dimensional vectors whose values ​​are all 1; According to the matching relationship, a record of the standing duration of the detected object is established, and only the target standing for more than a certain time is tracked to eliminate false detection.

2. The method according to claim 1, characterized in that In order to solve the optimal transmission problem, we define the distance matrix C, using a metric that combines location information and scale information. in are the nose coordinates of the two comparison objects, z1 and z2 are the pseudo depth coordinates of the two comparison objects, and the distance matrix C is constructed according to the metric. At the same time, the Sinkhorn method is used to solve the optimal solution T ★ ∈R N×M , for the i-th detected object in the previous frame, It is the serial number of the target corresponding to the detected object in the subsequent frame.

3. The method according to claim 1, characterized in that Further face recognition is performed on the object detected to be in a standing state, the face information of the standing object is matched with the stored face data, and the detailed information of the object with successful matching is displayed.

4. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

5. A computer-readable storage medium having a computer program stored thereon, the computer program being used to: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images