A sitting posture recognition method and system based on self-feedback learning

Abstract

The present invention provides a sitting posture recognition method and system based on self-feedback learning. The method includes: defining sitting posture representations; constructing paired data of labeled image frames and sitting posture representations; training a sitting posture representation extraction model, and obtaining the sitting posture representation of a sitting posture image frame to be recognized based on the trained sitting posture representation extraction model; implementing sitting posture type discrimination based on the sitting posture representation of the sitting posture image frame to be recognized and preset discrimination parameters; performing corresponding reminders according to the recognized sitting posture type, and performing discrimination of feedback actions after the reminders, and constructing a positive sample package and a negative sample package based on the change trend of the feedback actions; iteratively optimizing the preset discrimination parameters based on the positive sample package and the negative sample package, so as to improve the accuracy of sitting posture type discrimination. The present invention can achieve sitting posture classification without inputting standard reference sitting postures. Based on the self-feedback mechanism, the discrimination parameters can be continuously iteratively updated as the number of recognitions increases, improving the accuracy and sensitivity of recognition.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology. In particular, it relates to a sitting posture recognition method and system based on self-feedback learning. Background Art

[0002] In existing human body posture extraction solutions, the single-eye based extraction technology and the binocular based extraction technology are two important branches. Among them, the single-eye based technology often uses information such as the position and distance of key points of the human body that can characterize the posture, and compares it with the pre-set standard sitting posture representation, and discriminates whether the sitting posture is abnormal through a fixed threshold. For the extraction technology based on depth sensors such as binoculars and TOF, better representation effects have been achieved in terms of the accuracy of representing the posture. However, for the discrimination of sitting postures, the above-mentioned static method of "comparing with the pre-set standard sitting posture representation and discriminating whether the sitting posture is abnormal through a fixed threshold" is still mostly used. These methods face the problem of the effect brought by static discrimination, which is often manifested as that the fixed threshold is often set according to experience or population statistical data and cannot be adaptively adjusted according to the habit characteristics of specific users, resulting in insufficient sensitivity in judgment or easy misjudgment. Summary of the Invention

[0003] In view of the above-mentioned technical problem of low sensitivity in sitting posture recognition, a sitting posture recognition method and system based on self-feedback learning are provided. Based on the self-feedback incremental learning mechanism, the discrimination parameters can be continuously iteratively updated as the number of recognitions increases, improving the accuracy and sensitivity of recognition.

[0004] The technical means adopted by the present invention are as follows:

[0005] A sitting posture recognition method based on self-feedback learning, comprising:

[0006] Defining a sitting posture representation that can directly or indirectly express the situation of the sitting posture type, and the sitting posture representation is obtained based on an image frame;

[0007] Obtaining a non-continuous image data set and a continuous video frame data set, obtaining the annotation results of each image frame in the image data set and the video frame data set, so as to construct a matching pair data of the annotated image frame and the sitting posture representation;

[0008] Using the matching pair data of the annotated image frame and the sitting posture representation as training data to train a sitting posture representation extraction model, obtaining a sitting posture image frame to be recognized, and processing the sitting posture image to be recognized based on the trained sitting posture representation extraction model to obtain the sitting posture representation of the sitting posture image frame to be recognized;

[0009] Implementing sitting posture type discrimination based on the sitting posture representation of the sitting posture image frame to be recognized and preset discrimination parameters, and the discrimination parameters include a direction coefficient and an offset parameter;

[0010] Perform corresponding reminders according to the recognized sitting posture type, determine the feedback actions after the reminders, and construct positive and negative sample packages based on the changing trends of the feedback actions. The positive sample package includes all sitting posture representations within a period when the user's sitting posture shows a tendency to improve, and the negative sample package includes all sitting posture representations within a period when the user's sitting posture does not show a tendency to improve;

[0011] Iteratively optimize the preset discrimination parameters based on the positive and negative sample packages, so as to improve the accuracy of sitting posture type discrimination.

[0012] Further, the sitting posture representation is:

[0013] R seq =<box,keypoints,baseP,θ1,θ2,θ3>

[0014] Where, R seq(i) represents the sitting posture representation, box represents the bounding box of the human head, keypoints represents the key points of the human sitting posture, baseP represents the head reference point, θ1 represents the pitch rotation angle, θ2 represents the left - right swing angle, θ3 represents the left - right rotation angle, and baseP is the coordinate reference origin of θ1, θ2, and θ3.

[0015] Further, the sitting posture representation extraction model includes a single - frame extraction model for extracting the sitting posture representation of a single - frame data in an image dataset. The single - frame extraction model includes a first deep neural network module and a second deep neural network module;

[0016] The input of the first deep neural network module is the three - channel color image data obtained from the single - frame data. The output of the first deep neural network module is used as the input of the second deep neural network module. The output of the second neural network module includes the discrimination result of whether there is a human body in the single - frame data and the sitting posture representation.

[0017] Further, the output of the second neural network module also includes the auxiliary discrimination result of face attributes.

[0018] Further, the single - frame extraction model also includes a third deep neural network module and a fourth deep neural network module. The input of the third deep neural network module is the 3D point cloud data obtained from the single - frame data. The output of the third deep neural network module is used as the input of the second deep neural network module;

[0019] The input of the fourth deep neural network module is the output of the first deep neural network module, and the output of the fourth deep neural network module is the regressed 3D point cloud data.

[0020] Further, the sitting posture representation extraction model further includes a timing frame extraction model for extracting the sitting posture representation of consecutive frame data in the video frame dataset. The timing frame extraction model includes a FA module and an RNN module for feature aggregation;

[0021] The input of the FA module is the sitting posture representation of the current single-frame data extracted by the single-frame extraction model, the hidden layer output of the second deep neural network module when extracting the sitting posture representation of the current single-frame data by the single-frame extraction model, and the hidden layer output of the second deep neural network module when extracting the sitting posture representation of the previous single-frame data by the single-frame extraction model. The output of the FA module is the high-level feature of the sitting posture representation of the current frame. After cascading the high-level feature of the sitting posture representation of the current frame with the hidden layer output of the second deep neural network module when extracting the sitting posture representation of the current single-frame data by the single-frame extraction model, it serves as the input of the RNN module. The output of the RNN module is the sitting posture representation of the current frame of the video.

[0022] Further, the sitting posture type discrimination is realized based on the sitting posture representation of the sitting posture image frame to be recognized and the preset discrimination parameters, including:

[0023] Pre-set the unique original features and co-occurring original features of each sitting posture type;

[0024] Calculate the derived features based on the sitting posture representation of the sitting posture image frame to be recognized;

[0025] Based on the preset discrimination parameters, calculate the unique value of the derived feature, the co-occurrence value of the derived feature, and the change value of the derived feature in time series respectively;

[0026] Calculate the probability that the sitting posture image frame to be recognized belongs to a certain sitting posture type based on the unique value of the derived feature, the co-occurrence value of the derived feature, and the change value of the derived feature in time series.

[0027] Further, calculating the unique value of the derived feature, the co-occurrence value of the derived feature, and the change value of the derived feature in time series respectively based on the preset discrimination parameters includes:

[0028] Calculate the unique value of the derived feature based on the following formula:

[0029]

[0030] where, f1 represents the unique value of multiple derived features, D R(n) represents the nth derived feature of the sitting posture representation, f a represents the activation function, W f1(n) represents the direction coefficient of the nth derived feature, b f1(n) is the offset coefficient of the nth derived feature;

[0031] Calculate the co-occurrence value of the derived features based on the following formula:

[0032]

[0033] where f2 represents the co-occurrence value of the feature of K groups of derived features, and each group has M j co-occurring features, W f2(j,n) represents the direction coefficient of the nth feature in the jth group, D R(j,n) represents the jth derived feature, b f2(j,n) represents its offset coefficient;

[0034] Calculate the change value of the derived features over time based on the following formula:

[0035]

[0036] where f3 represents the change value of the derived features over time. A certain period of time T is divided into T1 period, T2 period and T3 period according to a preset rule. The T1 period represents the current period, the T2 period represents the intermediate period, and the T3 period represents the past period. D R,t (n) represents the value of the derived feature in the t period, W f3(i) represents the weight value of the derived feature value.

[0037] Furthermore, calculate the probability that the sitting posture image frame to be recognized belongs to a certain sitting posture type based on the unique value of the derived feature, the co-occurrence value of the derived feature, and the change value of the derived feature over time, including:

[0038] Calculate the probability that the sitting posture image frame to be recognized belongs to a certain sitting posture type based on the following formula:

[0039] f = sigmoid(w_norm * (f1 + f2 + f3))

[0040] where f represents the probability that the sitting posture image frame to be recognized belongs to a certain sitting posture type, f1 represents the unique value of the derived feature, f2 represents the co-occurrence value of the feature of the derived feature, f3 represents the change value of the derived feature over time, and w_norm represents the linear adjustment factor, which is a neural network learning parameter.

[0041] The present invention also provides a sitting posture recognition system based on self-feedback learning, including:

[0042] A sitting posture characterization definition module, which is used to define a sitting posture characterization that can directly or indirectly express the situation of the sitting posture type, and the sitting posture characterization is obtained based on the image frame;

[0043] A matching pair data construction module, which is used to obtain a discontinuous image data set and a continuous video frame data set, obtain the annotation results of each image frame in the image data set and the video frame data set, so as to construct matching pair data of annotated image frames and sitting posture representations;

[0044] A sitting posture representation extraction module, which is used to use the matching pair data of the annotated image frames and the sitting posture representations as training data to train a sitting posture representation extraction model, obtain a sitting posture image frame to be recognized, and process the sitting posture image to be recognized based on the trained sitting posture representation extraction model to obtain the sitting posture representation of the sitting posture image frame to be recognized;

[0045] A sitting posture type recognition module, which is used to perform sitting posture type discrimination based on the sitting posture representation of the sitting posture image frame to be recognized and preset discrimination parameters, and the discrimination parameters include a direction coefficient and an offset parameter;

[0046] A reminder module, which is used to give corresponding reminders according to the recognized sitting posture type, perform discrimination on feedback actions after the reminder, and construct positive sample packets and negative sample packets based on the change trend of the feedback actions, where the positive sample packet includes all sitting posture representations within a period of time when the user's sitting posture has a tendency to improve, and the negative sample packet includes all sitting posture representations within a period of time when the user's sitting posture has no tendency to improve;

[0047] A self-feedback module, which is used to iteratively optimize the preset discrimination parameters based on the positive sample packet and the negative sample packet, so as to improve the accuracy of sitting posture type discrimination.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] 1. The present invention trains the sitting posture representation extraction model in a multi-task learning manner, which can efficiently complete the extraction of various sitting posture representations. At the same time, for multiple frames, information fusion is performed at the feature layer and the representation layer, which not only improves the generalization of the model, but also captures the continuity information between image frames, further enhancing the stability.

[0050] 2. The present invention constructs a unified confidence sitting posture recognition model based on an empirical threshold, overcoming the instability caused by the strong dependence on a single feature in existing methods. For example, many methods use key points such as faces or human bodies, and when the key points are not visible, the recognition usually fails.

[0051] 3. The present invention provides an incremental learning method combined with the user's feedback actions, continuously iteratively updating the initial parameters, improving the accuracy and sensitivity of recognition, making the sitting posture recognition result fully conform to the characteristics of subjective judgment, and eliminating the need for the user to input a preset standard sitting posture. Description of the Drawings

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0053] Figure 1 This is a flowchart of a sitting posture recognition method based on self-feedback learning according to the present invention.

[0054] Figure 2 This is a schematic diagram of the sitting posture representation according to the present invention.

[0055] Figure 3 This is an architecture diagram of a single-frame extraction model according to the present invention.

[0056] Figure 4 This is an architecture diagram of a sequential-frame extraction model according to the present invention.

[0057] Figure 5 This is an architecture diagram of a self-feedback learning model according to the present invention.

[0058] Figure 6 This is a schematic diagram of the preset partition of recognition parameters according to the present invention. Detailed implementation manners

[0059] In order to enable those skilled in the art of the present technology to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0060] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above accompanying drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0061] Such as Figure 1As shown in the figure, the present invention provides a sitting posture recognition method based on self-feedback learning, including:

[0062] S1. Define a sitting posture representation that can directly or indirectly express the sitting posture type, and the sitting posture representation is obtained based on image frames.

[0063] In this application, the following representation data is used as the sitting posture representation:

[0064] R seq(i) =<box i ,keypoints i ,baseP i ,θ1 i ,θ2 i ,θ3 i >

[0065] Among them, R seq(i) represents the sitting posture representation implied by the i-th image frame, and the image frame can be RGB data extracted by a monocular / binocular camera or point cloud data of a depth camera.

[0066] As Figure 2 shown, box represents the bounding box of the human head, which is used to determine the position and relative distance of the head. Keypoints consists of three key points k1, k2, and k3, which respectively represent the position of the cervical root, the position of the left shoulder, and the position of the right shoulder. BaseP represents the head reference point, and in this application, the center point of the nose tip with obvious saliency is used. In the specific implementation process, a point that has a rigid relationship with the whole head can be used, such as the center of the eyebrows, the center of the chin, etc. θ1 represents the pitch rotation angle, that is, the angle of looking up and down. When looking straight ahead, θ1 = 0, <0 when looking up, and >0 when looking down. θ2 represents the roll rotation angle, that is, the angle of the head swaying from side to side. When there is no skew, θ2 = 0, <0 when skewing to the right, and >0 when skewing to the left. θ3 represents the yaw rotation angle, that is, the angle of the head rotating from side to side. When there is no rotation, θ3 = 0, <0 when rotating to the right, and >0 when rotating to the left. The coordinate reference origin of θ1, θ2, and θ2 is baseP.

[0067] S2. Obtain a discontinuous image data set and a continuous video frame data set, and obtain the manual annotation results of each image frame in the image data set and the video frame data set, so as to construct the matching pair data of the annotated image frame and the sitting posture representation.

[0068] Existing methods mostly use single frames and connect multiple sub-feature extraction modules to complete the extraction of sitting posture representations, or send multiple frames of images at one time and use algorithms such as 3D convolution to make full use of the information in the time series to more accurately extract sitting posture representations. In the present invention, we use the method of multi-task learning to efficiently complete the extraction of various sitting posture representations, and at the same time, for multiple frames, information fusion is performed at the feature layer and the representation layer.

[0069] In this embodiment, the set ImageData = {img_1, img_2, img_N} is a discontinuous image data set. Discontinuous image data is often more easily obtained, but there is no content continuity between images. The set VideoData = {v_1_img_1, v_1_img2, v1_img_M, v_2_img1…} is a continuous video frame data set. For each element in the above set, the sitting posture representation label is determined by manual annotation. Specifically, it can be in the form of a labeling tool or constructed using auxiliary devices.

[0070] S3. Use the matching pair data of the labeled image frame and the sitting posture representation as training data to train the sitting posture representation extraction model, obtain the sitting posture image frame to be recognized, and process the sitting posture image to be recognized based on the trained sitting posture representation extraction model to obtain the sitting posture representation of the sitting posture image frame to be recognized.

[0071] Specifically, the sitting posture representation extraction model in this application includes a single-frame extraction model for extracting the sitting posture representation of single-frame data in the image data set and a temporal frame extraction model for extracting the sitting posture representation of continuous frame data in the video frame data set. The single-frame extraction model is used to extract the sitting posture representation of discontinuous image data, and the temporal frame extraction model is used to extract the sitting posture representation of continuous video frame data.

[0072] As a preferred embodiment of the present invention, the single-frame extraction model includes a first deep neural network module and a second deep neural network module. The input of the first deep neural network module is the three-channel color image data obtained from single-frame data. The output of the first deep neural network module is used as the input of the second deep neural network module. The output of the second neural network module includes the discrimination result of whether there is a human body in the single-frame data and the sitting posture representation. Further, the output of the second neural network module also includes the face attribute auxiliary discrimination result. Furthermore, the single-frame extraction model also includes a third deep neural network module and a fourth deep neural network module. The input of the third deep neural network module is the 3D point cloud data obtained from single-frame data. The output of the third deep neural network module is used as the input of the second deep neural network module. The input of the fourth deep neural network module is the output of the first deep neural network module, and the output of the fourth deep neural network module is the regressed 3D point cloud data.

[0073] As Figure 3As shown, it is the preferred single-frame extraction model architecture in the present invention. Among them, RGB represents the three-channel color image data of imaging, and xyz represents the 3D point cloud data with the camera as the origin. net1 to net4 represent deep neural network modules, obj represents the network branch for determining the presence of a person, h1 represents the branch for regressing the set of points composed of box, keypoints, and baseP, h2 represents the branch for regressing the set of angles composed of θ1, θ2, and θ3, and aux serves as an auxiliary branch to supervise the regression of human face attributes such as gender and emotion. The dotted line represents an optional connection structure, and there are the following typical use cases to cover various application scenarios.

[0074] Application Example 1: There is no xyz data in the training data. At this time, the third deep neural network module net3 and the fourth deep neural network module net4 are not gated. Only rely on the first deep neural network module net1 and the second deep neural network module net2 to regress the sitting posture representation and the auxiliary attribute aux. The training method and the basic network architecture (net1-net4) are well-known in the field, such as VGG, Inception, ResNet, MobileNet, ShuffleNet, VIT and their variants, etc.

[0075] Application Example 2: Based on Application Example 1, there are full or partial xyz and RGB matching pair data during training. At this time, net3 and net4 are gated. net4 is used to learn from RGB data to estimate its corresponding xyz to establish the model's sensitivity to depth. During inference, according to the different hardware, it can be selected whether to remove the xyz input and the gating of net3. For example, for hardware with low control cost, only the RGB input is input separately.

[0076] The goal of this step is to train and determine the network parameters of net1-net4 on the image dataset, and the model can predict and output the sitting posture representation. Here, the sitting posture representation output by the single-frame extraction model is denoted as: R img It represents the sitting posture representation of the image img output by the single-frame model. According to different use cases, the img contains the aforementioned RGB and XYZ data.

[0077] As a preferred embodiment of the present invention, the timing frame extraction model includes a FA module and an RNN module for feature aggregation; the input of the FA module is the current single-frame data sitting posture representation extracted by the single-frame extraction model, the hidden layer output of the second deep neural network module when the current single-frame data sitting posture representation is extracted by the single-frame extraction model, and the hidden layer output of the second deep neural network module when the previous single-frame data sitting posture representation is extracted by the single-frame extraction model. The output of the FA module is the high-level feature of the sitting posture representation of the current frame. The high-level feature of the sitting posture representation of the current frame is cascaded with the hidden layer output of the second deep neural network module when the current single-frame data sitting posture representation is extracted by the single-frame extraction model as the input of the RNN module, and the output of the RNN module is the sitting posture representation of the current frame of the video.

[0078] Specifically, the video frame dataset is different from the aforementioned image dataset. Its frames have continuity, and the actual system scenario runs in the video frame mode to obtain a better user experience. In this step, we use the hidden layer output of the aforementioned net2 and the sitting posture representation output of the previous frame as inputs to output the sitting posture representation of the current frame. As above Figure 4 shown, where R seq(i) is the sitting posture representation of the i-th frame of the video, R img(i) is the image-based sitting posture representation output by the aforementioned single-frame extraction model (which only relies on a single image as the inference basis and often has unstable effects), h img(i) is the hidden layer output result of the net2 network module in the single-frame extraction model. Typical dimensions are such as 2048, 1024, depending on the specific network design, which is well-known in this field. The part in the dashed box is the RNN model, typically such as LSTM, GRU, for capturing long-term information dependencies. The part in the shaded box is the FA module, which is a feature aggregation module based on the attention mechanism to enhance the information transfer between adjacent frames.

[0079] Furthermore, the specific expression of the FA (feature aggregation) module is as follows:

[0080] Q i = W q .h img(i)

[0081] Q i-1 = W q .h img(i-1)

[0082] T i = W t .tanh(Q i *Q i-1 ))

[0083] V i= W v .R img(i) + W v .(T i * R seq(i-1))

[0084] R seq(i) = RNN(concat(V i , h img(i) )

[0085] where Q i and Q i-1 represent the query vectors obtained by linearly mapping the current frame and the previous frame with the hidden feature h img respectively, T i represents the correlation vector of information between two frames, and V i represents the high-dimensional feature of the sitting posture representation of the current frame, which, after being concatenated with h img , is fed into the RNN layer to finally obtain the sitting posture representation R seq(i) in terms of time series. Where W q , W t , W v are learnable parameters.

[0086] The present invention can be compatible with inputs of various data forms, RGB and XYZ. Thus, it can adapt to the differences between different hardware at one time. By training the corresponding designed models for ImageData and VideoData successively, the problems of data construction cost and full utilization are solved. The sitting posture representation in the video time series performs information fusion and inter-frame correlation measurement in the hidden feature layer, solving the problem that the recognition on video usually requires a large amount of computing power.

[0087] S4. Implement sitting posture type discrimination based on the sitting posture representation of the sitting posture image frame to be recognized and the preset discrimination parameters, where the discrimination parameters include a direction coefficient and an offset parameter. The method includes:

[0088] 1) Preset the unique original features and co-occurring original features of each sitting posture type.

[0089] 2) Calculate the derived features based on the sitting posture representation of the sitting posture image frame to be recognized;

[0090] 3) Calculate the unique value of the derived features, the co-occurring value of the derived features, and the change value of the derived features in terms of time series respectively based on the preset discrimination parameters;

[0091] 4) Calculate the probability that the sitting posture image frame to be recognized belongs to a certain sitting posture type based on the unique value of the derived features, the co-occurring value of the derived features, and the change value of the derived features in terms of time series.

[0092] Existing methods mostly use comparison or threshold methods to determine the sitting posture type, which have the characteristics of poor adaptability and incomplete subjective judgment experience. To solve this problem, this application proposes an incremental learning method combined with user feedback actions to improve this problem, and at the same time, there is no need for users to input pre-set standard sitting postures.

[0093] In this application, it completes the determination of the aforementioned sitting posture type according to the sitting posture representation R of the current frame and historical frames. seq(i) And the preset discrimination parameters. Specifically, denote S T _(R seq(i) ) as the sitting posture representation data in the past T seconds. Similarly, S T _(keypoints (i) ) represents the sitting posture key points in the past T seconds.

[0094] For example, for the determination of head-down, generally speaking, head-down is typically reflected in the decrease of the pitch (θ1) angle, the position of the reference point baseP becomes lower, and if the imaging device is placed in front of the user, the box usually becomes larger. And the determination period is usually during the user's static period, that is, during the user's movement, no determination is given. Given multiple corresponding thresholds V θ1 , V baseP , V box , note that the meanings of different Vs here are different, and the degree of head-down can be expressed by the following formula.

[0095] f1 = relu(θ1 - V θ1 ) + relu(baseP.y - V baseP ) + relu(sizeof(box) - V box )

[0096] f2 = relu(θ1 - v θ1 ) * relu(baseP.y - V baseP ) * relu(sizeof(box) - V box )

[0097] f = f1 + f2

[0098] Among them, baseP.y represents the height position of the reference point, and sizeof(box) represents the size of the box. They are derived features of the sitting posture representation R seq(i) . relu is an activation function that is well-known in the field and can also be replaced by tanh. f1 represents the feature unique value, f2 represents the feature co-value, and the larger the value of f, the more obvious the head-down feature.

[0099] For the discrimination confidence of any sitting posture type, we propose the following algorithm:

[0100]

[0101]

[0102]

[0103] f = sigmoid(w_norm * (f1 + f2 + f3))

[0104] where f1 represents the unique value of multiple derived features D R(n) among which D R(n) represents the nth derived feature of the sitting posture representation. For example, in the above-mentioned example of lowering the head, three derived features are used, and f a represents the activation function, which is commonly known in the field, such as functions like RELU, tanh, etc. W f1(n) is the direction coefficient of the nth derived feature, that is, for example, some features do not necessarily mean that the smaller the value, the greater the non-standard degree of the sitting posture, and it may also be the opposite. b f1(n) is the offset coefficient of the nth derived feature, such as the predefined threshold mentioned above. Among them, the unique value of the derived feature represents the feature value that appears alone in a certain sitting posture type. For example, when lowering the head, it often shows a change in the pitch angle, and the pitch angle is the so-called unique feature.

[0105] where f2 represents the co-occurrence value of K groups of derived features, and each group has M j co-occurring features, and W f2(j,n) represents the direction coefficient of the nth feature in the jth group, and b f2(j,n) represents its offset coefficient. The co-occurrence value of the derived features is a group of features with an associated relationship. For example, when lowering the head slightly to the lower right, there are both changes in the pitch angle and changes in the yaw rotation angle, etc.

[0106] where f3 represents the change value of the derived feature in time series. A certain period of time T is divided into T1 period, T2 period and T3 period according to a preset rule. Among them, the T1 period represents the current period, the T2 period represents the intermediate period, and the T3 period represents the past period. Typically, in this embodiment, T is divided into 100 equal parts, 0 to 50 equal parts are T1, 50 to 80 equal parts are T2, and 80 to 100 equal parts are T3. D R，t (n) represents the derived feature value at time t, and W f3(i)The weight value representing the derived eigenvalue. f3 can be understood as when there is a clear sitting posture transfer, the sitting posture representation extracted during the T3 period has a large change compared with the sitting posture representation extracted during the T1 period, while the change compared with the intermediate T2 period is smaller and in a steady state. That is, here it is set that the current T1 eigenvalue has a change amount d(T1, T3) compared with the past T3, and the change amount compared with the intermediate T2 is d(T1, T2), and its value is close to zero, and the typical value is d(T1, T2) < 0.2; during the implementation process, d(T1, T3) / d(T1, T2) > C is used as the condition for judging the sitting posture transfer, where C takes 2.0, 4.0, 8.0, etc. to represent the degree of sitting posture transfer respectively. Then, based on the above formula, the change value of the temporal derivative characteristic in the T period is obtained. In practical applications, the T3 period often takes the period when the sitting posture is correct. The change value of the derived feature is a comprehensive feature further derived by using operations on multiple original features, such as logical operations like "AND" and "OR".

[0107] f is the scaled sum of the first three, and then normalized to the probability space through a sigmoid function. For a certain type of sitting posture recognition, Table 1 shows the corresponding relationship between the sitting posture type and features in this invention.

[0108] Table 1: Sitting Posture Type and Corresponding Features

[0109]

[0110] Moving, unmanned, etc. are not within the discrimination range. For these two types, the output results of the sitting posture representation extraction model are used to judge whether there is a change in the box coordinates between the front and back frames. If it exceeds a certain degree, it is considered moving, and if the box is less than a certain degree, it is determined to be outside the discrimination area, so as to identify the moving state and unmanned state that are not discriminated. And the situation where the discrimination result is a correct sitting posture is considered to be the state where the above incorrect sitting posture types are not recognized.

[0111] After defining the feature appearance and co-occurrence groups, the probability of this type occurring can be realized by presetting appropriate W and b, that is, the respective direction coefficients and offset coefficients included in the aforementioned f1, f2, f3. For example, when f > 0.5, it is considered that this sitting posture type occurs.

[0112] Such as Figure 5As shown, the imaging area of the present invention is preferably divided into 9 (in specific implementation, it can also be divided into more). They are divided into a near area, a middle area, and a far area according to the distance from the image extraction device, and into a left and right area and a middle area according to the angle and the horizontal position of the user. According to the different positions of the box and the different yaw angles, the user is divided into different areas, and different parameter groups W and b are used for sitting posture discrimination. During implementation, W and b can either be preset empirically (for example, in the middle area a2, when acting on the features related to distance, the relevant factor in W is 1.0, then in the b1 area, the relevant factor in W should be less than 1.0, for example, taking 0.8), or randomly initialized and determined by self-adjustment through model training.

[0113] And some of the foregoing parameters need to be determined during operation. We adopt a tracking and sorting method to determine them, which is based on the fact that the user is in a normal sitting posture with a high probability. Specifically, for example, for the low position caused by the back not being straight, the determination of the reference standard height h is as follows:

[0114] 1) Take the sitting posture representations that have been historically determined to be correct sitting postures, determine the areas they are in, record their height positions, and form a set H for each area separately;

[0115] 2) Sort the set H, and take the highest h at the 90th percentile as the reference standard height.

[0116] Similar methods are also used for other dynamic parameters.

[0117] S5. Give corresponding reminders according to the recognized sitting posture type. After the reminder, judge the feedback actions, and construct positive sample packets and negative sample packets based on the change trends of the feedback actions. The positive sample packet includes all sitting posture representations within a period of time when the user's sitting posture shows a tendency to improve, and the negative sample packet includes all sitting posture representations within a period of time when the user's sitting posture does not show a tendency to improve.

[0118] The results of parameter presetting are affected by incremental learning. The main motivation is that the preset parameters are often the average values of the population, and there are individual experience differences. The method in this article uses the user's feedback actions and increments to update the preset parameters to solve the problems faced by existing methods. Specifically, in this step, corresponding reminders are given according to the different recognized sitting posture types. If the sitting posture is normal, no reminder is given, and for the rest, corresponding sound reminders are given through the speaker.

[0119] After the reminder, judge the feedback actions. For example, after reminding the user to lower their head, judge whether they have produced an action of raising their head. The reminder is usually based on continuous incorrect sitting postures within a period of time. After the reminder sound is emitted, if it is detected that the user's sitting posture shows a tendency to improve within a period of time, then all sitting posture representations that constitute the reminder during this period are recorded as positive samples S RThe P packet is usually triggered by corresponding post-adjustment parameters when a reminder is issued. For example, if there are 40 seconds of incorrect sitting postures in the past minute and the current sitting posture is incorrect, that is, there are multiple R ∈ S R P makes the aforementioned f confidence value greater than the decision threshold V f . As mentioned above, V f The typical value is 0.5, which is a concept of sensitivity. The smaller it is, the more sensitive it is, and the easier it is to determine a certain type of incorrect sitting posture. Take several historical sitting postures characterized as correct to form the negative sample packet S R N.

[0120] S6. Iteratively optimize the preset discrimination parameters based on the positive sample packet and the negative sample packet, so as to improve the accuracy of sitting posture type discrimination.

[0121] This step mainly performs incremental learning. The goal is to optimize and learn new W and b based on the noisy S R P and S R N to better perform discrimination. The meaning of "noisy" is that the judgments in the two sets are not completely accurate. The motivation is that the old or preset W and b are not the optimal experience parameters for individuals. For example, in the S R P, for the confidence value performance of the sitting posture representation, the average value may be only slightly higher than V f .

[0122] For one update of the incremental learning module, take T positive samples and the negative sample packet through multiple feedback results. Minimize the following objective function:

[0123]

[0124]

[0125]

[0126] Among them, p(y i |R; W, b) represents the probability that the sitting posture representation R is determined to be the sitting posture type y under the parameters W, b i , obtained based on the aforementioned calculation method of calculating the probability f that the sitting posture image frame to be recognized belongs to a certain sitting posture type, S R N(i, k) represents the kth sitting posture representation in the ith negative sample packet, S R P(i, j) represents the jth sitting posture representation in the ith positive sample packet, represents the number of sitting posture representations in the ith positive sample packet, represents the number of sitting posture representations in the jth negative sample packet.

[0127] Minimizing the above formula means selecting the sitting posture representation with the maximum confidence in the positive sample package to further maximize its confidence, and selecting the sitting posture representation with the minimum confidence in the negative sample package to further minimize its confidence.

[0128] Specifically, a mini-batch gradient descent method, such as an optimizer such as Adam, can be used until the objective function converges. Such methods are well known. The optimized W * ,b * The module then updates the old parameters to the optimal judgment parameters, and the subsequent judgments use these parameters. The subsequent recognition and reminders repeat this feedback cycle.

[0129] The above method primarily extracts sitting posture representations from videos, resolving the issue of individualized user experience differences caused by inherent experience thresholds, without requiring the user to enter a reference standard sitting posture for comparison. However, in preferred embodiments of the present invention, a user-defined reference standard sitting posture can also be entered into the above method to achieve an experience more tailored to the user's needs.

[0130] Specifically, in a preferred embodiment, the sitting posture recognition working mode can be divided into an automatic mode and a custom mode.

[0131] The automatic mode uses preset empirical parameters W and b to perform sitting posture recognition, and automatically iterates the parameters to a better one during user use.

[0132] The custom mode is to manually input a static sitting posture image of a specified type by the user, and the algorithm analyzes the sitting posture representation of the image data and records it as a reference sitting posture. In the incremental learning stage of work recognition, the selection of samples in the corresponding feature package is based on the input sitting posture representation. The static sitting posture referred to by the specified type can include a standard correct sitting posture. The user sits in the standard sitting posture that he or she subjectively believes, and then collects the image for input. This application records the sitting posture representation set of the correct sitting posture example input by the user as U R N, the set of incorrect sitting posture representations is recorded as: U R P

[0133] Minimize the following objective function:

[0134]

[0135]

[0136]

[0137]

[0138] in It is consistent with the above formula 2.1, except that SR Replace P with U R P, S R Replace N with U R N.

[0139] L2 considers the original S R Take J elements similar to U from P R The sitting posture representations similar to the elements in P are included in the loss calculation. From S R Take K elements similar to U from N R The sitting posture representations similar to the elements in N are included in the loss calculation.

[0140] The function similar_to(x, S) represents returning the indices of elements in x that are similar to those in S. The Euclidean distance or L1 distance is commonly used to measure the similarity between elements. λ is a preset balance parameter used to balance whether to take more custom inputs or more from the feedback. It can also be 0, that is, completely according to the sitting posture representations preset by the user.

[0141] The present invention also provides a sitting posture recognition system based on self-feedback learning, including:

[0142] A sitting posture representation definition module, which is used to define sitting posture representations that can directly or indirectly express the sitting posture type situation, and the sitting posture representations are obtained based on image frames;

[0143] A matching pair data construction module, which is used to obtain a discontinuous image data set and a continuous video frame data set, obtain the manual annotation results of each image frame in the image data set and the video frame data set, so as to construct matching pair data of the annotated image frames and sitting posture representations;

[0144] A sitting posture representation extraction module, which is used to use the matching pair data of the annotated image frames and sitting posture representations as training data to train a sitting posture representation extraction model, obtain a sitting posture image frame to be recognized, and process the sitting posture image to be recognized based on the trained sitting posture representation extraction model to obtain the sitting posture representation of the sitting posture image frame to be recognized;

[0145] A sitting posture type recognition module, which is used to implement sitting posture type discrimination based on the sitting posture representation of the sitting posture image frame to be recognized and preset discrimination parameters, and the discrimination parameters include a direction coefficient and an offset parameter;

[0146] A reminder module, which is used to give corresponding reminders according to the recognized sitting posture type, make a discrimination of the feedback action after the reminder, and construct a positive sample package and a negative sample package based on the change trend of the feedback action. The positive sample package includes all sitting posture representations within a period of time when the user's sitting posture has a tendency to improve, and the negative sample package includes all sitting posture representations within a period of time when the user's sitting posture has no tendency to improve;

[0147] A self-feedback module, which is used to iteratively optimize preset discrimination parameters based on the positive sample package and the negative sample package, so as to improve the accuracy of sitting posture type discrimination.

[0148] For the embodiments of the present invention, since it corresponds to the above embodiments, the description is relatively simple. For relevant similarities, please refer to the description in the above embodiments, and details will not be repeated here.

[0149] In the embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the units or modules can be in an electrical or other form.

[0150] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0152] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. And the aforementioned storage medium includes: USB flash drive, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other various media that can store program codes.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sitting posture recognition method based on self-feedback learning, characterized in that: include: A sitting posture representation that can directly or indirectly express the type of sitting posture is defined. The sitting posture representation is obtained based on an image frame. The sitting posture representation is: R seq =<box,keypoints,baseP,θ1,θ2,θ3> Among them, R seq Represents the sitting posture representation, box represents the bounding box of the human head, keypoints represents the key points of the human sitting posture, baseP represents the head reference point, θ1 represents the pitch angle, θ2 represents the left and right sway angle, θ3 represents the left and right rotation angle, and baseP is the coordinate reference origin of θ1, θ2, and θ2; Acquire a discontinuous image dataset and a continuous video frame dataset, obtain annotation results of each image frame in the image dataset and the video frame dataset, and thereby construct matching pair data of the annotated image frames and the sitting posture representation; Using the matching pair data of the annotated image frame and the sitting posture representation as training data to train a sitting posture representation extraction model, obtaining a sitting posture image frame to be identified, and processing the sitting posture image to be identified based on the trained sitting posture representation extraction model to obtain a sitting posture representation of the sitting posture image frame to be identified; Implementing sitting posture type discrimination based on the sitting posture representation of the sitting posture image frame to be identified and preset discrimination parameters, wherein the discrimination parameters include a direction coefficient and an offset parameter; Prompt the user accordingly based on the identified sitting posture type, identify the feedback action after the reminder, and construct positive and negative sample packages based on the changing trend of the feedback action. The positive sample package includes all the sitting posture representations during a period of time when the user's sitting posture has shown a trend of improvement, and the negative sample package includes all the sitting posture representations during a period of time when the user's sitting posture has not shown a trend of improvement. The preset discrimination parameters are iteratively optimized based on the positive sample package and the negative sample package. The confidence of the sitting posture representation with the maximum confidence in the positive sample package is further maximized, and the confidence of the sitting posture representation with the minimum confidence in the negative sample package is further minimized, thereby improving the accuracy of sitting posture type discrimination.

2. The method for sitting posture recognition based on self-feedback learning according to claim 1, characterized in that: The sitting posture representation extraction model includes a single-frame extraction model for extracting the sitting posture representation of a single frame of data in an image dataset, and the single-frame extraction model includes a first deep neural network module and a second deep neural network module; The input of the first deep neural network module is three-channel color image data obtained from single-frame data, and the output of the first deep neural network module serves as the input of the second deep neural network module. The output of the second deep neural network module includes the judgment result of whether there is a human body in the single-frame data and the sitting posture representation.

3. The method for sitting posture recognition based on self-feedback learning according to claim 2, characterized in that: The output of the second deep neural network module also includes auxiliary facial attribute discrimination results.

4. The method for sitting posture recognition based on self-feedback learning according to claim 2, characterized in that: The single-frame extraction model also includes a third deep neural network module and a fourth deep neural network module, the input of the third deep neural network module is 3D point cloud data obtained from the single-frame data, and the output of the third deep neural network module serves as the input of the second deep neural network module; The input of the fourth deep neural network module is the output of the first deep neural network module, and the output of the fourth deep neural network module is the regressed 3D point cloud data.

5. The method for sitting posture recognition based on self-feedback learning according to claim 2, characterized in that: The sitting posture representation extraction model also includes a time sequence frame extraction model for extracting the sitting posture representation of continuous frame data of the video frame dataset, and the time sequence frame extraction model includes an FA module and an RNN module for feature aggregation; The input of the FA module is the sitting posture representation of the current single frame data extracted by the single frame extraction model, the hidden layer output of the second deep neural network module when the sitting posture representation of the current single frame data is extracted by the single frame extraction model, and the hidden layer output of the second deep neural network module when the sitting posture representation of the previous single frame data is extracted by the single frame extraction model. The output of the FA module is the high-order feature of the sitting posture representation of the current frame. The high-order feature of the sitting posture representation of the current frame is cascaded with the hidden layer output of the second deep neural network module when the sitting posture representation of the current single frame data is extracted by the single frame extraction model as the input of the RNN module. The output of the RNN module is the sitting posture representation of the current frame of the video.

6. The method for sitting posture recognition based on self-feedback learning according to claim 1, characterized in that: The sitting posture type discrimination is realized based on the sitting posture representation of the sitting posture image frame to be identified and the preset discrimination parameters, including: Pre-set the unique raw features and co-occurrence raw features for each sitting posture type; Calculating derived features based on the sitting posture representation of the sitting posture image frame to be identified; Based on the preset discriminant parameters, the unique occurrence value of the derived features, the co-occurrence value of the derived features, and the change value of the derived features in time series are calculated respectively; The probability that the to-be-identified sitting posture image frame belongs to a certain sitting posture type is calculated based on the unique occurrence value of the derived features, the co-occurrence value of the derived features, and the temporal variation value of the derived features.

7. The method for sitting posture recognition based on self-feedback learning according to claim 6, characterized in that: Based on the preset discriminant parameters, the unique occurrence value of the derived feature, the co-occurrence value of the derived feature, and the change value of the derived feature in the time series are calculated respectively, including: The unique present value of the derived feature is calculated based on the following formula: Among them, f1 represents the unique value of multiple derived features, D R(n) represents the nth derived feature of the sitting posture representation, f a represents the activation function, W f1(n) represents the directional coefficient of the nth derived feature, b f1(n) is the offset coefficient of the nth derived feature; The co-occurrence value of the derived features is calculated based on the following formula: Among them, f2 represents the feature co-occurrence value of K groups of derived features, where each group has M j co-occurring features, W f2(j,n) represents the directional coefficient of the nth feature in the jth group, D R(j,n) represents the jth derived feature, b f2(j,n) represents the j-th derived feature offset coefficient; The change value of the derived feature over time is calculated based on the following formula: Where f3 represents the change value of the derived feature in the time series. A period of time T is divided into T1 period, T2 period and T3 period according to the preset rules. T1 period represents the present period, T2 period represents the middle period, and T3 period represents the past period. R,t (n) represents the derived characteristic value of period t, W f3(i) The weight value representing the derived feature value.

8. The method for sitting posture recognition based on self-feedback learning according to claim 6, characterized in that: Calculating the probability that the to-be-identified sitting posture image frame belongs to a certain sitting posture type based on the unique occurrence value of the derived feature, the co-occurrence value of the derived feature, and the temporal variation value of the derived feature, including: The probability that the image frame to be identified as sitting posture belongs to a certain sitting posture type is calculated based on the following formula: f=sigmoid(w_norm*(f1+f2+f3)) Among them, f represents the probability that the sitting posture image frame to be identified belongs to a certain sitting posture type, f1 represents the unique occurrence value of the derived feature, f2 represents the feature co-occurrence value of the derived feature, f3 represents the change value of the derived feature in time series, and w_norm represents the linear adjustment factor.

9. A sitting posture recognition system based on self-feedback learning, used to implement the sitting posture recognition method based on self-feedback learning as claimed in claim 1, characterized in that: include: A sitting posture representation definition module, which is used to define a sitting posture representation that can directly or indirectly express the type of sitting posture, wherein the sitting posture representation is obtained based on an image frame; a matching pair data construction module, which is used to obtain a non-continuous image dataset and a continuous video frame dataset, obtain the annotation results of each image frame in the image dataset and the video frame dataset, and thus construct matching pair data of the annotated image frames and the sitting posture representation; a sitting posture representation extraction module, configured to train a sitting posture representation extraction model using the matching pair data of the annotated image frame and the sitting posture representation as training data, obtain a sitting posture image frame to be identified, and process the sitting posture image to be identified based on the trained sitting posture representation extraction model to obtain a sitting posture representation of the sitting posture image frame to be identified; A sitting posture type recognition module, which is used to realize sitting posture type discrimination based on the sitting posture representation of the sitting posture image frame to be identified and preset discrimination parameters, wherein the discrimination parameters include a direction coefficient and an offset parameter; A reminder module is used to issue a corresponding reminder based on the identified sitting posture type, identify the feedback action after the reminder, and construct a positive sample package and a negative sample package based on the changing trend of the feedback action. The positive sample package includes all sitting posture representations of the user's sitting posture during a period of time when the user's sitting posture has shown a trend of improvement, and the negative sample package includes all sitting posture representations of the user's sitting posture during a period of time when the user's sitting posture has not shown a trend of improvement. The self-feedback module is used to iteratively optimize the preset discrimination parameters based on the positive sample package and the negative sample package, thereby improving the accuracy of sitting posture type discrimination.

Images