Reading monitoring method, device and equipment

Abstract

The invention provides a reading monitoring method, device and equipment, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a user reading image acquired by an image acquisition unit arranged at a preset calibration position; and inputting the user reading image into a pre-trained face recognition model to determine a face detection information set corresponding to the target user. And determining a distance set corresponding to each face key point based on the face detection information set and a key point distance calculation formula. And determining a deflection angle set corresponding to each face key point based on the face detection information set and a key point deflection angle calculation formula. The deflection angle set at least comprises deflection angles of a straight line formed by the selected face key point and other face key points relative to a preset direction. And based on the distance set, the deflection angle set and a reading posture comparison standard, determining whether the target user has a to-be-corrected posture, generating reading monitoring prompt information according to the to-be-corrected posture, and sending the reading monitoring prompt information to a reading monitoring alarm unit for alarm prompt.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a reading monitoring method, device and equipment. Background Art

[0002] In recent years, myopia has become a social problem that endangers the eye health of children and adolescents. The exact cause of myopia is still unclear, and most scholars believe that it is mainly related to genetic and environmental factors. Bad reading habits such as reading at close range and incorrect reading posture are considered to be important environmental factors leading to myopia.

[0003] At present, we have generally recognized the harm of long-term bad reading habits to eye health, but relying solely on personal awareness and self-control is far from enough. After all, people's habits and behavior patterns are difficult to change. For example, when we are reading or working, we often unconsciously shorten the distance between us and the book or computer screen.

[0004] Therefore, it is particularly important to use modern artificial intelligence technology to monitor bad reading habits. However, most existing technologies are designed for external wearable devices, which can easily affect their anti-myopia effect if worn improperly. In addition, they mainly monitor individual users, with high investment costs and low versatility.

[0005] Based on this, there is an urgent need for a technical solution that can efficiently, stably and cost-effectively monitor users' bad reading habits and assist users in healthy reading. Summary of the invention

[0006] The embodiments of the present application provide a reading monitoring method, device and equipment, which are used to solve the technical problems that the existing reading monitoring technology is not efficient and stable enough, has high cost, and affects the user experience.

[0007] On the one hand, an embodiment of the present application provides a reading monitoring method, the method comprising:

[0008] Acquire a user reading image acquired by an image acquisition unit disposed at a preset calibration position; wherein the preset calibration position is obtained based on a preset association relationship corresponding to a position of a reading object;

[0009] Inputting the user reading image into a pre-trained facial recognition model to determine a facial detection information set corresponding to the target user; wherein the facial detection information set at least includes effective detection area information and facial key point information;

[0010] Based on the facial detection information set and the key point distance calculation formula, determining a distance set corresponding to each facial key point; the distance set at least includes distance values ​​between a plurality of facial key points;

[0011] Based on the facial detection information set and the key point deflection angle calculation formula, determine the deflection angle set corresponding to each of the facial key points; the deflection angle set at least includes the deflection angle of the straight line formed by the selected facial key point and other facial key points relative to a preset direction;

[0012] Based on the distance set, the deflection angle set and the reading posture comparison standard, it is determined whether the target user has a posture to be corrected, so as to generate reading monitoring prompt information according to the posture to be corrected and send it to the reading monitoring alarm unit for alarm prompting.

[0013] In one implementation of the present application, the user reading image is input into a pre-trained facial recognition model to determine a facial detection information set corresponding to the target user, specifically including:

[0014] Determine the target user pixel area in the user reading image by using the pre-trained facial recognition model; wherein the facial recognition model is a multi-task convolutional neural network MTCNN model;

[0015] Determine the facial area and facial key points in the target user pixel area; wherein each facial key point includes at least key points corresponding to the left eye, the right eye, the nose, the left corner of the mouth, and the right corner of the mouth;

[0016] The upper left corner boundary coordinates, boundary length, and boundary height corresponding to the facial area are used as the effective detection area information, and the key point coordinates corresponding to each of the facial key points are used as the facial key point information to generate the facial detection information set.

[0017] In one implementation of the present application, before inputting the user reading image into a pre-trained facial recognition model, the method further includes:

[0018] Obtain user information corresponding to the target user; the user information includes at least the following user attributes: user age, user height, and user gender;

[0019] The user information is compared with a preset comparison standard association list to determine the reading posture comparison standard corresponding to the user information based on the comparison result; wherein the preset comparison standard association list includes the reading posture comparison standards corresponding to different attribute groups respectively; the reading posture comparison standard includes an interval set of posture association data of the user under the standard reading posture corresponding to the corresponding attribute group.

[0020] In one implementation of the present application, based on the facial detection information set and the key point distance calculation formula, determining the distance set corresponding to each facial key point specifically includes:

[0021] According to the key point coordinates of each of the facial key points in the facial detection information set, the Euclidean distances between the multiple facial key points are calculated using the key point distance calculation formula to obtain the distance values ​​corresponding to the multiple key point binary groups;

[0022] According to the correspondence between the key point binary group and the distance value, each distance value is added to the distance set in sequence.

[0023] In one implementation of the present application, based on the facial detection information set and the key point deflection angle calculation formula, determining the deflection angle set corresponding to each of the facial key points specifically includes:

[0024] Taking the selected facial key point as the coordinate origin, a coordinate axis vector group located on the same straight line in the preset direction is established within the effective detection area; the coordinate axis vector group includes a first vector and a second vector opposite to the first vector;

[0025] Determine, based on the face detection information set, key point vectors between the selected facial key points and other facial key points;

[0026] According to the key point deflection angle calculation formula, the coordinate axis vector group and each key point vector, the deflection angle of each key point vector relative to the preset direction is calculated to add each deflection angle to the deflection angle set.

[0027] In one implementation of the present application, before determining whether the target user has a posture to be corrected based on the distance set, the deflection angle set, and the reading posture comparison standard, the method further includes:

[0028] Acquire a plurality of reading posture-related data samples; wherein the plurality of reading posture-related data samples correspond to a plurality of different attribute groups with preset differences in user attributes; each of the reading posture-related data samples at least includes a marked posture type label; the posture type label at least includes a posture to be corrected and a standard reading posture;

[0029] Input each of the reading posture associated data samples corresponding to the same attribute group into the posture classification sub-model to be trained, so as to train the posture classification sub-model, until the output accuracy of the posture classification sub-model is greater than a first preset threshold, and determine the posture classification sub-model that has been trained; wherein one of the posture classification sub-models corresponds to one of the attribute groups, and includes one or more posture associated data corresponding to the corresponding reading posture comparison standard; and the reading posture comparison standard corresponds to at least one of the attribute groups;

[0030] The posture classification sub-model that has been trained corresponding to each of the reading posture comparison standards is used as the posture classification model.

[0031] In one implementation of the present application, based on the distance set, the deflection angle set, and the reading posture comparison standard, determining whether the target user has a posture to be corrected specifically includes:

[0032] Determining the reading posture comparison standard corresponding to the target user, so as to determine a matching posture classification sub-model in the posture classification model according to the reading posture comparison standard;

[0033] Through the calling interface of the matching posture classification sub-model, the posture association data in the distance set and the deflection angle set are input into the matching posture classification sub-model to determine the reading distance and reading posture of the target user, and judge whether the reading distance and the reading posture are the posture to be corrected according to the corresponding standard reading posture.

[0034] In one implementation of the present application, the method further includes:

[0035] Determine the number of posture corrections of the target user within a preset reading monitoring time;

[0036] When the number of posture corrections is greater than a second preset threshold, reducing the preset reading monitoring time according to a preset first gradient time adjustment rule; wherein the first gradient time adjustment rule includes a gradient adjustment time that is positively correlated with the preset reading monitoring time;

[0037] Otherwise, after the preset reading monitoring time is reached, based on the user's preset operation, a rest reminder message is generated or the preset reading monitoring time is increased.

[0038] On the other hand, an embodiment of the present application further provides a reading monitoring device, the device comprising:

[0039] An acquisition module, used to acquire a user reading image acquired by an image acquisition unit disposed at a preset calibration position; wherein the preset calibration position is obtained based on a preset association relationship corresponding to a position of a reading object;

[0040] An input module, used for inputting the user reading image into a pre-trained facial recognition model to determine a facial detection information set corresponding to the target user; wherein the facial detection information set at least includes effective detection area information and facial key point information;

[0041] A first determination module is used to determine a distance set corresponding to each facial key point based on the facial detection information set and a key point distance calculation formula; the distance set at least includes distance values ​​between a plurality of facial key points;

[0042] A second determination module is used to determine a set of deflection angles corresponding to each of the facial key points based on the facial detection information set and a key point deflection angle calculation formula; the deflection angle set at least includes a deflection angle of a straight line formed by a selected facial key point and other facial key points relative to a preset direction;

[0043] The third determination module is used to determine whether the target user has a posture to be corrected based on the distance set, the deflection angle set, and the reading posture comparison standard, so as to generate reading monitoring prompt information according to the posture to be corrected, and send it to the reading monitoring alarm unit for alarm prompting.

[0044] On the other hand, the embodiment of the present application further provides a reading monitoring device, the device comprising:

[0045] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a reading monitoring method as described above.

[0046] Compared with the prior art, the present invention has the following significant effects:

[0047] Through the above technical solution, the target user can monitor his / her reading without wearing any external equipment. The facial recognition model is used to process the user's reading image, thereby obtaining data related to the user's reading posture and monitoring the user's reading posture in real time. This application can monitor the user's bad reading habits efficiently, stably and at low cost, and assist the user in healthy reading. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0049] Figure 1 A schematic diagram of a flow chart of a reading monitoring method in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of processing a user reading image in a reading monitoring method in an embodiment of the present application;

[0051] Figure 3A schematic diagram of a user posture change in a reading monitoring method in an embodiment of the present application;

[0052] Figure 4 Another schematic diagram of processing a user's reading image in a reading monitoring method in an embodiment of the present application;

[0053] Figure 5 This is a structural schematic diagram of a reading monitoring device in an embodiment of the present application;

[0054] Figure 6 This is a structural schematic diagram of a reading monitoring device in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0056] The embodiments of the present application provide a reading monitoring method, device and equipment, which are used to solve the technical problems that the existing reading monitoring technology is not efficient and stable enough, has high cost, and affects the user experience.

[0057] The following describes in detail various embodiments of the present application in conjunction with the accompanying drawings.

[0058] The present application embodiment provides a reading monitoring method, such as Figure 1 As shown, the method may include steps S101-S105:

[0059] S101, a microcontroller acquires a user's reading image captured by an image capture unit disposed at a preset calibration position.

[0060] The preset calibration position is obtained based on a preset association relationship corresponding to the reading position. The preset association relationship can be preset by a user or a developer, that is, a corresponding relationship in which a distance should be maintained between the preset calibration position and the reading position. For example, if the reading position is at the first position of the desk, then the preset calibration position should be at the second position 30 cm away from the reading position to ensure that the user's reading image can be normally collected. The present application can also set a variety of preset association relationships, such as preset association relationships corresponding to 30 cm, 40 cm, 50 cm, etc.

[0061] In addition, when the distance between the preset calibration position and the reading position changes, a new reading posture comparison standard can be matched through a preset association relationship. That is, at different distances between the preset calibration position and the reading position, the microcontroller can use different reading posture comparison standards to verify whether the user is in a standard reading posture. This increases the flexibility of the device during reading monitoring, and the image acquisition unit does not need to be fixed in a fixed position for use, and different positions can be flexibly selected for placement for reading monitoring.

[0062] The above reading posture comparison standard may correspond to the posture association data of the user in the standard reading posture recorded by the microcontroller, such as the distance between the two eyes, the distance from the left eye to the nose, the distance from the right eye to the nose, the distance from the center of the two eyes to the nose, the distance from the nose tip to the left corner of the mouth, the distance from the nose tip to the right corner of the mouth, and the angle between the nose tip and other parts relative to the preset direction. The present application may also classify the reading posture by training a posture classification model so that it includes the numerical interval of the posture association data corresponding to the reading posture comparison standard. The posture classification model may be an artificial neural network model.

[0063] For example, if the distance between the preset calibration position and the reading position is 30 cm, this is the first preset association relationship, corresponding to the first reading posture comparison standard, and the numerical interval of the eye distance of the user reading image corresponding to the first reading posture comparison standard is [a1, b1]. The determination of the numerical interval can be determined by the microcontroller according to f / d=A*h / H, where f is the focal length of the image acquisition unit, d is the standard distance interval of the user's eye distance from the reading position preset by the user or developer, such as [d1, d2], A is the preset weight corresponding to the first preset association relationship, and different preset association relationships are set with different weights, which are specifically set by the user or developer according to the actual use scenario, and this application does not make specific restrictions on this. h is the eye distance (i.e., imaging distance) in the user's reading image, and H is the actual eye distance of the target user, and H can be preset. The specific value interval of h is obtained based on known data.

[0064] For another example, the distance between the preset calibration position and the reading position is 50 cm, and the numerical interval of the eye distance in the user reading image corresponding to the first reading posture comparison standard is [a2, b2]. When determining whether the user's reading posture is standard, the present application will compare the above-mentioned posture-related data to determine whether the data obtained by analyzing the user's reading image is within the numerical interval of the posture-related data under the standard reading posture, and determine the posture of the target user at this time based on the comparison result, such as leaning back, leaning forward, shaking the head left and right, tilting the head left and right, etc.

[0065] It should be noted that the microcontroller is used as the execution subject of the reading monitoring method for exemplary purposes only, and the execution subject is not limited to the microcontroller, and this application does not make specific limitations on this. The microcontroller is electrically connected to the image acquisition unit and the reading monitoring alarm unit, the image acquisition unit can be a camera, and the reading monitoring alarm unit can be a speaker or a light emitting diode, etc.

[0066] S102, the microcontroller inputs the user reading image into a pre-trained facial recognition model to determine a facial detection information set corresponding to the target user.

[0067] The facial detection information set includes at least effective detection area information and facial key point information.

[0068] In the embodiment of the present application, the user reading image is input into a pre-trained facial recognition model to determine a facial detection information set corresponding to the target user, specifically including:

[0069] The target user pixel area in the user reading image is determined by a pre-trained facial recognition model. The facial recognition model is a multi-task convolutional neural network (MTCNN) model. The facial area and facial key points in the target user pixel area are determined. Each facial key point includes at least the key points corresponding to the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth. The upper left corner boundary coordinates, boundary length, and boundary height corresponding to the facial area are used as effective detection area information, and the key point coordinates corresponding to each facial key point are used as facial key point information to generate a facial detection information set.

[0070] In other words, the present application uses the MTCNN model as a facial recognition model to extract a facial detection information set of a target user in a user reading image. The user reading image may contain multiple users, and the microcontroller will select the number of target users as target users according to the number of target users selected by the user operation, and determine the facial detection information set respectively, and each facial detection information set carries a corresponding target user identifier.

[0071] The MTCNN model can identify the face detection frame (i.e., the facial area) and five key points. Figure 2 As shown, the output facial area has the upper left corner boundary coordinate Box left_top (x0,y0), boundary length Box length and the border height Box height The area of ​​the facial region can be calculated based on the boundary length and boundary height. The key point coordinates at least include the left eye center coordinate L eye (x1,y1), right eye center coordinate Reye (x2,y2), the coordinates of the center points of both eyes Nose center coordinates C nose (x3,y3), the coordinate point of the left corner of the mouth is L mouth (x4,y4), the coordinate point of the right corner of the mouth is R mouth (x5, y5). The above coordinates are obtained by the rectangular coordinate system established on the plane where the facial area is located.

[0072] In the embodiment of the present application, before the user reading image is input into the pre-trained facial recognition model, the method further includes:

[0073] Obtain user information corresponding to the target user. The user information includes at least the following user attributes: user age, user height, and user gender. Compare the user information with the preset comparison standard association list to determine the reading posture comparison standard corresponding to the user information based on the comparison result. The preset comparison standard association list includes reading posture comparison standards corresponding to different attribute groups. The reading posture comparison standard includes an interval set of each posture association data of the corresponding attribute group corresponding to the user in the standard reading posture.

[0074] Among them, the user information of the target user can be pre-input into the microcontroller through the user terminal or the peripheral connected to the microcontroller. User information is such as [(age 9), (height 130), (gender male)]. Different user information may have different reading posture comparison standards. For example, users of different heights need to maintain a standard reading posture. The microcontroller will collect different posture association data of the angle between the nose tip and other key points. At this time, the microcontroller should monitor the reading of different target users according to different reading posture comparison standards. Different user attribute groups can be obtained by the developer based on a number of different attribute groups and sample data of standard reading postures by cluster analysis. The same attribute group corresponds to the same standard reading posture, that is, it has the same reading posture comparison standard, that is, the interval set corresponding to each posture association data is the same. The reading posture comparison standard corresponds to the interval set composed of each numerical interval corresponding to the posture association data of each dimension. The dimension of posture association data can be understood as, for example, the distance between the two eyes is one dimension, the angle between the nose tip and the center point of the left eye relative to the preset direction is one dimension...

[0075] In addition, user information can also be obtained by inputting an image of the target user in a standing state taken by an image acquisition unit into a pre-trained user information recognition model. The user information recognition model can be a machine learning model, which can be set in a cloud platform connected to the microcontroller network.

[0076] S103, the microcontroller determines a distance set corresponding to each facial key point based on the facial detection information set and the key point distance calculation formula.

[0077] The distance set at least includes distance values ​​between multiple facial key points.

[0078] In the embodiment of the present application, based on the facial detection information set and the key point distance calculation formula, the distance set corresponding to each facial key point is determined, specifically including:

[0079] According to the key point coordinates of each facial key point in the face detection information set, the Euclidean distance between multiple facial key points is calculated by the key point distance calculation formula to obtain the distance values ​​corresponding to multiple key point binary groups. According to the corresponding relationship between the key point binary group and the distance value, each distance value is added to the distance set in turn.

[0080] In other words, the present application can use the key point coordinates of each facial key point in the facial detection information set to perform Euclidean distance calculations between each selected facial key point, thereby obtaining the distance value between two facial key points.

[0081] For example, to calculate the interocular distance Calculate the distance from the left eye to the tip of the nose (i.e. the center point of the nose) Calculate the distance from the right eye to the tip of the nose Calculate the distance from the center of the two eyes to the tip of the nose Calculate the distance from the nose tip to the left corner of the mouth Calculate the distance from the nose tip to the right corner of the mouth The microcontroller can add the above distance value to the distance set and record the dimension of the posture-related data corresponding to each element (ie, the distance value) in the distance set.

[0082] S104, the microcontroller determines a set of deflection angles corresponding to each facial key point based on the facial detection information set and a key point deflection angle calculation formula.

[0083] The deflection angle set at least includes the deflection angle of the straight line formed by the selected facial key point and other facial key points relative to the preset direction.

[0084] It should be noted that when a user reads, the head may tilt forward and backward or left and right, which will affect the change of the above distance value. The reading posture cannot be well detected by the distance value alone. Figure 3 As shown, the head has changes in pitch angle, roll angle, yaw angle, etc. Based on this, the present application provides a technical solution for calculating the deflection angle.

[0085] Specifically, based on the facial detection information set and the key point deflection angle calculation formula, a deflection angle set corresponding to each facial key point is determined, specifically including:

[0086] Taking the selected facial key points as the coordinate origin, a coordinate axis vector group on the same straight line in the preset direction is established within the effective detection area. The coordinate axis vector group includes a first vector and a second vector opposite to the first vector. According to the facial detection information set, each key point vector between the selected facial key points and other facial key points is determined. According to the key point deflection angle calculation formula, the coordinate axis vector group and each key point vector, the deflection angle of each key point vector relative to the preset direction is calculated to add each deflection angle to the deflection angle set.

[0087] like Figure 4 As shown, the present application uses the nose tip of the selected facial key point as the coordinate origin, and establishes a coordinate axis vector group on the same straight line in the preset direction within the effective detection area 401, including the first vector and the second vector It is the tip of the nose C nose The vector from (x3,y3) to the left boundary (x0,y3), It is the tip of the nose C nose (x3,y3) to the right boundary (x0+Box length ,y3), The key point vector from the tip of the nose to the midpoint of the left eye is (x1-x3, y1-y3), the key point vector from the tip of the nose to the midpoint of the right eye is (x2-x3, y2-y3), the key point vector from the tip of the nose to the left corner of the mouth is (x4-x3, y4-y3), and the key point vector from the tip of the nose to the right corner of the mouth is (x5-x3, y5-y3).

[0088] The deflection angle is obtained by calculating the arc cosine function value of the first vector and the second vector respectively with the corresponding key point vector. For example, the arc cosine function value of the first vector and the key point vector from the nose tip to the left eye midpoint is the deflection angle corresponding to the nose tip and the left eye midpoint; for another example, the arc cosine function value of the second vector and the key point vector from the nose tip to the right corner of the mouth is the angle between the preset directions of the nose tip and the right corner of the mouth. In actual use, the deflection angle will be different when the user's face is in different postures. The user's reading posture can be monitored by this difference. Therefore, the application uses the deflection angle as posture-related data and calculates the deflection angle.

[0089] Further, the deflection angle is calculated as shown in the following table:

[0090]

[0091] S105, the microcontroller determines whether the target user has a posture to be corrected based on the distance set, the deflection angle set and the reading posture comparison standard, generates reading monitoring prompt information according to the posture to be corrected, and sends it to the reading monitoring alarm unit for alarm prompting.

[0092] The alarm prompt is issued by the reading monitoring alarm unit, such as a sound alarm prompt, which can instruct the user how to correct the posture to achieve the standard reading posture. The specific setting is made by the user or developer in the actual usage scenario, and this application does not make specific limitations on this.

[0093] That is to say, after the microcontroller of the present application obtains the distance set and the deflection angle set based on the above user reading image, it can judge whether the reading posture of the target user belongs to the posture to be corrected according to the above posture association data. Specifically, the microcontroller can determine the reading posture comparison standard, and determine the interval set of posture association data corresponding to the reading posture comparison standard, and judge whether the above posture association data conforms to the standard reading posture through each interval. If the posture association data corresponding to the above distance set and the deflection angle set are all within the corresponding interval, it belongs to the standard reading posture and there is no posture to be corrected. Otherwise, there is a posture to be corrected.

[0094] In the embodiment of the present application, before determining whether the target user has a posture to be corrected based on the distance set, the deflection angle set, and the reading posture comparison standard, the method further includes:

[0095] Acquire several reading posture associated data samples. Among them, several reading posture associated data samples correspond to multiple different attribute groups with preset differences in user attributes. Each reading posture associated data sample includes at least a marked posture type label. The posture type label includes at least a posture to be corrected and a standard reading posture. Input each reading posture associated data sample corresponding to the same attribute group into the posture classification sub-model to be trained to train the posture classification sub-model until the output accuracy of the posture classification sub-model is greater than the first preset threshold, and determine the trained posture classification sub-model. Among them, one posture classification sub-model corresponds to one attribute group, and includes one or more posture associated data corresponding to the corresponding reading posture comparison standard. The reading posture comparison standard corresponds to at least one attribute group. The trained posture classification sub-model corresponding to each reading posture comparison standard is used as the posture classification model.

[0096] In other words, the present application can train a posture classification model that can flexibly classify postures through reading posture-associated data samples corresponding to different attribute groups, wherein the posture classification model is composed of posture classification sub-models corresponding to multiple different reading posture comparison standards, and the posture classification sub-model can be an artificial neural network model. By training the model for posture classification, the data processing efficiency can be improved and the real-time reading monitoring process can be efficiently realized. The above-mentioned first preset threshold is set by the user or developer according to the actual usage scenario, and the present application does not make specific limitations on this.

[0097] In the embodiment of the present application, based on the distance set, the deflection angle set, and the reading posture comparison standard, determining whether the target user has a posture to be corrected specifically includes:

[0098] Determine the reading posture comparison standard corresponding to the target user, and determine the matching posture classification sub-model in the posture classification model according to the reading posture comparison standard. Through the calling interface of the matching posture classification sub-model, input the posture association data in the distance set and the deflection angle set into the matching posture classification sub-model to determine the reading distance and reading sitting posture of the target user, and judge whether the reading distance and reading sitting posture are postures to be corrected according to the corresponding standard reading posture.

[0099] That is to say, the microcontroller can obtain the target user's reading posture comparison standard through the technical solution of step S102, and match a posture classification sub-model according to the reading posture comparison standard as a matching posture classification sub-model. The matching posture classification sub-model is called to process each posture-related data in the distance set and the deflection angle set, and then each posture-related data is compared with the corresponding data comparison interval, and whether the target user's posture in the current user's reading image is the posture to be corrected is obtained according to the comparison result.

[0100] In the embodiment of the present application, in order to realize reading monitoring more intelligently, it also includes:

[0101] Determine the number of posture corrections of the target user within the preset reading monitoring time. When the number of posture corrections is greater than the second preset threshold, reduce the preset reading monitoring time according to the preset first gradient time adjustment rule. The first gradient time adjustment rule includes a gradient adjustment time that is positively correlated with the preset reading monitoring time. Otherwise, after reaching the preset reading monitoring time, generate a rest reminder message or increase the preset reading monitoring time based on the user's preset operation.

[0102] That is to say, the present application can pre-set a reading monitoring time, such as half an hour, one hour, etc. The reading monitoring time can be understood as a period of time for the user to read continuously. If the user does not take a break after exceeding the reading monitoring time, the reading monitoring alarm unit can remind the user to take a break. At the same time, the reading monitoring alarm unit can give different prompt signals to the user by emitting different sounds or different colors of light.

[0103] At the same time, the microcontroller can accumulate the number of posture corrections of the target user within the preset reading monitoring time, and compare the number of posture corrections with the second preset threshold value. The second preset threshold value can be set by the user during actual use, and this application does not make specific restrictions on this. If the number of posture corrections is greater than the second preset threshold value, the preset reading monitoring time can be gradually reduced, and the reduced time is obtained from the first gradient time adjustment rule. Different preset reading monitoring time lengths correspond to different gradient adjustment time lengths. The shorter the preset reading monitoring time length, the shorter the corresponding gradient adjustment time length.

[0104] If the number of posture corrections is not greater than the second preset threshold, the microcontroller can determine whether there is a user preset operation. The user preset operation can be understood as extending the preset reading monitoring time or generating a rest reminder message when the rest reminder condition is met. In other words, when the number of posture corrections is not greater than the second preset threshold, and the reading time reaches the preset reading monitoring time, the user preset operation is to generate a rest reminder message, then the rest reminder message is issued; the user preset operation is to increase the preset reading monitoring time, then the microcontroller increases the preset reading monitoring time, and the increased time can be set by the user or increased by the default time. This application does not make specific restrictions on this. If the user preset operation does not exist, the microcontroller generates a rest reminder message by default.

[0105] Through the above technical solution, the target user can monitor his / her reading without wearing any external equipment. The facial recognition model is used to process the user's reading image, thereby obtaining data related to the user's reading posture and monitoring the user's reading posture in real time. This application can monitor the user's bad reading habits efficiently, stably and at low cost, and assist the user in healthy reading.

[0106] Figure 5 A structural diagram of a reading monitoring device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the reading monitoring device 500 includes:

[0107] The acquisition module 501 is used to acquire the user reading image acquired by the image acquisition unit set at the preset calibration position. The preset calibration position is obtained based on the preset association relationship corresponding to the reading position. The input module 502 is used to input the user reading image into the pre-trained facial recognition model to determine the facial detection information set corresponding to the target user. The facial detection information set at least includes effective detection area information and facial key point information. The first determination module 503 is used to determine the distance set corresponding to each facial key point based on the facial detection information set and the key point distance calculation formula. The distance set at least includes the distance values ​​between multiple facial key points. The second determination module 504 is used to determine the deflection angle set corresponding to each facial key point based on the facial detection information set and the key point deflection angle calculation formula. The deflection angle set at least includes the deflection angle of the straight line formed by the selected facial key point and other facial key points relative to the preset direction. The third determination module 505 is used to determine whether the target user has a posture to be corrected based on the distance set, the deflection angle set, and the reading posture comparison standard, so as to generate reading monitoring prompt information according to the posture to be corrected, and send it to the reading monitoring alarm unit for alarm prompting.

[0108] The input module 502 is specifically used for:

[0109] The target user pixel area in the user reading image is determined by a pre-trained facial recognition model. The facial recognition model is a multi-task convolutional neural network MTCNN model. The facial area and each facial key point in the target user pixel area are determined. Each facial key point includes at least the key points corresponding to the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth. The upper left corner boundary coordinates, boundary length, and boundary height corresponding to the facial area are used as the effective detection area information, and the key point coordinates corresponding to each facial key point are used as the facial key point information to generate a facial detection information set.

[0110] The above device can also:

[0111] Obtain user information corresponding to the target user. The user information includes at least the following user attributes: user age, user height, and user gender. Compare the user information with the preset comparison standard association list to determine the reading posture comparison standard corresponding to the user information based on the comparison result. The preset comparison standard association list includes reading posture comparison standards corresponding to different attribute groups. The reading posture comparison standard includes an interval set of each posture association data of the corresponding attribute group corresponding to the user in the standard reading posture.

[0112] The first determining module 503 is specifically used for:

[0113] According to the key point coordinates of each facial key point in the face detection information set, the Euclidean distance between multiple facial key points is calculated by the key point distance calculation formula to obtain the distance values ​​corresponding to multiple key point binary groups. According to the corresponding relationship between the key point binary group and the distance value, each distance value is added to the distance set in turn.

[0114] The second determining module 504 is specifically used for:

[0115] Taking the selected facial key points as the coordinate origin, a coordinate axis vector group on the same straight line in the preset direction is established within the effective detection area. The coordinate axis vector group includes a first vector and a second vector opposite to the first vector. According to the facial detection information set, each key point vector between the selected facial key points and other facial key points is determined. According to the key point deflection angle calculation formula, the coordinate axis vector group and each key point vector, the deflection angle of each key point vector relative to the preset direction is calculated to add each deflection angle to the deflection angle set.

[0116] The above device can also:

[0117] Acquire several reading posture associated data samples. Among them, several reading posture associated data samples correspond to multiple different attribute groups with preset differences in user attributes. Each reading posture associated data sample includes at least a marked posture type label. The posture type label includes at least a posture to be corrected and a standard reading posture. Input each reading posture associated data sample corresponding to the same attribute group into the posture classification sub-model to be trained to train the posture classification sub-model until the output accuracy of the posture classification sub-model is greater than the first preset threshold, and determine the trained posture classification sub-model. Among them, one posture classification sub-model corresponds to one attribute group, and includes one or more posture associated data corresponding to the corresponding reading posture comparison standard. The reading posture comparison standard corresponds to at least one attribute group. The trained posture classification sub-model corresponding to each reading posture comparison standard is used as the posture classification model.

[0118] The third determination module 505 is specifically used for:

[0119] Determine the reading posture comparison standard corresponding to the target user, and determine the matching posture classification sub-model in the posture classification model according to the reading posture comparison standard. Through the calling interface of the matching posture classification sub-model, input the posture association data in the distance set and the deflection angle set into the matching posture classification sub-model to determine the reading distance and reading sitting posture of the target user, and judge whether the reading distance and reading sitting posture are postures to be corrected according to the corresponding standard reading posture.

[0120] The above device can also:

[0121] Determine the number of posture corrections of the target user within the preset reading monitoring time. When the number of posture corrections is greater than the second preset threshold, reduce the preset reading monitoring time according to the preset first gradient time adjustment rule. The first gradient time adjustment rule includes a gradient adjustment time that is positively correlated with the preset reading monitoring time. Otherwise, after reaching the preset reading monitoring time, generate a rest reminder message or increase the preset reading monitoring time based on the user's preset operation.

[0122] Figure 6 A structural diagram of a reading monitoring device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device includes:

[0123] At least one processor; and a memory in communication with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0124] A user reading image acquired by an image acquisition unit set at a preset calibration position is obtained. The preset calibration position is obtained based on a preset association relationship corresponding to a reading position. The user reading image is input into a pre-trained facial recognition model to determine a facial detection information set corresponding to a target user. The facial detection information set includes at least effective detection area information and facial key point information. Based on the facial detection information set and a key point distance calculation formula, a distance set corresponding to each facial key point is determined. The distance set includes at least distance values ​​between multiple facial key points. Based on the facial detection information set and a key point deflection angle calculation formula, a deflection angle set corresponding to each facial key point is determined. The deflection angle set includes at least a deflection angle of a straight line formed by a selected facial key point and other facial key points relative to a preset direction. Based on the distance set, the deflection angle set, and a reading posture comparison standard, it is determined whether the target user has a posture to be corrected, so as to generate reading monitoring prompt information according to the posture to be corrected, and send it to a reading monitoring alarm unit for an alarm prompt.

[0125] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0126] The devices and equipment provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and equipment also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and equipment will not be repeated here.

[0127] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0128] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A reading monitoring method, characterized in that: The method comprises: Acquire a user reading image acquired by an image acquisition unit disposed at a preset calibration position; wherein the preset calibration position is obtained based on a preset association relationship corresponding to a position of a reading object; Inputting the user reading image into a pre-trained facial recognition model to determine a facial detection information set corresponding to the target user; wherein the facial detection information set at least includes effective detection area information and facial key point information; Based on the facial detection information set and the key point distance calculation formula, determining a distance set corresponding to each facial key point; the distance set at least includes distance values ​​between a plurality of facial key points; Based on the facial detection information set and the key point deflection angle calculation formula, determine the deflection angle set corresponding to each of the facial key points; the deflection angle set at least includes the deflection angle of the straight line formed by the selected facial key point and other facial key points relative to a preset direction; Based on the distance set, the deflection angle set and the reading posture comparison standard, it is determined whether the target user has a posture to be corrected, so as to generate reading monitoring prompt information according to the posture to be corrected and send it to the reading monitoring alarm unit for alarm prompting.

2. A reading monitoring method according to claim 1, characterized in that: The user reading image is input into a pre-trained facial recognition model to determine a facial detection information set corresponding to the target user, specifically including: Determine the target user pixel area in the user reading image by using the pre-trained facial recognition model; wherein the facial recognition model is a multi-task convolutional neural network MTCNN model; Determine the facial area and facial key points in the target user pixel area; wherein each facial key point includes at least key points corresponding to the left eye, the right eye, the nose, the left corner of the mouth, and the right corner of the mouth; The upper left corner boundary coordinates, boundary length, and boundary height corresponding to the facial area are used as the effective detection area information, and the key point coordinates corresponding to each of the facial key points are used as the facial key point information to generate the facial detection information set.

3. A reading monitoring method according to claim 1, characterized in that: Before inputting the user reading image into the pre-trained facial recognition model, the method further includes: Obtain user information corresponding to the target user; the user information includes at least the following user attributes: user age, user height, and user gender; The user information is compared with a preset comparison standard association list to determine the reading posture comparison standard corresponding to the user information based on the comparison result; wherein the preset comparison standard association list includes the reading posture comparison standards corresponding to different attribute groups respectively; the reading posture comparison standard includes an interval set of posture association data of the user under the standard reading posture corresponding to the corresponding attribute group.

4. A reading monitoring method according to claim 1, characterized in that: Based on the facial detection information set and the key point distance calculation formula, determining the distance set corresponding to each facial key point specifically includes: According to the key point coordinates of each of the facial key points in the facial detection information set, the Euclidean distances between the multiple facial key points are calculated using the key point distance calculation formula to obtain the distance values ​​corresponding to the multiple key point binary groups; According to the correspondence between the key point binary group and the distance value, each distance value is added to the distance set in sequence.

5. A reading monitoring method according to claim 1, characterized in that: Based on the facial detection information set and the key point deflection angle calculation formula, determining the deflection angle set corresponding to each of the facial key points specifically includes: Taking the selected facial key point as the coordinate origin, a coordinate axis vector group located on the same straight line in the preset direction is established within the effective detection area; the coordinate axis vector group includes a first vector and a second vector opposite to the first vector; Determine, based on the face detection information set, key point vectors between the selected facial key points and other facial key points; According to the key point deflection angle calculation formula, the coordinate axis vector group and each key point vector, the deflection angle of each key point vector relative to the preset direction is calculated to add each deflection angle to the deflection angle set.

6. A reading monitoring method according to claim 1, characterized in that: Before determining whether the target user has a posture to be corrected based on the distance set, the deflection angle set, and the reading posture comparison standard, the method further includes: Acquire a plurality of reading posture-related data samples; wherein the plurality of reading posture-related data samples correspond to a plurality of different attribute groups with preset differences in user attributes; each of the reading posture-related data samples at least includes a marked posture type label; the posture type label at least includes a posture to be corrected and a standard reading posture; Input each of the reading posture associated data samples corresponding to the same attribute group into the posture classification sub-model to be trained, so as to train the posture classification sub-model, until the output accuracy of the posture classification sub-model is greater than a first preset threshold, and determine the posture classification sub-model that has been trained; wherein one of the posture classification sub-models corresponds to one of the attribute groups, and includes one or more posture associated data corresponding to the corresponding reading posture comparison standard; and the reading posture comparison standard corresponds to at least one of the attribute groups; The posture classification sub-model that has been trained corresponding to each of the reading posture comparison standards is used as the posture classification model.

7. A reading monitoring method according to claim 6, characterized in that: Based on the distance set, the deflection angle set, and the reading posture comparison standard, determining whether the target user has a posture to be corrected specifically includes: Determining the reading posture comparison standard corresponding to the target user, so as to determine a matching posture classification sub-model in the posture classification model according to the reading posture comparison standard; Through the calling interface of the matching posture classification sub-model, the posture association data in the distance set and the deflection angle set are input into the matching posture classification sub-model to determine the reading distance and reading posture of the target user, and judge whether the reading distance and the reading posture are the posture to be corrected according to the corresponding standard reading posture.

8. A reading monitoring method according to claim 1, characterized in that: The method further comprises: Determine the number of posture corrections of the target user within a preset reading monitoring time; When the number of posture corrections is greater than a second preset threshold, reducing the preset reading monitoring time according to a preset first gradient time adjustment rule; wherein the first gradient time adjustment rule includes a gradient adjustment time that is positively correlated with the preset reading monitoring time; Otherwise, after the preset reading monitoring time is reached, based on the user's preset operation, a rest reminder message is generated or the preset reading monitoring time is increased.

9. A reading monitoring device, characterized in that: The device comprises: An acquisition module, used to acquire a user reading image acquired by an image acquisition unit disposed at a preset calibration position; wherein the preset calibration position is obtained based on a preset association relationship corresponding to a position of a reading object; An input module, used for inputting the user reading image into a pre-trained facial recognition model to determine a facial detection information set corresponding to the target user; wherein the facial detection information set at least includes effective detection area information and facial key point information; A first determination module is used to determine a distance set corresponding to each facial key point based on the facial detection information set and a key point distance calculation formula; the distance set at least includes distance values ​​between a plurality of facial key points; A second determination module is used to determine a set of deflection angles corresponding to each of the facial key points based on the facial detection information set and a key point deflection angle calculation formula; the deflection angle set at least includes a deflection angle of a straight line formed by a selected facial key point and other facial key points relative to a preset direction; The third determination module is used to determine whether the target user has a posture to be corrected based on the distance set, the deflection angle set, and the reading posture comparison standard, so as to generate reading monitoring prompt information according to the posture to be corrected, and send it to the reading monitoring alarm unit for alarm prompting.

10. A reading monitoring device, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a reading monitoring method as described in any one of claims 1 to 8.

Images