Back posture inspection method and device, equipment and storage medium

Abstract

The invention provides a back posture checking method and device, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: determining a normal vector of each point in a back area according to a depth image of a to-be-detected human body; performing first detection on the back area according to the depth information of each point in the depth image to obtain a first detection result; performing second detection on the back area according to the normal vector of each point to obtain a second detection result; and determining a posture detection result of the back area according to the first detection result and the second detection result. Compared with the prior art, the problem that the accuracy of back posture collection is insufficient is solved.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular, to a method, device, equipment and storage medium for inspecting the back posture. Background Art

[0002] Back massage is a common physical massage method, which can effectively relieve back muscle fatigue and pain.

[0003] Before a massage robot massages a human back, it is necessary to collect a human back photo and depth information from above the human back, and then use the position and depth of the area to be massaged in the photo to guide the position of the massage end of the robotic arm, and the point cloud normal vector to guide the offset angle of the massage end of the robotic arm, simulating the human hand to massage the human back.

[0004] In the prior art, depth cameras generally select active light sensors, which are affected by sunlight, reflective materials, object slopes, etc., resulting in impaired accuracy. The above problems will cause problems in determining the massage area by the robot, affect the massage effect, and even cause physical injuries. Summary of the Invention

[0005] The purpose of the present application is to provide a method, device, equipment and storage medium for inspecting the back posture, aiming at the deficiencies in the prior art, so as to solve the problem of insufficient accuracy in collecting the back posture in the prior art.

[0006] To achieve the above purpose, the technical solutions adopted in the embodiments of the present application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for inspecting the back posture, the method including:

[0008] Determine the normal vector of each point in the back area according to the depth image of the human body to be detected;

[0009] Perform a first detection on the back area according to the depth information of each point in the depth image to obtain a first detection result;

[0010] Perform a second detection on the back area according to the normal vector of each point to obtain a second detection result;

[0011] Determine the posture detection result of the back area according to the first detection result and the second detection result.

[0012] In a second aspect, another embodiment of the present application provides an apparatus for inspecting the back posture, the apparatus including: a determination module and a detection module, wherein:

[0013] The determining module is configured to determine the normal vectors of each point in the back region according to the depth image of the human body to be detected;

[0014] The detecting module is configured to perform a first detection on the back region according to the depth information of each point in the depth image to obtain a first detection result; and perform a second detection on the back region according to the normal vectors of each point to obtain a second detection result;

[0015] The determining module is specifically configured to determine the posture detection result of the back region according to the first detection result and the second detection result.

[0016] In a third aspect, another embodiment of the present application provides an inspection device for back posture, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the inspection device for back posture runs, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the method according to any one of the above first aspects.

[0017] In a fourth aspect, another embodiment of the present application provides a storage medium, on which a computer program is stored. When the computer program is run by a processor, it performs the steps of the method according to any one of the above first aspects.

[0018] The beneficial effects of the present application are as follows: By using the inspection method for back posture provided by the present application, after obtaining the depth image of the human body to be detected, the normal vectors of each point in the back region are determined according to the depth image, and a first detection is respectively performed on the depth information of each point in the depth image, and a second detection is performed on the normal vector of each point, so as to determine the final posture detection result of the back region according to the first detection result and the second detection result. When the posture detection result of the back region indicates that the detection is passed, the massage robot will be controlled to perform subsequent actions. Otherwise, it is determined that there is a problem with the depth image of the human body to be detected obtained just now and normal massage cannot be performed. Such a setting method does not require collecting a data set for training, and can directly perform back posture detection based on the obtained depth image, which can adapt to people of different body types, and enables multiple-dimensional detection of the human body image to be detected before massage, and checks the rationality of the data in the depth image from the depth angle and the normal vector angle, so as to prepare for subsequent massage. Description of the Drawings

[0019] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and thus should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0020] Figure 1 Schematic flow chart of the back posture inspection method provided by an embodiment of the present application;

[0021] Figure 2 Schematic flow chart of the back posture inspection method provided by another embodiment of the present application;

[0022] Figure 3 Schematic structural diagram of the back posture inspection device provided by an embodiment of the present application;

[0023] Figure 4 Schematic structural diagram of the back posture inspection device provided by another embodiment of the present application;

[0024] Figure 5 Schematic structural diagram of the back posture inspection device provided by an embodiment of the present application. Detailed implementation manners

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application.

[0026] Generally, the components of the embodiments of the present application described and illustrated in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0027] In addition, the flowcharts used in the present application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations in the flowchart may not be implemented in sequence, and steps without logical context relationships may be reversed or implemented simultaneously. In addition, those of ordinary skill in the art can add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of the present application.

[0028] The following combines multiple specific application examples to explain a method for checking the back posture provided by the embodiments of the present application. Figure 1 It is a schematic flowchart of a method for checking the back posture provided by an embodiment of the present application. As Figure 1 shown, the method includes:

[0029] S101: Determine the normal vectors of each point in the back region according to the depth image of the human body to be detected.

[0030] Among them, the back region is the back region of the human body to be detected. In the embodiments of the present application, the application scenario of the present application can be, for example, after the human body to be detected lies prone on a preset bed surface, the massage robot takes a top-down photo of the human body to be detected from directly above the human body to be detected, and takes a depth image of the human body to be detected including the complete back of the human body to be detected. That is to say, the depth image of the human body to be detected can be, for example, a depth image collected after the back of the human body to be detected is photographed by the camera of the massage robot. Each pixel point in the depth image has its corresponding depth information, which is used to indicate the depth value (thickness value) of each point.

[0031] In some possible embodiments, for example, according to the depth image of the human body to be detected and the internal and external parameters of the camera of the massage robot for taking the depth image, according to the depth information corresponding to each pixel point in the depth image, the depth image is converted into point cloud data, and then for each point in the point cloud data, a plane is solved according to a preset iterative method to obtain the normal vector of each point; the preset iterative method can be, for example, RANdom SAmple Consensus.

[0032] The specific process of obtaining the normal vector of each point can be, for example, first select a point in the point cloud and randomly select two other points, and these three points form a triangle; then according to the coordinates of these three points, a plane is fitted using the least squares method; for all other points, calculate the distance from these points to the plane; in order to ensure that the fitted plane reflects the overall situation of the point cloud as much as possible, if the mean or median of the distances is less than a predetermined threshold, then accept the plane; after the plane is accepted, update the normal vector of the point according to the accepted plane.

[0033] S102: Perform a first detection on the back region according to the depth information of each point in the depth image to obtain a first detection result.

[0034] Among them, the first detection is to detect the depth information (depth value) of each point in the area of the back of the human body to be detected in the depth image, so as to obtain the depth values of the pixel points in the area of the back of the human body to be detected in the depth image, and to determine whether there is a depth drop in the current area of the back of the human body to be detected due to light, or the back area of the human body is smeared with massage oil or essential oil, or due to a steep side slope, etc.; among them, depth drop means that in image processing, the depth value of a pixel suddenly drops significantly, that is, the difference between the pixel values of adjacent pixel points in the image is too large, resulting in an obvious depth drop phenomenon in the image; in the embodiments of the present application, for example, due to reasons such as side slope jitter, the depth values of the pixels near the side slope in the depth image may suddenly change (the difference in depth values between adjacent pixel points is too large), thereby causing a depth drop phenomenon in the depth image.

[0035] If the first detection of the depth area indicates that there is a problem with the depth information of the current back area, it is determined that the first detection result fails.

[0036] S103: Perform a second detection on the back area according to the normal vectors of each point to obtain a second detection result.

[0037] Among them, the second detection is to detect the normal vector of each point in the back area after determining the normal vectors of each point, so as to determine whether there is an abnormality in the normal vectors of each point in the back area according to the normal vectors of each pixel point. If there is at least one abnormality in the normal vectors of each point, it is determined that the second detection fails. If there is no abnormality in the normal vectors of each point, it is determined that the second detection passes.

[0038] In the embodiments of the present application, after performing the first detection on the back area according to the depth information of each point in the depth image, it is also necessary to perform a second detection on the normal vectors of each point in the back area to determine the accuracy of the back area image acquisition and the normal vector acquisition result, and to determine whether there is an abnormality in the inclination angle of the normal vectors of each point in the back area, so as to ensure the safety and stability of subsequent massage and improve the accuracy of the detection result.

[0039] S104: Determine the posture detection result of the back area according to the first detection result and the second detection result.

[0040] In the embodiments of the present application, if at least one of the first detection result and / or the second detection result fails, it is determined that the posture detection result of the back area of the human body to be detected fails. That is, currently, due to incorrect lying posture of the human body to be detected or inaccurate depth, etc., the collected depth image of the human body to be detected cannot be applied, and it is necessary to re-collect the depth image of the human body to be detected.

[0041] By using the back posture inspection method provided in this application, after obtaining the depth image of the human body to be detected, the normal vectors of each point in the back region are determined according to the depth image, and the depth information of each point in the depth image is respectively subjected to a first inspection, and the normal vector of each point is subjected to a second inspection, so as to determine the final back region posture inspection result according to the first inspection result and the second inspection result. When the back region posture inspection result indicates that the inspection is passed, the massage robot will be controlled to perform subsequent actions. Otherwise, it is determined that there is a problem with the depth image of the human body to be detected obtained just now and normal massage cannot be performed. Such a setting method does not require collecting a data set for training, and can directly detect the back posture based on the obtained depth image, which can adapt to people of different body types, and enables multiple-dimensional detection of the human body image to be detected before massage, and checks the rationality of the data in the depth image from the depth angle and the normal vector angle, so as to prepare for subsequent massage.

[0042] Optionally, on the basis of the above embodiments, the embodiments of this application can also provide a back posture inspection method. The implementation process of the above method is illustrated below with reference to the accompanying drawings. Figure 2 It is a schematic flowchart of a back posture inspection method provided in another embodiment of this application, as Figure 2 shown, this method may further include:

[0043] S111: Generate a perspective transformation matrix for the back region of the human body to be detected according to the color image of the human body to be detected.

[0044] In the embodiments of this application, the method for determining the perspective transformation matrix may be, for example: determining a quadrilateral corresponding to the back region according to the color image of the human body to be detected and multiple target key points; generating a perspective transformation matrix for the back region of the human body to be detected according to the quadrilateral corresponding to the back region and a matrix image of a preset size; such a processing method can unify the target key points on each back region to a matrix image of a preset size regardless of whether the quadrilateral composed of the target key points on multiple back regions is the same in size or regular in size, so as to make subsequent processing more convenient.

[0045] In some possible embodiments, the color image of the human body to be detected may be, for example, an RGB image, and the multiple target key points may be, for example, multiple human body key points, and the multiple human body key points may be, for example: key points of other human body parts such as the left shoulder, right shoulder, and buttocks, or key points of human body acupoints, etc.

[0046] The method for collecting target key points can be, for example, inputting the color image of the human body to be detected into a human body key point detection model. The human body key point detection model is trained with a large number of sample human body images containing multiple target key point identifiers. After training, the human body key point model can directly determine the information of multiple target key points included in the color image based on the color image including the back region of the human body. That is to say, after detecting the color image according to the human body key point detection model provided in this application, the information of multiple target key points in the color image is output, such as the position information of multiple target key points in the color image. It should be understood that the above embodiments are only for illustrative purposes. The specific content included in the target key points, the method for determining the target key points in the color image, etc. can all be flexibly adjusted according to user needs and are not limited to those given in the above embodiments.

[0047] S102 may include:

[0048] S112: Perform a first detection on the back region according to the perspective transformation matrix and the depth information of each point to obtain a first detection result.

[0049] In some possible embodiments, the determination method of the first detection result can be, for example: perform a conversion process on the depth information of each point according to the perspective transformation matrix to obtain a first grid; where each point in the first grid has its corresponding depth information; and the first grid includes multiple sub-grids; then, based on the depth information of each point in the first grid, determine the first eigenvalue of each sub-grid in the first grid; perform a first detection on the first grid according to the first eigenvalue of each sub-grid in the first grid to obtain a first detection result.

[0050] That is to say, in the embodiments of this application, first, it is necessary to perform a conversion process on the depth information of each point according to the perspective change matrix to convert the depth information of each point onto a matrix image of a preset size. Then, based on the grid of a preset unit, the converted depth information is divided into grids to divide the converted depth information into multiple sub-grids. Each sub-grid contains multiple points and the depth information corresponding to the multiple points. According to the depth information corresponding to each point in each sub-grid, the first eigenvalue corresponding to each sub-grid is determined.

[0051] The determination method of the first eigenvalue of each sub-grid can be, for example: according to the depth information of each point in each sub-grid, sort the depth information in each sub-grid in descending order. Among the sorted depth information, take the depth information ranked 10% and the depth information ranked 90%, and take the difference between the above two depth information as the first eigenvalue of the sub-grid; it should be understood that the above embodiments are only illustrative. For example, the determination method of the first eigenvalue of each specific sub-grid can also be to take the average value of the depth information in the sub-grid as the first eigenvalue, or, after sorting from large to small, take the difference between the depth information ranked 20% and the depth information ranked 80% as the first eigenvalue of the sub-grid (of course, the depth information of 15% and 85%, or 25% and 75%, or any other combination of depth information, etc. can also be taken). The determination method of the first eigenvalue of each specific sub-grid can be flexibly adjusted according to user needs and is not limited to those given in the above embodiments.

[0052] In the embodiments of the present application, if the number of valid points in a sub-grid is less than a preset percentage, for example, less than 90%, then the eigenvalue of the sub-grid is determined to be 0, where a valid point is a point with depth information, and an invalid point is a point without depth information.

[0053] The determination method of the first detection result can be, for example, to divide the first grid into regions. Based on the central axis of the first grid, the first grid is divided into a left half-grid and a right half-grid. Since the human back region is generally higher on both sides and lower in the middle, after dividing the first grid based on its central axis, the human back region of the left half-grid generally shows a decreasing trend from left to right, and the human back region of the right half-grid generally shows a decreasing trend from right to left; that is, generally, the eigenvalue of the left sub-grid in the left half-grid is greater than that of the right sub-grid; and the eigenvalue of the right sub-grid in the right half-grid is generally greater than that of the left sub-grid.

[0054] Therefore, the determination method of the difference between adjacent sub-grids in the left grid is different from that between adjacent sub-grids in the right grid. In the embodiments of the present application, the detection method for each sub-grid in the left half-grid is: for each sub-grid in the left half-grid, the difference calculation method between adjacent grids is to subtract the eigenvalue of the right sub-grid from the eigenvalue of the left sub-grid in the adjacent sub-grids; for each sub-grid in the right half-grid, the difference calculation method between adjacent grids is to subtract the eigenvalue of the left sub-grid from the eigenvalue of the right sub-grid in the adjacent sub-grids; then, according to the size relationship between the difference and the preset first eigenvalue threshold, it is determined whether each sub-grid passes the first detection.

[0055] For example, for the left - hand side grid, if the difference in eigenvalue between two adjacent sub - grids is less than a preset first eigenvalue threshold, it indicates that there is no pattern of significant eigenvalue change from left to right for the left - hand side grid (since the general trend of the left - hand side of the back region is lower from left to right, i.e., the left side is higher than the right side). Then, it is determined that the first detection fails. If the difference is greater than or equal to the preset first eigenvalue threshold, it is determined that the first detection passes. Similarly, for the right - hand side grid, when the difference in eigenvalue between adjacent sub - grids is less than the first preset eigenvalue threshold, it also indicates that the current two adjacent sub - grids fail the detection. When the difference in eigenvalue between adjacent sub - grids is greater than or equal to the first preset eigenvalue threshold, it indicates that the current two adjacent sub - grids pass the detection.

[0056] For example, the size of the first grid is a 10 * 10 grid, which is divided into two grid combinations of 10 * 5 on the left and right respectively. For each sub - grid within the left - hand side 10 * 5, when determining the difference in eigenvalue between adjacent sub - grids, the calculation method is to subtract the first eigenvalue of the right - hand sub - grid from the first eigenvalue of the left - hand sub - grid, and compare the calculation result of the eigenvalue difference with the preset eigenvalue threshold. For each sub - grid combination within the right - hand side 10 * 5, when determining the difference in eigenvalue between adjacent sub - grids, the calculation method is to subtract the first eigenvalue of the left - hand sub - grid from the first eigenvalue of the right - hand sub - grid, and compare the eigenvalue difference result with the preset eigenvalue threshold. Among the comparison results of the above two eigenvalue differences, if there is at least one difference result less than the preset first eigenvalue threshold, it is determined that the current depth image fails the first detection. If each of the two difference results is greater than or equal to the preset first eigenvalue threshold, it is determined that the current depth image passes the first detection.

[0057] In the embodiments of the present application, to ensure the effectiveness of the detection results, for example, it is first necessary to perform an effectiveness detection on each point in the depth image to filter out the invalid points in the depth image. Among them, the points in the depth image that do not have depth information are invalid points. That is to say, before detecting the depth image, it is necessary to first filter out (remove) the invalid points without depth information in the depth image, and only retain the valid points with depth information, and perform subsequent detections based on the valid points in the depth image.

[0058] In some other possible embodiments, the method for determining the first detection result may also be, for example: obtaining the body thickness information of the human body to be detected according to the depth image to be detected; filtering the depth image according to the body thickness information to obtain a filtered depth image; performing a first detection on the back region according to the depth information of each point in the depth image and the depth information of each point in the filtered depth image to obtain the first detection result.

[0059] That is to say, in the embodiments of the present application, the first thickness mean value of the back region in the depth image can be determined according to the depth information of each point in the depth image; the second thickness mean value of the back region in the filtered depth image can be determined according to the depth information of each point in the filtered depth image; and the back region can be subjected to a first detection according to the difference between the first thickness mean value and the second thickness mean value to obtain a first detection result.

[0060] In some possible embodiments, if the difference between the first thickness mean value B1 and the second thickness mean value B2 is greater than a preset threshold, it is considered that the first detection result of the depth image fails the first detection; if the difference between the first thickness mean value B1 and the second thickness mean value B2 is less than or equal to the preset threshold, it is considered that the first detection result of the depth image passes the first detection.

[0061] Among them, the method of filtering the depth image can be, for example: filtering each sub-region in the back region of the human body to be detected in the depth image respectively. Among them, in order to improve the filtering accuracy, different sub-regions may correspond to different filtering methods.

[0062] That is, according to the human body thickness information, the range of each sub-region in the back region is determined; and the depth image to be detected is filtered according to the depth information of each sub-region in the depth image to be detected to obtain a filtered depth image.

[0063] Among them, the method for determining the human body thickness information in the above method can be, for example: according to the depth image to be detected, the mask of the back region and the plane mask where the back region is located are determined; and the human body thickness information is determined according to the mask of the back region, the plane mask, and the depth image to be detected.

[0064] Specifically, according to the first depth information of each point, the first average depth information is calculated; according to the plane mask, the second depth information of each point is extracted from the depth image to be detected; according to the second depth information of each point, the second average depth information is calculated; and the human body thickness information is determined according to the first average depth information and the second average depth information.

[0065] That is to say, in the embodiments of the present application, the method for determining the body thickness information of the back region of the human body to be detected may be as follows: First, for the depth image of the human body to be detected, a preset plane segmentation method can be used to extract the depth information belonging to the bed surface in the depth image, and then the depth information of the depth image and the bed surface is removed to segment out the point cloud data corresponding to the human body to be detected, and the point cloud data corresponding to the human body to be detected is projected onto a 2D plane to obtain a mask M1 of the back region of the human body to be detected; Subsequently, according to a preset extraction rule, for example, the depth information within 50% of the center position of the mask M1 of the back region can be extracted as the back center region M2 of the human body to be detected (it should be understood that the above embodiments are only for illustrative purposes, and specifically, the depth information within 20%, 25%, or 30% of the center can also be extracted, or the preset extraction rule can also be to determine the region of the quadrilateral formed by the target feature points as the back center region, etc., which can be flexibly adjusted according to user needs and is not limited to the above embodiments), and the mean value B1 of the depth information of the back center region M2 is determined. Then, the mean value of the depth information of the bed surface is obtained by calculating the mean value of the bed surface mask except for the mask of the back region of the human body to be detected. By taking the difference between the mean value of the depth information of the back center region and the mean value of the bed surface depth information, the difference result is the body thickness information.

[0066] Among them, the division method of different regions of the back of the human body to be detected may be as follows: Calculate the height occupied by the edge of the human body to be detected according to the body thickness information. According to the height occupied by the edge of the human body to be detected, the point cloud data belonging to the human body to be detected in the depth image is re-segmented. The point cloud in the upper region (that is, the point cloud above the bed point cloud is determined as the point cloud in the upper region) is projected onto a 3D plane to obtain a mask M3 of the back center region of the human body to be detected, and the edge region M4 of the human body to be detected (where M4 = M1 - M3). Extract the region that intersects the edge region M4 of the human body to be detected and the mask M3 of the back center region of the human body to be detected, and has a certain preset area and covers the boundary line of the center region as the region to be fused M5.

[0067] Still taking the back region including the planar region (bed surface region) where the back region is located, the back edge region, the back edge region M4, and the region M5 to be fused between the back center regions as an example for illustration. In the embodiments of the present application, the filtering method can be, for example, a multi-core iterative filtering method. First, in the back region of the human body to be detected, the depth information of the points exceeding the depth threshold is replaced with the bed surface depth information. Among them, the above method of replacing the depth information of the points exceeding the depth threshold with the bed surface depth information is a method for processing interference points. The depth threshold can be adjusted according to the actual bed surface depth. For example, when the actual bed surface depth is 0.8 cm, the depth threshold can be preset to 1 cm to filter out the depth information belonging to the bed surface in the depth information. It should be understood that the above embodiments are only for illustrative purposes, and the setting of the specific depth threshold can also be any depth threshold such as 2 cm, 3 cm, 5 cm, or 10 cm, etc. It can be preset according to the actual bed surface depth, as long as the depth threshold always remains greater than or equal to the actual bed surface depth, and it is not limited to the above embodiments.

[0068] For the determination of the depth information of each point in the region M5 to be fused, it is determined after fusing the original depth information and the filtered depth information. For the depth information of each point in the part of the back edge region M4 that is not M5, the original depth information can be directly retained; after each filtering, calculate the difference between the back edge region before and after filtering. If the difference is large (for example, when it is greater than the preset difference threshold), it is considered that the current back edge is not smooth and needs to continue filtering; otherwise, it is considered that the current back edge is smooth and does not need to continue filtering; after repeating the above steps of filtering and fusion processing multiple times, the filtered depth image is obtained.

[0069] Exemplarily, in some possible embodiments of the present application, the method of fusing the original depth information and the filtered depth information can be, for example, fusing according to a preset weight. The preset weight can be determined according to the position of each point, and the weights corresponding to different point positions are different. The specific configuration method of the preset weight can be: the weight of the original depth information of the part closer to the back edge region is greater, and the weight of the filtered depth information of the part closer to the back center region is greater. The sum of the weight of the original depth information and the weight of the filtered depth information is 1. It should be understood that the above embodiments are only for illustrative purposes, and the setting of the specific fusion method can be flexibly adjusted according to user needs, and it is not limited to the above embodiments. It can also set a weight for all points in the back edge region and a weight for all points in the back center region, and determine different weights for fusion according to different points corresponding to different regions, and it is not limited to the above embodiments.

[0070] In an embodiment of the present application, in order to ensure the accuracy of the detection result and improve the safety during the user's use process, when detecting the depth information of each point in the depth image, it is jointly determined based on the depth information of each point in the depth image before and after filtering, the perspective transformation matrix, and the depth information of each point. Only when both of the above two detections pass, it is determined that the depth image passes the first detection; otherwise, as long as at least one of the above two detections fails, it is determined that the depth image fails the first detection; it should be understood that the above embodiment is only for exemplary illustration, and specifically, whether the first detection information is determined based on the depth information of each point in the depth image before and after filtering; or based on the perspective transformation matrix and the depth information of each point; or based on the depth information of each point in the depth image before and after filtering, the perspective transformation matrix, and the depth information of each point jointly; can be flexibly adjusted according to user needs and is not limited by the above embodiment.

[0071] Correspondingly, S103 may include:

[0072] S113: Perform a second detection on the back region according to the perspective transformation matrix and the normal vector of each point to obtain a second detection result.

[0073] In the embodiment of the present application, the method for obtaining the second detection result may be, for example: performing a transformation process on the normal vector of each point according to the perspective transformation matrix to obtain a second grid; wherein, each point in the second grid has a corresponding normal vector; determining the second eigenvalue of each sub-grid in the second grid according to the normal vector with the largest included angle with the preset reference direction; performing a second detection on the second grid according to the second eigenvalue of each sub-grid in the second grid to obtain a second detection result.

[0074] It should be noted that the method for determining the second eigenvalue of each sub-grid in the second grid may be, for example: for each sub-grid, determining the target normal vector with the largest included angle with the vertical direction of the back of the human body to be detected, and using the included angle between the target normal vector and the vertical direction of the back of the human body to be detected as the second eigenvalue of the sub-grid.

[0075] In the embodiment of the present application, if the number of valid points in a sub-grid is less than a preset percentage, for example, less than 90%, then it is determined that the eigenvalue of the sub-grid is 0, where a valid point is a point with depth information, and an invalid point is a point without depth information.

[0076] For example, the method for determining the second detection result may be as follows: divide the second grid into regions. Based on the central axis of the second grid, divide the second grid into a left half grid and a right half grid. Since the human back region is generally higher on both sides and lower in the middle, after dividing the second grid based on its central axis, the human back region in the left half grid generally shows a decreasing trend from left to right, and the human back region in the right half grid generally shows a decreasing trend from right to left. According to the magnitude relationship between the eigenvalue differences of adjacent sub - grids within the left half grid and the preset second eigenvalue threshold, and the magnitude relationship between the eigenvalue differences of adjacent sub - grids within the right half grid and the preset second eigenvalue threshold, determine whether the second detection is passed according to the magnitude of the difference from the preset second eigenvalue threshold. For example, if the difference is less than the preset second eigenvalue threshold, it is determined that the second detection fails; if the difference is greater than or equal to the preset second eigenvalue threshold, it is determined that the second detection passes.

[0077] In another embodiment of the present application, the method for obtaining the second detection result may also be, for example: among the normal vectors of each point, determine the normal vector corresponding to the target key point; where the target key point may be, for example, multiple human key points, such as human acupoint key points, or human body part key points, etc., and each target key point has a preset normal vector; according to the angle between the normal vector of each target key point and the preset normal vector, perform a second detection on the back region to obtain the second detection result.

[0078] Among them, the method for determining the preset normal vector corresponding to the target key point may be, for example: according to the normal vectors of each target key point in a large number of sample depth images that pass the first detection and the second detection in the current application scenario, determine the average value of the normal vectors of each target key point, and determine that the average value of the normal vectors of each target key point is the preset normal vector of each target key point.

[0079] The method for performing the second detection on the back region may be, for example, respectively determine the angle between the normal vector of each target key point in the back region and the preset normal vector of this target key point, determine whether the angle value of the angle of each target key point is greater than the preset angle threshold. If there is at least one target key point among the target key points whose angle is greater than the preset angle threshold, it indicates that there is a problem with the normal vector of the depth image of the currently to - be - detected human body, and it is determined that the depth image of the currently to - be - detected human body fails the second detection; if for each target key point among the target key points, the angle of each target key point is less than or equal to the preset angle threshold, it indicates that there is no problem with the normal vector of the depth image of the currently to - be - detected human body, and the second detection is passed.

[0080] In one embodiment of the present application, in order to ensure the accuracy of the detection results and improve the safety during user operation, when performing the second detection on the normal vectors of each point in the depth image, it is determined jointly based on the angle between the normal vector of each target key point and the preset normal vector, as well as the perspective transformation matrix and the normal vectors of each point. Only when both of the above two detections pass can it be determined that the depth image passes the second detection; otherwise, if at least one of the above two detections fails, it is determined that the depth image fails the second detection. It should be understood that the above embodiment is only for illustrative purposes. Specifically, whether the second detection information is determined based on the angle between the normal vector of each target key point and the preset normal vector; or based on the perspective transformation matrix and the normal vectors of each point; or based on the angle between the normal vector of each target key point and the preset normal vector, as well as the perspective transformation matrix and the normal vectors of each point; can be flexibly adjusted according to user needs and is not limited to the above embodiment.

[0081] For the convenience of understanding the present application, the following uses a complete embodiment to explain the present application:

[0082] First, perform human back region segmentation on the depth image collected by the robot. The segmentation method can be, for example: use the plane segmentation method to extract the depth information of the bed surface part in the depth image, and then segment out the point cloud of the human body to be detected in the depth image, and project the segmented point cloud of the human body to be detected onto the 2D plane to obtain the back region mask M1 of the human body to be detected.

[0083] Subsequently, based on the preset extraction rules, divide the back region mask M1 of the human body to be detected to obtain the depth information of the back center region M2 of the human body to be detected. According to the depth information of each point cloud in M2, determine the mean value B1 of the depth information within the range of the M2 region, and according to the depth information corresponding to the bed surface mask, determine the mean value of the depth information within the range of the bed surface region, and calculate the difference between the mean value of the bed surface depth information and the mean depth value of the back center region of the human body to be detected to obtain the human body thickness information.

[0084] According to the human body thickness information calculated in the above steps, calculate the approximate height occupied by the human body edge of the human body to be detected. According to this segmentation height, re-segment the point cloud in the depth image, and project the point cloud in the upper region (the point cloud in the region above the bed surface) onto the 2D plane to obtain the back center region mask M3 of the human body to be detected. The human body edge region of the human body to be detected is M4 (M1 - M3), and extract the region with a certain area at the junction of the human body edge region of the human body to be detected and the back center region of the human body to be detected, which covers the boundary line of the center region, as the region to be fused M5.

[0085] The color image of the human body to be detected is input into a human body key point detection model to obtain human body key points, including the left shoulder, right shoulder, and hip key points. Calculate the quadrilateral frame where the back is located, and solve the perspective transformation matrix W for converting it to a fixed-size rectangular image. In the embodiments of the present application, to ensure the accuracy of the data, the four vertices of the back quadrilateral frame can be respectively selected as: the left shoulder, the right shoulder, the hip shifted left by about half of the left-right shoulder distance, and the hip shifted right by about half of the left-right shoulder distance. It should be understood that the above embodiments are only illustrative. The selection of the four vertices of the specific back quadrilateral frame can also be determined according to multiple preset acupoints, or determined according to other key points, and is not limited to those given in the above embodiments.

[0086] According to the region division result in the above steps, use a multi-core iterative filtering method. First, replace the depth information of the points in the depth image whose depth information exceeds the preset depth threshold with the depth information of the bed surface; in M5, fuse the original depth and the filtered depth. For the depth information of each point in other regions except the M5 region in M4, keep the original depth information unchanged. After each filtering, calculate the difference between the filtered and unfiltered human body edge regions. If the difference is large, it is considered that the edge is not smooth and needs filtering, otherwise the edge is smooth and does not need filtering. Repeat the filtering and fusion process multiple times.

[0087] Subsequently, in the embodiments of the present application, for the inspection of depth information, for example, the depth of the human back can be converted to WD by the perspective transformation matrix W, and then the WD is grid-divided to obtain the first grid. Sort the depth information of each point in each sub-grid of the first grid from small to large. For the depth information in each sorted sub-grid, take the difference between the 10%th depth information and the 90%th depth information as the first eigenvalue of the sub-grid. If the number of valid points in a sub-grid in the first grid is less than 90%, the eigenvalue is 0. For the left half of the first grid, subtract the first eigenvalue of the right grid from the first eigenvalue of the left grid; for the right half of the first grid, subtract the first eigenvalue of the left grid from the first eigenvalue of the right grid; if the subtraction value between any two grids is less than a certain threshold, it is considered that there is a problem, and it is considered that the depth information fails the first detection.

[0088] In addition, the present application also extracts the depth mean value B2 in M2. If the difference between B2 and B1 is greater than a certain threshold, it is considered that there is a problem, and it is considered that the depth information fails the first detection.

[0089] For the detection of the normal vector, in the embodiments of the present application, for example, the depth image can be converted into a point cloud by combining the internal parameters of the camera, and the plane is solved for each point on the point cloud by the ransac method to obtain the normal vector of this point.

[0090] Subsequently, the depth of the human back is transformed by the perspective transformation matrix W to obtain WN. Then, a grid is divided for WN to obtain a second grid. For each sub-grid in each second grid, the normal vector with the largest angle with the vertical direction of the human back can be solved, and this angle is used as the second eigenvalue of each sub-grid. If the number of valid points in a sub-grid is less than 90%, the second eigenvalue of this sub-grid is 0.

[0091] For the left half of the second grid, subtract the second eigenvalue of the right grid from the second eigenvalue of the left grid. For the right half of the second grid, subtract the second eigenvalue of the left grid from the second eigenvalue of the right grid. If the subtraction value between any two grids is less than a certain threshold, it is considered problematic, and the normal vector is considered to pass the second detection.

[0092] In addition, the present application also selects human body data in the back area of the existing dataset, determines the average normal vector of each target key point according to the human body data, and determines the average normal vector of each target key point as the preset normal vector corresponding to this key point.

[0093] Determine the normal vector corresponding to the target key point among the normal vectors of multiple points, and calculate the angle between the normal vector of each key point and the preset normal vector of this key point. Different key points may have different thresholds. If it is greater than a certain threshold, an error will be reported indicating that there is a problem with the normal vector, and the normal vector is considered to pass the second detection.

[0094] Using the back posture inspection method provided by the present application, it can determine the normal vector of each point in the back area of the human body to be detected according to the depth image of the human body to be detected, and according to the depth information of each point in the depth image and the normal vector information of each point in the back area, perform the first detection and the second detection on the back area respectively according to the depth information and normal vector information of each point, so as to determine the posture detection result of the back area based on the first detection result and the second detection result. And the depth data in the present application is obtained by dividing the boundary of the human body to be detected according to the color image of the human body to be detected, then dividing the back area of the human body to be detected according to the back thickness, and performing regional ratio fusion to output the repaired human body depth, thereby further improving the accuracy of the data in the present application and the accuracy of the posture detection result of the back area. Using the method provided by the present application, it can solve the problems of depth drop caused by the steep slope on the side of the human body, inaccurate depth caused by a small amount of sunlight affecting the depth sensor, and inaccurate depth caused by oil coating and reflection on the back. And it can ensure the smoothness of the back while not losing the inclination angle of the normal vector in the edge area, retaining the authenticity of the data, so as to obtain the back depth information and normal vector information of the human body to be detected close to the real scene, so as to guide the subsequent robot to perform massage according to the above data.

[0095] The following explains the back posture inspection device provided by the present application in conjunction with the accompanying drawings. This back posture inspection device can execute the above-mentioned Figure 1 - Figure 2 any back posture inspection method. For its specific implementation and beneficial effects, please refer to the above, and will not be elaborated below.

[0096] Figure 3 It is a schematic structural diagram of the back posture inspection device provided by an embodiment of the present application. As Figure 3 shown, the device includes: a determination module 201 and a detection module 202, where:

[0097] The determination module 201 is used to determine the normal vectors of each point in the back area according to the depth image of the human body to be detected;

[0098] The detection module 202 is used to perform a first detection on the back area according to the depth information of each point in the depth image to obtain a first detection result; and perform a second detection on the back area according to the normal vectors of each point to obtain a second detection result;

[0099] The determination module 201 is specifically used to determine the posture detection result of the back area according to the first detection result and the second detection result.

[0100] Optionally, on the basis of the above embodiment, the embodiment of the present application can also provide a back posture inspection device. The following explains the implementation process of the above Figure 3 given device by way of example in conjunction with the accompanying drawings. Figure 4 It is a schematic structural diagram of the back posture inspection device provided by another embodiment of the present application. As Figure 4 shown, the device further includes: a generation module 203, which is used to generate a perspective transformation matrix for the back area of the human body to be detected according to the color image of the human body to be detected;

[0101] The detection module 202 is specifically used to perform a first detection on the back area according to the perspective transformation matrix and the depth information of each point to obtain a first detection result; and perform a second detection on the back area according to the perspective transformation matrix and the normal vectors of each point to obtain a second detection result.

[0102] Optionally, the determination module 201 is specifically used to determine the quadrilateral corresponding to the back area according to the color image of the human body to be detected and multiple target key points;

[0103] The generation module 203 is specifically used to generate a perspective transformation matrix for the back area of the human body to be detected according to the quadrilateral corresponding to the back area and the matrix image of a preset size.

[0104] Optionally, the detection module 202 is specifically configured to perform conversion processing on the depth information of each point according to the perspective transformation matrix to obtain a first grid; wherein, each point in the first grid has corresponding depth information; determine the first eigenvalue of each sub-grid in the first grid according to the depth information of each point in the first grid; perform a first detection on the first grid according to the first eigenvalue of each sub-grid in the first grid to obtain a first detection result.

[0105] Optionally, the determination module 201 is specifically configured to obtain the body thickness information of the human body to be detected according to the depth image to be detected; filter the depth image according to the body thickness information to obtain a filtered depth image;

[0106] The detection module 202 is specifically configured to perform a first detection on the back region according to the depth information of each point in the depth image, the depth information of each point in the filtered depth image, and obtain a first detection result.

[0107] Optionally, the determination module 201 is specifically configured to determine the mask of the back region and the plane mask where the back region is located according to the depth image to be detected; determine the body thickness information according to the mask of the back region, the plane mask, and the depth image to be detected.

[0108] Optionally, the determination module 201 is specifically configured to extract the first depth information of each point from the depth image to be detected according to the mask of the back region; calculate the first average depth information according to the first depth information of each point; extract the second depth information of each point from the depth image to be detected according to the plane mask; calculate the second average depth information according to the second depth information of each point; determine the body thickness information according to the first average depth information and the second average depth information.

[0109] Optionally, the determination module 201 is specifically configured to determine the range of each sub-region in the back region according to the body thickness information; filter the depth image to be detected according to the depth information of each sub-region in the depth image to be detected to obtain a filtered depth image.

[0110] Optionally, the determination module 201 is specifically configured to determine the first thickness mean value of the back region in the depth image according to the depth information of each point in the depth image; determine the second thickness mean value of the back region in the filtered depth image according to the depth information of each point in the filtered depth image; perform a first detection on the back region according to the difference between the first thickness mean value and the second thickness mean value to obtain a first detection result.

[0111] Optionally, the determination module 201 is specifically configured to perform a transformation process on the normal vectors of each point according to the perspective transformation matrix to obtain a second grid; wherein, each point in the second grid has a corresponding normal vector; determine the second eigenvalue of each sub-grid in the second grid according to the normal vector with the largest included angle with the preset reference direction.

[0112] The detection module 202 is specifically configured to perform a second detection on the second grid according to the second eigenvalue of each sub-grid in the second grid to obtain a second detection result.

[0113] Optionally, the determination module 201 is specifically configured to determine the normal vector corresponding to the target key point among the normal vectors of each point; wherein, each target key point has a preset normal vector.

[0114] The detection module 202 is specifically configured to perform a second detection on the back region according to the included angle between the normal vector of each target key point and the preset normal vector to obtain a second detection result.

[0115] By using the back posture inspection device provided in this application, after obtaining the depth image of the human body to be detected, the normal vectors of each point in the back region are determined according to the depth image, and the depth information of each point in the depth image is respectively subjected to a first detection, and the normal vector of each point is subjected to a second detection, so as to determine the final back region posture detection result according to the first detection result and the second detection result. When the back region posture detection result indicates that the detection is passed, the massage robot will be controlled to perform subsequent actions. Otherwise, it is determined that there is a problem with the depth image of the human body to be detected obtained currently and normal massage cannot be performed. Such a setting method does not require collecting a data set for training, and can directly perform the detection of the back posture based on the obtained depth image, can adapt to people of different body types, and enables multiple-dimensional detection of the human body image to be detected before massage, and checks the rationality of the data in the depth image from the depth angle and the normal vector angle to prepare for the subsequent massage.

[0116] The above modules may be one or more integrated circuits configured to implement the above methods. For example: one or more Application Specific Integrated Circuits (ASICs), or one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs), etc. For another example, when a certain above module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processors that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0117] Figure 5 FIG. 4 is a schematic structural diagram of a back posture inspection device provided by an embodiment of the present application. The back posture inspection device may be integrated into a terminal device or a chip of the terminal device.

[0118] As Figure 5 shown, the back posture inspection device includes: a processor 501, a bus 502, and a storage medium 503.

[0119] The processor 501 is used to store a program. The processor 501 calls the program stored in the storage medium 503 to execute the above-mentioned Figure 1 - Figure 2 corresponding method embodiment.

[0120] Optionally, the present application further provides a program product, such as a storage medium. The storage medium stores a computer program, including a program that executes the corresponding embodiment of the above method when run by a processor.

[0121] In several embodiments provided by the present application, it should be understood that the disclosed device and method may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces. The indirect coupling or communication connection of the device or unit may be in an electrical, mechanical or other form.

[0122] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0123] In addition, each functional unit in various embodiments of the present application may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a hardware plus software functional unit.

[0124] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit stored in a storage medium includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to execute some steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk, or an optical disc that can store program codes.

Claims

1. A method for checking the back posture, characterized in that, The method includes: Determining the normal vectors of each point in the back region according to the depth image of the human body to be detected; Performing a first detection on the back region according to the depth information of each point in the depth image to obtain a first detection result; Performing a second detection on the back region according to the normal vectors of each point to obtain a second detection result; Determining the pose detection result of the back region according to the first detection result and the second detection result.

2. The method according to claim 1, wherein The method further includes: Generating a perspective transformation matrix for the back region of the human body to be detected according to the color image of the human body to be detected; The performing a first detection on the back region according to the depth information of each point in the depth image to obtain a first detection result includes: Performing a first detection on the back region according to the perspective transformation matrix and the depth information of each point to obtain the first detection result; The performing a second detection on the back region according to the normal vectors of each point to obtain a second detection result includes: Performing a second detection on the back region according to the perspective transformation matrix and the normal vectors of each point to obtain the second detection result.

3. The method according to claim 2, wherein The generating a perspective transformation matrix for the back region of the human body to be detected according to the color image of the human body to be detected includes: Determining the quadrilateral corresponding to the back region according to the color image of the human body to be detected and multiple target key points; Generating the perspective transformation matrix for the back region of the human body to be detected according to the quadrilateral corresponding to the back region and the matrix image of a preset size.

4. The method according to claim 2, characterized in that The performing a first detection on the back region according to the perspective transformation matrix and the depth information of each point to obtain the first detection result includes: Performing conversion processing on the depth information of each point according to the perspective transformation matrix to obtain a first grid; wherein, each point in the first grid has corresponding depth information; Determining the first eigenvalue of each sub-grid in the first grid according to the depth information of each point in the first grid; Performing a first detection on the first grid according to the first eigenvalue of each sub-grid in the first grid to obtain the first detection result.

5. The method according to claim 1, wherein Before the performing a first detection on the back region according to the depth information of each point in the depth image to obtain a first detection result, the method further includes: Obtaining the body thickness information of the human body to be detected according to the depth image to be detected; Filtering the depth image according to the body thickness information to obtain a filtered depth image; The performing a first detection on the back region according to the depth information of each point in the depth image to obtain a first detection result includes: Performing a first detection on the back region according to the depth information of each point in the depth image, the depth information of each point in the filtered depth image, and obtaining the first detection result.

6. The method according to claim 5, wherein The obtaining the body thickness information of the human body to be detected according to the depth image to be detected includes: Determine a mask of the back region and a plane mask where the back region is located according to the depth image to be detected; Determine the body thickness information according to the mask of the back region, the plane mask, and the depth image to be detected.

7. The method according to claim 6, wherein The determining the body thickness information according to the mask of the back region, the plane mask, and the depth image to be detected includes: Extract first depth information of each point from the depth image to be detected according to the mask of the back region; Calculate first average depth information according to the first depth information of each point; Extract second depth information of each point from the depth image to be detected according to the plane mask; Calculate second average depth information according to the second depth information of each point; Determine the body thickness information according to the first average depth information and the second average depth information.

8. The method according to claim 6, characterized in that, The filtering the depth image according to the body thickness information to obtain a filtered depth image includes: Determine the range of each sub-region in the back region according to the body thickness information; Filter the depth image to be detected according to the depth information of each sub-region in the depth image to be detected to obtain the filtered depth image.

9. The method according to claim 5, characterized in that The performing a first detection on the back region according to the depth information of each point in the depth image, the depth information of each point in the filtered depth image, and obtaining the first detection result includes: Determine a first thickness mean value of the back region in the depth image according to the depth information of each point in the depth image; Determine a second thickness mean value of the back region in the filtered depth image according to the depth information of each point in the filtered depth image; Perform a first detection on the back region according to the difference between the first thickness mean value and the second thickness mean value to obtain the first detection result.

10. The method according to claim 2, wherein The performing a second detection on the back region according to the perspective transformation matrix and the normal vectors of each point and obtaining the second detection result includes: Perform a conversion process on the normal vectors of each point according to the perspective transformation matrix to obtain a second grid; wherein, each point in the second grid has a corresponding normal vector; Determine a second eigenvalue of each sub-grid in the second grid according to the normal vector with the largest included angle with the preset reference direction; Perform a second detection on the second grid according to the second eigenvalue of each sub-grid in the second grid to obtain the second detection result.

11. The method according to claim 1, wherein, The performing a second detection on the back region according to the normal vectors of each point and obtaining a second detection result further includes: Determine the normal vectors corresponding to the target key points among the normal vectors of each point; wherein, each of the target key points has a preset normal vector; Perform a second detection on the back region according to the included angle between the normal vectors of each target key point and the preset normal vector to obtain the second detection result.

12. An inspection device for the back posture, characterized in that, The device includes: a determination module and a detection module, wherein: The determining module is configured to determine the normal vectors of each point in the back region according to the depth image of the human body to be detected; The detecting module is configured to perform a first detection on the back region according to the depth information of each point in the depth image to obtain a first detection result; and perform a second detection on the back region according to the normal vectors of each point to obtain a second detection result; The determining module is specifically configured to determine the posture detection result of the back region according to the first detection result and the second detection result.

13. An inspection device for the back posture, characterized in that, The device includes: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the back posture inspection device runs, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the method according to any one of claims 1-11 above.

14. A storage medium, characterized in that, A computer program is stored on the storage medium, and when the computer program is run by the processor, it performs the method according to any one of claims 1-11 above.

Images