Back acupoint recognition method and device, electronic equipment and readable storage medium

Abstract

The application provides a back acupoint recognition method and device, electronic equipment and readable storage medium, comprising: acquiring a back color image corresponding to a target object collected by a first visual sensor, and determining an initial back acupoint coordinate based on the back color image; acquiring a back depth image corresponding to the target object collected by a second visual sensor, and determining a back mask based on the back depth image; acquiring contact data corresponding to a process in which an end of the mechanical arm contacts the back of the target object, and determining a target skeleton position based on the contact data; and determining a target back acupoint coordinate corresponding to the target object according to the initial back acupoint coordinate, the back mask and the target skeleton position. The application can effectively improve the accuracy and reliability of recognizing the back acupoint coordinate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of acupoint recognition, and in particular to a back acupoint recognition method and device, electronic equipment and a readable storage medium. BACKGROUND

[0002] In massage robot or intelligent massage product technology, accurately identifying the acupoint of the human body is the premise of effective massage. At present, the related technology proposes to directly predict the pixel coordinates of the acupoint through a color image, but different clothing in the color image has a great influence on the acupoint recognition accuracy, such as it will be difficult to judge the contour information of the human body in the case of wearing loose clothes, so the existing technology has the problems of low accuracy and low reliability in the process of acupoint prediction. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a back acupoint recognition method and device, electronic equipment and a readable storage medium, which can effectively improve the accuracy and reliability of recognizing back acupoint coordinates.

[0004] In a first aspect, the present application provides a back acupoint recognition method, comprising: acquiring a back color image of a target object corresponding to a first visual sensor, and determining an initial back acupoint coordinate based on the back color image; acquiring a back depth image of the target object corresponding to a second visual sensor, and determining a back mask based on the back depth image; acquiring contact data corresponding to the process of the end of the mechanical arm contacting the back of the target object, and determining a target bone position based on the contact data; and determining a target back acupoint coordinate corresponding to the target object according to the initial back acupoint coordinate, the back mask and the target bone position.

[0005] In a second aspect, the present application also provides a back acupoint recognition device, comprising: an initial acupoint recognition module for acquiring a back color image of a target object corresponding to a first visual sensor, and determining an initial back acupoint coordinate based on the back color image; a back mask determination module for acquiring a back depth image of the target object corresponding to a second visual sensor, and determining a back mask based on the back depth image; a bone position determination module for acquiring contact data corresponding to the process of the end of the mechanical arm contacting the back of the target object, and determining a target bone position based on the contact data; and a target acupoint recognition module for determining a target back acupoint coordinate corresponding to the target object according to the initial back acupoint coordinate, the back mask and the target bone position.

[0006] In a third aspect, an electronic device is provided, including a processor and a memory, the memory storing computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of the first aspect.

[0007] In a fourth aspect, a computer readable storage medium is provided, the computer readable storage medium storing computer executable instructions, and the computer executable instructions, when invoked and executed by a processor, cause the processor to implement the method according to any one of the first aspect.

[0008] The method, device, electronic device and readable storage medium provided by the embodiments of the present application first acquire a back color image of a target object collected by a first visual sensor, and determine initial back acupoint coordinates based on the back color image; acquire a back depth image of the target object collected by a second visual sensor, and determine a back mask based on the back depth image; acquire contact data corresponding to the process in which an end of a mechanical arm contacts the back of the target object, and determine a target skeleton position based on the contact data; and finally determine target back acupoint coordinates of the target object based on the initial back acupoint coordinates, the back mask and the target skeleton position. The above method simulates visual and tactile senses in the process of artificial massage through the method of multi-sensor fusion of the first visual sensor, the second visual sensor and the end of the mechanical arm, and through the fusion of the initial back acupoint coordinates, the back mask and the target skeleton position, target back acupoint coordinates with high accuracy can be obtained. Compared with the prior art of predicting acupoint coordinates based on color images only, the embodiments of the present application can effectively improve the accuracy and reliability of identifying back acupoint coordinates.

[0009] Other features and advantages of the present application will be set forth in the descriptions below, and in part will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the present application. The objects and other advantages of the present application can be realized and attained by the structure particularly pointed out in the description and the drawings.

[0010] In order to make the above objectives, features and advantages of the present application more apparent, the following will describe preferred embodiments in detail, and the accompanying drawings will be described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings described below are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0012] Figure 1 A flowchart of a back acupoint recognition method provided by an embodiment of the present application is shown in FIG. 1.

[0013] Figure 2 A structural diagram of a back acupoint detection sub-model provided by an embodiment of the present application is shown in FIG. 2.

[0014] Figure 3 A flowchart of another back acupoint recognition method provided by an embodiment of the present application is shown in FIG. 3.

[0015] Figure 4 A structural diagram of a back acupoint recognition device provided by an embodiment of the present application is shown in FIG. 4.

[0016] Figure 5 A structural diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0017] To make the objectives, technical solutions and advantages of embodiments of the present application clearer, the technical solutions of the present application will be described below in connection with embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of the present application.

[0018] In traditional Chinese medicine, the positioning of acupoints on the back of the human body needs to rely on bony landmarks. For example, the positioning of the bladder meridian needs to rely on the bony joints of the spine and the scapula. However, in a robot system, it is difficult to accurately determine the bony landmarks and the acupoint positioning by using a single sensor. For example, it is difficult to determine the position of the scapula by using only a color image captured by a camera, and different clothes have a strong deceptive effect on the visual algorithm. For another example, it is difficult for a force sensor and an end control of a mechanical arm to simulate the tactile sensation of a human hand, and it is very inefficient to determine the bony joints on the spine by using a force sensor. Not only is the danger high, but it is also difficult to determine the bony joints on the spine.

[0019] For example, the related art discloses a method for recognizing acupoints on the back of a human body, a recognition device and a computer storage medium, which uses a deep learning technology. Specifically, an OpenPose model is used to recognize acupoints on a single RGB (Red Green Blue) image. The related art also discloses a method for recognizing acupoints on the back of a human body, a system, a device and a storage medium, which also uses a deep learning technology. A top-down key point detection model is used to segment a human body by using a semantic segmentation model, and then the key points of the human body are detected by using a key point detection model, and finally the acupoint is predicted by using the key points.

[0020] However, in the back acupoint positioning theory of traditional Chinese medicine, the bone markers of the human body are judged by touch to accurately position the acupoints, and it is difficult to judge the contour of the scapula and the thoracic vertebra or lumbar vertebra of the spine by relying on a color image alone, that is, the related art is difficult to accurately position the acupoints.

[0021] Based on this, the present application provides a back acupoint recognition method, device, electronic equipment and readable storage medium, which can effectively improve the accuracy and reliability of recognizing back acupoint coordinates.

[0022] In order to facilitate the understanding of the present embodiment, first of all, a back acupoint recognition method disclosed by the present embodiment is introduced in detail, referring to the flowchart of a back acupoint recognition method shown in Figure 1 The method mainly includes the following steps S102 to S108:

[0023] Step S102, acquiring the back color image corresponding to the target object collected by the first visual sensor, and determining the initial back acupoint coordinates based on the back color image. Wherein, the first visual sensor can be a device with image collection function, such as camera, camera, smart phone, etc., the target object is the human body to be identified acupoint coordinates, the back color image is the RGB image of the human back, and optionally, the back color image can be the RGB image collected by the first visual sensor at the pitch angle. In an embodiment, the back color image of the target object can be collected by controlling the first visual sensor, and the back color image uploaded by the first visual sensor is acquired, and the acupoint coordinates and the position of the spine vertebra of the target object are preliminarily predicted based on the back color image by using deep learning algorithm, so as to obtain the initial back acupoint coordinates and the initial position of the spine vertebra, which can be used to correct the back acupoint coordinates.

[0024] Step S104, acquiring the back depth image corresponding to the target object collected by the second visual sensor, and determining the back mask based on the back depth image. Wherein, the second visual sensor can be a depth camera or a laser radar device, and the back depth image collected by the second visual sensor is also the depth image of the human back, which can reflect the depth information of each pixel point in the image, and the back mask can approximately reflect the back region of the human body without wearing clothes. In an embodiment, the back depth image of the target object can be collected by controlling the second visual sensor, and the back depth image uploaded by the second visual sensor is acquired, the depth difference between the two initial back acupoint coordinates is judged, the region near the acupoint with similar depth is retained, and the region with large depth difference is removed, so as to achieve the purpose of removing the influence of loose clothes, and obtain the back mask corresponding to the target object.

[0025] In step S106, contact data corresponding to the process of the end of the mechanical arm contacting the back of the target object is acquired, and the target bone position is determined based on the contact data. The contact data can include contact coordinates and contact force, and the target bone position can include the scapula lateral position and the hip bone position. In an embodiment, the end of the mechanical arm can be controlled to contact the back of the human body along a preset contact path, such as by acting on the area near the scapula by pulling, by acting on the area near the waist by gently stroking the muscle near the bladder meridian, and by analyzing the softness of each position of the back (referred to as back softness) and the back curve of the human body through the contact coordinates and the contact force, so as to distinguish the bones and muscles of the target object based on the softness and the back curve of the human body, and accurately identify the scapula lateral position and the hip bone position.

[0026] In step S108, the target back acupoint coordinates corresponding to the target object are determined according to the initial back acupoint coordinates, the back mask, and the target bone position. In an embodiment, the initial back acupoint coordinates and the initial spine position can be adjusted by using the back mask, and the adjusted initial back acupoint coordinates and the adjusted initial spine position can be used to assist the end of the mechanical arm to determine the action area (including the area near the scapula and the area near the waist). The target back acupoint coordinates can be obtained by adjusting the adjusted initial back acupoint coordinates and the adjusted initial spine position again using the target bone position.

[0027] The back acupoint recognition method provided by the embodiment of the present application can simulate the visual and tactile senses in the artificial massage process through the multi-sensor fusion method of the first visual sensor, the second visual sensor, and the end of the mechanical arm. By fusing the initial back acupoint coordinates, the back mask, and the target bone position, the target back acupoint coordinates with high accuracy can be obtained. Compared with the prior art of predicting acupoint coordinates based only on color images, the embodiment of the present application can effectively improve the accuracy and reliability of recognizing back acupoint coordinates.

[0028] In order to facilitate the understanding of the foregoing step S102, an embodiment of determining initial back acupoint coordinates based on a back color image is provided. The initial spine position and the initial back acupoint coordinates corresponding to the target object can be predicted based on the back color image by using a pre-trained acupoint detection model. The acupoint detection model includes a human body detection sub-model and a back acupoint detection sub-model. The human body detection sub-model is used to detect the human body region in the back color image to obtain a human body detection frame, and the input of the human body detection sub-model is the back color image, and the output of the human body detection sub-model is the human body detection frame. The back acupoint detection sub-model is used to further detect the initial back acupoint coordinates in the human body region, and the input of the back acupoint detection sub-model is the human body detection frame, and the output of the back acupoint detection sub-model is the initial back acupoint coordinates. In the specific implementation, steps a to c can be referred to.

[0029] Step a, human body detection is performed on the back color image to obtain a human body detection frame by a human body detection sub-model. In an embodiment, the human body detection sub-model can use a YOLOX single-stage detection network to predict the position of the detection frame (i.e., the human body detection frame) where the human body is located in the back color image by using a top-down algorithm. Since the "human" target is included in almost all publicly available data sets related to detection, the embodiment of the present application fuses multiple public data sets such as MS COCO, Pascal VOC, CityScapes, and data in business scenarios as training samples, only retains the human class, and re-trains the YOLOX single-stage detection network to increase the diversity of the scene and improve the generalization of the model.

[0030] When training the YOLOX single-stage detection network, the training parameters and the loss function can be set based on actual needs. Optionally, in order to enhance the generalization of the YOLOX single-stage detection network, the training samples can be subjected to conventional data augmentation such as rotation, random scaling and cropping, and the training samples can also be subjected to mosaic data augmentation and Mixup data augmentation. Optionally, the embodiment of the present application can also use SimOTA for automatic positive and negative sample allocation. Optionally, the embodiment of the present application uses an IoU (Intersection over Union) loss as the detection frame regression loss, and uses a binary cross-entropy loss as the confidence loss and the class prediction loss. The input of the subsequent back acupoint detection sub-model is obtained by processing the human body detection frame.

[0031] Step b, adjusting the human body detection frame by using an unbiased data estimation algorithm. In an embodiment, the embodiment of the present application can use an unbiased data processing (UDP, Unbiased Data Processing) algorithm to pre-process and post-process the human body detection frame to obtain the input of the back acupoint detection sub-model and the coordinates of the corresponding back color image according to the initial back acupoint coordinates output by the back acupoint detection sub-model.

[0032] Step c, predicting the initial spine segment position and the initial back acupoint coordinates corresponding to the target object based on the adjusted human body detection frame by using the back acupoint detection sub-model. In an embodiment, the back acupoint detection sub-model can use a ViTPose model, and the training method of the ViTPose model can be selected based on actual needs. For example, see Figure 2A structure diagram of a back acupoint detection sub-model shown, a backbone network and a decoding head (i.e., a decoder), the backbone network includes a plurality of Transformer modules, each Transformer module includes a first layer normalization, a multi-head self-attention layer, a second layer normalization and a feedforward layer, the decoding head includes a first deconvolution layer, a first batch normalization, a first Relu, a second deconvolution layer, a second batch normalization, a second Relu and a convolution layer.

[0033] On the basis of the above structure, the embodiment of the application provides a training method of a back acupoint detection sub-model, specifically: first, use images in human body posture estimation public data sets such as MS COCO, AI Challenger, MPII and CrowdPose as training samples, train the Vision Transformer backbone network in the ViTPose model in a self-supervised training manner of Masked Autoencoders (MAE), through 75% of the images are occluded, through the encoder encoding and then through the decoder decoding, the prediction result of the decoder and the occluded part are used to calculate the mean square error loss to perform self-supervised training. Only random cropping is used as a data enhancement strategy during the training process, and only the encoder part, i.e., the backbone network part, is retained after the self-supervised training is completed. The backbone network part is connected to the head structure of human key point recognition, and the entire network is fine-tuned through the MS COCO human key point data set. All parameters of the entire network are updated during fine-tuning, and finally a small amount of acupoint data collected in a business scenario is fine-tuned again to prevent overfitting of the model.

[0034] In actual application, since there is no constraint between key points in the original ViTPose model, this will occasionally cause the left and right to be unable to be determined when only part of the human body is seen, for example, the left shoulder is predicted to be on the left side of the right shoulder (facing away from the camera) and the left waist is on the right side of the right waist (facing the camera) in the image. This situation is adjusted by first performing network prediction and then performing some prior logic to exchange left and right, for example, the business scenario is definitely facing away from the camera, thereby ensuring the reliability of the algorithm prediction.

[0035] In actual application, due to the lack of constraints between acupoints, a small number of cases will cause the prediction result to not conform to the human physiological model, for example, the lateral bladder meridian is basically distributed on the same straight line, so the initial back acupoint coordinates output by the ViTPose model are combined with the human physiological model to perform post-processing optimization.

[0036] To facilitate the understanding of the foregoing step S104, the embodiment of the present application provides an implementation of determining the back mask based on the back depth image, which can remove the area with too large depth difference to retain the back area, thereby achieving the purpose of removing the influence of loose clothes. For details, please refer to the following steps 1 to 5:

[0037] Step 1, determining the depth information corresponding to the initial back acupoint coordinate based on the back depth image. In an implementation, since the back depth image collected by the depth camera has depth information, the depth information corresponding to each acupoint can be directly read from the corresponding coordinate in the back depth image according to the foregoing initial back acupoint coordinate.

[0038] Step 2, estimating the target back thickness of the target object according to the basic physical data corresponding to the target object. The basic physical data can include height data and weight data. In an implementation, the approximate back thickness of the target object can be estimated according to the height data and weight data by using an existing algorithm.

[0039] Step 3, determining the target depth range corresponding to the target back thickness according to the mapping relationship between the back thickness and the depth range. In an implementation, the mapping relationship between the back thickness and the depth range can be preconfigured, so as to retrieve the corresponding target depth range with the target back thickness as the retrieval condition. In another optional implementation, the target depth range can be pre-determined according to prior experience, and the target depth range determined by this method is a fixed value.

[0040] Step 4, determining the adjacent back acupoint coordinate corresponding to the initial back acupoint coordinate, and determining the initial back acupoint coordinate as an abnormal back acupoint coordinate if the difference between the depth information corresponding to the initial back acupoint coordinate and the depth information corresponding to the adjacent back acupoint coordinate exceeds the target depth range. In actual application, the number of initial back acupoint coordinates is multiple. For each back acupoint coordinate, it is necessary to determine whether the difference between the depth information corresponding to the initial back acupoint coordinate and the depth information corresponding to the adjacent back acupoint coordinate exceeds the target depth range, and determine the initial back acupoint coordinate as an abnormal back acupoint coordinate if the difference exceeds the target depth range.

[0041] In an implementation, the adjacent back acupoint coordinate can be directly obtained through the definition of acupoints in traditional Chinese medicine. Further, the difference between the depth information of two back acupoint coordinates is determined. If the difference exceeds the target depth range, the initial back acupoint coordinate is determined as an abnormal back acupoint coordinate affected by the clothes shielding. If the difference does not exceed the target depth range, the initial back acupoint coordinate is determined as a normal back acupoint coordinate not affected by the clothes shielding.

[0042] Step 5, determining the back mask based on the initial back acupoint coordinates after removing the abnormal back acupoint coordinates. In an embodiment, by removing the abnormal back acupoint coordinates and retaining the normal back acupoint coordinates, it is equivalent to retaining the regions with similar depth near the acupoint points and removing the regions with large depth difference, so as to obtain the back mask for representing the back region of the human body which is approximately not wearing clothes.

[0043] Optionally, the initial back acupoint coordinates can be adjusted according to the back mask, and the regions for emphasis adjustment are the lateral contours of the shoulders, and the distance between the waist and the vertical direction of the spine is adjusted according to the left-right symmetry restriction.

[0044] To facilitate the understanding of the foregoing step S106, the embodiment of the present application provides an implementation manner of acquiring contact data corresponding to the process that the end of the mechanical arm contacts the back of the target object, and determining the target bone position based on the contact data, wherein the contact data includes first contact coordinates, second contact coordinates and contact force, and details can be referred to (1) to (2) as follows:

[0045] (1) controlling the end of the mechanical arm to contact the back of the target object according to a preset first contact path, acquiring the first contact coordinates corresponding to the process that the end of the mechanical arm contacts the back of the target object, and determining the back height curve corresponding to the target object according to the first contact coordinates. In an embodiment, the human back height curve is first acquired according to the contact position of the end of the mechanical arm. Specifically, the first contact path can be from the shoulder to the pelvic bone, that is, the mechanical arm is controlled to pass the bladder meridian from the shoulder to the pelvic bone position, and the human back height curve is obtained through the values of the height direction of the end of the mechanical arm and the head-to-foot direction of the human body.

[0046] Optionally, the longitudinal position of the adjusted initial back acupoint coordinates can be optimized and adjusted according to the human spine model of traditional Chinese medicine through a gradient descent algorithm.

[0047] (2) determining a second contact path based on the initial back acupoint coordinates, and controlling the end of the mechanical arm to contact the back of the target object according to the second contact path, acquiring the second contact coordinates and the contact force corresponding to the process that the end of the mechanical arm contacts the back of the target object; determining the back softness corresponding to the target object according to the second contact coordinates and the contact force; and determining the target bone position according to the back height curve and the back softness. In an embodiment, the position of the lateral scapula can be determined by controlling the mechanical arm to pluck the lateral bladder meridian. Specifically, the positions of the first thoracic vertebra to the seventh thoracic vertebra are determined according to the foregoing adjusted initial back acupoint coordinates, and the path is determined as the second contact path, so as to control the end of the mechanical arm to pluck the first thoracic vertebra to the seventh thoracic vertebra, the end of the mechanical arm is provided with a force sensor, so as to determine the hardness of the contact surface of the end according to the contact force collected by the force sensor and the second contact position of the end of the mechanical arm, and thus the position of the lateral scapula is distinguished.

[0048] Optionally, the lateral bladder meridian transverse position can be adjusted according to the bone landmark positioning method of traditional Chinese medicine.

[0049] To facilitate the understanding of the foregoing step S108, the embodiment of the present application provides an implementation of determining the target back acupoint coordinates corresponding to the target object according to the initial back acupoint coordinates, the back mask and the target skeleton position. Specifically, first, the initial back acupoint coordinates are adjusted according to the back mask to obtain intermediate back acupoint coordinates, and then the intermediate back acupoint coordinates are adjusted according to the target skeleton position to obtain the target back acupoint coordinates. The intermediate back acupoint coordinates are the initial back acupoint coordinates after adjustment. Optionally, when the initial back acupoint coordinates are adjusted according to the back mask, the initial back acupoint coordinates can be adjusted according to the back mask, and the areas to be adjusted include the lateral contour of the shoulder and the distance between the waist and the vertical direction of the spine according to the left-right symmetry restriction; when the intermediate back acupoint coordinates are adjusted according to the target skeleton position, the longitudinal coordinates of the intermediate back acupoint coordinates can be adjusted according to the back height curve corresponding to the target object, and the lateral coordinates of the intermediate back acupoint coordinates can be adjusted according to the target skeleton position corresponding to the target object to obtain the target back acupoint coordinates. In the specific implementation, the intermediate back acupoint coordinates can be optimized by the gradient descent algorithm according to the human spine model and the back height curve of traditional Chinese medicine, and the lateral bladder meridian transverse position can be adjusted according to the bone landmark positioning method of traditional Chinese medicine.

[0050] In an implementation, before the identification method of back acupoints provided by the embodiment of the present application is executed, the following preparation work needs to be performed: the person to be massaged / physiotherapied needs to be well laid on the bed, and the mechanical arm needs to be moved to a proper position away from the person to be massaged. On this basis, the embodiment of the present application further provides an application example of the identification method of back acupoints, which is shown in another flowchart of the identification method of back acupoints in FIG. 4. Figure 3 The method mainly includes the following steps S302 to S312.

[0051] The core idea and method steps of the present application are as follows:

[0052] Step S302, a back color image is acquired. In an implementation, the overhead view image of the human back can be acquired by a camera first.

[0053] Step S304, the back color image is input into the human detection sub-model to obtain a human detection frame.

[0054] Step S306, input the human body detection frame to the back acupoint detection sub-model through the UDP algorithm to obtain the preliminary back acupoint coordinates. In an embodiment, the initial spine segment position and the initial back acupoint coordinates can be preliminarily predicted through a deep learning algorithm (ViTPose model). The approximate initial back acupoint coordinates obtained through the image can improve the speed of acupoint recognition, and the approximate judgment of the back acupoint position of the human body can be made in a very short time. Compared with the pure tactile method, the efficiency can be greatly improved.

[0055] Step S308, obtain the back depth image. In an embodiment, the human body back overhead depth map can be obtained through a depth camera or a laser radar.

[0056] Step S310, obtain the back mask according to the back depth image, and update the initial back acupoint coordinates through the shoulder boundary and the waist contour in the back mask. In an embodiment, the depth value of the entire back is estimated through the depth value of the initial back acupoint coordinates identified in the previous step, so as to obtain the mask of the entire back region, and then the initial spine segment position and the initial back acupoint coordinates are updated according to the shoulder boundary and the waist contour in the back mask.

[0057] Since in the robot system, the pixel coordinates of the image cannot obtain the coordinates in the robot world coordinate system, the depth information is an essential part. In the second step, the existing depth information is used, and since the person being massaged is in a prone position, the clothes will collapse along the back contour. The clothes region with a large difference from the depth value of the human back can be filtered out to exclude the visual interference of soft and loose clothes.

[0058] Step S312, update the initial back acupoint coordinates again by rolling and plucking the bladder meridian through the mechanical arm to obtain the back height curve and the lateral contour of the scapula.

[0059] In an embodiment, the mechanical arm control can be used to act on the region near the scapula identified in the previous step through plucking and rolling, to act on the region near the waist identified in the previous step through the method of gently rolling the muscles near the bladder meridian, and to analyze the softness of each position and the curve of the human back through the position information of the mechanical arm and the force sensor at the end of the mechanical arm. The embodiment of the application identifies the scapula contour, the iliac bone position and the back curve of the human body through the close-to-massage method, reduces the long waiting time and the uncomfortable experience caused by acupoint positioning, and improves the experience of the person being massaged.

[0060] In one embodiment, the bone and muscle are distinguished according to the softness, so as to determine the accurate scapula lateral position and the iliac bone position. The lateral coordinate of the acupoint obtained in the third step is adjusted by the scapula lateral position, and the longitudinal coordinate of the acupoint obtained in the third step is adjusted by the iliac bone position and the back curve of the human body. The embodiments of the present application are based on the bone markers in the positioning of the back acupoints in traditional Chinese medicine, and the lateral coordinate of the bladder meridian is positioned according to the scapula lateral position, the longitudinal coordinate of the fourth lumbar vertebra is positioned according to the iliac bone position, and the accurate acupoint coordinates are positioned according to the bone markers.

[0061] In summary, considering that the person being massaged usually wears relatively loose clothes during the massage therapy to achieve a relatively relaxed state, and the loose clothes will cause visual errors and interfere with the visual algorithm, for example, the raised clothes will make it difficult to accurately determine the shoulder position and the body shape, and the pelvis will be covered to make it difficult to accurately distinguish the waist and the iliac bone position, so that the acupoint positioning result is not accurate enough. In addition, different postures of the human body will cause changes in the scapula and other bone markers, so that the position of the acupoint changes, and it is difficult to find the accurate scapula lateral position and the accurate acupoint position only by the visual algorithm. The embodiments of the present application simulate human vision, touch and other senses by a multi-sensor fusion method, exclude the interference of some loose clothes and low light environment, and thus achieve the purpose of accurate recognition of the back acupoints of the human body.

[0062] For the back acupoint recognition method provided by the foregoing embodiments, the embodiments of the present application provide a back acupoint recognition device, which refers to a structure schematic diagram of a back acupoint recognition device shown in Figure 4 The device mainly includes the following parts:

[0063] The initial acupoint recognition module 402 is configured to acquire the back color image of the target object collected by the first visual sensor, and determine the initial back acupoint coordinates based on the back color image;

[0064] The back mask determination module 404 is configured to acquire the back depth image of the target object collected by the second visual sensor, and determine the back mask based on the back depth image;

[0065] The bone position determination module 406 is configured to acquire the contact data corresponding to the process that the end of the mechanical arm contacts the back of the target object, and determine the target bone position based on the contact data;

[0066] The target acupoint recognition module 408 is configured to determine the target back acupoint coordinates of the target object according to the initial back acupoint coordinates, the back mask and the target bone position.

[0067] The back acupoint recognition device provided by the embodiment of the present application can simulate the vision, touch and other senses in the artificial massage process through the multi-sensor fusion method of the first vision sensor, the second vision sensor and the mechanical arm end, and can obtain the target back acupoint coordinates with high accuracy by fusing the initial back acupoint coordinates, the back mask and the target skeleton position. Compared with the prior art which only predicts the acupoint coordinates based on color images, the embodiment of the present application can effectively improve the accuracy and reliability of recognizing the back acupoint coordinates.

[0068] In an implementation manner, the initial acupoint recognition module 402 is further configured to: predict the initial spine vertebra position and the initial back acupoint coordinates corresponding to the target object based on the back color image through a pre-trained acupoint detection model; and the acupoint detection model comprises a human body detection sub-model and a back acupoint detection sub-model.

[0069] In an implementation manner, the initial acupoint recognition module 402 is further configured to: perform human body detection on the back color image to obtain a human body detection frame through the human body detection sub-model; adjust the human body detection frame by using an unbiased data estimation algorithm; and predict the initial spine vertebra position and the initial back acupoint coordinates corresponding to the target object based on the adjusted human body detection frame through the back acupoint detection sub-model.

[0070] In an implementation manner, the back mask determination module 404 is further configured to: determine the depth information corresponding to the initial back acupoint coordinates based on the back depth image; estimate the target back thickness of the target object according to the basic physical data of the target object; determine the target depth range corresponding to the target back thickness according to the mapping relationship between the back thickness and the depth range which is pre-configured; determine the adjacent back acupoint coordinates corresponding to the initial back acupoint coordinates, and if the difference between the depth information corresponding to the initial back acupoint coordinates and the depth information corresponding to the adjacent back acupoint coordinates exceeds the target depth range, determine that the initial back acupoint coordinates are abnormal back acupoint coordinates; and determine the back mask based on the initial back acupoint coordinates after the abnormal back acupoint coordinates are removed.

[0071] In an implementation, the skeleton position determination module 406 is further configured to: control the robot arm end to contact the back of the target object according to a preset first contact path, and obtain first contact coordinates corresponding to a process in which the robot arm end contacts the back of the target object; determine a back height curve corresponding to the target object according to the first contact coordinates; determine a second contact path based on the initial back acupoint coordinates, and control the robot arm end to contact the back of the target object according to the second contact path, and obtain second contact coordinates and contact force corresponding to a process in which the robot arm end contacts the back of the target object; determine back softness corresponding to the target object according to the second contact coordinates and the contact force; and determine a target skeleton position according to the back height curve and the back softness; wherein the contact data comprises the first contact coordinates, the second contact coordinates and the contact force.

[0072] In an implementation, the target acupoint identification module 408 is further configured to: adjust the initial back acupoint coordinates according to the back mask to obtain intermediate back acupoint coordinates; and adjust the intermediate back acupoint coordinates according to the target skeleton position to obtain target back acupoint coordinates.

[0073] In an implementation, the target acupoint identification module 408 is further configured to: adjust a longitudinal coordinate of the intermediate back acupoint coordinates according to the back height curve corresponding to the target object, and adjust a transverse coordinate of the intermediate back acupoint coordinates according to the target skeleton position corresponding to the target object to obtain the target back acupoint coordinates.

[0074] The device provided in the embodiments of the present application has the same implementation principle and technical effects as the foregoing method embodiments, and for brevity of description, the part not mentioned in the device embodiment part can be referred to the corresponding content in the foregoing method embodiments.

[0075] The embodiments of the present application provide an electronic device, specifically, the electronic device comprises a processor and a storage device; the storage device stores a computer program, and the computer program performs the following when being run by the processor:

[0076] A back acupoint identification method comprises: obtaining a back color image of a target object collected by a first visual sensor, and determining initial back acupoint coordinates based on the back color image; obtaining a back depth image of the target object collected by a second visual sensor, and determining a back mask based on the back depth image; obtaining contact data corresponding to a process in which a robot arm end contacts the back of the target object, and determining a target skeleton position based on the contact data; and determining target back acupoint coordinates corresponding to the target object according to the initial back acupoint coordinates, the back mask and the target skeleton position.

[0077] In an embodiment, determining the initial back acupoint coordinates based on the back color image comprises: predicting the initial spine segment position and the initial back acupoint coordinates corresponding to the target object based on the back color image by using a pre-trained acupoint detection model; wherein the acupoint detection model comprises a human body detection sub-model and a back acupoint detection sub-model.

[0078] In an embodiment, predicting the initial spine segment position and the initial back acupoint coordinates corresponding to the target object based on the back color image by using a pre-trained acupoint detection model comprises: performing human body detection on the back color image to obtain a human body detection frame by using a human body detection sub-model; adjusting the human body detection frame by using an unbiased data estimation algorithm; and predicting the initial spine segment position and the initial back acupoint coordinates corresponding to the target object based on the adjusted human body detection frame by using a back acupoint detection sub-model.

[0079] In an embodiment, determining the back mask based on the back depth image comprises: determining the depth information corresponding to the initial back acupoint coordinates based on the back depth image; estimating the target back thickness of the target object according to the basic physical data corresponding to the target object; determining the target depth range corresponding to the target back thickness according to a pre-configured mapping relationship between the back thickness and the depth range; determining the adjacent back acupoint coordinates of the initial back acupoint coordinates, and determining the initial back acupoint coordinates as abnormal back acupoint coordinates if the difference between the depth information corresponding to the initial back acupoint coordinates and the depth information corresponding to the adjacent back acupoint coordinates exceeds the target depth range; and determining the back mask based on the initial back acupoint coordinates after removing the abnormal back acupoint coordinates.

[0080] In an embodiment, the contact data corresponding to the process of the end of the mechanical arm contacting the back of the target object is obtained, and the target bone position is determined based on the contact data, comprising: controlling the end of the mechanical arm to contact the back of the target object according to a pre-set first contact path, and obtaining the first contact coordinates corresponding to the process of the end of the mechanical arm contacting the back of the target object; determining the back height curve corresponding to the target object according to the first contact coordinates; determining the second contact path based on the initial back acupoint coordinates, and controlling the end of the mechanical arm to contact the back of the target object according to the second contact path, and obtaining the second contact coordinates and the contact force corresponding to the process of the end of the mechanical arm contacting the back of the target object; determining the back softness corresponding to the target object according to the second contact coordinates and the contact force; and determining the target bone position according to the back height curve and the back softness; wherein the contact data comprises the first contact coordinates, the second contact coordinates and the contact force.

[0081] In an implementation, the target back acupoint coordinates corresponding to the target object are determined according to the initial back acupoint coordinates, the back mask and the target skeleton position, including: adjusting the initial back acupoint coordinates according to the back mask to obtain intermediate back acupoint coordinates; and adjusting the intermediate back acupoint coordinates according to the target skeleton position to obtain the target back acupoint coordinates.

[0082] In an implementation, the target back acupoint coordinates are determined according to the target skeleton position, including: adjusting the longitudinal coordinates of the intermediate back acupoint coordinates according to the back height curve corresponding to the target object, and adjusting the transverse coordinates of the intermediate back acupoint coordinates according to the target skeleton position corresponding to the target object to obtain the target back acupoint coordinates.

[0083] The electronic device provided by the embodiment of the present application can simulate the vision and touch in the artificial massage process through the multi-sensor fusion method of the first visual sensor, the second visual sensor and the mechanical arm end, and can obtain the target back acupoint coordinates with high accuracy by fusing the initial back acupoint coordinates, the back mask and the target skeleton position. Compared with the prior art of predicting the acupoint coordinates based on only color images, the embodiment of the present application can effectively improve the accuracy and reliability of identifying the back acupoint coordinates.

[0084] Figure 5 A structural schematic diagram of an electronic device provided by the embodiment of the present application is shown in the figure, and the electronic device 100 includes a processor 50, a memory 51, a bus 52 and a communication interface 53, wherein the processor 50, the communication interface 53 and the memory 51 are connected through the bus 52; the processor 50 is used to execute the executable modules stored in the memory 51, such as a computer program.

[0085] The memory 51 may contain a high-speed random access memory (RAM) and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 53 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.

[0086] The bus 52 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 5 only one bidirectional arrow is used, but it does not mean that there is only one bus or only one type of bus.

[0087] The memory 51 is configured to store a program, and the processor 50 executes the program after receiving an execution instruction. The method performed by the device for defining the flow process disclosed in any of the embodiments of the present application can be applied to the processor 50 or implemented by the processor 50.

[0088] The processor 50 can be an integrated circuit chip with a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 50 or the instruction in the form of software. The processor 50 described above can be a general processor, including a central processing unit (CPU) and a network processor (NP); can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory 51, and the processor 50 reads the information in the memory 51 and combines the hardware to complete the steps of the above method.

[0089] The computer program product of the readable storage medium provided by the embodiments of the present application includes a computer readable storage medium storing program codes, and the program codes include instructions for executing the following steps:

[0090] A back acupoint recognition method, comprising: acquiring a back color image corresponding to a target object collected by a first vision sensor, and determining an initial back acupoint coordinate based on the back color image; acquiring a back depth image corresponding to the target object collected by a second vision sensor, and determining a back mask based on the back depth image; acquiring contact data corresponding to a process in which an end of the mechanical arm contacts the back of the target object, and determining a target bone position based on the contact data; and determining a target back acupoint coordinate corresponding to the target object according to the initial back acupoint coordinate, the back mask and the target bone position.

[0091] In an implementation, determining the initial back acupoint coordinates based on the back color image includes: predicting the initial spine segment position and the initial back acupoint coordinates corresponding to the target object based on the back color image by a pre-trained acupoint detection model; wherein the acupoint detection model includes a human body detection sub-model and a back acupoint detection sub-model.

[0092] In an implementation, predicting the initial spine segment position and the initial back acupoint coordinates corresponding to the target object based on the back color image by a pre-trained acupoint detection model includes: performing human body detection on the back color image to obtain a human body detection frame by a human body detection sub-model; adjusting the human body detection frame by an unbiased data estimation algorithm; predicting the initial spine segment position and the initial back acupoint coordinates corresponding to the target object based on the adjusted human body detection frame by a back acupoint detection sub-model.

[0093] In an implementation, determining the back mask based on the back depth image includes: determining the depth information corresponding to the initial back acupoint coordinates based on the back depth image; estimating the target back thickness of the target object according to the basic physical data corresponding to the target object; determining the target depth range corresponding to the target back thickness according to the pre-configured mapping relationship between the back thickness and the depth range; determining the adjacent back acupoint coordinates corresponding to the initial back acupoint coordinates, and determining the initial back acupoint coordinates as abnormal back acupoint coordinates if the difference between the depth information corresponding to the initial back acupoint coordinates and the depth information corresponding to the adjacent back acupoint coordinates exceeds the target depth range; and determining the back mask based on the initial back acupoint coordinates after removing the abnormal back acupoint coordinates.

[0094] In an implementation, obtaining the contact data corresponding to the process that the end of the mechanical arm contacts the back of the target object, and determining the target bone position based on the contact data includes: controlling the end of the mechanical arm to contact the back of the target object according to a pre-set first contact path, and obtaining the first contact coordinates corresponding to the process that the end of the mechanical arm contacts the back of the target object; determining the back height curve corresponding to the target object according to the first contact coordinates; determining the second contact path based on the initial back acupoint coordinates, and controlling the end of the mechanical arm to contact the back of the target object according to the second contact path, and obtaining the second contact coordinates and the contact force corresponding to the process that the end of the mechanical arm contacts the back of the target object; determining the back softness corresponding to the target object according to the second contact coordinates and the contact force; and determining the target bone position according to the back height curve and the back softness; wherein the contact data includes the first contact coordinates, the second contact coordinates and the contact force.

[0095] In an implementation, the target back acupoint coordinates corresponding to the target object are determined according to the initial back acupoint coordinates, the back mask and the target skeleton position, including: adjusting the initial back acupoint coordinates according to the back mask to obtain intermediate back acupoint coordinates; and adjusting the intermediate back acupoint coordinates according to the target skeleton position to obtain the target back acupoint coordinates.

[0096] In an implementation, the target back acupoint coordinates are determined according to the target skeleton position, including: adjusting the longitudinal coordinates of the intermediate back acupoint coordinates according to the back height curve corresponding to the target object, and adjusting the transverse coordinates of the intermediate back acupoint coordinates according to the target skeleton position corresponding to the target object to obtain the target back acupoint coordinates.

[0097] The readable storage medium provided by the embodiment of the present application can simulate visual sense and tactile sense in the artificial massage process through the multi-sensor fusion method of the first visual sensor, the second visual sensor and the mechanical arm end, and can obtain target back acupoint coordinates with high accuracy by fusing the initial back acupoint coordinates, the back mask and the target skeleton position. Compared with the prior art of predicting acupoint coordinates based on a color image only, the embodiment of the present application can effectively improve the accuracy and reliability of identifying back acupoint coordinates.

[0098] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device) to execute all or part of the steps of the method described in the embodiments of the present application. The storage medium mentioned above includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk and various program code storage media.

[0099] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit the present application, and the protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features therein, within the technical scope disclosed by the present application. Such modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for identifying acupoints on the back, characterized in that, include: Acquire a color image of the back of the target object collected by the first visual sensor, and determine the initial coordinates of the acupoints on the back based on the color image of the back; Acquire the back depth image corresponding to the target object collected by the second vision sensor, and determine the back mask based on the back depth image; The method involves acquiring contact data corresponding to the process of a robotic arm's end-effector contacting the back of a target object, and determining the target bone position based on the contact data. This acquisition includes: controlling the robotic arm's end-effector to contact the back of the target object along a preset first contact path; acquiring first contact coordinates corresponding to the process of the robotic arm's end-effector contacting the back of the target object; determining the back height curve corresponding to the target object based on the first contact coordinates; determining a second contact path based on initial back acupoint coordinates; controlling the robotic arm's end-effector to contact the back of the target object along the second contact path; acquiring second contact coordinates and contact force corresponding to the process of the robotic arm's end-effector contacting the back of the target object; determining the back flexibility corresponding to the target object based on the second contact coordinates and the contact force; and determining the target bone position based on the back height curve and the back flexibility. The contact data includes the first contact coordinates, the second contact coordinates, and the contact force. The target back acupoint coordinates of the target object are determined based on the initial back acupoint coordinates, the back mask, and the target bone position.

2. The method for identifying acupoints on the back according to claim 1, characterized in that, Determining the initial back acupoint coordinates based on the back color image includes: Using a pre-trained acupoint detection model, the initial spinal segment positions and initial back acupoint coordinates of the target object are predicted based on the back color image; wherein, the acupoint detection model includes a human body detection sub-model and a back acupoint detection sub-model.

3. The method for identifying acupoints on the back according to claim 2, characterized in that, Using a pre-trained acupoint detection model, the initial spinal segment positions and initial back acupoint coordinates of the target object are predicted based on the back color image, including: The human detection sub-model is used to perform human detection from the back color image to obtain a human detection box. The human detection box is adjusted using an unbiased data estimation algorithm; Using the back acupoint detection sub-model, the initial spinal segment position and initial back acupoint coordinates of the target object are predicted based on the adjusted human body detection box.

4. The method for identifying acupoints on the back according to claim 1, characterized in that, Determining the back mask based on the back depth image includes: Determine the depth information corresponding to the initial back acupoint coordinates based on the back depth image; Based on the basic physical data of the target object, estimate the target back thickness of the target object; The target depth range corresponding to the target back thickness is determined based on the pre-configured mapping relationship between back thickness and depth range; Determine the coordinates of adjacent back acupoints corresponding to the initial back acupoint coordinates. If the difference between the depth information corresponding to the initial back acupoint coordinates and the depth information corresponding to the adjacent back acupoint coordinates exceeds the target depth range, then determine that the initial back acupoint coordinates are abnormal back acupoint coordinates. The back mask is determined based on the initial back acupoint coordinates after removing the abnormal back acupoint coordinates.

5. The method for identifying acupoints on the back according to claim 1, characterized in that, Determining the target back acupoint coordinates corresponding to the target object based on the initial back acupoint coordinates, the back mask, and the target bone position includes: The initial back acupoint coordinates are adjusted according to the back mask to obtain the intermediate back acupoint coordinates; The coordinates of the acupoints on the middle back are adjusted according to the location of the target bone to obtain the coordinates of the acupoints on the target back.

6. The method for identifying acupoints on the back according to claim 5, characterized in that, The coordinates of the acupoints on the middle back are adjusted according to the location of the target bone to obtain the coordinates of the acupoints on the target back, including: Based on the back height curve corresponding to the target object, the vertical coordinate of the middle back acupoint is adjusted, and based on the target bone position corresponding to the target object, the horizontal coordinate of the middle back acupoint is adjusted to obtain the target back acupoint coordinates.

7. A device for identifying acupoints on the back, characterized in that, include: The initial acupoint recognition module is used to acquire a back color image corresponding to the target object collected by the first visual sensor, and determine the initial back acupoint coordinates based on the back color image; The back mask determination module is used to acquire the back depth image corresponding to the target object collected by the second vision sensor, and determine the back mask based on the back depth image; A skeletal position determination module is used to acquire contact data corresponding to the process of a robotic arm end-effector contacting the back of a target object, and to determine the target skeletal position based on the contact data. Specifically, the skeletal position determination module is used to: control the robotic arm end-effector to contact the back of the target object along a preset first contact path, acquire first contact coordinates corresponding to the process of the robotic arm end-effector contacting the back of the target object; determine the back height curve corresponding to the target object based on the first contact coordinates; determine a second contact path based on the initial back acupoint coordinates, and control the robotic arm end-effector to contact the back of the target object along the second contact path, acquire second contact coordinates and contact force corresponding to the process of the robotic arm end-effector contacting the back of the target object; determine the back flexibility corresponding to the target object based on the second contact coordinates and the contact force; and determine the target skeletal position based on the back height curve and the back flexibility. The contact data includes the first contact coordinates, the second contact coordinates, and the contact force. The target acupoint recognition module is used to determine the target back acupoint coordinates corresponding to the target object based on the initial back acupoint coordinates, the back mask, and the target bone position.

8. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images