Traditional Chinese medicine finger diagnosis gesture recognition method and device based on region segmentation

Abstract

The invention provides a traditional Chinese medicine finger diagnosis gesture recognition method and device based on region segmentation, and belongs to the technical field of image recognition. A patient back image including a doctor finger diagnosis gesture is collected, human skeleton key points in the patient back image are obtained based on a human body posture evaluation model, and a back closed contour is drawn; generating a back segmentation image, and performing primary recognition on the back image of the patient based on a hand recognition model to obtain all gesture targets; judging whether each gesture target is located outside the back closed contour based on the back segmentation image, and cutting off the gesture target located outside the back closed contour from the patient back image to obtain a new patient back image; and performing secondary recognition on the back image of the new patient based on the hand recognition model to obtain a final gesture target. Therefore, the detection rate of the finger diagnosis gestures in the human body area is improved through secondary recognition for scenes with interference gestures.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a traditional Chinese medicine finger diagnosis gesture recognition method and device based on region segmentation. BACKGROUND

[0002] Traditional Chinese medicine physiotherapy robots, such as massage robots and cognitive assessment robots, generally carry gesture recognition functions to assist in locating pain points or assessing cognitive functions. A common gesture recognition method is to use a deep learning neural network recognition method. First, a network model is obtained by using a large number of hand-labeled images as training materials. The commonly used hand recognition model is the open source model mediapipe provided by Google. The image is extracted by a convolution kernel and enters the fully connected layer in the form of a feature vector. The feature vector is linearly changed by the trained weight and bias to obtain the classification probability of different positions of the image. Finally, a classifier such as a SoftMax classifier is used to output the recognition result. This method has high recognition rate and universality, and can recognize hands in a cluttered environment.

[0003] However, the existing hand recognition method has weak recognition ability for special gestures (such as three fingers bent and two fingers stretched). Especially in the case where a special gesture target and a regular hand target exist in the image at the same time, the actual detection rate of the special gesture target is often very low. The reason is that the classification probability of the special gesture target is much lower than that of the regular hand target. When both exist in the picture at the same time, the special gesture target is often filtered out as noise. In the treatment process of traditional Chinese medicine physiotherapy robots (such as massage robots and shock wave robots), doctors will use special finger diagnosis gestures, i.e. the above-mentioned special gesture target, to find pain points on patients. The robot needs to recognize the doctor's finger diagnosis gesture to locate the pain point. However, the patient's hands are also in the picture in the treatment scene, and the patient's hands are in a stretched state, i.e. the above-mentioned regular hand target. If the traditional hand recognition method is used, the finger diagnosis gesture cannot be recognized. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a traditional Chinese medicine finger diagnosis gesture recognition method and device based on region segmentation, which can improve the detection rate of finger diagnosis gestures in the human body region through secondary recognition.

[0005] The traditional Chinese medicine finger diagnosis gesture recognition method based on region segmentation provided by the embodiments of the present application comprises the following steps: Collecting a patient's back image including a doctor's finger diagnosis gesture, and obtaining human body skeleton key points in the patient's back image based on a human body posture evaluation model; Using the human body skeleton key points to outline a back closed contour and generate a back segmentation image, and performing primary recognition on the patient's back image based on a hand recognition model to obtain all gesture targets; determining whether each of the gesture targets is outside the closed contour of the back based on the back segmentation image, and cropping the gesture targets outside the closed contour of the back from the patient back image to obtain a new patient back image; performing secondary recognition on the new patient back image based on a hand recognition model to obtain a final gesture target.

[0006] In some embodiments, after the patient back image of the doctor's palpation gesture is collected, the method further comprises the following steps: aligning the patient back image to obtain a three-dimensional coordinate of each pixel point.

[0007] In some embodiments, the human body skeletal points include a left neck key point, a right neck key point, a left shoulder key point, a right shoulder key point, a left elbow key point, a right elbow key point, a left crotch key point, and a right crotch key point, and the closed contour of the back is outlined by using the human body skeletal points to generate a back segmentation image, comprising the following steps: creating a full black image with the same size as the patient back image, and determining a left waist line key point and a right waist line key point based on the left elbow key point and the right elbow key point; connecting the left neck key point, the right neck key point, the right shoulder key point, the right waist line key point, the right crotch key point, the left crotch key point, the left waist line key point, and the left shoulder key point in sequence to form a closed contour of the back; filling the closed contour of the back with white in the full black image to obtain a back segmentation image.

[0008] In some embodiments, the determination of whether each of the gesture targets is outside the closed contour of the back based on the back segmentation image comprises the following steps: mapping each of the gesture targets to a corresponding position of the back segmentation image, and obtaining a gray value of each position; determining whether the corresponding gesture target is outside the closed contour of the back or inside the closed contour of the back according to the gray value.

[0009] In some embodiments, the cropping of the gesture targets outside the closed contour of the back from the patient back image to obtain a new patient back image comprises the following steps: taking the gesture target outside the closed contour of the back as a center point, and expanding according to a set boundary parameter to construct a mask region; filling the mask region with black in the patient back image to obtain a new patient back image.

[0010] In some embodiments, the method further comprises the steps of: The hand gesture target is obtained by performing secondary recognition on the new patient back image based on the hand recognition model, and the hand gesture target is mapped to a corresponding position of the back segmentation image. It is determined whether the gray value of the position is the same as the gray value of the pixels within the back closed contour. If the gray values are the same, the hand gesture target is determined as the final hand gesture target.

[0011] In some embodiments, the method further comprises the steps of: If the hand gesture targets obtained by the primary recognition are all within the back closed contour, the hand gesture targets are directly determined as the final hand gesture target.

[0012] In some embodiments, a device for recognizing a traditional Chinese finger-pressing gesture based on region segmentation is also provided. The device comprises: An acquisition module is configured to acquire a patient back image including a doctor finger-pressing gesture, and obtain human body skeleton key points in the patient back image based on a human body posture evaluation model; A primary recognition module is configured to outline a back closed contour using the human body skeleton key points, generate a back segmentation image, and perform primary recognition on the patient back image based on a hand recognition model to obtain all hand gesture targets; A judgment module is configured to determine whether each hand gesture target is located outside the back closed contour based on the back segmentation image, and cut the hand gesture targets located outside the back closed contour from the patient back image to obtain a new patient back image; A secondary recognition module is configured to perform secondary recognition on the new patient back image based on a hand recognition model to obtain a final hand gesture target.

[0013] In some embodiments, an electronic device is also provided. The electronic device comprises a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory through the bus. The machine-readable instructions are executed by the processor to perform the steps of any one of the above-mentioned methods for recognizing a traditional Chinese finger-pressing gesture based on region segmentation.

[0014] In some embodiments, a computer-readable storage medium is also provided. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of any one of the above-mentioned methods for recognizing a traditional Chinese finger-pressing gesture based on region segmentation are performed.

[0015] The application discloses a traditional Chinese medicine finger diagnosis gesture recognition method and device based on region segmentation. A patient back image including a doctor finger diagnosis gesture is collected, and human body skeleton key points in the patient back image are acquired based on a human body posture evaluation model; the human body skeleton key points are used to outline a back closed contour, and a back segmentation image is generated, and a hand recognition model is used to recognize the patient back image once to obtain all gesture targets; whether each gesture target is located outside the back closed contour is judged based on the back segmentation image, and the gesture target located outside the back closed contour is cut from the patient back image to obtain a new patient back image; the hand recognition model is used to recognize the new patient back image twice to obtain final gesture targets. Therefore, for a scene with an interference gesture, the interference item is cut through secondary recognition combined with human body region segmentation, and the gesture recognition rate of the target region is significantly improved compared with a traditional method. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the application, and therefore should not be regarded as a limitation to the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0017] Figure 1 A flow chart of the traditional Chinese medicine finger diagnosis gesture recognition method based on region segmentation is shown; Figure 2 A schematic diagram of the human body skeleton key points is shown; Figure 3 A schematic diagram of the left elbow key point and the right elbow key point determining the left waist line key point and the right waist line key point is shown; Figure 4 A deep histogram constructed by the embodiment of the application is shown; Figure 5 A schematic diagram of the back closed contour is shown; Figure 6 A schematic diagram of the back segmentation image is shown; Figure 7 A schematic diagram of the new patient back image is shown; Figure 8 A structural schematic diagram of the traditional Chinese medicine finger diagnosis gesture recognition device based on region segmentation is shown; Figure 9 A structural schematic diagram of the electronic device is shown. DETAILED DESCRIPTION

[0018] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description, and are not used to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flowcharts used in the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts under the guidance of the content of the present application.

[0019] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0020] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0021] In view of the technical problems proposed in the background art, the present application provides a traditional Chinese finger diagnosis gesture recognition method and device based on region segmentation, an electronic device and a storage medium, which can improve the detection rate of finger diagnosis gestures in the human body region through secondary recognition.

[0022] Referring to the drawings accompanying the Figure 1 The traditional Chinese finger diagnosis gesture recognition method based on region segmentation provided by the present application comprises the following steps: S1, collecting a patient back image including a doctor finger diagnosis gesture, and obtaining human body skeleton key points in the patient back image based on a human body posture evaluation model; S2, using the human body skeleton key points to outline a back closed contour, generating a back segmentation image, and performing primary recognition on the patient back image based on a hand recognition model to obtain all gesture targets; S3, judging whether each gesture target is located outside the back closed contour based on the back segmentation image, and cutting the gesture target located outside the back closed contour from the patient back image to obtain a new patient back image. S4, performing secondary recognition on the new patient back image based on the hand recognition model to obtain a final gesture target.

[0023] Specifically, in step S1, after collecting the patient back image including the doctor's palpation gesture, the patient back image also needs to be aligned with three-dimensional point cloud to convert two-dimensional image pixel coordinates into three-dimensional coordinates. Further, data basis is provided for subsequent human region segmentation positioning.

[0024] In an embodiment, the scene point cloud is directly obtained using a three-dimensional vision device, i.e. the depth information of each point in the scene is , and the pixel coordinates of each pixel in the image are . Assuming that the three-dimensional vision device has been calibrated by the camera intrinsic parameters, i.e. the camera focal length fx, fy and the optical center coordinates px, py are known, then the point cloud corresponding to each pixel point is The three-dimensional coordinates are:

[0025]

[0026]

[0027] In addition, in step S1, the open source human pose evaluation model AlphaPose can be called to process the collected patient back image to obtain a pixel coordinate set of human skeleton key points. The process of processing the patient back image by the human pose evaluation model is a technical means known to those skilled in the art, and will not be described here. Meanwhile, referring to the accompanying drawings Figure 2 , the obtained human skeleton key points include left neck key point p3, right neck key point p4, left shoulder key point p5, right shoulder key point p6, left elbow key point p7, right elbow key point p8, left wrist key point P9, right wrist key point P10, left crotch key point p11, and right crotch key point p12.

[0028] In step S2, in the process of using the human skeleton key points to outline the back closed contour to generate the back segmentation image, first, a full black image contourImg is created, and the size of the full black image contourImg is the same as that of the patient back image. Then, the line segment between the left neck key point p3 and the left shoulder key point p5 , the line segment between the right neck key point p4 and the right shoulder key point p6 , and the line segment between the left neck key point p3 and the right neck key point p4 are drawn in the full black image contourImg. The two-point connection formula is as follows.

[0029]

[0030] in,( ) represents the pixel coordinates of the starting point of the line segment. ) represents the pixel coordinates of the endpoint of the line segment. () represents the pixel coordinates of any point on the line segment. In one embodiment, the above line segment can be drawn on a completely black image contourImg using open-source drawing tools such as OpenCV.

[0031] Then, the key points of the left and right waistlines are determined. These key points are based on the key points of the left and right elbows, p7 and p8, respectively. (See the appendix of the instruction manual for details.) Figure 3 The key points of the left and right waistlines are located on the line connecting the key points of the left and right elbows, p7 and p8. superior. You can substitute the formula from the previous step into the calculation.

[0032] Specifically, when determining the key points of the left and right waistlines, ( Starting from ), ( ) is the endpoint for traversal Points on the line are used to input the aligned point cloud to obtain line segments. Record the depth of each point and create a depth histogram (see the instruction manual appendix). Figure 4 For line segments Points on :

[0033]

[0034] ( ) / ))+

[0035] Among them, line segment The depth corresponding to each point can be directly read by the 3D vision device; the GetDist function interface is provided by the device vendor. , and thus depth Create a depth histogram, with the horizontal axis representing the point index and the vertical axis representing the point depth. The depth histogram shows that the depth decreases continuously at the edge of the left waistline key point, then stabilizes, and gradually increases at the edge of the right waistline key point. The method for locating the left waistline key point is to iterate through the depth values ​​of points starting from the beginning of the histogram. When more than 15 consecutive points show a decrease in depth, the last decreasing point is recorded as the left waistline key point. The right waist line point positioning method is to traverse the depth value of the point from the histogram end position, and when the depth of 15 consecutive points is exceeded, the last falling point is recorded as the right waist line key point .

[0036] Then connect the left shoulder key point p5 and the left waist line key point Get the line segment , connect the right shoulder key point p6 and the right waist line key point Get the line segment , connect the left waist line key point And the left crotch key point p11 to get the line segment , connect the left crotch key point p11 and the right crotch key point p12 to get the line segment , connect the right crotch key point p12 and the right waist line key point Get the line segment . Thus, the sequential connection of the left neck key point, the right neck key point, the right shoulder key point, the right waist line key point, the right crotch key point, the left crotch key point, the left waist line key point and the left shoulder key point is completed, forming a closed contour of the back (see the accompanying drawings Figure 5 ).

[0037] Finally, referring to the accompanying drawings Figure 6 , the open source image processing tool can be used to fill the outlined closed contour of the back with white in the all-black image to obtain a back segmentation image segmImg.

[0038] It should be noted that in other embodiments, different colors can be selected for the generated all-black image and the white-filled closed contour of the back according to specific needs. The present application does not limit and fix this.

[0039] In addition, in step S2, a gesture recognition is also performed to preliminarily search all possible targets in the entire scene, that is, to identify all gesture targets in the original image (patient back image). In an embodiment, the open source hand recognition model mediapipe can be used to identify gesture targets in the patient back image. As described in the background art, in actual application, the patient's hand is often identified, while the doctor's gesture on the patient's back is not detected.

[0040] In step S3, all gesture targets identified once are judged for interference items by the generated back segmentation image segmImg. For example, the present application aims to detect the doctor's gesture, and the patient's hand is an interference item.

[0041] In an embodiment, for the gesture target identified once The corresponding gray value on the back segmentation segmImg is If the gray value is 255, it is considered that the gesture target is located within the back closed contour; if the gray value is 0, it is considered that the gesture target is located outside the back closed contour, that is, the gesture target is an interference item.

[0042] Further, after judging the interference item, the interference item needs to be cut out in the patient back image to generate a new patient back image for secondary recognition.

[0043] In an embodiment, a square with 20 pixels in width and height can be constructed with the interference item coordinate as the center point, and the square is used as a mask area to paint the original image (patient back image) black, and saved as a new recognition image as a new patient back image, as shown in the accompanying drawings. Figure 7

[0044] In step S4, the secondary recognition of the gesture target is performed on the obtained new patient back image, and it is re-judged whether it is an interference item to obtain the final gesture target. Among them, the open-source hand recognition model mediapipe is also used to recognize the gesture target in the new patient back image, and the recognized gesture target is mapped to the corresponding position of the back segmentation image, and the gray value of the position is used to judge whether the gesture target is located outside the back closed contour or inside the back closed contour. If it is judged that the gesture target is outside the back closed contour, it is determined that the gesture target of the doctor is not detected; if it is judged that the gesture target is inside the back closed contour, it is taken as the final gesture target.

[0045] It should be noted that in step S3, if the gesture target recognized once does not exist outside the back closed contour, that is, if no interference item is judged, the recognized gesture target is directly taken as the final gesture target.

[0046] The application provides a traditional Chinese finger diagnosis gesture recognition method based on region segmentation, which provides depth information through three-dimensional point cloud alignment, combines with human body posture evaluation model to segment human back region, and significantly improves the gesture recognition rate of the target region through twice recognition and cutting out of interference items. It shields external interference by human region segmentation, accurately positions through secondary recognition, solves the problem that special gestures are filtered due to low classification probability, and provides key technical support for accurately positioning pain points of traditional Chinese medicine physiotherapy robots.

[0047] ​Based on the same inventive concept, the embodiment of the present application also provides a device for recognizing a TCM finger diagnosis gesture based on region segmentation. Since the principle of the device for solving the problem in the embodiment of the present application is similar to the above-mentioned method for recognizing a TCM finger diagnosis gesture based on region segmentation, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described herein.

[0048] As shown in the accompanying drawings of the specification, Figure 8 The embodiment of the present application also provides a device for recognizing a TCM finger diagnosis gesture based on region segmentation, which comprises: An acquisition module 801 is configured to collect a patient back image including a doctor finger diagnosis gesture, and acquire human body skeleton key points in the patient back image based on a human body posture evaluation model; A first recognition module 802 is configured to outline a back closed contour by using the human body skeleton key points, generate a back segmentation image, and perform first recognition on the patient back image based on a hand recognition model to obtain all gesture targets. A judgment module 803 is configured to judge whether each gesture target is located outside the back closed contour based on the back segmentation image, and cut the gesture target located outside the back closed contour from the patient back image to obtain a new patient back image. A second recognition module 804 is configured to perform second recognition on the new patient back image based on the hand recognition model to obtain a final gesture target.

[0049] In some embodiments, after the acquisition module 801 collects the patient back image including the doctor finger diagnosis gesture, the method further comprises: performing three-dimensional point cloud alignment on the patient back image to obtain three-dimensional coordinates of each pixel point.

[0050] In some embodiments, the human body skeleton key points include a left neck key point, a right neck key point, a left shoulder key point, a right shoulder key point, a left elbow key point, a right elbow key point, a left crotch key point and a right crotch key point, and the first recognition module 802 outlines the back closed contour by using the human body skeleton key points to generate the back segmentation image, which comprises: creating a full black image with the same size as the patient back image, and determining a left waist line key point and a right waist line key point based on the left elbow key point and the right elbow key point; connecting the left neck key point, the right neck key point, the right shoulder key point, the right waist line key point, the right crotch key point, the left crotch key point, the left waist line key point and the left shoulder key point in sequence to form the back closed contour; and performing full white filling on the back closed contour in the full black image to obtain the back segmentation image.

[0051] In some embodiments, the determining module 803 determines whether each of the gesture targets is located outside the closed contour of the back based on the back segmented image, including: mapping each of the gesture targets to a corresponding position of the back segmented image, and obtaining a gray value of each position; and determining, according to the gray value, whether the corresponding gesture target is located outside the closed contour of the back or inside the closed contour of the back.

[0052] In some embodiments, the determining module 803 clips the gesture target located outside the closed contour of the back from the patient back image to obtain a new patient back image, including: taking the gesture target located outside the closed contour of the back as a center point, and expanding according to a set boundary parameter to construct a mask region; and performing full black filling of the mask region in the patient back image to obtain a new patient back image.

[0053] In some embodiments, the secondary recognition module 804 performs secondary recognition on the new patient back image based on a hand recognition model to obtain a final gesture target, including: performing secondary recognition on the new patient back image based on a hand recognition model to obtain a gesture target, and mapping the gesture target to a corresponding position of the back segmented image, re-determining whether a gray value of the position is the same as a gray value of a pixel inside the closed contour of the back, and if so, taking the gesture target as the final gesture target.

[0054] The TCM finger diagnosis gesture recognition device based on region segmentation provided in the application acquires a patient back image including a doctor finger diagnosis gesture through an acquisition module, and acquires human body skeleton key points in the patient back image based on a human body posture evaluation model; a primary recognition module outlines a closed contour of the back by using the human body skeleton key points, generates a back segmented image, and performs primary recognition on the patient back image based on a hand recognition model to obtain all gesture targets; a determining module determines whether each of the gesture targets is located outside the closed contour of the back based on the back segmented image, and clips the gesture target located outside the closed contour of the back from the patient back image to obtain a new patient back image; and a secondary recognition module performs secondary recognition on the new patient back image based on a hand recognition model to obtain a final gesture target. Thus, for a scene in which there is an interfering gesture, the secondary recognition is combined with human body region segmentation, the interfering item is clipped, and then recognition is performed again, and compared with a traditional method, the target region gesture recognition rate is significantly improved.

[0055] Based on the same concept of the application, as described in the description and the drawings Figure 9As shown, the electronic device 900 provided by the embodiment of the present application includes at least one processor 901, at least one network interface 904 or other user interface 903, a memory 905, and at least one communication bus 902. The communication bus 902 is used to realize the connection and communication between the components. The electronic device 900 can optionally include a user interface 903, including a display (for example, a touch screen, an LCD, a CRT, holographic imaging (Holographic) or a projector, etc.), a keyboard or a clicking device (for example, a mouse, a trackball, a touchpad or a touch screen, etc.).

[0056] The memory 905 can include read-only memory and random access memory, and provide instructions and data for the processor 901. A part of the memory 905 can also include a non-volatile random access memory (NVRAM).

[0057] In some embodiments, the memory 905 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: An operating system 9051 containing various system programs, used to realize various basic services and process hardware-based tasks; Application modules 9052 containing various application programs, such as a desktop (launcher), a media player (MediaPlayer), a browser (Browser), etc., used to realize various application services.

[0058] In the embodiment of the present application, the processor 901 is used to execute the steps of a kind of Chinese medicine finger diagnosis gesture recognition method based on region segmentation by calling the program or instruction stored in the memory 905.

[0059] The present application also provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to execute the steps of a kind of Chinese medicine finger diagnosis gesture recognition method based on region segmentation.

[0060] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc., and the computer program on the storage medium can improve the detection rate of finger diagnosis gestures in human body region through secondary identification when executed.

[0061] In the embodiments provided by the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. The embodiments described above are merely exemplary, for example, the division of units is only a logical function division, and there can be another division manner in actual implementation; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between units can be indirect coupling or communication connection through some interfaces, and electrical, mechanical or other forms.

[0062] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. In actual implementation, some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0063] In addition, each functional unit in the embodiments provided by the present application can be integrated in a processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit.

[0064] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially or partly, or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various other media that can store program codes.

[0065] Finally, it should be noted that the above examples are merely specific embodiments of the present application, and are used to illustrate the technical solutions of the present application, but are not limiting thereof, and the protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing examples, it should be understood by those skilled in the art that any person skilled in the art can modify or easily think of changes to the technical solutions recorded in the foregoing examples, or make equivalent replacements to some of the technical features, within the technical range disclosed by the present application, and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. All should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for recognizing TCM finger-prick gestures based on region segmentation, characterized in that, The method includes the following steps: Images of the patient's back, including doctor's finger examination gestures, are collected, and key points of the human skeleton in the patient's back images are obtained based on a human posture evaluation model. The closed outline of the back is delineated using the key points of the human skeleton, and a segmented image of the back is generated. Based on the hand recognition model, the patient's back image is recognized once to obtain all the gesture targets. Based on the back segmentation image, it is determined whether each gesture target is located outside the back closed contour, and the gesture targets located outside the back closed contour are removed from the patient's back image to obtain a new patient back image; The hand recognition model is used to perform secondary recognition on the back image of the new patient to obtain the final gesture target.

2. The method for recognizing TCM finger-prick gestures based on region segmentation according to claim 1, characterized in that, After acquiring images of the patient's back, including those showing the doctor's finger-prick gestures, the process also includes the following steps: The patient's back image was aligned using a 3D point cloud to obtain the 3D coordinates of each pixel.

3. The method for recognizing TCM finger-prick gestures based on region segmentation according to claim 2, characterized in that, The key points of the human skeleton include the key points of the left neck, right neck, left shoulder, right shoulder, left elbow, right elbow, left hip, and right hip. The process of using these key points to outline the closed contour of the back and generate a segmented back image includes the following steps: Create a completely black image of the same size as the patient's back image, and determine the left waistline key point and the right waistline key point based on the left elbow key point and the right elbow key point; Connect the key points of the left neck, right neck, right shoulder, right waistline, right hip, left hip, left waistline, and left shoulder in sequence to form a closed outline of the back. The closed outline of the back is filled with white in the all-black image to obtain a back segmentation image.

4. The method for recognizing TCM finger-prick gestures based on region segmentation according to claim 3, characterized in that, The step of determining whether each gesture target is located outside the closed contour of the back based on the back segmentation image includes the following steps: Each gesture target is mapped to a corresponding position in the back segmentation image, and the grayscale value of each position is obtained; Based on the grayscale value, the corresponding gesture target is determined to be located either outside or inside the closed back contour.

5. The method for recognizing TCM finger-prick gestures based on region segmentation according to claim 4, characterized in that, The step of cropping the gesture target located outside the closed contour of the back from the patient's back image to obtain a new patient's back image includes the following steps: The gesture target located outside the closed outline of the back is taken as the center point and expanded according to the set boundary parameters to construct the mask area; The masked area is filled with black in the patient's back image to obtain a new patient's back image.

6. The method for recognizing TCM finger-prick gestures based on region segmentation according to claim 5, characterized in that, The process of performing secondary recognition on the new patient's back image based on a hand recognition model to obtain the final gesture target includes the following steps: The hand recognition model is used to perform secondary recognition on the back image of the new patient to obtain the gesture target. The gesture target is then mapped to the corresponding position in the back segmentation image. The gray value of the position is then re-evaluated to see if it is the same as the gray value of the pixels within the closed contour of the back. If they are the same, the gesture target is taken as the final gesture target.

7. The method for recognizing TCM finger-prick gestures based on region segmentation according to claim 6, characterized in that, The method further includes the following steps: If all the gesture targets identified in one recognition are within the closed contour of the back, the gesture targets are directly determined as the final gesture targets.

8. A TCM finger-based diagnostic gesture recognition device based on region segmentation, characterized in that, The device includes: The acquisition module is used to acquire images of the patient's back, including doctor's finger examination gestures, and to acquire key points of the human skeleton in the patient's back images based on a human posture evaluation model. The primary recognition module is used to outline the closed contour of the back using the key points of the human skeleton and generate a segmented image of the back. It also performs primary recognition on the patient's back image based on the hand recognition model to obtain all gesture targets. The judgment module is used to determine whether each gesture target is located outside the closed contour of the back based on the back segmentation image, and to remove the gesture targets located outside the closed contour of the back from the patient's back image to obtain a new patient's back image; The secondary recognition module is used to perform secondary recognition on the new patient's back image based on the hand recognition model to obtain the final gesture target.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of a TCM finger-diagnosis gesture recognition method based on region segmentation as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of a TCM finger-diagnosis gesture recognition method based on region segmentation as described in any one of claims 1 to 7.

Images