Multi-method fusion hand acupoint positioning method and system, and electronic device

Abstract

The application discloses a multi-method fusion hand acupoint positioning method and system and electronic equipment, and belongs to the technical field of medical image processing. The method is based on machine learning and digital image processing technology, combined with the acupoint positioning theory of traditional Chinese medicine, and adopts a targeted method for accurate positioning of acupoints for different acupoints. In the application, the hand acupoints are divided into four categories. The first category is acupoints that can be positioned by hand key points according to the positioning method of traditional Chinese medicine. The second category is acupoints that need to be positioned by nail positions according to the positioning method of traditional Chinese medicine. The third category is acupoints located at the edge position of the hand. The fourth category is acupoints without obvious image features. According to the different categories of hand acupoints, different positioning methods are selected, and the augmented reality method is used to superimpose the acupoint information on the real hand image. It is verified that the application can more accurately position the hand acupoints and has a more intuitive display effect.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of stereovision, and particularly relates to a multi-method fusion hand acupoint positioning method and system and electronic equipment. BACKGROUND

[0002] Common two-dimensional images lack depth information, and the depth information is extremely important for us to perceive the three-dimensional world. By simulating the binocular perception mechanism of human beings, binocular stereovision acquires left and right images with certain angle and position deviation, calculates the disparity of each pixel point in the images, and then uses a triangular formula to restore the depth information of the photographed scene, and has great application prospects in the fields of three-dimensional reconstruction, automatic driving, self-navigation and augmented reality.

[0003] Traditional stereomatching methods are fast and easy to implement, but have low precision, while learning-based stereomatching methods can often achieve high matching precision. However, many learning-based stereomatching methods regard the channel and spatial features of each pixel point as equally important in the feature extraction stage, which is not conducive to the full expression of effective features for the stereomatching task, and gives the opportunity for invalid features to hinder the stereomatching process, resulting in a decrease in the final matching precision. At the same time, stereomatching for weak texture regions is still an important direction for research on binocular stereomatching.

[0004] Therefore, the application provides a multi-method fusion hand acupoint positioning method and electronic equipment to solve the above problems. SUMMARY

[0005] The purpose of the embodiments of the application is to provide a multi-method fusion hand acupoint positioning method and electronic equipment, which solves the problems of large hand acupoint positioning granularity and inaccuracy in the prior art.

[0006] To solve the above technical problems, the application is implemented as follows:

[0007] In a first aspect, the embodiments of the application provide a multi-method fusion hand acupoint positioning method, comprising:

[0008] Collecting hand image information, performing data labeling, including left and right hand information and nail position information, and pre-processing the data to form sample data;

[0009] Generating a data set for training a nail positioning model according to the formed sample data;

[0010] According to the formed sample data, a key point positioning model is used to locate the hand key points of the sample to generate a data set for training a direct acupoint positioning model;

[0011] The data set for training the nail positioning model is proportionally divided to form a training set and a test set, and the nail positioning model is trained;

[0012] The data set for training the direct acupoint positioning model is proportionally divided to form a training set and a test set, and the direct acupoint positioning model is trained;

[0013] The hand image or real-time video is obtained through the AR glasses, and the image information is transmitted to the hand positioning system;

[0014] Different models or methods are adopted for different acupoints, and the acupoint positioning is performed in combination with the traditional Chinese medicine acupoint positioning theory;

[0015] The acupoint position information and name information are displayed on the corresponding positions of the hand through augmented reality.

[0016] Optionally, the hand image information includes:

[0017] The left hand palm image, the left hand back image, the right hand palm image, and the right hand back image;

[0018] The four images are collected in equal proportions, and the posture of the hand needs to be placed flat;

[0019] The nail position information includes:

[0020] A rectangular frame is used for labeling;

[0021] The data preprocessing includes:

[0022] Image rotation, image enhancement, contrast adjustment, and brightness adjustment.

[0023] Optionally, the hand key point of the sample is positioned by the key point positioning model, and includes:

[0024] The MediaPipe hand key point positioning algorithm is adopted to position 21 hand key points, and the hand key point coordinates are (x i ,y i ,z i ), wherein the subscript i represents the index of the key point; x i represents the proportional position of the i-th key point in the horizontal direction of the image, and the proportion multiplied by the length of the horizontal direction of the picture is the horizontal coordinate of the i-th key point; y i represents the proportional position of the i-th key point in the vertical direction of the image, and the proportion multiplied by the height of the vertical direction of the picture is the vertical coordinate of the i-th key point; z i represents the depth information of the i-th key point relative to the wrist.

[0025] Optionally, the training of the nail positioning model comprises:

[0026] The YOLOv5 algorithm is used to train the nail positioning model.

[0027] The nail positioning model comprises:

[0028] For all existing nail positions in the image, for each nail position, a rectangular frame formed by (x begin ,y begin ) and (x end ,y end ) is used to represent the position, wherein (x begin ,y begin ) represents the coordinates of the starting point of the rectangular frame, and (x end ,y end ) represents the coordinates of the end point of the rectangular frame.

[0029] Optionally, the direct acupoint positioning model comprises:

[0030] An input layer having 63 neurons;

[0031] Two hidden layers;

[0032] An output layer for positioning the x and y coordinates of the acupoint.

[0033] Optionally, the different models or methods comprise:

[0034] According to different acupoint characteristics, different positioning methods are used, comprising:

[0035] A MediaPipe hand key point positioning model is called to locate 21 hand key points;

[0036] According to the information, the hand region in the image is obtained;

[0037] The length of the finger and the body inch is calculated through the 10th and 11th key points;

[0038] A nail recognition model is called, and the result of the model recognition is saved; if the recognition result is not empty, it indicates that the current image is the back of the hand; if the recognition result is empty, it indicates that the current image is the palm of the hand; if the image is the palm of the hand, the positioning of the palm acupoint is performed; if it is the back of the hand, the positioning of the back acupoint is performed.

[0039] Optionally, the positioning of the acupoint in combination with the traditional Chinese medicine acupoint positioning theory comprises:

[0040] For the first type of acupoint, the hand key point is used as the reference, the acupoint position is positioned according to the traditional Chinese medicine acupoint positioning method, and the finger and the body inch are offset;

[0041] For the second type of acupoint, according to the hand position to which the target acupoint belongs, the nail position information belonging to the position is obtained. Taking the nail position as a reference, according to the acupoint positioning method of traditional Chinese medicine, the acupoint position is positioned according to the offset of the finger to the body inch;

[0042] For the third type of acupoint, the Canny operator is used for hand edge detection to obtain hand contour information in the current image, according to the hand position to which the target acupoint belongs, the hand key point closest to the position is selected, the search direction is selected according to the acupoint positioning method of traditional Chinese medicine, the key point is taken as the starting point, and the search is performed along the search direction, and the search is stopped when the hand contour is reached, and the position of the acupoint is obtained.

[0043] For the fourth type of acupoint, the x, y, z data of the 21 hand key points of the target acupoint are input into the direct acupoint positioning model, and after the model calculation, the x, y coordinates of the acupoint are obtained.

[0044] In a second aspect, the embodiments of the present application provide a multi-method fusion hand acupoint positioning system, comprising:

[0045] A sample data forming module is configured to collect hand image information, perform data labeling, including left and right hand information and nail position information, and pre-process the data to form sample data;

[0046] A first data set generating module is configured to generate a data set for training a nail positioning model according to the formed sample data;

[0047] A second data set generating module is configured to generate a data set for training a direct acupoint positioning model by using a key point positioning model to position hand key points of the sample according to the formed sample data;

[0048] A nail positioning model training module is configured to divide the data set for training the nail positioning model according to a proportion to form a training set and a test set, and train the nail positioning model;

[0049] A direct acupoint positioning model training module is configured to divide the data set for training the direct acupoint positioning model according to a proportion to form a training set and a test set, and train the direct acupoint positioning model;

[0050] An image information transmission module is configured to obtain a hand image or real-time video through AR glasses, and transmit image information to the hand positioning system;

[0051] An acupoint positioning module is configured to adopt different models or methods for different acupoints, and combine the acupoint positioning theory of traditional Chinese medicine to position the acupoints;

[0052] A display module is configured to display acupoint position information and name information to the corresponding position of the hand through augmented reality.

[0053] In a third aspect, an electronic device is provided, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the method according to the first aspect.

[0054] In a fourth aspect, a readable storage medium is provided, which stores a program or instructions, and the program or instructions, when executed by a processor, implement the steps of the method according to the first aspect.

[0055] In the embodiments of the present application, a more fine-grained hand acupoint positioning method is proposed, and different positioning methods are adopted for different categories of hand acupoints. On the basis of considering hand key point information, further consideration is given to nail information and hand contour information, and an acupoint positioning method without obvious image features is proposed, thereby improving the positioning accuracy of hand acupoints. Secondly, the hand acupoints are divided into four categories in the present application, and positioning methods are proposed for each category, rather than different positioning methods for each acupoint. On the premise of ensuring positioning accuracy, the workload of system development is reduced. Thirdly, the present application uses augmented reality technology to display acupoint information, which is superimposed and displayed in the real field of view, is more intuitive, and does not need to be held for shooting when positioning the hand acupoints, which is more convenient. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a positioning method implementation step diagram provided by the embodiments of the present application;

[0057] Figure 2 is a schematic diagram of 21 hand key points provided by the embodiments of the present application;

[0058] Figure 3 is a positioning method diagram when positioning each acupoint according to the embodiments of the present application;

[0059] Figure 4 (a) to (d) are hand acupoint positioning schematic diagrams provided by the embodiments of the present application;

[0060] Figure 5 is a structural block diagram of a multi-method fusion hand acupoint positioning system provided by the embodiments of the present application;

[0061] Figure 6 is a structural block diagram of an electronic device provided by the embodiments of the present application;

[0062] Figure 7 is a hardware structure diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0063] With reference to the drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of them. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0064] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a category and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the objects before and after are in an "or" relationship.

[0065] The multi-method fusion hand acupoint positioning method provided by the embodiments of the present application will be described in detail below in combination with the drawings, through specific embodiments and their application scenarios.

[0066] Please refer to Figure 1 The multi-method fusion hand acupoint positioning method provided by the embodiments of the present application, comprising:

[0067] Step S100, collecting hand image information, performing data labeling, including left and right hand information and nail position information, and pre-processing the data to form sample data;

[0068] Step S101, generating a data set for training a nail positioning model according to the formed sample data;

[0069] Step S102, positioning hand key points of the sample using a key point positioning model according to the formed sample data, and generating a data set for training a direct acupoint positioning model;

[0070] Step S103, dividing the data set for training the nail positioning model according to a proportion to form a training set and a test set, and training the nail positioning model;

[0071] Step S104, dividing the data set for training the direct acupoint positioning model according to a proportion to form a training set and a test set, and training the direct acupoint positioning model;

[0072] Step S105, obtaining a hand image or real-time video through AR glasses, and transmitting the image information to the hand positioning system;

[0073] Step S106, different models or methods are taken for different acupoints, and acupoint positioning is performed in combination with the theory of acupoint positioning in traditional Chinese medicine;

[0074] Step S107, acupoint position information and name information are displayed to the corresponding positions of the hands in an augmented reality manner.

[0075] In step S100, the hand image information includes:

[0076] Four images of a left hand palm image, a left hand back image, a right hand palm image, and a right hand back image; wherein the four images are collected in equal proportions, and the posture of the hand needs to be placed flat, that is, the hand does not perform bending, fist closing, and other actions.

[0077] The nail position information includes:

[0078] A rectangular box is used for labeling;

[0079] The data preprocessing includes but is not limited to:

[0080] Image rotation, image enhancement, contrast adjustment, brightness adjustment, and other methods.

[0081] It should be noted that special cases such as deformities and disabilities are not considered, and each image sample contains one hand, and the position information of five nails is labeled for each hand.

[0082] In step S102, the key point positioning model is used to locate the key points of the hand of the sample, including:

[0083] The MediaPipe hand key point positioning algorithm is used to locate 21 hand key points, and details are shown in Figure 2 The hand key point coordinates are (x i ,y i ,z i ), wherein the subscript i represents the index of the key point; x i represents the proportional position of the i-th key point in the horizontal direction of the image, and the proportion multiplied by the length of the horizontal direction of the picture is the horizontal coordinate of the i-th key point; y i represents the proportional position of the i-th key point in the vertical direction of the image, and the proportion multiplied by the height of the vertical direction of the picture is the vertical coordinate of the i-th key point; z i represents the depth information of the i-th key point relative to the wrist.

[0084] It should be noted that, considering the relevance of human body structure, although the fourth type of acupoint does not have obvious image features, it is still associated with other parts of the hand. Therefore, the data set constructed in step S102 includes x, y, z axis information of the 21 key points of the hand and x, y axis information of the acupoint to be located.

[0085] In step S103, the training of the nail positioning model is performed, including:

[0086] The YOLOv5 algorithm is used to train the nail positioning model. The trained nail positioning model has two functions: one is to locate the position of the nail, and the other is to determine whether the hand image is a palm or a back according to the model output.

[0087] The nail positioning model includes:

[0088] The positions of all existing nails in the image are located. For the position of each nail, a rectangular frame is formed using (x begin ,y begin ) and (x end ,y end ), where (x begin ,y begin ) represents the coordinates of the starting point of the rectangular frame, and (x end ,y end ) represents the coordinates of the end point of the rectangular frame.

[0089] It should be noted that in step S103, the data set is randomly divided to avoid the influence of data order on the model, and to ensure that the left / right hand and palm / back images are distributed equally in the training set and test set.

[0090] In step S104, the direct acupoint positioning model includes:

[0091] The input layer has 63 neurons;

[0092] The hidden layer has two layers;

[0093] The output layer is used to locate the x, y coordinates of the acupoint.

[0094] In addition, the learning rate lr of the model is 0.001, and the loss function is MSE.

[0095] The model trained in step S104 can directly locate acupoints without obvious image features, and the model runs quickly and can accurately locate the fourth type of acupoint.

[0096] It should be noted that in step S104, the data set is divided randomly to avoid the influence of data order on the model, and to ensure that the left / right hand, palm / back of the hand four images are distributed in the training set and test set at equal proportion.

[0097] In step S105, the head-mounted AR glasses are used to obtain hand image information, which has the advantage of mobile platform and frees the hands, so that hand-held shooting is not needed, and hand image information can be obtained more conveniently.

[0098] Referring to Figure 3 and 4 (a)~(d) shown in step S106, different models or methods are adopted, including:

[0099] According to different characteristics of the acupoints, different positioning methods are adopted, including:

[0100] The MediaPipe hand key point positioning model is called to locate 21 hand key points;

[0101] According to the information, the hand region in the image is obtained;

[0102] The length of the finger and the body inch is calculated through the 10th and 11th key points;

[0103] The nail recognition model is called, and the result of model recognition is saved; if the recognition result is not empty, it indicates that the current image is the back of the hand; if the recognition result is empty, it indicates that the current image is the palm; if the image is the palm, the palm acupoint positioning is performed; if it is the back of the hand, the back of the hand acupoint positioning is performed.

[0104] In addition, it should be noted that in the case of being able to distinguish the palm and the back of the hand, the current image can be judged to be the left hand or the right hand according to the key point information, and the information needs to be used when performing coordinate offset.

[0105] In step S106, the acupoint positioning is performed in combination with the traditional Chinese medicine acupoint positioning theory, including:

[0106] For the first type of acupoint, the hand key point is taken as the reference, the acupoint position is located according to the traditional Chinese medicine acupoint positioning method, and the hand finger and body inch are offset;

[0107] For the second type of acupoint, the nail position information belonging to the position of the target acupoint is obtained according to the position of the hand belonging to the target acupoint. The nail position is taken as the reference, the acupoint position is located according to the traditional Chinese medicine acupoint positioning method, and the hand finger and body inch are offset;

[0108] For the third type of acupoint, the Canny operator is used for hand edge detection to obtain the hand contour information in the current image. According to the hand position of the target acupoint, the nearest hand key point to the position is selected. According to the acupoint positioning method of traditional Chinese medicine, the search direction is selected. The key point is taken as the starting point, and the search is carried out along the search direction. The search is stopped when the hand contour is reached, and the position of the acupoint is obtained.

[0109] For the fourth type of acupoint, the x, y, z data of the 21 hand key points of the target acupoint are input into the direct acupoint positioning model. After the model calculation, the x, y coordinates of the acupoint are obtained.

[0110] In the following, a multi-method fusion hand acupoint positioning method provided by the embodiment of the present application is described in detail with specific embodiment 1.

[0111] Embodiment 1

[0112] The first type of acupoint takes the Shàofǔ acupoint as an example. In the theory of traditional Chinese medicine acupoints, the Shàofǔ acupoint is located at the proximal end of the 5th metacarpal joint, between the 4th and 5th metacarpal bones. According to the 13th and 17th key points, the Shàofǔ acupoint can be positioned. The 13th key point coordinate is (x 13 ,y 13 ), and the 17th key point coordinate is (x 17 ,y 17 ). The Shàofǔ acupoint coordinate can be obtained as:

[0113]

[0114] The second type of acupoint takes the left hand Shāngyáng acupoint as an example. In the theory of traditional Chinese medicine acupoints, the Shāngyáng acupoint is located at the radial side of the distal segment of the index finger, 0.1 cùn above the nail root angle. Therefore, the acupoint can be positioned by the index finger nail position and the same body length. Through the 10th and 11th key points, the same body length is calculated as:

[0115]

[0116] The left hand index finger nail coordinate positioned is (x e ,y e ). The Shāngyáng acupoint coordinate can be obtained as:

[0117]

[0118] The third type of acupoint takes the Zhōngchōng acupoint as an example. In the theory of traditional Chinese medicine acupoints, the Zhōngchōng acupoint is located at the highest point of the middle finger tip. Therefore, it can be positioned according to the key point 12 and the hand contour extracted by the Canny operator. Let the extracted hand contour be c. Then, take the key point 12 as the starting point, search upwards, and stop searching when the contour c is detected. The position coordinate at this time is the coordinate of the Zhōngchōng acupoint.

[0119] The fourth type of acupoint takes Hegu acupoint as an example. In the acupoint theory of traditional Chinese medicine, Hegu acupoint is located at the midpoint of the radial side of the second metacarpal bone. However, the length of the second metacarpal bone cannot be determined by only 21 key points, and thus the midpoint position cannot be determined, and the acupoint cannot be directly located according to the hand key points. Therefore, for this acupoint, the x, y and z coordinate information of the 21 key points is input into the direct acupoint positioning model, and the coordinates of the acupoint are output through model calculation.

[0120] Referring to Figure 5 The embodiment of the present application provides a multi-method fusion hand acupoint positioning system, which comprises a sample data forming module 1, a first data set generating module 2, a second data set generating module 3, a nail positioning model training module 4, a direct acupoint positioning model training module 5, an image information transmission module 6, an acupoint positioning module 7 and a display module 8.

[0121] The sample data forming module 1 is used to collect hand image information, perform data labeling, including left and right hand information and nail position information, and pre-process the data to form sample data.

[0122] The first data set generating module 2 is used to generate a data set for training a nail positioning model according to the formed sample data.

[0123] The second data set generating module 3 is used to locate the hand key points of the sample by using a key point positioning model according to the formed sample data, and generate a data set for training a direct acupoint positioning model.

[0124] The nail positioning model training module 4 is used to divide the data set for training the nail positioning model according to a proportion to form a training set and a test set, and train the nail positioning model.

[0125] The direct acupoint positioning model training module 5 divides the data set for training the direct acupoint positioning model according to a proportion to form a training set and a test set, and trains the direct acupoint positioning model.

[0126] The image information transmission module 6 is used to obtain a hand image or real-time video through AR glasses, and transmit the image information to the hand positioning system.

[0127] The acupoint positioning module 7 is used to adopt different models or methods for different acupoints, and combine the acupoint positioning theory of traditional Chinese medicine to position the acupoints.

[0128] The display module 8 is used to display the acupoint position information and name information to the corresponding position of the hand through augmented reality.

[0129] As Figure 6As shown, the embodiment of the present application further provides an electronic device 600, which comprises a processor 601, a memory 602, a program or instruction stored in the memory 602 and executable on the processor 601, the program or instruction is executed by the processor 601 to realize the processes of the above-mentioned multi-method fusion hand acupoint positioning method embodiments and achieve the same technical effects. To avoid repetition, details are not described here.

[0130] It should be noted that the first electronic device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0131] Figure 7 To realize the hardware structure of an electronic device in the embodiment of the present application.

[0132] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710, etc.

[0133] Those skilled in the art can understand that the electronic device 700 can further include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 710 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the figure, or combine certain components, or different component arrangements, which are not described here.

[0134] It should be understood that in the embodiments of the present application, the input unit 704 can include a graphics processor (GPU) 7041 and a microphone 7042. The graphics processor 7041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 706 can include a display panel 7061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 can include two parts of a touch detection device and a touch controller. The other input devices 7072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), a trackball, a mouse, a joystick, and the like, which will not be described here. The memory 709 can be used to store software programs and various data, including but not limited to application programs and operating systems. The processor 710 can integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 710.

[0135] The embodiments of the present application also provide a readable storage medium, which stores programs or instructions, and the programs or instructions are executed by a processor to realize the processes of the above-mentioned multi-method fusion hand acupoint positioning method embodiments and achieve the same technical effects. To avoid repetition, details are not described here.

[0136] The processor is the processor in the electronic device described in the above-mentioned embodiments. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0137] The embodiments of the present application further provide a chip, which includes a processor and a communication interface. The communication interface is coupled with the processor. The processor is used to run programs or instructions to realize the processes of the above-mentioned multi-method fusion hand acupoint positioning method embodiments and achieve the same technical effects. To avoid repetition, details are not described here.

[0138] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip, etc.

[0139] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element. Furthermore, it is to be understood that the method and apparatus of the present application can be carried out by more than one process, method, article, or apparatus either simultaneously, concurrently, or with intervening action that are carried out at the same time, either in a simultaneous fashion or in a fashion that is interleaved in time. For example, the described methods can be performed in a different order from that described, and / or various steps can be combined or omitted, and / or additional steps can be added, without departing from the scope of the present application. Also, features described with respect to certain examples can be combined in other examples.

[0140] From the above description of the embodiments, it is apparent that the above-described method of the embodiments can be realized by means of software and general-purpose hardware platforms, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such an understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a number of instructions for causing a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the methods described in the various embodiments of the present application.

[0141] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are merely illustrative, rather than limiting, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the protection scope of the claims, and all of them belong to the protection scope of the present application.

Claims

1. A multi-method fusion hand acupoint positioning method, characterized in that, The method comprises the following steps: Collect hand image information, perform data labeling, including left and right hand information and nail position information, and pre-process the data to form sample data; According to the formed sample data, generate a data set for training a nail positioning model; According to the formed sample data, use a key point positioning model to locate the hand key points of the sample, and generate a data set for training a direct acupoint positioning model; Divide the data set for training the nail positioning model according to a proportion to form a training set and a test set, and train the nail positioning model; Divide the data set for training the direct acupoint positioning model according to a proportion to form a training set and a test set, and train the direct acupoint positioning model; Obtain a hand image or real-time video through AR glasses, and transmit the image information to a hand acupoint positioning system; For different acupoints, different models or methods are adopted, and the acupoint positioning theory of traditional Chinese medicine is combined to locate the acupoints; The acupoint position information and name information are displayed on the corresponding position of the hand through augmented reality.

2. The method of claim 1, wherein: The hand image information comprises: Four images of a left hand palm image, a left hand back image, a right hand palm image, and a right hand back image; The four images are collected in equal proportions, and the posture of the hand needs to be flat; The nail position information comprises: A rectangular frame is used for labeling; The data pre-processing comprises: Image rotation, image enhancement, contrast adjustment, and brightness adjustment.

3. The method of claim 1, wherein: The key point positioning model is used to locate the hand key points of the sample, which comprises: The 21 hand key points are located by using the MediaPipe hand key point positioning algorithm, and the hand key point coordinates are (x i , y i ,z i ), wherein the subscript i represents the index of the key point; x i represents the proportional position of the i-th key point in the horizontal direction of the image, and the proportion multiplied by the length of the horizontal direction of the picture is the horizontal coordinate of the i-th key point; y i represents the proportional position of the i-th key point in the vertical direction of the image, and the proportion multiplied by the height of the vertical direction of the picture is the vertical coordinate of the i-th key point; z i represents the depth information of the i-th key point relative to the wrist.

4. The method of claim 1, wherein: The training of the nail positioning model comprises: The YOLO v5 algorithm is used to train the nail positioning model; The nail positioning model comprises: Positioning all the present nail positions in the image, for each nail position, using a rectangular frame formed by (x begin , y begin ) and (x end , y end ), where (x begin , y begin ) represents the coordinates of the starting point of the rectangular frame and (x end , y end ) represents the coordinates of the end point of the rectangular frame.

5. The method of claim 1, wherein: The direct acupoint positioning model comprises: An input layer with 63 neurons; Two hidden layers; An output layer for positioning the x and y coordinates of the acupoints.

6. The method of claim 1, wherein: The different models or methods comprise: Different positioning methods are adopted according to different acupoint characteristics, which comprise: The MediaPipe hand key point positioning model is called to locate 21 hand key points; According to the information, the hand region in the image is obtained; The length of the finger and the body inch is calculated through the 10th and 11th key points; The nail recognition model is called, and the recognition result of the model is saved; if the recognition result is not empty, it indicates that the current image is the back of the hand; if the recognition result is empty, it indicates that the current image is the palm of the hand; if the image is the palm of the hand, the palm acupoint is positioned; if it is the back of the hand, the back acupoint is positioned.

7. The method of claim 1, wherein: The acupoint positioning theory of traditional Chinese medicine is combined to locate the acupoints, which comprises: For the first type of acupoint, the hand key points are used as the reference, the acupoint position is located according to the traditional Chinese medicine acupoint positioning method, and the hand finger and body inch are offset; For the second type of acupoint, the nail position information belonging to the position of the target acupoint is obtained according to the position of the target acupoint; the nail position is used as the reference, the acupoint position is located according to the traditional Chinese medicine acupoint positioning method, and the hand finger and body inch are offset. For the third type of acupoint, the Canny operator is used for hand edge detection to obtain the hand contour information in the current image. According to the hand position of the target acupoint, the nearest hand key point to the position is selected. According to the acupoint positioning method of traditional Chinese medicine, the search direction is selected. The key point is taken as the starting point, and the search is carried out along the search direction. The search is stopped when the hand contour is reached, and the position of the acupoint is obtained. For the fourth type of acupoint, the x, y, z data of the 21 hand key points of the target acupoint are input into the direct acupoint positioning model. After the model calculation, the x, y coordinates of the acupoint are obtained.

8. A multi-method fusion hand acupoint positioning system, characterized in that, It comprises: A sample data forming module is used to collect hand image information, perform data labeling, including left and right hand information and nail position information, and pre-process the data to form sample data; A first data set generation module is used to generate a data set for training a nail positioning model according to the formed sample data; A second data set generation module is used to generate a data set for training a direct acupoint positioning model by positioning the hand key points of the sample using a key point positioning model according to the formed sample data; A nail positioning model training module is used to divide the data set for training the nail positioning model according to a proportion to form a training set and a test set, and train the nail positioning model; A direct acupoint positioning model training module is used to divide the data set for training the direct acupoint positioning model according to a proportion to form a training set and a test set, and train the direct acupoint positioning model; An image information transmission module is used to obtain hand images or real-time videos through AR glasses, and transmit image information to a hand acupoint positioning system; An acupoint positioning module is used to adopt different models or methods for different acupoints, and combine the acupoint positioning theory of traditional Chinese medicine to position the acupoints; A display module is used to display the acupoint position information and name information to the corresponding position of the hand through augmented reality.

9. An electronic device, comprising: It comprises a processor, a memory, and a program or instruction stored on the memory and executable on the processor, which implements the steps of the multi-method fusion hand acupoint positioning method according to any one of claims 1 to 8 when executed by the processor.

10. A readable storage medium, characterized by, The readable storage medium stores a program or instruction, which implements the steps of the multi-method fusion hand acupoint positioning method according to any one of claims 1 to 8 when executed by the processor.

Images