Auricular point real-time detection and auxiliary positioning method and system and storage medium

Abstract

The invention belongs to the technical field of auricular point recognition, and provides an auricular point real-time detection and auxiliary positioning method and system and a storage medium, and the method specifically comprises the following steps: S1, obtaining an ear image of a user; s2, constructing and deploying a real-time reasoning model, and generating a corresponding ear bounding box and a plurality of acupoint feature points according to detection processing of the real-time reasoning model on the user ear image; and S3, mapping the ear bounding box and the acupoint feature points to an ear image of the user, and outputting an auxiliary positioning image with an auricular point label. According to the auricular point real-time detection and auxiliary positioning method provided by the invention, the auricular point can be efficiently and accurately detected, identified and positioned in real time, the auricular point identification and positioning difficulty is reduced, and a user can be conveniently assisted in rapidly and accurately taking the auricular point.

Description

Technical Field

[0001] The present invention belongs to the technical field of ear acupoint recognition, and in particular relates to a method, system and storage medium for real-time detection and auxiliary positioning of ear acupoints. Background Art

[0002] There are many acupuncture points on the ear, and the distances between each acupuncture point are relatively dense. Currently, it is difficult to accurately locate acupuncture points through manual judgment methods, and it often takes a long time. In addition, due to the differences in the ear structure and specific shapes of different human bodies, and the dense distribution of ear acupuncture points that are difficult to distinguish, the accuracy of positioning is limited by the physician's experience level and the influence of external environments such as light and temperature. As a result, it is extremely difficult for physicians to accurately locate ear acupuncture points. If the wrong acupuncture points are determined, it may lead to failure to achieve the ideal treatment effect. Especially for inexperienced physicians, ear acupuncture point recognition is difficult to get started, and experience accumulation is slow, so relevant experience needs to be accumulated in a large amount of clinical practice.

[0003] Therefore, it is necessary to design a method for real-time detection and auxiliary positioning of ear acupoints that can effectively solve the above problems. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a method for real-time detection and auxiliary positioning of ear acupoints, which can efficiently and accurately detect and identify and locate ear acupoints in real time, reduce the difficulty of ear acupoint identification and positioning, and facilitate auxiliary users to quickly and accurately locate acupoints.

[0005] The technical solution of the present invention is:

[0006] The present invention provides a method for real-time detection and auxiliary positioning of ear acupoints, which specifically comprises the following steps:

[0007] S1, obtaining a user's ear image;

[0008] S2, constructing and deploying a real-time inference model, and generating a corresponding ear bounding box and a plurality of acupoint feature points located in the ear bounding box according to detection and processing of the user's ear image by the real-time inference model;

[0009] S3, mapping the ear boundary box and the acupoint feature points to the user's ear image, and outputting an auxiliary positioning image with ear acupoint annotations.

[0010] Preferably, in step S2, the real-time inference model includes a serially connected RTMDet training model and a RTMPose training model, the RTMDet training model generates the ear bounding box based on the user ear image, and the RTMPose training model generates a plurality of the acupoint feature points located within the ear bounding box based on the user ear image.

[0011] Preferably, in step S2, constructing and deploying a real-time reasoning model includes the following steps:

[0012] Collect natural optical images of the human ear and construct the original dataset;

[0013] The images in the original data set are processed, the ear bounding boxes are annotated to form a sample data set, which is then input into the RTMDet model for learning and training to obtain the RTMDet training model;

[0014] The images in the original data set are processed, and the feature points of acupoints are marked to form a sample data set, which is then input into the RTMPose model for learning and training to obtain the RTMPose training model;

[0015] The RTMDet training model and the RTMPose training model are connected in series to build a real-time inference model.

[0016] Preferably, the learning and training process of the RTMDet training model includes a first iterative training phase using a strong data enhancement algorithm and a second iterative training phase using a weak data enhancement algorithm;

[0017] The first iterative training stage includes Mosaic+MixUp cache acceleration processing, random rotation processing, and random cropping processing, and the second iterative training stage includes random rotation processing and random cropping processing.

[0018] Preferably, the Mosaic+MixUp cache acceleration process comprises the following steps:

[0019] Construct a Mosaic cache queue, load a randomly selected image from the sample data set to initialize the Mosaic cache queue, load an image from the sample data set, update the Mosaic cache queue, and randomly select an image from it for Mosaic processing;

[0020] Construct a MixUp cache queue, load a randomly selected image from the sample data set to initialize the MixUp cache queue, load an image from the sample data set, update the MixUp cache queue, and randomly select an image from it for MixUp processing.

[0021] Preferably, the learning and training process of the RTMPose training model includes a first iterative training phase processed using a strong data enhancement algorithm, and a second iterative training phase processed using a weak data enhancement algorithm;

[0022] The first iterative training stage includes a scale transformation process, a random rotation process, and a random cropping process, and the second iterative training stage includes a random rotation process and a random cropping process.

[0023] Preferably, in step S1, the user ear image is an image frame captured sequentially frame by frame in the video stream.

[0024] Preferably, step S2 also includes image post-processing, and the image post-processing includes frame skip detection processing, posture non-maximum suppression processing, and Euro-filter processing.

[0025] The present invention also proposes a real-time auricular acupoint detection and auxiliary positioning system, which specifically includes:

[0026] An image acquisition module is used to detect and acquire an ear image of a user;

[0027] A real-time reasoning module, used for detecting and processing the user's ear image, generating a corresponding ear boundary box and a plurality of acupoint feature points located within the ear boundary box;

[0028] An auxiliary positioning module, used for mapping the ear boundary box and the acupoint feature points to the user's ear image, and outputting an auxiliary positioning image with ear acupoint annotations in real time;

[0029] The real-time reasoning module includes an image pre-processing unit, an ear position detection unit, an ear acupoint feature point detection unit, and an image post-processing unit.

[0030] The image pre-processing unit pre-processes the acquired user ear image, such as adjusting the image size and denoising; the ear position detection unit uses the RTMDet training model to process the user ear image and generate an ear bounding box; the ear acupoint feature point detection unit uses the RTMPose training model to process the user ear image and generate a number of ear acupoint feature points located within the ear bounding box; the image post-processing unit performs frame skipping detection processing, posture non-maximum suppression processing, and one-Euro filtering processing on the user ear image that has completed the ear bounding box and ear acupoint feature point annotation, so as to improve the smoothness of the real-time output result image.

[0031] The present invention also proposes a computer-readable storage medium on which a computer program is stored. The computer program is loaded by a processor to execute the steps in the above-mentioned ear acupoint real-time detection and auxiliary positioning method.

[0032] Compared with the prior art, the present invention has the following advantages and effects:

[0033] (1) The real-time inference model including the serially connected RTMDet training model and RTMPose training model can improve the positioning accuracy and real-time positioning efficiency of ear acupoints, thereby assisting operators to quickly and accurately locate acupoints, reducing the difficulty of ear acupoint identification and positioning, and making ear acupoint identification, positioning, and acupoint selection clearer, more intuitive, and more efficient;

[0034] (2) The RTMDet training model and the RTMPose training model, which include the first iterative training stage and the second iterative training stage, are used. That is, both the RTMDet and RTMPose training models adopt a "two-stage" data augmentation strategy to ensure that the model is more robust to complex pattern images and videos with various noises and object combinations in the real-world environment, while making the model more focused on the features of the real training data to adapt to normal test scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the process of the real-time detection and auxiliary positioning method of ear acupoints in an embodiment of the present invention;

[0036] Figure 2 It is a flow chart of constructing and deploying a real-time reasoning model in the method for real-time detection and auxiliary positioning of auricular points in an embodiment of the present invention;

[0037] Figure 3 A detailed flow chart of a method for real-time detection and auxiliary positioning of ear acupoints in an embodiment of the present invention;

[0038] Figure 4 It is a schematic diagram of the process of Mosaic (or MixUp) cache acceleration processing in the real-time detection and auxiliary positioning method of ear acupoints in an embodiment of the present invention;

[0039] Figure 5 It is a schematic diagram of manually marking ear boundary boxes and acupoint feature points using labelme software in the real-time detection and auxiliary positioning method of auricular acupoints in an embodiment of the present invention;

[0040] Figure 6 Schematic diagram of actual use effect of the ear acupoint real-time detection and auxiliary positioning method in an embodiment of the present invention;

[0041] Figure 7 It is another practical use effect diagram of the ear acupoint real-time detection and auxiliary positioning method in an embodiment of the present invention;

[0042] Figure 8 Schematic diagram of the structure of the real-time detection and auxiliary positioning system of ear acupoints in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make those skilled in the art better understand the present invention, the present invention is further described in conjunction with specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention.

[0044] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative work shall fall within the scope of protection of the present invention.

[0045] like Figures 1 to 8 As shown, the present invention provides a method for real-time detection and auxiliary positioning of ear acupoints, which specifically includes the following steps:

[0046] S1, obtain the user's ear image; further, in this embodiment, the user's ear image is an image frame captured frame by frame in sequence in the video stream, and the captured image frames are pre-processed, including but not limited to image resizing, image denoising, etc., to obtain the user's ear image of a set size (taking 640×640×3 as an example).

[0047] S2, constructing and deploying a real-time inference model, and generating a corresponding ear bounding box and a number of acupoint feature points within the ear bounding box based on the detection and processing of the user's ear image by the real-time inference model.

[0048] Among them, the real-time inference model includes the RTMDet training model and the RTMPose training model connected in series. The RTMDet training model quickly identifies the position of the human ear in the image or video based on the user's ear image, and generates an accurate ear bounding box (if there are multiple human ear targets in the image or video, multiple ear bounding boxes or multiple regions of interest can be formed), providing a key reference range for the subsequent detection and positioning of acupoint feature points; the RTMPose training model detects and accurately locates specific acupoints on the human ear in real time based on the user's ear image, and generates several acupoint feature points located within the ear bounding box.

[0049] refer to Figure 2 As shown in the figure, building and deploying a real-time inference model specifically includes the following steps:

[0050] Natural optical images of the human ear under multi-resolution and different lighting conditions were collected to construct an original dataset. The inclusion criteria for the natural optical images of the human ear in the original dataset were as follows: the gender of the subjects was evenly distributed, the age of the subjects was ≥18 years old, the area of ​​the real bounding box of the ear accounted for ≥40% of the total image area, and the image acquisition device was perpendicular to the sagittal plane of the subjects.

[0051] It should be noted that by acquiring natural optical images of the human ear in multiple resolutions and different lighting environments, the diversity of data patterns can be effectively increased, making the training model adaptable to real test scenarios with a variety of noises. In addition, because the ear development and acupoint distribution patterns of children are different from those of adults, the subjects are required to be ≥18 years old when collecting data.

[0052] Process the images in the above original dataset, refer to Figure 5 As shown, the ear bounding boxes are manually annotated using the labelme open source annotation software to form a sample data set, which is then input into the RTMDet model for learning and training to obtain the RTMDet training model.

[0053] Furthermore, the above sample data set is divided into a training set and a test set in a ratio of 7:3.

[0054] Among them, the learning and training process of the RTMDet training model includes a first iterative training stage using a strong data enhancement algorithm and a second iterative training stage using a weak data enhancement algorithm; specifically, in this embodiment, taking 200 training iterations as an example, the first iterative training stage is specifically the first 180 training iterations, and the second iterative training stage is specifically the last 20 training iterations.

[0055] Furthermore, the first iterative training stage includes Mosaic+MixUp cache acceleration processing, random rotation processing, and random cropping processing, and the second iterative training stage includes random rotation processing and random cropping processing.

[0056] refer to Figure 4 As shown in the figure, the Mosaic+MixUp cache acceleration process specifically includes the following steps:

[0057] Construct a Mosaic cache queue and load 20 randomly selected images from the sample data set to initialize the Mosaic cache queue. After initialization, each subsequent training step only needs to load a new image and its label from the sample data set stored on the hard disk, update the Mosaic cache queue (the images in the cache queue are repeatable), and randomly select 4 images from the Mosaic cache queue for Mosaic processing.

[0058] Build a MixUp cache queue and load 10 randomly selected images from the sample data set to initialize the MixUp cache queue. After initialization, each subsequent training step only needs to load a new image and its label from the sample data set stored on the hard disk, update the MixUp cache queue (the images in the cache queue are repeatable), and randomly select 2 images from the MixUp cache queue for MixUp processing.

[0059] Compared with the existing traditional hard disk loading images, it is not necessary to load a specific number of images from the hard disk each time (Mosaic requires 4 images, MixUp requires 2 images). Figure 4As shown in the figure: the cache is located inside or close to the CPU, the physical distance of data transmission is very short, and the cache uses high-speed storage media, so the access delay is very low, usually in nanoseconds; the solid-state drive needs to transmit data through the I / O bus and storage controller, and the delay is in microseconds; the data set is stored in the hard disk, and the traditional Mosaic+MixUp algorithm loads several pictures from the hard disk in each training step (usually 4 pictures for Mosaic and 2 pictures for MixUp, Figure 3 Taking 4 images of Mosaic as an example), the time consumed is 4 times that of other common data enhancement algorithms (such as random cropping, etc.). The cache mechanism initializes a cache queue. The images in the initial queue are randomly selected from the data set. The maximum number of images in the cache queue for Mosaic is 20, and the maximum number of images in MixUp is 10. In each training step, only one new image needs to be loaded from the hard disk, and then an existing image is randomly popped out from the cache queue, and then 4 (or 2) images are randomly selected from the updated queue for Mosaic (or MixUp); the speed comparison of reading images using traditional reading methods and cache acceleration mechanisms on the ImageNet large data set is shown in the following table:

[0060] Whether to use cache acceleration mechanism ms / 100 sheets Mosaic no 87.1 Mosaic yes 24.0 MixUp no 19.3 MixUp yes 12.4

[0061] It should be noted that if the cache queue length exceeds the preset length (the preset length of the Mosaic cache queue is 20, and the preset length of the MixUp cache queue is 10), the image that was first added to the cache queue will be popped up. At the same time, when Mosaic and MixUp enhancement processing is required, only a specific number of images need to be randomly selected from the corresponding cache queue for stitching, weighting, mixing, etc.

[0062] Process the images in the above original dataset, refer to Figure 5 As shown in the figure, after manually annotating the real bounding box of the human ear using the labelme open source annotation software, experienced professional physicians annotated the acupoint feature points according to the standard auricular acupoint map (refer to the international GB / T13734-2008 "Names and Locations of Auricular Acupoints") to form a sample data set, which was input into the RTMPose model for learning and training to obtain the RTMPose training model.

[0063] Furthermore, the above sample data set is divided into a training set and a test set in a ratio of 7:3.

[0064] Among them, the learning and training process of the RTMPose training model includes a first iterative training stage using a strong data enhancement algorithm and a second iterative training stage using a weak data enhancement algorithm; specifically, in this embodiment, taking 200 training iterations as an example, the first iterative training stage is specifically the first 180 training iterations, and the second iterative training stage is specifically the last 20 training iterations.

[0065] Furthermore, the first iterative training stage includes a scale transformation process, a random rotation process, and a random cropping process, and the second iterative training stage includes a random rotation process and a random cropping process. Specifically, in the first iterative training stage, a scale transformation process with a large range ([0.6, 1.4]), a random rotation process with a large random rotation factor (80), and a random cropping process with a probability of the Cutout algorithm set to 1 are applied, and in the second iterative training stage, a random rotation process with a small random rotation factor (30) is applied, and a random cropping process with a probability of the Cutout algorithm set to 0.5 is applied.

[0066] The RTMDet training model and the RTMPose training model obtained by the above learning and training are connected in series to build a real-time reasoning model. Specifically, in this embodiment, the RTMDet training model and the RTMPose training model are trained under the PyTorch framework, and the trained RTMDet training model and the RTMPose training model are streamlined and exported as a model file in .pth format. Through the MMDeploy model deployment toolbox and the ONNX-Runtime reasoning backend, the RTMDet and RTMPose models are converted to the ONNX format using the software development kit SDK provided by MMDeploy to adapt them to the ONNX-Runtime reasoning backend, completing the construction and deployment of the overall real-time reasoning model.

[0067] It should be noted that ONNX (Open Neural Network Exchange) is an open standard that provides a standardized way to represent deep learning models, making model conversion more consistent; ONNX makes the interoperability of models between different deep learning frameworks (such as PyTorch, TensorFlow, Caffe2, etc.) and inference engines (such as ONNX-Runtime, TensorRT, OpenVINO, etc.) more convenient, thereby achieving flexible and efficient deployment; ONNX models are cross-platform and can run in a variety of environments (C / C++, etc.), not limited to the Python environment used in this embodiment.

[0068] Furthermore, step S2 also includes image post-processing, and image post-processing includes frame skipping detection processing, posture non-maximum suppression processing, and Euro-filter processing.

[0069] refer to Figure 3 As shown, by adopting the frame skipping detection technology, human ear target detection is performed once every several frames (taking 3 frames as an example), and the ear bounding box result generated by the last RTMDet training model output is used in the interval frame (intermediate frame), that is, for the intermediate frame, the RTMDet training model is no longer used to detect and generate the ear bounding box, but the ear bounding box (region of interest ROI) of the last output frame is directly used, thereby saving computing costs while enhancing the smoothness of real-time reasoning and maintaining the video processing process of the flow.

[0070] Furthermore, by adopting posture non-maximum suppression processing and one-Euro filtering processing, the ear bounding box results and acupoint feature point results generated by erroneous predictions can be filtered out, saving calculation costs and reducing erroneous prediction results.

[0071] S3, maps the ear bounding box and acupoint feature points to the user's ear image, and outputs an auxiliary positioning image with ear acupoint annotations. For details, refer to Figure 6 and Figure 7 As shown, the average processing speed can reach about 20 frames per second.

[0072] In summary, the ear acupoint real-time detection and auxiliary positioning method provided by the present invention can efficiently and accurately detect and identify and locate ear acupoints in real time, reduce the difficulty of ear acupoint identification and positioning, and facilitate auxiliary users to quickly and accurately locate acupoints.

[0073] Optionally, in some embodiments, the present invention also proposes a real-time detection and auxiliary positioning system for ear acupoints, which specifically includes: an image acquisition module, used to detect and acquire the user's ear image; a real-time reasoning module, used to detect and process the user's ear image, generate a corresponding ear bounding box and a number of acupoint feature points located within the ear bounding box; an auxiliary positioning module, used to map the ear bounding box and the acupoint feature points to the user's ear image, and output an auxiliary positioning image with ear acupoint annotations in real time.

[0074] The real-time reasoning module includes an image pre-processing unit, an ear position detection unit, an ear point feature point detection unit, and an image post-processing unit. The image pre-processing unit pre-processes the acquired user ear image, such as adjusting the image size, denoising, etc.; the ear position detection unit uses the RTMDet training model to process the user ear image and generate an ear boundary box; the ear point feature point detection unit uses the RTMPose training model to process the user ear image and generate a number of ear point feature points located within the ear boundary box; the image post-processing unit performs frame skipping detection processing, posture non-maximum suppression processing, and one-euro filtering processing on the user ear image that has completed the ear boundary box and ear point feature point annotation, so as to improve the smoothness of the real-time output result image.

[0075] Optionally, in some embodiments, the present invention further proposes a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the steps in the above-mentioned method for real-time detection and auxiliary positioning of ear acupoints.

[0076] It should be noted that if the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0077] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. For those skilled in the art, the present invention may have various changes and modifications. All equivalent changes and modifications made within the scope of the present invention should still be within the scope of the present invention.

Claims

1. A method for real-time detection and auxiliary positioning of ear acupoints, characterized in that: The following steps are involved: S1, obtaining a user's ear image; S2, constructing and deploying a real-time inference model, and generating a corresponding ear bounding box and a plurality of acupoint feature points located in the ear bounding box according to detection and processing of the user's ear image by the real-time inference model; S3, mapping the ear boundary box and the acupoint feature points to the user's ear image, and outputting an auxiliary positioning image with ear acupoint annotations.

2. The method for real-time detection and auxiliary positioning of auricular points according to claim 1, characterized in that: In step S2, the real-time inference model includes a serially connected RTMDet training model and a RTMPose training model, the RTMDet training model generates the ear bounding box based on the user ear image, and the RTMPose training model generates a plurality of the acupoint feature points located within the ear bounding box based on the user ear image.

3. The method for real-time detection and auxiliary positioning of auricular points according to claim 2, characterized in that: In step S2, building and deploying a real-time inference model includes the following steps: Collect natural optical images of the human ear and construct the original dataset; The images in the original data set are processed, the ear bounding boxes are annotated to form a sample data set, which is then input into the RTMDet model for learning and training to obtain the RTMDet training model; The images in the original data set are processed, and the feature points of acupoints are marked to form a sample data set, which is then input into the RTMPose model for learning and training to obtain the RTMPose training model; The RTMDet training model and the RTMPose training model are connected in series to build a real-time inference model.

4. The method for real-time detection and auxiliary positioning of auricular points according to claim 3, characterized in that: The learning and training process of the RTMDet training model includes a first iterative training phase using a strong data enhancement algorithm and a second iterative training phase using a weak data enhancement algorithm; The first iterative training stage includes Mosaic+MixUp cache acceleration processing, random rotation processing, and random cropping processing, and the second iterative training stage includes random rotation processing and random cropping processing.

5. The method for real-time detection and auxiliary positioning of auricular points according to claim 4, characterized in that: The Mosaic+MixUp cache acceleration process includes the following steps: Construct a Mosaic cache queue, load a randomly selected image from the sample data set to initialize the Mosaic cache queue, load an image from the sample data set, update the Mosaic cache queue, and randomly select an image from it for Mosaic processing; Construct a MixUp cache queue, load a randomly selected image from the sample data set to initialize the MixUp cache queue, load an image from the sample data set, update the MixUp cache queue, and randomly select an image from it for MixUp processing.

6. The method for real-time detection and auxiliary positioning of auricular points according to claim 3, characterized in that: The learning and training process of the RTMPose training model includes a first iterative training phase using a strong data enhancement algorithm and a second iterative training phase using a weak data enhancement algorithm; The first iterative training stage includes a scale transformation process, a random rotation process, and a random cropping process, and the second iterative training stage includes a random rotation process and a random cropping process.

7. The method for real-time detection and auxiliary positioning of auricular points according to claim 1, characterized in that: In step S1, the user ear image is an image frame captured sequentially frame by frame in the video stream.

8. The method for real-time detection and auxiliary positioning of auricular points according to claim 1, characterized in that: Step S2 also includes image post-processing, and the image post-processing includes frame skipping detection processing, posture non-maximum suppression processing, and Euro-filter processing.

9. A real-time detection and auxiliary positioning system for ear acupoints, characterized in that: include: An image acquisition module, used to detect and acquire an ear image of a user; A real-time reasoning module, used for detecting and processing the user's ear image, generating a corresponding ear boundary box and a plurality of acupoint feature points located within the ear boundary box; An auxiliary positioning module, used for mapping the ear boundary box and the acupoint feature points to the user's ear image, and outputting an auxiliary positioning image with ear acupoint annotations in real time; The real-time reasoning module includes an image pre-processing unit, an ear position detection unit, an ear acupoint feature point detection unit, and an image post-processing unit.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the method for real-time detection and auxiliary positioning of ear acupoints as described in any one of claims 1 to 8.

Images