Face acne skin lesion detection matching method, system and equipment

Abstract

The invention provides a face acne skin lesion detection matching method, system and device, and relates to the field of image processing, and the method comprises the steps: calling a face key point detection interface, and obtaining face key point coordinates on face images at different angles; obtaining a first mapping transformation reference point of each face key point corresponding to the side face and the front face; according to the first mapping transformation reference point, generating a perspective transformation matrix of mapping the side face to the front face; according to the perspective transformation matrix and the human face key point coordinates, mapping the skin lesion detection frame on the side face to a position corresponding to the front face; matching the skin lesion detection frame on the front face with the skin lesion detection frame mapped to the front face by the side face; and unifying the skin lesion type labels of the skin lesion with the front face and the side face successfully matched. According to the invention, the accuracy of acne evaluation results can be improved.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method, system, and device for detecting and matching acne lesions on the face. Background Technology

[0002] When conducting online facial acne assessments, it is recommended that users take facial images from three angles—front, left, and right—to gain a more complete understanding of the patient's facial condition. However, since the image features of the same lesion may differ in images from different angles, the detection results for the same lesion at different angles may be inconsistent when using detection algorithms. Furthermore, existing acne assessment methods mainly combine the type and number of acne lesions. Therefore, if the same lesion is detected as having different lesion type labels from different angles, it will affect the determination of acne assessment results and treatment plans. Summary of the Invention

[0003] The purpose of this invention is to provide a method, system, and device for detecting and matching acne lesions on the face, so as to solve the problem of inconsistent detection results of the same lesion from different angles, which leads to low accuracy of acne assessment results.

[0004] To achieve the above objectives, the present invention provides the following solution:

[0005] A method for detecting and matching facial acne lesions, comprising:

[0006] The facial landmark detection interface is called to obtain the coordinates of facial landmarks on facial images from different angles; the facial images from different angles include frontal and side views; the side views include the left and right sides; the coordinates of facial landmarks generated in the same area have the same sequence number;

[0007] Obtain the first mapping transformation reference point for each facial key point corresponding to the side face and the front face; the first mapping transformation reference point is the nearest neighbor key point directly connected to the facial key point in the topology;

[0008] Based on the first mapping transformation reference point, a perspective transformation matrix is ​​generated that maps the side profile to the front profile;

[0009] Based on the perspective transformation matrix and the coordinates of the facial key points, the skin lesion detection box on the side face is mapped to the corresponding position on the front face;

[0010] Match the skin lesion detection box on the front face with the skin lesion detection box mapped from the side face to the front face;

[0011] Unify the skin lesion type label for the skin lesions that are successfully matched between the frontal face and the side face.

[0012] Optionally, based on the first mapping transformation reference point, a perspective transformation matrix mapping the side profile to the frontal face is generated, specifically including:

[0013] Using the least squares method, the optimal perspective transformation matrix between the first mapping transformation reference points corresponding to the coordinates of the same number of facial key points at different angles is calculated; the optimal perspective transformation matrix is ​​the perspective transformation matrix that maps the side face to the front face.

[0014] Optionally, based on the perspective transformation matrix and the coordinates of the facial key points, the skin lesion detection box on the side face is mapped to the corresponding position on the front face, specifically including:

[0015] Face images from different angles are input into a target detection model, which outputs acne lesion detection results. The acne lesion detection results include the coordinate information of the lesion detection box, the lesion type label, and the confidence level.

[0016] The center point of each skin lesion detection box on the side face is used as the mapping point;

[0017] Based on the coordinates of the facial key points, the distance between each facial key point and the mapping point is calculated, and the facial key point with the shortest distance is selected as the second mapping transformation reference point.

[0018] Based on the perspective transformation matrix corresponding to the second mapping transformation reference point, calculate the coordinates of the facial key points mapped from the center point of the skin lesion detection box to the frontal face.

[0019] Optionally, matching the skin lesion detection box on the frontal face with the skin lesion detection box mapped from the side face to the frontal face specifically includes:

[0020] Calculate the detection frame distance between the skin lesion detection frame on the front face and the skin lesion detection frame mapped from the side face to the front face;

[0021] Determine whether the distance of the detection frame is less than the set reference distance;

[0022] If so, obtain the skin lesions on the front face and the skin lesions on the side face corresponding to the detection box distance that is less than the set reference distance;

[0023] The Hungarian algorithm was used to match the skin lesions on the frontal face and the skin lesions on the lateral face.

[0024] If not, it is determined that the lesion detection box on the front face and the lesion corresponding to the lesion detection box mapped from the side face to the front face are not the same lesion.

[0025] Optionally, the skin lesion type label for the successfully matched frontal and side facial lesions is unified, specifically including:

[0026] Determine whether the skin lesion type label on the front face is the same as the skin lesion type label that was successfully matched on the side face;

[0027] If so, keep the skin lesion type label unchanged;

[0028] If not, compare the confidence levels of the acne lesion detection results for the frontal face and the acne lesion detection results for the side face, and adjust the lesion type label corresponding to the acne lesion detection results with low confidence to the lesion type label corresponding to the acne lesion detection results with high confidence.

[0029] A facial acne lesion detection and matching system includes:

[0030] The facial landmark coordinate acquisition module is used to call the facial landmark detection interface to obtain the coordinates of facial landmarks on facial images from different angles; the facial images from different angles include frontal and side views; the side views include the left and right sides; the sequence numbers of the facial landmark coordinates generated in the same location are the same;

[0031] The first mapping transformation reference point acquisition module is used to acquire the first mapping transformation reference point of each facial key point corresponding to the side face and the front face; the first mapping transformation reference point is the nearest key point that is directly topologically connected to the facial key point.

[0032] The perspective transformation matrix generation module is used to generate a perspective transformation matrix that maps the side face to the front face based on the first mapping transformation reference point.

[0033] The mapping module is used to map the skin lesion detection box on the side face to the corresponding position on the front face according to the perspective transformation matrix and the coordinates of the facial key points;

[0034] The matching module is used to match the skin lesion detection box on the front face with the skin lesion detection box mapped from the side face to the front face;

[0035] The lesion type labeling module is used to unify the lesion type labels of lesions that are successfully matched between the frontal and side views.

[0036] Optional, the perspective transformation matrix generation module specifically includes:

[0037] The perspective transformation matrix generation unit is used to calculate the optimal perspective transformation matrix between the first mapping transformation reference points corresponding to the coordinates of the same index facial key points at different angles using the least squares method; the optimal perspective transformation matrix is ​​the perspective transformation matrix that maps the side face to the front face.

[0038] Optional, mapping modules, specifically including:

[0039] The acne lesion detection result output unit is used to input face images from different angles into the target detection model and output acne lesion detection results; the acne lesion detection results include the coordinate information of the lesion detection box, the lesion type label, and the confidence level;

[0040] A mapping point determination unit is used to take the center point of each skin lesion detection box on the side face as a mapping point;

[0041] The second mapping transformation reference point determination unit is used to calculate the distance between each facial key point and the mapping point based on the coordinates of the facial key points, and select the facial key point with the shortest distance as the second mapping transformation reference point.

[0042] The mapping unit is used to calculate the coordinates of the facial key points after the center point of the skin lesion detection box is mapped to the frontal face based on the perspective transformation matrix corresponding to the second mapping transformation reference point.

[0043] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor running the computer program to enable the electronic device to perform the above-described facial acne lesion detection and matching method.

[0044] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for detecting and matching facial acne lesions.

[0045] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: The present invention obtains the coordinates of facial key points on facial images from different angles and the first mapping transformation reference point of each facial key point by calling the facial key point detection interface, and generates a perspective transformation matrix mapping the side face to the front face, thereby ensuring that the skin lesions on the side face can be accurately mapped to the front face; and matches the skin lesion detection boxes on the front face and the skin lesion detection boxes mapped from the side face to the front face with the corresponding skin lesion types, and unifies the skin lesion type labels of the skin lesions that are successfully matched between the front face and the side face, thereby making the skin lesions on facial images from different angles consistent and improving the accuracy of acne assessment results. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 The flowchart illustrates the facial acne lesion detection and matching method provided by this invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] The purpose of this invention is to provide a method, system, and device for detecting and matching facial acne lesions, which can improve the accuracy of acne assessment results.

[0050] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] Example 1

[0052] like Figure 1 As shown, the present invention provides a method for detecting and matching facial acne lesions, comprising:

[0053] Step 101: Call the facial landmark detection interface to obtain the coordinates of facial landmarks on facial images from different angles; the facial images from different angles include frontal and side views; the side views include left and right faces; the coordinates of facial landmarks generated in the same location have the same sequence number.

[0054] In practical applications, the Mediapipe facial landmark detection interface is called to obtain the coordinates of facial landmarks on face images from different angles. Mediapipe is an open-source machine vision algorithm tool library proposed by Google. The Mediapipe facial landmark detection interface can calculate and generate 478 key points of the face in the image, ensuring the accuracy of calculating the perspective transformation matrix and lesion localization and matching.

[0055] Step 102: Obtain the first mapping transformation reference point for each facial key point corresponding to the side profile and the front profile; the first mapping transformation reference point is the nearest neighbor key point directly topologically connected to the facial key point. Specifically, the first mapping transformation reference point is the nearest neighbor key point directly topologically connected to a certain key point, i.e., a set of key points.

[0056] In practical applications, the input is a face image from three angles: frontal, left, and right. For each image, the Mediapipe face key point detection interface is called to obtain the corresponding sequence number of the facial coordinate network connection. This allows us to obtain the nearest key point that is directly connected to all key points in the topology, and use it as the first mapping transformation reference point.

[0057] Mediapipe generates the same keypoint coordinates for the same part of a face from different angles. Facial keypoint mapping involves calculating the coordinates of keypoints on the profile face mapped to the front face. Specifically, for each keypoint on both the profile and front face, the first mapping transformation reference point is obtained, and the optimal perspective transformation matrix between these two sets of reference points is calculated using the least squares method.

[0058] Step 103: Generate a perspective transformation matrix that maps the side profile to the front profile based on the first mapping transformation reference point.

[0059] In practical applications, step 103 specifically includes: using the least squares method to calculate the optimal perspective transformation matrix between the first mapping transformation reference points corresponding to the coordinates of facial key points with the same index at different angles; the optimal perspective transformation matrix is ​​the perspective transformation matrix mapping the side face to the front face. The perspective transformation matrix ensures that skin lesions on the side face can be accurately mapped to the front face.

[0060] Step 104: Based on the perspective transformation matrix and the coordinates of the facial key points, map the skin lesion detection box on the side face to the corresponding position on the front face.

[0061] In practical applications, step 104 specifically includes: inputting facial images from different angles into a target detection model and outputting acne lesion detection results; the acne lesion detection results include the coordinate information of the lesion detection box, the lesion type label, and the confidence level; using the center point of each lesion detection box on the side face as a mapping point; calculating the distance between each facial key point and the mapping point based on the facial key point coordinates, and selecting the facial key point with the shortest distance as the second mapping transformation reference point; calculating the coordinates of the facial key points after mapping the center point of the lesion detection box to the frontal face according to the perspective transformation matrix corresponding to the second mapping transformation reference point. The second mapping transformation reference point is the key point closest to the center of the detection box, i.e., a single point.

[0062] As an optional embodiment of the present invention, three face images from different angles are input into the target detection model. The acne lesion detection result output by the target detection model includes the coordinate information of the lesion detection box, the lesion type label, and the confidence score. The confidence score is a decimal between 0 and 1, and the higher the confidence score, the more reliable the detection result.

[0063] For the acne lesion detection results on the face image from each angle, the center point of each lesion detection box is used as the mapping point. By calculating the distance L between it and other key points, the perspective transformation matrix corresponding to the nearest key point is obtained. Based on the perspective transformation matrix, the coordinates of the center point of the detection box after being mapped to the front face are calculated.

[0064] The formula for calculating distance L is as follows, where (x1,y1) and (x2,y2) are the coordinates of the two facial key points for which the distance needs to be calculated, and c is the distance L.

[0065]

[0066] Step 105: Match the skin lesion detection box on the front face with the skin lesion detection box mapped from the side face to the front face.

[0067] In practical applications, step 105 specifically includes: calculating the detection box distance between the skin lesion detection box on the front face and the skin lesion detection box mapped from the side face to the front face; determining whether the detection box distance is less than a set reference distance; if so, obtaining the skin lesions on the front face and the side face corresponding to the detection box distance less than the set reference distance; matching the skin lesions on the front face and the side face using the Hungarian algorithm; if not, determining that the skin lesion detection box on the front face and the skin lesion corresponding to the skin lesion detection box mapped from the side face to the front face are not the same skin lesion.

[0068] As an optional implementation of the present invention, a threshold is set for the detection results of each lesion type label. Detection boxes with a confidence level greater than the threshold are filtered and retained for matching operations. That is, a threshold is set for each type of lesion included in the acne detection results. For example, the target detection model can detect three types of lesions: comedones, papules, and pustules. Thresholds are set for these three types of lesions respectively. From the coordinate information of the lesion detection boxes, the lesion type label, and the confidence level included in the acne lesion detection results, it can be seen that if the lesion type label of the lesion detection box is comedones, the confidence level of the lesion detection box is compared with the threshold for comedones. Those that are greater than the threshold are retained.

[0069] For both the original detection results from the frontal face and the detection results mapped from the side face, a Hungarian algorithm is used to match the original frontal face detection results and the side face-mapped frontal face detection results where the distance between the center points of the frontal and side face lesion detection boxes is less than a set baseline distance. This is because if the distance between the center points of the frontal and side face lesion detection boxes is greater than the baseline distance, they are considered not to be the same lesion from different angles on the face. The baseline distance is the average distance between all facial keypoints and their nearest keypoints. This baseline distance filters out lesion pairs that are very likely not the same lesion from different angles, avoiding duplicate calculations and reducing computational load.

[0070] A series of lesion detection boxes on the side face are mapped to the frontal face and denoted as a, b, c, d, e..., while the lesion detection boxes on the frontal face are denoted as A, B, C, D, E... Because some lesions may be densely distributed on the face, meaning that a may be less than the set reference distance to both A and B, c may be less than the set reference distance to both B and D, and d may be less than the reference distance to D, the Hungarian algorithm is chosen to achieve optimal pairwise matching, as the center points of any two lesion detection boxes with a distance less than the reference distance satisfy the matching condition and do not match repeatedly. The Hungarian algorithm can achieve optimal matching and has low computational complexity.

[0071] Step 106: Unify the skin lesion type labels of the skin lesions that are successfully matched between the frontal face and the side face.

[0072] In practical applications, step 106 specifically includes: determining whether the skin lesion type labels of the skin lesions on the front face and the skin lesions that were successfully matched on the side face are the same; if so, keeping the skin lesion type label unchanged; if not, comparing the confidence level of the acne skin lesion detection results on the front face and the confidence level of the acne skin lesion detection results on the side face, and adjusting the skin lesion type label corresponding to the acne skin lesion detection results with low confidence to the skin lesion type label corresponding to the acne skin lesion detection results with high confidence.

[0073] As an optional embodiment of the present invention, for two successfully matched lesions, after mapping a lesion A on the side face onto the frontal face image and denoting it as A', and if a lesion B on the frontal face is successfully matched with A', then A on the side face and B on the frontal face are considered to be the same lesion. The successfully matched A and B will undergo further processing: 1) If the labels of A and B are consistent, they remain unchanged; 2) If the labels of A and B are inconsistent, the label of the detection result with lower confidence is adjusted to that of the detection result with higher confidence.

[0074] After the above matching operation, skin lesion detection results are obtained from three angles, ensuring that the same skin lesion maintains consistent labeling across different angles. The multi-angle skin lesion matching algorithm proposed in this invention can ensure consistent detection results for the same skin lesion from different angles, thereby improving the scientific rigor and reliability of facial acne assessment.

[0075] Example 2

[0076] In order to implement the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, a facial acne lesion detection and matching system is provided below.

[0077] A facial acne lesion detection and matching system includes:

[0078] The facial landmark coordinate acquisition module is used to call the facial landmark detection interface to obtain the coordinates of facial landmarks on facial images from different angles; the facial images from different angles include frontal and side views; the side views include left and right faces; the sequence numbers of the facial landmark coordinates generated at the same location are the same.

[0079] The first mapping transformation reference point acquisition module is used to acquire the first mapping transformation reference point of each facial key point corresponding to the side face and the front face; the first mapping transformation reference point is the nearest key point that is directly topologically connected to the facial key point.

[0080] The perspective transformation matrix generation module is used to generate a perspective transformation matrix that maps the side profile to the front profile based on the first mapping transformation reference point.

[0081] The mapping module is used to map the skin lesion detection box on the side face to the corresponding position on the front face based on the perspective transformation matrix and the coordinates of the facial key points.

[0082] The matching module is used to match the skin lesion detection box on the front face with the skin lesion detection box mapped from the side face to the front face.

[0083] The lesion type labeling module is used to unify the lesion type labels of lesions that are successfully matched between the frontal and side views.

[0084] In practical applications, the perspective transformation matrix generation module specifically includes: a perspective transformation matrix generation unit, used to calculate the optimal perspective transformation matrix between the first mapping transformation reference points corresponding to the coordinates of the same index facial key points at different angles using the least squares method; the optimal perspective transformation matrix is ​​the perspective transformation matrix mapping the side face to the front face.

[0085] In practical applications, the mapping module specifically includes: an acne lesion detection result output unit, used to input facial images from different angles into the target detection model and output acne lesion detection results; the acne lesion detection results include the coordinate information of the lesion detection box, the lesion type label, and the confidence level; a mapping point determination unit, used to take the center point of each lesion detection box on the side face as the mapping point; a second mapping transformation reference point determination unit, used to calculate the distance between each facial key point and the mapping point based on the coordinates of the facial key points, and select the facial key point with the shortest distance as the second mapping transformation reference point; and a mapping unit, used to calculate the coordinates of the facial key points after the center point of the lesion detection box is mapped to the frontal face according to the perspective transformation matrix corresponding to the second mapping transformation reference point.

[0086] This invention calculates the perspective transformation matrix from the side face to the front face based on facial key points, then maps the skin lesions on the side face to the front face through the perspective transformation matrix, and finally uses a matching algorithm to ensure that the detection results of the same skin lesion from different angles are consistent.

[0087] Example 3

[0088] This invention provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the facial acne lesion detection and matching method provided in Embodiment 1.

[0089] In practical applications, the aforementioned electronic devices can be servers.

[0090] In practical applications, electronic devices include: at least one processor, memory, bus, and communication interface.

[0091] The processor, communication interface, and memory communicate with each other via a communication bus.

[0092] A communication interface is used to communicate with other devices.

[0093] The processor is used to execute programs, specifically the methods described in the above embodiments.

[0094] Specifically, the program may include program code, which includes computer operation instructions.

[0095] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The electronic device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0096] Memory is used to store programs. Memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive.

[0097] Based on the description of the above embodiments, this application provides a storage medium storing computer program instructions thereon, which can be executed by a processor to implement the methods described in any embodiment.

[0098] The facial acne lesion detection and matching system provided in this application exists in various forms, including but not limited to:

[0099] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0100] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access capabilities. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0101] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

[0102] (4) Other electronic devices with data interaction functions.

[0103] Specific embodiments of the subject matter have now been described. Other embodiments are within the scope of the appended claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous.

[0104] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0105] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0109] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0110] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0111] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0112] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0113] This application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific transactions or implement specific abstract data types. This application can also be practiced in distributed computing environments where transactions are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0114] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0115] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting and matching acne lesions on the face, characterized in that, include: The facial landmark detection interface is called to obtain the coordinates of facial landmarks on facial images from different angles; the facial images from different angles include frontal and side views; the side views include the left and right sides; the coordinates of facial landmarks generated in the same area have the same sequence number; Obtain the first mapping transformation reference point for each facial key point corresponding to the side face and the front face; the first mapping transformation reference point is the nearest neighbor key point directly connected to the facial key point in the topology; Based on the first mapping transformation reference point, a perspective transformation matrix is ​​generated that maps the side profile to the front profile; Based on the perspective transformation matrix and the coordinates of the facial key points, the skin lesion detection box on the side face is mapped to the corresponding position on the front face; Match the skin lesion detection box on the front face with the skin lesion detection box mapped from the side face to the front face; Unify the skin lesion type label for the skin lesions that are successfully matched between the frontal face and the side face.

2. The facial acne lesion detection and matching method according to claim 1, characterized in that, Based on the first mapping transformation reference point, a perspective transformation matrix is ​​generated that maps the side profile to the frontal face, specifically including: Using the least squares method, the optimal perspective transformation matrix between the first mapping transformation reference points corresponding to the coordinates of the same number of facial key points at different angles is calculated; the optimal perspective transformation matrix is ​​the perspective transformation matrix that maps the side face to the front face.

3. The facial acne lesion detection and matching method according to claim 1, characterized in that, Based on the perspective transformation matrix and the coordinates of the facial key points, the skin lesion detection box on the side face is mapped to the corresponding position on the front face, specifically including: Face images from different angles are input into a target detection model, which outputs acne lesion detection results. The acne lesion detection results include the coordinate information of the lesion detection box, the lesion type label, and the confidence level. The center point of each skin lesion detection box on the side face is used as the mapping point; Based on the coordinates of the facial key points, the distance between each facial key point and the mapping point is calculated, and the facial key point with the shortest distance is selected as the second mapping transformation reference point. Based on the perspective transformation matrix corresponding to the second mapping transformation reference point, calculate the coordinates of the facial key points mapped from the center point of the skin lesion detection box to the frontal face.

4. The facial acne lesion detection and matching method according to claim 1, characterized in that, Matching the skin lesion detection bounding box on the front face with the skin lesion detection bounding box mapped from the side face to the front face specifically includes: Calculate the detection frame distance between the skin lesion detection frame on the front face and the skin lesion detection frame mapped from the side face to the front face; Determine whether the distance of the detection frame is less than the set reference distance; If so, obtain the skin lesions on the front face and the skin lesions on the side face corresponding to the detection box distance that is less than the set reference distance; The Hungarian algorithm was used to match the skin lesions on the frontal face and the skin lesions on the lateral face. If not, it is determined that the lesion detection box on the front face and the lesion corresponding to the lesion detection box mapped from the side face to the front face are not the same lesion.

5. The facial acne lesion detection and matching method according to claim 1, characterized in that, Unify the skin lesion type label for the skin lesions that successfully match the frontal and side views, specifically including: Determine whether the skin lesion type label on the front face is the same as the skin lesion type label that was successfully matched on the side face; If so, keep the skin lesion type label unchanged; If not, compare the confidence levels of the acne lesion detection results for the frontal face and the acne lesion detection results for the side face, and adjust the lesion type label corresponding to the acne lesion detection results with low confidence to the lesion type label corresponding to the acne lesion detection results with high confidence.

6. A facial acne lesion detection and matching system, characterized in that, include: The facial landmark coordinate acquisition module is used to call the facial landmark detection interface to obtain the coordinates of facial landmarks on facial images from different angles; the facial images from different angles include frontal and side views; the side views include the left and right sides; the sequence numbers of the facial landmark coordinates generated in the same location are the same; The first mapping transformation reference point acquisition module is used to acquire the first mapping transformation reference point of each facial key point corresponding to the side face and the front face; the first mapping transformation reference point is the nearest key point that is directly topologically connected to the facial key point. The perspective transformation matrix generation module is used to generate a perspective transformation matrix that maps the side face to the front face based on the first mapping transformation reference point. The mapping module is used to map the skin lesion detection box on the side face to the corresponding position on the front face according to the perspective transformation matrix and the coordinates of the facial key points; The matching module is used to match the skin lesion detection box on the front face with the skin lesion detection box mapped from the side face to the front face; The lesion type labeling module is used to unify the lesion type labels of lesions that are successfully matched between the frontal and side views.

7. The facial acne lesion detection and matching system according to claim 6, characterized in that, The perspective transformation matrix generation module specifically includes: The perspective transformation matrix generation unit is used to calculate the optimal perspective transformation matrix between the first mapping transformation reference points corresponding to the coordinates of the same index facial key points at different angles using the least squares method; the optimal perspective transformation matrix is ​​the perspective transformation matrix that maps the side face to the front face.

8. The facial acne lesion detection and matching system according to claim 6, characterized in that, The mapping module specifically includes: The acne lesion detection result output unit is used to input face images from different angles into the target detection model and output acne lesion detection results; the acne lesion detection results include the coordinate information of the lesion detection box, the lesion type label, and the confidence level; A mapping point determination unit is used to take the center point of each skin lesion detection box on the side face as a mapping point; The second mapping transformation reference point determination unit is used to calculate the distance between each facial key point and the mapping point based on the coordinates of the facial key points, and select the facial key point with the shortest distance as the second mapping transformation reference point. The mapping unit is used to calculate the coordinates of the facial key points after the center point of the skin lesion detection box is mapped to the frontal face based on the perspective transformation matrix corresponding to the second mapping transformation reference point.

9. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the facial acne lesion detection and matching method as described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the facial acne lesion detection and matching method as described in any one of claims 1-5.

Images