An auricular point detection method, device and computer equipment

Abstract

The application provides an auricular point detection method, device and computer equipment, which comprises the following steps: acquiring ear image data sequences collected under multiple perspectives, each image frame in the ear image data sequences being an RGBD multi-modal image; according to a TSDF algorithm and a multi-perspective surface structure analysis method, a three-dimensional model of an auricle is reconstructed by using input multi-perspective auricle images, the real-time performance is ensured while a high-precision reconstructed model is obtained; the illumination, albedo and photometric normal of a shooting environment are estimated according to the input images, and then the three-dimensional model is inversely rendered by using the information, so that a high-precision multi-perspective fusion auricle reconstruction result is obtained; finally, an auricular point recognition model is used to learn the structural feature information and medical feature information of the auricle images, so as to determine the positions of auricular points in each auricle image, realize high-precision two-dimensional auricular point positioning, and accurately realize three-dimensional auricular point alignment and display in cooperation with the three-dimensional auricle reconstruction result obtained before.

Description

TECHNICAL FIELD

[0001] The present application relates to ear acupoint positioning technology field, and particularly relates to ear acupoint detection method, device and computer equipment. BACKGROUND

[0002] Ear acupoint therapy is an important part of traditional Chinese medicine. Accurate stimulation of different ear acupoints can have a therapeutic effect on the corresponding disease.

[0003] Ear acupoint positioning is an important prerequisite for ear acupoint therapy. However, due to the small area of the auricle, the large individual difference of human ears, and the lack of texture of the ear image, the positioning of ear acupoints is very complex. In the prior art, an ear acupoint model is usually constructed by collecting an RGB image of the ear and identifying the features of the image. However, the two-dimensional RGB image cannot reflect the three-dimensional spatial position information of the ear acupoint, resulting in an incomplete model and insufficient accuracy. SUMMARY

[0004] In view of the above problems, the present application is proposed to provide an ear acupoint detection method, device and computer equipment to solve the above technical problems or at least partially solve the above technical problems.

[0005] In one aspect of the present application, an ear acupoint detection method is provided, which comprises:

[0006] Obtaining an ear image data sequence collected under multiple perspectives, each image frame in the ear image data sequence being an RGBD multi-modal image;

[0007] Fusing the ear image data sequence based on a TSDF truncated zone symbol distance function to obtain a preliminary auricle three-dimensional model;

[0008] Obtaining the illumination, albedo and photometric normal of the corresponding image acquisition environment from each image frame of the ear image data sequence, respectively, and mapping the illumination, albedo and photometric normal of the image frame as the optical feature information of the current image to the preliminary auricle three-dimensional model to obtain an auricle three-dimensional model;

[0009] Extracting the structural feature information and medical feature information of each auricle image in the ear image data sequence, and learning the structural feature information and medical feature information using a pre-constructed ear acupoint recognition model to determine the position of each ear acupoint in each auricle image;

[0010] Positioning and marking each ear acupoint position in the auricle three-dimensional model in each auricle image to realize three-dimensional ear acupoint alignment and display.

[0011] Further, the obtaining of the illumination, albedo and photometric normal of the corresponding image acquisition environment from each image frame of the ear image data sequence comprises:

[0012] estimating initial diffuse reflectance illuminance of the acquisition environment of the current image according to the depth information and the color information of each image frame;

[0013] calculating the lighting degree of the current image according to the color information of each image frame and the corresponding initial diffuse reflectance illuminance;

[0014] constructing a diffuse reflectance illuminance function represented by albedo, photometric normal and lighting degree, and solving the albedo estimation value and the photometric normal estimation value of the current image by an iterative optimization algorithm, so that the difference between the calculated diffuse reflectance illuminance and the initial diffuse reflectance illuminance is minimized.

[0015] Further, the constructing a diffuse reflectance illuminance function represented by albedo, photometric normal and lighting degree, and solving the albedo estimation value and the photometric normal estimation value of the current image by an iterative optimization algorithm, comprises:

[0016] In constructing a diffuse reflectance illuminance function represented by albedo, photometric normal and lighting degree, the photometric normal is converted into a photometric normal function represented by depth information, and the albedo estimation value and the optimized value of the depth information of the current image are solved by an iterative optimization algorithm;

[0017] The photometric normal estimation value is calculated based on the photometric normal function according to the optimized value of the depth information.

[0018] Further, the mapping the lighting degree, albedo and photometric normal of the image frame as the optical characteristic information of the current image to the auricle three-dimensional model, comprises:

[0019] rotating the photometric normal of the current image frame so that the photometric normal direction of the current image is consistent with the photometric normal direction of the previous image frame;

[0020] The lighting degree, albedo and photometric normal length information of the current image frame are weightedly averaged with the corresponding data of the original model according to a preset weight, so as to realize the fusion of the optical characteristic information.

[0021] Further, the extracting the structural characteristic information of each auricle image in the ear image data sequence, comprises:

[0022] extracting the gray scale feature and the depth feature of the image from the color channel and the depth channel of each auricle image;

[0023] learning the color channel attention information and the depth channel attention information of the current input auricle image from the gray scale feature and the depth feature of the auricle image respectively, so as to obtain the attention feature matrix of the color channel as V r , K r , Q r , and obtain the attention feature matrix of the depth channel as V d, K d , Q d ;

[0024] In the process of obtaining the color channel feature map, the Softmax function is used to perform cross-modal attention calculation on the K r matrix of the color channel and the Q d matrix of the depth channel, and the obtained calculation result and the V r matrix of the color channel are subjected to Softmax calculation again to obtain the fused grayscale feature. In the process of obtaining the depth channel feature map, the Softmax function is used to perform cross-modal attention calculation on the K d matrix of the depth channel and the Q r matrix of the color channel, and the obtained calculation result and the V d matrix of the depth channel are subjected to Softmax calculation again to obtain the fused depth feature, so as to realize the aggregation of the depth information and the grayscale information.

[0025] The obtained grayscale feature and depth feature are respectively sent into the corresponding stacked network layer for processing, and the processed grayscale feature and depth feature are superimposed to obtain the structural feature information of the auricle image.

[0026] Further, the method further comprises:

[0027] The step of pre-training the auricular point recognition model specifically comprises:

[0028] According to the structural information of each image sample in the preset auricle image sample set, structure self-supervised training is performed using a Transformer network to realize structure reconstruction of the image sample. According to the identified auricular point positions in each image sample, auricular point position identification is performed on a plurality of structure reconstruction images corresponding to the image sample. The image sample is an RGBD multi-modal image.

[0029] Each image sample in the auricle image sample set and the plurality of structure reconstruction images corresponding to each image sample are taken as a training data set. Structure feature information is extracted from each training sample in the training data set, and first disease index identification is performed on each training sample in the training data set and / or a second disease index uploaded by a user is received to realize medical feature information extraction.

[0030] An initial recognition network model is used to train the structural information and medical feature information of each training sample in the training data set to obtain an auricular point recognition model.

[0031] Further, the structure self-supervised training using the Transformer network according to the structural information of each image sample in the preset auricle image sample set to realize structure reconstruction of the image sample comprises:

[0032] each image sample in the ear auricle image sample set is divided into a plurality of small blocks and sent into the encoding structure of the Transformer network to learn the grayscale features and depth features of the image sample in the color channel and the depth channel respectively;

[0033] based on the grayscale features and depth features of each image sample in the color channel and the depth channel, the position region with a grayscale value of 0 in the mask image input into the decoding structure of the Transformer network is estimated to realize the structure reconstruction of the image sample, and the mask data is a mask image obtained by randomly selecting a position region of the image sample input into the Transformer network and assigning a value of 0 to the grayscale value of the selected position region.

[0034] In another aspect of the present application, an auricular point detection device is provided, which comprises functional modules for implementing the auricular point detection method according to any one of the above aspects;

[0035] The device comprises:

[0036] The acquisition module is configured to acquire an ear image data sequence collected under multiple perspectives, each image frame in the ear image data sequence being an RGBD multi-modal image.

[0037] The model construction module is configured to fuse the ear image data sequence based on a TSDF (Truncated Signed Distance Function) to obtain a preliminary auricle three-dimensional model.

[0038] The feature mapping module is configured to acquire the illumination, albedo and photometric normal of the corresponding image acquisition environment from each image frame of the ear image data sequence, and map the illumination, albedo and photometric normal of the image frame as the optical feature information of the current image to the preliminary auricle three-dimensional model to obtain an auricle three-dimensional model.

[0039] The recognition module is configured to extract the structural feature information and medical feature information of each auricle image in the ear image data sequence, and learn the structural feature information and medical feature information by using a pre-constructed auricular point recognition model to determine each auricular point position in each auricle image.

[0040] The positioning module is configured to position and identify each auricular point position in each auricle image in the auricle three-dimensional model to realize three-dimensional auricular point alignment and display.

[0041] Further, the device further comprises:

[0042] The model training module is configured to pre-train an auricular point recognition model, and specifically configured to: perform structural self-supervised training on each image sample in a preset auricle image sample set by using a Transformer network according to structural information of the image sample, so as to realize structural reconstruction of the image sample; identify each auricular point in the image sample according to the auricular point position points identified in each image sample, wherein the image sample is an RGBD multi-modal image; use each image sample in the auricle image sample set and the multiple structural reconstruction images corresponding to each image sample as a training data set, extract structural feature information of each training sample in the training data set, and perform first disease index identification on each training sample in the training data set and / or receive a second disease index uploaded by a user, so as to realize medical feature information extraction; and train the structural information and the medical feature information of each training sample in the training data set by using a preset initial recognition network model, so as to obtain the auricular point recognition model.

[0043] In another aspect of the present application, a computer device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor; when the computer program is executed by the processor, the steps of the auricular point detection method according to any one of the above aspects are implemented.

[0044] The auricular point detection method, device and computer device provided by the embodiments of the present application can realize three-dimensional model reconstruction of an auricle by using input multi-view ear images according to the TSDF algorithm and the multi-view surface structure analysis method, can ensure real-time performance while obtaining a reconstructed model with high precision, can estimate relevant optical information of a shooting environment, such as illumination, albedo and photometric normal, according to input images, and then perform inverse rendering on the three-dimensional model by using the information to obtain a high-precision multi-view fused auricle reconstruction result, and finally, can learn structural feature information and medical feature information by using an auricular point recognition model to determine each auricular point in each auricle image, realize high-precision two-dimensional auricular point positioning, and accurately realize three-dimensional auricular point alignment and display in cooperation with the three-dimensional auricle reconstruction result obtained before.

[0045] The above description is only a summary of the technical solutions of the present application. In order to enable one skilled in the art to better understand the technical means of the present application, the contents of the specification can be implemented, and in order to enable the above and other purposes, features and advantages of the present application to be more apparent and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0046] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the present application. In the drawings:

[0047] Figure 1 Flow chart of ear acupoint detection method of embodiment of the present application;

[0048] Figure 2 Implementation principle diagram for modifying and refining the model using relevant features of subsequent frame images in the embodiment of the present application;

[0049] Figure 3 Network structure diagram of the Transform structure for realizing RGBD fusion in the embodiment of the present application;

[0050] Figure 4 Structure schematic diagram of ear acupoint detection device of the embodiment of the present application. DETAILED DESCRIPTION

[0051] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood, and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0052] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0053] Embodiment one

[0054] The embodiment of the present application provides an ear acupoint detection method, as shown in the drawings, the ear acupoint detection method provided by the present application comprises the following steps: Figure 1 As shown in the drawings, the ear acupoint detection method provided by the present application comprises the following steps:

[0055] S1, acquiring ear image data sequence collected under multiple perspectives, each image frame in the ear image data sequence is an RGBD multi-modal image; wherein RGB refers to the gray scale information of the red, green and blue three channels of the image; D, namely Depth, refers to the depth information of each pixel point in the image corresponding in the three-dimensional space.

[0056] S2, fusing the ear image data sequence based on the TSDF truncated band signed distance function, and obtaining a preliminary auricle three-dimensional model by fusion;

[0057] S3, obtain the illumination, albedo and photometric normal of the corresponding image acquisition environment from each image frame of the ear image data sequence respectively, map the illumination, albedo and photometric normal of the image frame to the preliminary auricle three-dimensional model as the optical characteristic information of the current image, and obtain the auricle three-dimensional model.

[0058] In the embodiment, the refined depth information is first sent to a TSDF regular space, that is, a cube large enough to accommodate the object to be built, and the cube is composed of a plurality of small cubes, each of which is called a voxel; for each image point in the two-dimensional ear image plane, the coordinates of the point in the three-dimensional space are calculated according to the intrinsic matrix and pose matrix of the camera and the depth information corresponding to the point, and are mapped to the corresponding voxel, so as to realize preliminary reconstruction of the auricle appearance. The albedo, illumination and photometric normal of the scene are also stored in the voxel at the same time, so as to realize geometric integration of data.

[0059] Specifically, based on the diffuse surface assumption theory, the illumination of the environment is preliminarily estimated from the depth information of the image by using 9 basis functions, and then the albedo and photometric normal information of the image are solved based on the iterative optimization algorithm according to the illumination, and the depth information directly obtained during shooting is optimized; then the estimated albedo and photometric normal value are used to inversely optimize the illumination; then the iterative optimization and inverse optimization process is continuously iterated, so as to realize rough to fine estimation of the three data, and ensure the reliability of the optical information of the environment obtained from the image.

[0060] S4, extract the structural feature information and medical feature information of each auricle image in the ear image data sequence, and learn the structural feature information and medical feature information by using a pre-constructed ear point recognition model, to determine the position points of each ear point in each auricle image;

[0061] S5, position and identify each ear point position point in each auricle image in the auricle three-dimensional model, to realize three-dimensional ear point alignment and display.

[0062] The ear point detection method provided by the embodiment of the application realizes three-dimensional model reconstruction of the auricle by using the input multi-view ear image according to the TSDF algorithm and the multi-view surface structure analysis method, can ensure real-time performance while obtaining a reconstructed model with high precision, estimates the related optical information of the shooting environment such as illumination, albedo and photometric normal according to the input image, then inversely renders the three-dimensional model by using the information, obtains a high-precision auricle reconstruction result of multi-view fusion, finally learns the structural feature information and medical feature information by using an ear point recognition model, to determine the position points of each ear point in each auricle image, realizes high-precision two-dimensional ear point positioning, and cooperates with the three-dimensional auricle reconstruction result obtained before, to accurately realize three-dimensional ear point alignment and display.

[0063] In the embodiment of the present application, the step of obtaining the illumination, albedo and photometric normal of the corresponding image acquisition environment from each image frame of the ear image data sequence in step S3 specifically comprises the following steps not shown in the drawings:

[0064] S31, estimating the initial diffuse reflection illumination of the acquisition environment of the current image according to the depth information and color information of each image frame;

[0065] S32, calculating the illumination of the current image according to the color information of each image frame and the corresponding initial diffuse reflection illumination;

[0066] Specifically, the calculation method of the illumination of the current image is as follows:

[0067] E(L t )=E Idata +λ Itemp E Itemp

[0068]

[0069]

[0070] Wherein, N x , N y are the number of pixels in the length direction and the width direction of the image respectively; B(i,j) is the initial diffuse reflection illumination of each pixel point, I(x,y) is the color information of each pixel point of the image, in the RGB color space, the color value I(x,y) is represented by the gray value of the three channels of R, G and B, E(L t ) is a set of l k of a frame of image; E Idata is the shadow term, E Itemp is the time illumination regularization term, λ Itemp is the corresponding weight, which is a preset value.

[0071] The least square method is used to solve the above simultaneous equations to calculate the illumination l k .

[0072] S33, constructing a diffuse reflection illumination function represented by albedo, photometric normal and illumination, and solving the estimated value of the albedo and the estimated value of the photometric normal of the current image by an iterative optimization algorithm, so that the difference between the calculated diffuse reflection illumination and the initial diffuse reflection illumination is minimized.

[0073] Specifically, the diffuse reflection illumination function represented by albedo, photometric normal and illumination is as follows:

[0074]

[0075] Wherein, u=(i,j) represents a certain pixel in the image; B is diffuse reflectance illumination; a is albedo, n is photometric normal, and l is illumination. k is the illumination, H k is a set of spherical harmonic basis functions, K takes 0-8, indicating 9 functions that constitute a set of spherical harmonic basis functions, H k (n) The calculation formula is as follows:

[0076] H0=1.0, H1=n y , H2=n z , H3=n x , H4=n x n y

[0077] H5=n y n z , H6=-n x n x -n y n y +2n z n z

[0078] H7=n z n x , H8=n x n x -n y n y

[0079] Wherein, n x , n y , n z are the x-axis component, y-axis component and z-axis component of the photometric normal respectively.

[0080] Further, the diffuse reflectance illumination function represented by albedo, photometric normal and illumination in step S33 is constructed, and the albedo estimation value and the photometric normal estimation value of the current image are solved by an iterative optimization algorithm, specifically including: when the diffuse reflectance illumination function represented by albedo, photometric normal and illumination is constructed, the photometric normal is converted into a photometric normal function represented by depth information, and the albedo estimation value and the optimized value of the depth information of the current image are solved by an iterative optimization algorithm; the photometric normal estimation value is calculated based on the photometric normal function according to the optimized value of the depth information.

[0081] Specifically, the illumination can be expressed as a function of the color and the diffuse reflectance of the object. The color information can be directly obtained by the gray value of the color channel; the diffuse reflectance can be expressed as a function of the albedo, the photometric normal and the illumination. The initial diffuse reflectance is replaced by the average gray value of the image, and then is calculated by the albedo, the photometric normal and the illumination obtained before. Then the estimation of the illumination can be realized by the least square method. It can be seen that the diffuse reflectance can be directly estimated by the depth information and the color information of the image, and the diffuse reflectance can also be expressed as a function of the albedo, the photometric normal and the illumination. Therefore, the albedo and the photometric normal can be estimated when the illumination and the diffuse reflectance are known. However, it is a non-linear problem to estimate two unknowns by one equation, so it is necessary to solve by using an iterative optimization algorithm, that is, the albedo and the photometric normal are regarded as independent variables, and the values of the two variables are adjusted constantly, so that the difference between the calculated diffuse reflectance and the given diffuse reflectance is minimized, that is, the optimization purpose is to minimize the difference between the two diffuse reflectances. In the actual optimization process, the photometric normal is a vector, so the optimization complexity is high. Therefore, it is rewritten as a function of the depth value, wherein the depth information is a scalar, so that the calculation amount can be greatly reduced. After the optimization is completed, the estimated value of the albedo and the optimized value of the depth can be obtained, and then the photometric normal can be calculated by using the depth optimized value. The photometric normal can be calculated by using the cross product of the depth value and the surrounding points. The relationship between the photometric normal and the depth information constructed in this way is as follows:

[0082]

[0083] n(i,j)=(p(i,j-1)-p(i,j))×(p(i-1,j)-p(i,j))

[0084] where p(i,j) denotes a point in the three-dimensional space mapping to the image point in the imaging plane, (u x , u y ) T represents the principal point position of the camera, D(i,j) represents the depth information of the image point (i,j) T , f x and f y are the focal lengths of the camera in the x and y directions, and n(i,j) (i.e. n(u)) represents the photometric normal corresponding to the pixel point u=(i,j).

[0085] where the iterative optimization process means that the albedo and the photometric normal are estimated by using the updated illumination; the reverse optimization means that the illumination is optimized by using the obtained albedo and photometric normal information, the diffuse reflectance is updated by using these numerical values, and then the value of the illumination is updated.

[0086] In the embodiment of the present application, the step of mapping the illumination, albedo and photometric normal of the image frame to the optical characteristic information of the current image in step S3 comprises: rotating the photometric normal of the current image frame so that the direction of the photometric normal of the current image is consistent with the direction of the photometric normal of the previous image frame; and performing weighted average on the illumination, albedo and photometric normal length information of the current image frame and the corresponding data of the original model according to a preset weight, so as to realize the fusion of the optical characteristic information.

[0087] In the embodiment, as shown in the figure, Figure 2 For the new frame of input data, the depth information is used to integrate the data into the three-dimensional model built before; in order to ensure that the balance of the original model is not broken during the integration process, the photometric normal of the new frame is rotated so that it coincides with the original photometric normal in direction, that is, the transformation matrix is calculated so that the two photometric normals are consistent in direction; then the albedo, illumination and photometric normal length are weighted and averaged with the corresponding data of the original model according to a certain weight, so as to realize the fusion of the texture characteristic information and complete the refinement of the model. The above processing is performed on all input images, and the model is continuously modified and refined, so as to obtain a high-precision multi-view fused auricle reconstruction result.

[0088] Specifically, the purpose of the transformation matrix is to rotate the newly input photometric normal to the direction of the original photometric normal. That is, given two vectors with different directions, how to calculate the rotation matrix so that the two vectors are consistent in direction. First, the included angle between the two vectors can be calculated according to the definition of dot product, and the plane on which the rotation angle lies is the plane formed by the two vectors, so the rotation axis must be perpendicular to the plane. Therefore, the rotation axis of the vector can be calculated according to the definition of cross product, and then the specific rotation matrix can be determined according to the Rodrigues rotation matrix, that is, the transformation matrix mentioned before.

[0089] wherein the mixing weight of each frame is determined by the current frame weight and the predefined upper limit weight, and the smaller one of the two is taken. The predefined upper limit weight is used to constrain the upper limit of the mixing weight value, and is artificially set according to the actual situation. The current frame weight is the sum of the current frame camera pose weight and the previous frame weight. The current frame camera pose weight is calculated from the camera shooting area, camera angle and occlusion area. The initial weight can be designed artificially.

[0090] In the embodiment of the present application, the step of extracting the structural characteristic information of each auricle image in the ear image data sequence in step S4 comprises the following steps not shown in the accompanying drawings:

[0091] S41, extracting the gray scale features and depth features of the image from the color channel and depth channel of each auricle image;

[0092] S42, learn color channel attention information and depth channel attention information of the current input auricle image from the gray feature and the depth feature of the auricle image respectively, obtain the attention feature matrix of the color channel as V r , r , Q r , obtain the attention feature matrix of the depth channel as V d , d , Q d ;

[0093] S43, in the process of obtaining the color channel feature map, the Softmax function is used for cross-modal attention calculation of the K r matrix of the color channel and the Q d matrix of the depth channel, and the obtained calculation result and the V r matrix of the color channel are subjected to Softmax calculation again to obtain the fused gray feature; in the process of obtaining the depth channel feature map, the Softmax function is used for cross-modal attention calculation of the K d matrix of the depth channel and the Q r matrix of the color channel, and the obtained calculation result and the V d matrix of the depth channel are subjected to Softmax calculation again to obtain the fused depth feature, realizing the aggregation of the depth information and the gray information.

[0094] S44, the obtained gray feature and the depth feature are respectively sent into corresponding stacked network layers for processing, and the processed gray feature and the depth feature are superimposed to obtain the structural feature information of the auricle image.

[0095] In this embodiment, first, the gray feature and the depth feature of the image are extracted from the color channel and the depth channel of the image respectively, i.e., F′ r and F′ d ; then the color channel attention information and the depth channel attention information of the input image are learned from the two features respectively, wherein Q, K and V are the learned attention feature matrices; then the Q matrix is used for cross-modal attention calculation, realizing the preliminary aggregation of the depth information and the gray information; Softmax is used for normalizing the data to [0, 1] for subsequent processing; then the obtained gray feature and the depth feature are respectively sent into different stacked network layers for processing, including the elimination of redundant information, the unification of data dimensions, etc.; finally, the obtained gray feature and the depth feature are superimposed, thereby realizing the fusion of the RGBD multi-modal data; the feature superposition is the superposition of the gray feature and the depth feature. Referring to Figure 3 , Figure 3It can be seen from the illustrated RGBD fusion Transformer structure diagram that, in the process of obtaining the color channel feature map, the input Softmax is a K matrix calculated using RGB information and a Q matrix calculated using depth information; in the process of obtaining the depth channel feature map, the input Softmax is a K matrix calculated using depth information and a Q matrix calculated using RGB information, and the cross process realizes cross-modal attention calculation through the Q matrix.

[0096] Further, the method further comprises:

[0097] The step of pre-training the ear point recognition model specifically comprises:

[0098] According to the structural information of each image sample in the preset auricle image sample set, structure self-supervised training is performed using a Transformer network to realize structure reconstruction of the image sample, ear point position points in each image sample are identified, and a plurality of structure reconstruction images corresponding to the image sample are identified as ear point position points, and the image sample is an RGBD multi-modal image;

[0099] Each image sample in the auricle image sample set and the plurality of structure reconstruction images corresponding to each image sample are taken as a training data set, structural feature information of each training sample in the training data set is extracted, first disease index identification of each training sample in the training data set is performed, and / or a second disease index uploaded by a user is received to realize medical feature information extraction.

[0100] An initial identification network model is used to train the structural information and medical feature information of each training sample in the training data set to obtain an ear point recognition model.

[0101] In the embodiment of the application, the structure self-supervised training using the Transformer network according to the structural information of each image sample in the preset auricle image sample set to realize structure reconstruction of the image sample comprises: dividing each image sample in the auricle image sample set into small blocks and inputting the small blocks into the encoding structure of the Transformer network to learn the gray scale features and depth features of the image sample in the color channel and the depth channel respectively; based on the gray scale features and the depth features of each image sample in the color channel and the depth channel, estimating the position region with a gray scale value of 0 in the mask image input into the decoding structure of the Transformer network to realize structure reconstruction of the image sample, and the mask data is a mask image obtained by randomly selecting a position region of the current image sample input into the Transformer network and assigning a value of 0 to the gray scale value of the selected position region.

[0102] In this embodiment, first, the image is divided into several small blocks and sent into the Encoder structure, and the texture features and structural features of the image color channel and depth channel are learned respectively; for the mask data input into the Decoder, the black area is estimated using the learned features, the completion of the all-zero area is completed, and the reconstruction of the image structure is realized, deepening the learning and understanding of the ear contour and structure of the network.

[0103] Wherein, the adaptive mask refers to the random position of the input image is blocked with a black area, that is, the gray value of this part of the region is assigned as 0, and then the network is used to recover this part, realize the ear self-supervised pre-training in the unmarked scene, solve the data shortage problem, and deepen the understanding of the network to the image, that is, deepen the understanding and memory of the network to the auricle structure. Further, the adaptive mask can be directly generated by the network.

[0104] The application uses structure self-supervised training, uses the structural information of the RGBD image to perform self-supervised training on the Transformer network, generates low-dimensional structure points through the Encoder, and combines the adaptive mask to perform image structure reconstruction, realizes the ear self-supervised pre-training in the unmarked scene, and solves the data shortage problem.

[0105] The ear point detection method provided by the application can realize the estimation of the three-dimensional surface structure of the auricle based on the surface structure analysis method of multi-view SFS (shape from shading) and through the rapid solution of image luminosity information.

[0106] The ear point detection method provided by the application solves the problem of difficult measurement of depth information in the auricle wrinkle area based on the auricle reconstruction algorithm of the RGBD multi-view information fusion of the Transformer, and realizes high-precision and complete three-dimensional modeling of the ear.

[0107] The ear point detection method provided by the application is based on the ear point positioning algorithm of the structure self-supervised training, and the sample quantity of the auricle pathological area and the ear point positioning is insufficient, and the model training is prone to overfitting, and the automatic recognition results of papules, dandruff and blood vessels can be integrated, and the medical feature extraction model through the human-in-the-loop is used to realize the accurate positioning of the ear point.

[0108] For the method embodiment, in order to simply describe, all are expressed as a series of action combinations, but those skilled in the art should know that the embodiment of the application is not limited by the described action sequence, because according to the embodiment of the application, certain steps can be adopted in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the embodiment of the application.

[0109] Embodiment two

[0110] Another embodiment of the present application also provides an auricular point detection device, which comprises functional modules for implementing the auricular point detection method according to any one of the above. Figure 4 The structure of the auricular point detection device provided by the embodiment of the present application is schematically shown in the structure diagram, and the auricular point detection device of the embodiment of the present application specifically comprises an acquisition module 401, a model construction module 402, a feature mapping module 403, an identification module 404 and a positioning module 405, wherein: Figure 4

[0111] The acquisition module 401 is configured to acquire an ear image data sequence collected under multiple perspectives, wherein each image frame in the ear image data sequence is an RGBD multi-modal image.

[0112] The model construction module 402 is configured to fuse the ear image data sequence based on a TSDF truncated zone symbol distance function, and obtain a preliminary pinna three-dimensional model through the fusion.

[0113] The feature mapping module 403 is configured to acquire illumination, albedo and photometric normals of a corresponding image acquisition environment from each image frame of the ear image data sequence respectively, map the illumination, albedo and photometric normals of the image frame as optical feature information of a current image to the preliminary pinna three-dimensional model, and obtain a pinna three-dimensional model.

[0114] The identification module 404 is configured to extract structural feature information and medical feature information of each pinna image in the ear image data sequence, learn the structural feature information and the medical feature information by using a pre-constructed auricular point identification model, and determine each auricular point position in each pinna image.

[0115] The positioning module 405 is configured to position and identify each auricular point position in each pinna image in the pinna three-dimensional model, and realize three-dimensional auricular point alignment and display.

[0116] ​In the embodiment of the present application, the device further comprises a model training module not shown in the drawings, the model training module is used for pre-training an auricular point recognition model, and is specifically used for: performing structural self-supervised training on each image sample in a preset auricle image sample set by using a Transform network according to structural information of the image sample, so as to realize structural reconstruction of the image sample; identifying each auricular point position of the image sample according to the auricular point position in each image sample, the image sample is an RGBD multi-modal image; taking each image sample in the auricle image sample set and the multiple structural reconstruction images corresponding to each image sample as a training data set, performing structural feature information extraction on each training sample in the training data set, and performing first disease index identification on each training sample in the training data set and / or receiving a second disease index uploaded by a user, to realize medical feature information extraction; and training the structural information and the medical feature information of each training sample in the training data set by using a preset initial identification network model, to obtain the auricular point recognition model.

[0117] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts refer to the part of the method embodiment, and have corresponding technical effects.

[0118] Embodiment three

[0119] The embodiment of the present application provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in each auricular point detection method embodiment described above when executing the computer program, for example, steps S1-S5 shown in the figure. Figure 1 Alternatively, the processor implements the functions of each module in each auricular point detection device embodiment described above when executing the computer program, for example, the acquisition module 401, the model construction module 402, the feature mapping module 403, the identification module 404 and the positioning module 405 shown in the figure. Figure 4

[0120] For the computer device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts refer to the part of the method embodiment, and have corresponding technical effects.

[0121] In addition, those skilled in the art can understand that although some embodiments herein include certain features instead of other features included in other embodiments, the combination of features of different embodiments means to be within the scope of the present application and form different embodiments. For example, any one of the claimed embodiments can be used in any combination.

[0122] ​It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An auricular point detection method, characterized by, The method comprises: acquiring ear image data sequences collected under multiple perspectives, each image frame in the ear image data sequences being an RGBD multi-modal image; fusing the ear image data sequences based on a TSDF truncated signed distance function to obtain a preliminary auricle three-dimensional model; obtaining the illumination, albedo and photometric normal of the image acquisition environment from each image frame of the ear image data sequences, and mapping the illumination, albedo and photometric normal of the image frame as optical characteristic information of the current image to the preliminary auricle three-dimensional model to obtain an auricle three-dimensional model; extracting structural characteristic information and medical characteristic information of each auricle image in the ear image data sequences, and learning the structural characteristic information and medical characteristic information by using a pre-constructed ear point recognition model to determine each ear point position in each auricle image; positioning and marking each ear point position in each auricle image in the auricle three-dimensional model to realize three-dimensional ear point alignment and display.

2. The method of claim 1, wherein, The method further comprises: estimating the initial diffuse reflectance illumination of the image acquisition environment according to the depth information and color information of each image frame; calculating the illumination of the current image according to the color information of each image frame and the corresponding initial diffuse reflectance illumination; constructing a diffuse reflectance illumination function represented by albedo, photometric normal and illumination, and solving the albedo estimation value and photometric normal estimation value of the current image by an iterative optimization algorithm to minimize the difference between the calculated diffuse reflectance illumination and the initial diffuse reflectance illumination.

3. The method of claim 2, wherein, The method further comprises: when constructing the diffuse reflectance illumination function represented by albedo, photometric normal and illumination, converting the photometric normal into a photometric normal function represented by depth information, and solving the albedo estimation value and optimized value of the depth information of the current image by an iterative optimization algorithm; calculating the photometric normal estimation value based on the photometric normal function according to the optimized value of the depth information.

4. The method of claim 1, wherein, The method further comprises: rotating the photometric normal of the current image frame to make the photometric normal direction of the current image consistent with the photometric normal direction of the previous image frame; performing weighted average on the illumination, albedo and photometric normal length information of the current image frame and the corresponding data of the original model according to a preset weight to realize fusion of the optical characteristic information.

5. The method of claim 1, wherein, The method further comprises: extracting the gray scale features and depth features of the image from the color channel and depth channel of each auricle image; The color channel attention information and the depth channel attention information of the current input auricle image are learned from the gray scale features and the depth features of the auricle image respectively, and the attention feature matrix of the color channel is V r , K r , Q r , and the attention feature matrix of the depth channel is V d , K d , Q d ; In the process of obtaining the color channel feature map, the Softmax function is used for cross-modal attention calculation of K r matrix of the color channel and Q d matrix of the depth channel, and the obtained calculation result and V r matrix of the color channel are subjected to Softmax calculation again to obtain the fused grayscale feature. In the process of obtaining the depth channel feature map, the Softmax function is used for cross-modal attention calculation of K d matrix of the depth channel and Q r matrix of the color channel, and the obtained calculation result and V d matrix of the depth channel are subjected to Softmax calculation again to obtain the fused depth feature, so as to realize the aggregation of the depth information and the grayscale information. sending the obtained gray scale features and depth features into corresponding stacked network layers for processing, superimposing the processed gray scale features and depth features to obtain the structural characteristic information of the auricle image.

6. The method of claim 1, wherein, The method further comprises: a step of pre-training the ear point recognition model, specifically comprising: The structure information of each image sample in the preset auricle image sample set is used for structure self-supervised training by using a Transformer network, so as to realize structure reconstruction of the image sample, and each auricular point position in each image sample is marked, and the multiple structure reconstruction images corresponding to the image sample are marked with auricular point positions, and each image sample is an RGBD multi-modal image; Each image sample in the auricle image sample set and the multiple structure reconstruction images corresponding to each image sample are taken as a training data set, structure feature information of each training sample in the training data set is extracted, a first disease index of each training sample in the training data set is identified, and / or a second disease index uploaded by a user is received, and medical feature information is extracted; An initial identification network model is used to train the structure information and the medical feature information of each training sample in the training data set, and an auricular point identification model is obtained.

7. The method of claim 6, wherein, The structure information of each image sample in the preset auricle image sample set is used for structure self-supervised training by using a Transformer network, so as to realize structure reconstruction of the image sample, and each auricular point position in each image sample is marked, and the multiple structure reconstruction images corresponding to the image sample are marked with auricular point positions, and each image sample is an RGBD multi-modal image; Each image sample in the auricle image sample set is divided into small blocks, and is input into an encoding structure of the Transformer network, and gray scale features and depth features of the image sample in a color channel and a depth channel are learned respectively; Based on the gray scale features and the depth features of each image sample in the color channel and the depth channel, a position region with a gray scale value of 0 in a mask image input into a decoding structure of the Transformer network is estimated, and structure reconstruction of the image sample is realized, and the mask data is a mask image obtained by randomly selecting a position region of a current image sample input into the Transformer network and assigning a value of 0 to a gray scale value of the selected position region.

8. An auricular point detection device, characterized by, The device comprises: An acquisition module is configured to acquire ear image data sequences collected under multiple perspectives, each image frame in the ear image data sequences being an RGBD multi-modal image; A model construction module is configured to fuse the ear image data sequences based on a TSDF (Truncated Signed Distance Function), and obtain a preliminary auricle three-dimensional model through fusion; A feature mapping module is configured to acquire illumination, albedo and photometric normals of a corresponding image acquisition environment from each image frame of the ear image data sequences, map the illumination, the albedo and the photometric normals of the image frame as optical feature information of a current image to the preliminary auricle three-dimensional model, and obtain an auricle three-dimensional model; An identification module is configured to extract structure feature information and medical feature information of each auricle image in the ear image data sequences, and learn the structure feature information and the medical feature information by using a pre-constructed auricular point identification model, so as to determine each auricular point position in each auricle image; A positioning module is configured to position and mark each auricular point position in each auricle image in the auricle three-dimensional model, and realize three-dimensional auricular point alignment and display.

9. The apparatus of claim 8, wherein, The device further comprises: The model training module is configured to pre-train an auricular point recognition model, and specifically comprises: performing structural self-supervised training on each image sample in a preset auricle image sample set by using a Transformer network according to structural information of the image sample, so as to realize structural reconstruction of the image sample; identifying each auricular point position in each image sample, and identifying auricular point positions in a plurality of structural reconstruction images corresponding to the image sample, wherein each image sample is an RGBD multi-modal image; taking each image sample in the auricle image sample set and the plurality of structural reconstruction images corresponding to each image sample as a training data set, extracting structural feature information of each training sample in the training data set, and performing first disease index identification on each training sample in the training data set and / or receiving a second disease index uploaded by a user, so as to realize medical feature information extraction; and training structural information and medical feature information of each training sample in the training data set by using a preset initial recognition network model, so as to obtain the auricular point recognition model.

10. A computer device, comprising: A computer program product comprising a memory and a processor, and a computer program stored on the memory and loadable on the processor; The computer program is executed by the processor to implement the steps of the method according to any one of claims 1-7.

Images