Ear tip optimization service providing system using shape of ear

Abstract

To provide an ear tip optimization service providing system using the shape of an ear.SOLUTION: An ear tip optimization service providing system includes: a receiving unit configured to receive a photograph of the shape of the ears from the user device; a detecting unit configured to detect, identify, and identify an object corresponding to the ears in the photograph; a restoring unit configured to restore the detected object to 3D information; an extracting unit configured to extract ear tips corresponding to the sizes and shapes of the insides of the ears restored in the 3D database; and a recommending unit configured to recommend the extracted ear tips to the user terminals.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a system for providing an ear tip optimization service using ear shape, and provides a system that can provide ear tips that fit a user's ears by comparing ear shape data and ear tip data based on a photograph of the ear shape. [Background technology]

[0002] Earphones are electro-acoustic transducers that are used to acoustically couple closely to the ear and obtain acoustic signals from electrical signals. There are headphone, hook, kernel, and open types, and earphones are also categorized into touch and button types depending on the driving method of the earphone device. Headphone types are worn on the head, while hook types are worn on the auricle. Hook-type products can be divided into those where the device covers the ear when hung on the auricle, and those where the device is worn inside the ear when hung on the auricle. Kernel types are worn by inserting wireless earphones into the ear canal, while open types are worn by wearing wireless earphones on the ear canal side, i.e., on the concha of the ear. When comparing where the ear tips of earphones are located, kernel types are worn deeper than open types.

[0003] At this time, methods for making earphones custom-made to the shape of the ear have been researched and developed. In this regard, prior art Patent Document 1: Korean Patent Publication No. 2021-0070624 (published on June 15, 2021) and Patent Document 2: Korean Patent Publication No. 2018-0085152 (published on July 26, 2018) disclose a configuration for collecting image information of the ear shape using a measurement sample to measure the ear shape and proceeding with a design corresponding to the ear shape, and a configuration for making earphones that correspond to the shape of the ear canal through an impression of the ear taken by imitating the ear shape.

[0004] However, in both the former and latter cases, the impression of each individual's ear is cast and molded, and it is merely a one-to-one custom-made type, and is not designed to produce earphones for the general public. Earphones developed in various shapes can cause pain and inconvenience because they do not fit the shape of the user's ear. Most global companies that develop earphones have not developed ear tips that fit the shape of Korean ears, so if they are too large, it can cause pain, and if they are too small, the earphones can easily fall off without being fixed in place. In addition, ear impressions need to be collected to develop new products, but the variety and amount of data on Korean ear impressions makes it difficult to develop new products. For this reason, there is a need to research and develop a method to collect photos of the shape of each user's ear and analyze the ear shape using deep learning and 3D to recommend the appropriate ear tip. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Korean Patent Publication No. 2021-0070624 (Published on June 15, 2021) [Patent Document 2] Korean Patent Publication No. 2018-0085152 (Published on July 26, 2018) Summary of the Invention [Problem to be solved by the invention]

[0006] In one embodiment of the present invention, when a photo of an ear shape is uploaded from a user terminal, an object corresponding to the ear within the ear shape is detected, identified, and extracted, the extracted object is restored as 3D data, and an ear tip corresponding to the size and shape of the inside of the ear restored in the 3D data is extracted and recommended, thereby reducing the return rate caused by ordering ear tips that do not fit the size and shape of the user's ear, and by collecting photos and 3D data including the ear shape, big data is constructed and ear impressions can be collected, which can be used as Korean ear impression data for new product development, and feedback is collected, and ear shape data, ear tip data, and feedback data are stored in a data set, and an artificial intelligence algorithm is modeled to output ear tip data in order of high feedback when ear shape data is input, and when ear shape data is input using the modeled artificial intelligence algorithm, ear tips are recommended in order of predicted good ratings and reviews, thereby increasing customer satisfaction. However, the technical problem to be achieved by this embodiment is not limited to the above technical problem, and other technical problems may exist. [Means for solving the problem]

[0007] As a technical means for achieving the above-mentioned technical object, one embodiment of the present invention includes a user terminal that photographs and uploads an ear shape and receives recommendations for ear tips corresponding to the ear shape, and an optimization service providing server including a receiving unit that receives a photograph of the ear shape from the user terminal, a detection unit that detects, identifies and detects an object corresponding to the ear in the photograph, a restoration unit that restores the detected object as 3D data, an extraction unit that extracts ear tips corresponding to the internal size and shape of the ear restored in the 3D data, and a recommendation unit that recommends the extracted ear tips to the user terminal. Effect of the Invention

[0008] According to any one of the above-mentioned problem solving means of the present invention, when a photo of an ear shape is uploaded from a user terminal, an object corresponding to the ear within the ear shape is detected, identified and extracted, the extracted object is restored as 3D data, and ear tips corresponding to the size and shape of the inside of the ear restored in the 3D data are extracted and recommended, thereby reducing the return rate caused by orders of ear tips that do not fit the size and shape of the user's ear, big data is constructed by collecting photos and 3D data including the ear shape, and impressions of the ears can be collected, which can be used as Korean ear impression data for development of new products, feedback is collected and ear shape data, ear tip data and feedback data are stored in a data set, an artificial intelligence algorithm is modeled to output ear tip data in order of high feedback when ear shape data is input, and when ear shape data is input to the modeled artificial intelligence algorithm, ear tips are recommended in order of predicted good ratings and reviews, thereby increasing customer satisfaction. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating a system for providing an ear tip optimization service using an ear shape according to an embodiment of the present invention. [Diagram 2] 2 is a block diagram illustrating an optimization service providing server included in the system of FIG. 1. [Figure 3a] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 3b] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 3c] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 3d]1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 3e] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 3f] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 3g] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 3h] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 3i] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 3j] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 4a] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 4b] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 4c] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 4d] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 4e] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 4f]1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Figure 4g] 1 is a diagram illustrating an embodiment of an ear tip optimization service that uses an ear shape according to an embodiment of the present invention; [Diagram 5] 1 is an operational flowchart illustrating a method for providing an ear tip optimization service using an ear shape according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains can easily carry out the present invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are designated by similar reference numerals throughout the specification.

[0011] Throughout the specification, when a part is described as being "connected" to another part, this includes not only when the part is "directly connected" to another part, but also when the part is "electrically connected" to another part via another element in between. Furthermore, when a part is described as "comprising" a certain component, this does not mean excluding other components, but means that the part may further include other components, unless otherwise specified to the contrary, and does not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0012] The terms "about," "substantially," and the like, as used throughout the specification, are used in the sense of a numerical value or close to a numerical value when the manufacturing and material tolerances inherent in the referred meaning are given, and are used to prevent unscrupulous infringers from taking unfair advantage of the disclosure in which precise or absolute numerical values ​​are recited to aid in the understanding of the present invention. The terms "step of" or "step of" as used throughout the specification of the present invention do not mean "step for".

[0013] In this specification, the term "module" includes a unit implemented by hardware, a unit implemented by software, and a unit implemented by using both. Also, one unit may be implemented by using two or more pieces of hardware, and two or more units may be implemented by one piece of hardware. Meanwhile, the term "module" is not limited to software or hardware, and the "module" may be configured to be in an addressable storage medium or to execute one or more processors. Thus, as an example, the "module" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The components and functions provided within the "module" may be combined into a smaller number of components and "modules" or further separated into additional components and "modules". Moreover, the components and "modules" may be embodied to execute one or more CPUs within a device or a security multimedia card.

[0014] Some of the operations and functions described herein as being performed by a terminal, apparatus, or device may instead be performed by a server connected to the terminal, apparatus, or device. Similarly, some of the operations and functions described herein as being performed by a server may instead be performed by a terminal, apparatus, or device connected to the server.

[0015] In this specification, some of the operations or functions described as mapping or matching with a terminal may be interpreted as mapping or matching a terminal's unique number or personal identification information, which is identifying data of the terminal.

[0016] The present invention will now be described in detail with reference to the accompanying drawings. Fig. 1 is a diagram for explaining an ear tip optimization service providing system using ear shape according to an embodiment of the present invention. Referring to Fig. 1, the ear tip optimization service providing system 1 using ear shape may include at least one user terminal 100, an optimization service providing server 300, and at least one sales terminal 400. However, the ear tip optimization service providing system 1 using ear shape in Fig. 1 is merely one embodiment of the present invention, and the present invention should not be interpreted as being limited by Fig. 1.

[0017] At this time, each component of Fig. 1 is generally connected through a network (Network 200). For example, as shown in Fig. 1, at least one user terminal 100 may be connected to an optimization service providing server 300 through the network 200. And, the optimization service providing server 300 may be connected to at least one user terminal 100 and at least one sales terminal 400 through the network 200. And, at least one sales terminal 400 may be connected to the optimization service providing server 300 through the network 200.

[0018] Here, a network refers to a connected structure that allows information exchange between multiple nodes such as terminals and servers. Examples of such networks include a local area network (LAN), a wide area network (WAN), the Internet (WWW), wired / wireless data communication networks, telephone networks, and wired / wireless television communication networks. Examples of wireless data communication networks include, but are not limited to, 3G, 4G, 5G, 3GPP (registered trademark) (3rd Generation Partnership Project), 5GPP (5th Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth network, NFC (Near-Field Communication) network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc.

[0019] In the following, the term "at least one" is defined as a term including singular and plural, and it is obvious that even if the term "at least one" is not present, each component can exist in singular or plural and can mean singular or plural. Also, whether each component is provided in singular or plural may be variable depending on the embodiment.

[0020] At least one user terminal 100 may be a terminal of a user who photographs and uploads the shape of his / her ears using a web page, app page, program, or application related to an ear tip optimization service that uses ear shape, and receives ear tip recommendations.

[0021] Here, at least one user terminal 100 may be implemented as a computer that can connect to a remote server or terminal through a network. Here, the computer may include, for example, a notebook computer, a desktop, a laptop, etc. equipped with a navigation system and a web browser. At this time, at least one user terminal 100 may be implemented as a terminal that can connect to a remote server or terminal through a network. At least one user terminal 100 may be, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminals, smartphones, smartpads, tablet PCs, etc.

[0022] The optimization service providing server 300 may be a server that provides an ear tip optimization service web page, app page, program, or application using the ear shape. The optimization service providing server 300 may be a server that creates a database of ear tip data including at least one ear tip shape and size. The optimization service providing server 300 may be a server that detects, identifies, and extracts an object corresponding to the ear in a photograph of the ear when the ear shape is photographed and uploaded from the user terminal 100, and restores the object in 3D data. The optimization service providing server 300 may be a server that detects at least one feature point inside the ear using an AR ruler in the 3D data, and extracts the size and shape of a preset area within the feature point. The optimization service providing server 300 may be a server that extracts and recommends an optimal ear tip by comparing the ear shape data and ear tip data of the user terminal 100.

[0023] Here, the optimization service providing server 300 may be implemented as a computer that can be connected to a remote server or terminal through a network. Here, the computer may include, for example, a notebook computer, desktop, laptop, etc. equipped with a navigation system and a web browser.

[0024] At least one sales terminal 400 may be a sales terminal that uses a web page, an app page, a program, or an application related to an ear tip optimization service using an ear shape. The sales terminal 400 may be a terminal that uploads inventory information to the optimization service providing server 300.

[0025] Here, at least one sales terminal 400 may be implemented as a computer that can connect to a remote server or terminal through a network. Here, the computer may include, for example, a notebook computer, a desktop, a laptop, etc. equipped with a navigation system and a web browser. At this time, at least one sales terminal 400 may be implemented as a terminal that can connect to a remote server or terminal through a network. At least one sales terminal 400 may be, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminals, smartphones, smartpads, tablet PCs, etc.

[0026] FIG. 2 is a block diagram illustrating an optimization service providing server included in the system of FIG. 1, and FIG. 3 and FIG. 4 are diagrams illustrating an embodiment of an ear tip optimization service using ear shape in accordance with an embodiment of the present invention.

[0027] Referring to FIG. 2, the optimization service providing server 300 may include a receiving unit 310, a detecting unit 320, a restoring unit 330, an extracting unit 340, a recommending unit 350, a sensor unit 360, a measuring unit 370, a database unit 380, a constructing unit 390, a feedback storing unit 391, a modeling unit 393, a sentiment analysis unit 395, a new product development unit 397, and a purchasing linking unit 399.

[0028] When another server (not shown) operating in conjunction with the optimization service providing server 300 according to an embodiment of the present invention transmits an ear tip optimization service application, program, application page, web page, etc. using the ear shape to at least one user terminal 100 and at least one sales terminal 400, at least one user terminal 100 and at least one sales terminal 400 can install or open an ear tip optimization service application, program, application page, web page, etc. using the ear shape. In addition, the service program may be driven in at least one user terminal 100 and at least one sales terminal 400 using a script executed in a web browser. Here, the web browser is a program that enables the use of a web (WWW) service, and refers to a program that receives and displays hypertext described in HTML (Hyper Text Mark-up Language), and includes, for example, Netscape, Explorer, Chrome, etc. In addition, the application refers to an application on a terminal, and includes, for example, an app executed in a mobile terminal (smartphone).

[0029] Before describing Fig. 2, the basic concept of converting 2D data into 3D data will be described. The contents described below will not be repeated when describing Fig. 2.

[0030] <Point cloud>

[0031] A point cloud is a collection of points that belong to a certain coordinate system. In a 3D coordinate system, points are generally defined by (X, Y, Z), and a point cloud can be applied to a depth image to convert a 2D coordinate system to a 3D coordinate system. First, to convert a depth image, which is a 2D coordinate system, to a 3D coordinate system, the focal length, the coordinates of the 2D depth image, and the depth value, which are among the internal parameter information of the camera that captured the depth image, are used. Here, the focal length is the distance from the camera lens to the image sensor. For example, when a camera captures a ball, the light reflected from the ball passes through the lens. The light that passes through the lens is imaged on the image sensor, and a 2D depth image of the ball is obtained. The 2D depth image must be used to convert from the 2D coordinate system to the 3D coordinate system. Here, the Similar TriAngles Rule can be used to easily convert. Assuming that f is the focal length, u is the coordinate where the object is located in the depth image, the origin of the coordinates is not the upper left corner of the image but the midpoint of the image, z is the Depth Value that corresponds to the 2D coordinate in the 2D depth image, and x is the X-axis coordinate of the object when converted to the 3D coordinate system, then by substituting this into the following mathematical formula 1, the x-coordinate can be estimated because the value obtained by dividing the x-axis distance u from the midpoint of the 2D image to the corresponding pixel by the focal length f is the same as the value obtained by dividing the x-coordinate by the Depth Value in the 3D coordinate system. Therefore, u in the 2D depth image can be converted to a 3D coordinate for the X-axis, and by repeating the above method for the Y-axis, the 2D coordinate system can be converted to a 3D coordinate system.

[0032]

number

[0033] <Coordinate conversion>

[0034] Coordinate transformation is the process of converting the position of one point to another in a coordinate system. In computer vision, rotation transformation, scaling transformation, and translation transformation can be used to model the matching relationship between many images.

[0035] <3D coordinate conversion>

[0036] 3D coordinate transformation is the transformation of coordinate axes in 3D space. 3D coordinate transformation can be used to change the 3D world coordinate system to the camera coordinate system or the camera coordinate system to the 3D world coordinate system. In this process, the coordinate axes are always rotated through a rotation transformation, and then translation transformation is performed to obtain a point (X1, Y1, Z1) in the camera coordinate system as a point (X, Y, Z) in the world coordinate system. In the 3D world coordinate system, the plane of the X and Y axes represents the ground, and the Z axis is defined as being perpendicular to the plane. In the camera coordinate system, the Zc axis is defined as the direction of the optical axis of the camera, the Xc axis is vertical to the right of the optical axis, and the Yc axis is perpendicular to the Xc and Zc axes.

[0037] Rotation transformation rotates a point (X, Y, Z) in 3D space around the X-axis, Y-axis, and Z-axis in a counterclockwise direction using a rotation matrix by θ, with the direction of each axis as the reference. The rotation matrices for rotating each axis are as shown in Equation 2, Equation 3, and Equation 4. Equation 2 is a matrix for rotating around the X-axis, Equation 3 is a matrix for rotating around the Y-axis, and Equation 4 is a matrix for rotating around the Z-axis. By combining the rotation matrices in Equations 2 to 4, any 3D rotation can be expressed as shown in Equation 5.

[0038]

number

[0039]

number

[0040]

number

[0041]

number

[0042]

number

[0043] The translation transformation is a rotation transformation to convert a point (XC, YC, ZC) on the camera coordinate system into a point (X, Y, Z) on the world coordinate system, and then the camera coordinate system is translated in the direction of the world coordinate system using Equation 6.

[0044] Based on the above-described basic concept, referring to Fig. 2, the receiving unit 310 can receive a photograph of an ear shape from the user terminal 100. The user terminal 100 can upload the photograph of the ear shape after taking the photograph.

[0045] The detection unit 320 can detect, identify, and detect an object corresponding to an ear in a photograph. At this time, object detection is to analyze a given image when a specific image is input and grasp the location and class of a specific object. An object detection algorithm must first find the location of an object in an image and classify what kind of object the object is at that location. Various methods have been attempted to accomplish this, but recently they have been narrowed down to a single-stage detection method and a two-stage detection method. The single-stage detection method simultaneously performs location detection and classification for all areas. The two-stage detection method performs rough location detection and classification for selected candidates. The single-stage detection method simultaneously performs object location detection and classification for all areas, so it is fast, but has the disadvantage of being slightly less accurate than the two-stage detection method. On the other hand, the two-stage detector has a disadvantage that it is relatively accurate but slow because object position detection and classification are performed sequentially. In this regard, in one embodiment of the present invention, the two-stage detection method is used because the single-stage detection method is fast but has low accuracy. Of course, the single-stage detection method is not excluded.

[0046] <Ear recognition model that is robust to pose changes>

[0047] In a real environment, to prevent performance degradation due to pose changes, the user must take certain actions to obtain a frontal image of the ear, which may be burdensome for the user. Therefore, to be robust to pose changes, a database of photos and images of ears in various environments with pose changes, lighting changes, and partial occlusions from the front can be constructed. In addition, to take into account unique characteristics such as the shape of the ear, zero padding is performed on the input image as preprocessing, and feature extraction based on VGG-16 and ResNet50 models and score-level based ensemble techniques can be used to improve recognition performance.

[0048] <Pretreatment>

[0049] In most cases, data augmentation is performed to add missing images by rotating images, adjusting brightness or intensity, cropping, applying Gaussian blur or noise, etc., but preprocessing that takes into account ear characteristics such as the width-to-height ratio and shape of the ear is rare. Therefore, when resizing the input image to fit the square CNN model input layer, geometric deformation may occur in the shape and structure of the ear. In one embodiment of the present invention, zero padding may be applied so that the width and height of the original image are the same in order to maintain the ratio of the original image and prevent geometric deformation that may occur when resizing the input image to fit the size of the CNN square input layer. Augmentation techniques that cut out a certain area in an image are often used to secure sufficient data for CNN model training, but when an image is randomly cut out, data with many areas that are not necessary for training, such as hair and background, may be obtained. Therefore, in one embodiment of the present invention, data can be secured by performing a cut based on the center of the image, and the data can be set as training and validation data sets by dividing it into 8:2 or 7:3.

[0050] <cnn>

[0051] In order to construct an ear recognition model that is robust to unconstrained environments based on deep learning, various models such as AlexNet, VGG, and SqueezeNet can be used. In one embodiment of the present invention, VGG-16 and ResNet50, which have achieved excellent results in the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC), can be used to research CNN-based ear recognition that is robust to pose changes. In order to construct a feature vector that represents an input image, VGG-16 extracts N-dimensional features in the second fully connected layer, and ResNet50 extracts M-dimensional features by performing an average pooling operation after passing through the last convolutional layer. When there is little available training data, it is widely known that fine-tuning a model that has been pre-trained with a large database such as ImageNet is an effective training method. Accordingly, in one embodiment of the present invention, VGG-16 and ResNet50 models that have been pre-trained with the ImageNet database can be used as initial models to retrain the pre-constructed ear database.

[0052] <Ensemble>

[0053] Ensemble is a technique for improving performance by combining various models and fusing information held by each model. Ensemble can be performed at various levels such as data level and feature level, but the most commonly used method when combining feature vectors or models of different types is score and decision level ensemble. Ensemble can be performed in various ways such as Product Rule, Sum Rule, Min / Max Rule, Median Rule, Majority Voting, etc., and it is generally known that the Sum Rule method shows good performance. In one embodiment of the present invention, score level ensemble can be used to combine VGG-16 and ResNet50. For this, pairs of same person and different person can be formed in the learning data set, and the cosine distance of each pair can be calculated. Then, the average and standard deviation of the cosine distance can be calculated and used as parameters of the Z-Score normalization technique performed to normalize the cosine distance distribution of both models in the validation data. The normalized cosine distance can be used as a score for performing the ensemble to calculate the final score based on the Sum Rule.

[0054] The restoration unit 330 can restore the detected object as 3D data. At this time, as described later, since ToF is used when the user terminal 100 takes an image, depth information and distance information can be obtained, which is the same as data collected from various sensors and technologies that acquire 3D information of the real world, such as a stereo camera, a 3D scanner, an RGB-D sensor, etc., including a LiDAR (Light Detection And Ranging) when generating 3D data. At this time, the RGBD (RGB+Depth) image captured by the ToF is converted into a 3D point cloud, and after removing parts other than the ear and noise, the point cloud is matched using the camera position value obtained by ARKit of the entire ear and an ICP (Iterative Closet Point) algorithm. After transmitting one point cloud and restoring the surface, the matched ear can be converted into a mesh. At this time, it is assumed that the user terminal 100 is a terminal provided with a ToF or Lidar sensor, and RGBD can be converted into a point cloud using MetalKit, and 3D data restoration can be processed using SceneKit. Point cloud processing can be done by embedding PythonKit to take advantage of the functionality of the Open3D library.

[0055] <Individual point cloud generation>

[0056] Using ARKit, you can access the ToF and Color cameras in front of the user device 100. At this time, activate the Gravity option so that the Y axis of the world coordinate system of the user device 100 is parallel to the Earth's gravity. Then, in ARFrame, you can obtain a point cloud using the RGBD obtained by the Color camera and ToF and the camera's intrinsic and extrinsic parameters.

[0057] <Ear region segmentation extraction>

[0058] The point cloud obtained using ARKit includes hair, neck, face, etc. other than the ears, so the rest must be removed except for the vertices corresponding to the ears. The base of the point cloud obtained when using the Gravity option in ARKit is perpendicular to the Y axis. Using this, most of the vertices can be removed. You can also use the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm to remove noise. This process can be done using the Python-based Open3D and Numpy libraries.

[0059] <Point Cloud Matching>

[0060] A global camera parameter estimation method using 3D feature points from the ear can be used. Alternatively, when the ARFrame is generated, the camera information estimated during shooting is provided, but errors may occur in the position and orientation information. If this is used as is, the accuracy of point cloud matching may be excessively low. For this reason, matching can be performed using an ICP algorithm that considers the similarity of overlapping parts after applying the camera position information. The above process can also be performed using PythonKit, and the Open3D and Numpy libraries can be used. The point clouds matched in this way can be converted into a 3D mesh, and 3D data can be generated accordingly.

[0061] The extraction unit 340 can extract an ear tip corresponding to the size and shape of the inside of the ear restored in 3D data. The size of a preset area of ​​the ear converted into 3D data must be measured, and for this, the feature points of the ear must be extracted in the same way as the feature points of the face are extracted. Since the area where the ear tip is inserted is fixed, the shape and size of this area must be measured. For this, feature points can be extracted to extract the size and shape of the area where the ear tip is inserted. Referring to FIG. 4f, the part to be photographed is the outer ear, which is made up of the pinna and the ear canal. Here, the part from the inside of the pinna to the part that connects to the ear canal is the part where the ear tip can be inserted. In order to extract this part, feature points of the inner area of ​​the pinna and the area where the ear canal begins can be extracted, and then the size and shape can be extracted.

[0062] <Features>

[0063] First, it detects only the edges that can be features in the 3D data. Canny Edge Detection can be used to detect edges, and they can be expressed as binary images. A feature extraction model based on a convolutional neural network autoencoder can be used, which consists of two convolutional layers, one fully connected layer, and three hidden layers. The input terminal can use the binary image where the edges are detected, and stride 2 is applied from each convolutional layer, and pooling is not performed. The output of the last hidden layer can be used as the feature of the ear image to extract features. After that, the decoding part is configured in the reverse order of the above configuration, but the fully connected layer in one-dimensional form can be reconstructed in two-dimensional form. Also, the encoding process can proceed using DeConvolution instead of convolution. When learning, dropout can be applied to each layer to prevent overfitting, and Relu can be applied as the activation function of the convolutional neural network structure, and Sigmoid can be applied to the deep neural network after the fully connected layer. In training a model thus completed, the Adam optimizer of the backpropagation algorithm can be applied.

[0064] <lidar>

[0065] When using Lidar, 3D modeling requires classifying and dividing ear objects in a situation where point cloud data of various objects other than ears, such as face, lips, and nose, is mixed, and the ear objects can be found by searching for Model Key Feature in the separated ear objects or by connecting them in the form of Polygon Mesh. Model Key Feature refers to important points, lines, and faces that have characteristics in point cloud data, and for 3D modeling, it is necessary to detect important features and to have topological structure information on the order and relationship in which they should be connected. Polygon Mesh refers to performing 3D modeling with a set of polygons and vertices that make up a polyhedron, and in point cloud data, all points are vertices, and the relationships between them are found and connected to create polygons to generate a 3D model. The Model Key Feature method detects only the important features of the point cloud to perform 3D modeling, so the resulting data is small in size and the shape can be expressed simply and clearly, but it is difficult to accurately detect each characteristic element and find the topological structure relationship. On the other hand, the Polygon Mesh method searches for all points in a point cloud as vertices and connects them in the form of polygons to perform 3D modeling, so that a 3D model can be generated relatively easily using a Polygon Mesh generation algorithm such as TIN (Triangulated Irregular Network). For this, for example, https: / / poly.cam / can be used, but since the iPhone has recently been equipped with a Lidar sensor, a method can be used in which the Lidar sensor in the iPhone is used to convert the data into 3D as shown in Figures 3b to 3j, and the values ​​between each feature point are measured as shown in Table 1 below. Of course, the applications that can be used are not limited to those mentioned above. It goes without saying that a 3D model can be generated from Lidar data in various ways other than the above-mentioned methods.

[0066] The recommendation unit 350 can recommend the extracted ear tip to the user terminal 100. The user terminal 100 can receive recommendations of ear tips corresponding to the shape of the ear. At this time, the shape and size of the ear tip are also stored in a database, and since 3D data exists when designing the ear tip, it is possible to provide an ear tip with the most similar shape and size through a comparison between the 3D data. At this time, it is possible to virtually wear the earphone using characteristic points or landmarks marked in the design of the earphone and landmarks of 3D data corresponding to the shape of the ear, and then analyze the pressure and free space. In addition, it is possible to grasp whether the height and width of the ear tip correspond to the shape of the user's ear by using the distance from the starting point of the ear tip to the wall surface where the ear canal starts. The landmarks and dimensions used at this time can be those listed in "Wonsup Lee, Xiaopeng Yang, Hayoung Jung, Ilgeun Bok, Chulwoo Kim, Ochae Kwon & Heecheon You (2018) Anthropometric analysis of 3D ear scans of Koreans and Caucasians for ear product design, Ergonomics, 61:11, 1480-1495," referring to FIG. 4g. These may be the same as, but are not limited to, those in Table 1 below.

[0067] [Table 1] JPEG2025015519000009.jpg50166

[0068] When the sensor unit 360 photographs the shape of the ear with the user terminal 100, it can measure the distance between the camera of the user terminal 100 and the shape of the ear using ToF (Time of Flight) using infrared rays built into the user terminal 100. At this time, ToF measures the signal reflected back from the object after reflecting the signal, and estimates the depth of the object using the time it takes. The image sensor can detect depth information for all pixels to generate a reliable depth map, and using this, it is possible to obtain depth information from a 2D photo without using a depth camera or a 3D scanner, and thus obtain 3D data like Lidar. This has been described above, so a detailed description will not be given. Also, although the description has been given based on ToF using infrared rays based on Android (registered trademark), when based on iOS that uses Lidar sensor with ToF, a method using Lidar other than the above-mentioned ToF may be possible. After restoring the 3D data, the measurement unit 370 may detect at least one feature point inside the ear using an AR ruler and extract the size and shape of a pre-defined area within the feature point. At this time, the feature point may be the same as the landmarks or dimensions of the paper in Table 1, but is not limited thereto.

[0069] The database unit 380 can create a database of at least one size and shape of ear tip.

[0070] The construction unit 390 may accumulate photos including ear shapes taken from a plurality of user terminals 100, 3D data, and ear sizes and shapes, and construct the big data. In this case, the user terminals 100 may be a plurality of user terminals 100 belonging to a plurality of subjects. Of course, representative ear image databases such as USTB and IITD databases may be used, but if Koreans are the target, a Korean database must be constructed separately. Accordingly, after identifying the country in which the product was purchased using the country of sale, payment currency, etc., a database may be constructed by race, nationality, etc.

[0071] When a rating and a review are written after purchasing ear tips from a plurality of user terminals 100, the feedback storage unit 391 may map and store a plurality of ear shape data of a plurality of user terminals 100, ear tip data of the purchased ear tips, rating and reviews. At this time, even if an ear tip is extracted as being ergonomically suitable, it may be inconvenient or not inconvenient depending on the usage form, period of use, and time of use of each user. Accordingly, after collecting the usage form, period of use, time of use, and purpose of use in addition to demographic data, data on whether the same ear tip is inconvenient or comfortable in what usage form can be analyzed. For example, suppose that A and B are identical twins and have the same Z-shaped ears, and purchase C ear tips for the same ear shape, but A listens while working and B listens while exercising. Assuming that the time is the same, if A finds it inconvenient and B finds it comfortable, the Z-shaped ear tip C may be an ear tip that is good for listening while working, and may be suitable for a user who purchases it in such a situation. In this way, by continuously accumulating the shape of each ear, recommended and purchased ear tips, usage patterns, ratings, and purchase dates, it is possible to collect big data that can serve as the basis for determining which types of ear tips are best suited for which situations and for whom.

[0072] The modeling unit 393 can map a plurality of ear shape data, ear tip data of purchased ear tips, ratings, and reviews to construct a data set, and can learn and verify to extract ear tip data with high ratings and positive reviews by inputting ear shape data using at least one artificial intelligence algorithm. At this time, the at least one artificial intelligence algorithm is not limited to any one algorithm since various algorithms can be used.

[0073] The sentiment analysis unit 395 may use at least one language model for analyzing positive or negative sentiment from the reviews. The language model for this may be, but is not limited to, KoBERT (Korean Bidirectional Encoder Representations from Transformers), a natural language processing-based Korean sentiment analysis model trained to enable multiple classification. Although KoBERT is used as a basic embedding model of a sentence, a fully-connected layer may be additionally connected to extract a lower-dimensional final embedding vector. After that, a classifier may be added to proceed with learning of a detailed sentiment classification task. In a baseline model for comparing the performance of loss functions, a classifier composed of a fully connected layer and a softmax function may be used, and in a model to which a metric learning-based loss function is added, a classifier that predicts the closest class as the correct answer based on the distance from the central embedding of each class may be used.

[0074] The new product development department 397 can provide big data to be used as raw data for the development of new products.

[0075] The purchase connection unit 399 extracts at least one online shopping mall selling the recommended ear tips and provides it to the user terminal 100, and extracts offline stores that have the ear tips in stock, and then lists them in order of proximity based on the location of the user terminal 100 and provides them to the user terminal 100.

[0076] Hereinafter, the operation process of the optimization service providing server according to the configuration of Fig. 2 will be described in detail with reference to Fig. 3 and Fig. 4. However, it is obvious that the embodiment is only one of various embodiments of the present invention and is not limited thereto.

[0077] Referring to FIG. 3a, (a) the optimization service providing server 300 stores at least one ear tip size and shape in a database, and when a photo of the ear shape is uploaded from the user terminal 100 as in (b), it detects, identifies and extracts the ear, which is an object in the photo, and restores 3D data from it. The optimization service providing server 300 can extract the ear size and shape as in (c) and then extract and recommend the most suitable ear tip as in (d). Referring to FIG. 3b (a), the optimization service providing server 300 can collect feedback from the user terminal 100 that purchased the ear tip, and model a recommendation algorithm as in (b). As in (c), the optimization service providing server 300 can connect an online shopping mall or an offline store for purchasing the recommended ear tip, and as in (d), the accumulated data on the ear impression can be used for the development of a new product. This is summarized as in FIG. 4a. In the case of FIG. 4b, 3D data is restored from an ear shape image and a corresponding ear tip is recommended, but the flow is not limited thereto. Currently, the applicant of the present invention manufactures, produces and sells a product as shown in the left side of Fig. 4c, but when carrying out an embodiment of the present invention as shown in the right side of Fig. 4c and Figs. 4d and 4e, ear tips can be automatically selected, which induces quick purchase decisions and reduces the return rate, which can help improve profits. It should be made clear that Figs. 4g and 4f are not drawings according to an embodiment of the present invention, but are drawings attached as examples to explain the anatomical structure, features, size, landmarks, etc. of the ear.

[0078] Matters not described in the method of providing an ear tip optimization service using ear shape in Figures 2 to 4 are the same as or can be easily inferred from the content previously described in the method of providing an ear tip optimization service using ear shape through Figure 1, so they will not be described below.

[0079] Fig. 5 is a diagram showing a process of transmitting and receiving data between each component included in the system for providing an ear tip optimization service using an ear shape of Fig. 1 according to an embodiment of the present invention. Hereinafter, an example of a process of transmitting and receiving data between each component will be described with reference to Fig. 5, but the present application is not limited to such an embodiment, and it is obvious to those skilled in the art that the process of transmitting and receiving data shown in Fig. 5 can be changed according to the various embodiments described above.

[0080] Referring to FIG. 5, an optimization service providing server receives a photograph of an ear shape from a user terminal (S5100), and detects, identifies, and identifies an object corresponding to the ear in the photograph (S5200).

[0081] In addition, the optimization service providing server restores the detected object as 3D data (S5300), and extracts an ear tip corresponding to the size and shape of the inside of the ear restored from the 3D data (S5400).

[0082] Then, the optimization service providing server recommends the extracted ear tips to the user terminal (S5500).

[0083] The order of the above steps (S5100 to S5500) is merely an example and is not limited thereto, that is, the order of the above steps (S5100 to S5500) may be changed, and some of the steps may be performed simultaneously or deleted.

[0084] Matters not described in the method of providing an ear tip optimization service using ear shape in Figure 5 are the same as or can be easily inferred from the contents previously described in the method of providing an ear tip optimization service using ear shape through Figures 1 to 4, so they will not be described below.

[0085] The method for providing an ear tip optimization service using the shape of an ear according to an embodiment described in FIG. 5 may also be embodied in the form of a recording medium including computer executable instructions such as an application or a program module executed by a computer. A computer readable medium may be any available medium that can be accessed by a computer, and includes all volatile and non-volatile media, and separate and non-separate media. Also, a computer readable medium may include all computer storage media. A computer storage medium includes all volatile and non-volatile, separate and non-separate media embodied in any method or technology for storing information such as computer readable instructions, data structures, program modules, or other data.

[0086] The method for providing an ear tip optimization service using the shape of the ear according to an embodiment of the present invention described above may be executed by an application (which may include a program included in a platform or operating system that is basically installed in the terminal) that is installed in the terminal, or may be executed by an application (i.e., a program) that the user directly installs in the master terminal through an application providing server such as an application store server, an application, or a web server related to the service. In this sense, the method for providing an ear tip optimization service using the shape of the ear according to an embodiment of the present invention described above may be embodied as an application (i.e., a program) that is installed in the terminal or directly installed by the user, and may be recorded in a computer-readable recording medium such as a terminal.

[0087] The above description of the present invention is for illustrative purposes only, and those skilled in the art to which the present invention pertains may easily modify the present invention into other specific forms without changing the technical spirit or essential features of the present invention. Therefore, the above-described embodiments should be understood to be illustrative and not limiting in all respects. For example, each component described as a single type may be implemented in a distributed form, and similarly, each component described as a distributed type may be implemented in a combined form.

[0088] The scope of the present invention is defined by the claims set forth below rather than the above detailed description, and all modifications and variations that fall within the meaning and scope of the claims and their equivalent concepts are also included within the scope of the present invention.< / lidar> < / cnn>

Claims

1. A user terminal that photographs and uploads the shape of the ear and receives recommendations of ear tips corresponding to the shape of the ear; and an optimization service providing server including a receiving unit for receiving a photograph of an ear shape from the user terminal; a detecting unit for detecting, identifying and detecting an object corresponding to the ear in the photograph; a restoring unit for restoring the detected object as 3D data; an extracting unit for extracting an ear tip corresponding to the size and shape of the inside of the ear restored from the 3D data; and a recommending unit for recommending the extracted ear tip to the user terminal; The detection unit is performing position detection related to the ear in the photograph, and performing classification on the selected candidate group to detect an object corresponding to the ear in the photograph; The extraction unit is extracting feature points of the inner region of the auricle and the region where the ear canal begins in order to extract a region where an ear tip can be inserted from the inside of the auricle to a region connecting to the ear canal, extracting the size and shape of the inside of the ear, which is ear shape data of the user terminal, and comparing ear tip data including at least one shape and size of an ear tip stored in a database in the optimization service providing server with the ear shape data of the user terminal to extract the ear tip corresponding to the extracted size and shape of the inside of the ear; The recommendation unit, The earphones are virtually worn using the characteristic points and landmarks marked in the design and the 3D data landmarks corresponding to the shape of the ears, and then the pressure and free space are analyzed, and whether the height and width of the ear tips correspond to the shape of the user's ears is determined using the distance from the start of the ear tips to the wall where the ear canal begins. An ear tip optimization service providing system that utilizes the shape of the ear.

2. The restoration unit is Lidar is used to reconstruct the detected object into 3D data. The ear tip optimization service providing system using the ear shape according to claim 1.

3. The optimization service providing server includes: a sensor unit for measuring a distance between a camera of the user terminal and the shape of the ear by using infrared rays built into the user terminal and ToF (Time of Flight) when the shape of the ear is photographed by the user terminal; The ear tip optimization service providing system using the ear shape according to claim 1.

4. The optimization service providing server includes: After the 3D data is restored, the ear detection unit detects at least one feature point inside the ear using an AR ruler, and extracts a size and a shape of a predetermined area within the feature point. The ear tip optimization service providing system using the ear shape according to claim 1.

5. The optimization service providing server includes: A database unit for creating a database of at least one size and shape of the ear tip. The ear tip optimization service providing system using the ear shape according to claim 1.

6. The user terminal is a plurality of user terminals belonging to a plurality of entities, The optimization service providing server includes: a construction unit that accumulates the photos including the ear shapes taken from the plurality of user terminals, the 3D data, and the sizes and shapes of the ears, and constructs the photos into big data; The ear tip optimization service providing system using the ear shape according to claim 1.

7. The optimization service providing server includes: A feedback storage unit that maps and stores a plurality of ear shape data of the plurality of user terminals, ear tip data of the purchased ear tips, ratings, and reviews when the ear tips are purchased from the plurality of user terminals and a rating and review are written. The ear tip optimization service providing system using the ear shape according to claim 6.

8. The optimization service providing server includes: A modeling unit that constructs a data set by mapping a plurality of ear shape data, ear tip data of purchased ear tips, ratings and reviews, and learns and validates by using at least one artificial intelligence algorithm to extract ear tip data with high ratings and positive reviews when the ear shape data is input; The ear tip optimization service providing system using the ear shape according to claim 7.

9. The optimization service providing server includes: a sentiment analysis unit that utilizes at least one language model for positive or negative sentiment analysis of the reviews; The ear tip optimization service providing system using the ear shape according to claim 7.

10. The optimization service providing server includes: A new product development department that provides the big data for use as raw data for new product development; The ear tip optimization service providing system using the ear shape according to claim 6.

11. The optimization service providing server includes: a purchase connection unit that extracts at least one online shopping mall selling the recommended ear tips and provides it to the user terminal, extracts offline stores that have the ear tips in stock, and lists them in order of proximity based on the location of the user terminal and provides them to the user terminal; The ear tip optimization service providing system using the ear shape according to claim 1.

Images