Electronic device for processing image and operating method of electronic device

Abstract

A method includes: obtaining a first image of an object including a surface having a non-flat shape; identifying a region corresponding to the surface as a region of interest by applying the first image to a first artificial intelligence model; obtaining data on a three-dimensional (3D) shape type of the object by applying the first image to a second AI model; obtaining a set of values of a 3D parameter associated with the object, the surface or the first camera based on the region and the data; estimating the non-flat shape of the surface based on the set of values of the 3D parameter; and obtaining a flat surface image in which the non-flat shape of the surface is planarized by performing a perspective transformation on the surface.

Description

Technical Field

[0001] The present disclosure relates to an electronic device to which an algorithm for removing distortion of a region of interest in an image is applied, and an operating method of the electronic device. Background Art

[0002] In a digital image obtained by photographing a three-dimensional (3D) object, there is physical distortion due to a non-flat (e.g., curved) surface of the 3D object, distortion due to a photographing viewing angle, etc. Various techniques using 3D information have been developed to remove distortion caused by 3D characteristics. Among image distortion removal methods using 3D information, an algorithm for inferring 3D information of an object by using an algorithm and removing distortion in an image without hardware such as a sensor for obtaining 3D information has recently been used. Summary of the invention

[0003] According to one aspect of the present invention, a method for processing an image performed by an electronic device is provided. The method may include obtaining a first image of a three-dimensional (3D) object including at least one surface by using a first camera, and at least one surface has a non-flat shape. The method may include identifying an area corresponding to at least one surface as a region of interest (ROI) by applying the first image to a first artificial intelligence (AI) model. The method may include obtaining data about the 3D shape type of the object by applying the first image to a second AI model. The method may include obtaining a set of values ​​of 3D parameters related to at least one of the object, at least one surface, or the first camera based on the area identified as the ROI and the data about the 3D shape type. The method may include estimating the non-flat shape of the at least one surface based on the set of values ​​of the 3D parameters. The method may include obtaining a flat surface image in which the non-flat shape of at least one surface is flattened by performing a perspective transformation on at least one surface.

[0004] According to one aspect of the present disclosure, an electronic device for processing an image is provided. The electronic device may include a first camera, a memory storing one or more instructions, and one or more processors configured to execute the one or more instructions stored in the memory. The one or more processors may be configured to execute the one or more instructions to obtain a first image of a three-dimensional (3D) object including at least one surface by using the first camera, and the at least one surface has a non-flat shape. The one or more processors may also be configured to execute one or more instructions to identify an area corresponding to at least one surface as a region of interest (ROI) by applying the first image to a first artificial intelligence (AI) model. The one or more processors may also be configured to execute one or more instructions to obtain data about the 3D shape type of the object by applying the first image to a second AI model. The one or more processors may also be configured to execute one or more instructions to obtain a set of values ​​of 3D parameters related to at least one of the object, at least one surface, or the first camera 2200 based on the area identified as the ROI and the data about the 3D shape type. The one or more processors may also be configured to execute one or more instructions to estimate the non-flat shape of at least one surface based on the set of values ​​of the 3D parameters. The one or more processors may also be configured to execute the one or more instructions to obtain a flat surface image in which a non-flat shape of the at least one surface is flattened by performing a perspective transformation on the at least one surface.

[0005] According to an aspect of the present invention, a method for processing an image performed by an electronic device is provided. The method may include obtaining a partial image of the object including the surface of the object by using a first camera. The method may include identifying an area corresponding to the surface of the object as an ROI by applying the partial image of the object to a first AI model. The method may include obtaining an entire image of the object by using a second camera having a wider viewing angle than the first camera. The method may include identifying the 3D shape type of the object by applying the entire image of the object to a second AI model. The method may include obtaining a set of values ​​of 3D parameters corresponding to the 3D shape type of the object. The method may include obtaining a flat surface image in which the curved shape of the surface has been flattened by performing a perspective transformation of the surface based on information about the ROI and a set of values ​​of the 3D parameters. The method may include obtaining information related to the object from the flat surface image.

[0006] According to one aspect of the present disclosure, an electronic device for processing an image is provided. The electronic device may include a first camera, a second camera, a memory storing one or more instructions, and one or more processors configured to execute the one or more instructions stored in the memory. The one or more processors may be configured to execute the one or more instructions to obtain a partial image of the object including the surface of the object by using the first camera. The one or more processors may also be configured to execute one or more instructions to identify an area corresponding to the surface of the object as an ROI by applying a partial image of the object to a first AI model. The one or more processors may also be configured to execute the one or more instructions to obtain the entire image of the object by using a second camera having a wider viewing angle than the first camera. The one or more processors may also be configured to execute one or more instructions to obtain the 3D shape type of the object by applying the entire image of the object to the second AI model. The one or more processors may also be configured to execute one or more instructions to obtain a set of values ​​of 3D parameters corresponding to the 3D shape type of the object. The one or more processors may also be configured to execute one or more instructions to obtain a flat surface image in which the curved shape of the surface has been flattened by performing a perspective transformation of the surface based on information about the ROI and a set of values ​​of the 3D parameters. The one or more processors may also be configured to execute one or more instructions to obtain information related to the object from the flat surface image.

[0007] According to one aspect of the present disclosure, there is provided a computer-readable recording medium having recorded thereon a program for executing one of the above-described and below-described methods of removing distortion from an image by processing the image, which is performed by an electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 2 is a diagram illustrating that an electronic device removes image distortion according to an embodiment of the present disclosure.

[0009] Figure 2 is a flowchart of a method of processing an image performed by an electronic device according to an embodiment of the present disclosure.

[0010] Figure 3 is a diagram for explaining an operation of processing an image performed by an electronic device according to an embodiment of the present disclosure.

[0011] Figure 4 is a diagram for explaining an operation of recognizing a three-dimensional (3D) shape of an object performed by an electronic device according to an embodiment of the present disclosure.

[0012] Figure 5is a diagram for explaining an operation of recognizing a region of interest (ROI) on a surface of an object, performed by an electronic device according to an embodiment of the present disclosure.

[0013] Fig. 6A is a diagram for explaining an operation of obtaining 3D information of an object, performed by an electronic device according to an embodiment of the present disclosure.

[0014] Figure 6B is a diagram for explaining an operation of removing distortion of a ROI based on 3D information of an object, performed by an electronic device according to an embodiment of the present disclosure.

[0015] Figure 7 is a view for explaining an operation of extracting information in a ROI performed by an electronic device according to an embodiment of the present disclosure.

[0016] Fig. 8A 2 is a view for explaining a first example in which an electronic device obtains a distortion-free image by obtaining 3D information according to an embodiment of the present disclosure.

[0017] Figure 8B 2 is a view for explaining a second example in which the electronic device obtains a distortion-free image by obtaining 3D information according to an embodiment of the present disclosure.

[0018] Figure 8C is a view for explaining a third example in which the electronic device obtains a distortion-free image by obtaining 3D information according to an embodiment of the present disclosure.

[0019] Fig.9A is a view for explaining a first example in which an electronic device extracts information from an undistorted image according to an embodiment of the present disclosure.

[0020] Fig. 9B 2 is a view for explaining a second example in which the electronic device extracts information from an undistorted image according to an embodiment of the present disclosure.

[0021] Fig. 10A is a diagram for explaining an operation of training an object 3D shape recognition model performed by an electronic device according to an embodiment of the present disclosure.

[0022] Fig. 10B is a diagram for explaining another operation of training an object 3D shape recognition model performed by an electronic device according to an embodiment of the present disclosure.

[0023] Fig. 10C 3 is a diagram for explaining an embodiment in which an electronic device recognizes a 3D shape of an object according to an embodiment of the present disclosure.

[0024] Fig. 10D 3 is a diagram for explaining an embodiment in which an electronic device recognizes a 3D shape of an object according to an embodiment of the present disclosure.

[0025] Fig.11 is a diagram for explaining an operation of training a ROI recognition model performed by an electronic device according to an embodiment of the present disclosure.

[0026] Fig.12 is a diagram for explaining an operation of training a distortion removal model performed by an electronic device according to an embodiment of the present disclosure.

[0027] Fig.13 is a diagram for explaining a plurality of cameras included in an electronic device according to an embodiment of the present disclosure.

[0028] Fig.14A is a flowchart of an operation using multiple cameras performed by an electronic device according to an embodiment of the present disclosure.

[0029] Fig. 14B It is used for Fig.14A Further explanation of the flow chart is supplemented in the Fig.

[0030] Fig.15A is a flowchart of an operation using multiple cameras performed by an electronic device according to an embodiment of the present disclosure.

[0031] Fig. 15B It is used for Fig.15A Further explanation of the flow chart is supplemented in the Fig.

[0032] Fig.16A is a flowchart of an operation using multiple cameras performed by an electronic device according to an embodiment of the present disclosure.

[0033] Fig. 16B It is used for Fig.16A Further explanation of the flow chart is supplemented in the Fig.

[0034] Fig. 16C It is used for Fig.16A Further explanation of the flow chart is supplemented in the Fig.

[0035] Fig.17 is a diagram for explaining an operation of processing an image and providing extracted information, performed by an electronic device according to an embodiment of the present disclosure.

[0036] Fig.18 is a diagram for explaining an example of a system related to an operation of processing an image performed by an electronic device according to an embodiment of the present disclosure.

[0037] Fig.19 is a diagram for explaining an example of a system related to an operation of processing an image by using a server, performed by an electronic device according to an embodiment of the present disclosure.

[0038] Fig. 20 is a block diagram of an electronic device according to an embodiment of the present disclosure.

[0039] Fig.21 is a block diagram of a structure of a server according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] Throughout the disclosure, the expression "at least one of a, b, or c" means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0041] Although currently widely used general terms are selected to describe the present disclosure in consideration of the functions of the present disclosure, these general terms may vary according to the intention of ordinary technicians in the field, case precedents, the emergence of new technologies, etc. Terms arbitrarily selected by the applicant of the present disclosure may also be used in specific cases. In this case, their meanings are provided in the detailed description of the present disclosure. Therefore, these terms must be defined based on their meanings and the contents of the entire specification, rather than simply stating these terms.

[0042] Unless it has a clearly different meaning in the context, an expression used in the singular may encompass an expression in the plural. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs. In this specification, although terms such as "first", "second" and the like may be used to describe various components, these components must not be limited to the above terms. The above terms are only used to distinguish one component from another component.

[0043] When used in this specification, the terms "comprises" and / or "comprising" or "includes" and / or "including" specify the presence of the elements described, but do not exclude the presence or addition of one or more other elements. When used in this specification, the terms "unit", "machine" and "module" refer to a unit in which at least one function or operation is performed, and can be implemented as hardware, software, or a combination of hardware and software.

[0044] Embodiments of the present disclosure are described in detail herein with reference to the accompanying drawings so that a person of ordinary skill in the art to which the present disclosure belongs can easily perform the present disclosure. However, the present disclosure can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. In the accompanying drawings, in order to simplify the description, parts irrelevant to the description are omitted, and the same reference numerals always represent the same elements. In addition, the reference numerals used in each of the accompanying drawings are only used to describe each of the accompanying drawings, and the different reference numerals used in different accompanying drawings do not indicate different elements. Embodiments of the present disclosure will now be described more fully with reference to the accompanying drawings.

[0045] Figure 1 2 is a diagram illustrating an example in which an electronic device removes image distortion according to an embodiment of the present disclosure.

[0046] Reference Figure 1 , the electronic device 2000 according to an embodiment of the present disclosure may include a camera and a display. The electronic device 2000 may be a device that captures images (still images and / or videos) through a camera and outputs images through a display. For example, the electronic device 2000 may include, but is not limited to, a smart TV, a smart phone, a tablet personal computer (PC), a laptop PC, etc. The electronic device 2000 may be implemented by using various kinds and types of electronic devices including a camera and a display. The electronic device 2000 may also include a speaker for outputting audio.

[0047] According to an embodiment of the present disclosure, a user of the electronic device 2000 may photograph the object 100 by using a camera of the electronic device 2000. The electronic device 2000 may obtain an image 110 including at least a portion of the object 100.

[0048] In the present disclosure, when there is information to be recognized on the surface of the object 100 in the image, this is referred to as a region of interest (ROI) 120. For example, a region of the surface of the object 100 (e.g., a label region attached to the surface of the object 100) may be the ROI. According to an embodiment of the present disclosure, the electronic device 2000 may extract information related to the object 100 from the ROI 120 of the object 100.

[0049] In the present disclosure, the distortion of removing the "surface (e.g., label)" of the product will be described as an example of ROI 120. Here, the label is made of paper, sticker, fabric, etc. and attached to the product, and the trademark or product name of the product can be printed on the label. The surface (e.g., label) of the product can include various information related to the product, such as the ingredients, usage, usage, handling precautions, price, volume, capacity, etc. of the product. In the present disclosure, the surface (e.g., label) is only an example of an area on the surface of the object 100. For example, those texts, images, logos and other text / visual elements can be printed, engraved or etched on the surface of the object 100 without using a label. For example, embodiments of the present disclosure can be applicable to any text, image, logo and other text / visual elements on the surface of the object 100.

[0050] In the present disclosure, the electronic device 2000 may identify an area corresponding to at least one surface (e.g., a label) included in the object 100 as the ROI 120, and may obtain information related to the object 100 from the area corresponding to the at least one surface (e.g., the label). When the object 100 has a 3D shape, the shape of the surface (e.g., the label) of the object 100 may be distorted in the two-dimensional (2D) image 110. Therefore, the accuracy of the information (e.g., a logo, an icon, or a text) obtained by the electronic device 200 from the surface (e.g., the label) of the object 100 may be deteriorated. In order to extract accurate information from the ROI 120 (e.g., at least one surface, the label), the electronic device 2000 according to an embodiment of the present disclosure may obtain an undistorted image 130 by using the image 110 of the object 100. The undistorted image 130 refers to an image in which the distortion of the ROI 120 of the object 100 is reduced and / or removed. For example, the undistorted image 130 may be a flattened image obtained by reducing or removing the bending distortion of the surface (e.g., the label) area. In this disclosure, the undistorted image 130 may also be referred to as a flat surface (eg, label) image.

[0051] The electronic device 2000 according to an embodiment of the present disclosure may estimate 3D information of the object 100 so as to generate an undistorted image 130. The electronic device 2000 obtains the undistorted image 130 by transforming the ROI 120 into a plane based on the 3D information of the object 100. The 3D information of the object 100 may include 3D parameters related to the 3D shape of the object 100 or 3D parameters related to a camera photographing the object. The 3D shape may include, but is not limited to, a sphere, a cube, a cylinder, and the like.

[0052] In the present disclosure, 3D parameters refer to elements representing geometric characteristics related to the 3D shape of the object 100. The 3D parameters may include, for example, height and radius information (or horizontal and vertical information) of the object 100, translation and rotation information for 3D geometric transformation on the 3D space of the object 100, and focal length information of the camera of the electronic device 2000 that has photographed the object 100, but the embodiments of the present disclosure are not limited thereto. The 3D parameters are variables, and the 3D shape may also change as the value of any one of the 3D parameters changes. The 3D parameter elements may be collected to form a 3D parameter set. The information capable of representing the 3D shape of the object 100 determined according to the 3D parameter set is referred to as "3D information" in the present disclosure.

[0053] In the present disclosure, “3D information of the object 100” refers to a set of 3D parameter values ​​(e.g., horizontal value, vertical value, height value, and radius value) representing the 3D shape of the object 100 included in the image 110. The 3D information of the object 100 does not necessarily include 3D parameters representing values ​​such as the absolute width, height, height, radius, etc. of the object 100, and may be composed of 3D parameters representing relative values ​​representing the 3D ratio of the object 100. In other words, when the 3D information of the object 100 is present, the electronic device 2000 may render the object 100 in a 3D shape having the same ratio as the object 100.

[0054] In order to perform an image processing operation of removing distortion of the ROI 120, the electronic device 2000 according to an embodiment of the present disclosure may identify the ROI 120 from the image 110 including at least a portion of the object 100, identify the 3D shape type of the object 100, and estimate 3D information of the object 100 based on the ROI 120 of the object 100 and the 3D shape type of the object 100. The electronic device 2000 may create an undistorted image 130 based on the 3D information of the object 100.

[0055] According to an embodiment of the present disclosure, the electronic device 2000 may extract the object information 140 from the undistorted image 130 , and may provide the undistorted image 130 and / or the object information 140 extracted from the undistorted image 130 to a user.

[0056] Detailed operations of removing distortion of the ROI 120 or extracting information from the undistorted image 130 through image processing operations performed by the electronic device 2000 will now be described in more detail with reference to the following drawings.

[0057] Figure 2 is a flowchart of a method of processing an image performed by an electronic device according to an embodiment of the present disclosure.

[0058] In operation S210, the electronic device 2000 according to an embodiment of the present disclosure obtains a first image of an object including at least one surface (e.g., a label) by using a first camera. The electronic device 2000 can activate the first camera by a user's manipulation. For example, the user can activate the camera of the electronic device 2000 to shoot an object in order to obtain information about the object. The user can activate the camera by touching a hardware button or icon for executing the camera, or can activate the camera by a voice command (e.g., turning on the Hi-Bixby camera and displaying surface (e.g., label) information by capturing a Hi-Bixby picture).

[0059] According to an embodiment of the present invention, the first camera may be one of a telephoto camera, a wide-angle camera, and an ultra-wide-angle camera, and the first image may be one of an image captured by the telephoto camera, an image captured by the wide-angle camera, and an image captured by the ultra-wide-angle camera.

[0060] According to an embodiment of the present disclosure, the electronic device 2000 may include one or more cameras. For example, the electronic device 2000 may include a multi-camera consisting of a first camera and a second camera. When the electronic device 2000 includes multiple cameras, the multiple cameras may have different specifications. For example, the multiple cameras may include a telephoto camera, a wide-angle camera, and an ultra-wide-angle camera with different focal lengths and different viewing angles.

[0061] However, the type of camera included in the electronic device 2000 is not limited to the above examples. When the electronic device 2000 includes a plurality of cameras, the first image may be an image obtained by synthesizing images obtained via a plurality of cameras. The first image may be a preview image captured and stored to be displayed on the screen of the electronic device 2000, an image that has been captured and stored in the electronic device 2000, or an image obtained from outside the electronic device 2000. The first image may be an image obtained by photographing a portion of an object including at least one surface (e.g., a label), or may be an image obtained by photographing the entire object. According to an embodiment of the present disclosure, the first image may be a panoramic image continuously captured by the first camera.

[0062] In operation S220, the electronic device 2000 according to an embodiment of the present disclosure identifies an area corresponding to at least one surface (e.g., a label) in the first image as an ROI by applying the first image to a first artificial intelligence (AI) model. For example, when the first image is obtained by the first camera, the electronic device 2000 may apply the first image to the first AI model. At this time, the first AI model may infer the ROI within the first image and output data related to the ROI. In the present disclosure, applying the first image to the first AI model may include not only applying the entire first image itself to the first AI model, but also preprocessing the first image and applying the result of the preprocessing to the first AI model.

[0063] For example, the electronic device 2000 may apply to the first AI model a cropped image obtained by cropping a partial area from the first image, an image obtained by adjusting the size of the first image, or an image obtained by cropping a portion of the first image and adjusting the size of a portion of the first image.

[0064] In the present disclosure, the first AI model may be referred to as a ROI recognition model. The ROI recognition model may be an AI model trained to receive an image and output data related to the ROI of an object in the image. For example, the ROI recognition model may be an AI model trained to infer an area corresponding to a surface (e.g., a label) in an image as an ROI. According to some embodiments of the present disclosure, the electronic device 2000 may recognize an ROI on the surface of an object (e.g., a label attached to a product) by using the ROI recognition model. According to some embodiments of the present invention, the electronic device 2000 may recognize key points (referred to as first key points in the present invention) representing the ROI of an object by using the ROI recognition model. For example, the first AI model may output information about key points (or coordinate values) indicating the edges of at least one surface (e.g., a label) in the first image. Reference will be made to Figure 5 The operations performed by the first AI model to estimate the ROI in the first image are described in more detail.

[0065] In the present disclosure, the surface (eg, label) region is exemplified as the ROI of the object, but the ROI is not limited thereto. Other regions of information to be extracted from the object may be set as ROIs by the electronic device 2000, and the embodiments of the present disclosure may be applied in the same / similar manner.

[0066] In operation S230, the electronic device 2000 according to an embodiment of the present disclosure obtains data related to the 3D shape type of the object by applying the first image to the second AI model. For example, when the first image is obtained through the first camera, the electronic device 2000 may apply the first image to the second AI model. At this time, the second AI model may infer the 3D shape type of the object within the first image, and may output data related to the 3D shape type of the object. In the present disclosure, the second AI model may be referred to as an object 3D shape recognition model. The object 3D shape recognition model may be an AI model trained to receive an image and output data related to the 3D shape type of the object in the image. For example, the object 3D shape recognition model may be an AI model trained to infer the 3D shape type of the object in the image. According to some embodiments of the present disclosure, the electronic device 2000 may identify the 3D shape type (e.g., sphere, cube, cylinder, etc.) of the object included in the first image by using the object 3D shape recognition model. This will be referred to later. Figure 4 An operation of recognizing a 3D shape type of an object by using the object 3D shape recognition model, performed by the electronic device 2000, is described.

[0067] When the object in the image has a 3D shape, the ROI attached to the surface of the 3D object in the 2D image may be distorted, so the recognition accuracy of the information in the ROI (e.g., logo, icon, text, etc.) may be reduced. For example, when the object is a cylindrical type product, because the label of the product attached to the cylindrical surface is attached to the curved surface of the object, the label of the product as the ROI is distorted in the image of the cylindrical type product. The electronic device 2000 according to an embodiment of the present disclosure can recognize the 3D shape of the object, and can use data about the 3D shape type of the recognized object to remove the distortion of the ROI. In the present disclosure, the cylindrical type product is only an example of an object. In the present disclosure, the object can be any product or material with a non-flat surface. Therefore, the curved surface is only an example of a non-flat surface discussed in the present disclosure.

[0068] According to an embodiment of the present disclosure, an operation S220 of identifying an area corresponding to at least one surface (e.g., a label) in the first image as an ROI by applying the first image to the first AI model, and an operation S230 of obtaining data about a 3D shape type of an object included in the first image by applying the first image to the second model can be performed in parallel. For example, when the first image is obtained through the first camera, the electronic device 2000 can input the first image into each of the first AI model and the second AI model. At this time, the operation of inferring the area corresponding to at least one surface (e.g., a label) in the first image as an ROI performed by the first AI model, and the operation of inferring the 3D shape type of the object included in the first image performed by the second AI model can be performed in parallel.

[0069] According to an embodiment of the present disclosure, any one of operations S220 and S230 may be performed first. For example, the electronic device 2000 may first input the first image to the first AI model to check the result of inferring the ROI through the first AI model, and then may input the first image to the second AI model. On the other hand, the electronic device 2000 may first input the first image to the first AI model to check the result of inferring the 3D shape type of the object included in the first image through the second AI model, and then may input the first image to the first AI model.

[0070] In operation S240, the electronic device 2000 according to an embodiment of the present disclosure obtains a set of 3D parameter values ​​related to at least one of the object, the at least one surface (e.g., the label), or the first camera based on an area corresponding to the at least one surface (e.g., the label) identified as the ROI and data related to the 3D shape type of the object. According to some embodiments of the present disclosure, the elements of the 3D parameter may include width, length, height, and radius information related to the 3D shape of the object.

[0071] According to some embodiments of the present disclosure, the elements of the 3D parameters may include translation and rotation information for 3D geometric transformation on the 3D space of the object. The translation and rotation information may be information indicating the position and angle at which the camera of the electronic device 2000 views and photographs the object.

[0072] According to some embodiments of the present disclosure, the elements of the 3D parameters may include focal length information of the camera of the electronic device 2000 that has photographed the object. However, the 3D parameters are not limited to the above examples, and the electronic device 2000 may also include other information for identifying the 3D geometric characteristics of the object and removing distortion of the ROI.

[0073] According to an embodiment of the present disclosure, the 3D parameters are determined to correspond to the 3D shape of the object. In other words, the elements of the 3D parameters corresponding to each type of 3D shape (hereinafter referred to as 3D shape type) may be different.

[0074] For example, when the 3D shape is a cylindrical type, the 3D parameter corresponding to the cylindrical type may include a radius, but when the 3D shape is a cube type, the 3D parameter corresponding to the cube type may not include a radius. The 3D parameter corresponding to the 3D shape type of the object obtained in operation S230 may be set to an initial value for obtaining accurate 3D information of the object. The electronic device 2000 may obtain the 3D parameter representing the 3D information of the object by finely adjusting the parameter value so that the 3D parameter having the initial value represents the 3D information of the object.

[0075] According to an embodiment of the present disclosure, when the 3D shape type of the object is a cylinder (or a bottle), the elements of the 3D parameters may include, but are not limited to, width, length, height, and radius information of the object, translation and rotation information about the 3D space of the object, and focal length information of a camera of the electronic device 2000 that has photographed the object. As described above, when the 3D shape type of the object is a cuboid, the elements of the 3D parameters corresponding to the cuboid type may be different from the elements of the 3D parameters corresponding to the cylinder type.

[0076] According to an embodiment of the present disclosure, the electronic device 2000 may obtain 3D information representing a curved shape of at least one level. The electronic device 2000 finely adjusts the initial value of the 3D parameter to approximate or match the correct value of the 3D parameter of the object, so that the adjusted final value of the 3D parameter represents the 3D information of the object. Continuing to describe the case where the 3D shape type is a cylinder (or bottle) as the above example, the electronic device 2000 may adjust the width, length, height, and radius of the object in the value of the 3D parameter to indicate the relative percentage or absolute value of the width, length, and height of the object.

[0077] The electronic device 2000 may also adjust the translation and rotation values ​​among the values ​​of the 3D parameters to become values ​​representing the translation and rotation degrees of the object in the 3D space. The electronic device 2000 may also adjust the focal length value among the values ​​of the 3D parameters to represent the focal length of the camera of the electronic device 2000 that has photographed the object.

[0078] According to an embodiment of the present disclosure, the electronic device 2000 may set an arbitrary virtual object to estimate the 3D information of the object. The virtual object may be an object having the same shape type as the 3D shape type of the object identified in operation S230 and capable of being rendered using 3D parameters having initial parameter values. The electronic device 2000 may project the 3D virtual object in 2D and may set a key point (referred to as a second key point in the present disclosure) of the 3D virtual object.

[0079] The electronic device 2000 may finely adjust the 3D parameter value so that the key point of the virtual object matches the key point (first key point) of the object obtained in operation S220. When the fine adjustment of the 3D parameter is repeatedly performed, the final value of the 3D parameter is determined, and when the final value of the 3D parameter represents the 3D information of the object, the second key point obtained from the virtual object is matched with the first key point of the object. Fig. 6A An operation performed by the electronic device 2000 to change the value of the 3D parameter through fine adjustment to indicate the 3D information of the object is further described.

[0080] The electronic device 2000 obtaining the 3D parameter value described in operation S240 refers to obtaining the final value of the 3D parameter obtained through the above adjustment.

[0081] In operation S250 , the electronic device 2000 according to an embodiment of the present disclosure estimates a non-flat shape (eg, a curved shape) of at least one surface (eg, a tag) based on a 3D parameter value.

[0082] The 3D parameters whose values ​​have been adjusted by the above operation indicate 3D information of the object within the image (e.g., the width, length, height, and radius of the object, and the curvature (angle) of the surface or a label attached to the surface of the object). The electronic device 2000 can generate a 2D mesh representing a surface (e.g., a label) that is an ROI on the surface of the object by using the 3D parameters. The 2D mesh data is the result of projecting the coordinates of the surface (e.g., a label) on the 3D space in a 2D manner by using the 3D parameter values, and can refer to the surface (e.g., label) distortion information in the first image.

[0083] In operation S260, the electronic device 2000 according to an embodiment of the present disclosure obtains a flat surface (e.g., label) image in which a non-flat shape (e.g., a curved shape) of at least one surface (e.g., label) has been flattened by performing a perspective transformation on at least one surface (e.g., label).

[0084] The electronic device 2000 can transform a non-flat shape (e.g., a curved shape) of a surface (e.g., a label) into a flat shape through perspective transformation. Since the image of the flattened surface (e.g., a label) is an image in which distortion during shooting due to the 3D shape of the object has been removed and / or reduced, the image of the flattened surface (e.g., a label) may be referred to as a distortion-free image or a flat surface (e.g., a label) image in the present disclosure.

[0085] A distortion removal model may be used in operations S240 to S260. The distortion removal model may be an AI model trained to output a distortion-free image by receiving information of an ROI in an object and 3D parameter values ​​associated with the object. The information of the ROI may include an image of the ROI and coordinates of key points of the ROI. For example, the distortion removal model may obtain a flat label image including a flattened label by receiving an image including a label attached to a surface of a 3D object including a curved surface and captured when curved.

[0086] According to an embodiment of the present disclosure, the electronic device 2000 can obtain information related to an object from a flat surface (e.g., a label) image. The electronic device 2000 can identify a logo, icon, text, etc. within the ROI by using an information detection model for extracting information within the ROI. The information detection model can be stored in a memory of the electronic device 2000, or can be stored in an external server.

[0087] Through the above operations, the electronic device 2000 can infer 3D information of an object in an image, and can remove distortion from the ROI by performing accurate perspective transformation using the inferred 3D information of the object, thereby extracting information within the ROI with improved accuracy. Figure 7 An operation performed by the electronic device 2000 to obtain information related to an object from a flat surface (eg, label) image by using an information detection model is described.

[0088] Now refer to Figure 3 The operation of obtaining a flat surface (e.g., label) image in which distortion has been removed from a first image including geometric distortion by using a first AI model (ROI recognition model) and a second AI model (object 3D shape recognition model) performed by the electronic device 2000 is described in more detail.

[0089] Figure 3 is a diagram for explaining an operation of processing an image performed by an electronic device according to an embodiment of the present disclosure.

[0090] Reference Figure 3According to an embodiment of the present disclosure, the electronic device 2000 may obtain an image of the object 300, hereinafter referred to as the object image 304. The object 300 may include at least one tag.

[0091] According to an embodiment of the present disclosure, the electronic device 2000 may obtain an image of the object 300 by photographing the object 300 using a user's camera. Alternatively, the electronic device 2000 may receive an already captured image of the object 300 from another electronic device (eg, a server or another user's electronic device).

[0092] According to an embodiment of the present disclosure, the electronic device 2000 may identify the ROI 312 by using the ROI recognition model 310. The ROI recognition model 310 may be an AI model trained to receive an image and output data related to the ROI 312 of the object 300 in the image. The data related to the ROI 312 may be, for example, key points and / or coordinates of the ROI 312, but the embodiments of the present disclosure are not limited thereto. For ease of description, the data related to the ROI 312 will now be referred to as the ROI 312. Figure 3 In the example of FIG. 3 , the ROI 312 is a label attached to the surface of the object 300. However, the type of the ROI 312 is not limited thereto.

[0093] According to an embodiment of the present disclosure, the electronic device 2000 may use the object image 304 as input data of the ROI recognition model 310. The electronic device 2000 may process the ROI 312 by applying a certain preprocessing algorithm to the object image 304 so that the ROI 312 is suitable for being recognized. For example, the electronic device 2000 may use the cropped object image 302 as input data of the ROI recognition model 310, and the cropped object image 302 is obtained by cropping out a part of the object image 304 and adjusting the size of the cropped object image 304. In this case, the cropped area of ​​the object image 304 may be an area other than the ROI. At least a part of the object 300 may be included in the cropped object image 302, and the ROI 312 of the object 300 may be included in the cropped object image 302.

[0094] According to an embodiment of the present disclosure, the electronic device 2000 may recognize the 3D shape type 322 of the object by using the object 3D shape recognition model 320. The object 3D shape recognition model 320 may be an AI model trained to receive an image and output data related to the 3D shape type 322 of the object 300 in the image. Figure 3The 3D shape type 322 is shown to be a cylinder, but the embodiments of the present disclosure are not limited thereto. For example, the 3D shape type 322 may be a sphere, a cube, etc. For ease of description, data related to the 3D shape type 322 will now be referred to as the 3D shape type 322.

[0095] The electronic device 2000 may obtain an initial value of the 3D parameter 324 based on the 3D shape type 322. The 3D parameter 324 may be determined based on the 3D shape type 322. For example, when the 3D shape type 322 is a cylinder type, elements of the 3D parameter 324 corresponding to the cylinder type may include at least one of a height, a radius, an angle of an ROI on an object surface, a translation coordinate and a rotation coordinate on a 3D space, or a focal length of a camera.

[0096] According to an embodiment of the present disclosure, the electronic device 2000 can obtain the undistorted image 332 by using the distortion removal model 330. The distortion removal model 330 may be an AI model trained to receive the ROI 312, the 3D parameters 324, and the object image 304 (or the cropped object image 302) and output the undistorted image 332. Figure 3 In the example of , since ROI 312 is a label and object 300 is a bottle, undistorted image 332 may be a flat label image in which distortion of the label attached to the surface of the bottle has been removed. However, undistorted image 332 is not limited to a flat label image. Undistorted image 332 may include all types of images obtainable according to the type of ROI 312 and 3D shape type 322.

[0097] According to an embodiment of the present disclosure, the distortion removal model 330 may adjust the initial value of the 3D parameter 324 so that the final value of the 3D parameter 324 represents the 3D information of the object 300. For example, the distortion removal model 330 may obtain relative values ​​or absolute values ​​such as the width, length, height, and radius of the object 300 and the curvature (angle) of the label attached to the surface of the object 300. The distortion removal model 330 may create an undistorted image 332 based on the final value of the 3D parameter 324 representing the 3D information of the object 300.

[0098] For example, the distortion removal model 330 may obtain a flat label image from which label distortion has been removed as the undistorted image 332 by transforming the curvature of the label attached to the surface of the (curved) object 300 to be flattened based on the final value of the 3D parameter 324 .

[0099] According to an embodiment of the present disclosure, the electronic device 2000 may replace the operation of the distortion removal model 330 with a series of data processing / calculations. The electronic device 2000 may obtain the distortion-free image 332 by performing a series of data processing / calculations without using the distortion removal model 330. For example, the electronic device 2000 may set an arbitrary virtual object to estimate the 3D information of the object. The arbitrary virtual object may be created based on the initial value of the 3D parameter 324. The electronic device 2000 may set an arbitrary ROI from an arbitrary virtual object and adjust the value of the 3D parameter so that the arbitrary ROI of the arbitrary virtual object matches the ROI 312 of the object 300, thereby obtaining the final value of the 3D parameter 324. The electronic device 2000 may create the distortion-free image 332 based on the final value of the 3D parameter 324.

[0100] Will refer to it later Fig. 6A An operation of setting an arbitrary virtual object to estimate 3D information of the object, performed by the electronic device 2000, is described in more detail.

[0101] Figure 4 is a diagram for explaining an operation of recognizing a 3D shape of an object performed by an electronic device according to an embodiment of the present disclosure.

[0102] According to an embodiment of the present disclosure, the electronic device 2000 can recognize the 3D shape type 420 of the object by using the object 3D shape recognition model 410. The electronic device 2000 can recognize the 3D shape type 420 of the object by a neural network operation of the object 3D shape recognition model 410 for receiving the image 400 of the object and extracting features.

[0103] The object 3D shape recognition model 410 may be trained based on a training data set consisting of various images including 3D objects. The 3D shape type 420 of the object may be labeled on the object image of the training data set of the object 3D shape recognition model 410. The 3D shape type 420 of the object may include, for example, a sphere, a cube, a pyramid, a cone, a truncated cone, a hemisphere, and a cuboid, but embodiments of the present disclosure are not limited thereto.

[0104] According to an embodiment of the present disclosure, the electronic device 2000 may obtain a 3D parameter 430 corresponding to the identified 3D shape type 420 of the object based on the identified 3D shape type 420. The 3D parameter 430 refers to an element representing geometric characteristics related to the 3D shape of the object.

[0105] For example, when the 3D shape type 420 is a "sphere", the 3D parameters 430 of the "sphere" shape are obtained, and when the 3D shape type 420 is a "cube", the 3D parameters 430 of the "cube" shape may be obtained. Elements constituting the 3D parameters 430 may be different for different 3D shape types 420. For example, the 3D parameters 430 of the "sphere" shape may include elements such as radius and / or diameter, and the 3D parameters 430 of the "cube" shape may include elements such as width, length, and height.

[0106] For ease of explanation, Figure 4 The 3D parameters 430 shown in FIG. 4 only include elements such as width, length, radius, and depth, which are geometric features, but the 3D parameters 430 are not limited thereto. The 3D parameters 430 may also include rotation coordinate information of an object in space, translation coordinate information of an object in space, focal length information of a camera that has photographed an object, and 3D information about an ROI of the object (e.g., width, length, and curvature of the ROI). In other words, the 3D parameters 430 are only examples to help visual understanding, and the 3D parameters 430 may also include any type of elements that can be used to estimate 3D information of an object in an image in addition to the aforementioned examples, and some elements may be excluded from the aforementioned examples.

[0107] For example, the electronic device 2000 according to an embodiment of the present disclosure applies the image 400 to the object 3D shape recognition model 410 to recognize the cylinder type 422, which is the 3D shape type 420 of the object in the image 400. The electronic device 2000 may obtain an initial value of a 3D parameter 432 of a cylindrical shape corresponding to the cylinder type 422. The 3D parameter 432 of the cylindrical shape may include, for example, a diameter D of the cylinder, a radius r of the cylinder, rotation information R of the cylinder in the 3D space, translation information T of the cylinder in the 3D space, a height h of the cylinder, a height h' of the ROI on the surface of the cylinder, an angle θ at which the ROI (e.g., a label of a product) is positioned on the surface of the cylinder, and focal length information F of the camera, but the embodiments of the present disclosure are not limited thereto.

[0108] According to an embodiment of the present disclosure, each element included in the 3D parameter 430 may have a set initial value representing the 3D information of an arbitrary object. The electronic device 2000 according to an embodiment of the present disclosure may be matched so that the 3D parameter 430 represents the 3D information of the object. For example, the electronic device 2000 may adjust the value of the 3D parameter 432 of the cylindrical shape so that the value of the 3D parameter 432 of the cylindrical shape represents the 3D information of the object in the image 400. In other words, the electronic device 2000 may obtain the value of the 3D parameter 430 representing the 3D information of the object in the image 400. This will be Fig. 6A is further described in the description of .

[0109] The drawings of the present disclosure show that the object in the image 400 is “wine” and the ROI is “wine label”, but the present disclosure is not limited thereto.

[0110] For example, in the present disclosure, the 3D shape type 420 of a wine bottle is recognized as a cylinder type 422. However, the wine bottle may be recognized as a bottle type according to the training and adjustment of the object 3D shape recognition model 410, and the correspondingly obtained 3D parameters may be 3D parameters corresponding to the bottle type.

[0111] For another example, the object in the image may be an object such as a sphere, a cone, or a cuboid, which is another type of 3D shape. In this case, the electronic device 2000 may identify the 3D shape type 420 of each object and may obtain 3D parameters 430.

[0112] As another example, the ROI in the image may be an area representing information related to a product (object), such as the raw materials of the product, how to use the product, and how much to use the product, rather than a label of the product. In this case, the electronic device 2000 may perform a distortion removal operation according to an embodiment of the present disclosure to accurately identify information included in the ROI of the object, and may obtain object-related information from the undistorted image.

[0113] Figure 5 is a diagram for explaining an operation of recognizing a ROI on a surface of an object, performed by an electronic device according to an embodiment of the present disclosure.

[0114] According to an embodiment of the present disclosure, the electronic device 2000 may recognize the ROI 520 by using the ROI recognition model 510. The electronic device 2000 may recognize the ROI 520 through a neural network operation of the ROI recognition model 510 for receiving the object image 500 and extracting features.

[0115] According to an embodiment of the present disclosure, the electronic device 2000 may pre-process the object image 500 to be input to the ROI recognition model 510. The electronic device 2000 may use an input image 502 obtained by cropping a portion of the object image 500 and adjusting the size of the cropped object image 500 as input data of the ROI recognition model 510. According to an embodiment of the present disclosure, the electronic device 2000 may obtain an image to be input to the ROI recognition model 510 by using another camera.

[0116] For example, when the user photographs the object, the electronic device 2000 can obtain a high-resolution image of the ROI by using another high-resolution camera. In this case, the image captured by the user can have the same format as the object image 500, and the image stored separately by the electronic device 2000 to identify the ROI can have the same format as the input image 502.

[0117] The ROI recognition model 510 may be trained based on a training data set consisting of various images including the ROI. Key points representing the ROI may be marked on the ROI image of the training data set of the ROI recognition model 510. The ROI 520 recognized by the electronic device 2000 by using the ROI recognition model 510 may include, but is not limited to, an image on which the detected ROI 520 is displayed, key points representing the ROI 520, and / or coordinates of the key points in the image.

[0118] The ROI recognition model 510 may include a backbone network and a regression module. The backbone network may use a known neural network (e.g., a convolutional neural network (CNN)) algorithm to extract various features from the input image 502. For example, the backbone network may be a pre-trained network model, and may be changed to another type of neural network to improve the performance of the ROI recognition model 510. The regression module performs the task of detecting the ROI 520. For example, the regression module may include a regression algorithm for performing learning so that the bounding box, key points, etc. representing the ROI converge to the correct answer value. The regression module may include a neural network layer and weights for detecting the ROI 520, for example, the regression module may be configured with a region having a convolutional neural network (R-CNN) feature for detecting the ROI, but the embodiments of the present disclosure are not limited thereto. The electronic device 2000 may train the layers of the regression module by using the training data set of the ROI recognition model 510.

[0119] Fig. 6A is a diagram for explaining an operation of obtaining 3D information of an object, performed by an electronic device according to an embodiment of the present disclosure.

[0120] When describing Fig. 6A When describing the 3D shape type of an object, for ease of description, a case where the 3D shape type of the object is identified as a cylinder is taken as an example. However, the 3D shape type of the object is not limited to a cylinder, and can be applied to any 3D shape type in which geometric features can be represented as 3D parameters, including the above examples.

[0121] The electronic device 2000 according to an embodiment of the present disclosure may perform operations to be described later to obtain 3D information of an object. Because the electronic device 2000 performs perspective transformation based on the 3D information of the object, the electronic device 2000 may remove distortion in the image more accurately than when perspective transformation is usually performed without the 3D information of the object. The distortion in the image may include distortion of the ROI due to the curved surface of the surface of the 3D object, etc. For example, a label attached to the surface of an object may be shown as distorted in a 2D image due to the curved surface of the 3D shape of the object, but the embodiments of the present disclosure are not limited thereto.

[0122] According to an embodiment of the present disclosure, the electronic device 2000 may obtain a 3D parameter 610 corresponding to a cylinder, which is the identified 3D shape type, from 3D parameters corresponding to various pre-stored 3D shape types (e.g., cylinders, spheres, and cubes). The 3D parameter 610 corresponding to the cylinder type may include, for example, the radius r of the cylinder, the rotation information R of the cylinder in 3D space, the translation information T of the cylinder in 3D space, the height h of the ROI, the angle θ at which the ROI (e.g., the label of the product) is positioned on the surface of the cylinder, and the focal length information F of the camera, but the embodiment of the present disclosure is not limited thereto. Each element included in the 3D parameter 610 may have a set initial value.

[0123] According to an embodiment of the present disclosure, the electronic device 2000 may set a virtual object 620 to estimate 3D information of an object within an image. The virtual object 620 may be an object that is set to the same shape type as the 3D shape type of the object in the image and rendered as the initial value of the 3D parameter 610. In other words, in Fig. 6A In the example of FIG. 6 , the virtual object 620 is of a cylindrical type and is an object using the initial values ​​(r, R, T, h, θ, and F) of the 3D parameters 610 as 3D information. The virtual object 620 may include an initial ROI 622 arbitrarily set for the virtual object.

[0124] The electronic device 2000 may finely adjust the value of the 3D parameter 610 so that the value of the 3D parameter 610 representing the 3D information of the virtual object 620 represents the 3D information of the object in the image.

[0125] The electronic device 2000 may project the virtual object 620 in two dimensions, and may set a key point 630 (also referred to as a second key point 630) indicating an ROI (e.g., a label) of the virtual object 620. The electronic device 2000 may finely adjust the value of the 3D parameter 610 so that the second key point 630 matches the key point 640 (also referred to as a first key point 640) indicating the ROI of the object in the image. Since the operation of obtaining the first key point 640 indicating the ROI of the object in the image performed by the electronic device 2000 has been described above, its redundant description will be omitted.

[0126] The electronic device 2000 may adjust the second key point 630 based on the loss function to match the first key point 640. The function f may be a function including the initial values ​​r, R, T, h, θ and f of the 3D parameter 610 of the cylinder as variables. The electronic device 2000 may estimate the second key point 630 of the virtual object 620 using the function f, and may adjust the second key point 630 by using the loss function to minimize the difference between the second key point 630 and the first key point 640. The electronic device 2000 may change the value of the 3D parameter 610 so that the second key point 630 matches the first key point 640. The electronic device 2000 may recreate (update) the virtual object 620 based on the changed value of the 3D parameter 610, and may repeat the above operation.

[0127] In other words, by repeatedly adjusting the value of the 3D parameter 610 and creating a virtual object of 3D information having the adjusted value of the 3D parameter 610, the electronic device 2000 can obtain the value of the 3D parameter 610 by which the difference between the second key point 630 obtained by projecting the virtual object 620 in two dimensions and the first key point 640 indicating the ROI of the object in the image is minimized. When the above adjustment is repeated, the initial value of the 3D parameter 610 set for the virtual object 620 can be adjusted to approximate the correct value of the 3D parameter 610 of the object. When the second key point 630 matches the first key point 640, in this case, the value of the 3D parameter 610 corresponding to the virtual object 620 represents the 3D information of the object in the image. The electronic device 2000 can finally obtain the 3D parameter 610 representing the 3D information of the object in the image.

[0128] Figure 6B is a diagram for explaining an operation of removing distortion of a ROI based on 3D information of an object, performed by an electronic device according to an embodiment of the present disclosure.

[0129] exist Figure 6B In the description, the above reference will be continued. Fig. 6A Content described as an example. Figure 6BThe electronic device 2000 according to an embodiment of the present disclosure may obtain the value of the 3D parameter 610 value representing the 3D information of the object in the image through a process of finely adjusting the value of the 3D parameter 610 .

[0130] The electronic device 2000 can create 2D mesh data 650 representing the ROI on the surface of the object within the image by using the value of the 3D parameter 610. The 2D mesh data 650 refers to data created by projecting the coordinates of the ROI of the object in two dimensions on the 3D space based on the obtained value of the 3D parameter 610, and includes distortion information of the ROI of the object.

[0131] For example, the ROI attached to the surface of a "wine bottle" which is a 3D object having a curved shape may be a "wine label." In this case, the 2D mesh data 650 is the result of 2D projection of coordinates on the 3D space of the wine label attached to the surface of the wine bottle, which is the ROI within the image including the wine bottle, and may represent distortion information of the wine label.

[0132] The electronic device 2000 may convert the 2D mesh data 650 that has reflected the bending distortion into flat data 660. In this case, various algorithms for data conversion may be applied. For example, the electronic device 2000 may use, but is not limited to, a perspective transformation algorithm.

[0133] The electronic device 2000 according to an embodiment of the present disclosure may obtain an undistorted image 670 corresponding to the flat data 660 by creating the flat data 660. For example, the undistorted image 670 may be, but is not limited to, an image in which a wine label having a curved shape and attached to a curved surface of a wine bottle is flattened. According to some embodiments of the present disclosure, the electronic device 2000 may perform inter-pixel interpolation when obtaining the undistorted image 670, thereby improving image quality.

[0134] The electronic device 2000 can extract information within the ROI by using the undistorted image 670 of the ROI. Since the undistorted image 670 is created based on the result of inferring accurate 3D information of the object, even when a general information detection model (e.g., an optical character recognition (OCR) model) for extracting information within an image is used, a logo, icon, text, etc. within the ROI can be more accurately detected.

[0135] In other words, even when the information detection model is not separately trained to extract information from the distorted image by reflecting the distortion in the image, accurate information extraction can be performed even by the general information detection model. However, the above-mentioned general information detection model is only an example, and the electronic device 2000 can also use a detection model trained by training data including distortion in a logo, icon, text, etc.

[0136] Figure 7 is a view for explaining an operation of extracting information in a ROI performed by an electronic device according to an embodiment of the present disclosure.

[0137] It will be explained under the following premise Figure 7 According to the above embodiment, the object is included in the image, at least a portion of the entire area of ​​the object is the ROI, and the electronic device 2000 obtains the undistorted image 700 of the ROI. In detail, the undistorted image 700 may be a flat label image from which distortion (e.g., distortion due to curvature) of the product label has been removed.

[0138] According to an embodiment of the present disclosure, the electronic device 2000 may extract intra-ROI information 720 from the undistorted image 700 of the ROI by using the information detection model 710. The intra-ROI information 720 may be information related to the object. For example, the electronic device 2000 may obtain the undistorted image 700 of the product label included in the object, and may obtain the intra-ROI information 720 related to the object included in the product label.

[0139] According to an embodiment of the present disclosure, since the information detection model 710 extracts information by using the undistorted image 700, a known detection model for information extraction can be used. For example, the information detection model 710 can be an OCR model. The electronic device 2000 can detect text within the ROI by using the OCR model. The OCR model can recognize general characters, special characters, symbols, etc.

[0140] However, the intra-ROI information 720 is not limited thereto, and various detection models for detecting logos, icons, images, etc. within the ROI may be used. Specifically, a logo detection model, an icon detection model, an image detection model, an object detection model, etc. may be included.

[0141] According to an embodiment of the present disclosure, the information detection model 710 may be an AI model trained based on the undistorted image 700. In order to ensure the accuracy of extracting information from the undistorted image 700 obtained according to the above embodiment, the electronic device 2000 may also train the information detection model 710 by including the undistorted image 700 and the intra-ROI information 720 in the training data set.

[0142] In this case, the electronic device 2000 can use a known detection model as a pre-trained model to train the information detection model 710, so that the ROI intra-information 720 is more accurately extracted. According to some embodiments of the present disclosure, the electronic device 2000 can use one or more information detection models 710. For example, the electronic device 2000 can independently display / provide information obtained from each of two or more information detection models 710, or can create new auxiliary information by combining and / or processing the information obtained from each of the two or more information detection models 710, and display / provide the created auxiliary information.

[0143] Fig. 8A 2 is a view for explaining a first example in which an electronic device obtains a distortion-free image by obtaining 3D information according to an embodiment of the present disclosure.

[0144] exist FIG. 8A to FIG. 8C , viewpoint is a term arbitrarily selected to indicate a direction and / or angle at which a camera of the electronic device 2000 views the object 800.

[0145] Reference Fig. 8A , the electronic device 2000 according to an embodiment of the present disclosure may identify a ROI 812 from an object image 810 obtained by photographing the object 800 at a first viewpoint, and may obtain an undistorted image 814 (eg, a flat label image).

[0146] According to an embodiment of the present disclosure, the first viewpoint may be a direction in which the camera of the electronic device 2000 views the object 800 from the front. In this case, even when the electronic device 2000 photographs the object 800 from the front, since the image obtained by photographing the object in a 3D shape is 2D, the surface of the object 800 or the label attached to the object 800 may have distortion due to the curved surface existing in the object 800.

[0147] The electronic device 2000 according to an embodiment of the present disclosure may crop ROI 812 from the object image 810 and may obtain an undistorted image 814 including ROI 812. The electronic device 2000 may use 3D information of the object 800 in order to obtain the undistorted image 814. The 3D information may consist of 3D parameter values ​​adjusted for the object 800.

[0148] For example, the 3D information may include the radius of the cylindrical-shaped object 800, the rotation coordinates of the object 800 in the 3D space, the translation coordinates of the object 800 in the 3D space, the angle at which the ROI 812 is positioned on the surface of the object 800 (i.e., the angle from the central axis of the cylindrical shape (which is the 3D shape of the object 800) to both ends of the ROI 812), and the focal length of the camera when the electronic device 2000 captures the object image 810.

[0149] Based on the 3D information, the electronic device 2000 may perform perspective transformation so that the ROI 812 may be expressed on a 2D plane without distortion. Since the detailed operation of the perspective transformation has been described above, a redundant description thereof will be omitted.

[0150] As the viewpoint at which the electronic device 2000 views the object 800 changes, the degree of distortion occurring in the ROI 812 may vary. The electronic device 2000 according to an embodiment of the present disclosure may perform robust distortion removal by utilizing 3D information regardless of the degree of distortion. Figure 8B and 8C Describe this in more detail.

[0151] Figure 8B 2 is a view for explaining a second example in which the electronic device obtains a distortion-free image by obtaining 3D information according to an embodiment of the present disclosure.

[0152] Reference Figure 8B , the electronic device 2000 according to an embodiment of the present disclosure may identify a ROI 822 from an object image 820 obtained by photographing the object 800 at a second viewpoint, and may obtain an undistorted image 826 (eg, a flat label image).

[0153] According to an embodiment of the present disclosure, the second viewpoint may be a direction in which the camera of the electronic device 2000 is tilted in a vertical upward direction to view the object 800. In this case, not only distortion due to the 3D shape of the object 800 but also distortion due to the viewpoint of the camera of the electronic device 2000 may exist in the ROI 822 included in the object image 820. The electronic device 2000 may obtain an undistorted image 826 from which the distortion due to the 3D shape of the object 800 and the distortion due to the viewpoint of the camera of the electronic device 2000 have been removed by using the 3D information of the object 800.

[0154] For example, the transformed image 824 is an image created by performing a perspective transformation on the ROI 822 to achieve flattening. Because a known perspective transformation algorithm can be used for the perspective transformation, its detailed description will be omitted. Referring to the transformed image 824, even when the ROI 822 is transformed to be flat, the distortion caused by the 3D shape of the object 800 and / or the distortions 824-1 and 824-2 caused by the viewpoint of the camera may remain. ( Figure 8B The distortions 824-1 and 824-2 in the figure exemplarily represent the distortion that the character is bent into a curve compared to the reference straight line.

[0155] According to an embodiment of the present disclosure, the 3D information may be composed of 3D parameter values ​​adjusted to represent the object 800. For example, the 3D information may include the radius of the object 800, the rotation coordinates of the object 800 on the 3D space, the translation coordinates of the object 800 on the 3D space, the angle at which the ROI 812 is positioned on the surface of the object 800 (i.e., the angle from the central axis of the cylindrical shape that is the 3D shape of the object 800 to both ends of the ROI 812), and the focal length of the camera when the electronic device 2000 captures the object image 820. The electronic device 2000 according to an embodiment of the present disclosure may obtain a distortion-free image 826 in which distortion due to the 3D shape of the object 800 and distortion due to the shooting viewpoint of the camera of the electronic device 2000 are eliminated by accurately performing perspective transformation using the 3D information.

[0156] Figure 8C is a view for explaining a third example in which the electronic device obtains a distortion-free image by obtaining 3D information according to an embodiment of the present disclosure.

[0157] Reference Figure 8C , the electronic device 2000 according to an embodiment of the present disclosure may identify a ROI 832 from an object image 830 obtained by photographing the object 800 at a third viewpoint, and may obtain an undistorted image 836 (eg, a flat label image).

[0158] According to an embodiment of the present disclosure, the third viewpoint may be a direction in which the camera of the electronic device 2000 is tilted in a vertical downward direction to view the object 800. In this case, not only distortion due to the 3D shape of the object 800 but also distortion due to the viewpoint of the camera of the electronic device 2000 may exist in the ROI 832 included in the image of the object 800.

[0159] For example, the transformed image 834 is an image created by performing a perspective transformation on the ROI 832 to achieve flattening. Referring to the transformed image 834, even when the ROI 832 is transformed to be flat, distortion due to the 3D shape of the object 800 and / or distortions 834-1 and 834-2 due to the viewpoint of the camera may remain. Figure 8C The distortions 834-1 and 834-2 in FIG. 8 exemplarily represent the distortion that the character is bent into a curve compared to the reference straight line. )

[0160] By using the 3D information of the object 800, the electronic device 2000 can obtain an undistorted image 836 from which distortion has been accurately removed. Figure 8B Description has been made, so its redundant description will be omitted.

[0161] According to an embodiment of the present disclosure, the 3D parameters included in the 3D information may include rotation coordinates of the object 800 in the 3D space, translation coordinates of the object 800 in the 3D space, etc. Therefore, when creating the undistorted image 836, the electronic device 2000 may translate and rotate the ROI 832 and perform perspective transformation.

[0162] According to an embodiment of the present disclosure, the 3D parameters included in the 3D information may include the focal length of the camera when the electronic device 2000 captures the object image 830. Therefore, when creating the undistorted image 836, the electronic device 2000 may pre-process the image including the ROI 832 based on the focal length and may perform perspective transformation.

[0163] In other words, when creating the undistorted image 836, the electronic device 2000 removes distortion due to the 3D shape of the object 800 and / or distortion due to the viewpoint of the camera by using the 3D information. Therefore, the electronic device 2000 can perform robust distortion removal regardless of the degree of distortion of the ROI 832 within the image.

[0164] Fig.9A is a view for explaining a first example in which an electronic device extracts information from an undistorted image according to an embodiment of the present disclosure.

[0165] refer to Fig.9A , an original image 910 , a cropped image 920 , and an undistorted image 930 are shown.

[0166] According to an embodiment of the present disclosure, the electronic device 2000 can extract information present in the image by using an information detection model. When obtaining the undistorted image 930, the electronic device 2000 can detect the information within the ROI by using a general information detection model. In other words, even when the electronic device 2000 does not separately train a detection model to extract information from a distorted image by reflecting the distortion in the image, the electronic device 2000 can create the undistorted image 930 and apply the general detection model to the undistorted image 930. Therefore, the electronic device 2000 can save computing resources for separately training / updating the information detection model.

[0167] For example, the electronic device 2000 may detect text existing in the image by using an OCR model. Extracting text from an image by using an OCR model by the electronic device 2000 will now be described as an example.

[0168] According to an embodiment of the present disclosure, the original image 910 is an original image obtained by the electronic device 2000 by using a camera. The original image 910 may include distortion of the ROI due to the 3D shape of the object, and may also include blank space in the image other than the ROI. In other words, noise pixels outside the ROI may be included. When the electronic device 2000 applies OCR to the original image 910, due to the characteristics of the original image 910 described above, at least some text in the ROI may not be recognized or may be misrecognized. For example, in the original image 910, the detection area of ​​the text is outlined by a box, and the detected text that is misrecognized within the detection area in the area where the text is detected is indicated by a shaded arrow (in the case of misrecognition).

[0169] In addition, text that exists but is not recognized as a detection area is indicated by a black arrow (in the case of non-recognition). For example, when the number of text blocks to be detected in the ROI is 14, as a result of applying OCR to the original image 910 (i.e., referring to the text 911 detected from the original image 910), 8 text blocks may be detected, and at least some of the 8 text blocks may provide inaccurate text detection results.

[0170] To help with clearer understanding, the unrecognized and erroneous recognized cases exemplarily described in the present disclosure will be further described with reference to text 911 detected from the original image 910 , and exemplary results of extracting information from the cropped image 920 and the undistorted image 930 will be described.

[0171] According to an embodiment of the present disclosure, the OCR model may detect text in an image based on the fact that the confidence is equal to or greater than a threshold value (eg, 0.5), recognize the detected text, and output a recognition result.

[0172] In the examples of the present disclosure, the unrecognized case may mean that even by performing text detection and recognition on the image, no text detection and recognition results are output from the image. For example, the unrecognized case may include case 1) where no text is detected, and case 2) where no recognition result is output because the text is detected and text recognition has been performed but the confidence of the recognition result is less than a threshold value (e.g., 0.5).

[0173] In the examples of the present disclosure, the recognition situation may include the following situation: because text is detected, text recognition has been performed, and the confidence of the recognition result is equal to or greater than a threshold value (e.g., 0.5), so the recognition result is output. The recognition situation can be classified into a good recognition situation and an incorrect recognition situation. In the examples of the present disclosure, the good recognition situation and the incorrect recognition situation can be used as relative concepts.

[0174] For example, a false recognition situation may refer to a situation where the confidence of the recognition result is low (e.g., the confidence is greater than or equal to 0.5 and less than 0.8), and a good recognition situation may refer to a situation where the confidence of the recognition result is relatively higher than the false recognition situation (e.g., the confidence is 0.8 or greater). Therefore, the text recognition result corresponding to the false recognition situation may not be an accurate recognition result of the actual text, although the recognition result is output. For example, because "2: "A**" mfr~y*D" representing the second recognized text in the recognition result of the text 911 detected from the original image 910 has a recognition result confidence of 0.598, the recognition result confidence is a relatively low value and the recognition result is also an inaccurate text, so "2: "A**" mfr~y*D" can be called a false recognition situation.

[0175] Similarly, because "1:ELEVE" representing the first recognized text in the recognition result of the text 911 detected from the original image 910 has a recognition result confidence of 0.888, the recognition result confidence is a relatively high value and the recognition result is also an accurate text, so "1:ELEVE" can be called a good recognition case.

[0176] Even when the confidence of the result of text detection / recognition of the OCR model is high, the result of text detection / recognition may be inaccurate due to the distortion of the image itself. For example, "3:pour cette cuv6e", which represents the third recognized text in the recognition result of the text 911 detected from the original image 910, has a recognition result confidence of 0.960, but the actual accurate text is "pour cette cuvee". This is due to the distortion of the curved surface present on the original image 910 itself, and may be due to the use of a general OCR model instead of learning features related to the distortion separately. Because the electronic device 2000 according to an embodiment of the present disclosure creates an undistorted image 930 and performs OCR on the undistorted image 930, accurate text can be detected even when a general OCR model is used.

[0177] An example of detecting text by using a general OCR model with respect to the cropped image 920 and the undistorted image 930 as images having different features will now be further described. The above description related to non-recognition / misrecognition can also be applied to text 921 detected from the cropped image 920 and text 931 detected from the undistorted image 930, which will be described later.

[0178] The above description related to non-recognition / misrecognition can also be applied to the text 913 detected from the original image 912, the text 923 detected from the cropped image 922, and the text 933 detected from the undistorted image 932, which will be referred to later. Fig. 9BGive a description.

[0179] According to an embodiment of the present disclosure, the cropped image 920 is an image obtained by detecting the ROI from the original image 910 and cropping only the ROI. The cropped image 920 may include a distortion of the ROI due to the 3D shape of the object. When the electronic device 2000 applies OCR to the cropped image 920, at least some of the text in the ROI may not be recognized or may be misrecognized due to the characteristics of the cropped image 920 described above. For example, when the number of text blocks to be detected in the ROI is 14, as a result of applying OCR to the cropped image 920 (i.e., referring to the text 921 detected from the cropped image 920), 9 text blocks may be detected, and at least some of the 9 text blocks may provide inaccurate text detection results.

[0180] According to an embodiment of the present disclosure, according to the above-mentioned embodiment, the 3D shape of the object is recognized by the electronic device 2000, the ROI is recognized, the 3D parameter value representing the 3D information of the object is obtained, and the perspective transformation is performed based on the 3D parameter value to obtain the undistorted image 930. Because the undistorted image 930 is an accurate 2D perspective transformation image obtained based on the 3D information, the electronic device 2000 can obtain a more accurate text detection result. When the electronic device 2000 applies OCR to the undistorted image 930, the text within the ROI can be accurately detected. For example, when the number of text blocks to be detected in the ROI is 14, as a result of applying OCR to the undistorted image 930 (i.e., referring to the text 931 detected from the undistorted image 930), 14 text blocks can be detected, and an accurate text detection result can be obtained.

[0181] The number of text blocks to be detected, unrecognized text blocks, and misrecognized text blocks described above are merely examples for ease of description and are not intended to determine text recognition results. In other words, it should be understood that these are intended to explain that the results of detecting text relative to the undistorted image 930 are relatively more accurate than the results of detecting text relative to the original image 910 and the cropped image 920.

[0182] Fig. 9B 2 is a view for explaining a second example in which the electronic device extracts information from an undistorted image according to an embodiment of the present disclosure.

[0183] refer to Fig. 9B , an original image 912 , a cropped image 922 , and an undistorted image 932 are shown.

[0184] According to an embodiment of the present disclosure, in addition to distortion due to the 3D shape of the object, the original image 912 and the cropped image 922 may also have distortion due to the viewpoint (distance, angle, etc.) from which the electronic device 2000 photographs the object.

[0185] The electronic device 2000 can obtain the undistorted image 930 by recognizing the 3D shape of the object, recognizing the ROI, obtaining a 3D parameter value representing the 3D information of the object, and performing a perspective transformation based on the 3D parameter value. Since the 3D parameters may include the rotation coordinates of the object in the 3D space, the translation coordinates of the object in the 3D space, and the focal length of the camera, the electronic device 2000 can translate and / or rotate the ROI and can perform a perspective transformation.

[0186] In detail, in the original image 912, when the object is not located at the center of the image obtained by shooting the 3D space, the electronic device 2000 can move the object based on the translation information of the object on the space included in the 3D parameters. In detail, in the original image 912, when the object rotates within the image obtained by shooting the 3D space, the electronic device 2000 can rotate the object to be arranged horizontally / vertically based on the rotation information of the object on the space included in the 3D parameters. The electronic device 2000 can supplement the degree of translation / rotation of the object by using the focal length of the camera that has captured the original image 912. According to an embodiment of the present disclosure, in the above-mentioned embodiment, the translation / rotation of the object can be included in the operation of obtaining the 3D parameter value representing the 3D information of the object. In other words, when the electronic device 2000 performs a fine adjustment operation for obtaining the 3D parameter value representing the 3D information of the object, the translation information, the rotation information and the focal length information can be utilized.

[0187] Therefore, if Fig. 9B As shown, even when the object in the original image 912 is photographed obliquely, a distortion-free image 932 with horizontally / vertically arranged ROIs can be obtained.

[0188] According to an embodiment of the present disclosure, the result of detecting text with respect to the undistorted image 932 is relatively more accurate than the result of detecting text with respect to the original image 912 and the cropped image 922. In other words, referring to the text 913 detected from the original image 912, the text 933 detected from the cropped image 922, and the text 933 detected from the undistorted image 932, it can be seen that the text 933 detected from the undistorted image 932 is most accurately recognized.

[0189] The unrecognized text blocks and the misrecognized text blocks are merely examples for ease of description and are not intended to determine the text recognition results. In other words, it should be understood that these are intended to explain that the results of detecting text relative to the undistorted image 930 are relatively more accurate than the results of detecting text relative to the original image 910 and the cropped image 920.

[0190] Fig. 10A is a diagram for explaining an operation of training an object 3D shape recognition model performed by an electronic device according to an embodiment of the present disclosure.

[0191] According to some embodiments of the present disclosure, the electronic device 2000 may train the object 3D shape recognition model 1000. The electronic device 2000 may train the object 3D shape recognition model 1000 by using a training data set consisting of various images including 3D objects. The training data set may include a training image 1010, which includes the entire 3D shape of the object.

[0192] According to an embodiment of the present disclosure, the electronic device 2000 may use a training image 1012 including a portion of the 3D shape of an object in order to improve the inference performance of the object 3D shape recognition model 1000. The training image 1012 including a portion of the 3D shape of an object may be obtained by photographing the entirety or a portion of the object at various angles and distances. For example, an image obtained by photographing the entirety or a portion of the object in a first direction 1012-1 may be obtained, and an image obtained by photographing the entirety or a portion of the object in a second direction 1012-2 may be obtained. As in the aforementioned example, an image obtained by photographing the entirety or a portion of the object in all directions in which the object may be photographed may be included in the training image 1012 and used as training data.

[0193] According to some embodiments of the present disclosure, the training image 1012 including a portion of the 3D shape of the object may have been included in the training data set. According to some embodiments of the present disclosure, the electronic device 2000 may receive the training image 1012 including a portion of the 3D shape of the object from an external device (e.g., a server). According to some embodiments of the present disclosure, the electronic device 2000 may obtain the training image 1012 including a portion of the 3D shape of the object by using a camera. For example, the electronic device 2000 may provide an interface for guiding the user to shoot a portion of the object.

[0194] The electronic device 2000 according to an embodiment of the present disclosure can infer the 3D shape of the object by using the object 3D shape recognition model that is trained using the training image 1010 including the entire 3D shape of the object and the training image 1012 including a portion of the 3D shape of the object. For example, even when only the input image 1020 obtained by photographing only a portion of the object is input, the electronic device 2000 can infer that the 3D shape type of the object in the input image 1020 is a cylinder 1030.

[0195] Fig. 10B is a diagram for explaining another operation of training an object 3D shape recognition model performed by an electronic device according to an embodiment of the present disclosure.

[0196] refer to Fig. 10B, the electronic device 2000 can create training data for training the object 3D shape recognition model 1000.

[0197] According to an embodiment of the present disclosure, the training data set may include a training image 1010 including the entire 3D shape of an object. The electronic device 2000 may create training data by performing a specific data augmentation operation on the image included in the training data set.

[0198] For example, the electronic device 2000 may create a training image 1014 including a portion of the 3D shape of the object so as to crop the training image 1010 including the entire 3D shape of the object. For example, the electronic device 2000 may divide the training image 1010 into six parts to expand the data so that one piece of training data becomes six pieces of training data. For example, when the first area 1014-1 of the training image 1010 is determined as a segmented area, the cropped first image 1014-2 may be used as training data. Although in Fig. 10B Only cropping is shown as an example in , but various other data augmentation methods such as rotation and flipping can be applied.

[0199] The electronic device 2000 according to an embodiment of the present disclosure can infer the 3D shape of the object by using the object 3D shape recognition model trained using the training image 1010 including the entire 3D shape of the object and the training image 1014 including a portion of the 3D shape of the object. For example, even when only the input image 1020 obtained by photographing only a portion of the object is input, the electronic device 2000 can infer that the 3D shape type of the object in the input image 1020 is a cylinder 1030.

[0200] The electronic device 2000 may also perform a specific data augmentation task on the plurality of training data, and train the object 3D shape recognition model 1000 by using the augmented data, thereby improving the inference performance of the object 3D shape recognition model 1000. For example, the electronic device 2000 may apply various data augmentation methods, such as cropping, rotation, and flipping, with respect to the training image 1010 including the entire 3D shape of the object and the training images 1012 and 1014 including a portion of the 3D shape of the object, and may include the augmented data in the training data set.

[0201] Fig. 10C 3 is a diagram for explaining an embodiment in which an electronic device recognizes a 3D shape of an object according to an embodiment of the present disclosure.

[0202] According to an embodiment of the present disclosure, the electronic device 2000 may input an input image 1020 (hereinafter referred to as an input image) obtained by photographing only a part of an object to the object 3D shape recognition model 1000, and may obtain an object 3D shape inference result 1026. In this case, since the input image 1020 does not include the entire shape of the object, the object 3D shape inference result 1026 may need to be supplemented. For example, the object 3D shape inference result 1026 may be a probability (50%) of a cylinder type and a probability (50%) of a truncated cone type. And, the threshold for the object 3D shape recognition model 1000 to determine the object 3D shape may be a probability value: 80% or more. In this case, since the probability (50%) of the cylinder type and the probability (50%) of the cone type do not exceed the threshold (80%) for determining the object 3D shape, the electronic device 2000 may supplement the object 3D shape inference result 1026.

[0203] According to an embodiment of the present disclosure, the electronic device 2000 may perform an information detection operation for supplementing the object 3D shape inference result 1026 based on the fact that the value of the object 3D shape inference result 1026 is less than a preset threshold. The information detection operation may be, for example, detection of a logo, icon, text, etc., but the embodiments of the present disclosure are not limited thereto.

[0204] For example, the electronic device 2000 may perform OCR on the input image 1020 to detect text in the input image 1020. In this case, the detected text may be "ABCDE", which is a product name. The electronic device 2000 may search for products from a database or through an external server based on the detected text. For example, the electronic device 2000 may search for products of "ABCDE" from a database. The electronic device 2000 may determine the weight of the 3D shape type based on the result of the product search. For example, as a result of searching for the product "ABCDE", it may be identified that 95% or more of the products "ABCDE" on the market are of the cylindrical type. In this case, the electronic device 2000 may determine to apply a weight to the cylindrical type. The electronic device 2000 may apply the determined weight to the object 3D shape inference result 1026. As a result of applying the weight, it may be determined that the final determined 3D shape type of the object is a cylinder 1030.

[0205] According to an embodiment of the present disclosure, the electronic device 2000 may perform an information detection operation in parallel with inputting the input image 1020 to the object 3D shape recognition model 1000. For example, the electronic device 2000 may perform OCR on the input image 1020. The electronic device 2000 may determine a weight to be applied to the object 3D shape inference result 1026 based on the result of the OCR performed in parallel.

[0206] Fig. 10D 3 is a diagram for explaining an embodiment in which an electronic device recognizes a 3D shape of an object according to an embodiment of the present disclosure.

[0207] According to an embodiment of the present disclosure, the electronic device 2000 may input an input image 1024 to the object 3D shape recognition model 1000 , and may obtain an object 3D shape inference result 1026 .

[0208] Before applying the input image 1024 to the object 3D shape recognition model 1000, the electronic device 2000 may display a user interface for selecting an object search domain. For example, the electronic device 2000 may display selectable domains such as dairy products, wine, and canned food, and may receive a user input for selecting a domain.

[0209] The electronic device 2000 may determine the weight of the 3D shape type based on the user input for selecting the search domain. For example, when the user selects a wine label search, it may be recognized that 95% or more of the wine products on the market are of the cylindrical type. In this case, the electronic device 2000 may determine to apply a weight to the cylindrical type. The electronic device 2000 may apply the determined weight to the object 3D shape inference result 1026. As a result of applying the weight, it may be determined that the final determined 3D shape type of the object is a cylinder 1030.

[0210] Fig.11 is a diagram for explaining an operation of training a ROI recognition model performed by an electronic device according to an embodiment of the present disclosure.

[0211] According to an embodiment of the present disclosure, the electronic device 2000 may train the ROI recognition model 1120. The electronic device 2000 may train the ROI recognition model 1120 based on a training data set 1110 consisting of various images containing ROIs. Key points representing the ROI may be marked on the ROI image of the training data set 1110. The ROI recognized by the electronic device 2000 by using the ROI recognition model 1120 may include, but is not limited to, an image on which the detected ROI is displayed, key points representing the ROI, and / or coordinates of key points in the image.

[0212] According to an embodiment of the present disclosure, the electronic device 2000 may store the trained ROI recognition model 1120. When the electronic device 2000 performs an operation of removing distortion in an image according to the above embodiment, the electronic device 2000 may execute the trained ROI recognition model 1120. According to an embodiment of the present disclosure, the electronic device 2000 may upload the trained ROI recognition model 1120 in an external server.

[0213] Fig.12is a diagram for explaining an operation of training a distortion removal model performed by an electronic device according to an embodiment of the present disclosure.

[0214] According to an embodiment of the present disclosure, the electronic device 2000 may train the distortion removal model 1220. The training data set 1210 for training the distortion removal model 1220 may include ROI data and 3D parameter data. The ROI data may include, for example, an image including an ROI and key points representing the ROI, but the embodiments of the present disclosure are not limited thereto. The 3D parameter data may include, for example, width, length, height, and radius information of an object, translation and rotation information of a 3D geometric transformation on a 3D space of the object 100, and focal length information of a camera of the electronic device 2000 that has photographed the object, but the embodiments of the present disclosure are not limited thereto.

[0215] According to an embodiment of the present disclosure, the distortion removal model 1210 can receive ROI data and 3D parameter data and can output a distortion-free image. Therefore, the distortion removal model 1220 can use a neural network to learn, for an object with a specific 3D shape, which part of the object is the ROI and what the value of the 3D information of the object is.

[0216] According to an embodiment of the present disclosure, the electronic device 2000 may store the trained distortion removal model 1220. When the electronic device 2000 performs an operation of removing distortion in an image according to the above embodiment, the electronic device 2000 may execute the trained distortion removal model 1220. According to an embodiment of the present disclosure, the electronic device 2000 may upload the trained distortion removal model 1220 in an external server.

[0217] Fig.13 is a diagram for explaining a plurality of cameras in an electronic device according to an embodiment of the present disclosure.

[0218] According to an embodiment of the present disclosure, the electronic device 2000 may include a plurality of cameras. For example, the electronic device 2000 may include a first camera 1310, a second camera 1320, and a third camera 1330. Fig.13 Three cameras are shown in FIG. 1 , but embodiments of the present disclosure are not limited thereto, and a plurality of cameras refers to two or more cameras.

[0219] The corresponding specifications of the plurality of cameras may be different from each other. For example, the first camera 1310 may be a telephoto camera, the second camera 1320 may be a wide-angle camera, and the third camera 1330 may be an ultra-wide-angle camera. However, the type of camera is not limited thereto and may include a standard camera, etc.

[0220] The plurality of cameras may obtain images of different characteristics. For example, the first image 1312 obtained by the first camera 1310 may be an image including a portion of an object by zooming in and photographing the object. The second image 1322 obtained by the second camera 1320 may be an image including the entire object by photographing the object at a wider viewing angle than the first camera 1310. The third image 1332 obtained by the third camera 1330 may be an image including the entire object and a wide area of ​​the scene by photographing the object at a wider viewing angle than the first camera 1310 and the second camera 1320.

[0221] According to an embodiment of the present disclosure, since images obtained by a plurality of cameras included in the electronic device 2000 have different characteristics, the results of the electronic device 2000 extracting information from an object in an image according to the above operation may also be different from each other depending on which cameras are used to obtain the image to be used. In order to recognize an object included in an image and extract information from the ROI of the object, the electronic device 2000 may determine which camera of the plurality of cameras to activate.

[0222] According to an embodiment of the present disclosure, the electronic device 2000 may obtain the first image 1312 by activating the first camera 1310 and photographing the object using the first camera 1310. The electronic device 2000 may recognize the 3D shape type of the object in the image and the ROI of the object by using the first image 1312. According to some embodiments of the present invention, in the above example, the first image 1312 may be an image obtained using the first camera 1310 (which is a telephoto camera).

[0223] In this case, because the first image 1312 includes only a portion of the object, the ROI of the object in the first image 1312 may be identified with sufficient confidence (e.g., a predetermined value or more), but the 3D shape type of the object in the first image 1312 may be identified with insufficient confidence. The electronic device 2000 may activate the second camera 1320 and / or the third camera 1330 to obtain the second image 1322 and / or the third image 1332 each including the entirety of the object, and may identify the 3D shape type of the object by using the second image 1322 and / or the third image 1332. In other words, the electronic device 2000 may selectively use an image suitable for identifying the ROI and the 3D shape type of the object.

[0224] According to an embodiment of the present disclosure, the electronic device 2000 may obtain a first image 1312 and a second image 1322 by activating the first camera 1310 and the second camera 1320 and photographing the object by using the first camera 1310 and the second camera 1320. The electronic device 2000 may recognize an ROI of the object by using the first image 1312 including a portion of the object, and may recognize a 3D shape type of the object by using the second image 1322 and / or the third image 1332.

[0225] The operation of activating the camera performed by the electronic device 2000 according to an embodiment of the present disclosure is not limited to the above examples. The electronic device 2000 can use all possible combinations of multiple cameras. For example, the electronic device 2000 can activate only the second camera 1320 and the third camera 1330, or can activate all of the first camera 1310, the second camera 1320, and the third camera 1330.

[0226] The operations of recognizing the ROI of an object, recognizing the 3D shape type of an object, and removing the distortion of the ROI performed by the electronic device 2000 according to an embodiment of the present disclosure may use the above-mentioned AI models (e.g., object 3D object shape recognition model, ROI recognition model, and distortion removal model). A redundant description thereof will be omitted.

[0227] Detailed operations of processing an image and removing distortion by using a plurality of cameras, performed by the electronic device 2000, will be described in more detail in drawings and descriptions thereof to be described later.

[0228] Fig.14A is a flowchart of an operation using multiple cameras performed by an electronic device according to an embodiment of the present disclosure.

[0229] As in Figure 2 In operation S210, the electronic device 2000 according to an embodiment of the present disclosure may obtain a first image of an object including at least one surface (e.g., a label) by using a first camera. Since the operation of obtaining the first image of the object performed by the electronic device 2000 has been described in detail above, its repeated description will be omitted. Operation S230 may be performed after operation S210, and may be followed by operation S1410.

[0230] In operation S1410, the electronic device 2000 according to an embodiment of the present disclosure checks whether the 3D shape type of the object has been identified from the first image of the object obtained using the first camera. For example, when the first image obtained using the first camera includes only a portion of the object, even when the electronic device 2000 inputs the first image into the second AI model, the second AI model can accurately infer the 3D shape type of the object. At this time, the second AI model can output a result indicating that the 3D shape type of the object cannot be inferred, or output a low confidence value for inferring the 3D shape type. When a result with a confidence value equal to or less than a threshold is output from the second AI model, the electronic device 2000 can determine that the 3D shape type of the object has not been identified from the first image.

[0231] According to an embodiment of the present disclosure, when the 3D shape type of the object has not been recognized from the first image, the electronic device 2000 may perform operation S1420. Operation S1420 may be selectively or redundantly applied together with an operation of determining a weight of the 3D shape type and recognizing the 3D shape by applying the weight, as described above with reference to FIG. Fig. 10C and Fig. 10D When the 3D shape type of the object is recognized, the electronic device 2000 may perform operation S1450 to continue the distortion removal operation.

[0232] In operation S1420, the electronic device 2000 according to an embodiment of the present disclosure activates the second camera. The second camera may have a wider viewing angle than the first camera. The second camera may be, for example, a wide-angle camera or an ultra-wide-angle camera, but the embodiments of the present disclosure are not limited thereto.

[0233] In operation S1430, the electronic device 2000 according to an embodiment of the present disclosure obtains a second image by using the second camera. Because the second camera has a wider viewing angle than the first camera, even when the first image obtained by using the first camera includes only a portion of the 3D shape of the object, the second image obtained by using the second camera can also be included in the entire 3D shape of the object.

[0234] In operation S1440, the electronic device 2000 according to an embodiment of the present disclosure obtains data related to the 3D shape type of the object by applying the second image to the second AI model. The second image may include the entire 3D shape of the object. Figure 2 The operation S230 is the same as that of the operation S230, and thus a detailed description thereof will be omitted.

[0235] In operation S1450, the electronic device 2000 according to an embodiment of the present disclosure recognizes a 3D shape of an object by applying at least one of the first image or the second image to the first AI model.

[0236] According to an embodiment of the present disclosure, even when only a portion of the 3D shape of the object is included in the first image, the ROI may be completely included. The electronic device 2000 may identify an area corresponding to at least one surface (e.g., a label) in the first image as an ROI by applying the first image to a first AI model (ROI recognition model).

[0237] According to an embodiment of the present disclosure, since the entire 3D shape of the object is included in the second image, the ROI may also be completely included. The electronic device 2000 may identify an area corresponding to at least one surface (e.g., a label) in the second image as an ROI by applying the second image to the first AI model (ROI recognition model).

[0238] According to an embodiment of the present disclosure, the electronic device 2000 may identify the ROI by applying the first image and the second image to the first AI model (ROI identification model) and selecting or combining the ROI identification results obtained from the first image and the second image, respectively. After performing operation S1450, the electronic device 2000 may perform Figure 2 In this case, Figure 2 The operations / data related to the first camera in operations S240 to S270 may also be similarly applied to the second camera.

[0239] Fig. 14B It is used for Fig.14A Further explanation of the flow chart is supplemented in the Fig.

[0240] According to an embodiment of the present disclosure, the first image 1410 obtained by the electronic device 2000 by using the first camera may include only a part of the object. In this case, the object 3D shape recognition model 1400 may not be able to recognize the 3D shape type of the object from the first image 1410. In this case, the electronic device 2000 may perform operation S1420 to activate a second camera having a wider viewing angle than the first camera, and may obtain a second image 1420 by using the activated second camera. The electronic device 2000 may input the second image 1420 to the object 3D shape recognition model 1400 to recognize the 3D shape type of the object.

[0241] The operation of recognizing the 3D shape type of the object by using the second image performed by the electronic device 2000 may be selectively or redundantly applied together with the operation of determining the weight of the 3D shape type and recognizing the 3D shape by applying the weight performed by the electronic device 2000, as described above with reference to Fig. 10C and Fig. 10D described.

[0242] Fig.15Ais a flowchart of an operation using multiple cameras performed by an electronic device according to an embodiment of the present disclosure.

[0243] In operation S1510, the electronic device 2000 according to an embodiment of the present disclosure obtains a first image including a portion of an object (e.g., a surface or a label) using a first camera, and obtains a second image including the entire object by using a second camera. The second camera may have a wider viewing angle than the first camera. For example, the first camera may be a telephoto camera, and the second camera may be a wide-angle camera or an ultra-wide-angle camera, but embodiments of the present invention are not limited thereto. According to an embodiment of the present disclosure, a camera of the electronic device 2000 may be activated to photograph an object. The user may activate the camera by touching a hardware button or icon for executing the camera, or may activate the camera by a voice command.

[0244] When the user adjusts the position of the electronic device 2000 so that the surface (e.g., label) generally appears on the preview area corresponding to the first camera in order to extract information from the surface of the object (e.g., label), the surface of the object (e.g., label) may clearly appear on the first image obtained by the electronic device 2000 using the first camera, but the entire shape of the object may not appear. However, the entire shape of the object may appear on the second image obtained using the second camera having a wider field of view than the first camera.

[0245] In operation S1520, the electronic device 2000 according to an embodiment of the present disclosure applies the first image to the first AI model (ROI recognition model) to recognize the ROI (e.g., the area corresponding to at least one label) of the surface of the object. Since the first image is an image on which the ROI is focused, the ROI can be accurately recognized by applying the first image to the first AI model. Since operation S1520 corresponds to Figure 2 Operation S220 is described below, and thus a detailed description thereof will be omitted.

[0246] In operation S1530, the electronic device 2000 according to an embodiment of the present disclosure recognizes the 3D shape type of the object by applying the second image to the second AI model. Since operation S1530 corresponds to Figure 2 The step S230 is described below, and therefore its redundant description will be omitted.

[0247] In operation S1540, the electronic device 2000 according to an embodiment of the present disclosure obtains a 3D parameter value corresponding to the 3D shape type of the object. Figure 2 Operation S240 is described below, and thus a detailed description thereof will be omitted.

[0248] Fig. 15B It is used for Fig.15AFurther explanation of the flow chart is supplemented in the Fig.

[0249] According to an embodiment of the present disclosure, the first image 1502 obtained by the electronic device 2000 using the first camera may be an image obtained using a telephoto camera. Since the first image 1502 does not include the entire 3D shape of the object but includes an enlarged ROI, the first image 1502 may be an image suitable for identifying the ROI. In this case, the electronic device 2000 may identify a region corresponding to at least one surface (e.g., a label) in the first image as an ROI by inputting the first image 1502 into the ROI recognition model 1510.

[0250] According to an embodiment of the present disclosure, the second image 1504 obtained by the electronic device 2000 using the second camera may be an image obtained using a wide-angle camera and / or an ultra-wide-angle camera. Since the second image 1504 contains the entire 3D shape of the object, the second image 1504 may be an image suitable for recognizing the 3D shape of the object. In this case, the electronic device 2000 may input the second image 1504 into the object 3D shape recognition model 1520 to recognize the 3D shape type of the object within the second image 1504.

[0251] Fig.16A is a flowchart of an operation using multiple cameras performed by an electronic device according to an embodiment of the present disclosure.

[0252] In operation S1610, the electronic device 2000 according to an embodiment of the present disclosure applies a first image captured in real time by using a first camera to a first AI model (ROI recognition model) to obtain a confidence level of the ROI. The first camera may be a telephoto camera.

[0253] According to an embodiment of the present disclosure, when a user of the electronic device 2000 wants to identify an object (for example, when the user wants to search for a label of a product), the user can activate a camera application. The user can continuously adjust the field of view of the camera so that the camera looks at the object while viewing a preview image or the like displayed on the screen of the electronic device 2000. The electronic device 2000 can input each first image frame obtained in real time by the first camera into the ROI recognition model. The electronic device 2000 can obtain the confidence of the ROI, which indicates the accuracy of identifying the ROI of each of the first image frames.

[0254] In operation S1620, the electronic device 2000 according to an embodiment of the present disclosure obtains the confidence of the 3D shape type of the object by applying the second image captured in real time by using the second camera to the second AI model. The second camera may be a wide-angle camera or an ultra-wide-angle camera.

[0255] According to an embodiment of the present disclosure, the electronic device 2000 may input each second image frame obtained in real time by the second camera into the object 3D shape estimation model. The electronic device 2000 may obtain the confidence level of the 3D shape type of the object, which indicates the accuracy of estimating the 3D shape of the object for each second image frame.

[0256] In operation S1630, the electronic device 2000 according to an embodiment of the present disclosure determines whether the confidence of the ROI exceeds a first threshold. The first threshold may be a preset threshold of the ROI. When the confidence of the ROI is equal to or less than the first threshold, the electronic device 2000 may continue to perform operation S1610 until a confidence exceeding the first threshold is obtained.

[0257] In operation S1640, the electronic device 2000 according to an embodiment of the present disclosure determines whether the confidence of the 3D shape type of the object exceeds a second threshold. The second threshold may be a preset threshold for the 3D shape of the object. When the confidence of the 3D shape type of the object is equal to or less than the second threshold, the electronic device 2000 may continue to perform operation S1620 until a confidence exceeding the second threshold is obtained.

[0258] In operation S1650, the electronic device 2000 according to an embodiment of the present disclosure captures a first image and a second image.

[0259] According to an embodiment of the present disclosure, the condition for performing operation S1650 is an AND condition that the confidence of the ROI exceeds the first threshold and the confidence of the 3D shape type exceeds the second threshold. The electronic device 2000 can capture and store the first image and the second image, and can perform operation S1520 and subsequent operations thereof. In this case, the electronic device 2000 can identify the ROI of the surface of the object by applying the first image to the ROI recognition model, and can identify the 3D shape of the object by applying the second image to the object 3D shape recognition model. Because its detailed operation has been described above, its redundant description will be omitted.

[0260] Fig. 16B It is used for Fig.16A Further explanation of the flow chart is supplemented in the Fig.

[0261] In the description Fig. 16B and 16C When describing the method of identifying a wine label, a case where a user wants to identify a wine label will be described as an example.

[0262] Reference Fig. 16B, the electronic device 2000 according to an embodiment of the present disclosure may display a first screen image 1600 for object recognition. The first screen image 1600 may include an interface for guiding a user of the electronic device 2000 to perform object recognition. For example, the electronic device 2000 may display a rectangular frame 1606 for guiding the ROI of the object included in the first screen image 1600 (however, the rectangular frame 1606 is not limited to a rectangle and has another shape capable of performing a similar function, such as a circle), and may display guidance such as “Search for wine labels (indicated by 1608)”. According to some embodiments of the present disclosure, when an object is not recognized from the image displayed on the first screen 1600, the electronic device 2000 may display guidance such as “Please view the product through the camera”.

[0263] According to an embodiment of the present disclosure, the electronic device 2000 may display a second screen image 1602 representing a preview image obtained by a camera. When the user is viewing the second screen image 1602, the user may adjust the field of view of the camera so that the object is completely included in the image. The electronic device 2000 may calculate the confidence of the ROI and the 3D shape type of the object when displaying the second screen image 1602 as a preview image of the camera. Because this has been described above, its redundant description will be omitted.

[0264] When the confidence of the ROI exceeds the first threshold and the confidence of the 3D shape type of the object exceeds the second threshold, the electronic device 2000 may obtain a 3D parameter value related to the object based on the area corresponding to at least one surface (e.g., label) identified as the ROI and the data related to the 3D shape type of the object. The electronic device 2000 may obtain a flat surface (e.g., label) image in which the curved shape of at least one surface (e.g., label) has been flattened by estimating the curved shape of at least one surface (e.g., label) using the 3D parameter value related to the object and performing a perspective transformation. When the flat surface (e.g., label) image is obtained and information related to the object is extracted from the flat surface (e.g., label) image (i.e., when the product is recognized), the electronic device 2000 may output a notification such as "wine information has been retrieved (indicated by 1610)" to the preview image. The electronic device 2000 may output information 1604 related to the object extracted from the flat surface (e.g., label) image. For example, the electronic device 2000 may output a wine label image and detailed information about the wine.

[0265] Fig. 16C It is used for Fig.16A Further explanation of the flow chart is supplemented in the Fig.

[0266] Reference Fig. 16C, the electronic device 2000 according to an embodiment of the present disclosure may display a first screen image 1600 for object recognition. The first screen image 1600 may include an interface for guiding a user of the electronic device 2000 to perform object recognition. For example, the electronic device 2000 may display a rectangular frame 1606 for guiding the ROI of the object included in the first screen image 1600 (however, the rectangular frame 1606 is not limited to a rectangle and has another shape capable of performing a similar function, such as a circle), and may display guidance such as “Search for wine labels (indicated by 1608)”. According to some embodiments of the present disclosure, when an object is not recognized from the image displayed on the first screen 1600, the electronic device 2000 may display guidance such as “Please view the product through the camera”.

[0267] According to an embodiment of the present disclosure, the electronic device 2000 may calculate the confidence of the ROI and the 3D shape type of the object while displaying the second screen image 1602 as a preview image of the camera. Only when the confidence of the ROI exceeds the first threshold and the confidence of the 3D shape type of the object exceeds the second threshold, the electronic device 2000 performs a subsequent operation for removing distortion from the image. Therefore, when the confidence of the ROI is less than or equal to the first threshold and / or the confidence of the 3D shape type of the object is less than or equal to the second threshold, the electronic device 2000 may output a notification for guiding the user to adjust the camera field of view in order to obtain the first image and the second image. For example, the electronic device 2000 may display on the screen or output a notification such as an inability to recognize a wine label as audio. Please adjust the camera angle (indicated by 1612).

[0268] Fig.17 is a diagram for explaining an operation of processing an image and providing extracted information, performed by an electronic device according to an embodiment of the present disclosure.

[0269] According to an embodiment of the present disclosure, the electronic device 2000 may create a flat surface (eg, label) image as an undistorted image, extract information related to an object from the flat surface (eg, label) image, and provide the extracted information to a user.

[0270] According to an embodiment of the present disclosure, the electronic device 2000 may display a first screen image 1700 for starting object recognition. The first screen image 1700 may include a user interface such as “wine label scanning 1701.” The user of the electronic device 2000 may start an object recognition operation through the user interface.

[0271] According to an embodiment of the present disclosure, the electronic device 2000 may display a second screen image 1702 for performing object recognition. The second screen image 1702 may include an interface for guiding a user of the electronic device 2000 to perform object recognition. For example, the electronic device 2000 may display a guide area 1702-1 for guiding an ROI of an object to be included in the second screen image 1702, and may display a guide phrase (1702-2) such as "Take a picture of the front label of the wine".

[0272] The electronic device 2000 may obtain multiple images (e.g., a telephoto image, a wide-angle image, and an ultra-wide-angle image) through multiple cameras, and may perform a distortion removal operation based on 3D information according to the above-described embodiments. In other words, the electronic device 2000 creates a distortion-free wine label image by extracting a wine label region from the image and performing correction to remove distortion. The electronic device 2000 may extract multiple pieces of wine-related information by applying OCR to the distortion-free wine label image. The electronic device 2000 may search for wine information by using text information recognized from the wine label.

[0273] According to an embodiment of the present disclosure, when the electronic device 2000 extracts / corrects a wine label area and searches for wine information by using text information recognized from a wine label, the electronic device 2000 may display a third screen image 1704 indicating object recognition and search results. An undistorted image created by the electronic device 2000 according to the above-described embodiment may be displayed on the third screen image 1704. Fig.17 The undistorted image in the example of may be a wine label image. The wine label image may be a flat surface (eg, label) image obtained by converting a wine label attached to a wine bottle in a curved shape into a flat wine label.

[0274] The object related information obtained by the electronic device 2000 according to the above-described embodiment may be displayed on the third screen image 1704 . Fig.17 The object related information in the example of can be wine detailed information. In this case, the wine name, origin, production year, etc. as a result of performing OCR on the wine label image can be displayed.

[0275] According to an embodiment of the present disclosure, in addition to the object-related information obtained from the wine label image, additional information related to the object obtained from the server or from the database of the electronic device 2000 may be displayed on the third screen image 1704. For example, the acidity, body, and alcohol content of the wine that cannot be obtained from the wine label image may be displayed.

[0276] According to an embodiment of the present disclosure, information obtained from another electronic device and / or information obtained based on user input may be further displayed on the third screen image 1704. For example, the nickname, storage date, storage location, etc. of the wine may be displayed.

[0277] However, the information obtainable from the wine label image and the information obtained from a path other than the wine label image have been described by way of example and are not limited to the above description.

[0278] According to an embodiment of the present disclosure, the electronic device 2000 may display the fourth screen image 1706 in which object recognition and search results are performed into the database. In this case, the electronic device 2000 may display the flat surface (e.g., label) image as an undistorted image in a preview form 1708. When each flat surface (e.g., label) image is selected, the wine information corresponding to the selected flat surface (e.g., label) image may be displayed again, as in the third screen image 1704.

[0279] Fig.18 is a diagram for explaining an example of a system related to an operation of processing an image performed by an electronic device according to an embodiment of the present disclosure.

[0280] According to an embodiment of the present disclosure, the model used by the electronic device 2000 can be trained in another electronic device suitable for performing neural network calculations (e.g., a local personal computer (PC)). For example, an object 3D shape estimation model, a ROI recognition model, a distortion removal model, an information extraction model, etc. can be trained by another electronic device and stored in a training state.

[0281] According to an embodiment of the present disclosure, the electronic device 2000 may receive a training model stored in another electronic device. The electronic device 2000 may perform the above-mentioned image processing operation based on the received model. In this case, the electronic device 2000 may perform an inference operation by executing the trained model, and may create a flat surface (e.g., label) image and surface (e.g., label) information. The created flat surface (e.g., label) image and the created surface (e.g., label) information may be provided to a user through an application, etc. Fig.18 In the embodiment, it has been described that the model is stored and used in a mobile phone as an example of the electronic device 2000. However, the embodiments of the present disclosure are not limited thereto. The electronic device 2000 may include any electronic device that can execute an application and is equipped with a display and a camera, such as a TV, a tablet PC, and a smart refrigerator.

[0282] As described above in the description of the previous drawings, the model used by the electronic device 2000 may be trained using the computing resources of the electronic device 2000. Since this has been described in detail above, a redundant description thereof will be omitted.

[0283] Fig.19 is a diagram for explaining an example of a system related to an operation of processing an image by using a server, performed by an electronic device according to an embodiment of the present disclosure.

[0284] According to an embodiment of the present disclosure, a model used by the electronic device 2000 may be trained in another electronic device (e.g., a local PC) suitable for performing neural network calculations. For example, an object 3D shape estimation model, an ROI recognition model, a distortion removal model, an information extraction model, etc. may be trained by another electronic device and stored in a trained state. The model trained in another electronic device (e.g., a local PC) may be sent to and stored in another electronic device (e.g., a server).

[0285] According to an embodiment of the present disclosure, the electronic device 2000 can perform image processing operations by using a server. The electronic device 2000 can capture an object image (e.g., a telephoto image, a wide-angle image, and an ultra-wide-angle image) by using a camera, and can send the object image to the server. In this case, the server can perform an inference operation by executing a trained model, and can create a flat surface (e.g., label) image and surface (e.g., label) information. The electronic device 2000 can receive a flat surface (e.g., label) image and surface (e.g., label) information from the server. The received flat surface (e.g., label) image and the received surface (e.g., label) information can be provided to a user through an application, etc. Fig.19 In the embodiment, it has been described that the model is stored and used in a mobile phone as an example of the electronic device 2000. However, the embodiments of the present disclosure are not limited thereto. The electronic device 2000 may include any electronic device that can execute an application and is equipped with a display and a camera, such as a TV, a tablet PC, and a smart refrigerator.

[0286] As described above in the description of the previous drawings, the model used by the electronic device 2000 may be trained using the computing resources of the electronic device 2000. Since this has been described in detail above, a redundant description thereof will be omitted.

[0287] Fig. 20 is a block diagram of an electronic device 2000 according to an embodiment of the present disclosure.

[0288] The electronic device 2000 according to an embodiment of the present disclosure may include a communication interface 2100 , a camera 2200 , a memory 2300 , and a processor 2400 .

[0289] The communication interface 2100 may perform data communication with other electronic devices under the control of the processor 2400 .

[0290] The communication interface 2100 may include a communication circuit. The communication interface 2100 may include a communication circuit that can perform data communication between the electronic device 2000 and other electronic devices by using at least one of the data communication methods, including, for example, a wired local area network (LAN), a wireless LAN, Wi-Fi, Bluetooth, Zigbee, Wi-Fi Direct (WFD), infrared communication (IrDA), Bluetooth Low Energy (BLE), near field communication (NFC), wireless broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and radio frequency (RF) communication.

[0291] The communication interface 2100 may transmit / receive data for performing image processing operations of the electronic device 2000 to / from an external electronic device. For example, the communication interface 2100 may transmit / receive an AI model used by the electronic device 2000, or transmit / receive a training data set of the AI ​​model to / from a server, etc. The electronic device 2000 may obtain an image from which distortion is to be removed from a server, etc. The electronic device 2000 may transmit / receive data to / from a server, etc., in order to search for information related to an object.

[0292] The camera 2200 can obtain a video and / or an image by photographing an object. One or more cameras 2200 may be included. The camera 220 may include, for example, an RGB camera, a telephoto camera, a wide-angle camera, and an ultra-wide-angle camera, but the embodiments of the present invention are not limited thereto. The camera 2200 can obtain a video including a plurality of frames. The specific type and detailed functions of the camera 2200 can be clearly inferred by a person of ordinary skill in the art, and thus a description thereof is omitted.

[0293] Instructions, data structures, and program codes that can be read by the processor 2400 can be stored in the memory 2300. One or more memories 2300 can be included. According to the disclosed embodiments, the operations performed by the processor 2400 can be implemented by executing instructions or codes of the program stored in the memory 2300.

[0294] The memory 2300 may include a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (e.g., a secure digital (SD) or extreme digital (XD) memory), and may include a non-volatile memory and a volatile memory, the non-volatile memory including at least one of a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, or an optical disk, and a volatile memory such as a random access memory (RAM) or a static random access memory (SRAM).

[0295] The memory 2300 according to an embodiment of the present disclosure may store one or more instructions and / or programs for causing the electronic device 2000 to operate to remove distortion in an image. For example, the memory 2300 may store a ROI recognition module 2310, an object 3D shape recognition module 2320, a 3D information acquisition module 2330, a distortion removal module 2340, and an information extraction module 2350.

[0296] The processor 2400 may control the overall operation of the electronic device 2000. For example, the processor 2400 may control the overall operation of the electronic device 2000 for removing distortion from an image by executing one or more instructions of a program stored in the memory 2300. One or more processors 2400 may be included.

[0297] According to the present disclosure, one or more processors 2400 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), an integrated many-core (MIC), a digital signal processor (DSP), or a neural processing unit (NPU). One or more processors 2400 may be implemented in the form of an integrated system on chip (SoC) including one or more electronic components. Each of the one or more processors 2400 may be implemented as separate hardware (H / W).

[0298] In this case, the processor 2400 may identify a region corresponding to at least one surface (e.g., a label) in the image as a ROI by executing the ROI recognition module 2310. The ROI recognition module 2310 may include the ROI recognition model. Since specific operations related to the ROI recognition module 2310 have been described in detail with reference to the previous drawings, their redundant description will be omitted.

[0299] The processor 2400 executes the object 3D shape recognition module 2320 to obtain data related to the 3D shape type of the object in the image. The object 3D shape recognition module 2320 may include an object 3D shape recognition model. Since the specific operations related to the object 3D shape recognition module 2320 have been described in detail with reference to the previous drawings, their redundant description will be omitted.

[0300] The processor 2400 can infer the 3D information of the object in the image by executing the 3D information acquisition module 2330. The processor 2400 obtains a 3D parameter value related to at least one of the object, at least one surface (e.g., a label), or the first camera based on the ROI and the data related to the 3D shape type of the object. Obtaining the 3D parameter value can represent the 3D information of the object by finely adjusting the initial value of the 3D parameter corresponding to the 3D shape of the object. Since the specific operations related to the 3D shape acquisition module 2330 have been described in detail with reference to the previous drawings, their redundant description will be omitted.

[0301] The processor 2400 may remove distortion from the image by executing the distortion removal module 2340. The distortion removal module 2340 may include a distortion removal model. The processor 2400 may estimate the curved shape of at least one surface (e.g., a label) based on the 3D parameters. The processor 2400 may obtain a flat surface (e.g., a label) image in which the curved shape of the surface (e.g., a label) has been flattened by performing a perspective transformation on at least one surface (e.g., a label). Since specific operations related to the distortion removal module 2340 have been described in detail with reference to the previous drawings, their redundant description will be omitted.

[0302] The processor 2400 may extract information from the undistorted image by executing the information extraction module 2350. The information extraction module 2350 may include an information extraction model. The processor 2400 may extract information within the ROI by using the information extraction module 2350, and may identify, for example, a logo, an icon, and text within the ROI. Since specific operations related to the information extraction module 2350 have been described in detail with reference to the previous drawings, their redundant description will be omitted.

[0303] The modules stored in the memory 2300 are for convenience of description, but the embodiments of the present disclosure are not limited thereto. Other modules may be added to implement the above embodiments, and some of the above modules may be implemented as one module.

[0304] When the method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when the first operation, the second operation, and the third operation are performed by the method according to an embodiment of the present disclosure, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor), and the third operation may be performed by the second processor (e.g., an AI processor). An AI-specific processor as an example of a second processor may perform operations for training / inferring an AI model. However, the embodiments of the present disclosure are not limited thereto.

[0305] One or more processors according to the present disclosure may be implemented as a single-core processor or a multi-core processor.

[0306] When the method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core or by a plurality of cores included in one or more processors.

[0307] Although not in Fig. 20 , but the electronic device 2000 may further include a user interface. The user interface may include an input interface for receiving user input and an output interface for outputting information.

[0308] An output interface is provided to output an audio signal or a video signal. The output interface may include a display, a sound output interface, a vibration motor, etc. When the display forms a layer structure with the touch pad to construct a touch screen, the display can be used as an input device and an output device. The display may include at least one selected from a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT-LCD), a light emitting diode (LED), an organic light emitting diode (OLED), a flexible display, a 3D display, and an electrophoretic display. According to an embodiment of the electronic device 2000, the electronic device 2000 may include at least two displays.

[0309] The audio output interface may output an audio signal received from the communication interface 2100 or stored in the memory 2300. The sound output interface may output a sound signal related to a function performed by the electronic device 2000. The audio output interface may include, for example, a speaker and a buzzer.

[0310] The input interface is used to receive input from the user. The input interface may include, but is not limited to, at least one of a keypad, a dome switch, a touch pad (e.g., a capacitive overlay type, a resistive overlay type, an infrared beam type, an integrated strain gauge type, a surface acoustic wave type, a piezoelectric type, etc.), a scroll wheel, or a micro switch.

[0311] The input interface may include a speech recognition module. For example, the electronic device 2000 may receive a speech signal as an analog signal through a microphone, and convert the speech signal into a computer-readable text by using an automatic speech recognition (ASR) model. The electronic device 2000 may also obtain the user's speech intention by interpreting the converted text using a natural language understanding (NLU) model. The ASR model or the NLU model may be an AI model. Language understanding is a technology for recognizing and applying / processing human language / characters, and therefore includes natural language processing, machine translation, dialogue systems, question-answering, and speech recognition / speech recognition / synthesis, etc.

[0312] Fig.21 is a block diagram of a structure of a server according to an embodiment of the present disclosure.

[0313] According to an embodiment of the present disclosure, the operation of the electronic device 2000 may be performed by the server 3000 .

[0314] The server 3000 according to an embodiment of the present disclosure may include a communication interface 3100, a memory 3200, and a processor 3300. The communication interface 3100, the memory 3200, and the processor 3300 of the server 3000 correspond to Fig. 20 The communication interface 2100, the memory 2300 and the processor 2400 of the electronic device 2000 are described above, and thus a redundant description thereof will be omitted.

[0315] The server 3000 according to an embodiment of the present disclosure may have higher computing performance than the electronic device 2000, so that it can perform calculations with a larger amount of calculations than the electronic device 2000. The server 3000 may perform training of an AI model, which requires a relatively large amount of calculations compared to inference. The server 3000 may perform interference by using the AI ​​model and send the result of the interference to the electronic device 2000.

[0316] The present disclosure aims to propose an image processing method in an image distortion removal method using 3D information, the image processing method being used to infer 3D information of an object and remove distortion in an image by using an algorithm without requiring hardware such as a sensor for obtaining 3D information.

[0317] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0318] According to one aspect of the present disclosure, a method for processing an image performed by an electronic device 2000 is provided. The method may include obtaining a first image of a three-dimensional (3D) object including at least one surface (e.g., a label) by using a first camera. The method may include identifying an area corresponding to at least one surface (e.g., a label) in the first image as an ROI by applying the first image to a first AI model. The method may include obtaining data related to the 3D shape type of the object by applying the first image to a second AI model. The method may include obtaining a set of values ​​of 3D parameters related to the object, at least one surface (e.g., a label), or at least one of the first cameras based on an area corresponding to at least one surface (e.g., a label) identified as an ROI and data related to the 3D shape type of the object. The method may include estimating a non-flat shape of at least one surface (e.g., a label) based on the value of the 3D parameter. The method may include obtaining a flat surface (e.g., a label) image in which the non-flat shape of at least one surface (e.g., a label) has been flattened by performing a perspective transformation on at least one surface (e.g., a label).

[0319] The set of values ​​of the 3D parameter may include at least one of width value, length value, height value and radius value related to the 3D shape of the object, angle value of ROI of the surface of the object, translation value and rotation value of 3D geometric transformation or focal length value of the camera.

[0320] The first AI model can be trained to infer the region corresponding to the surface (e.g., label) in the image as the ROI. The second AI model can be trained to infer the 3D shape type of the object in the image.

[0321] Obtaining data related to the 3D shape type of the object may include receiving user input related to the 3D shape type of the object from a user. Obtaining data related to the 3D shape type of the object may also include identifying the 3D shape type of the object by applying a weight to a 3D shape type corresponding to the user input among a plurality of 3D shape types.

[0322] Identifying an area corresponding to at least one surface (e.g., a label) as an ROI may include identifying a first keypoint representing an area corresponding to at least one surface (e.g., a label). Obtaining a set of values ​​of 3D parameters may include obtaining a virtual object corresponding to the 3D shape type of the object and a set of initial values ​​of the 3D parameters of the virtual object. Obtaining the values ​​of the 3D parameters may further include adjusting the set of initial values ​​of the 3D parameters of the virtual object based on the first keypoint. Obtaining the values ​​of the 3D parameters may also include obtaining a set of adjusted initial values ​​of the 3D parameters of the virtual object as a set of values ​​of the 3D parameters associated with at least one of the object, at least one surface, or a camera.

[0323] Adjusting the initial values ​​of the 3D parameters of the virtual object based on the first keypoint may include setting a second keypoint representing an area corresponding to a virtual surface (e.g., a label) of the virtual object. Adjusting the initial values ​​of the 3D parameters of the virtual object based on the first keypoint may include adjusting the second keypoint to match the first keypoint so that the set of initial values ​​of the 3D parameters of the virtual object approximates the ground truth of the set of values ​​of the 3D parameters of the object.

[0324] Obtaining information related to the object from the flat surface (eg, label) image may include applying OCR to the flat surface (eg, label) image.

[0325] The method may further include obtaining a second image of the object by using a second camera having a wider viewing angle than the first camera.

[0326] Obtaining data related to the 3D shape type of the object may also include obtaining information related to the 3D shape type of the object by further applying the second image to the second AI model.

[0327] The method may also include obtaining a confidence of the ROI by applying the first image to the first AI model using the first camera. The method may also include obtaining a confidence of the 3D shape type of the object by applying the second image to the second AI model using the second camera. The method may also include capturing the first image and the second image based on corresponding thresholds of the confidence of the 3D shape type of the object and the confidence of the ROI, respectively.

[0328] The method may also include searching for matching data in a database based on the flat surface (e.g., label) image or information obtained from the flat surface (e.g., label) image. The method may also include displaying the results of the search, and the database may store other flat surface (e.g., label) images previously obtained by the electronic device and information related to other objects.

[0329] According to one aspect of the present disclosure, an electronic device for processing an image is provided. The electronic device may include a first camera, a memory storing one or more instructions, and one or more processors configured to execute the one or more instructions stored in the memory. One or more processors may be configured to execute one or more instructions to obtain a first image of a 3D object including at least one surface (e.g., a label) by using a first camera. One or more processors may also be configured to execute one or more instructions to identify an area corresponding to at least one surface (e.g., a label) in the first image as an ROI by applying the first image to a first AI model. One or more processors may also be configured to execute one or more instructions to obtain data related to the 3D shape type of the object by applying the first image to a second AI model. One or more processors may also be configured to execute one or more instructions to obtain a set of 3D parameters related to the object, at least one surface (e.g., a label), or at least one of the first cameras based on an area corresponding to at least one surface (e.g., a label) identified as an ROI and data related to the 3D shape type of the object. One or more processors may further be configured to execute one or more instructions to estimate the non-flat shape of at least one surface (e.g., a label) based on a set of values ​​of 3D parameters. The one or more processors may also be configured to execute one or more instructions to obtain a flat surface (e.g., label) image in which a non-flat shape of at least one surface (e.g., label) has been flattened by performing a perspective transformation on at least one surface (e.g., label).

[0330] The one or more processors may also be configured to execute one or more instructions to receive user input related to the 3D shape type of the object from the user. The one or more processors may also be configured to execute one or more instructions to identify the 3D shape type of the object by applying a weight to a 3D shape type corresponding to the user input in a plurality of 3D shape types.

[0331] The one or more processors may be further configured to execute the one or more instructions to identify a first key point representing the area corresponding to the at least one surface (e.g., a label). The one or more processors may also be configured to execute one or more instructions to obtain a virtual object corresponding to the 3D shape type of the object and a set of initial values ​​of 3D parameters of the virtual object. The one or more processors may be further configured to execute the one or more instructions to adjust the set of initial values ​​of the 3D parameters of the virtual object based on the first key point. The one or more processors may also be configured to execute one or more instructions to obtain a set of adjusted initial values ​​of the 3D parameters of the virtual object as a set of values ​​of the 3D parameters associated with at least one of the object, at least one surface, or the camera.

[0332] The one or more processors may be further configured to execute one or more instructions to set a second keypoint representing an area of ​​a virtual surface (e.g., a label) corresponding to the virtual object. The one or more processors may be further configured to execute the one or more instructions to adjust the second keypoint to match the first keypoint so that the set of initial values ​​of the 3D parameters of the virtual object approximates the ground truth of the set of values ​​of the 3D parameters of the object.

[0333] The one or more processors may also be configured to execute one or more instructions to apply OCR to flat surface (eg, label) images.

[0334] The electronic device may further include a second camera having a wider viewing angle than the first camera, and the one or more processors may be further configured to execute the one or more instructions to obtain a second image of the object through the second camera.

[0335] The one or more processors may also be configured to execute the one or more instructions to obtain information related to the 3D shape type of the object by applying the second image to the second AI model.

[0336] The method for processing an image performed by an electronic device according to an embodiment of the present disclosure may include obtaining a partial image of an object including at least one surface (e.g., a label) by using a first camera. The method may include identifying an area corresponding to the surface (e.g., a label) of the object as an ROI by applying the partial image of the object to a first AI model. The method may include obtaining an entire image of the object by using a second camera. The method may include identifying a 3D shape type of the object by applying the entire image of the object to a second AI model. The method may include obtaining a set of values ​​of 3D parameters corresponding to the 3D shape type of the object. The method may include obtaining a flat surface (e.g., a label) image in which a curved shape of the surface (e.g., a label) has been flattened by performing a perspective transformation of the surface (e.g., a label) based on information about the ROI and a set of values ​​of the 3D parameters. The method may include obtaining information related to the object from the flat surface (e.g., a label) image.

[0337] Embodiments of the present disclosure may also be embodied as a storage medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media may be any available media accessible by a computer, and includes all volatile / non-volatile and removable / non-removable media. In addition, computer-readable media may include all computer storage and communication media. Computer storage media include all volatile / non-volatile and removable / non-removable media embodied by a method or technology for storing information such as computer-readable instruction codes, data structures, program modules, or other data. Communication media may typically include computer-readable instructions, data structures, or other data in a modulated data signal, such as a program module.

[0338] In addition, the computer-readable storage medium may be provided in the form of a non-transitory storage medium. A "non-transitory storage medium" is a tangible device and simply means that it does not contain signals (e.g., electromagnetic waves). The term does not distinguish between a case where data is semi-permanently stored in a storage medium and a case where data is temporarily stored. For example, a non-transitory recording medium may include a buffer that temporarily stores data.

[0339] According to an embodiment of the present disclosure, a method according to various disclosed embodiments can be provided by being included in a computer program product. A computer program product as a commodity can be traded between a seller and a buyer. The computer program product is distributed in the form of a device-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or can be distributed (e.g., downloaded or uploaded) directly and online through an application store or between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) can be at least temporarily stored in a device-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or a relay server, or can be temporarily generated.

[0340] Although the present disclosure has been specifically shown and described with reference to the exemplary embodiments of the present disclosure, it should be understood that various changes in form and detail may be made therein without departing from the spirit and scope of the present disclosure. Therefore, the above-described embodiments should be considered only in a descriptive sense, and not for the purpose of limitation. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

[0341] The scope of the present disclosure is indicated by the scope of claims described later rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and concepts of their equivalents should be construed as being included in the scope of the present disclosure.

Claims

1. A method for processing an image performed by an electronic device, the method include: obtaining a first image of a three-dimensional (3D) object including at least one surface having a non-flat shape by using a first camera (S210); identifying a region corresponding to the at least one surface as a region of interest (ROI) by applying the first image to a first artificial intelligence (AI) model (S220); obtaining data on a 3D shape type of the object by applying the first image to a second AI model (S230); obtaining a set of values ​​of 3D parameters associated with at least one of the object, the at least one surface, or the first camera based on the region identified as the ROI and the data about the 3D shape type (S240); estimating the non-planar shape of the at least one surface based on the set of values ​​of the 3D parameters (S250); as well as A flat surface image in which the non-flat shape of the at least one surface is flattened is obtained by performing perspective transformation on the at least one surface (S260).

2. The method according to claim 1, in, The set of values ​​of the 3D parameter includes at least one of the following: a height value associated with the 3D shape of the object, The radius value associated with the 3D shape of the object, an angular value of the ROI of the at least one surface of the object, The translation value of the 3D geometric transformation, the rotation value of the 3D geometric transformation, or The focal length of the first camera.

3. The method according to any one of claims 1 and 2, in, training the first AI model to infer a region corresponding to a surface in an image as a ROI, and Wherein, the second AI model is trained to infer the 3D shape type of the object in the image.

4. The method according to any one of claims 1 to 3, in, Obtaining data about the 3D shape type of the object includes: receiving input from a user relating to a 3D shape type of the object; and The 3D shape type of the object is identified by applying a weight to a 3D shape type corresponding to the user's input among a plurality of 3D shape types.

5. The method according to any one of claims 1 to 4, in, Identifying the region corresponding to the at least one surface as the ROI comprises identifying a first keypoint representing the region corresponding to the at least one surface, and Obtaining a set of values ​​of the 3D parameters includes: Obtaining a virtual object corresponding to the 3D shape type of the object, and obtaining a set of initial values ​​of 3D parameters of the virtual object; adjusting a set of initial values ​​of 3D parameters of the virtual object based on the first key point; and A set of adjusted initial values ​​of the 3D parameters of the virtual object is obtained as a set of values ​​of the 3D parameters associated with at least one of the object, the at least one surface, or the first camera.

6. The method according to claim 5, in, The set of initial values ​​of the 3D parameters of the virtual object adjusted based on the first key point includes: Setting a second keypoint representing the area corresponding to the virtual surface of the virtual object; and The second keypoint is adjusted to match the first keypoint such that the set of initial values ​​of the 3D parameters of the virtual object approximates the ground truth of the set of values ​​of the 3D parameters of the object.

7. The method according to any one of claims 1 to 6, in, Obtaining information related to the object from the flat surface image includes applying optical character recognition (OCR) to the flat surface image.

8. An electronic device for processing an image, the electronic device include: First camera 2200; A memory 2300 storing one or more instructions; as well as one or more processors 2400 configured to execute the one or more instructions stored in the memory, The one or more processors 2400 are configured to execute the one or more instructions to: obtaining a first image of a three-dimensional (3D) object including at least one surface having a non-flat shape by using a first camera 2200; identifying a region corresponding to the at least one surface as a region of interest (ROI) by applying the first image to a first artificial intelligence (AI) model; obtaining data about a 3D shape type of the object by applying the first image to a second AI model; obtaining a set of values ​​of 3D parameters associated with at least one of the object, the at least one surface, or the first camera 2200 based on the area identified as the ROI and the data regarding the 3D shape type; estimating the non-planar shape of the at least one surface based on the set of values ​​of the 3D parameters; and By performing perspective transformation on the at least one surface, a flat surface image is obtained in which the non-flat shape of the at least one surface is flattened.

9. The electronic device according to claim 8, in, The set of values ​​of the 3D parameter includes at least one of the following: a height value associated with the 3D shape of the object, The radius value associated with the 3D shape of the object, an angular value of the ROI of the at least one surface of the object, The translation value of the 3D geometric transformation, the rotation value of the 3D geometric transformation, or The focal length of the first camera.

10. The electronic device according to any one of claims 8 to 9, in, training the first AI model to infer a region corresponding to a surface in an image as a ROI, and Wherein, the second AI model is trained to infer the 3D shape type of the object in the image.

11. The electronic device according to any one of claims 8 to 10, in, The one or more processors are further configured to execute the one or more instructions to: receiving input from a user related to a 3D shape type of the object; as well as The 3D shape type of the object is identified by applying a weight to a 3D shape type corresponding to the user's input among a plurality of 3D shape types.

12. The electronic device according to any one of claims 8 to 11, in, The one or more processors are further configured to execute the one or more instructions to: identifying a first keypoint representing the area corresponding to the at least one surface; Obtaining a virtual object corresponding to the 3D shape type of the object and a set of initial values ​​of 3D parameters of the virtual object; adjusting a set of initial values ​​of 3D parameters of the virtual object based on the first key point; as well as A set of adjusted initial values ​​of the 3D parameters of the virtual object is obtained as a set of values ​​of the 3D parameters associated with at least one of the object, the at least one surface, or the first camera.

13. The electronic device according to claim 12, in, The one or more processors are further configured to execute the one or more instructions to: setting a second keypoint representing the area corresponding to the virtual surface of the virtual object; as well as The second keypoint is adjusted to match the first keypoint such that the set of initial values ​​of the 3D parameters of the virtual object approximates the ground truth of the set of values ​​of the 3D parameters of the object.

14. The electronic device according to any one of claims 8 to 13, in, The one or more processors are further configured to execute the one or more instructions to apply optical character recognition (OCR) to the planar surface image.

15. A computer-readable recording medium having recorded thereon a computer program, the computer program performing the method according to any one of claims 1 to 7 when executed by a computer.

Images