Method and server for acquiring text from image

Abstract

A method in which a server acquires text from an image is provided. The method may include the steps of: acquiring an image including a first text and a second text overlapping the first text; separating a first text region corresponding to the first text from the image; extracting pixels corresponding to the first text from the first text region so as to obtain a damaged part and an undamaged part of the first text; and repairing, in the image of the first text, a damaged portion of the first text in which the first text is overlaid by the second text, thereby restoring the first text.

Description

Technical Field

[0001] A server and operating method are provided for separating, extracting and recognizing overlapping text from an image by using a text separation network. Background Art

[0002] Since there are various types of products, text for describing each product is provided thereon. For products with text, the text is printed on the surface of the product, on a label sticker, etc. Some commercially available product labels can be created using a method of additionally printing specific product information on a label template, which has public product information pre-printed thereon. In the process of printing such labels, overlapping text may exist due to errors (e.g., errors in printing coordinates, etc.). When overlapping text is recognized by using a general text recognition method, the text recognition result may be inaccurate. When recognizing text, an algorithm for separating and accurately recognizing overlapping text has recently been used. Summary of the invention

[0003] Technical solutions to the problem

[0004] According to one aspect of the present disclosure, a method for obtaining text from an image performed by a server may include: obtaining an image including a first text and a second text overlapping the first text; separating a first text region corresponding to the first text from the image; extracting pixels corresponding to the first text from the first text region to obtain an undamaged portion and a damaged portion of the first text; and reconstructing the first text by repairing the damaged portion of the first text in the image, wherein the first text overlaps with the second text in the damaged portion.

[0005] According to one aspect of the present disclosure, a server for obtaining text from an image may include: a memory storing one or more instructions; and one or more processors configured to execute the one or more instructions stored in the memory to obtain an image including a first text and a second text overlapping the first text, separate a first text region corresponding to the first text from the image, extract pixels corresponding to the first text region to obtain an undamaged portion and a damaged portion of the first text, and reconstruct the first text by repairing the damaged portion of the first text in the image, wherein the first text overlaps with the second text in the damaged portion.

[0006] According to an aspect of the present disclosure, there may be provided a computer-readable recording medium having recorded thereon a program for executing any one of the methods of obtaining text from an image performed by a server as described above and below. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a schematic diagram illustrating an operation of extracting text from an image by a server according to an embodiment of the present disclosure.

[0008] Figure 2 is a flowchart illustrating an operation of a server obtaining text from an image according to an embodiment of the present disclosure.

[0009] Figure 3 is a diagram for describing an image including overlapping text obtained by a server according to an embodiment of the present disclosure.

[0010] Figure 4 is a diagram for describing an operation of a server preprocessing an image for text separation according to an embodiment of the present disclosure.

[0011] Figure 5 is a diagram for describing a text separation network used by a server according to an embodiment of the present disclosure.

[0012] Figure 6 is a diagram for describing an operation of a server recognizing text in an image according to an embodiment of the present disclosure.

[0013] Figure 7a is a diagram for describing an operation of a server performing text separation using color information according to an embodiment of the present disclosure.

[0014] Figure 7b is a diagram for describing an operation of a server performing text separation using color information according to an embodiment of the present disclosure.

[0015] Figure 8 is a diagram for describing an operation of a server detecting a text area in an image according to an embodiment of the present disclosure.

[0016] Fig. 9 is a diagram for describing an operation of separating a text region from an image by a server according to an embodiment of the present disclosure.

[0017] Fig.10 is a diagram for describing an operation of a server extracting a first text from a text area according to an embodiment of the present disclosure.

[0018] Fig.11a is a diagram for describing an operation of a server repairing an image including damaged text according to an embodiment of the present disclosure.

[0019] Fig.11b is a diagram for describing an operation of a server generating training data for a repairer according to an embodiment of the present disclosure.

[0020] Fig.12 is a diagram for describing a text separation network according to an embodiment of the present disclosure.

[0021] Fig.13ais a diagram for describing an example in which a server according to an embodiment of the present disclosure operates in conjunction with a refrigerator as a home appliance.

[0022] Fig.13b is a diagram for further describing an example in which a server operates in conjunction with a refrigerator as a home appliance.

[0023] Fig.13c is a diagram for describing an operation of a home appliance in conjunction with a server.

[0024] Fig.14a is a diagram for describing an example in which a server according to an embodiment of the present disclosure operates in conjunction with an oven as a home appliance.

[0025] Fig.14b is a diagram for further describing an example in which a server operates in conjunction with an oven as a home appliance.

[0026] Fig.15 is a flowchart of an operating method of a server operating in conjunction with a home appliance according to an embodiment of the present disclosure.

[0027] Fig.16 is a block diagram of a server according to an embodiment of the present disclosure.

[0028] Fig.17 is a block diagram of an electronic device according to an embodiment of the present disclosure.

[0029] Fig.18 is a block diagram of a refrigerator according to an embodiment of the present disclosure.

[0030] Fig.19 is a block diagram of an oven according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Throughout this disclosure, the expression “at least one of a, b, or c” refers to only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0032] The terms used in the present disclosure may be general terms currently widely used in the art based on the functions described in the present disclosure, but may be changed according to the intention of a person skilled in the art, precedent situations, the emergence of new technologies, etc. In addition, the applicant may arbitrarily select some specific terms, and in this case, the meaning of the selected terms will be described in detail in the detailed description of the present disclosure. Therefore, the terms used herein should not be defined by their simple titles, but rather defined based on the meaning of the terms and the overall description of the present disclosure.

[0033] Unless the context clearly states otherwise, the singular expression used herein is also intended to include plural expressions.All terms used herein (including technical or scientific terms) may have the same meaning as those of ordinary skill in the art generally understood. In addition, although various elements or components may be described herein using terms including ordinal numbers such as "first", "second", etc., these elements or components should not be limited by the terms. These terms are only used to distinguish an element or component from another element or component.

[0034] Throughout the specification, when a component "includes" or "comprises" an element, unless there is a specific description to the contrary, it should be understood that the component may also include other elements without excluding other elements. In addition, terms such as "part", "module", etc. described in the specification refer to a unit for processing at least one function or operation, and can be implemented as hardware or software, or a combination of hardware and software.

[0035] Embodiments of the present disclosure will be described more fully below with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, embodiments of the present disclosure may have different forms and should not be construed as being limited to the embodiments thereof set forth herein. In addition, parts irrelevant to the description of the present disclosure are omitted to clearly explain the present disclosure in the accompanying drawings, and the same reference numerals always represent the same elements. In addition, the reference numerals used in the various drawings are only used to describe the various drawings, and the different reference numerals used in the different drawings are not intended to indicate different elements. Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

[0036] In an embodiment of the present disclosure, when an image includes overlapping text, the overlapping text can be separated via color clustering and / or neural networks using information of color channels and text attributes (e.g., text color, text shape, and text font). In addition, any missing pixel information generated by the text separation process can be restored or reconstructed by implantation.

[0037] Figure 1 is a schematic diagram illustrating an operation of extracting text from an image by a server according to an embodiment of the present disclosure.

[0038] refer to Figure 1, the server 2000 according to an embodiment of the present disclosure may obtain the image 100. In this case, the image 100 may include text, and different texts may overlap each other within the image 100. For example, the image 100 obtained by the server 2000 may be a product label image. Here, a product label may be created by printing detailed product information on a basic product label template. As a result, the text included in the product label template (e.g., product type) and the text printed on the product label template (e.g., product name) may overlap each other. The product may be, for example, food (meat, etc.), wine, clothing, textile detergent, etc., but is not limited thereto.

[0039] The server 2000 may separate and extract overlapping texts from the image 100. The server 2000 may separate and extract overlapping texts by using a text separation network, thereby respectively obtaining unit text segments 110 such as letters, words, corpora, sentences, etc. For example, when the image 100 is a product label image, the text segments 110 represent a plurality of pieces of information related to the product.

[0040] In an embodiment of the present disclosure, the server 2000 may recognize text and generate product-related information based on the text segment 110 separated from the image 100. The server 2000 may control a home appliance (e.g., a smart home appliance such as a refrigerator or an oven) by using the product-related information, or provide the product-related information to a user's electronic device (e.g., a user's smart phone).

[0041] In addition, in the present disclosure, the operations performed by the server 2000 may also be performed by the user's electronic device. The electronic device may be a device that captures images (still images and / or videos) through a camera and outputs images on a display, and may receive images from an external device (e.g., an image scanner). For example, the electronic device may include, but is not limited to, a smart TV, a smart phone, a tablet PC, a laptop PC, etc. The electronic device may be implemented as any of various types and forms of electronic devices including a camera and a display. In addition, the electronic device may include a speaker for outputting audio. The electronic device 3000 may separate and extract text directly from the image 100 obtained using the camera of the electronic device 3000, or send the image 100 to the server 2000 to allow the server 2000 to separate and extract text from the image 100.

[0042] An example of the server 2000 performing text separation and extraction operations is described in more detail below with reference to the accompanying drawings.

[0043] Figure 2 is a flowchart illustrating an operation of a server obtaining text from an image according to an embodiment of the present disclosure.

[0044] In operation S210 , the server 2000 obtains an image including a first text and a second text overlapping the first text.

[0045] For example, the server 2000 may receive an image from a user's electronic device (eg, a smartphone, a PC, etc.). For example, the server 2000 may obtain an image stored in a memory within the server 2000.

[0046] The server 2000 may obtain an image including overlapping texts. In the present disclosure, when there are repeated texts in the image, the separated and extracted text is referred to as a first text, and another text overlapping with the first text is referred to as a second text.

[0047] The difference between the first text and the second text is for the convenience of description, and any unit text in the overlapping text can be the first text. For example, when the text "ABC" and the text "DEF" overlap each other, it can be determined that the first text is "ABC" and the second text is "DEF", and the operations for text separation and extraction described below are performed to obtain the first text "ABC" from the overlapping text. However, because the server 2000 separates / extracts all texts in the image, the same or similar operations are performed on all texts in the image. In other words, in the same or similar manner, the server 2000 can identify the first text as "DEF" and the second text as "ABC", thereby separating the first text "DEF" from the overlapping text.

[0048] There may be one or more second texts overlapping with the first text. For example, when a unit text segment overlapping with the first text is called second text-A, another unit text segment overlapping with the first text may be called second text-B.

[0049] In operation S220, the server 2000 separates a text region corresponding to the first text (first text region) from the image. In the present disclosure, a software module used by the server 2000 to separate the text region corresponding to the first text is referred to as a separator.

[0050] The server 2000 may perform text segmentation by using a separator. The server 2000 may determine a plurality of text regions by segmenting the text in the image into a plurality of character groups. The text region refers to a region including characters classified in the same group in the image. The separator may include a first separator and a second separator. The first separator may perform a relatively light text separation operation compared to the second separator, and the second separator may perform a relatively heavy operation compared to the first separator. Therefore, the second separator may achieve a higher text separation accuracy than the first separator.

[0051] In an embodiment of the present disclosure, the server 2000 may obtain a text area map indicating the position of the text in the image, and determine a plurality of text areas based on the text area map. The server 2000 may determine a plurality of text areas based on at least one of a language, a font, a corpus (e.g., weight (g)), or a logo of each text.

[0052] In an embodiment of the present disclosure, the first separator may determine the plurality of text regions by applying a color clustering algorithm so that characters of adjacent colors are grouped.

[0053] In an embodiment of the present disclosure, the second separator may be implemented as an artificial intelligence (AI) model. The separator may be a deep neural network model that receives an original image including text as input and outputs an image including a unit (e.g., a character group) of text. The separator may be implemented using various known deep neural network architectures and algorithms or by modifications of various known deep neural network architectures and algorithms.

[0054] In an embodiment of the present disclosure, when the server 2000 separates a text region corresponding to a first text, the separated text region includes the first text and all or a portion of a second text overlapping the first text.

[0055] In operation S230, the server 2000 extracts pixels related to one or more characters in the first text from the text area corresponding to the first text, thereby obtaining an image of the first text with a damaged portion. That is, the server 2000 extracts pixels corresponding to the first text from the text area of ​​the first text to obtain an undamaged portion and a damaged portion of the first text. In the present disclosure, the software module of the server 2000 for extracting pixels corresponding to characters in the first text is referred to as an extractor.

[0056] The server 2000 may mark pixels corresponding to one or more characters in the first text in the text area separated in operation S220. For example, the server 2000 may mark pixels corresponding to one or more characters in the first text with a label value of 1, and may not mark pixels corresponding to characters in the second text overlapping the first text. Alternatively, for example, the server 2000 may mark pixels corresponding to one or more characters in the first text with a label value of 1, while marking pixels corresponding to characters in the second text overlapping the first text with a label value of 2.

[0057] Server 2000 can obtain an image of the first text by extracting pixels corresponding to characters in the first text from the text area. In this case, because server 2000 does not extract pixels corresponding to characters in the second text overlapping the first text, some pixels corresponding to the characters in the first text may be damaged.

[0058] In operation S240, the server 2000 reconstructs the first text by repairing the area of ​​the first text overlapped by the second text in the image of the first text. That is, the server 2000 repairs the damaged portion of the first text where the first text overlaps with the second text in the image. In the present disclosure, the software module used by the server 2000 to reconstruct the damaged pixels of the characters in the first text is referred to as a repairer.

[0059] In an embodiment of the present disclosure, a repairer for reconstructing damaged pixels may be implemented as an AI model. The repairer may be a deep neural network model that receives an image including damaged pixels as input and outputs an image filled with damaged pixels. The repairer may be implemented using various known deep neural network architectures and algorithms or by modifications of various known deep neural network architectures and algorithms.

[0060] In the following, reference Figure 3 Describes the image including overlay text.

[0061] Figure 3 is a diagram for describing an image including overlapping text obtained by a server according to an embodiment of the present disclosure.

[0062] In an embodiment of the present disclosure, the image including the overlapping text obtained by the server 2000 may be a product label image. Figure 3 , a first image 310 containing Korean text and a second image 320 containing English text are shown as examples.

[0063] The first image 310 is an example of a product label image including Korean text. The first image 310 may show a product label template on which text indicating detailed information of a product is printed. The first image 310 is a product label image including a plurality of text overlapping areas 300. For example, a numeric text indicating an actual price of "9700" may be printed over a Korean text area. (which means "price (KRW)" in English), which is one of the items in the product label template. The text overlap area 300 may occur due to a printing error, resulting in the existence of Korean text in the product label template. Overlapping numbers "9700". In addition, numerical text indicating the actual weight "176" can be printed in Korean text within the product label template (meaning "weight (g)" in English), and in this case, due to an error in the printing process, there may be a text overlap area 300 with overlapping text.

[0064] The second image 320 is an example of a product label image containing English text. The second image 320 may be an image in which text indicating detailed information of the product is printed on the product label template. For example, text indicating detailed information of the product, such as packaging date, shelf life, net weight, unit price, and total price, may be printed on corresponding items in the product label template, and information such as the name / address of a mall selling the product may be printed on the product label template.

[0065] According to an embodiment of the present disclosure, when an image including overlapping text is obtained, the server 2000 can separate the overlapping text to recognize the text. For example, when the text "ABC" and the text "DEF" overlap so that the text becomes unrecognizable as in the above-mentioned text overlapping area 300, the server 2000 can separate the text "ABC" and the text "DEF".

[0066] In addition, including Figure 3 In the present disclosure, Korean text, English text, digital text, etc. will be described as examples of overlapping text. However, this is just an example for convenience of description, and the language of the text is not limited thereto.

[0067] Figure 4 is a diagram for describing an operation of a server preprocessing an image for text separation according to an embodiment of the present disclosure.

[0068] In an embodiment of the present disclosure, the server 2000 may pre-process an image including overlapping text. Figure 4 , a first image 410 including overlapping text and a second image 420 including overlapping text are shown as examples. The server 2000 may pre-process the image based on color channel information of the image.

[0069] In an embodiment of the present disclosure, the server 2000 may erase a specific color in an image. For example, when preprocessing the first image 410, the server 2000 may erase a logo 412 of a specific color in the first image 410. The server 2000 may identify the color of the logo 412 so as to remove the logo 412 of the specific color. The server 2000 may set a first threshold and a second threshold based on the color of the logo 412, and erase pixels having a color between the first threshold and the second threshold, thereby obtaining a preprocessed first image 414.

[0070] In an embodiment of the present disclosure, the server 2000 may keep only a specific color intact in the image. For example, when preprocessing the second image 420, the server 2000 may leave only black or non-color intact in the second image 420 while erasing the remaining colors therein.

[0071] For example, the server 2000 may obtain the preprocessed second image 422 by erasing pixels having a saturation less than a first threshold and a value less than a second threshold in a hue, saturation, and value (HSV) color space. In this case, items 424 having colors other than black may be deleted. Items 424 having other colors may include, for example, text, a logo 426, a background color, etc., but are not limited thereto.

[0072] For example, the server 2000 may leave only non-colors in a red, green, and blue (RGB) color space. The server 2000 may obtain a pre-processed second image 422 by leaving only pixels whose ratios of R, G, and B colors are within a certain range based on a preset threshold. In this case, items 424 having colors other than non-colors may be deleted. Items 424 having other colors may include, for example, text, a logo 426, a background color, etc., but are not limited thereto.

[0073] In an embodiment of the present disclosure, the server 2000 may selectively preprocess the image including the overlapping text. For example, the server 2000 may preprocess the image including the overlapping text as in the above example, and separate the overlapping text from the preprocessed image. Alternatively, for example, the server 2000 may directly separate the overlapping text without preprocessing the image including the overlapping text.

[0074] The operation of separating text by the server 2000 is further described with reference to the following figures. In all the following descriptions, when the server 2000 separates text, preprocessing may be selectively applied.

[0075] Figure 5 is a diagram for describing a text separation network used by a server according to an embodiment of the present disclosure.

[0076] refer to Figure 5 , the server 2000 may separate text from the original image 500 by using the text separation network 510 .

[0077] In an embodiment of the present disclosure, the text separation network 510 may receive the original image 500 as an input and separate overlapping text from the original image 500. The text separation network 510 may include a separator 520, an extractor 530, and a repairer 540, which are modules for processing the original image 500. However, separation into a plurality of modules is merely an example for describing the detailed operation of the text separation network 510, and the present disclosure is not limited thereto. For example, two or more modules included in the text separation network 510 may be combined into one module, or at least one module may be omitted.

[0078] The separator 520 processes the original image 500 to separate text regions corresponding to text in the original image 500. That is, the separator 520 may receive an image and determine a plurality of text regions in the image. Each of the plurality of text regions may be a region including characters in the image that are classified in the same group. For example, one text region may include Korean characters classified in the same group. (meaning "weight (g)" in English) 522. However, Figure 5 Only one of the plurality of text regions is shown, and the separator 520 separates all texts in the image into the text regions.

[0079] In an embodiment of the present disclosure, the separator 520 may be a module that clusters adjacent colors in an image using a color clustering algorithm and groups characters based on the clustering result. The separator 520 may group the colors in the image into K clusters using, for example, a K-means algorithm, but is not limited thereto.

[0080] In an embodiment of the present disclosure, the separator 520 may include one or more AI models.

[0081] The separator 520 may include a text detection model. The text detection model may be an AI model that receives an image as input and outputs a text region map indicating the result of detecting the location of the text in the image. The text region map may be two-dimensional (2D) Gaussian score data indicating the presence of text, but is not limited thereto. In addition, the text detection model may be trained based on training data that includes annotations of ground truth values ​​for text detection, such as characters from the text in the image, correlations between characters, and the like. The text detection model may be implemented using various known deep neural network architectures and algorithms or by modifications of various known deep neural network architectures and algorithms.

[0082] The separator 520 may include a text separation model. The text separation model may be an AI model that receives an image as an input and outputs a result of separating text from the image (eg, a text region image, etc.).

[0083] In an embodiment of the present disclosure, a text separation model can be trained based on annotated training data including ground truth, which is a criterion for text separation, including at least one of the language, font, corpus, or logo of each text. The text separation model can be implemented using various known deep neural network architectures and algorithms or by modifications of various known deep neural network architectures and algorithms.

[0084] In addition, since the original image 500 is an image including overlapping texts, the texts may overlap in the text area separated by the separator 520. For example, when a character group included in a text area is referred to as a first text, the first text and a part of a second text which is a text classified in another character group may be included in the separated text area.

[0085] In an embodiment of the present disclosure, the separator 520 may mark pixels corresponding to characters in the first text. For example, the separator 520 may mark pixels corresponding to characters in the first text with a certain value. The separator 520 may not mark characters in the second text that overlap with the first text, or may mark characters with a value other than the value of the first text.

[0086] The extractor 530 can extract pixels corresponding to the first text in the text area to obtain an image in which a portion of the first text is damaged. Here, when a portion of the first text is damaged, it is understood that only pixels corresponding to the first text are extracted, and pixels corresponding to the second text overlapping with the first text are not extracted.

[0087] In an embodiment of the present disclosure, the extractor 530 may extract only pixels corresponding to the first text based on labels of the pixels corresponding to the first text in the text region, thereby obtaining an image 532 of the first text having a damaged portion.

[0088] The repairer 540 can reconstruct the damaged pixels in the image 532 of the first text. The repairer 540 can be implemented as a deep neural network model that receives an image including damaged pixels as input and outputs an image filled with damaged pixels. For example, the repairer 540 can receive the damaged image 532 of the first text and output the reconstructed image 542 of the first text.

[0089] The repairer 540 can be trained based on training data obtained by overlapping the noise text with the real text. The repairer 540 can be implemented using various known deep neural network architectures and algorithms or by modifications of various known deep neural network architectures and algorithms.

[0090] refer to Figure 6 to Figure 11bThe operation of the server 2000 to separate overlapping texts by using the text separation network 510 and the modules (separator 520, extractor 530, and repairer 540) of the text separation network 510 is further described.

[0091] Figure 6 is a diagram for describing an operation of a server recognizing text in an image according to an embodiment of the present disclosure.

[0092] In an embodiment of the present disclosure, the text separation network 600 may include a first separator 610 and a second separator 620 .

[0093] The first separator 610 may be a module that clusters adjacent colors in an image using a color clustering algorithm and groups characters based on the clustering result. The color clustering algorithm may be used to detect text from an image by grouping pixels with similar colors (e.g., adjacent colors in a color space) together. As a method for quantifying color similarity, the Euclidean distance between different colors in a color space may be calculated, and colors with a distance smaller than a predetermined distance may be considered similar colors and assigned to the same color group. Alternatively, or in combination with a method based on Euclidean distance, a color histogram representing the distribution of colors in an image may be used to determine similar colors. The first separator 610 may group the colors in an image into K clusters using, for example, a K-means algorithm, but is not limited thereto. The first separator 610 may send a text region including grouped characters to an extractor 630.

[0094] The second separator 620 may include a text separation model. The text separation model may be an AI model that receives an image as input and outputs a result of separating text from the image (e.g., a text region including grouped characters). The second separator 620 may send the text region including a group of characters to the extractor 630.

[0095] The first separator 610 performs a relatively light text separation operation compared to the second separator 620, and the second separator 620 performs a relatively heavy operation compared to the first separator 610. Therefore, the second separator 620 can achieve higher text separation accuracy than the first separator 610.

[0096] The server 2000 may determine whether to use the first separator 610 or the second separator 620 of the text separation network 600 based on a preset condition. Hereinafter, the preset conditions for using the first separator 610 and the second separator 620 are described.

[0097] In an embodiment of the present disclosure, the server 2000 may obtain an input image. The input image may contain overlapping text. The server 2000 may receive a user input for specifying a specific area of ​​the image (S610). For example, the server 2000 may receive a user input for specifying a specific area of ​​the image from an electronic device of the user (e.g., a smart phone or a PC). The user input may be an input for specifying a text area containing overlapping text within the image. The server 2000 may perform text separation based on the user input for specifying a text area containing overlapping text. In an embodiment of the present disclosure, the server 2000 may first perform text recognition (e.g., optical character recognition (OCR) etc.) on the input image. When the text recognition result of the input image is not obtained due to text overlap, the server 2000 may determine to use the first separator 610 to obtain the text recognition result. When the server 2000 uses the text separation network 600 for the first time, the priority of the first separator 610 may be set to be higher than the priority of the second separator 620.

[0098] The server 2000 may perform text separation by using the first separator 610. For example, the server 2000 may perform color clustering by using the first separator 610 and separate texts of different colors based on the color clustering. For example, the server 2000 may separate black text and red text.

[0099] When there is no user input, the server 2000 extracts a text area map of the input image (S620). The server 2000 may obtain the text area map by using a text detection model. The text detection model may be an AI model that receives an input image and outputs a text area map indicating the result of detecting the position of text in the image.

[0100] The server 2000 may apply a color clustering algorithm to the text region in the image based on the text region map (S630). The server 2000 may obtain color clusters for each of the text regions present in the image based on the text region map, and determine whether to use the first separator 610 or the second separator 620 based on whether the number of the obtained color clusters is greater than or equal to a preset number N (e.g., three).

[0101] When the number of color clusters is greater than or equal to the preset number N, the server 2000 may use the first separator 610. When the number of color clusters is greater than or equal to the preset number N, it means that there are three or more texts of different colors in the corresponding text area. In this case, the server 2000 can separate the text based on the color of the text by using the first separator 610. For example, when the second text overlaps with the first text, and the first text and the second text have different colors, the server 2000 can separate the first text and the second text. The server 2000 can extract pixels corresponding to characters in the first text by using the extractor 630. Here, the server 200 may not extract pixels corresponding to characters in the second text overlapping with the first text, and as a result, an image in which a part of the first text is damaged or incomplete due to the overlapping second text can be generated. In order to solve this problem, the server 2000 can reconstruct the damaged part of the first text in the image by using the repairer 640. The repairer 640 can reconstruct the first text by filling the missing part of the damaged part of the first text, and an image including the reconstructed first text can be output. According to an embodiment of the present disclosure, the server 2000 may use the extractor 630 to separate the first text from the image while ignoring the overlapping second text, and the server 2000 may use the repairer 640 to repair and reconstruct any damaged areas of the first text caused by the overlap, thereby generating an image showing the restored first text.

[0102] The server 2000 may perform text recognition (S640) based on the reconstructed image obtained from the repairer 640. For example, the server 2000 may recognize the text (e.g., common characters, special characters, symbols, etc.) included in the image by using an OCR model. The text obtained by the server 2000 via text recognition may include information related to the product. For example, when the input image is an image of a product label, the server 2000 may obtain the text related to the product included in the product label as product information. In an embodiment of the present disclosure, text recognition may be performed by an external device connected to the server 2000. For example, the server 2000 may send an image including text obtained by the text separation network 600 to an electronic device of the user (e.g., a smart phone, etc.). In this case, OCR may be performed by the electronic device of the user.

[0103] In an embodiment of the present disclosure, the server 2000 can verify the validity of the text recognition result (S650). For example, the server 2000 can determine the validity of the OCR result based on whether the recognition confidence included in the OCR result is greater than or equal to a preset value. For example, the server 2000 can determine whether the text recognition result is valid based on a preset standard. Specifically, the server 2000 can determine whether the recognized text satisfies a predetermined format, i.e., a 20-digit number representing an identification number for identifying a product. For example, the server 2000 can verify whether the recognized text numbers, corpora, words, etc. are correctly recognized. When it is determined that the text recognition result is invalid, the server 2000 can use the second separator 620 to perform more accurate text separation and recognition again.

[0104] When the number of color clusters is less than a preset number, the server 2000 may use the second separator 620. In addition, the server 2000 may use the second separator 620 based on the validity of the text recognition result, as described above.

[0105] By using the second separator 620, the server 2000 can separate the text by determining the text area based on at least one of the language, font, corpus, or logo of each text. The second separator 620 can be an AI model that receives an original image including text as input and outputs an image including a unit (e.g., a character group) of the text. The output of the second separator 620 is sequentially sent to the extractor 630 and the repairer 640, so that the same operation as the case of separating and recognizing the text by using the first separator 610 can be performed again.

[0106] Figure 7a is a diagram for describing an operation of a server performing text separation using color information according to an embodiment of the present disclosure.

[0107] refer to Figure 7a and Figure 7b The separator described can correspond to Figure 6 The first separator 610 is provided.

[0108] In an embodiment of the present disclosure, the server 2000 can filter out the color of objects other than text (e.g., background, logo, etc.) from the image by using a color clustering algorithm. By using a separator, the server 2000 can perform color clustering by highlighting the black color in the original image 710 including the overlapping text.

[0109] In an embodiment of the present disclosure, the server 2000 may convert the original image 710 into a cyan, magenta, yellow, and black (CYMK) color space. In the CYMK color space of the original image 710, the server 2000 may filter out the color based on the threshold of the K channel representing black to obtain a first filtered image 720 (step 1). As a result of the first filtering of the original image 710 by the server 2000, bright colors (e.g., yellow, etc.) and the like may be removed from the original image 710.

[0110] In an embodiment of the present disclosure, the server 2000 may convert the first filtered image 720 into a Lab color space. The server 2000 may perform color clustering in the Lab color space and extract information about clusters adjacent to black in the color clusters, thereby obtaining a secondary filtered image 730 (step 2).

[0111] According to an embodiment of the present disclosure, the server 2000 may extract pixels corresponding to text and / or characters in the text from the secondary filtered image 730 and reconstruct damaged pixels.

[0112] Figure 7b is a diagram for describing an operation of a server performing text separation using color information according to an embodiment of the present disclosure.

[0113] In an embodiment of the present disclosure, the server 2000 can filter the color of objects (e.g., background, logo, etc.) other than text in the image by using a color clustering algorithm. For example, the server 2000 can obtain an original image 712 including overlapping text, and obtain a color distribution 714 of the original image 712. The server 2000 can obtain, for example, a color distribution graph, an RGB histogram, etc., but the present disclosure is not limited thereto. The server 2000 can determine a target number N based on the color distribution 700 to group the colors in the image into N clusters. For example, the server 2000 can determine N primary colors in the image based on the color distribution to perform color clustering, but the present disclosure is not limited thereto. For example, the server 2000 can receive a target number N from a user.

[0114] The server 2000 may perform color clustering iteratively. For example, the server 2000 may perform a first color clustering on the original image 712 to obtain a first filtered image 722 having a reduced number of colors compared to the original image 710. Specifically, the server 2000 may group the colors in the original image 712 into three clusters and filter out the colors not included in the three clusters. Then, the server 2000 may perform a second color clustering on the first filtered image 722 to accurately filter out the colors from the first filtered image 722. In detail, the server 2000 may group the colors in the first filtered image 722 into three clusters and filter out the colors not included in the three clusters, thereby obtaining a secondary filtered image 732.

[0115] According to an embodiment of the present disclosure, the server 2000 may extract pixels corresponding to text and / or characters in the text from the secondary filtered image 732 and reconstruct damaged pixels.

[0116] Figure 8 is a diagram for describing an operation of a server detecting a text area in an image according to an embodiment of the present disclosure.

[0117] In an embodiment of the present disclosure, the server 2000 may detect an area corresponding to text in an image. The server 2000 may detect text by using a separator. The text detection model may be an AI model that receives an input image 800 and outputs a text area map 810 indicating a result of detecting a position of text in the input image 800.

[0118] In an embodiment of the present disclosure, the server 2000 may train a text detection model based on a training data set including training images. The training images may be marked with text boxes around characters from the text in the image. In addition, the training images may be marked with affinity boxes for linking related characters from the text in the image. The related characters may be characters that constitute a word or a sentence, but are not limited thereto.

[0119] The text detection model may calculate a 2D Gaussian score indicating the presence of text in the input image 800. The server 2000 may output a text region map 810 indicating a region where text exists in the input image 800 based on the 2D Gaussian score.

[0120] In an embodiment of the present disclosure, the server 2000 may obtain a text area map 810 of the input image 800 and perform a text separation operation with a separator based on the text area map 810. In this case, the separator used by the server 2000 may also receive the text area map 810 as well as the input image 800.

[0121] In an embodiment of the present disclosure, a text detection model may be included in the separator. For example, the first separator and / or the second separator may include a text detection model to first perform text detection and then perform the operation of the first separator and / or the second separator.

[0122] Fig. 9 is a diagram for describing an operation of separating a text region from an image by a server according to an embodiment of the present disclosure.

[0123] refer to Fig. 9 The separator 900 described may correspond to Figure 6 The second separator 620 is provided.

[0124] In an embodiment of the present disclosure, the server 2000 may separate a plurality of text regions from the input image 902 by using the separator 900 .

[0125] For example, the server 2000 can obtain the first text area 910 from the input image 902. The first text area 910 may include a first text and a second text. Here, the first text refers to a text corresponding to a character group (e.g., a word, a sentence, a corpus, etc.), and the second text refers to a text separated together when the first text area is separated from the input image 902. The second text may overlap with the first text. Therefore, when reconstructing the first text, the second text is considered to be noise relative to the first text. For example, at least a portion of the second text may overlap with the first text. In order to identify the first text in the first text area 910, the server 2000 may only extract pixels corresponding to the first text and reconstruct pixels damaged due to the overlap of the first text with the second text.

[0126] The server 2000 separates text regions respectively corresponding to all texts included in the input image 902. For example, the server 2000 may obtain a second text region 912. The second text region 912 may include a first text corresponding to one complete character group and a second text regarded as noise relative to the first text in the second text region 912. In addition, the server 2000 may obtain a third text region 914. The third text region 914 may include a first text and a second text.

[0127] According to an embodiment of the present disclosure, when separating a plurality of text regions by using the separator 900, the server 2000 may distinguish between the first text and the second text included in each of the text regions. For example, the separator 900 may be an AI model trained to assign different labels to the first text and the second text.

[0128] In an embodiment of the present disclosure, separator 900 can be trained to distinguish the language of the text. For example, separator 900 can distinguish the language of the text (e.g., English, Korean, etc.) to separate text areas based on different languages. In some embodiments of the present disclosure, server 2000 can obtain user input for specifying a language. In this case, separator 900 can additionally use user input as input data.

[0129] In an embodiment of the present disclosure, the separator 900 may be trained to distinguish the fonts of the text. For example, the separator 900 may distinguish the fonts of the text to separate text regions based on different fonts.

[0130] In an embodiment of the present disclosure, the separator 900 can be trained to distinguish corpora of text. For example, the separator 900 can distinguish corpora of text (e.g., weight (g)) to separate text regions based on different corpora. In this case, even when the characters included in the corpus are characters of different languages ​​(e.g., Korean and English alphabets) or have different fonts, the corpus can be distinguished as a single corpus.

[0131] In order to distinguish the language, font, corpus, and logo of each text from the language, font, corpus, and logo of other texts through the separator 900, the server 2000 can train the separator 900 based on training data annotated by applying labels to pixels corresponding to characters in a character group.

[0132] According to an embodiment of the present disclosure, the server 2000 may obtain a plurality of text regions from an image, and extract and / or reconstruct the first text included in each of the text regions for each text region.

[0133] Fig.10 is a diagram for describing an operation of a server extracting a first text from a text area according to an embodiment of the present disclosure.

[0134] In an embodiment of the present disclosure, when the server 2000 separates text by using a separator, a text region is obtained. The server 2000 determines the text region by determining a character group including characters based on a word, a sentence, a corpus, etc. The server 2000 may extract pixels corresponding to the text from the text region by using the extractor 1000. For ease of description, Fig.10 Only one character from the group of characters in the text area is shown instead of all characters.

[0135] For example, a first text region 1010 containing English text may include a first text 1002 and a second text 1004. Here, the first text 1002 may be an alphabetic letter "A", and the second text 1004 overlapping the first text 1002 may be an alphabetic letter "L". The first text region 1010 may include information 1020 about the first text 1002 and the second text 1004. For example, different labels may be assigned to characters from the first text 1002 and characters from the second text 1004.

[0136] Based on the first text region 1010 and the information 1020 about the first text 1002 and the second text 1004 included in the first text region 1010 , the server 2000 may obtain an image 1030 in which a portion of the first text 1002 in the first text region 1010 is damaged.

[0137] In the same manner, the server 2000 may obtain a second text region 1012 including Korean text. The second text region 1012 may include the first text 1002 and the second context 1004. In addition, the server 2000 may obtain information 1022 about the first text 1002 and the second text 1004 included in the second text region 1012. Based on the second text region 1012 and / or the information 1022 about the first text 1002 and the second text 1004 included in the second text region 1012, the server 2000 may obtain an image 1032 in which a portion of the first text 1002 in the second text region 1012 is damaged.

[0138] The server 2000 can repair the image in which a portion of the first text 1002 in the text area is damaged. Fig.11a and Fig.11b Describe the repair further.

[0139] Fig.11a is a diagram for describing an operation of a server repairing an image including damaged text according to an embodiment of the present disclosure.

[0140] In an embodiment of the present disclosure, the server 2000 may reconstruct damaged pixels in an image by using the repairer 1100 .

[0141] The repairer 1100 can reconstruct the first text by repairing the area where the first text is overlapped by the second text. Fig.10In the depicted example, the first text may be an alphabetic letter "A", and the second text overlapping the first text may be an alphabetic letter "L". In this case, the repairer 1100 may receive as input an image 1110 in which a partial area of ​​the first text is damaged, and output a repaired image 1120 in which the damaged portion is reconstructed.

[0142] In the same manner, the server 2000 may reconstruct the Korean text. The server 2000 may input the image 1112 in which a portion of the Korean characters in the first text is damaged to the repairer 1100, and obtain a repaired image 1122 in which the damaged portion is reconstructed.

[0143] In an embodiment of the present disclosure, the server 2000 may perform preprocessing before reconstructing the image including the damaged portion by using the repairer 1100. For example, the server 2000 may reduce the number of color channels to one by converting the image including the damaged portion into grayscale. Therefore, the computational complexity may be reduced by reducing the number of color channels.

[0144] The repairer 1100 may be implemented using various known deep neural network architectures and algorithms or by modifications of various known deep neural network architectures and algorithms. According to an embodiment of the present disclosure, the server 2000 may generate training data for training the repairer 1100 and train the repairer 1100 by using the training data set. Fig.11b The generation of training data for the repairer 1100 is further described.

[0145] Fig.11b is a diagram for describing an operation of a server generating training data for a repairer according to an embodiment of the present disclosure.

[0146] In an embodiment of the present disclosure, the server 2000 may generate training data 1150 for training the repairer 1100. The server 2000 may generate the training data 1150 based on the ground truth data 1130 and the noise data 1140.

[0147] The server 2000 may generate the training data 1150 by overlapping the noise data 1140 with the ground truth data 1130. For example, the server 2000 may generate the training data 1150 by removing pixels in the noise data 1140 that overlap with the ground truth data 1130 from the ground truth data 1130, wherein a portion of the ground truth data 1130 is damaged. Here, the ground truth data 1130 corresponds to the first text in the above-described embodiment of the present disclosure, and the noise data 1140 corresponds to the second text.

[0148] In an embodiment of the present disclosure, when generating training data 1150, server 2000 may generate training data 1150 based on at least one of the language, font, or corpus of the text. Server 2000 may generate training data 1150 by using text with a high probability of overlap as ground truth data 1130 and noise data 1140. Text with a high probability of overlap may be determined based on the correlation between texts. For example, text with a high probability of overlap may be preset based on at least one of the language, font, or corpus of each text. For example, based on user input, text with a high probability of overlap may have been set as text with a high probability of overlap. For example, in a case where a sirloin product is sold in a shopping mall and the product label has an overlap between the brand name "AMart" and the product name "sirloin", "sirloin" and "A-MART" may be set as text with a high probability of overlap. Although Fig.11b An example of generating the training data 1150 using "SIRLOIN" as the ground truth data 1130 and using "A-MART" as the noise data 1140 is shown, but the training data 1150 may be generated by setting "SIRLOIN" as the noise data 1140 and setting "A-MART" as the ground truth data 1130. In the same manner, the server 2000 may generate the training data 1150 by using the ground truth data 1130 and the noise data 1140, which are Korean texts and are set based on the correlation between the texts.

[0149] Fig.12 is a diagram for describing a text separation network according to an embodiment of the present disclosure.

[0150] In an embodiment of the present disclosure, the text separation network may be divided into a first text separation network 1200 and a second text separation network 1202 .

[0151] The first text separation network 1200 may include a separator 1210, an extractor 1220, and a repairer 1230. Since the separator 1210, the extractor 1220, and the repairer 1230 included in the first text separation network 1200 have been described above with reference to the aforementioned drawings, their description will not be repeated below.

[0152] The second text separation network 1202 may include only a separator 1210 and an extractor 1220. The separator 1210 and the extractor 1220 included in the second text separation network 1200 perform the same functions as the separator 1210 and the extractor 1220 in the first text separation network 1200, respectively. Because the second text separation network 1202 does not include a repairer 1230, the second text separation network 1202 requires a relatively small amount of computation for the text separation operation compared to the first text separation network 1200. That is, the second text separation network 1202 may be a simplified version of the first text separation network 1200.

[0153] The server 2000 may selectively use the first text separation network 1200 or the second text separation network 1202 .

[0154] In an embodiment of the present disclosure, the server 2000 may give priority to using the second text separation network 1202 and use the first text separation network 1200 as a second priority. The server 2000 may obtain an image including separated text by using the second text separation network 1202. The server 2000 may recognize the text (e.g., common characters, special characters, symbols, etc.) included in the image by using an OCR model. The server 2000 may determine the validity of the OCR result as a text recognition result based on whether the recognition confidence of the OCR result is greater than or equal to a preset value. When it is determined that the text recognition result is invalid, the server 2000 may use the first text separation network 1200 further including a repairer 1230 to perform more accurate text separation and text recognition again.

[0155] In an embodiment of the present disclosure, in order to output the calculation result faster, the server 2000 may perform text separation by using the second text separation network 1202 instead of the first text separation network 1200. In this case, the server 2000 may provide the user with an image of the unrepaired text, or provide the user with the result of recognizing the text based on the image of the unrepaired text.

[0156] In an embodiment of the present disclosure, the server 2000 may determine whether to use the first text separation network 1200 or the second text separation network 1202 based on user input. For example, the user may choose to use the second text separation network 1202 to use less data.

[0157] In an embodiment of the present disclosure, the server 2000 may determine whether to use the first text separation network 1200 or the second text separation network 1202 based on user information. For example, the server 2000 may provide the second text separation network 1202 as a trial version to free users.

[0158] In an embodiment of the present disclosure, text separation and recognition can be performed by the user's electronic device (e.g., a smart phone). Because the user's electronic device has relatively low computing performance compared to the server 2000, the user's electronic device can perform text separation and recognition by using the second text separation network 1202. For example, when the user captures an image of a label including text by using a smart phone, the smart phone can separate overlapping text by using the second text separation network 1202. When the overlapping text is separated, the smart phone can recognize the separated text by using OCR.

[0159] Fig.13a is a diagram for describing an example in which a server according to an embodiment of the present disclosure operates in conjunction with a refrigerator as a home appliance.

[0160] refer to Fig.13a , the server 2000 can perform data communication with the user's electronic device 3000 and the refrigerator 4000.

[0161] In an embodiment of the present disclosure, a user may capture an image of a product label by using the user's electronic device 3000. Figures 13a to 13c An example is described in which the product label of which the user has captured an image is a label of a meat product, but the type of product is not limited thereto. For example, the product may be a food product. The server 2000 may receive an image of the product label from the user's electronic device 3000. In this case, the image of the product label may include overlapping text.

[0162] The server 2000 may separate the overlapping text from the image and recognize the separated text to generate information related to the product. For example, the server 2000 may generate information related to the meat product, such as product name, weight, price, barcode, product identification number, manufacturing date, manufacturer, distributor, expiration date, etc. The server 2000 may send the information related to the product to the user's electronic device 3000 and / or refrigerator 4000.

[0163] The server 2000 may also control the refrigerator 4000 based on the information related to the product. For example, the server 2000 may receive an input of a location within the refrigerator 4000 where meat is to be stored, or determine a location where meat is to be stored (e.g., an empty space in the refrigerator 4000). The server 2000 may also control the refrigerator 4000 to operate in a meat storage mode that sets an optimal temperature, etc., at the determined storage location.

[0164] Fig.13b is a diagram for further describing an example in which a server operates in conjunction with a refrigerator as a home appliance.

[0165] In an embodiment of the present disclosure, a user can capture an image of a meat product 1301 by using an electronic device 3000 and obtain a label image of the meat product 1301. The server 2000 can obtain the label image of the meat product 1301 from the electronic device 3000. The server 2000 can perform text separation and text recognition on the label image of the meat product 1301 to provide information related to the meat product 1301 to the electronic device 3000.

[0166] In an embodiment of the present disclosure, the user's electronic device 3000 may provide an application that can remotely manage various home appliances at home. The application may be, for example, an application that can manage / control the refrigerator 4000, but is not limited thereto.

[0167] Referring to the first screen 1310 of the application installed on the electronic device 3000, the application may include an interface for product label recognition. For example, the first screen 1310 may include a meat label scanning button 1312, but is not limited thereto.

[0168] In response to the user selecting the meat tag scanning button 1312, the electronic device 3000 may display a second screen 1320 that allows product tag identification to be performed. For example, the electronic device 3000 may display a quadrilateral frame 1322 (however, its shape is not limited to a quadrilateral, and may include other shapes that can provide similar functions, such as a circle) for guiding the product tag to be included inside the second screen 1320, and display a guide such as "Capture an image of a meat tag". In some embodiments of the present disclosure, when an object is not recognized in the image displayed on the second screen 1320, the electronic device 3000 may display a guide such as "Please point the camera at the product". The electronic device 3000 may display a preview image obtained from the camera on the second screen 1320. While viewing the second screen 1320, the user may adjust the field of view of the camera so that the product label is completely included in the image of the quadrilateral frame 1322.

[0169] The electronic device 3000 may send an image of a product label to the server 2000. For example, a user of the electronic device 3000 may send an image of a meat label to the server 2000 via an application. The electronic device 3000 may receive information related to a product (e.g., meat product 1301) generated by the server 2000 based on the image of the product label. The electronic device 3000 may display product-related information received from the server 2000 on a third screen 1330. For example, the electronic device 3000 may display information 1332 indicating the meat product 1301, such as the type of meat, country of origin, cut, weight, individual traceability number, etc. For example, the electronic device 3000 may display action information 1334 related to the control of a household appliance, such as storing meat. The household appliance may be, for example, a refrigerator 4000, but is not limited thereto.

[0170] The electronic device 3000 may display a fourth screen 1340 showing action information 1334 about a household appliance associated with a product. For example, when the user selects an action of "storing meat" on the third screen 1330 to store meat in the refrigerator 4000, the electronic device 3000 may display a fourth screen 1340 including items related to storing meat. For example, a photo of meat, a product name, a receipt date, a storage location, etc. may be displayed on the fourth screen 1340, but is not limited thereto. In addition, the fourth screen 1340 may include an interface via which the electronic device 3000 or the server 2000 controls the refrigerator 4000. For example, the electronic device 3000 may display a button 1342 for controlling the refrigerator 4000 (e.g., "start storing in the refrigerator") on the fourth screen 1340, but is not limited thereto.

[0171] When the user selects the button 1342 for controlling the refrigerator 4000, the electronic device 3000 may provide the user with a guide 1352 related to the manipulation of the refrigerator 4000. For example, the electronic device 3000 may display the guide 1352 on the fifth screen 1350, which enables the user to store meat in the refrigerator 4000 (e.g., "The refrigerator multi-storage chamber has activated the meat storage mode. Please put the meat in").

[0172] The electronic device 3000 may transmit a control command for the refrigerator 4000 to control the refrigerator 4000 to operate the multi-storage room space in the meat storage mode. The electronic device 3000 may transmit the control command for the refrigerator 4000 to the refrigerator 4000 via the server 2000 or directly to the refrigerator 4000.

[0173] Fig.13c is a diagram for describing an operation of a home appliance in conjunction with a server.

[0174] In an embodiment of the present disclosure, the server 2000 may operate in conjunction with the refrigerator 4000, as described above with reference to Fig.13a and Fig.13b In detail, the server 2000 may transmit the information related to the product to the refrigerator 4000. In this case, the information related to the product may be displayed on the display 4910 of the refrigerator 4000.

[0175] For example, when the product label is a meat product label as in the above example, information indicating the meat product, such as the type of meat, country of origin, cut, weight, individual traceability number, etc. can be displayed on the display 4910 of the refrigerator 4000.

[0176] For example, when the user stores meat corresponding to the identified meat product label in the refrigerator 4000, the display 4910 of the refrigerator 4000 may display information related to the meat product, such as the location where the meat is stored, the storage time period, the storage temperature, the operation mode, etc. Alternatively, the display 4910 of the refrigerator 4000 may display various information related to the product, such as text, images, and videos indicating meat product recipes, recommendations for ingredients related to meat dishes, and links to purchase ingredients.

[0177] Fig.14a is a diagram for describing an example in which a server according to an embodiment of the present disclosure operates in conjunction with an oven as a home appliance.

[0178] refer to Fig.13a , the server 2000 can perform data communication with the user's electronic device 3000 and the oven 5000.

[0179] In an embodiment of the present disclosure, a user may capture an image of a product label by using the user's electronic device 3000. Fig.14a and Fig.14b An example is described in which the product label of which the user captures an image is a label of a meat product, but the type of product is not limited thereto. For example, the product may be a food product. The server 2000 may receive an image of the product label from the user's electronic device 3000. In this case, the image of the product label may include overlapping text.

[0180] The server 2000 may separate the overlapping text from the image and recognize the separated text to generate information related to the product. For example, the server 2000 may generate information related to the meat product, such as product name, weight, price, barcode, product identification number, manufacturing date, manufacturer, distributor, expiration date, etc. The server 2000 may send the information related to the product to the user's electronic device 3000 and / or the oven 5000.

[0181] Server 2000 may control oven 5000 based on the information related to the product. For example, server 2000 may determine temperature, time, and operation mode (e.g., preheating mode, convection mode, etc.) as settings for steak cooking. Server 2000 may transmit the determined settings related to the control operation for oven 5000 to oven 5000 so that the settings for steak cooking are automatically applied in oven 5000.

[0182] Fig.14b is a diagram for further describing an example in which a server operates in conjunction with an oven as a home appliance.

[0183] In an embodiment of the present disclosure, a user can capture an image of a meat product 1401 by using an electronic device 3000 and obtain a label image of the meat product 1401. The server 2000 can obtain the label image of the product from the electronic device 3000 and perform text separation and text recognition on the label image to provide information related to the product to the electronic device 3000.

[0184] In an embodiment of the present disclosure, the user's electronic device 3000 may provide an application that can remotely manage various home appliances at home. The application may be, for example, an application that can manage / control the oven 5000, but is not limited thereto.

[0185] Referring to the first screen 1410 of the application installed on the electronic device 3000, the application may include an interface for product label recognition. For example, the first screen 1410 may include a meat label scanning button 1412, but is not limited thereto.

[0186] In response to the user selecting the meat tag scanning button 1412, the electronic device 3000 may display a second screen 1420 that allows product tag identification to be performed. For example, the electronic device 3000 may display a quadrilateral frame 1422 (however, its shape is not limited to a quadrilateral, and other shapes that can provide similar functions, such as a circle) for guiding the product tag to be included inside the second screen 1420, and display a guide such as "Capture an image of a meat tag". In an embodiment of the present disclosure, when an object is not recognized in the image displayed on the second screen 1420, the electronic device 3000 may display a guide such as "Please point the camera at the product". The electronic device 3000 may display a preview image obtained from the camera on the second screen 1420. While viewing the second screen 1420, the user may adjust the field of view of the camera so that the product tag is completely included in the image of the quadrilateral frame 1422.

[0187] The electronic device 3000 may transmit an image of a product label to the server 2000, and display information related to the product received from the server 2000 on a third screen 1430. For example, the electronic device 3000 may display information 1432 indicating the meat product 1401, such as the type of meat, country of origin, cut, weight, individual traceability number, etc. For example, the electronic device 3000 may display action information 1434 related to the control of a home appliance, such as thawing meat, cooking, recipes, etc. The home appliance may be, for example, an oven 5000, but is not limited thereto.

[0188] The electronic device 3000 may display a fourth screen 1440 showing action information 1434 about a home appliance associated with a product. For example, when the user selects the "cooking" action on the third screen 1430 to cook a steak with meat, the electronic device 3000 may display a fourth screen 1440 including information related to steak cooking. For example, recipes and settings and preparation items for cooking may be displayed on the fourth screen 1440, but the present disclosure is not limited thereto. In addition, the fourth screen 1440 may include an interface via which the electronic device 3000 or the server 2000 controls the oven 5000. For example, the electronic device 3000 may display a button 1442 (e.g., "start cooking in the oven") for controlling the oven 5000 on the fourth screen 1440, but is not limited thereto.

[0189] When the user selects button 1442 for controlling oven 5000, electronic device 3000 may provide the user with guide 1452 related to the manipulation of oven 5000. For example, electronic device 3000 may display guide 1452 “Oven setting for steak cooking is complete. Please put meat in the oven” on fifth screen 1450, enabling the user to cook steak.

[0190] Electronic device 3000 may transmit a control command for oven 5000 to server 2000 or oven 5000 to set a cooking operation mode of oven 5000 .

[0191] Fig.15 is a flowchart of an operating method of a server operating in conjunction with a home appliance according to an embodiment of the present disclosure.

[0192] refer to Fig.15 According to an embodiment of the present disclosure, the server 2000 may perform data communication with the electronic device 3000 and the home appliance 1500. The electronic device 3000 may include a smart phone as the above example, but is not limited thereto. The home appliance 1500 may include a refrigerator 4000 and an oven 5000, which are the above examples, but are not limited thereto.

[0193] In operation S1510, the electronic device 3000 may capture a product label image. The electronic device 3000 may capture the product label image via a user's camera manipulation. In this case, the product label image may include overlapping text.

[0194] In operation S1515 , the electronic device 3000 may transmit the product label image to the server 2000 .

[0195] In operation S1520, the server 2000 may separate the text from the product label image. The server 2000 may separate and extract overlapping text from the product label image by using a text separation network. The server 2000 may separate the text by using a first text separation network including a separator, an extractor, and a repairer. The server 2000 may separate the text by using a second text separation network including a separator and an extractor. The separator may include a first separator using a color clustering algorithm and a second separator including a text separation model as an AI model. The server 2000 may recognize the separated text. The server 2000 may generate information related to the product based on the recognized text. The information related to the product may include, for example, product name, weight, price, barcode, product identification number, manufacturing date, manufacturer, distributor, expiration date, etc., but is not limited thereto.

[0196] In operation S1525 , the server 2000 may transmit information related to the product to the electronic device 3000 .

[0197] In operation S1530, the electronic device 3000 may display information related to the product. Fig.13b and Fig.14b The operation of the electronic device 3000 displaying the information related to the product has been described, so the description already provided above will be omitted here.

[0198] In operations S1532, S1534, and S1536, the electronic device 3000 and / or the server 2000 may control the home appliance 1500. For example, the electronic device 3000 may send a home appliance control request to the server 2000 (S1532), and the server 2000 may send a home appliance control command to the home appliance 1500 (S1534). Alternatively, the electronic device 2000 may send a home appliance control command to the home appliance 1500 (S1536).

[0199] In operation S1540, the home appliance 1500 may perform an operation corresponding to the home appliance control command. For example, when the home appliance 1500 is the refrigerator 4000, the temperature thereof may be set according to the operation mode of the refrigerator 4000, and information related to the stored product may be stored in the storage location. For example, when the home appliance 1500 is the oven 5000, settings related to cooking, such as temperature, etc. may be performed according to the operation mode of the oven 5000.

[0200] Fig.16 is a block diagram of a server according to an embodiment of the present disclosure.

[0201] According to an embodiment of the present disclosure, the server 2000 may include a communication interface 2100 , a memory 2200 , and a processor 2300 .

[0202] The communication interface 2100 may include a communication circuit. The communicator 2100 may include, for example, a communication circuit capable of performing data communication between the server 2000 and other devices by using at least one of a data communication method including a wired local area network (LAN), a wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), an infrared data association (IrDA), Bluetooth low energy (BLE), a near field communication (NFC), wireless broadband Internet (WiBro), a world interoperability for microwave access (WiMAX), a shared wireless access protocol (SWAP), a wireless gigabit alliance (WiGig), and a radio frequency (RF) communication.

[0203] The communication interface 2100 can send data required to perform the operation of the server 2000 to the external electronic device and receive data required to perform the operation of the server 2000 from the external electronic device. For example, the server 2000 can use the communication interface 2100 to receive an image including overlapping text from an external electronic device (e.g., a user's smart phone, etc.), and send product-related information generated by text separation and recognition to the external electronic device. In addition, the server 2000 can send data required to control the home appliance to the home appliance and receive data required to control the home appliance from the home appliance via the communication interface 2100.

[0204] The memory 2200 may store instructions, data structures, and program codes that may be read by the processor 2300. The memory 2200 may be configured as one or more memories. In an embodiment of the present disclosure, the operations performed by the processor 2300 may be implemented by executing instructions or codes of a program stored in the memory 2200.

[0205] Memory 2200 may include nonvolatile memory, such as read-only memory (ROM) (e.g., programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM)), flash memory (e.g., memory cards and solid-state drives (SSDs)), and analog recording-type memory (e.g., hard disk drives (HDDs), magnetic tapes, optical disks), as well as volatile memory, such as random access memory (RAM) (e.g., dynamic RAM (DRAM) and static RAM (SRAM)).

[0206] The memory 2200 may store one or more instructions and programs that cause the server 2000 to operate to separate and extract overlapping texts. For example, the separator 2100, the extractor 2220, and the repairer 2230 may be stored in the memory 2200. As described above in the aforementioned figures, the first text separation network may include the separator 2100, the extractor 2200, and the repairer 2230, and the second text separation network may include the separator 2100 and the extractor 2200. The separator 2100 may include a first separator using a color clustering algorithm and a second separator including a text separation model as an AI model.

[0207] The processor 2300 may control all operations of the server 2000. For example, the processor 2300 may execute one or more instructions of the program stored in the memory 2200 to control all operations of the server 2000 so that the server 2000 separates and recognizes overlapping texts. The processor 2300 may be configured as one or more processors.

[0208] The one or more processors 2300 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), an integrated many-core (MIC), a digital signal processor (DSP), or a neural processing unit (NPU). The one or more processors 2300 may be implemented in the form of an integrated system on chip (SoC) including one or more electronic components. The one or more processors 2300 may each be implemented as separate hardware (H / W).

[0209] The processor 2300 may separate a plurality of text regions within the input image by using the separator 2210. The processor 2300 may determine the text region by determining a character group including characters based on a word, a sentence, a corpus, etc. In this case, overlapping texts may exist in the input image. For example, a second text may overlap with a first text. The processor 2300 may separate text regions corresponding to all texts included in the input image, thereby obtaining text regions corresponding to overlapping texts, respectively. Because a description related to the operation of the separator 2210 has been provided with reference to the aforementioned figures, the description will not be repeated for the sake of brevity.

[0210] The processor 2300 may extract pixels corresponding to text in the text region by using the extractor 2220. For example, the processor 2300 may extract only pixels corresponding to the first text but not corresponding to the second text overlapping the first text from the first text region including the first text. In this case, when the second text overlapping the first text is removed from the first text region, an image in which a portion of the first text is damaged may be obtained. Since a description related to the operation of the extractor 2220 has been provided with reference to the aforementioned drawings, the description will not be repeated for the sake of brevity.

[0211] The processor 2300 may reconstruct damaged pixels in the image by using the repairer 2230. The processor 2300 may reconstruct the first text by repairing a region where the first text overlaps with the second text in the first text region by using the repairer 2230. Because a description related to the operation of the repairer 2230 has been provided with reference to the aforementioned drawings, the description is omitted for the sake of brevity.

[0212] In addition, the modules stored in the memory 2200 and executed by the processor 2300 are for ease of description and are not necessarily limited thereto. Other modules may be added to implement the above-mentioned embodiments of the present disclosure, and the modules may be subdivided into multiple modules distinguished according to their detailed functions, and some of the above-mentioned modules may be combined to form a single module. For example, the separator 2210, the extractor 2220, and the repairer 2230 may be combined into one and collectively referred to as a first text separation network, and the separator 2210 and the extractor 2220 may be combined into one and collectively referred to as a second text separation network, but the present disclosure is not limited thereto.

[0213] When the method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or multiple processors. For example, when the method according to an embodiment of the present disclosure performs a first operation, a second operation, and a third operation, the first operation, the second operation, and the third operation may all be performed by a first processor, and the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor), and the third operation may be performed by a second processor (e.g., a dedicated AI processor). Here, a dedicated AI processor as an example of a second processor may perform calculations for training / inference of an AI model. However, embodiments of the present disclosure are not limited thereto.

[0214] One or more processors 2300 according to the present disclosure may be implemented as a single-core processor or a multi-core processor.

[0215] When the method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core or a plurality of cores included in one or more processors 2300 .

[0216] Fig.17 is a block diagram of an electronic device according to an embodiment of the present disclosure.

[0217] In an embodiment of the present disclosure, the above operations of the server 2000 may be performed by the electronic device 3000 .

[0218] According to an embodiment of the present disclosure, the electronic device 3000 may include a communication interface 3100, a display 3200, a camera 3300, a memory 3400, and a processor 3500. Because the communication interface 3100, the memory 3400, and the processor 3500 of the electronic device 3000 correspond to Fig.16 The communication interface 2100, memory 2200 and processor 2300 of the server 2000 are described above, so the description already provided about the components is omitted.

[0219] The display 3200 may output information processed by the electronic device 3000. In addition, when the display 3200 and the touch panel form a layer structure to construct a touch screen, the display may be used as an input device as well as an output device. The display 3200 may include at least one of a liquid crystal display (LCD), a thin film transistor LCD (TFT LCD), an organic light emitting diode (OLED) display, a flexible display, a three-dimensional (3D) display, a micro display, or a head mounted display (HMD). The electronic device 3000 may display information related to a product obtained by identifying a product tag on the display 3200.

[0220] The camera 3300 may obtain a video and / or an image by capturing an image of an object. The electronic device 3000 may include one or more cameras 3300. The camera 3300 may include, for example, an RGB camera, a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, etc., but is not limited thereto. The camera 3300 may obtain a video including a plurality of frames. A person of ordinary skill in the art may clearly infer the specific type and detailed functions of the camera 3300, and thus a description thereof is omitted.

[0221] Although not in Fig.17 , but the electronic device 3000 may further include an input / output (I / O) interface. The I / O interface may include an input interface for receiving a user's input and an output interface for outputting a signal other than an image / video signal output from the display 3200.

[0222] The input interface is used to receive input from the user. The input interface may include, but is not limited to, at least one of a keypad, a dome switch, a touch pad (capacitive stacking type, resistive stacking type, infrared beam type, surface acoustic wave type, integrated strain gauge type, piezoelectric type, etc.), a scroll wheel, or a scroll wheel switch.

[0223] The output interface may include a speaker. The speaker may output an audio signal received from the communication interface 3100 or stored in the memory 3400.

[0224] Fig.18 is a block diagram of a refrigerator according to an embodiment of the present disclosure.

[0225] The refrigerator 4000 according to an embodiment of the present disclosure may include a main body 4010 , a storage chamber 4100 , a door 4200 , a cold air supply device 4300 , a power module 4400 , a processor 4500 , a memory 4600 , a communication module 4700 , an input interface 4800 , and an output interface 4900 .

[0226] The body 4010 may include an inner shell, an outer shell disposed outside the inner shell, and a thermal insulator disposed between the inner shell and the outer shell.

[0227] The inner shell may include at least one of a shell, a plate, a panel, or a liner forming the storage chamber 4100. The inner shell may be formed as a single body, or may be formed by assembling a plurality of plates. The outer shell may form the exterior of the body 4010, and may be coupled to the exterior of the inner shell so that a thermal insulation is provided between the inner shell and the outer shell.

[0228] The thermal insulation member may insulate the inside of the storage room 4100 and the outside of the storage room 4100 so that the temperature inside the storage room 4100 may be maintained at a set appropriate temperature without being affected by the environment outside the storage room 4100. According to an embodiment of the present disclosure, the thermal insulation member may include a foam insulation member. The foam insulation member may be molded by injecting polyurethane foam between the inner shell and the outer shell and foaming the polyurethane foam, in which polyurethane and a foaming agent are mixed.

[0229] According to one embodiment of the present disclosure, in addition to the foam insulation, the insulation may also include a vacuum insulation, or may include only a vacuum insulation instead of the foam insulation. The vacuum insulation may include a core material and an outer covering material, the outer covering material accommodating the core material and sealing the interior to a vacuum or a pressure close to a vacuum. However, the thermal insulation is not limited to the foam insulation or the vacuum insulation, and may include various materials that can be used for the insulation.

[0230] The storage chamber 4100 may include a space defined by an inner shell. The storage chamber 4100 may further include an inner chamber defining a space corresponding to the storage chamber 4100. Various items such as food, medicine, and cosmetics may be stored in the storage chamber 4100, and at least one side of the storage chamber 4100 may be opened to put in or take out items.

[0231] The refrigerator 4000 may include one or more storage chambers 4100. When two or more storage chambers 4100 are provided in the refrigerator 4000, each of the storage chambers 4100 may have different purposes and may be maintained at different temperatures. To this end, the storage chambers 4100 may be separated from each other by partition walls including heat insulating materials.

[0232] The storage chamber 4100 may be arranged to be maintained within an appropriate temperature range according to its use, and include a refrigerating chamber 4120, a freezing chamber 4130, or a variable temperature chamber 4140 distinguished according to its use and / or temperature range. The refrigerating chamber 4120 may be maintained at a temperature suitable for refrigerated items, and the freezing chamber 4130 may be maintained at a temperature suitable for frozen items. Refrigeration may mean cooling items without freezing items, and for example, the refrigerating chamber 4120 may be maintained within a range between 0°C and 7°C. Freezing may mean freezing items or cooling items to keep them frozen, and for example, the freezing chamber 4130 may be maintained within a range between -20°C and -1°C. The variable temperature chamber 4140 may be used as a refrigerating chamber 4120 or a freezing chamber 4130 based on a user's selection or regardless of a user's selection.

[0233] The storage chamber 4100 may be referred to by various names, such as a vegetable chamber, a fresh-keeping chamber, a cooling chamber, and an ice-making chamber, as well as names such as a refrigerating chamber 4120, a freezing chamber 4130, and a temperature-changing chamber 4140, and as used herein, terms such as a refrigerating chamber 4120, a freezing chamber 4130, and a temperature-changing chamber 4140 should be understood to each cover a storage chamber 4100 having a corresponding use and temperature range.

[0234] According to an embodiment of the present disclosure, the refrigerator 4000 may include at least one door 4200 configured to open and close one open side of the storage chamber 4100. At least one door 4200 may be provided to open and close each of one or more storage chambers 4100, or one door 4200 may be provided to open and close a plurality of storage chambers 4100. The door 4200 may be rotatably or slidably installed on the front surface of the body 4010.

[0235] The door 4200 may be configured to seal the storage chamber 4100 when the door 4200 is closed. The door 4200 may include a heat insulating material similar to that in the body 4010 to insulate the storage chamber 4100 when the door 4200 is closed.

[0236] According to an embodiment of the present disclosure, the door 4200 may include a door outer panel forming a front surface of the door 4200, a door inner panel forming a rear surface of the door 4200 and facing the storage chamber 4100, an upper cover, a lower cover, and a door insulation material disposed therein.

[0237] The gasket may be provided at the edge of the door inner panel and seals the storage chamber 4100 by closely contacting the front surface of the body 4010 when the door 4200 is closed. The door inner panel may include a dyke protruding rearward to install a door basket capable of storing items.

[0238] According to an embodiment of the present disclosure, the door 4200 may include a door body and a front panel detachably coupled to the front side of the door body and forming a front surface of the door 4200. The door body may include a door output plate forming the front surface of the door body, a door inner plate forming the rear surface of the door body and facing the storage chamber 4100, an upper cap, a lower cap, and a door heat insulating material disposed therein.

[0239] According to the arrangement of the door 4200 and the storage chamber 4100, the refrigerator 4000 may be classified into a French door type, a side-by-side type, a bottom mounted freezer (BMF) refrigerator, a top mounted freezer (TMF) refrigerator, a single door refrigerator, etc.

[0240] According to an embodiment of the present disclosure, the refrigerator 4000 may include a cold air supply device 4300 configured to supply cold air to the storage chamber 4100 .

[0241] The cold air supply device 4300 may include a machine, a device, an electronic device, and / or a system as a combination thereof that is capable of generating cold air and guiding the cold air to cool the storage compartment 4100 .

[0242] According to an embodiment of the present disclosure, the cold air supply device 4300 can generate cold air through a refrigeration cycle including compression, condensation, expansion and evaporation of a refrigerant. To this end, the cold air supply device 4300 may include a refrigeration cycle system having a compressor, a condenser, an expansion device and an evaporator capable of driving a refrigeration cycle. According to an embodiment of the present disclosure, the cold air supply device 4300 may include a semiconductor such as a thermoelectric element. The thermoelectric element can cool the storage chamber 4100 by heating and cooling using the Peltier effect.

[0243] According to an embodiment of the present disclosure, the refrigerator 4000 may include a machine room in which at least some components belonging to the cold air supply device 4300 are arranged.

[0244] The machine room may be separated and insulated from the storage room 4100 to prevent heat generated in components arranged in the machine room from being transferred to the storage room 4100. The inside of the machine room may be configured to communicate with the outside of the body 4010 to dissipate heat from the components inside the machine room.

[0245] According to an embodiment of the present disclosure, the refrigerator 4000 may include a dispenser installed in the door 4200 to provide water and / or ice. The dispenser may be provided in the door 4200 so that a user can access the dispenser without opening the door 4200.

[0246] According to an embodiment of the present disclosure, the refrigerator 4000 may include an ice maker provided to generate ice. The ice maker may include an ice tray for storing water, an ice breaker for separating ice from the ice tray, and an ice bucket for storing ice generated in the ice tray.

[0247] According to an embodiment of the present disclosure, the refrigerator 4000 may include a controller for controlling the refrigerator 4000 .

[0248] The controller may include a memory 4600 for storing or recording a program and / or data for controlling the refrigerator 4000 and a processor 4500 configured to output a control signal for controlling the cold air supply device 4300, etc. according to the program and / or data recorded in the memory 4600.

[0249] The memory 4600 stores or records various information, data, instructions, programs, etc. required for the operation of the refrigerator 4000. The memory 4600 may store temporary data generated when a control signal for controlling components included in the refrigerator 4000 is generated. The memory 4600 may include at least one of a volatile memory or a nonvolatile memory, or a combination thereof.

[0250] The processor 4500 controls all operations of the refrigerator 4000. The processor 4500 may execute a program stored in the memory 4600 to control the components of the refrigerator 4000. The processor 4500 may include a separate NPU that performs the operation of the AI ​​model. The processor 4500 may also include a CPU, a dedicated graphics processor (GPU), etc. The processor 4500 may generate a control signal for controlling the operation of the cold air supply device 4300. For example, the processor 4500 may receive temperature information of the storage chamber 4100 from a temperature sensor, and generate a cooling control signal for controlling the operation of the cold air supply device 4300 based on the temperature information of the storage chamber 4100.

[0251] In addition, the processor 4500 can process user input received from the user interface and control the operation of the user interface according to the program and / or data recorded / stored in the memory 4600. The user interface can be provided using the input interface 4800 and the output interface 4900. The processor 4500 can receive user input from the user interface. The processor 4500 can also send a display control signal and image data to the user interface for displaying an image on the user interface in response to the user input.

[0252] The processor 4500 may be provided integrally or separately from the memory 4600. The processor 4500 may include one or more processors. For example, the processor 4500 may include a main processor and at least one sub-processor. The memory 4600 may include one or more memories.

[0253] According to an embodiment of the present disclosure, the refrigerator 4000 may include a memory 4600 and a processor 4500 for controlling all components included in the refrigerator 4000 and the memory 4600, or include a plurality of memories 4600 and a plurality of processors 4500 for individually controlling the components of the refrigerator 4000. For example, the refrigerator 4000 may include a memory 4600 and a processor 4500, and the processor 4500 is used to control the operation of the cold air supply device 4300 according to the output of the temperature sensor. In addition, the refrigerator 4000 may separately include a memory 4600 and a processor 4500 for controlling the operation of the user interface according to the user input.

[0254] The communication module 4700 can communicate with an external device such as a server, a mobile device, or another home appliance via a nearby access point (AP). The AP can connect the LAN to which the refrigerator 4000 or the user device is connected to a wide area network (WAN) to which the server is connected. The refrigerator 4000 or the user device can be connected to the server via the WAN.

[0255] The input interface 4800 may include a key 4810 , a touch screen 4820 , a microphone, etc. The input interface 4800 may receive a user input and transmit the user input to the processor 4500 .

[0256] The output interface 4900 may include a display 4910 , a speaker 4920 , etc. The output interface 4900 may output various notifications, messages, information, etc. generated by the processor 4500 .

[0257] In an embodiment of the present disclosure, the refrigerator 4000 may perform data communication with the server 2000 and / or the electronic device 3000. The refrigerator 4000 may receive data related to a control operation (e.g., operation mode setting) from the server 2000 and / or the electronic device 3000, and perform a function corresponding to the received data in the refrigerator 4000. Since a description of the refrigerator 4000 being controlled by the server 2000 and / or the electronic device 3000 has been provided above, the description will not be repeated.

[0258] Fig.19 is a block diagram of an oven according to an embodiment of the present disclosure.

[0259] The oven 5000 according to an embodiment of the present disclosure may include a processor 5100 , a driver 5200 , a sensor unit 5300 , a communication module 5400 , a user interface 5500 , lighting (lamp) 5600 , and a memory 5700 .

[0260] Processor 5100 may control all operations of oven 5000. Processor 5100 may execute a program stored in memory 5700 to control driver 5200, sensor unit 5300, communication module 5400, user interface 5500, lighting 5600, and memory 5700.

[0261] In an embodiment of the present disclosure, the oven 5000 may be a smart oven equipped with an artificial intelligence (AI) processor. The AI ​​processor may be manufactured in the form of a dedicated hardware chip for AI, or it may be manufactured as part of an existing general-purpose processor (e.g., a CPU or an application processor (AP)) or a dedicated graphics processor (e.g., a GPU) and installed in the oven 5000.

[0262] The memory 5700 may store programs required for processing and control performed by the processor 5100 and a plurality of input / output data (e.g., recipe information, area table, spacing table, size information of cropped areas, distortion correction values, brightness level table, etc.). The memory 5700 may store AI models. For example, the memory 5700 may store AI models for object recognition, AI models for recipe recommendations, etc.

[0263] The memory 5700 may include at least one type of storage medium, i.e., at least one of a flash memory, a hard disk memory, a multimedia card micro memory, a card memory (e.g., an SD card or an XD memory), a RAM, a SRAM, a ROM, an EEPROM, a PROM, a magnetic memory, a magnetic disk, or an optical disk. In addition, the oven 5000 may interact with a network storage or a cloud server that performs a storage function on the Internet.

[0264] The driver 5200 may include a heater 5210 , a circulation fan 132 , and a cooling fan 5230 , but is not limited thereto. The driver 5200 may vary according to the type of the oven 5000 .

[0265] A heater 5210 for heating food may be provided in an inner space (eg, a cooking chamber) of the oven 5000. The heater 5210 may be an electric heater including a resistor or a gas heater generating heat by burning gas.

[0266] A circulation fan 5220 for uniformly heating food by circulating the internal air and a circulation motor for driving the circulation fan 5220 may be provided at the rear of the internal space (cooking chamber). In addition, a fan cover covering the circulation fan 5220 may be provided in front of the circulation fan 5220 and have a through hole formed therein to allow air flow.

[0267] The cooling fan 5230 may be a centrifugal fan that draws air from above and discharges it radially. The cooling fan 5230 may be disposed in a cooling channel. The cooling fan 5230 may include a flat rotating plate, a hub disposed at the center of the rotating plate and coupled to a rotating shaft of a cooling motor, and a plurality of wings arranged from the center of the rotating plate to an edge portion thereof. The hub may be formed in a conical shape with a radius increasing toward the bottom, so that the air drawn from the top is diffused in a downward direction.

[0268] The sensor unit 5300 may include, but is not limited to, a depth sensor 5310, a weight sensor 5320, an infrared sensor 5330, a humidity sensor 5340 for sensing the humidity of the interior space, a gas sensor 5350 for sensing the gas level in the interior space, and a temperature sensor 5360. Since a person of ordinary skill in the art can intuitively infer the function of each sensor from its name, a detailed description thereof will be omitted.

[0269] The communication module 5400 may include one or more components that implement communication between the oven 5000 and a server device (not shown) or between the oven 5000 and a mobile terminal (not shown). For example, the communication module 5400 may include a short-distance communication module 5410, a long-distance communication module 5420, and the like.

[0270] The short-range communication module 5410 may include, but is not limited to, a communication module using a communication technology such as Bluetooth, BLE, NFC, WLAN (Wi-Fi), ZigBee, IrDA, WFD, ultra-wideband (UWB), Ant+, etc. When the oven 5000 is remotely controlled by a server device (not shown) in an Internet of Things (IoT) environment, the remote communication module 5420 may be used to communicate with the server 2000. The remote communication module 5420 may include the Internet, a computer network (e.g., a LAN or WAN), and a mobile communication module. The mobile communication module may include, but is not limited to, a third generation (3G) module, a fourth generation (4G) module, a fifth generation (5G) module, a long term evolution (LTE) module, a narrowband Internet of Things (NB-IoT) module, an LTE machine (LTE-M) module, etc.

[0271] The user interface 5500 may include an input interface 5510 and an output interface 5520. The output interface 5510 is used to output a video signal or an audio signal, and may include a display, an audio output interface, and the like.

[0272] When the display and the touch panel form a layer structure to construct a touch screen, the display can be used as an input interface 5520 and an output interface 5510. The display may include at least one of an LCD, a TFT LCD, an LED display, an OLED display, a flexible display, a 3D display, or an electrophoretic display. In addition, the oven 5000 may include two or more displays according to the configuration implemented therein.

[0273] The audio output interface may output audio data received via the communication module 5400 or stored in the memory 5700. The audio output interface may also output a sound signal related to a function performed by the oven 5000. The audio output interface may include a speaker, a buzzer, and the like.

[0274] According to an embodiment of the present disclosure, the display may output a monitoring image of the inner space of the oven 5000, or output recipe information suitable for the ingredients. In addition, the display may output a correction value of the cooking temperature determined according to the height of the insertion tray.

[0275] The input interface 5520 is used to receive input from the user. The input interface 5520 can be at least one of a keypad, a dome switch, a touch pad (capacitive stacking type, resistive stacking type, infrared beam type, surface acoustic wave type, integrated strain gauge type, piezoelectric type, etc.), a micro wheel or a micro switch, but is not limited thereto.

[0276] The input interface 5520 may include a speech recognition module. For example, the oven 5000 may receive a speech signal as an analog signal via a microphone, and convert a portion of the speech into computer-readable text by using an automatic speech recognition (ASR) model. The oven 5000 may obtain the intent in the user's speech by interpreting the text using a natural language understanding (NLU) model. Here, the ASR model or the NLU model may be an AI model. The AI ​​model may be processed by a dedicated AI processor designed with a hardware structure dedicated to processing the AI ​​model.

[0277] Lighting 5600 may be provided on one side of the interior space of oven 5000 and may be referred to as internal lighting. For example, lighting 5600 may be provided on the ceiling or on the side, but is not limited thereto. Lighting 5600 may be turned on when the door of oven 5000 is opened or oven 5000 is operated. Lighting 5600 may be protected by a glass cover.

[0278] According to an embodiment of the present disclosure, the lighting 5600 may have various brightness levels. For example, the lighting 5600 may emit light from a low brightness level to a high brightness level. The brightness of the lighting 5600 may be controlled by the processor 5100. The lighting 5600 may be a halogen lighting or an LED lighting, but is not limited thereto.

[0279] In an embodiment of the present disclosure, the oven 5000 may perform data communication with the server 2000 and / or the electronic device 3000. The oven 5000 may receive data related to a control operation (e.g., operation mode setting) from the server 2000 and / or the electronic device 3000, and perform a function corresponding to the received data in the oven 5000. Since a description of the oven 5000 being controlled by the server 2000 and / or the electronic device 3000 has been provided above, the description will not be repeated.

[0280] The present disclosure proposes a method for obtaining an image including complete text by determining a text area to separate overlapping text from an image including the overlapping text, extracting only pixels corresponding to a character group from the text area, and then reconstructing the overlapping text, and a method for generating information by recognizing the reconstructed text.

[0281] The technical solutions to be implemented in the present disclosure are not limited to those described above, and other technical solutions not described will be clearly understood by those of ordinary skill in the art from the following description.

[0282] According to one aspect of the present disclosure, a method for obtaining text from an image performed by a server may be provided. The method may include obtaining an image including a first text and a second text overlapping the first text. The method may include separating a first text region corresponding to the first text from the image. The method may include extracting pixels corresponding to the first text from the first text region to obtain a non-overlapping portion of the first text (a non-overlapping portion of the first text that does not overlap with any other text or feature) and a damaged portion (e.g., an overlapping portion where the first text overlaps with the second text) of the first text. The method may include reconstructing the first text by repairing the damaged portion of the first text where the first text overlaps with the second text in the image.

[0283] Separating the first text region may include determining a plurality of text regions by segmenting the first text and the second text in the image into a plurality of character groups.

[0284] Separating the text region may include separating the first text region from the plurality of text regions.

[0285] Determining the plurality of text regions may include determining the plurality of text regions by applying a color clustering algorithm to the image to group characters of adjacent colors in a color space.

[0286] Determining the plurality of text regions may include receiving a user input selecting a first text in the image.

[0287] Determining the plurality of text regions may include applying a color clustering algorithm to a first text region in the image corresponding to the user input.

[0288] Determining the plurality of text regions may include obtaining, based on the image, a text region map indicating locations of the first text and the second text in the image.

[0289] Determining the plurality of text regions may include determining the plurality of text regions based on a text region map.

[0290] Determining the plurality of text regions may include applying a color clustering algorithm based on the text region map.

[0291] Determining the plurality of text regions may include determining the plurality of text regions based on at least one of a language, a font, a format, a corpus, or a logo of each of the first text and the second text.

[0292] The method may include applying OCR to the first text.

[0293] The images may include labeled images related to food.

[0294] The first text may include food-related text.

[0295] The method may include providing food-related information based on a result of applying OCR to the first text.

[0296] The method may include sending information related to the food to an external device.

[0297] According to one aspect of the present disclosure, a server for obtaining text from an image may be provided. The server may include a communication interface, a memory storing one or more instructions, and one or more processors configured to execute the one or more instructions stored in the memory.

[0298] One or more processors may execute one or more instructions to obtain an image including a first text and a second text overlapping the first text. One or more processors may execute one or more instructions to separate a first text region corresponding to the first text from the image. One or more processors may execute one or more instructions to extract pixels corresponding to the first text region to obtain an undamaged portion and a damaged portion of the first text. One or more processors may execute one or more instructions to reconstruct the first text by repairing a damaged portion of the first text in the image, wherein the first text overlaps the second text in the damaged portion.

[0299] The one or more processors may execute the one or more instructions to determine a plurality of text regions by segmenting the first text and the second text in the image into a plurality of character groups.

[0300] The one or more processors may execute the one or more instructions to separate the first text region from the plurality of text regions.

[0301] The one or more processors may execute the one or more instructions to determine a plurality of text regions by applying a color clustering algorithm to the image to group characters of adjacent colors in a color space.

[0302] The one or more processors may execute the one or more instructions to receive user input selecting the first text in the image.

[0303] The one or more processors may execute the one or more instructions to apply a color clustering algorithm to a first text region in the image corresponding to the user input.

[0304] The one or more processors may execute the one or more instructions to obtain, based on the image, a text region map indicating locations of the first text and the second text in the image.

[0305] One or more processors may execute one or more instructions to determine a plurality of text regions based on the text region map.

[0306] The one or more processors may execute the one or more instructions to apply a color clustering algorithm based on the text region map.

[0307] The one or more processors may execute the one or more instructions to determine a plurality of text regions based on at least one of a language, a font, a format, a corpus, or a logo of each of the first text and the second text.

[0308] The one or more processors may execute the one or more instructions to apply OCR to the first text.

[0309] The images may include labeled images related to food.

[0310] The first text may include food-related text.

[0311] The one or more processors may execute the one or more instructions to provide food-related information based on a result of applying OCR to the first text.

[0312] In addition, embodiments of the present disclosure may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. A computer-readable recording medium may be any available medium accessible to a computer, and includes both volatile and non-volatile media and both removable and non-removable media. In addition, a computer-readable recording medium may include both computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Communication media typically embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal.

[0313] The computer-readable storage medium may be provided in the form of a non-transitory storage medium. In this regard, the term "non-transitory storage medium" simply means that the storage medium does not include a signal (e.g., an electromagnetic wave) and is a tangible device, and the term does not distinguish between a location where data is semi-permanently stored in the storage medium and a location where data is temporarily stored in the storage medium. For example, a "non-transitory storage medium" may include a buffer for temporarily storing data.

[0314] According to an embodiment of the present disclosure, when a computer program product is provided, the method according to an embodiment of the present disclosure may be included in the computer program product. The computer program product can be traded between a seller and a buyer as a product. The computer program product can be distributed in the form of a computer-readable storage medium (e.g., a CD-ROM), or distributed online (e.g., downloaded or uploaded) via an application store or distributed directly between two user devices (e.g., smart phones). For online distribution, at least a portion of the computer program product (e.g., a downloadable app) can be at least temporarily stored or temporarily generated in a computer-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a relay server).

[0315] The above description of the present disclosure is provided for illustration, and those skilled in the art will appreciate that changes in form and detail may be easily made therein without departing from the technical concept or basic features of the present disclosure. Therefore, the above-described embodiments of the present disclosure and all aspects thereof are merely illustrative and non-restrictive. For example, each component defined as an integrated component may be implemented in a distributed manner, and similarly, components defined as separate components may be implemented in an integrated form.

[0316] The scope of the disclosure is defined not by the detailed description thereof but by the appended claims, and all changes or modifications within the meaning and scope of the appended claims and their equivalents are to be construed as being included in the scope of the disclosure.

Claims

1. A method for obtaining text from an image performed by a server, the method comprising: obtaining an image including a first text and a second text overlapping the first text; separating a first text region corresponding to the first text from the image; Extracting pixels corresponding to the first text from the first text area to obtain an undamaged portion and a damaged portion of the first text; as well as The first text is reconstructed by repairing a damaged portion of the first text in the image, wherein the first text overlaps with the second text in the damaged portion.

2. The method according to claim 1, wherein Separating the first text region comprises: determining a plurality of text regions by segmenting the first text and the second text in the image into a plurality of character groups; as well as The first text region is separated from the plurality of text regions.

3. The method according to claim 2, wherein: Determining the plurality of text regions includes determining the plurality of text regions by applying a color clustering algorithm to the image to group characters of adjacent colors in a color space.

4. The method according to claim 3, wherein Determining the plurality of text regions comprises: receiving user input for selecting the first text in the image; as well as The color clustering algorithm is applied to the first text region in the image corresponding to the user input.

5. The method according to claim 3, wherein Determining the plurality of text regions comprises: obtaining, based on the image, a text area map indicating locations of the first text and the second text in the image; as well as The plurality of text regions are determined based on the text region map.

6. The method according to any one of claims 2 to 5, wherein: Determining the plurality of text regions includes determining the plurality of text regions based on at least one of a language, a font, a format, a corpus, or a logo of each of the first text and the second text.

7. The method according to any one of claims 1 to 6, wherein The image includes a label image related to food, The first text includes text related to the food, and The method further comprises: Based on a result of applying the OCR to the first text, information related to the food is provided.

8. A server for obtaining text from an image, the server comprising: a memory storing one or more instructions; as well as one or more processors configured to execute the one or more instructions stored in the memory to: obtaining an image including a first text and a second text overlapping the first text, separating a first text region corresponding to the first text from the image, extracting pixels corresponding to the first text area to obtain an undamaged portion and a damaged portion of the first text, and The first text is reconstructed by repairing a damaged portion of the first text in the image, wherein the first text overlaps with the second text in the damaged portion.

9. The server according to claim 8, wherein The one or more processors are further configured to execute the one or more instructions to: determining a plurality of text regions by segmenting the first text and the second text in the image into a plurality of character groups, and The first text region is separated from the plurality of text regions.

10. The server according to claim 9, wherein: The one or more processors are further configured to execute the one or more instructions to determine the plurality of text regions by applying a color clustering algorithm to the image to group characters of adjacent colors in a color space.

11. The server according to claim 10, wherein The one or more processors are further configured to execute the one or more instructions to: receiving user input for selecting the first text in the image, and The color clustering algorithm is applied to the first text region in the image corresponding to the user input.

12. The server according to claim 10, wherein The one or more processors are further configured to execute the one or more instructions to: obtaining a text area map indicating positions of the first text and the second text in the image based on the image, and The plurality of text regions are determined based on the text region map.

13. The server according to any one of claims 9 to 12, wherein: The one or more processors are further configured to execute the one or more instructions to determine the plurality of text regions based on at least one of a language, a font, a format, a corpus, or a logo of each of the first text and the second text.

14. A server according to any one of claims 8 to 13, wherein The image includes a label image related to food, The first text includes text related to the food, and The one or more processors are further configured to execute the one or more instructions to: Based on a result of applying the OCR to the first text, information related to the food is provided. 15 . A computer-readable recording medium having recorded thereon a program for executing the method according to claim 1 on a computer.

Images