Letter detection device, method and system

Abstract

According to one embodiment, a letter detection method is disclosed, which is performed by a letter detection device and is characterized in that it includes the following steps: obtaining an input image; and inputting the input image into a letter detection model including a neural network and processing it; obtaining at least one output image from the letter detection model, wherein the output image includes a probability value image, which represents the probability of the existence of a letter in the input image in an image space at a position corresponding to the input image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing. More specifically, the present disclosure relates to an apparatus, method and system for detecting letters included in an image by using a machine learning model. Background Art

[0002] Detecting the position of text in an image is a key element in human computer interaction (HCI). Detecting text from text images is relatively simple, but it is difficult to detect text in real-life photographic images such as signs, road signs, and menus due to various design factors and the effects of image transformation and light reflection.

[0003] A method has been proposed for detecting words consisting of multiple letters in an image. However, when detecting text centered on the word rather than on individual letters, there is a problem of reduced detection accuracy for text arranged in a curved form or text distorted depending on the viewpoint. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The technical problem to be solved by the letter detection device, method and system according to one embodiment is to detect each letter with higher accuracy, rather than detecting words composed of multiple letters.

[0006] Furthermore, a technical problem to be solved by a letter detection device, method, and system according to an embodiment is to increase a small amount of ground truth (GT).

[0007] Furthermore, the technical problem to be solved by the letter detection device, method and system according to an embodiment is to effectively train a letter detection model through supervised learning and weakly supervised learning processes.

[0008] (II) Technical solution

[0009] According to one embodiment, a letter detection method includes the following steps: obtaining an input image; inputting the input image into a letter detection model including a neural network and processing it; and obtaining at least one output image from the letter detection model, wherein the output image includes a probability value image, which represents the probability of the existence of a letter in the input image in an image space at a position corresponding to the input image.

[0010] (III) Beneficial effects

[0011] According to an embodiment, the letter detection device, method, and system can detect letters arranged in a curved line or letters distorted according to a viewpoint with high accuracy by detecting each letter instead of detecting a word composed of a plurality of letters.

[0012] Furthermore, the letter detection device, method, and system according to an embodiment can generate a ground truth (GT) that provides position information of a letter region through a learning image annotated with the position of the vocabulary region.

[0013] Furthermore, the letter detection device, method, and system according to an embodiment can effectively train a letter detection model through supervised learning and weakly supervised learning processes.

[0014] However, the effects that can be achieved by the letter detection device, method and system according to an embodiment are not limited to the above contents, and for the effects not mentioned above, ordinary technicians in the technical field to which the present disclosure belongs can clearly understand them based on the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to fully understand the drawings cited in this specification, a brief description of each drawing is provided.

[0016] Figure 1 is a diagram showing a letter detection device according to an embodiment.

[0017] Figure 2 is a flowchart for illustrating a letter detection method according to an embodiment.

[0018] Figure 3 is a diagram showing an image input to a letter detection model and an image output by the letter detection model.

[0019] Figure 4 is a diagram showing a neural network structure of a letter detection model according to an embodiment.

[0020] Figure 5 This is a flowchart for explaining a method of training a letter detection model using a learning image in which position values ​​of letter regions are labeled.

[0021] Figure 6 This is a diagram for explaining the training process of the letter detection model based on the learning image.

[0022] Figure 7 This is a diagram for explaining the process of generating a true value (GT) image from a learning image.

[0023] Figure 8This is a diagram for explaining a method of training a letter detection model using a learning image to which position values ​​of vocabulary areas are labeled.

[0024] Fig. 9 This is a diagram for explaining the training process of the letter detection model based on the learning image.

[0025] Fig.10 This is a diagram for explaining a method of specifying the position value of a letter region in a learning image to which the position value of a vocabulary region is labeled.

[0026] Fig.11 is a diagram showing an intermediate output image output when a vocabulary region image is input into a letter detection model at each epoch.

[0027] Fig.12 is a diagram for explaining another method of detecting letter regions from a vocabulary region image.

[0028] Fig.13 is a block diagram showing a configuration of a letter detection device according to an embodiment.

[0029] Fig.14 is a functional block diagram of a letter detection device according to an embodiment.

[0030] Fig.15 is a diagram showing a server device and a client device to which a letter detection device according to an embodiment can be applied.

[0031] Fig.16 This is a diagram for explaining the training process of the letter detection model based on the learning image.

[0032] Best Mode for Carrying Out the Invention

[0033] According to one embodiment, a letter detection method includes: obtaining an input image; inputting the input image into a letter detection model including a neural network and processing it; and obtaining at least one output image from the letter detection model, wherein the output image includes a probability value image, which represents the probability of the existence of a letter in the input image in an image space at a position corresponding to the input image.

[0034] According to one embodiment, a letter detection device includes: a processor; and a memory storing at least one instruction, wherein the processor can obtain an input image according to the at least one instruction, and obtain at least one output image from the letter detection model by inputting the input image into a letter detection model including a neural network, wherein the output image includes a probability value image, which represents the probability of the existence of a letter in the input image in an image space at a position corresponding to the input image.

[0035] According to one embodiment, a letter detection system includes: a server device, which trains a letter detection model including a neural network by using a learning image; and a client device, which receives data for executing the letter detection model from the server device and inputs the input image into the letter detection model when an input image is obtained to obtain at least one output image from the letter detection model, wherein the output image includes a probability value image, which represents the probability of the existence of a letter in the input image on an image space at a position corresponding to the input image. DETAILED DESCRIPTION

[0036] The present disclosure allows various changes and can have various embodiments, therefore, specific embodiments are illustrated in the drawings and described by specific embodiments. However, this is not to limit the present disclosure to specific embodiments, but should be understood to include all changes, equivalents and substitutes included in the ideas and technical scope of the present disclosure.

[0037] When describing the embodiments, if it is considered that the specific description of the known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof is omitted. In addition, the numbers (e.g., first, second, etc.) used in the process of describing the embodiments are merely identification symbols for distinguishing one constituent element from other constituent elements.

[0038] Furthermore, when the present specification mentions that a constituent element is “connected” or “linked” to other constituent elements, the constituent element may be directly connected or linked to the other constituent elements, but, unless there is an explicit statement to the contrary, it should be understood that the constituent element may be connected or linked to the other constituent elements by interposing another constituent element in between.

[0039] Furthermore, for the constituent elements represented as 'part (unit)', 'module', etc. in this specification, two or more constituent elements may be combined into one constituent element, or one constituent element may be divided into two or more constituent elements according to more subdivided functions. Furthermore, each constituent element described below may further perform a part or all of the functions that are performed by other constituent elements in addition to the main functions that it is responsible for. Of course, a part of the main functions that are performed by each constituent element may be exclusively performed by another constituent element.

[0040] In addition, in this specification, "letter" may refer to the basic unit of words or sentences. For example, for English, each English letter may be equivalent to a letter, for numbers, the numbers from "0" to "9" may be equivalent to letters, and for Korean, a word composed of a combination of consonants and vowels (for example, ), a combination of consonants, vowels, and consonants (e.g. ), individually recorded consonants (e.g. ), vowels recorded separately (e.g. ) etc. can be equivalent to letters.

[0041] Furthermore, in this specification, 'vocabulary' may refer to a unit of characters including at least one letter. 'Vocabulary' may also refer to a unit of characters including at least one letter and having a linguistic meaning. Furthermore, the letters constituting 'vocabulary' may not be separated from each other. Furthermore, 'vocabulary' may consist of one letter. For example, the indefinite article 'a' in English consists of one letter, but when it is separated from the surrounding letters, it may belong to 'vocabulary'.

[0042] Hereinafter, embodiments according to the technical idea of ​​the present disclosure are sequentially described in detail.

[0043] Figure 1 is a diagram showing a letter detection device 100 according to an embodiment.

[0044] According to an embodiment, the letter detection device 100 includes a letter detection model. The letter detection device 100 detects letters included in an input image through the letter detection model. The letter detection model may include a machine learning model, which includes a neural network. The letter detection model may detect letters in the input image through a neural network including an input layer, a hidden layer, and an output layer. The letter detection model may include a convolution neural network (CNN).

[0045] Reference Figure 1 For an image 11 including 'Cold', the existing detection device 10 can output the image 11 having a bounding box 15 corresponding to 'Cold' by detecting 'Cold' which is equivalent to a word. However, according to an embodiment, the letter detection device 100 can distinguish 'Cold' into letters 'C', 'o', 'l', and 'd' for detection, and can output the image 110 having a bounding box 115 for each letter.

[0046] Since the letter detection model according to an embodiment can detect letters included in an image in letter units rather than vocabulary units, accuracy can be improved compared to a method of detecting letters in vocabulary units.

[0047] In the following, reference is made to Figure 2 etc. describe in detail the operation of the letter detection device 100 according to one embodiment.

[0048] Figure 2 is a flowchart for illustrating a letter detection method according to an embodiment.

[0049] In step S210, the letter detection device 100 obtains an image including at least one letter. The letter detection device 100 may obtain an image stored internally or obtain an image received from an external device.

[0050] In step S220, the letter detection apparatus 100 inputs the image into a letter detection model. The letter detection model may be trained to detect the positions of each letter included in the image.

[0051] In step S230, the letter detection apparatus 100 obtains an output image output from the letter detection model. The size and resolution of the output image and the input image may be the same as each other, or the size and resolution of the output image may be smaller than that of the input image.

[0052] The output image represents the existence probability of the letters included in the input image on the image space. The output image may include at least one of a probability value image and a link image.

[0053] The probability value image can represent the existence probability of a letter in the image with pixel values. The pixels included in the probability value image can have values ​​corresponding to the probability that each pixel is equivalent to a letter. The pixels included in the probability value image can have pixel values ​​corresponding to the probability that the position of the pixel is equivalent to the center of the letter.

[0054] The link image indicates connectivity between letters in the image. The link image may include at least one line indicating whether the letters are adjacent to each other.

[0055] Reference Figure 3 When an image 310 including letters such as 'p', 'e', ​​'a', 'c', 'e', ​​etc. is input into the letter detection model, the letter detection model 310 can output: a probability value image 320, which spatially represents the existence probability of the letters within the image 310; and a link image 330, which represents the connectivity between the letters within the image 310.

[0056] The pixels of probability value image 320 may have pixel values ​​corresponding to the probability of existence of letters within image 310. Alternatively, the pixels of probability value image 320 may have pixel values ​​corresponding to the probability of being equivalent to the center of a letter. For example, in probability value image 320, pixels having a particular pixel value may indicate a higher probability of being equivalent to the center of a letter than pixels having other pixel values.

[0057] The location where the letter is likely to exist in the image 310 or the location which is likely to correspond to the center of the letter can be confirmed based on the pixel value of the probability value image 320, thereby detecting the position value of the letter included in the input image 310.

[0058] Furthermore, when centered at a location where the probability value is higher than a first threshold, the size of an adjacent area having a probability value greater than a second threshold less than the first threshold is proportional to the size of the letters included in image 310. Therefore, the size of the letters within image 310 can also be detected based on probability value image 320.

[0059] Link image 330 indicates connectivity between letters in image 310. Link image 330 may include lines 335 and 337, either of which may correspond to a line connecting center points of mutually adjacent letters.

[0060] A line 335 corresponding to 'e' may be included in the link image 330. 'W', 'H', 'Y' included in the image 310 are separated from 'p', 'e', ​​'a', 'c', 'e', ​​and thus a line 337 corresponding to 'W', 'H', 'Y' may be included in the link image 330 separately from the line 335.

[0061] The thickness of the line in the link image 330 may be proportional to the size of the letters included in the image 310. For example, the thickness of the line in the link image 330 may be proportional to the average of the diagonal lengths of the quadrilateral-shaped letter regions confirmed in the probability value image 320.

[0062] Figure 4 is a diagram showing a neural network structure of a letter detection model according to an embodiment.

[0063] Reference Figure 4 ,The letter detection model may include a convolutional layer, an upsampling layer, and an upconvolutional layer.

[0064] The image 405 is convolved in the first convolution layer 410, the second convolution layer 415, the third convolution layer 420, the fourth convolution layer 425, the fifth convolution layer 430, and the sixth convolution layer 435. The output of the sixth convolution layer 435 and the output of the fifth convolution layer 430 are concatenated and input to the first upper convolution layer 440, and the value input to the first upper convolution layer 440 is concatenated 441, batch normalization 442, convolution 443, and batch normalization 444 and input to the first upsampling layer 445. The output of the first upsampling layer 445 and the output of the fourth convolution layer 425 are concatenated and processed in the second upper convolution layer 450 and the second upsampling layer 455. The output of the second upsampling layer 455 and the output of the third convolution layer 420 are processed in the third upsampling layer 460 and the third upsampling layer 465 after being concatenated, and the processing result is concatenated with the output of the second convolution layer 415 and input to the fourth upsampling layer 470. In addition, the output of the fourth upsampling layer 470 and the output of the first convolution layer 410 are concatenated, and then output as a probability value image and a link image 495 after multiple convolution processes 475, 480, 485, 490.

[0065] Figure 4 The neural network structure of the letter detection model shown is merely exemplary, and the number of convolutional layers or the direction of data processing may be variously modified within a range obvious to those skilled in the art.

[0066] In the following, reference is made to Figures 5 to 12 Detailed description of the process of training a letter detection model.

[0067] Figure 5 This is a flowchart for explaining a method of training a letter detection model using a learning image in which position values ​​of letter regions are labeled. Figure 5 The process shown can be equivalent to supervised learning.

[0068] In step S510, the letter detection device 100 obtains a learning image with position values ​​of letter regions annotated thereon. The position values ​​of letter regions may be equivalent to coordinate values ​​of the letter regions on the learning image.

[0069] The learning image may include at least one letter. The letter detection device 100 may obtain the position value of each letter area including the letter in the learning image. When the letter area including the letter in the learning image has a quadrilateral shape, the letter detection device 100 may obtain the coordinate value of each corner in the quadrilateral letter area. In addition to the quadrilateral, the letter area may also have a polygonal shape such as a triangle, a pentagon, or a circular shape.

[0070] In step S520, the letter detection apparatus 100 generates a ground truth (GT) image corresponding to the learning image. The GT image may include at least one of a GT probability value image and a GT link image.

[0071] GT images are the correct answer data provided for learning the letter detection model. Figure 7 Describe the method for generating GT images.

[0072] In step S530, the letter detection device 100 inputs the learning image into the letter detection model. In response to the input of the learning image, the letter detection model outputs an output image corresponding to the learning image. The output image may include at least one of a probability value image and a link image.

[0073] In step S540, the letter detection model updates the internal weighting value according to the comparison result between the output image and the GT image.

[0074] The loss value can be calculated based on the comparison result between the output image and the GT image. For example, the loss value can be equivalent to the L2 Loss value. In addition, the loss value can also use various methods such as L1 loss and smooth L1 loss. The calculated loss value is input into the letter detection model, and the letter detection model can update the internal weight value according to the loss value.

[0075] Reference Figure 6 Describe the learning process of the letter detection model.

[0076] Reference Figure 6 , the learning image 610 is input to the letter detection model, and the letter detection model outputs an output image (at least one of a probability value image and a link image) 620. Furthermore, a GT image 640 is generated based on the position value 630 of the letter region in the learning image 610, and a loss 650 between the output image 620 and the GT image 640 is calculated. The calculated loss 650 is input to the letter detection model, thereby updating the internal weight value of the letter detection model.

[0077] Figure 7 7 is a diagram for explaining the process of generating a GT image from the learning image 710 .

[0078] The position value of each letter area 715 included in the learning image 710 may be marked. Figure 7Only the letter area 715 corresponding to '2' is shown, but letter areas may be determined for all letters. The letter detection device 100 may generate a GT probability value image 730 by warping a two-dimensional Gaussian map (e.g., a two-dimensional heat map) 720 according to the position values ​​of each letter area 715. For example, the letter detection device 100 may warp the two-dimensional Gaussian map 720 according to the coordinate values ​​of the corners of the quadrilateral-shaped letter area 715. Warping refers to geometric image processing that moves the position values ​​of pixels in an image. The size and resolution of the GT probability value image 730 may be the same as the size and resolution of the learning image 710, or the size and resolution of the GT probability value image 730 may be smaller than the size and resolution of the learning image 710.

[0079] Furthermore, the letter detection device 100 may generate a GT link image 740 by connecting the centers of the letter regions 715 using a line. The letter detection device 100 may generate a GT link image 740 by connecting the centers located within a preset distance from the adjacent centers using one line and connecting the centers located outside the preset distance from the adjacent centers using another line. The size and resolution of the GT link image 740 may be the same as the size and resolution of the learning image 710, or the size and resolution of the GT link image 740 may be smaller than the size and resolution of the learning image 710.

[0080] The letter detection device 100 may adjust the thickness of the line within the GT link image 740 based on the size of each letter region 715. For example, the letter detection device 100 may determine the thickness of the line in proportion to the average value of the diagonal length of the letter region 715. For example, when the average value of the diagonal length of a first letter region corresponding to one letter is 10, and the average value of the diagonal length of a second letter region corresponding to another letter is 5, the letter detection device 100 may determine the thickness of the line corresponding to the first letter region to be twice the thickness of the line corresponding to the second letter region.

[0081] On the other hand, refer to Figures 5 to 7 The learning process described is based on the case where the position values ​​of the letter area 715 are annotated on the learning image 710. Generally, the number of learning images annotated with the position values ​​of the letter area is not large, so according to one embodiment, the letter detection model can be learned by using the learning images annotated with the position values ​​of the vocabulary area.

[0082] Figure 8 This is a diagram for explaining a method of training a letter detection model using a learning image to which position values ​​of vocabulary areas are labeled. Figure 8 The process shown can be equivalent to the weakly supervised learning process.

[0083] In step S810, the letter detection device 100 obtains a learning image with a position value of a vocabulary region annotated thereon. The position value of the vocabulary region may be equivalent to a coordinate value of the vocabulary region in the learning image.

[0084] As described above, the vocabulary may include at least one letter, and the position value of the vocabulary area may refer to the position value of the polygon including the vocabulary. For example, when the vocabulary area is a quadrilateral, the position value of the vocabulary area may refer to the coordinate information of the corners of the quadrilateral. The shape of the vocabulary area may also be equivalent to a polygon or a circle other than a quadrilateral.

[0085] In step S820, letter detection apparatus 100 extracts vocabulary region images corresponding to vocabulary regions from the learning image. When the number of vocabulary regions included in the learning image is plural, letter detection apparatus 100 may extract plural vocabulary region images respectively corresponding to the plural vocabulary regions.

[0086] In step S830, the letter detection device 100 inputs the vocabulary region image into the letter detection model. In response to the input of the vocabulary region image, the letter detection model outputs an intermediate output image corresponding to the vocabulary region image. As described above, since the letter detection model is trained to detect the positions of each letter included in the image, the letter detection model can output an intermediate output image that represents the existence probability of each letter included in the vocabulary region image. The intermediate output image can include at least one of a probability value image and a link image.

[0087] When there are multiple vocabulary region images, the letter detection device 100 may further input the multiple vocabulary region images into the letter detection model respectively to obtain multiple intermediate output images.

[0088] In step S840, the letter detection device 100 checks the position value of the letter area in the vocabulary area image from the intermediate output image. The position value of the letter area may be equivalent to the coordinate value of the letter area on the learning image or the coordinate value of the letter area on the vocabulary area image.

[0089] When the position value of the letter area on the vocabulary area image is confirmed, the letter detection device 100 can change the position value of the confirmed letter area to the position value of the letter area on the learning image. The letter detection device 100 can change the position value of the letter area on the vocabulary area image to the position value of the letter area on the learning image in consideration of the coordinate relationship between the vocabulary areas included in the vocabulary area image and the learning image. For example, when the coordinates of the left top and right top of the vocabulary area image are confirmed and the coordinates of the left top and right top of the vocabulary area included in the learning image are confirmed, the letter detection device 100 can change the coordinate value of the letter area included in the vocabulary area image to the coordinate value on the learning image based on the relationship between the coordinate values ​​of the left top and right top in the vocabulary area image and the coordinate values ​​of the left top and right top of the vocabulary area on the learning image.

[0090] In step S850 , when the position value of the letter region included in the learning image is confirmed, the letter detection device 100 generates a pseudo GT image based on the position value of the letter region.

[0091] As reference Figure 7 As described above, the letter detection device 100 can generate a pseudo GT probability value image by distorting a two-dimensional Gaussian map according to the position value of each letter region detected from the learning image, and generate a pseudo GT link image by connecting the centers of each letter region.

[0092] According to an embodiment, the letter detection device 100 can omit the process of detecting the letter area from the intermediate output image and use the intermediate output image itself as a pseudo GT image. Since the intermediate output image is an image corresponding to the vocabulary area in the learning image, the letter detection device 100 can generate an image having a specific pixel value by making it correspond to other areas in the learning image except the vocabulary area, and can also generate a pseudo GT image by combining the generated image and the intermediate output image.

[0093] Reference Fig. 9 Describe the learning process of the alphabet learning model.

[0094] A vocabulary region image 920 corresponding to the vocabulary region is extracted from the learning image 910 on which the position value 915 of the vocabulary region is annotated, and the extracted vocabulary region image 920 is input to the letter detection model. Based on the input of the vocabulary region image 920, the letter detection model outputs an intermediate output image 930. Furthermore, the learning image 910 is input to the letter detection model, and the letter detection model outputs an output image 925 corresponding to the learning image 910.

[0095] The position value 935 of the letter region in the learning image 910 is determined from the intermediate output image 930. Based on the position value 935 of the letter region, a pseudo GT image 940 corresponding to the learning image 910 is generated.

[0096] Reference Fig. 9 , a calibration weight value 945 is applied to the loss value 950 between the output image 925 and the pseudo GT image 940, and the calibration weight value 945 is a value calculated in consideration of the accuracy of the intermediate output image 930. In the case where the amount of training of the letter detection model is insufficient, the detection accuracy of the intermediate output image 930 is necessarily low, and therefore, the calibration weight value 945 is applied to the loss value 950 in consideration of this.

[0097] The calibration weight value 945 may be calculated based on the result of comparing the number of letters detected from the intermediate output image 930 and the number of correct answers of letters included in the vocabulary area image 920. The number of correct answers of letters included in the vocabulary area image 920 may be calculated by analyzing the vocabulary area image 920 by the letter detecting apparatus 100, or the number of correct answers may be input into the letter detecting apparatus 100.

[0098] The letter detection device 100 may distinguish the intermediate output image 930 into at least one region by applying an image segmentation algorithm (e.g., a watershed algorithm) to locations in the intermediate output image 930 where the probability value is equal to or higher than a threshold value, and may determine the number of distinguished regions as the number of letters detected from the intermediate output image 930. In the case where the amount of training of the letter detection model is insufficient, there is a high probability that the number of letters detected from the intermediate output image 930 is less than the actual number, and therefore, the calibration weight value 945 may be calculated by comparing the number of letters detected from the intermediate output image 930 with the number of correct answers for the letters included in the vocabulary area image 920.

[0099] When the length (or number) of letters detected from the intermediate output image 930 is assumed to be detected_length, and the correct length (or number) of letters included in the vocabulary area image 920 is assumed to be gt_length, the error value may be calculated according to the following Math 1.

[0100] [Mathematical formula 1]

[0101] err=|gt_length-detected_length|

[0102] In the case where there are multiple vocabulary area images 920 extracted from the learning image 910, the detected_length may refer to the length (or number) of all letters detected from the multiple intermediate output images 930, and gt_length may refer to the correct answer length (or number) of all letters included in the multiple vocabulary area images 920.

[0103] Based on the error value, the calibration weight value can be calculated according to the following mathematical formula 2.

[0104] [Mathematical formula 2]

[0105] weight=alpha*|gt_length-err| / gt_length

[0106] In Mathematical Formula 2, alpha is a preset value, for example, it can be set to 0.9. In this case, when err is 0, the weight value is 0.9, and when only 7 letters are detected from 10 letters in the intermediate output image 930, the weight can be 0.9*7 / 10, that is, 0.63.

[0107] According to one embodiment, weight can be determined inversely proportional to err. In other words, when err is large, weight can be determined to be small; when err is small, weight can be determined to be large. As the weight value increases, the weighted loss increases, and the learning of the letter detection model occurs more frequently. Therefore, the learning intensity can be increased inversely proportional to the size of err.

[0108] The calibration weight value 945 calculated based on the result of comparing the number of letters detected from the intermediate output image 930 and the number of correct answers of letters included in the vocabulary area image 920 is multiplied by the loss value between the output image 925 as the output of the letter detection model and the pseudo GT image 950. Since the multiplied result, that is, the weighted value loss 955 is input to the letter detection model, the weighted value of the letter detection model can be updated.

[0109] Fig.16 It is shown that according to the above description Figure 6 and Fig. 9 The training process diagram of Fig.16, an image with word region position values ​​annotated (Word annotation) and a word region image cropped from the image are input into a letter detection model. Furthermore, a pseudo GT image can be generated based on an intermediate output image output by the letter detection model corresponding to the word region image, and a loss (Loss) between an output image output by the letter detection model and the pseudo GT image can be calculated, wherein the output image corresponds to the image annotated with the word region position values.

[0110] Furthermore, since the image with the position values ​​of the letter regions annotated (Character annotation) is input into the letter detection model, the letter detection model can generate an output image, and the Loss between the GT image generated based on the position values ​​of the letter regions and the output image can be calculated.

[0111] The calculated Loss is propagated to the letter detection model, thereby updating the internal weight value of the letter detection model.

[0112] Fig.10 The following is a diagram showing a process of specifying the position value of the letter region 1065 from the learning image 1010 to which the position value of the vocabulary region 1015 is labeled.

[0113] As described above, the vocabulary region image corresponding to the vocabulary region 1015 can be extracted from the learning image 1010 to which the position value of the vocabulary region 1015 is annotated.

[0114] According to one embodiment, the letter detection device 100 may deform the extracted vocabulary area image according to a preset image processing algorithm. For example, the preset image processing algorithm may include image distortion. Due to the distortion of the viewpoint, the arrangement direction of the letters or the size of the letters in the vocabulary area included in the actual image may have a large variability. Therefore, according to one embodiment, in order to easily detect the letters, the vocabulary area image is deformed into a vocabulary area image such as a rectangular shape. Fig.10 It can be seen that in the vocabulary region 1015 including the word "cold" in the learning image 1010, the height of the right corner is greater than the height of the left corner. The letter detection device 100 can deform the vocabulary region image extracted from the learning image 1010 by an image processing algorithm to obtain a deformed vocabulary region image 1020 in which the heights of the left corner and the right corner are the same.

[0115] In the case where there are multiple vocabulary regions 1015 , multiple vocabulary region images may be extracted, and each of the multiple vocabulary region images may be deformed according to a preset image processing algorithm, so that multiple deformed vocabulary region images 1020 may be obtained.

[0116] The letter detection apparatus 100 inputs the deformed vocabulary region image 1020 into the letter detection model, and obtains an intermediate output image 1030 output by the letter detection model.

[0117] In order to determine the letter region in the intermediate output image 1030, a region growing method may be applied. For example, the letter detection device 100 may determine a location corresponding to a probability value equal to or higher than a first threshold in the intermediate output image 1030. And, the letter detection device 100 may increase the region from the determined location to a location having a probability value above a second threshold value less than the first threshold. The letter detection device 100 may determine the polygonal region including the finally increased region 1045 as the letter region 1050.

[0118] According to an embodiment, the letter detection apparatus 100 may further apply an image segmentation algorithm (eg, a watershed method) to the intermediate output image 1030 to determine the region 1045 identified from the letter level map 1040 as the letter region 1050 .

[0119] Since the sizes and resolutions of the deformed vocabulary region image 1020 and the intermediate output image 1030 correspond to each other, the position values ​​of the letter regions determined in the intermediate output image 1030 can be directly used as the position values ​​of the letter regions 1050 in the deformed vocabulary region image 1020. The letter detection apparatus 100 can change the position values ​​of the letter regions 1050 in the deformed vocabulary region image 1020 to the position values ​​of the letter regions 1065 in the learning image 1010.

[0120] According to one embodiment, the letter detection device 100 can inversely deform (for example, de-distort) the deformed vocabulary area image 1020 in which the position value of the letter area 1050 is detected according to an image processing algorithm, and align the inversely deformed vocabulary area image with the learning image 1010 to determine the position value of the letter area 1065 on the learning image 1010.

[0121] According to another example, the letter detection device 100 may change the position value of the letter area 1050 detected from the deformed vocabulary area image 1020 to the position value of the learning image 1010 in consideration of the relationship between the coordinate value of the preset location of the deformed vocabulary area image 1020 and the coordinate value of the preset location of the vocabulary area 1015 included in the learning image 1010.

[0122] When the position values ​​of the letter regions 1065 in the learning image 1010 are labeled, the letter detection device 100 can generate a pseudo GT image based on the position values ​​of the letter regions 1065 in the learning image 1010 .

[0123] Fig.11 is a diagram showing an intermediate output image output when a vocabulary region image is input into a letter detection model at each epoch, Fig.12 is a diagram for explaining another method of detecting position values ​​of letter regions from a vocabulary region image. Fig.11 and Fig.12 , the following description is based on vocabulary region images, however, in addition to vocabulary region images, the description can also be applied to Fig.10 Described inflected vocabulary area image.

[0124] As mentioned above, when the amount of training for the letter detection model is insufficient, the accuracy of the intermediate output image output based on the vocabulary region image may be reduced. Fig.11 It can be confirmed that the accuracy of the intermediate output image output when the vocabulary area image including 'Alice' is input to the letter detection model increases with the increase of epochs. For example, only one letter can be detected in the first epoch, but five letters can be detected in the tenth epoch.

[0125] As described above, the letter detection device 100 may compare the number of letters detected from the intermediate output image and the number of correct solutions of letters included in the vocabulary area image to determine the calibration weight value to be applied to the loss value. However, in the case where the number of letters detected from the intermediate output image is less than a preset number, the letter detection device 100 may not use the intermediate output image when determining the position value of the letter area.

[0126] For example, in the case where the number of letters detected from the intermediate output image is less than a preset ratio (e.g., 0.5) of the number of correct answers to the letters included in the vocabulary region image, the letter detection device 100 may not use the intermediate output image. In the case where the number of letters detected from the intermediate output image is 4 and the number of correct answers to the letters included in the vocabulary region image is 10, the letter detection device 100 may confirm that only less than 5 (i.e., 0.5 of 10) letters are detected and the intermediate output image is not used.

[0127] Instead of identifying the letter region from the intermediate output image, the letter detection device 100 may segment the vocabulary region image according to the number of correct answers and determine each segmented region as a letter region. Fig.12 Since 'Alice' consists of five letters, the letter detection device 100 may evenly divide the vocabulary region image 1200 into five regions 1210, 1220, 1230, 1240, and 1250, and then determine each region as a letter region. In addition, the letter detection device 100 may determine the calibration weight value applied to the loss value to be a preset value (e.g., 0.5).

[0128] On the other hand, when the letter detection apparatus 100 obtains an output image including at least one of a probability value image and a link image according to an input of an image including letters, a bounding box can be represented in the input image based on the output image. The bounding box can be represented in letter units or vocabulary units.

[0129] The letter detection device 100 may determine a location corresponding to a probability value equal to or higher than a first threshold value in the output probability value image. Furthermore, the letter detection device 100 may increase an area starting from the determined location to a location having a probability value higher than a second threshold value less than the first threshold value. The letter detection device 100 may determine a polygonal area including the finally increased area as a letter area. The letter detection device 100 may represent a bounding box corresponding to the letter area determined in the probability value image in the input image and output it.

[0130] Furthermore, the letter detection device 100 may combine the letter regions determined in the probability value image in consideration of the connectivity of the link image, and may determine the region generated by the combination as the vocabulary region. For example, the letter detection device 100 may combine the letter regions corresponding to the same line in the link image among the letter regions determined in the probability value image. Furthermore, the letter detection device 100 may represent a bounding box corresponding to the vocabulary region on the input image and output it.

[0131] According to one embodiment, for combining letter areas determined in the probability value image, the letter detection device 100 can combine letter areas corresponding to locations in the probability value image corresponding to probability values ​​equal to or greater than a first threshold with locations in the link image having values ​​equal to or greater than a third threshold.

[0132] According to an embodiment, in addition to the letter detection model, the letter detection device 100 may further include a vocabulary detection model. When a learning image without any annotation is input, the letter detection model may detect the position value of the vocabulary area from the learning image through the vocabulary detection model, and the position value of the detected vocabulary area may be used to train the letter detection model.

[0133] Fig.13 2 is a diagram showing a configuration of a letter detection device 100 according to an embodiment.

[0134] Reference Fig.13 According to an embodiment, the letter detection device 100 may include a memory 1310 and a processor 1330. The memory 1310 may store at least one instruction, and the processor 1330 may control the detection of letters in an image and the training of a letter detection model according to the at least one instruction.

[0135] Although Fig.13 Only one memory 1310 and one processor 1330 are shown, but the letter detection device 100 may include a plurality of memories 1310 and may also include a plurality of processors 1330 .

[0136] The letter detection device 100 may be included in a server device, or in a client device, or in a standalone device.

[0137] Fig.14 is a functional block diagram of a letter detection device 100 according to an embodiment.

[0138] Reference Fig.14 , the letter detection device 100 may include a letter detection model 1410 , a data processing unit 1430 and a model training control unit 1450 .

[0139] At least one of the letter detection model 1410, the data processing unit 1430 and the model training control unit 1450 can be embodied as the processor 1330, and instructions for controlling the actions of at least one of the letter detection model 1410, the data processing unit 1430 and the model training control unit 1450 can be stored in the memory 1310.

[0140] According to one embodiment, the letter detection model 1410 may be stored in the memory 1310, and the data processing unit 1430 and the model training control unit 1450 may be embodied as the processor 1330. Alternatively, according to one embodiment, the letter detection model 1410 may be embodied as a neural network processor, and the data processing unit 1430 and the model training control unit 1450 may be embodied as a general processor.

[0141] The data processing unit 1430 obtains an image requiring letter detection and inputs it into the letter detection model 1410. In addition, in order to train the letter detection model, the data processing unit 1430 may obtain a learning image and transfer it to the model training control unit 1450. The learning image may include a learning image annotated with a position value of a vocabulary region or a learning image annotated with a position value of a letter region.

[0142] The model training control unit 1450 inputs the learning image into the letter detection model 1410. The model training control unit 1450 may perform preprocessing before inputting the learning image into the letter detection model 1410. The model training control unit 1450 may generate a GT image or a pseudo GT image from the learning image.

[0143] The letter detection model 1410 may receive input of a learning image and a correct image (i.e., a GT image or a pseudo GT image) from the model training control unit 1450 to update the internal weighting value. The letter detection model 1410 may compare the output image corresponding to the learning image with the correct image to update the internal weighting value.

[0144] When the output image is outputted corresponding to the image inputted into the letter detection model 1410, the data processing unit 1430 may also represent a bounding box on the input image based on the output image. The bounding box may be represented in letter units or vocabulary units.

[0145] Although not shown, the letter detecting device 100 may further include a communication unit that may transmit an output image output by the letter detecting model 1410 and an image processed by the data processing unit 1430 to an external device.

[0146] Fig.15 is a diagram showing a server device 1510 and a client device 1520 to which the letter detection device 100 according to an embodiment can be applied.

[0147] As described above, the letter detection device 100 according to an embodiment may be embodied as a server device 1510. In this case, the server device 1510 may receive an image to be processed from the client device 1520. The server device 1510 may obtain an output image by inputting the received image into a letter detection model.

[0148] The server device 1510 may transmit the output image to the client device 1520. Also, the server device 1510 may represent a bounding box corresponding to a letter region or a vocabulary region in the image received from the client device 1520 on the image and transmit it to the client device 1520.

[0149] The server device 1510 may receive learning data from an external device including the client device 1520 , or may control the training of the letter detection model using learning data stored internally.

[0150] The letter detection device 100 according to an embodiment may also be included in a client device 1520. The client device 1520 may receive data for executing a letter detection model from the server device 1510. The client device 1520 may input an image photographed by a camera module, an image stored in an internal memory, or an image received from an external device into the letter detection model to detect letters within the image.

[0151] The client device 1520 may receive learning data from an external device, or may control the training of the letter detection model using the learning data stored internally. According to an embodiment, the server device 1510 that provides the client device 1520 with data for executing the letter detection model may control the training of the letter detection model based on the learning data. In this case, the server device 1510 may transmit only the weighted value information updated according to the training result to the client device 1520, and the client device 1520 may update the letter detection model according to the received information.

[0152] Fig.15 A desktop computer is shown as the client device 1520, but not limited thereto, the client device 1520 may include a laptop computer, a smartphone, a tablet computer, an artificial intelligence (AI) robot, an AI speaker, a wearable device, etc.

[0153] According to an embodiment, the letter detection device 100 may be included in a standalone device, rather than in a server-client model. The standalone device may control the training of a built-in letter detection model, and may detect letters in an image by inputting the obtained image into the letter detection model.

[0154] On the other hand, the above-described embodiments of the present disclosure may be written as a program that can be executed on a computer, and the written program may be stored in a medium.

[0155] The medium may continue to store programs that can be executed on a computer, or temporarily store them for execution or downloading. Alternatively, the medium may be various recording devices or storage devices in the form of a single or multiple hardware combinations, and may be dispersed on a network, without being limited to media that directly access any computer system. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as compact disc read-only memory (CD-ROM) and high-density digital video disks (DVD), magneto-optical media such as floptical disks, and media that include read-only memory (ROM), random access memory (RAM), flash memory, etc., and are configured to store program instructions. In addition, as other examples of the medium, it may also include application stores that distribute application programs, sites that supply or distribute various other software, and recording media or storage media managed by servers, etc.

[0156] The technical concept of the present disclosure is described in detail above with reference to preferred embodiments, but the technical concept of the present disclosure is not limited to the embodiments, and a person skilled in the art can make various modifications and changes within the scope of the technical concept of the present disclosure.

Claims

1. A letter detection method, the letter detection method being performed by a letter detection device, characterized in that: The method comprises the following steps: training a letter detection model including a neural network; and; inputting an input image into the trained letter detection model to obtain at least one output image output from the letter detection model, The output image includes a probability value image, which represents the probability of the existence of letters in the input image. The training steps include: determining, from a first learning image annotated with a position value of a vocabulary region including at least one letter, a position value of a letter region within the vocabulary region; and generating a pseudo true value image corresponding to the first learning image based on the determined position value of the letter area, wherein the letter detection model is trained according to a comparison result between an output image output by the letter detection model corresponding to the first learning image and the pseudo true value image, The step of generating the pseudo true value image comprises the following steps: extracting a vocabulary region image corresponding to the position value of the vocabulary region from the first learning image; obtaining an intermediate output image corresponding to the vocabulary region image by inputting the extracted vocabulary region image into the letter detection model; determining, in each letter region detected from the intermediate output image, a position value of each letter region in the vocabulary region image; determining a position value of a letter region in the first learning image by using the position value of a letter region within the vocabulary region image; and A pseudo true value image corresponding to the first learning image is generated based on the determined position value of the letter region.

2. The letter detection method according to claim 1, characterized in that: The output image further comprises: A link image representing the connectivity between letters detected from the input image.

3. The letter detection method according to claim 2, characterized in that: The link image comprises lines connecting centers of adjacent letters detected from the input image, and, The thickness of the line represents the size of the letter area including each of the detected letters.

4. The letter detection method according to claim 1, characterized in that: The training step further comprises the following steps: A true value image is generated from a second learning image in which position values ​​of letter regions are annotated; wherein the letter detection model is trained based on a comparison result between an output image output by the letter detection model corresponding to the second learning image and the true value image.

5. The letter detection method according to claim 1, characterized in that: The training step further comprises the following steps: obtaining information on the number of correct answers for letters in the vocabulary area included in the first learning image; comparing the number of correct solutions for letters included in the vocabulary region with the number of letters detected from the intermediate output image to determine a calibration weight value; as well as applying the determined calibration weight value to a comparison result between an output image output by the letter detection model and the pseudo ground truth image, The intermediate output image is obtained based on a vocabulary region image corresponding to a position value of the vocabulary region.

6. The letter detection method according to claim 1, characterized in that: The step of determining the position value of each letter region in the vocabulary region image comprises the following steps: When the number of letters detected from the intermediate output image is less than a preset number, segmenting the vocabulary region image according to the number of correct answers for the letters; as well as The position values ​​of the respective divided regions divided from the vocabulary region image are determined as the position values ​​of the respective letter regions within the vocabulary region image.

7. The letter detection method according to claim 1, characterized in that: The step of obtaining the intermediate output image comprises the following steps: The vocabulary area image extracted by deformation according to an image processing algorithm; and obtaining an intermediate output image corresponding to the deformed vocabulary region image by inputting the deformed vocabulary region image into the letter detection model, The step of determining the position value of each letter region in the vocabulary region image comprises the following steps: determining the position value of each letter region within the deformed vocabulary region image, and, The step of determining the position value of the letter area in the first learning image comprises the following steps: The position value of the letter region in the first learning image is determined by using the position value of the letter region in the deformed vocabulary region image.

8. The letter detection method according to claim 2, characterized in that: The letter detection method further comprises the following steps: determining in the probability value image at least one location having a probability value equal to or greater than a first threshold; adding a boundary region from each of the at least one determined location to locations having a probability value above a second threshold value that is less than the first threshold value; as well as A bounding box corresponding to the added bounding region is represented on the input image.

9. A letter detection device, characterized in that: include: processor; as well as a memory storing at least one instruction, wherein the processor trains a letter detection model including a neural network according to the at least one instruction, By inputting an input image into the trained letter detection model to obtain at least one output image output from the letter detection model, The output image includes a probability value image, the probability value image indicating the probability of the presence of letters in the input image, To train the letter detection model, the processor determines position values ​​of letter regions within the vocabulary region from a first learning image annotated with position values ​​of the vocabulary region including at least one letter; and generates a pseudo-truth image corresponding to the first learning image based on the determined position values ​​of the letter regions, wherein the letter detection model is trained according to a comparison result between an output image output by the letter detection model corresponding to the first learning image and the pseudo true value image, Wherein, in order to generate the pseudo true value image, the processor extracts a vocabulary area image corresponding to the position value of the vocabulary area from the first learning image; obtains an intermediate output image corresponding to the vocabulary area image by inputting the extracted vocabulary area image into the letter detection model; determines the position value of each letter area in the vocabulary area image among each letter area detected from the intermediate output image; determines the position value of the letter area in the first learning image by using the position value of the letter area within the vocabulary area image; and generates a pseudo true value image corresponding to the first learning image based on the determined position value of the letter area.

10. A letter detection system, characterized in that: include: A server device that trains a letter detection model including a neural network by using a first learning image labeled with a position value of a vocabulary area including at least one letter; as well as a client device that receives data for executing the letter detection model from the server device, and when an input image is obtained, inputs the input image into the letter detection model to obtain at least one output image from the letter detection model, The output image includes a probability value image, which represents the probability of the existence of letters in the input image. In order to train the letter detection model, the server device determines the position value of the letter area in the vocabulary area from the first learning image; and generates a pseudo-truth image corresponding to the first learning image based on the determined position value of the letter area, wherein the letter detection model is trained according to a comparison result between an output image output by the letter detection model corresponding to the first learning image and the pseudo true value image, Wherein, in order to generate the pseudo true value image, the server device extracts a vocabulary area image corresponding to the position value of the vocabulary area from the first learning image; obtains an intermediate output image corresponding to the vocabulary area image by inputting the extracted vocabulary area image into the letter detection model; determines the position value of each letter area in the vocabulary area image among the letter areas detected from the intermediate output image; determines the position value of the letter area in the first learning image by using the position value of the letter area within the vocabulary area image; and generates a pseudo true value image corresponding to the first learning image based on the determined position value of the letter area.

Images