Electronic device and method for acquiring annotation information and storage medium

Abstract

The invention relates to an electronic device and method for acquiring annotation information and a storage medium. An electronic device may include a memory and a processor. The processor can obtain a first image containing a virtual license plate with a text object by using a template representing a license plate type; in a second image corresponding to a vehicle, based on identification of an actual license plate attached to the vehicle, position information of the actual license plate and color information of at least a part of the actual license plate are obtained; the color information can be utilized to change the color of at least one pixel contained in the first image; a first image including at least one pixel of which at least one color is changed can be combined with a second image based on position information, so that a third image of a synthesized vehicle including a virtual license plate is obtained; the third image, the text object, and a position information pair representing a position within the third image where the first image is combined may be stored as annotation information for training a neural network.

Description

Technical Field

[0001] The present disclosure relates to an electronic device, a method, and a non-transitory computer-readable storage medium for obtaining annotation information for neural network training. Background Art

[0002] With the development of image object recognition technology, various forms of services are emerging. The services can be applied to fields such as automatic driving, augmented reality, virtual reality, and metaverse, and can be provided through electronic devices owned by different users such as smartphones. The services may involve hardware and / or software mechanisms such as artificial intelligence that simulate human behavior and / or thinking. Technologies related to artificial intelligence may include neural network technologies that use simulated biological neural networks. Summary of the invention

[0003] Technical issues to be solved

[0004] The present disclosure aims to provide an electronic device, method and non-transitory computer-readable storage medium to obtain annotation information, thereby using a neural network to recognize a license plate attached to a vehicle.

[0005] The purpose of the present disclosure is to provide an electronic device, a method and a non-transitory computer-readable storage medium for identifying a license plate attached to a vehicle using a neural network.

[0006] The technical problems to be solved by the present disclosure are not limited to the above-mentioned technical problems, and ordinary technicians in the technical field to which the present disclosure belongs can clearly understand other technical problems not mentioned from the following description.

[0007] Technical Solution

[0008] An electronic device provided according to an embodiment may include a memory and a processor. The processor may use a template representing a license plate type to obtain a first image including a virtual license plate having a text object. The processor may, in a second image corresponding to a vehicle, obtain position information of the real license plate and color information of at least a portion of the real license plate based on identifying the real license plate attached to the vehicle. The processor may use the color information to change the color of at least one pixel included in the first image. The processor may combine the first image including at least one pixel whose color is changed with the second image based on the position information, thereby obtaining a third image of the vehicle synthesized with the virtual license plate. The processor may store a pair of the third image, the text object, and the position information representing the position in the third image to which the first image is combined as annotation information for training a neural network.

[0009] According to the method performed by the electronic device provided by the above embodiment, the following operations may be included. The method may use a template representing a license plate type to obtain a first image containing a virtual license plate having a text object. The method may, in a second image corresponding to a vehicle, obtain position information of the real license plate and color information of at least a portion of the real license plate based on identifying the real license plate attached to the vehicle. The method may use the color information to change the color of at least one pixel contained in the first image. The method may combine the first image containing at least one pixel whose color is changed with the second image based on the position information, thereby obtaining a third image of the vehicle synthesized with the virtual license plate. The method may store a pair of the third image, the text object, and the position information representing the position in the third image to which the first image is combined as annotation information for training a neural network.

[0010] According to one embodiment, a computer-readable storage medium is provided, the computer-readable storage medium being used to store one or more programs. The instructions contained in the one or more programs, when executed by a processor of an electronic device, enable the electronic device to obtain a first image containing a virtual license plate having a text object by using a template representing a license plate type. The instructions contained in the one or more programs, when executed by the processor of the electronic device, enable the electronic device to obtain, in a second image corresponding to a vehicle, position information of the real license plate and color information of at least a portion of the real license plate based on identifying the real license plate attached to the vehicle. The instructions contained in the one or more programs, when executed by the processor of the electronic device, enable the electronic device to change the color of at least one pixel contained in the first image by using the color information. The instructions contained in the one or more programs, when executed by the processor of the electronic device, enable the electronic device to combine the first image containing at least one pixel whose color is changed with the second image based on the position information, thereby obtaining a third image of the vehicle in which the virtual license plate is synthesized. The instructions contained in the one or more programs, when the one or more programs are executed by the processor of the electronic device, enable the electronic device to store a pair of position information of the third image, the text object and the position within the third image indicating the position where the first image is combined as annotation information for training a neural network.

[0011] Beneficial Effects

[0012] According to an embodiment of the present disclosure, an electronic device, method, and non-transitory computer-readable storage medium can obtain annotation information so as to use a neural network to identify a license plate attached to a vehicle.

[0013] According to an embodiment of the present disclosure, an electronic device, method, and non-transitory computer-readable storage medium can use a neural network to recognize a license plate attached to a vehicle.

[0014] The effects that can be obtained by the present disclosure are not limited to the above-mentioned effects, and ordinary technicians in the technical field to which the present disclosure belongs can clearly understand other effects that are not mentioned from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 An example of a block diagram of an electronic device according to an embodiment is shown.

[0016] Figure 2 An example of a virtual license plate template to be acquired by an electronic device according to an embodiment is shown.

[0017] Figure 3 An example of virtual license plate data acquired by an electronic device according to an embodiment is shown.

[0018] Figure 4 An example of a virtual license plate obtained by an electronic device according to an embodiment is shown.

[0019] Figure 5 An example of a neural network used by an electronic device to obtain a virtual license plate according to an embodiment is shown.

[0020] Figure 6 An example of a virtual license plate obtained by an electronic device using a neural network according to an embodiment is shown.

[0021] Figure 7 An example of a table of license plates used for recognition by an electronic device according to an embodiment is shown.

[0022] Figure 8a to Figure 8b An example of a table of license plate types used by an electronic device according to an embodiment is shown.

[0023] Fig. 9 An example of an image to be used by an electronic device to obtain location information according to an embodiment is shown.

[0024] Fig.10 An example of an operation of an electronic device according to an embodiment to obtain actual license plate position information attached to a vehicle is shown.

[0025] Fig.11 An example of an operation of an electronic device according to an embodiment using color information of an actual license plate to change the color of a virtual license plate is shown.

[0026] Fig.12 An example of an operation of an electronic device according to an embodiment to change the color of a virtual license plate using a histogram representing the color of an actual license plate is shown.

[0027] Fig.13 An example of a noise-based virtual license plate obtained by an electronic device according to an embodiment is shown.

[0028] Fig.14 An example of an operation of an electronic device projecting a first image onto a second image according to an embodiment is shown.

[0029] Fig.15An example of an operation in which an electronic device according to an embodiment combines a first image including a virtual license plate with a second image to obtain a third image and stores the third image as annotation information is shown.

[0030] Fig.16 An example of an operation flow chart of an electronic device provided according to an embodiment is shown.

[0031] Fig.17 is a diagram for explaining the operation of an electronic device for training a neural network based on a learning data set.

[0032] Fig.18 is a block diagram of an electronic device according to an embodiment.

[0033] Fig.19 An example of position information corresponding to a polygonal shape provided according to an embodiment is shown. DETAILED DESCRIPTION

[0034] Hereinafter, various embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0035] The various embodiments of the present disclosure and the terms used therein are not intended to limit the specific implementation of the technology described herein, but should be understood to include various changes, equivalents and / or substitutes of the relevant embodiments. Regarding the description of the drawings, similar constituent elements may use similar reference symbols. Unless the context has a clear different meaning, a singular expression may include a plural expression. In this article, expressions such as "A or B", "at least one of A and / or B", "A, B or C" or "at least one of A, B and / or C" may include all possible combinations of the enumerated items. Expressions such as "first", "second", "first" or "second" may modify the relevant constituent elements regardless of their order or importance, and are only used to distinguish one constituent element from another constituent element, and are not used to limit the relevant constituent elements. When referring to a certain (e.g., first) constituent element "(functionally or communicatively) connected" or "connected to" another (e.g., second) constituent element, this means that the certain constituent element may be directly connected to the other constituent element, or connected through other constituent elements (e.g., third constituent element).

[0036] The term "module" as used herein includes a unit composed of hardware, software or firmware, and can be used interchangeably with terms such as logic, logic block, component or circuit. A module can be an integral component, or a minimum unit or part thereof that performs one or more functions. For example, a module can be composed of an ASIC (application-specific integrated circuit).

[0037] Figure 1An example of a block diagram of an electronic device according to an embodiment is shown. Figure 1 According to an embodiment, the electronic device 101 provided may include at least one of a processor 120 or a memory 130. The processor 120 and the memory 130 may be electrically connected and / or operably coupled with each other through an electronic component such as a communication bus. Hereinafter, hardware operably coupled means that a direct connection or an indirect connection between hardware is established by a wired or wireless manner so that a second hardware is controlled by a first hardware in the hardware. Although the constituent elements of the electronic device 101 are shown in different blocks, the embodiments are not limited thereto. Figure 1 Some of the hardware in the electronic device 101 may be included in a single integrated circuit such as a SoC (system on a chip). The type and / or quantity of hardware included in the electronic device 101 is not limited to Figure 1 For example, the electronic device 101 may only include Figure 1 Part of the hardware shown.

[0038] According to one embodiment, the electronic device 101 may include hardware for processing data based on more than one instruction. The hardware for processing data may include a processor 120. For example, the hardware for processing data may include an arithmetic logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU) and / or an application processor (AP). The processor 120 may have a structure of a single-core processor, or may have a structure of a multi-core processor such as a dual core, a quad core, a hexa core or an octa core.

[0039] According to one embodiment, the memory 130 of the electronic device 101 may include a hardware element for storing data and / or instructions input to and / or output from the processor 120 of the electronic device 101. The memory 130, for example, may include a volatile memory such as a RAM (random-access memory) and / or a non-volatile memory such as a ROM (read-only memory). For example, the volatile memory may include at least one of a DRAM (dynamic RAM), a SRAM (static RAM), a Cache RAM, and a PSRAM (pseudo SRAM). For example, the non-volatile memory may include at least one of a PROM (programmable ROM), an EPROM (erasable PROM), an EEPROM (electrically erasable PROM), a flash memory, a hard disk, an optical disk, an SSD (solid state drive), and an eMMC (embedded multi media card).

[0040] For example, the memory 130 of the electronic device 101 may include a neural network (not shown). The electronic device 101 may recognize a license plate attached to a vehicle based on the neural network stored in the memory 130. For example, the electronic device 101 may train the neural network based on the image (or the annotation information 165) stored in the memory 130. The electronic device 101 may train the neural network based on a virtual license plate included in the image. The generation operation of the virtual license plate for the neural network learning by the electronic device 101 will be performed in Figures 2 to 6 Described in.

[0041] The memory 130 of the electronic device 101 provided according to one embodiment may include a template 150. The template 150 may include information about the type of license plate attached to the vehicle. The template 150 may take into account the license plate rules of various countries (e.g., "Notice on Standards for Motor Vehicle Registration Number Plates, etc."). For example, the type of template 150 may be different depending on the license plate rules of each country. The electronic device 101 may recognize different templates 150 under the license plate rules of different countries. For example, the template 150 may include a background template 151 representing the background of the license plate and / or a text template 153 representing a text object (e.g., numbers or letters) on the license plate. Figure 2 , the template 150 will be exemplarily described.

[0042] For example, the memory 130 of the electronic device 101 may include the first image 160. The first image 160 may include a virtual license plate obtained by the electronic device 101 using the template 150. The size of the first image 160 may correspond to the size of the virtual license plate. The operation of the electronic device 101 to obtain the first image 160 will refer to Figures 2 to 4 Give a description.

[0043] For example, the memory 130 of the electronic device 101 may include the second image 161. The second image 161 may be an example of an image of an actual environment including a vehicle. The second image 161 may include an actual license plate attached to the vehicle. The electronic device 101 may obtain position information and color information of the actual license plate using the second image 161. The electronic device 101 may change the color of the first image 160 using the color information, and combine the first image 160 with at least a portion of the second image 161 using the position information.

[0044] For example, the memory 130 of the electronic device 101 may include a third image 162. The third image 162 may include a vehicle synthesized with a virtual license plate by combining the second image 161 and the first image 160. For example, the electronic device 101 may store, in the memory 130, position information for indicating a position within the third image combined with the third image 162, the text object, and the first image 160 as annotation information 165. The position information may indicate a position (e.g., a coordinate value) at which the first image 160 overlaps within the third image. The annotation information 165 may be used to train a neural network for recognizing a license plate attached to a vehicle in an image.

[0045] As described above, the electronic device 101 provided according to an embodiment can obtain a virtual license plate for learning a neural network for recognizing a license plate. The electronic device 101 can avoid infringing user information corresponding to the actual license plate by using the virtual license plate for neural network learning, unlike the case of using the actual license plate of the vehicle included in the image corresponding to the actual environment.

[0046] The following will refer to Figure 2 , a template 150 for obtaining a virtual license plate is exemplarily described.

[0047] Figure 2 An example of a virtual license plate template to be acquired by an electronic device according to an embodiment is shown. Figure 2 The electronic device 101 may include Figure 1 The electronic device 101 in. Figure 2 At least one of the operations in Figure 1 The processor 120 in executes.

[0048] Reference Figure 2 , shows an example of a template 150 based on license plate rules of different countries. Template 150 may vary according to region, country and / or time. Template 150 may include a background template 151 representing a license plate background and / or a text template 153 representing a text object included in the license plate.

[0049] For example, the background template 151 may have a specified size. The specified size may vary according to the country or region. The specified size may also vary according to the purpose of the vehicle and / or the type of vehicle. The background template 151 may include visual objects, graphics and / or colors representing a region or country. The color of the background template 151 may vary according to the type of vehicle and / or the purpose of the vehicle. For example, when the electronic device 101 generates a virtual license plate corresponding to a first country (e.g., South Korea), the first background template 151-1 may be used to generate the virtual license plate. For another example, when the electronic device 101 generates a license plate corresponding to a second country (e.g., the United States), the second background template 151-2 may be used to generate the virtual license plate. The electronic device 101 may select the background template 151 based on the country or region corresponding to the virtual license plate to be generated.

[0050] For example, the text template 153 may include a text object to be included in the license plate. The text template 153 may include a text object 211 representing a letter and / or a text object 212 representing a number. The text object included in the text template 153 may have a specified size in order to be included in the background template 151. The specified size may be set according to the license plate rules of different countries. However, it is not limited to this. The text object included in the text template 153 may include letters based on a language (e.g., Chinese characters, Korean, letters and / or Hiragana). The text object may be identified based on a specified font. The language and / or specified font of the text object may vary according to region, country and / or time. For example, for the United States, the text object may include a text object based on a first font (e.g., penitentiary gothic), and for California, the text object may include a text object based on a second font (e.g., rage italic). However, it is not limited to this. If the national license plate rules change, the background template 151 and / or the text template 153 may also be changed accordingly.

[0051] According to an embodiment, the electronic device 101 can select at least one of the background templates 151 and combine at least one text object included in the text template 153 with the selected at least one background template, thereby obtaining a virtual license plate.

[0052] For example, the electronic device 101 can obtain the virtual license plate by combining text objects 211 representing letters and / or text objects 212 representing numbers. The number of text objects (e.g., text objects 211 representing letters and / or text objects 212 representing numbers) contained in the virtual license plate may vary according to region, country and / or time. For example, the electronic device 101 can select at least one of the text objects 211 representing letters based on the number of occurrences, and randomly select at least one of the text objects 212 representing numbers. However, it is not limited to this. For example, the electronic device 101 can combine the text objects in the text template in the background template based on the specified set values ​​(e.g., size, position and / or quantity) according to the license plate rules of different countries, thereby obtaining a virtual license plate.

[0053] Reference Figure 3 , exemplary data for obtaining a virtual license plate by an electronic device 101 provided according to an embodiment will be described.

[0054] Figure 3 An example of virtual license plate data acquired by an electronic device according to an embodiment is shown. Figure 3 The electronic device 101 may include Figure 1 to Figure 2 The electronic device 101 in. Figure 3 At least one of the operations in Figure 1 The processor 120 in executes.

[0055] Reference Figure 3 According to an embodiment, the electronic device 101 can utilize Figure 1 The virtual license plate 305 can be obtained based on the license plate rules of different countries. For example, the electronic device 101 can set a background template (e.g., Figure 1 The electronic device 101 can set the background template and the text template (for example, Figure 1 The electronic device 101 may set the position, size and / or type (e.g., letters or numbers) of the text objects to be included in the background template to obtain the virtual license plate 305. The electronic device 101 may set the type of the virtual license plate 305 according to the type of the license plate to be recognized by the neural network to be trained. The type of the license plate may be set based on the location (e.g., region or country) of the actual environment corresponding to the image containing the license plate.

[0056] For example, the electronic device 101 may identify data 310 for acquiring the virtual license plate 305. The data 310 may include information about the virtual license plate 305. For example, the information about the virtual license plate 305 may include the type of template, the color of the background template, the shape (e.g., rectangle) of the background template, the location (e.g., region or country) corresponding to the template, the size of the virtual license plate 305, and / or the text object (e.g., ). The data 310 may be included in the annotation information that the electronic device 101 enables the neural network for recognizing the license plate to learn. For example, the electronic device 101 may train the neural network using an image containing an actual license plate and data (e.g., data 310) containing information about the actual license plate.

[0057] For example, the data 310 may include information indicating the size of the virtual license plate 305. The line information 311 may indicate the center coordinates of a two-dimensional coordinate axis (e.g., Min X, Min Y). The center coordinates of the two-dimensional coordinate axis may be reference coordinates for identifying a position of the virtual license plate 305. For example, the line information 311 may be based on the shape of the virtual license plate 305 (e.g., Figure 3 ), including a position for representing the virtual license plate 305 (e.g., Figure 3 For example, line information 311 may represent the coordinates of position information 306 of the upper left corner of the virtual license plate 305 based on the two-dimensional coordinate axis (e.g., 1240.1408450704225, 1042.535211676057). Line information 312 may represent relative coordinates relative to the center coordinates (e.g., Max X, Max Y). For example, line information 312 may represent the coordinates of position information 307 of the lower right corner of the virtual license plate 305 based on the two-dimensional coordinate axis (e.g., 1493.661971830986, 1115.4225352112676). However, it is not limited to this. The electronic device 101 may use the coordinate values ​​of position information 306 and position information 307 to set the size of the virtual license plate. The size of the virtual license plate may vary depending on the type of the virtual license plate 305.

[0058] For example, data 310 may include information about the vehicle to be synthesized into a virtual license plate 305. For example, line information 313 representing information about the vehicle may include the type of the vehicle, the purpose of the vehicle, and / or the registration location of the vehicle. Line information 313 may be acquired based on the JSON format. The electronic device 101 may identify at least one information about the vehicle contained in line information 313 according to the type of virtual license plate 305 to be acquired. For example, for a first type (e.g., "type01"), the shape of the license plate may be a rectangle whose lateral length is greater than the longitudinal length. For example, the color of the text object contained in the license plate may be a first color (e.g., black). For a second type (e.g., "type02"), the shape of the license plate may be a rectangle whose lateral length is shorter than the lateral length of the rectangle of the first type and whose longitudinal length is longer than the longitudinal length of the rectangle of the first type. For example, the color of the text object contained in the license plate may be a first color (e.g., black). For a third type (e.g., "type03"), the shape of the license plate may be similar to that of the first type of license plate. For the third type, the color of the license plate may be different from the color of the license plate of the first type. Since the purpose of a vehicle corresponding to the first type of license plate may be different from that of a vehicle corresponding to the third type of license plate, the colors of the first type of license plate and the third type of license plate may be different. The color of the text object contained in the third type of license plate may be the second color (e.g., white). For the fourth type ("type04"), the shape of the license plate may be similar to the shape of the second type of license plate. The color of the text object contained in the fourth type of license plate may be the second color. However, it is not limited to this. The type of license plate may vary according to the region, country and / or purpose of the vehicle. The electronic device 101 can obtain information corresponding to the type from the identified license plate by identifying the type of license plate. For example, when selecting (e.g., true) information about the first type (e.g., "type01") and not selecting (e.g., false) information about other types (e.g., "type02" to "type ETC"), the electronic device 101 can obtain a virtual license plate 305 corresponding to the first type. The description of the JSON format will be described with reference to Table 1 below.

[0059]

Table 1

[0060]

[0061]

[0062] Table 1 according to an embodiment may include a portion of the information of the virtual license plate obtained by the electronic device 101 through the image. For example, the electronic device 101 may train a neural network using a portion of the information and a virtual license plate (or license plate image) corresponding to the information.

[0063] For example, referring to Table 1, "flags" may mean that the following information includes data (e.g., classification information) for indicating at least one license plate included in the image. The information corresponding to "flags" may be different in each part (or category) distinguished by "flags". "Flags" may be used to distinguish the information corresponding to "flags".

[0064] For example, "label" can represent a text object and / or a visual object that the electronic device 101 uses a neural network to identify. As an example, "label" can mean data that represents the correct answer. "Label" can be used as a reference for the electronic device 101 to guide the neural network to learn. As an example, when multiple license plates are identified in an image, data sets (e.g., label-to-flags) (or instructions) corresponding to each of the multiple license plates can be obtained. As an example, the data set below the "label" can be used to identify a text object corresponding to the "label". For example, the data set can be based on the position information contained in the license plate (e.g., the location information described later). Figure 4 The first position 411 in the license plate is obtained. The position information may vary depending on the shape of the license plate. For the description of the shape of the license plate, please refer to Figure 4 Since the location containing the text object and / or the visual object may vary depending on the shape of the license plate, the electronic device 101 may deduce the location information containing the text object by identifying the shape (or type) of the license plate. As an example, based on the type of virtual license plate to be generated, the number of text objects used by the electronic device 101 to set may vary. A data set based on the JSON format may be obtained based on the number of text objects.

[0065] For example, "point" can represent the position of the upper left corner of the rectangle (e.g., 1513.0806104480532, 923.088542783918) and / or the position of the lower right corner of the rectangle (e.g., 1653.8700841322636, 959.930480470897) based on the shape of the virtual license plate 305 (or a bounding box for identifying a position of the virtual license plate). As an example, "point" can vary depending on the shape of the virtual license plate 305 (or bounding box). As an example, when based on the shape of an n-gon (or polygon), "point" can contain n pieces of position information. For example, the position information based on the shape of an n-gon (or polygon) can be as follows: Fig.19 See Fig.19, the electronic device 101 may identify position information corresponding to corners 1910 - 1 , 1910 - 2 , 1910 - 3 , 1910 - 4 , 1910 - 5 , and 1910 - 6 of the polygon 1910 . However, this is not limited thereto.

[0066] For example, line information 311 and / or line information 312 may be an example of "point". For example, "shapetype" may represent a license plate shape corresponding to a license plate type. As an example, referring to Table 1, since the first type (e.g., "type01") in the information representing the license plate type is true and the others are false, the information contained in Table 1 may include a shape corresponding to the first type (e.g., a rectangle) and position information based on the first type (e.g., line information 311 and / or line information 312). As an example, referring to Table 1, "truck" and / or "passenger car" may be set based on the type of vehicle.

[0067] For example, when the color of the text object included in the virtual license plate is a first color (e.g., black), "NUMBERS" is true, and when it is a second color (e.g., non-black), "NUMBERS" is false. "num_n" can be used to identify a position (e.g., Figure 4 As an example, when the text object (e.g., number) contained in one position of the virtual license plate is 5, "num_5" can be set to true. "INVERSE_NUMBERS" is true when the color of the text object contained in the virtual license plate is the second color (e.g., white), and false when it is a color different from the second color (e.g., non-white). "num_inverse_n" can be used to identify the text object contained in one position of the virtual license plate (e.g., Figure 4 As an example, when the text object (e.g., number) contained in one position of the virtual license plate is 5, "num_inverse_5" can be set to true. However, this is not limited to this. Below, with reference to Tables 2 to 5, the information used to identify the text object (e.g., number, symbol, or letter) in the license plate will be described.

[0068]

Table 2

[0069]

[0070]

[0071] Table 2 may include information about text objects (e.g., letters) included in a license plate provided according to an embodiment. Referring to Table 2, the electronic device 101 provided according to an embodiment may identify a text object included in a position of a license plate. For example, when the color of the text object included in the license plate is a first color (e.g., black), "CHARACTERS" is set to true, and when it is a second color (e.g., non-black), it is set to false. to A variable indicating letter information contained in a position of a license plate may be represented.

[0072] For example, the variables may vary depending on the type of license plate, the region (or country) in which the license plate is used, and the language of the letters contained in the license plate. As an example, the variables may include words in Korean, English, Chinese characters, and / or Hiragana. As an example, the variables may include more than one text. As an example, when the electronic device 101 recognizes a text object corresponding to at least one of the variables, the variable corresponding to the text object may be set to true. The operation of the electronic device 101 to obtain a data set in which the variable corresponding to the text object is set to true will be described with reference to Tables 4 to 5.

[0073]

Table 3

[0074]

[0075]

[0076]

[0077] Table 3 may include information about text objects (e.g., letters) included in a license plate provided according to an embodiment. For example, when the color of the text object included in the license plate is a first color (e.g., white), "INVERSE_CHARACTERS" is set to true, and when it is a second color (e.g., non-white), it is set to false. to A variable that can represent letter information contained in a position of a license plate (e.g. or ). For example, the variables may vary depending on the type of license plate, the region (or country) in which the license plate is used, and the language of the letters contained in the license plate. As an example, the variables may include words in Korean, English, Chinese characters and / or Hiragana. As an example, the variables may contain more than one text. As an example, when the electronic device 101 recognizes a text object corresponding to at least one of the variables, the variable corresponding to the text object may be set to true. However, this is not limited to this. For example, when the electronic device 101 obtains a data set in JSON format based on the relevant position 305-1, the data set may be as shown in Table 4.

[0078]

Table 4

[0079]

[0080]

[0081]

[0082]

[0083]

[0084] Referring to Table 4, Table 4 may include information about the location 305-1 provided according to an embodiment. For example, "point" may include information related to the location information 306-1 (e.g., 1526.8438538205978, 929.5681063122923) and / or the location information 307-1 (e.g., 1540.1328903654485, 953.156146179402) about the location 305-1. For example, referring to Table 4, since the location 305-1 is an area for setting a text object, the information indicating the type of the license plate (e.g., "car", "truck", and / or "type01" to "typeETC") may be set to false. For example, referring to Table 4, since the location 305-1 contains a text object representing a number (e.g., 5), a variable corresponding to the number (e.g., "num_5") may be set to true. Other variables different from the variable corresponding to the number may be set to false. For example, referring to Table 4, the variable corresponding to the color of the text object (e.g., "NUMBERS") can be set to true, but is not limited thereto. For example, when the electronic device 101 obtains a data set in JSON format based on the relevant position 305-2, the data set of the relevant position 305-2 can be as shown in Table 5 described later.

[0085]

Table 5

[0086]

[0087]

[0088]

[0089]

[0090] Referring to Table 5 provided according to one embodiment, "point" can represent the relative coordinates of the position information 306 about the virtual license plate 305 based on the shape (e.g., rectangle) used to distinguish the position 305-2. "point" can represent the coordinate values ​​of the position information 306-2 and the position information 307-2. "point" can be set to the position coordinates of the upper left corner of the position (e.g., 1557.0764119601329, 930.5647840531561) and the position coordinates of the lower right corner (e.g., 1572.0265780730897, 953.4883720930231). However, it is not limited to this. For example, since the position 305-2 contains letters (e.g., ), so the variable related to "num_n" (or "num_inverse_n") can be set to false. For example, since position 305-2 contains a text object representing a letter (e.g., ), so the variable corresponding to that letter (e.g., ) is set to true, while other variables can be set to false. Referring to Table 5, variables representing license plate types (e.g., "car", "truck", "type01" to "typeETC") can be set to false. However, this is not limited to this. For example, the electronic device 101 can obtain data 310 using an image containing an actual license plate, and use the image and the data 310 to train a neural network for obtaining a virtual license plate. For example, the electronic device 101 can configure a text object and / or a visual object corresponding to the virtual license plate type at a specified position of the virtual license plate through a learned neural network, thereby obtaining a virtual license plate 305.

[0091] In addition, although the text objects configured at the specified position of the virtual license plate as an embodiment in this specification only include letters (Character) and numbers (Number), this is only described as an embodiment, and the embodiments of the present invention are not limited to this. For example, according to the present invention, the text that can be configured at the specified position of the virtual license plate may include text generated by a writing system for recording letters used globally, such as ASCII (American Standard Code for Information Interchange) code and / or Unicode, etc. The text that can be configured at the specified position of the virtual license plate may include text with specifications defined for information exchange through a computer.

[0092] Below, refer to Figure 4 , an example of a virtual license plate obtained by the electronic device 101 according to one embodiment is described.

[0093] Figure 4 An example of a virtual license plate obtained by an electronic device according to an embodiment is shown. Figure 4 The electronic device 101 may include Figures 1 to 3 The electronic device 101 in. Figure 4 At least one of the operations in Figure 1 The processor 120 in the embodiment is executed. Figure 4 , showing an example of a virtual license plate 401 obtained by an electronic device 101 according to an embodiment.

[0094] For example, the electronic device 101 may use Figure 3 The electronic device 101 may obtain the virtual license plate 401-1 based on the license plate rule corresponding to the first country (eg, South Korea).

[0095] For example, the virtual license plate 401-1 may be acquired based on a specified color (e.g., white). The text objects included in the virtual license plate 401-1 may include different information depending on the positions at which they are configured in the virtual license plate 401-1. Depending on the number and / or positions of the text objects included in the virtual license plate 401-1, the text objects may include time information (e.g., the year of revision of the license plate rules). A table representing text objects including different information depending on the positions at which they are configured will be referred to. Figures 7 to 8b Give a description.

[0096] For example, a text object representing a number (e.g., 12) located at the first position 411 may represent a type of vehicle (e.g., a passenger car). ) may indicate the purpose of the vehicle. For example, the text object representing a number located at the third position 413 may indicate a serial number for a specific vehicle. However, this is not limited to this. The position of the upper left corner and / or the position of the lower right corner of the first position 411 may be based on Figure 3 The number of text objects included in the first position 411 may be more than one. In the following, repeated descriptions of the position will be omitted.

[0097] According to an embodiment, the electronic device 101 may obtain the virtual license plate 401-2 using data corresponding to the virtual license plate 401-2 which is different from the virtual license plate 401-1. The virtual license plate 401-2 may include information about the region.

[0098] For example, the text object representing letters located at the first position 421 may represent a region name (eg, (Seoul))(region name, county name, province name). The text object representing a number (e.g., 52) located at the second position 422 may represent the type of vehicle. The text object representing a letter (e.g., ) may indicate the purpose of the vehicle. The text object (eg, 3108) representing a number located at the fourth position 424 may indicate a serial number for a vehicle to which a license plate is specifically attached. However, this is not limited thereto.

[0099] For example, the electronic device 101 may process the virtual license plate 401 - 1 or the virtual license plate 401 - 2 according to gray scaling based on the acquired virtual license plate 401 - 1 or the virtual license plate 401 - 2.

[0100] For example, the electronic device 101 may obtain a virtual license plate including a visual object (e.g., a Tai Chi pattern) representing a first country (e.g., South Korea) and / or a text object (e.g., "KOR") representing the first country. The color of one area of ​​the virtual license plate including the visual object and / or the text object may be different from the color of other areas.

[0101] According to an embodiment, the electronic device 101 may obtain the virtual license plate 401-3 according to the license plate rules corresponding to the second country (eg, China). For example, the electronic device 101 may generate the virtual license plate 401-3 using data corresponding to the virtual license plate 401-3.

[0102] For example, the virtual license plate 401-3 may include information about a region, information about time, and / or information for a specific vehicle. For example, a text object representing a letter (e.g., "Jing") located at the first position 431 may represent region information (e.g., Beijing). As an example, the text object representing a letter located at the first position 431 may be an abbreviation of a province within the second country. The text object representing a letter located at the second position 432 (e.g., "N") may include information about a region or use. The text object representing a letter located at the second position 432 may include information about the seat of the provincial government within the second country, a city of the province, a municipality directly under the central government, use, and / or time. However, it is not limited thereto. The text object representing a number or a letter located at the third position 433 (e.g., 888R8) may represent a serial number for a specific vehicle.

[0103] For example, the color of the virtual license plate 401-3 may include information about a vehicle according to the license plate rules corresponding to the second country. As an example, the first color (e.g., blue) may represent a passenger car. The second color (e.g., yellow) may represent a heavy passenger car, a bus, a truck, and / or a motorcycle. The third color (e.g., white) may represent the use of a vehicle (e.g., a police car, a military vehicle). The fourth color (e.g., black) may represent the use of a vehicle (e.g., a vehicle related to a foreign enterprise, an embassy, or a consulate). The fifth color (e.g., green) may represent the type of a vehicle (e.g., an electric vehicle). However, it is not limited thereto.

[0104] According to an embodiment, the electronic device 101 may obtain the virtual license plate 401-4 based on the license plate rules corresponding to a third country (e.g., Europe). For example, the electronic device 101 may obtain the virtual license plate 401-4 by using the data corresponding to the virtual license plate 401-4.

[0105] For example, virtual license plate 401-4 may include information about a region. For example, a visual object located at first position 441 may represent a united group (e.g., EU) associated with a third country. A text object (e.g., "EST") representing a letter located at second position 442 may represent information about a country. A blank (space) and a text object located at third position 443, fourth position 444, and fifth position 445 may represent a serial number for a particular vehicle. According to an embodiment, a visual object (e.g., a national flag) may be configured at third position 443, fourth position 444, or fifth position 445. According to an embodiment, the number of blank (space) and text objects located at third position 443, fourth position 444, and fifth position 445 may be different. The color of virtual license plate 401-4 may include information about a country (e.g., yellow or white for the United Kingdom). However, it is not limited thereto. For example, a virtual license plate corresponding to a fourth country (e.g., the United States) may also include a visual object.

[0106] According to an embodiment, the electronic device 101 may obtain a virtual license plate 401-5 based on a license plate rule corresponding to a fifth country (e.g., Japan). For example, the electronic device 101 may obtain the virtual license plate 401-5 using data corresponding to the virtual license plate 401-5. For example, the virtual license plate 401-5 may contain information about a region. For example, a text object located at a first position 451 may represent a region. A text object located at a second position 452 may represent a business purpose for which a vehicle is used. A text object located at a third position 453 (e.g., 500) may represent a type and / or purpose of a vehicle. A text object located at a fourth position 454 (e.g., 46-49) may represent information for identifying a vehicle. The color of the virtual license plate 401-5 may be distinguished as a first color (e.g., white), a second color (e.g., green), a third color (e.g., yellow), and / or a fourth color (e.g., black) according to the purpose of the vehicle. However, it is not limited thereto.

[0107] For example, the electronic device 101 can identify sub-positions 451-1, 451-2, 452, 453-1, 453-2, 453-3, 454-1, 454-2, 454-3, 454-4, 454-5 for setting text objects according to the type of the virtual license plate 401-5. The number of sub-positions may vary according to the type of the virtual license plate. As an example, referring to the accompanying drawings, the number of text objects included in the sub-position is shown as 1, but according to the embodiment, multiple text objects may also be included. The electronic device 101 can obtain the following information based on 1 data set related to the type (Type) of the virtual license plate 401-5 and the number of sub-positions 451-1, 451-2, 452, 453-1, 453-2, 453-3, 454-1, 454-2, 454-3, 454-4, and 454-5 in the virtual license plate 401-5 (for example, a total of 11 including 4 letters (451-1, 451-2, 452, and 454-3) and 7 numbers (453-1, 453-2, 453-3, 454-1, 454-2, 454-4, and 454-5)), a total of 12 data sets. Figure 3 The data sets based on JSON format shown in Tables 1 to 9. Each of the data sets may include at least partially different information based on the type (e.g., numbers or letters) of the text objects contained in the sub-positions 451-1, 451-2, 453-1, 453-2, 453-3, 454-1, 454-2, 454-3, 454-4, 454-5. The electronic device 101 may set the text objects (e.g., numbers or letters) specified according to the type of the virtual license plate 401-5 to each sub-position 451-1, 451-2, 452, 453-1, 453-2, 453-3, 454-1, 454-2, 454-3, 454-4, 454-5 based on the information corresponding to the type of the virtual license plate 401-5, so that the neural network can be trained to obtain the virtual license plate 401-5.

[0108] In other words, the electronic device 101 provided according to an embodiment can generate the coordinate values ​​of the virtual license plate 401-5 area based on the type of the virtual license plate 401-5 from the data file annotated in the JSON format, as well as the letters specified in the coordinate values ​​of the relevant sub-areas 451-1, 451-2, 452, 453-1, 453-2, 453-3, 454-1, 454-2, 454-3, 454-4, and 454-5 in the virtual license plate 401-5. In addition, the shape of the designated area in the virtual license plate 401-5 in this specification adopts a rectangle, so it is described that only two values ​​of the left-top coordinates and the right-bottom coordinates of the designated area in the virtual license plate 401-5 are used, but it is not limited to this. That is, if the shape of the designated area in the virtual license plate 401-5 adopts other shapes such as an N-gon or a circle, the value required for conveniently specifying the position of the shape area can be used.

[0109] According to an embodiment, the electronic device 101 may obtain virtual license plates 401 based on various types according to the license plate rules of the corresponding country. Figure 4 As an example of the virtual license plate 401, the license plates stipulated by countries such as South Korea, the United States, Europe, China, and Japan are shown, but this is only an embodiment and does not limit the embodiments of the present invention. In addition, the electronic device 101 can train a neural network for recognizing license plates to recognize virtual license plates based on various types of virtual license plates 401 based on the acquisition of the various types. The electronic device 101 can set visual objects and / or text objects in different positions according to the type of license plate (for example, virtual license plates 401-1, 401-2, 401-3, 401-4, 401-5) and the license plate rules of the corresponding country. The electronic device 101 can train a neural network to recognize (or set) text objects and / or visual objects contained in the license plate based on the position information according to the license plate type.

[0110] The virtual license plate 401 obtained by the electronic device 101 according to the embodiment provided as described above may not be an actually existing license plate. For example, the virtual license plate 401 obtained by the electronic device 101 may not match the vehicle contained in the image to be synthesized with the virtual license plate 401. In order to generate a virtual license plate 401 that does not actually exist, the electronic device 101 may randomly combine text objects contained in the virtual license plate 401. Based on generating a virtual license plate 401 that does not actually exist, the electronic device 101 can reduce the infringement of personal information.

[0111] Below, refer to Figure 5 , an example of a neural network for obtaining a virtual license plate provided according to an embodiment is described.

[0112] Figure 5 An example of a neural network used by an electronic device to obtain a virtual license plate according to an embodiment is shown. Figure 5 The electronic device may include Figures 1 to 4 The electronic device 101 in. Figure 5 At least one of the operations in Figure 1 The processor 120 executes.

[0113] Reference Figure 5 According to one embodiment, the electronic device may include a neural network 505 (e.g., a conditional generative adversarial network (CGAN)). The neural network 505 may include more than one neural network (e.g., a generator neural network or a discriminator neural network).

[0114] For example, the neural network 505 may include more than one layer. Each of the more than one layer may include functions and / or filters for computing data. Each of the more than one layer may be a convolutional filter, a fully connected layer, or various types of filters or layers combined based on specific functions or features. The neural network 505 may include more than one layer to form a deep neural network. The process of causing a deep neural network to learn is called deep learning.

[0115] For example, nodes included in more than one layer may be connected to each other by connecting lines having connection weights. The tuning of the neural network may be based on, for example, supervised learning, unsupervised learning, and / or adversarial learning. For example, the neural network 505 may include a generative neural network 505-1 for acquiring a license plate and / or a discriminative neural network 505-2 for discriminating the acquired license plate. The electronic device may train the neural network for generating a license plate by adversarial learning using the generative neural network 505-1 and / or the discriminative neural network 505-2. For example, in one embodiment, the electronic device may use a generative adversarial network (GAN) to generate a license plate by causing two neural networks to learn competitively at the same time.

[0116] According to one embodiment, the electronic device can recognize license plates based on a specified number (e.g., 30). The types of the license plates can be different. The electronic device can recognize information of at least one license plate 501 among the recognized license plates. In order to generate the license plate 501 among the recognized license plates, the electronic device can use a neural network 505.

[0117] For example, the electronic device may identify a data set 502 related to a license plate 501. The data set 502 may be input into a neural network 505. The electronic device may input the data set 502 into the neural network 505 so that the neural network 505 learns to obtain an image corresponding to the license plate 501. As an example, in order to obtain an image related to the license plate 501, the electronic device may identify a template corresponding to the license plate rules of the country related to the license plate 501. The electronic device may use the template, a condition vector 503 based on a specified number (e.g., 30) of license plates, and the license plate to be obtained to obtain the license plate through the neural network 505. However, this is not limited thereto.

[0118] For example, the electronic device can use the generative neural network 505-1 to input the acquired license plate into the discriminative neural network 505-2 based on the acquisition of the license plate. The electronic device can use the discriminative neural network 505-2 to determine whether the acquired license plate is acquired or whether it is based on one of a specified number of license plates. The electronic device can use the generative neural network 505-1 and the discriminative neural network 505-2 to acquire license plates based on various types based on performing adversarial learning.

[0119] Below, refer to Figure 6 , illustrating an example of a virtual license plate obtained by an electronic device using a neural network 505.

[0120] Figure 6 An example of a virtual license plate obtained by an electronic device using a neural network according to an embodiment is shown. Figure 6 The electronic device 101 may include Figures 1 to 5 The electronic device 101 in. Figure 6 At least one of the operations in Figure 1 The processor 120 in executes.

[0121] Reference Figure 6 According to one embodiment, the electronic device 101 Figure 5 The neural network 505 in the example can use the template to obtain a virtual license plate 611 based on the outline.

[0122] For example, the electronic device 101 can use the virtual license plate 611 based on the contour line to obtain a virtual license plate 612 similar to the actual license plate by at least partially changing the pixel values ​​contained in the virtual license plate 611 based on the contour line. The electronic device 101 can obtain the virtual license plate 612 through a neural network (e.g., cycle GAN). For example, the electronic device 101 can cause the neural network to learn to change the virtual license plate 611 based on the contour line into a virtual license plate 612 similar to the actual license plate. For example, the electronic device 101 can store the virtual license plate 612 obtained by the neural network as the annotation data 615 for learning the neural network for recognizing the license plate in the memory. The electronic device 101 can store the annotation data 615 containing more than one virtual license plate in the memory so that the neural network for recognizing the license plate can learn.

[0123] Below, refer to Figures 7 to 8b , illustrating an example table of at least a portion of license plate rules of various countries corresponding to a template used by the electronic device 101 to obtain a virtual license plate according to an embodiment.

[0124] Figure 7 An example of a table of license plates used for recognition by an electronic device according to an embodiment is shown. Figure 8a to Figure 8b An example of a table of license plate types used by an electronic device according to an embodiment is shown. Figures 7 to 8b The electronic device 101 may include Figures 1 to 6 The electronic device 101 in.

[0125] Reference Figure 7 According to an embodiment, table 700 may represent information of text objects to be used by the electronic device 101 to obtain a virtual license plate corresponding to a first country (e.g., South Korea). For example, referring to table 700, the classification column may be divided into vehicle types (e.g., by vehicle model) and vehicle purposes (e.g., by purpose).

[0126] For example, the category column can be divided into passenger cars, vans, trucks, special cars such as cranes and snowplows, and emergency cars such as fire trucks and police cars. The symbol column can be distinguished based on the number range assigned by vehicle type.

[0127] According to an embodiment, the electronic device 101 may place a text object representing at least one number included in the number range corresponding to the symbol column in the Figure 4 The first position (e.g., 411) or Figure 4in a second position (e.g., 422).

[0128] For example, passenger cars can be divided into non-business use, rental business use, and general business use. Passenger cars for non-business use and / or rental business use can correspond to a first range (e.g., 100 to 699). Passenger cars for general business use can correspond to a second range (e.g., 01 to 69).

[0129] For example, vans can be divided into non-business use, rental business use, and general business use. Non-business use and / or rental business use vans can correspond to the third range (e.g., 700 to 799). General business use vans can correspond to the fourth range (e.g., 70 to 79).

[0130] For example, trucks can be divided into non-commercial and general commercial. Non-commercial trucks can correspond to the fifth range (e.g., 800 to 979). General commercial trucks can correspond to the sixth range (e.g., 80 to 97).

[0131] For example, special vehicles can be divided into non-commercial, rental business and general commercial. Non-commercial and rental business special vehicles can correspond to the seventh range (eg, 980 to 997).

[0132] For example, emergency vehicles can be divided into police cars and fire trucks. Emergency vehicles can correspond to the eighth range (for example, 998 to 999). But it is not limited to the above examples.

[0133] According to an embodiment, the columns by purpose may be divided into non-commercial use, for automobile transportation businesses, and for diplomacy. The columns by purpose may correspond to text objects representing letters.

[0134] According to an embodiment, the electronic device 101 may configure at least one of the text objects in Figure 4 The second position 412 and / or Figure 4 The third position 423.

[0135] For example, non-commercial use can be classified as private car. Private car can correspond to the first text object (e.g., ).

[0136] For example, the automobile transportation business can be divided into general use (for general) and rental business use (for rental businesses). For example, the general use automobile transportation business can correspond to the second text object (for example, ). The car transportation business for the rental business may correspond to a third text object (for example, ).

[0137] For example, diplomatic use can be divided into diplomat use, consular use, quasi-diplomat use, quasi-consular use, international organization use, and other diplomatic use. As an example, text objects related to diplomatic use can be configured in Figure 4 For example, when the electronic device 101 obtains a virtual license plate using a text object related to foreign affairs, the color of the virtual license plate obtained using the text object may be a second color (eg, blue) different from the first color (eg, white). However, this is not limited thereto.

[0138] For example, the word diplomat can correspond to the fourth text object (e.g., (Diplomacy)). Consular use can correspond to the fifth text object (for example, (Consul)). Quasi-diplomats can correspond to the sixth text object (for example, (quasi-consular)). Quasi-consular can correspond to the seventh text object (for example, (Approved)). International organizations can correspond to the eighth text object (for example, (State Machinery). Other diplomatic uses can correspond to the ninth text object (for example, (Agreement, Representative)). However, it is not limited thereto.

[0139] According to an embodiment, the electronic device 101 may place a text object representing at least one number in a numerical range corresponding to a vehicle model and at least one text object representing letters corresponding to a purpose at a specified position in the background template, thereby obtaining a virtual license plate. The electronic device 101 may obtain a virtual license plate similar to an actual license plate by using a table 700 corresponding to license plate rules of various countries and based on identifying text objects representing numbers and text objects representing letters.

[0140] Reference Figure 8a , shows a table 800-1 showing license plate types according to an embodiment. Figure 8b, shows a table 800-2 indicating license plate types according to an embodiment. Tables 800-1 and 800-2 may include usage, specifications, and samples according to the types. As an example, the license plate colors included in the samples may vary according to the types. For example, the electronic device 101 may use the type-related information included in tables 800-1 and 800-2 to identify data for obtaining a virtual license plate. For example, the type-related information may be included in Figure 3 If the electronic device 101 selects information about the first type (e.g., Type01) in the line information 313, and does not select information about other types (e.g., Type02 to Type17), the electronic device 101 can obtain a virtual license plate corresponding to the first type (e.g., a virtual license plate similar to a sample corresponding to Type 1). However, this is not limited thereto.

[0141] For example, the electronic device 101 may utilize license plate rules of other countries that are different from the country associated with table 700 , table 800 - 1 , and / or table 800 - 2 (eg, South Korea) to obtain a virtual license plate that is similar to an actual license plate of the other country.

[0142] As described above, according to one embodiment, the electronic device 101 can obtain a virtual license plate based on a specified size by combining a text object representing at least one number included in table 700 and a text object representing at least one letter included in table 800-1 and table 800-2. The electronic device 101 can obtain a virtual license plate similar to an actual license plate using information included in table 700, table 800-1, and / or table 800-2 corresponding to license plate rules of various countries. The electronic device 101 can train a neural network for recognizing license plates based on obtaining a virtual license plate similar to an actual license plate.

[0143] Below, refer to Figures 9 and 10 , illustrating an example of an operation of the electronic device 101 according to one embodiment to obtain position information of an actual license plate attached to a vehicle contained in an image representing an actual environment.

[0144] Fig. 9 An example of an image to be used by an electronic device to obtain location information according to an embodiment is shown. Fig.10 An example of an operation of an electronic device according to an embodiment to obtain actual license plate position information attached to a vehicle is shown. Figures 9 and 10 The electronic device 101 may include Figures 1 to 8b An electronic device 101. Figures 9 and 10 At least one of the operations can be Figure 1 Executed by processor 120.

[0145] Reference Fig. 9 According to one embodiment, the electronic device 101 may store the Figure 1 The one or more images 900 including the vehicle are identified in the memory 130 of the electronic device 101. The one or more images 900 may represent images of the actual environment including the vehicle. The one or more images 900 may include the second image 161. Each of the one or more images 900 may be compared with the first image (e.g., the virtual license plate) including the virtual license plate acquired by the electronic device 101. Figure 1 The operation of combining the second image 161 in the one or more images 900 with the first image will refer to Figure 14 to Figure 15 Give a description.

[0146] According to one embodiment, the electronic device 101 can identify the vehicle 905 in the second image 161 corresponding to the vehicle. For example, the electronic device 101 can identify the actual license plate 910 attached to the vehicle 905. The electronic device 101 can identify vertices 901 of the actual license plate 910. For example, the electronic device 101 can identify input indicating the selection of the vertex. However, it is not limited to this. For example, the electronic device 101 can use the vertex 901 to identify the corners corresponding to the actual license plate 910 (for example, the corners constituting the bounding box).

[0147] Reference Fig.10 According to one embodiment, the electronic device 101 may change the position of the corner of the actual license plate 910 obtained using the vertex 901. For example, the electronic device 101 may identify the position information of the actual license plate 910 using the pixels included in the second image 161. For example, the electronic device 101 may identify the pixels including the actual license plate 910 from the pixels included in the second image 161 using the position information of the vertex 901. The electronic device 101 may identify the edges and / or vertices of the actual license plate 910 using the pixels including the actual license plate 910. For example, the electronic device 101 may identify the corners adjacent to the corners of the actual license plate 910 obtained using the vertex 901 (e.g., the corners obtained using the vertex 1010).

[0148] For example, the sub-image 1020 cropped from the second image 161 based on the edges of the actual license plate 910 may include the vertices 1010 of the actual license plate 910 .

[0149] For example, the electronic device 101 may utilize pixels containing the actual license plate 910 to Fig. 9 The first corner corresponding to the actual license plate 910 identified by the vertex 901 in the figure is changed to the second corner of the recognized actual license plate 910. The second corner can be based on Fig.10As an example, the electronic device 101 can Fig. 9 The position of each vertex 901 in is changed to Fig.10 The positions of the vertices 1010 in FIG. 10 are changed to the positions of the vertices 1010 in FIG. 10. For example, the position of the vertex 901-1 can be changed to the position of the vertex 1010-1. The position of the vertex 901-2 can be changed to the position of the vertex 1010-2. The position of the vertex 901-3 can be changed to the position of the vertex 1010-3. The position of the vertex 901-4 can be changed to the position of the vertex 1010-4.

[0150] For example, the electronic device 101 may identify Fig. 9 The position information (e.g., coordinate values) of each second corner of each first corner corresponding to the actual license plate 910 identified by using the vertex 901 in the image 161 is located at the shortest distance (e.g., Manhattan distance). The position information of the second corners may correspond to the position information of the actual license plate 910 included in the second image 161. For example, based on identifying each vertex 1010, the electronic device 101 may identify the position information of the actual license plate included in the second image 161. The position information may be used by the electronic device 101 to project a virtual license plate.

[0151] For example, the electronic device 101 may use Fig. 9 The electronic device 101 recognizes the first position information of the actual license plate 910 based on the input about the vertex 901 in the image 161, and recognizes the pixels including the actual license plate 910 from the pixels included in the second image 161 using the recognized approximate first position information of the actual license plate 910. The electronic device 101 can recognize the corners constituting the actual license plate 910 in the pixels including the actual license plate 910. The electronic device 101 can obtain the position information of the actual license plate 910 using the corners.

[0152] Below, refer to Figure 11 to Figure 12 , describes an example of an operation of the electronic device 101 provided according to an embodiment for changing the color of a virtual license plate obtained using a template.

[0153] Fig.11 An example of an operation of an electronic device according to an embodiment using color information of an actual license plate to change the color of a virtual license plate is shown. Fig.12 An example of an operation of an electronic device according to an embodiment to change the color of a virtual license plate using a histogram representing the color of an actual license plate is shown. Figure 11 to Figure 12 The electronic device 101 may include Figures 1 to 10 The electronic device 101 in.

[0154] Reference Fig.11According to one embodiment, the electronic device 101 may recognize the actual license plate 1110 in the second image 161. The electronic device 101 may recognize the color information of each pixel included in the actual license plate 1110 in the second image 161.

[0155] For example, the electronic device 101 can recognize at least a portion of the actual license plate 1110. The electronic device 101 can recognize a text object contained in the actual license plate 1110. The electronic device 101 can sample pixels contained in the at least a portion of the actual license plate that is different from other portions containing the text object. In order to temporarily avoid (for refraining) obtaining color information of other portions containing the text object, the electronic device 101 can identify RGB values ​​corresponding to other portions. As an example, the electronic device 101 can temporarily avoid (refraining) obtaining color information of pixels whose RGB values ​​of each pixel contained in the actual license plate 1110 are lower than a specified value (for example, 90). Based on temporarily avoiding (refraining) obtaining color information of pixels whose RGB values ​​are lower than the specified value, the electronic device 101 can temporarily avoid (refraining) obtaining color information corresponding to the other portions.

[0156] For example, the electronic device 101 may obtain color information 1120 including RGB values ​​1121 of a first region 1111 and RGB values ​​1122 of a second region 1112 in a portion of the actual license plate 1110. For example, the electronic device 101 may obtain color information 1120 including RGB values ​​(e.g., about 300) of a plurality of regions (including the first region 1111 and the second region 1112).

[0157] According to one embodiment, the electronic device 101 can identify the virtual license plate 1150 (or the information including the virtual license plate 1150). Figure 1 For example, the electronic device 101 may use the color information 1120 to change the color of at least one pixel included in the virtual license plate 1150.

[0158] For example, the electronic device 101 may change the color of at least a portion of the virtual license plate 1150. The electronic device 101 may change the color of at least a portion of the virtual license plate 1150 that is different from other portions that include text objects. For example, the electronic device 101 may obtain color information of the virtual license plate 1150, the color information including the RGB value 1131 of the first area 1141 and the RGB value 1132 of the second area 1142 included in at least a portion of the virtual license plate 1150. As an example, since the virtual license plate 1150 is generated by the electronic device 101, the color of at least a portion of the virtual license plate 1150 may be substantially the same. However, this is not limited to this.

[0159] For example, the electronic device 101 may change the color of at least one pixel included in the virtual license plate 1150 (or the first image 160 including the virtual license plate 1150) by using the color information 1120 representing the RGB value corresponding to the color of the actual license plate 1110, based on changing the RGB value (e.g., RGB value 1131 and RGB value 1132) corresponding to the color (e.g., color information 1130) of the virtual license plate 1150. For example, the electronic device 101 may change the RGB value of a certain area of ​​the virtual license plate 1150 by using at least one RGB value included in the color information 1120. As an example, the electronic device 101 may change (or replace) the RGB value 1132 of the second area 1142 of the virtual license plate 1150 by using the RGB value 1121 of the first area 1111 of the actual license plate 1110. However, it is not limited thereto.

[0160] Reference Fig.12 According to one embodiment, the electronic device 101 may obtain, in the second image 161, RGB values ​​of the first region 1111 having the largest RGB value and the second region 1112 having the smallest RGB value among the pixels included in the second image 161. For example, the RGB value 1221 of the first region 1111 may be the largest RGB value of the pixels included in the second image 161. For example, the RGB value 1222 of the second region 1112 may be the smallest RGB value of the pixels included in the second image 161. The electronic device 101 may obtain color information 1220 including the RGB value 1121 and the RGB value 1222. The color information 1220 may be represented by a histogram 1230 based on color and frequency.

[0161] For example, the electronic device 101 may identify color information about the virtual license plate 1150 (or the first image 160 including the virtual license plate 1150). The color information of the virtual license plate 1150 may be represented by a histogram (not shown). The electronic device 101 may change the histogram of the virtual license plate 1150 using the histogram 1230 of the actual license plate 1110. The electronic device 101 may change the histogram of the virtual license plate 1150 to a histogram 1230 similar to the actual license plate 1110. For example, based on the electronic device 101 changing the histogram of the virtual license plate 1150, the color information of the virtual license plate 1150 corresponding to the histogram may be changed. For example, the electronic device 101 may change the color of at least a portion 1240 of the virtual license plate 1150 using the histogram 1230. At least a portion 1240 may be distinguished from other portions including a text object in the virtual license plate 1150. For example, the electronic device 101 may use Math Formula 1 to change the color of at least one pixel included in the virtual license plate 1150 (or the first image 160 including the virtual license plate 1150 ).

[0162]

Mathematical formula 1

[0163] X_new(X_input-X_min / / X_max-X_min)*255

[0164] X_new may represent at least one of the RGB values ​​of at least one pixel in the virtual license plate 1150 after the color change. X_input may represent at least one of the RGB values ​​of the at least one pixel before the color change. X_min may represent the minimum value of the RGB values ​​obtained using the actual license plate 1110 (e.g., one of the RGB values ​​1222). X_max may represent the maximum value of the RGB values ​​obtained using the actual license plate 1110 (e.g., one of the RGB values ​​1221). For example, the electronic device 101 may change the color of at least one pixel included in the virtual license plate 1150 using Mathematical Formula 1.

[0165] According to the electronic device 101 provided by the embodiment as described above, the color of the virtual license plate 1150 can be changed based on the color information of the actual license plate 1110 to obtain the virtual license plate 1150 having a color similar to the actual license plate 1110 .

[0166] Below, refer to Fig.13 , describes an example of an operation of the electronic device 101 obtaining noise using the virtual license plate 1150 after the color is changed according to an embodiment.

[0167] Fig.13 An example of a noise-based virtual license plate obtained by an electronic device according to an embodiment is shown. Fig.13 The electronic device 101 may include Figures 1 to 12 An electronic device 101. Fig.13 At least one of the operations can be Figure 1 The virtual license plate 1300 may refer to the virtual license plate 1150 after the color is changed.

[0168] According to an embodiment, the electronic device 101 may combine the noise 1310 with the virtual license plate 1300. For example, the electronic device 101 may generate the noise 1310 with respect to the virtual license plate 1300 by changing parameters representing pixels included in the virtual license plate 1300.

[0169] For example, the electronic device 101 may generate noise 1310 on the virtual license plate 1300 based on salt and pepper noise for generating at least one point. The electronic device 101 may generate noise 1310 based on Gaussian noise. The electronic device 101 may generate noise 1310 based on uniform noise. The electronic device 101 may generate noise 1310 based on at least one of resizing, gradient, filter (e.g., bilateral filter), and / or morphology.

[0170] For example, the electronic device 101 may generate the noise 1310 using more than one operation. The electronic device 101 may obtain the noise 1310 using at least one of salt and pepper noise, gradient, resampling, and / or Gaussian noise. For example, the electronic device 101 may obtain the noise 1310 using at least one of resampling, morphological operations (e.g., dilation), filters (e.g., bilateral filters), salt and pepper noise, and / or gradient. For example, the electronic device 101 may use a filter to change the parameter values ​​of the pixels included in the virtual license plate 1300, thereby rendering the first image 160 including the noise 1310. The electronic device 101 may use the filter to perform the opposite operation of the denoising filter.

[0171] For example, the electronic device 101 may use a kernel (e.g., a kernel related to morphological operations) to change parameter values ​​of pixels related to the virtual license plate 1300. The electronic device 101 may use the kernel to generate noise 1310 related to the virtual license plate 1300 by changing the parameter value to at least one of 0 or 1. The electronic device 101 may generate noise 1310 related to the virtual license plate 1300 based on generating noise 1310. Figure 1A first image of a virtual license plate similar to the actual license plate in the second image 161 (e.g., Figure 1 The electronic device 101 may combine the first image with the second image based on the generated noise 1310 to store the combined image as labeling information for learning by a neural network for recognizing a license plate.

[0172] According to the embodiments described above, the electronic device 101 can obtain a virtual license plate similar to an actual license plate based on generating noise 1310 on the virtual license plate 1300 after the color is changed.

[0173] Below, refer to Fig.14 , describing an example of an operation in which the electronic device 101 projects a first image including noise 1310 and a virtual license plate 1300 onto at least a portion of a second image.

[0174] Fig.14 An example of an operation of an electronic device projecting a first image onto a second image according to an embodiment is shown. Fig.14 The electronic device 101 may include Figures 1 to 13 An electronic device 101.

[0175] refer to Fig.14 According to one embodiment, the electronic device 101 may combine the first image 160 including the virtual license plate with the second image 161. The electronic device 101 may project the first image 160 onto at least a portion of the second image 161 based on homography. Homography may represent the corresponding relationship between projected matching points when one image is projected onto another image.

[0176] For example, the electronic device 101 may utilize Fig.10 The electronic device 101 may project the first image 160 onto at least a portion of the second image 161 based on the position information of the actual license plate identified by the vertex 1010 in the first image 160. For example, the electronic device 101 may utilize the position information to project the virtual license plate (e.g., the virtual license plate included in the first image 160) onto the actual license plate included in the second image 161 by using a matrix that maps each vertex of the virtual license plate to each vertex 1010 of the actual license plate. For example, the electronic device 101 may project the first image 160 onto at least a portion of the second image 161 based on the planar surface 1400. For example, the matrix may be represented by Mathematical Formula 2.

[0177]

Mathematical formula 2

[0178]

[0179] For example, H may represent a scale factor based on a homogeneous coordinate system. H may correspond to the matrix. A homogeneous coordinate system may refer to a coordinate system that represents a first coordinate value (e.g., (x, y)) based on a two-dimensional coordinate system as a second coordinate value (e.g., (x, y, 1)). h11, h12, h13, h21, h22, h23, h31, h32, and / or h33 may represent elements of the matrix H. For example, referring to Fig.14 , the electronic device 101 may, based on homography, convert a vertex (eg, Fig.14 x in the second image 161) is mapped to a vertex of the actual license plate contained in the second image 161 (for example, Fig.14 For example, the electronic device 101 can map each vertex of the virtual license plate included in the first image 160 to each vertex 1110 of the actual license plate based on Mathematical Formula 2 and using the position information of the actual license plate.

[0180] Fig.15 An example of an operation in which an electronic device according to an embodiment combines a first image including a virtual license plate with a second image to obtain a third image and stores the third image as annotation information is shown. Fig.15 The electronic device 101 may include Figures 1 to 14 An electronic device 101. Fig.15 At least one of the operations in Figure 1 Executed by processor 120.

[0181] Reference Fig.15 According to one embodiment, the electronic device 101 may recognize the actual license plate (eg, Fig. 9 For example, the electronic device 101 may project each vertex of the first image 160 (or the virtual license plate included in the first image) onto each vertex of the actual license plate based on homography.

[0182] For example, the electronic device 101 may combine the first image 160 and the second image 161 based on the position information of the actual license plate. The electronic device 101 may obtain a third image 162 including the vehicle 905 synthesized with the virtual license plate included in the first image 160 .

[0183] For example, the electronic device 101 may include a text object (eg, ), position information indicating the position in the third image combined with the first image 160, and the third image 162 are stored as annotation information 165. The annotation information 165 can be used to train a neural network for recognizing license plates.

[0184] As described above, according to one embodiment, the electronic device 101 may utilize a template based on the license plate rules of various countries (e.g., Figure 1 The electronic device 101 can identify the color information of the actual license plate attached to the vehicle 905 contained in the second image 161, and the position information of the actual license plate in the second image 161. The electronic device 101 can change the color of at least one pixel contained in the virtual license plate using the color information. The electronic device 101 can generate noise by changing the parameters of the pixels contained in the virtual license plate whose color has been changed. The electronic device 101 can obtain the first image 160 containing the virtual license plate in which the noise is generated. The electronic device 101 can project the first image 160 onto the actual license plate contained in the second image 161 based on homography using the position information. The electronic device 101 can use the third image 162 obtained by combining the first image 160 and the second image 161 for learning the neural network. The electronic device 101 can store the third image 162 as annotation information 165 in the memory. The electronic device 101 can use the template to obtain the annotation information 165 that enables the neural network for recognizing the license plate to learn.

[0185] Fig.16 An example of an operation flow chart of an electronic device provided according to an embodiment is shown. Fig.16 At least one of the operations in Figure 1 The electronic device 101 and / or Figure 1 The processor 120 in the processor is executed. Fig.16 The operations in can be performed in sequence, but they do not necessarily have to be performed in sequence. For example, the order of the operations can be changed, and at least two operations can be performed in parallel.

[0186] Reference Fig.16 In operation 1610, a processor according to an embodiment may utilize a template representing a license plate type (e.g., Figure 1 template 150), obtaining a first image containing a virtual license plate having a text object (e.g., Figure 1The first image 160 of the vehicle. For example, the type of virtual license plate obtained by the electronic device 101 may vary based on the license plate rules of each country. The virtual license plate may include a text object representing a number, a text object representing a letter, a visual object representing a location (e.g., a country or region), and / or a graphic. The color of the virtual license plate may vary depending on the type of vehicle. For example, in order to obtain the virtual license plate, the processor may use a neural network based on adversarial learning (e.g., Figure 5 Neural network 505). However, it is not limited to this.

[0187] Reference Fig.16 In operation 1620, a processor according to an embodiment may, in conjunction with a vehicle (eg, Fig. 9 The second image corresponding to the vehicle 905) (eg, Figure 1 In the second image 161), based on identifying the actual license plate attached to the vehicle (e.g., Fig. 9 The processor may obtain the position information of the actual license plate and the color information of at least a part of the actual license plate. For example, the processor may use a histogram to represent the color information of at least a part of the actual license plate. The processor may obtain the color information including the RGB value of at least a part of the actual license plate. The histogram may be represented based on each of the RGB values. For example, the processor may obtain the RGB value of the pixel contained in at least a part of the actual license plate that is different from other parts corresponding to the text object contained in the actual license plate. The processor may sample the RGB value based on a specified number of pixels.

[0188] Reference Fig.16 In operation 1630, a processor according to an embodiment may change the color of at least one pixel included in the first image using color information. For example, the processor may change (or replace) each RGB value of the pixel included in the first image with each RGB value corresponding to the actual license plate. For example, the processor may change a histogram representing the colors of the pixels included in the first image based on a histogram corresponding to the actual license plate. For example, the processor may combine noise (e.g., color information) into the first image including at least one pixel whose color is changed based on at least one of salt and pepper noise, Gaussian noise, uniform noise, resampling, gradient, filter, and / or morphological operations. Fig.13 Noise in 1310).

[0189] Reference Fig.16 In operation 1640, the processor according to an embodiment may combine the first image including at least one pixel whose at least one color is changed with the second image based on the position information, thereby obtaining a third image of the vehicle synthesized with the virtual license plate (for example, Figure 1The processor may combine the first image and the second image by mapping each vertex of the first image to each vertex of the actual license plate contained in the second image using the position information. The processor may project each vertex of the first image to each vertex of the actual license plate contained in the second image based on homography.

[0190] Reference Fig.16 In operation 1650, the processor according to an embodiment may use a pair of the third image, the text object, and the position information in the third image indicating the position in which the first image is combined as annotation information for training the neural network (e.g., Figure 1 The processor may train a neural network for recognizing a license plate using the labeled information. The operation of the processor training the neural network may be an example of supervised learning. However, it is not limited thereto.

[0191] Fig.17 is a diagram for explaining the operation of the electronic device 101 for training a neural network based on a learning data set according to an embodiment. Fig.17 The operation described can be performed by Figures 1 to 16 The electronic device is executed.

[0192] refer to Fig.17 In operation 1702, according to an embodiment, the electronic device may obtain a learning data set. The electronic device may obtain a learning data set for supervised learning. The learning data may include input data and a ground truth data pair corresponding to the input data. The ground truth data may represent output data to be obtained from a neural network that receives input data of the ground truth data pair. The ground truth data may be obtained by referring to Figures 1 to 12 An electronic device is described for obtaining.

[0193] For example, when training a neural network to recognize an image, the learning data may include information about the image and one or more objects contained in the image. The information may include a category or class of an object that can be recognized by the image. The information may include the position, width, height, and / or size of a visual object corresponding to the object in the image. The learning data set identified by operation 1702 may include multiple learning data pairs. In the example of training a neural network to recognize an image, the learning data set identified by the electronic device may include multiple images and ground truth data corresponding to each of the multiple images.

[0194] Reference Fig.17 In operation 1704, according to one embodiment, the electronic device may train the neural network based on the learning data set. In one embodiment of training the neural network based on supervised learning, the electronic device may input the input data contained in the learning data into the input layer of the neural network. Fig.18 An example of a neural network including the input layer is described. From the output layer of the neural network that receives the input data through the input layer, the electronic device can obtain output data of the neural network corresponding to the input data.

[0195] In one embodiment, the training of operation 1704 may be performed based on the difference between the output data and the ground truth data corresponding to the input data included in the learning data. For example, the electronic device may adjust one or more parameters related to the neural network (e.g., the reference Fig.18 The electronic device may adjust the one or more parameters to reduce the difference. The operation of the electronic device to adjust the one or more parameters may be referred to as tuning the neural network. The electronic device may perform neural network tuning based on the output data by using a function defined for evaluating the performance of the neural network, such as a cost function. The difference between the output data and the ground truth data may be included in an example of the cost function.

[0196] refer to Fig.17 In operation 1706, according to an embodiment, the electronic device may identify whether the neural network trained by operation 1704 outputs valid output data. The output data is valid if the difference (or cost function) between the output data and the ground truth data satisfies the conditions set for using the neural network. For example, when the average value and / or the maximum value of the difference between the output data and the ground truth data is less than or equal to a specified threshold, the electronic device may determine that the neural network outputs valid output data.

[0197] If the neural network does not output valid output data (1706-No), the electronic device may repeatedly perform training of the neural network based on operation 1704. The embodiment is not limited thereto, and the electronic device may repeatedly perform operations 1702 and 1704.

[0198] In a state where valid output data is obtained from the neural network (1706-Yes), the electronic device according to one embodiment may use the trained neural network based on operation 1708. For example, the electronic device may input other input data different from the input data input to the neural network as learning data to the neural network. The electronic device may use the output data obtained from the neural network that receives the other input data as a result of reasoning the other input data based on the neural network.

[0199] Fig.18 is a block diagram of an electronic device 101 according to an embodiment. Fig.18 The electronic device 101 may include Figures 1 to 17 For example, refer to Fig.17 The operation described can be performed by Fig.18 The electronic device 101 and / or Fig.18 Executed by processor 1810.

[0200] Reference Fig.18 , the processor 1810 of the electronic device 101 can perform computations related to the neural network 1830 stored in the memory 1820. The processor 1810 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), or a neural processing unit (NPU). The NPU can be implemented as a chip separate from the CPU, or integrated on the same chip as the CPU in the form of a system on a chip (SoC). The NPU integrated in the CPU can be called a neural core and / or an artificial intelligence (AI) accelerator.

[0201] Reference Fig.18, the processor 1810 may identify a neural network 1830 stored in the memory 1820. The neural network 1830 may include a combination of an input layer 1832, one or more hidden layers 1834, and an output layer 1836. Each of the above layers (e.g., the input layer 1832, one or more hidden layers 1834, and the output layer 1836) may include a plurality of nodes. The number of hidden layers 1834 may vary depending on the embodiment, and a neural network 1830 including a plurality of hidden layers 1834 may be referred to as a deep neural network. The operation of training the deep neural network may be referred to as deep learning.

[0202] In one embodiment, if the neural network 1830 has a feed forward neural network structure, the first node included in a specific layer may be connected to all second nodes included in other layers before the specific layer. In the memory 1820, the parameters stored for the neural network 1830 may include a weight assigned to the connection between the second node and the first node. In the neural network 1830 having a feed forward neural network structure, the value of the first node may correspond to a weighted sum of values ​​assigned to the second node based on the weight assigned to the connection connecting the second node and the first node.

[0203] In one embodiment, if the neural network 1830 has a structure of a convolution neural network, the first node included in a specific layer may correspond to the weighted sum of some second nodes included in other layers before the specific layer. The part of the second nodes corresponding to the first node may be identified by the filter corresponding to the specific layer. In the memory 1820, the parameters stored for the neural network 1830 may include weights representing the filter. The filter may include one or more nodes in the second node for calculating the weighted sum of the first node, and weights corresponding to each of the one or more nodes.

[0204] According to one embodiment, the processor 1810 of the electronic device 101 may train the neural network 1830 using the learning data set 1840 stored in the memory 1820. Based on the learning data set 1840, the processor 1810 may perform the operation described with reference to FIG. 127 to adjust one or more parameters stored in the memory 1820 for the neural network 1830.

[0205] According to one embodiment, the processor 1810 of the electronic device 101 can perform object detection, object recognition and / or object classification using a neural network 1830 trained based on a learning data set 1840. The processor 1810 can input an image (or video) acquired through a camera 1850 into an input layer 1832 of the neural network 1830. Based on the input layer 1832 into which the image is input, the processor 1810 can sequentially acquire the values ​​of the nodes of each layer contained in the neural network 1830, thereby acquiring a set of values ​​of the nodes of the output layer 1836 (e.g., output data). The output data can be used as a result of reasoning about the information contained in the image using the neural network 1830. The embodiment is not limited thereto, and the processor 1810 can also acquire an image (or video) from an external electronic device connected to the electronic device 101 through a communication circuit 1860 and input it into the neural network 1830.

[0206] In one embodiment, the neural network 1830 trained to process an image can be used to identify an area in the image corresponding to an object (object detection), and / or identify a category of an object represented in the image (object recognition and / or object classification). For example, the electronic device 101 can use the neural network 1830 to segment the area in the image corresponding to the object based on a rectangular shape such as a bounding box. For example, the electronic device 101 can use the neural network 1830 to identify at least one category that matches the object from a plurality of specified categories.

[0207] As described above, an electronic device provided according to an embodiment may include a memory and a processor. The processor may use a template representing a license plate type to obtain a first image including a virtual license plate having a text object. The processor may, in a second image corresponding to a vehicle, obtain position information of the real license plate and color information of at least a portion of the real license plate based on identifying the real license plate attached to the vehicle. The processor may use the color information to change the color of at least one pixel included in the first image. The processor may combine the first image including at least one pixel whose color is changed with the second image based on the position information, thereby obtaining a third image of the vehicle synthesized with the virtual license plate. The processor may store a pair of the third image, the text object, and the position information representing the position in the third image to which the first image is combined as annotation information for training a neural network.

[0208] For example, the processor may project the virtual license plate contained in the first image onto the actual license plate attached to the vehicle using the position information representing the vertices of the actual license plate. The processor may train the neural network for recognizing other actual license plates different from the actual license plate in a fourth image different from the third image using the annotation information.

[0209] For example, the processor may acquire the color information by sampling pixels included in at least a portion of the actual license plate, wherein at least a portion of the actual license plate is different from other portions of the actual license plate that include other text objects different from the text object. The processor may use the color information to change the color of at least a portion of the virtual license plate, wherein at least a portion of the virtual license plate is different from other portions of the virtual license plate that include the text object of the virtual license plate. The processor may generate noise of the virtual license plate based on changing parameters of pixels included in the virtual license plate that represent at least a portion of which the color is changed.

[0210] For example, the processor can utilize a template including a background template representing the background of the virtual license plate and a text template representing the text object based on a specified font, and obtain the first image including the virtual license plate by combining at least one of the text objects including text objects representing numbers and text objects representing letters to a specified position of the background template.

[0211] For example, the processor may utilize the position information to project the virtual license plate onto the actual license plate in the second image by using a matrix that maps each vertex of the virtual license plate to each vertex of the actual license plate.

[0212] For example, the processor may use the color information representing the RGB value corresponding to the color of the actual license plate to change the color of at least one of the pixels of the first image based on changing the RGB values ​​corresponding to the color of the first image.

[0213] For example, the processor can identify the pixels corresponding to the actual license plate attached to the vehicle based on the pixels contained in the second image, and identify the position information of the actual license plate by using the area in the second image that contains the pixels corresponding to the actual license plate.

[0214] For example, the processor may use the template to combine the text objects having a specified size to obtain the first image of the virtual license plate including the combined text objects.

[0215] The method performed by the electronic device according to the embodiment described above may include the following operations. The method may use a template representing a license plate type to obtain a first image containing a virtual license plate having a text object. The method may, in a second image corresponding to a vehicle, obtain position information of the real license plate and color information of at least a portion of the real license plate based on identifying the real license plate attached to the vehicle. The method may use the color information to change the color of at least one pixel contained in the first image. The method may combine the first image containing at least one pixel whose color is changed with the second image based on the position information, thereby obtaining a third image of the vehicle in which the virtual license plate is synthesized. The method may store a pair of the third image, the text object, and the position information representing the position in the third image to which the first image is combined as annotation information for training a neural network.

[0216] For example, the method may project the virtual license plate contained in the first image onto the actual license plate attached to the vehicle using the position information representing the vertices of the actual license plate. The method may train the neural network for recognizing other actual license plates different from the actual license plate in a fourth image different from the third image using the annotation information.

[0217] For example, the method may acquire the color information by sampling pixels included in at least a portion of the actual license plate, wherein at least a portion of the actual license plate is different from other portions of the actual license plate that include other text objects different from the text object. The method may use the color information to change the color of at least a portion of the virtual license plate, wherein at least a portion of the virtual license plate is different from other portions of the virtual license plate that include the text object of the virtual license plate. The method may generate noise of the virtual license plate based on changing parameters of pixels included in the virtual license plate that represent at least a portion of which the color is changed.

[0218] For example, the method can utilize a template including a background template representing the background of the virtual license plate and a text template representing the text object based on a specified font, and obtain the first image including the virtual license plate by combining at least one of the text objects including text objects representing numbers and text objects representing letters to a specified position of the background template.

[0219] For example, the method may utilize the position information to project the virtual license plate onto the actual license plate in the second image by using a matrix that maps each vertex of the virtual license plate to each vertex of the actual license plate.

[0220] For example, the method may utilize the color information representing the RGB value corresponding to the color of the actual license plate to change the color of at least one of the pixels of the first image based on changing the RGB values ​​corresponding to the color of the first image.

[0221] For example, the method can be based on using the pixels contained in the second image to identify the pixels corresponding to the actual license plate attached to the vehicle, and using the area in the second image that contains the pixels corresponding to the actual license plate to identify the position information of the actual license plate.

[0222] According to one embodiment, a non-transitory computer-readable storage medium is provided, and the computer-readable storage medium is used to store one or more programs. The instructions contained in the one or more programs, when the one or more programs are executed by the processor of the electronic device, enable the electronic device to obtain a first image containing a virtual license plate (virtual license plate) with a text object using a template representing a license plate type. The instructions contained in the one or more programs, when the one or more programs are executed by the processor of the electronic device, enable the electronic device to obtain, in a second image corresponding to a vehicle, the position information of the actual license plate (real license plate) attached to the vehicle and the color information of at least a part of the actual license plate based on the recognition of the actual license plate. The instructions contained in the one or more programs, when the one or more programs are executed by the processor of the electronic device, enable the electronic device to change the color of at least one pixel contained in the first image using the color information. The instructions contained in the one or more programs, when the one or more programs are executed by the processor of the electronic device, enable the electronic device to combine the first image containing at least one pixel whose color is changed with the second image based on the position information, thereby obtaining a third image of the vehicle in which the virtual license plate is synthesized. The instructions contained in the one or more programs, when the one or more programs are executed by the processor of the electronic device, enable the electronic device to store a pair of position information of the third image, the text object and the position within the third image indicating the position where the first image is combined as annotation information for training a neural network.

[0223] For example, the instructions contained in the one or more programs, when executed by the processor of the electronic device, enable the electronic device to project the virtual license plate contained in the first image onto the actual license plate attached to the vehicle using the position information representing the vertices of the actual license plate. The instructions contained in the one or more programs, when executed by the processor of the electronic device, enable the electronic device to train the neural network for recognizing other actual license plates different from the actual license plate in a fourth image different from the third image using the annotation information.

[0224] For example, the instructions contained in the one or more programs, when executed by the processor of the electronic device, enable the electronic device to obtain the color information by sampling pixels contained in at least a portion of the actual license plate, wherein at least a portion of the actual license plate is different from other portions of the actual license plate containing other text objects different from the text object. The instructions contained in the one or more programs, when executed by the processor of the electronic device, enable the electronic device to use the color information to change the color of at least a portion of the virtual license plate, wherein at least a portion of the virtual license plate is different from other portions of the virtual license plate containing the text object of the virtual license plate. The instructions contained in the one or more programs, when executed by the processor of the electronic device, enable the electronic device to generate noise of the virtual license plate based on changing parameters of pixels contained in the virtual license plate used to indicate that at least a portion of the color is changed.

[0225] For example, the instructions contained in the one or more programs, when the one or more programs are executed by the processor of the electronic device, enable the electronic device to utilize a template including a background template representing the background of the virtual license plate and a text template representing the text object based on a specified font, to obtain the first image including the virtual license plate by combining at least one of the text objects including text objects representing numbers and text objects representing letters to a specified position of the background template.

[0226] For example, the instructions contained in the one or more programs, when the one or more programs are executed by the processor of the electronic device, enable the electronic device to utilize the position information to project the virtual license plate onto the actual license plate in the second image by using a matrix that maps the vertices of the virtual license plate to the vertices of the actual license plate.

[0227] The device described above can be implemented by hardware components, software components and / or a combination of hardware components and software components. For example, the device and components described in the embodiment can be implemented by a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor (digital signal processor), a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device that can execute and respond to instructions (instruction) and more than one general-purpose computer or a special-purpose computer. The processing device can execute an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device can access, store, operate, process and generate data in response to the execution of the software. For ease of understanding, although it is mentioned in the description that only one processing device is used, it will be understood by those of ordinary skill in the art that the processing device may include multiple processing elements (processing element) and / or multiple types of processing elements. For example, the processing device may include multiple processors or a processor and a controller. In addition, other processing configurations (processing configuration) such as parallel processors (parallelprocessor) can also be used.

[0228] Software may include a computer program, code, instructions, or any combination of these elements that can cause a processing device to operate as intended, or command a processing device individually or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device to be interpreted by or provide instructions or data to a processing device. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0229] The method according to the embodiment can be implemented in the form of program instructions that can be executed by various computer means, and recorded on a computer-readable medium. At this time, the medium can continuously store programs that can be executed on the computer, or temporarily store programs for execution or downloading. The medium can also be a recording or storage means in various forms of a single or multiple hardware combination, not limited to a medium directly connected to a computer system, but also a medium distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks and tapes, optical storage media such as CD-ROMs and DVDs, magneto-optical media such as optical floppy disks (Flopticaldisk), and means for storing program instructions such as ROM, RAM, and flash memory. In addition, examples of other media may also include recording media and even storage media managed by application stores that distribute application programs or other various websites, servers, etc. that supply or distribute software.

[0230] Although the embodiments are described above through limited examples and drawings, a person skilled in the art can make various modifications and variations based on the above description. For example, the described techniques can be performed in a different order from the described methods, and / or the described systems, structures, devices, circuits and other components can be combined or combined in a different form from the described methods, or replaced by other components or equivalents, and even so, appropriate results can be achieved.

[0231] Accordingly, other implementations, other embodiments, and their equivalents are also within the scope of the claims.

Claims

1. An electronic device, comprising: A memory, used to store instructions, and including one or more storage media; as well as at least one processor comprising processing circuitry; wherein the instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: Using a template representing a license plate type, obtaining a first image including a virtual license plate having a text object, In a second image corresponding to the vehicle, based on identifying an actual license plate attached to the vehicle, position information of the actual license plate and color information of at least a portion of the actual license plate are acquired, Using the color information, changing the color of at least one pixel included in the first image, Based on the position information, combining the first image including at least one pixel whose color is changed with the second image, thereby acquiring a third image of the vehicle in which the virtual license plate is synthesized, and The third image, the text object, and a position information pair representing a position in the third image where the first image is combined are stored as annotation information for training a neural network.

2. The electronic device according to claim 1, wherein: The instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: projecting the virtual license plate contained in the first image onto the actual license plate attached to the vehicle using the position information representing the vertices of the actual license plate, and The neural network for recognizing other actual license plates different from the actual license plates is trained in a fourth image different from the third image using the annotation information.

3. The electronic device according to claim 2, wherein: The instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: The color information is obtained by sampling pixels contained in at least a portion of the actual license plate, wherein the at least a portion of the actual license plate is different from other portions of the actual license plate that contain other text objects different from the text object, using the color information, changing the color of at least a portion of the virtual license plate, wherein the at least a portion of the virtual license plate is different from other portions of the virtual license plate including the text object of the virtual license plate, and Noise of the virtual license plate is generated based on changing a parameter of a pixel included in the virtual license plate indicating that at least a portion of the color is changed.

4. The electronic device as claimed in claim 3, characterized in that: The instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: Utilizing the template including a background template representing the background of the virtual license plate and a text template representing the text object based on a specified font, the first image including the virtual license plate is acquired by combining at least one of the text objects including a text object representing a number and a text object representing a letter to a specified position of the background template.

5. The electronic device as claimed in claim 2, characterized in that: The instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: Using the position information, the virtual license plate is projected onto the actual license plate in the second image by using a matrix that maps each vertex of the virtual license plate to each vertex of the actual license plate.

6. The electronic device as claimed in claim 1, wherein: The instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: The color of at least one of the pixels of the first image is changed based on the RGB values ​​corresponding to the color of the first image being changed using the color information representing the RGB value corresponding to the color of the actual license plate.

7. The electronic device as claimed in claim 1, characterized in that: The instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: Based on using the pixels included in the second image to identify the pixels corresponding to the actual license plate attached to the vehicle, and using the area in the second image containing the pixels corresponding to the actual license plate, the position information of the actual license plate is identified.

8. The electronic device as claimed in claim 1, wherein: The instructions, when executed individually or collectively by the at least one processor, cause the electronic device to: The template is used to combine the text objects having a specified size to obtain the first image of the virtual license plate including the combined text objects.

9. A method performed by an electronic device, comprising the following operations: Using a template representing a license plate type, obtaining a first image including a virtual license plate having a text object; In a second image corresponding to the vehicle, based on identifying an actual license plate attached to the vehicle, obtaining position information of the actual license plate and color information of at least a portion of the actual license plate; Using the color information, changing the color of at least one pixel included in the first image; Based on the position information, combining the first image including at least one pixel whose color is changed with the second image, thereby acquiring a third image of the vehicle synthesized with the virtual license plate; as well as The third image, the text object, and a position information pair representing a position in the third image where the first image is combined are stored as annotation information for training a neural network.

10. The method executed by an electronic device according to claim 9, wherein: The following operations are included: projecting the virtual license plate contained in the first image onto the actual license plate attached to the vehicle using the position information representing vertices of the actual license plate; as well as The neural network for recognizing other actual license plates different from the actual license plates is trained in a fourth image different from the third image using the annotation information.

11. The method executed by an electronic device according to claim 10, wherein: The following operations are included: Acquiring the color information by sampling pixels contained in at least a portion of the actual license plate, wherein the at least a portion of the actual license plate is different from other portions of the actual license plate that contain other text objects different from the text object; Using the color information, changing the color of at least a portion of the virtual license plate, wherein the at least a portion of the virtual license plate is different from other portions of the virtual license plate including the text object of the virtual license plate; and Noise of the virtual license plate is generated based on changing a parameter of a pixel included in the virtual license plate indicating that at least a portion of the color is changed.

12. The method executed by an electronic device as claimed in claim 11, characterized in that: The following operations are included: Utilizing the template including a background template representing the background of the virtual license plate and a text template representing the text object based on a specified font, the first image including the virtual license plate is acquired by combining at least one of the text objects including a text object representing a number and a text object representing a letter to a specified position of the background template.

13. The method executed by an electronic device as claimed in claim 10, characterized in that: The following operations are included: Using the position information, the virtual license plate is projected onto the actual license plate in the second image by using a matrix that maps each vertex of the virtual license plate to each vertex of the actual license plate.

14. The method executed by an electronic device as claimed in claim 9, characterized in that: The following operations are included: The color of at least one of the pixels of the first image is changed based on the RGB values ​​corresponding to the color of the first image being changed using the color information representing the RGB value corresponding to the color of the actual license plate.

15. The method executed by an electronic device as claimed in claim 9, characterized in that: The following operations are included: Based on using the pixels included in the second image to identify the pixels corresponding to the actual license plate attached to the vehicle, and using the area in the second image containing the pixels corresponding to the actual license plate, the position information of the actual license plate is identified.

16. A non-transitory computer-readable storage medium for storing one or more programs, The instructions contained in the one or more programs, when the one or more programs are executed individually or collectively by at least one processor of the electronic device, cause the electronic device to: Using a template representing a license plate type, obtaining a first image including a virtual license plate having a text object, In a second image corresponding to the vehicle, based on identifying an actual license plate attached to the vehicle, position information of the actual license plate and color information of at least a portion of the actual license plate are acquired, Using the color information, changing the color of at least one pixel included in the first image, Based on the position information, combining the first image including at least one pixel whose color is changed with the second image, thereby acquiring a third image of the vehicle in which the virtual license plate is synthesized, and The third image, the text object, and a position information pair representing a position in the third image where the first image is combined are stored as annotation information for training a neural network.

17. The non-transitory computer-readable storage medium of claim 16, wherein: The instructions contained in the one or more programs, when the one or more programs are executed individually or collectively by the at least one processor, cause the electronic device to: projecting the virtual license plate contained in the first image onto the actual license plate attached to the vehicle using the position information representing the vertices of the actual license plate, and The neural network for recognizing other actual license plates different from the actual license plates is trained in a fourth image different from the third image using the annotation information.

18. The non-transitory computer-readable storage medium of claim 17, wherein: The instructions contained in the one or more programs, when the one or more programs are executed individually or collectively by the at least one processor, cause the electronic device to: The color information is obtained by sampling pixels contained in at least a portion of the actual license plate, wherein the at least a portion of the actual license plate is different from other portions of the actual license plate that contain other text objects different from the text object, using the color information, changing the color of at least a portion of the virtual license plate, wherein the at least a portion of the virtual license plate is different from other portions of the virtual license plate including the text object of the virtual license plate, and Noise of the virtual license plate is generated based on changing a parameter of a pixel included in the virtual license plate indicating that at least a portion of the color is changed.

19. The non-transitory computer-readable storage medium of claim 18, wherein: The instructions contained in the one or more programs, when the one or more programs are executed individually or collectively by the at least one processor, cause the electronic device to: Utilizing the template including a background template representing the background of the virtual license plate and a text template representing the text object based on a specified font, the first image including the virtual license plate is acquired by combining at least one of the text objects including a text object representing a number and a text object representing a letter to a specified position of the background template.

20. The non-transitory computer-readable storage medium of claim 17, wherein: The instructions contained in the one or more programs, when the one or more programs are executed individually or collectively by the at least one processor, cause the electronic device to: Using the position information, the virtual license plate is projected onto the actual license plate in the second image by using a matrix that maps each vertex of the virtual license plate to each vertex of the actual license plate.

Images