Electronic device, method, and non-transitory computer readable storage medium for acquiring labeling information for training of neural network

Abstract

To provide an electronic device for acquiring labeling information for training of a neural network.SOLUTION: According to an embodiment, an electronic device may be configured to include a memory and a processor. The processor may be configured to acquire a first image including a virtual license plate including text objects, using a template representing a kind of license plates. The processor may be configured to, on the basis of identifying a real license plate attached to a vehicle in a second image corresponding to the vehicle, acquire position information on the real license plate, and color information on at least a part of the real license plate. The processor may be configured to change at least one color of pixels included in the first image, using the color information,.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an electronic device, a method, and a non-transitory computer-readable storage medium for obtaining labeling information for training a neural network. [Background technology]

[0002] As image object recognition technology develops, various forms of services are emerging. These services can be used for Automatic Driving, Augmented Reality, Virtual Reality, Metaverse, etc., and can be provided through electronic devices owned by different users, such as smartphones. The services can be related to hardware and / or software mechanisms that mimic human behavior and / or thinking, such as artificial intelligence. Technologies related to artificial intelligence can include technologies that utilize neural networks that simulate the neural networks of living organisms. Summary of the Invention [Problem to be solved by the invention]

[0003] What is needed is a method for identifying license plates attached to vehicles using neural networks.

[0004] The technical problems to be achieved in this specification are not limited to the above-mentioned technical problems, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the following description. [Means for solving the problem]

[0005] According to an embodiment, the electronic device may be configured to include a memory and a processor. The processor may be configured to obtain a first image including a virtual license plate including a text object using a template representing a type of license plate. The processor may be configured to obtain position information of the real license plate and color information of at least a part of the real license plate based on identifying a real license plate attached to the vehicle in a second image corresponding to the vehicle. The processor may be configured to change the color of at least one pixel included in the first image using the color information. The processor may be configured to combine the first image and the second image including at least one pixel with the at least one color changed based on the position information to obtain a third image including the vehicle with the virtual license plate synthesized. The processor may be configured to store a pair of the third image, the text object, and position information indicating a position in the third image where the first image is combined as labeling information for training a neural network.

[0006] In a method performed by an electronic device according to an embodiment, the method can include an operation of acquiring a first image including a virtual license plate including a text object using a template representing a type of license plate. The method can include an operation of acquiring position information of the real license plate and color information of at least a portion of the real license plate based on identifying a real license plate attached to the vehicle in a second image corresponding to the vehicle. The method can include an operation of changing a color of at least one pixel included in the first image using the color information. The method can include an operation of combining the first image including the at least one pixel with the at least one color changed and the second image based on the position information to acquire a third image including the vehicle with the virtual license plate combined. The method can include an operation of storing a pair of the third image, the text object, and position information indicating a position in the third image to which the first image is combined as labeling information for training a neural network.

[0007] In a computer-readable storage medium having one or more programs stored thereon according to an embodiment, the one or more programs may be configured to include instructions, when executed by a processor of an electronic device, to cause the electronic device to obtain a first image including a virtual license plate including a text object using a template representing a type of license plate. The one or more programs may be configured to include instructions, when executed by a processor of an electronic device, to cause the electronic device to obtain position information of the real license plate and color information of at least a portion of the real license plate based on identifying a real license plate attached to the vehicle in a second image corresponding to the vehicle. The one or more programs may be configured to include instructions, when executed by a processor of an electronic device, to cause the electronic device to use the color information to change a color of at least one of the pixels included in the first image. The one or more programs may be configured to include instructions, when executed by a processor of an electronic device, to cause the electronic device to combine the first image including at least one pixel with the at least one color changed based on the position information and the second image to obtain a third image including the vehicle with the virtual license plate combined. The one or more programs may be configured to include instructions that, when executed by a processor of an electronic device, cause the electronic device to store pairs of the third image, the text object, and position information indicating a position within the third image to which the first image is combined, as labeling information for training a neural network. Effect of the Invention

[0008] An electronic device, a method, and a non-transitory computer-readable storage medium according to an embodiment of the present disclosure can obtain labeling information for identifying a license plate attached to a vehicle using a neural network.

[0009] The effects obtained by the present disclosure are not limited to the effects described above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains from the following description. [Brief description of the drawings]

[0010] [Figure 1] 1 illustrates an example block diagram of an electronic device according to one embodiment. [Diagram 2] 1 illustrates an example of a template of a virtual license plate obtained by an electronic device according to an embodiment. [Diagram 3] 1 illustrates an example of data regarding a virtual license plate captured by an electronic device according to one embodiment. [Figure 4] 1 illustrates an example of a virtual license plate captured by an electronic device according to one embodiment. [Diagram 5] 1 illustrates an example of a neural network used by an electronic device to obtain a virtual license plate according to one embodiment. [Figure 6] 1 illustrates an example of a virtual license plate obtained by an electronic device using a neural network according to an embodiment. [Figure 7] 1 illustrates an example of a table showing license plates for identification by an electronic device according to one embodiment. [Figure 8a] 1 illustrates an example table showing license plate types for identification by an electronic device according to one embodiment. [Figure 8b] 1 illustrates an example table showing license plate types for identification by an electronic device according to one embodiment. [Figure 9] 1 illustrates an example of an image used by an electronic device to obtain location information according to one embodiment. [Figure 10]1 illustrates an example of an operation of an electronic device to obtain location information of a real license plate attached to a vehicle according to an embodiment. [Figure 11] 1 illustrates an example of an operation of an electronic device using color information of a real license plate to change the color of a virtual license plate according to one embodiment. [Figure 12] 1 illustrates an example of an electronic device modifying the color of a virtual license plate using a histogram representing the color of a real license plate, according to one embodiment. [Figure 13] 1 illustrates an example of a virtual license plate based on noise acquired by an electronic device according to one embodiment. [Figure 14] 1 illustrates an example of an operation for an electronic device to project a first image onto a second image according to an embodiment. [Figure 15] 1 illustrates an example of an operation of an electronic device according to an embodiment, in which a third image is obtained by combining a first image including a virtual license plate with a second image and storing the third image as labeling information. [Figure 16] 1 shows an example of a flowchart illustrating the operation of an electronic device according to one embodiment. [Figure 17] FIG. 1 illustrates the operation of an electronic device for training a neural network based on a training data set. [Figure 18] FIG. 1 is a block diagram of an electronic device, according to one embodiment. [Figure 19] 1 illustrates an example of position information corresponding to a polygonal shape according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Various embodiments of the present document will now be described with reference to the accompanying drawings.

[0012] It should be understood that the various embodiments and the terms used therein are not intended to limit the technology described in this document to a particular embodiment, but include various modifications, equivalents, or alternatives of the embodiment. With respect to the description of the drawings, similar components may use similar reference numerals. Unless the context clearly dictates otherwise, singular expressions may include plural expressions. In this document, expressions such as "A or B," "at least one of A and / or B," "A, B or C," or "at least one of A, B, and C" may include all possible combinations of the items listed together. Expressions such as "first," "second," or "first" or "second" may modify the corresponding components regardless of order or importance, and are used only to distinguish a certain component from other components, not to limit the corresponding components. When a (e.g., first) component is referred to as being "coupled (functionally or communicatively)" or "connected" to another (e.g., second) component, the component may be directly connected to the other component or may be connected through another component (e.g., a third component).

[0013] The term "module" as used in this document includes a unit configured in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integrated component or the smallest unit or portion of a component that performs one or more functions. For example, a module may be configured in an application-specific integrated circuit (ASIC).

[0014] FIG. 1 illustrates an example of a block diagram of an electronic device according to an embodiment. Referring to FIG. 1, an electronic device 101 according to an embodiment may include at least one of a processor 120 or a memory 130. The processor 120 and the memory 130 may be electrically and / or operably coupled with each other by an electronic component such as a communication bus. Hereinafter, the term "operably coupled" may mean that a direct connection between the hardware or an indirect connection between the hardware is established by wire or wirelessly such that a first piece of hardware controls a second piece of hardware. Although the components of the electronic device 101 are illustrated in different blocks, the embodiment is not limited thereto. Some of the hardware in FIG. 1 may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of hardware included in the electronic device 101 is not limited to that illustrated in FIG. 1. For example, the electronic device 101 may include only a part of the hardware illustrated in FIG. 1.

[0015] According to an embodiment, the electronic device 101 may include hardware for processing data based on one or more instructions. The hardware for processing data may include a processor 120. The hardware for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). The processor 120 may have a single-core processor structure, or may have a multi-core processor structure, such as a dual core, a quad core, a hexa core, or an octa core.

[0016] According to one embodiment, the memory 130 of the electronic device 101 may include hardware components for storing data and / or instructions input and / or output to the processor 120 of the electronic device 101. The memory 130 may include, for example, a volatile memory, such as a random-access memory (RAM), and / or a non-volatile memory, such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM (Cache RAM), and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disk, a solid state drive (SSD), and an embedded multimedia card (eMMC).

[0017] For example, the memory 130 of the electronic device 101 may include a neural network (not shown). The electronic device 101 may identify a license plate attached to a vehicle based on the neural network stored in the memory 130. For example, the electronic device 101 may train the neural network based on an image (or the labeling information 165) stored in the memory 130. The electronic device 101 may train the neural network based on a virtual license plate included in the image. The operation of the electronic device 101 to generate a virtual license plate for training the neural network will be described later with reference to FIGS. 2 to 6.

[0018] The memory 130 of the electronic device 101 according to an embodiment may include a template 150. The template 150 may include information regarding the type of license plate attached to the vehicle. The template 150 may take into account country-specific license plate rules (e.g., "Notice on Standards for Motor Vehicle Registration Plates, etc."). In one example, the type of template 150 may differ depending on the country-specific license plate rules. The electronic device 101 may identify different templates 150 for the country-specific license plate rules. For example, the template 150 may include a background template 151 representing the background of the license plate and / or a text template 153 representing a text object (e.g., a number or a letter) of the license plate. An exemplary description of the template 150 is described below with reference to FIG. 2.

[0019] For example, the memory 130 of the electronic device 101 may include a first image 160. The first image 160 may include a virtual license plate acquired by the electronic device 101 using the template 150. The size of the first image 160 may correspond to the size of the virtual license plate. The operation of the electronic device 101 acquiring the first image 160 will be described later with reference to FIGS. 2 to 4.

[0020] For example, the memory 130 of the electronic device 101 may include a second image 161. The second image 161 may be an example of an image corresponding to a real environment including a vehicle. The second image 161 may include a real license plate attached to the vehicle. The electronic device 101 may use the second image 161 to obtain position information and color information of the real license plate. The electronic device 101 may use the color information to change the color of the first image 160 and use the position information to combine the first image 160 with at least a portion of the second image 161.

[0021] For example, the memory 130 of the electronic device 101 may include a third image 162. The third image 162 may include a vehicle with a virtual license plate synthesized by combining the second image 161 and the first image 160. For example, the electronic device 101 may store in the memory 130 the third image 162, the text object, and position information indicating a position in the third image where the first image 160 is combined as labeling information 165. The position information may represent a position (e.g., coordinate value) where the first image 160 is superimposed in the third image. The labeling information 165 may be used to train a neural network to identify a license plate attached to a vehicle in an image.

[0022] As described above, the electronic device 101 according to an embodiment can obtain a virtual license plate for training a neural network to identify license plates. Based on using the virtual license plate to train a neural network, the electronic device 101 may not infringe on user information corresponding to a real license plate, unlike using a real license plate of a vehicle included in an image corresponding to a real environment.

[0023] An exemplary illustration of a template 150 used to obtain a virtual license plate is provided below with reference to FIG.

[0024] 2 shows an example of a template of a virtual license plate obtained by an electronic device according to an embodiment. The electronic device 101 of FIG. 2 may include the electronic device 101 of FIG. 1. At least one of the operations of FIG. 2 may be performed by the processor 120 of FIG. 1.

[0025] 2, an example of a template 150 based on country-specific license plate rules is shown. The template 150 may vary by region, country, and / or time. The template 150 may include a background template 151 that shows the background of the license plate and / or a text template 153 that shows a text object contained within the license plate.

[0026] For example, the background template 151 may have a specified size. The specified size may vary depending on a country or region. The specified size may vary depending on a vehicle use and / or a vehicle type. The background template 151 may include visual objects, figures, and / or colors representing a region or country. The color of the background template 151 may vary depending on a vehicle type and / or a vehicle use. As an example, when generating a virtual license plate corresponding to a first country (e.g., Korea), the electronic device 101 may generate the virtual license plate using a first background template 151-1. As an example, when generating a license plate corresponding to a second country (e.g., the United States), the electronic device 101 may generate the virtual license plate using a second background template 151-2. The electronic device 101 may select the background template 151 based on a country or region corresponding to the virtual license plate to be generated.

[0027] For example, the text template 153 may include text objects included in a license plate. The text template 153 may include text objects 211 representing letters and / or text objects 212 representing numbers. The text objects included in the text template 153 may have a designated size to be included in the background template 151. The designated size may be set based on license plate rules for each country. However, this is not limited thereto. The text objects included in the text template 153 may include characters based on a language (e.g., Kanji, Hangul, the alphabet, and / or Hiragana). The text objects may be identified based on a designated font. The language and / or designated font of the text object may vary by region, country, and / or time. As an example, the text objects may include text objects based on a first font (e.g., penitentiary gothic) in the United States and text objects based on a second font (e.g., rage italic) in the State of California in the United States. However, this is not limited thereto. If country-specific license plate rules change, the background template 151 and / or the text template 153 may be changed.

[0028] In one embodiment, the electronic device 101 can select at least one of the background templates 151 and combine at least one of the text objects included in the text template 153 into the selected at least one background template to obtain a virtual license plate.

[0029] For example, the electronic device 101 may combine each of the text objects 211 representing letters and / or the text objects 212 representing numbers to obtain a virtual license plate. The number of text objects (e.g., the text objects 211 representing letters and / or the text objects 212 representing numbers) included in the virtual license plate may vary by region, country, and / or time. For example, the electronic device 101 may select at least one of the text objects 211 representing letters based on the number of occurrences and at least one of the text objects 212 representing numbers randomly. However, this is not limited thereto. For example, the electronic device 101 may combine the text objects in the text template into the background template based on the specified settings (e.g., size, position, and / or number) according to the license plate rules for each country to obtain a virtual license plate.

[0030] A description of exemplary data used by the electronic device 101 to obtain a virtual license plate according to one embodiment is provided below with reference to FIG.

[0031] 3 shows an example of data of a virtual license plate acquired by an electronic device according to an embodiment. The electronic device 101 in FIG. 3 may include the electronic device 101 in FIG. 1 to FIG. 2. At least one of the operations in FIG. 3 may be executed by the processor 120 in FIG. 1.

[0032] Referring to FIG. 3, the electronic device 101 according to an embodiment can obtain a virtual license plate 305 using the template 150 of FIG. 1. The virtual license plate 305 can be obtained based on the license plate rules of each country. For example, the electronic device 101 can set the size, color, and / or inclusion or non-inclusion of a visual object of a background template (e.g., the background template 151 of FIG. 1). The electronic device 101 can combine the set background template with a text object included in a text template (e.g., the text template 153 of FIG. 1) to obtain a virtual license plate 305. The electronic device 101 can set the position, size, and / or type (e.g., letters or numbers) of the text object included in the background template to obtain the virtual license plate 305. The electronic device 101 can set the type of the virtual license plate 305 according to the type of license plate to be identified using the neural network to be trained. The type of license plate may be set based on the location (e.g., region or country) of the real environment corresponding to the image including the license plate.

[0033] For example, the electronic device 101 can identify data 310 for obtaining the virtual license plate 305. The data 310 can include information about the virtual license plate 305. For example, the information about the virtual license plate 305 can include information about a type of template, a color of the background template, a shape of the background template (e.g., rectangular), a location (e.g., a region or country) corresponding to the template, a size of the virtual license plate 305, and / or a text object (e.g., "58do0290") included in the virtual license plate 305. The data 310 can be included in labeling information that the electronic device 101 uses to train a neural network to identify license plates. For example, the electronic device 101 can train a neural network using data (e.g., data 310) including an image including a real license plate and information about the real license plate.

[0034] For example, the data 310 may include information indicating the size of the virtual license plate 305. The line information 311 may represent the center coordinates (e.g., Min X, Min Y) of the two-dimensional coordinate axis. The center coordinates of the two-dimensional coordinate axis may mean coordinates that are references for identifying one position of the virtual license plate 305. For example, the line information 311 may include information indicating one position (e.g., "points" in FIG. 3) of the virtual license plate 305 based on the shape (e.g., "shapes" in FIG. 3) of the virtual license plate 305. For example, the line information 311 may indicate coordinates (e.g., 1240.1408450704225, 1042.535211676057) of the position information 306 relative to the left-top edge of the virtual license plate 305 based on the two-dimensional coordinate axis. The line information 312 may represent relative coordinates (e.g., Max X, Max Y) relative to the center coordinates. For example, the line information 312 may represent the coordinates (e.g., 1493.661971830986, 1115.4225352112676) of the position information 307 relative to the right-bottom edge of the virtual license plate 305 based on the two-dimensional coordinate axis. However, the present invention is not limited to this. The electronic device 101 may set the size of the virtual license plate using the coordinate values ​​of the position information 306 and the position information 307. The size of the virtual license plate may be changed according to the type of the virtual license plate 305.

[0035] For example, the data 310 may include information about the vehicle on which the virtual license plate 305 is synthesized. For example, the line information 313 indicating information about the vehicle may include the type of the vehicle, the use of the vehicle, and / or the registered address of the vehicle. The line information 313 may be acquired based on a JSON format. The electronic device 101 may identify at least one of the information about the vehicle included in the line information 313 according to the type of the acquired virtual license plate 305. For example, in the case of a first type (e.g., “type01”), the shape of the license plate may have a rectangular shape with a horizontal length longer than a vertical length. In one example, the color of the text object included in the license plate may be a first color (e.g., black). In the case of a second type (e.g., “type02”), the shape of the license plate may have a rectangular shape with a horizontal length shorter than the horizontal length of the first type rectangle and a vertical length longer than the vertical length of the first type rectangle. In one example, the color of the text object included in the license plate may be a first color (e.g., black). In the case of the third type (e.g., "type03"), the form of the license plate may be similar to that of the first type license plate. In the case of the third type, the color of the license plate may be distinguished from that of the first type license plate. Since the use of a vehicle having a license plate corresponding to the first type may be different from that of a vehicle having a license plate corresponding to the third type, the colors of the first type license plate and the third type license plate may be different. The color of the text object included in the third type license plate may be a second color (e.g., white). In the case of the fourth type ("type04"), the form of the license plate may be similar to that of the second type license plate. The color of the text object included in the fourth type license plate may be a second color. However, it is not limited thereto. The type of license plate may vary depending on the region, country, and / or vehicle use.By identifying the type of license plate, the electronic device 101 can obtain information corresponding to the type from the identified license plate. For example, when information on a first type (e.g., "type01") is selected (e.g., true) and information on other types (e.g., "type02" to "typeETC") is not selected (e.g., false), the electronic device 101 can obtain a virtual license plate 305 corresponding to the first type. The JSON format will be described later with reference to Table 1 below. [Table 1] Table 1, according to one embodiment, may include a portion of information indicative of a virtual license plate captured via an image by electronic device 101. For example, electronic device 101 may train a neural network using the portion of information and a virtual license plate (or license plate image) corresponding to the information.

[0036] For example, referring to Table 1, the information following "flags" may mean including data representing at least one license plate included in the image (e.g., class information). The information corresponding to "flags" may differ depending on the part (or category) separated by "flags". "flags" may be used to distinguish the information corresponding to "flags".

[0037] For example, the "label" may represent a text object and / or a visual object for the electronic device 101 to identify through the neural network. In one example, the "label" may mean data representing a correct answer. The "label" may be referred to as guide information for the electronic device 101 to train the neural network. As an example, when multiple license plates are identified in an image, a data set (e.g., label-to-flags) (or instructions) corresponding to each of the multiple license plates may be obtained. In one example, the data set below the "label" may be used to identify a text object corresponding to the "label". For example, the data set may be obtained based on position information included in the license plate (e.g., the first position 411 in FIG. 4 described later). The position information may differ depending on the form of the license plate. The description of the form of the license plate will be described later with reference to FIG. 4. Since the position including the text object and / or visual object may differ depending on the form of the license plate, the electronic device 101 may infer the position information including the text object by identifying the form (or type) of the license plate. As an example, the number of text objects for the electronic device 101 to set may differ based on the type of virtual license plate to be generated. Based on the number of text objects, you can get a dataset based on the JSON format.

[0038] For example, the "point" may represent the position of the top left edge of a rectangle (e.g., 1513.0806104480532, 923088542783918) and / or the position of the bottom right edge of a rectangle (e.g., 1653.8700841322636, 959.930480470897) based on the shape of the virtual license plate 305 (or a bounding box for identifying one position of the virtual license plate). In one example, the "point" may change depending on the shape of the virtual license plate 305 (or a bounding box). As an example, in the case of a shape based on an n-gon (or polygon), the "point" may include n pieces of position information. For example, the position information related to the shape based on an n-gon (or polygon) may be shown as in FIG. 19. 19, the electronic device 101 may identify position information corresponding to edges 1910-1, 1910-2, 1910-3, 1910-4, 1910-5, and 1910-6 of a polygon 1910, but is not limited to such.

[0039] For example, line information 311 and / or line information 312 may be an example of "point". For example, "shape type" may represent the shape of a license plate corresponding to the type of license plate. As an example, referring to Table 1, among the information indicating the type of license plate, a first type (e.g., "type01") is true and the rest are false, so that the information included in Table 1 may include a shape (e.g., rectangle) corresponding to the first type and position information based on the first type (e.g., line information 311 and / or line information 312). As an example, referring to Table 1, "truck" and / or "passenger car" may be set based on the type of vehicle.

[0040] For example, "NUMBERS" can be set to true if the color of a text object included in the virtual license plate is a first color (e.g., black), and can be set to false if the color is a second color (e.g., a color different from black). "num_n" can be used to identify a text object (e.g., a number) included in one position of the virtual license plate (e.g., position information described later in FIG. 4). As an example, if a text object (e.g., a number) included in one position of the virtual license plate is 5, "num_5" can be set to true. "INVERSE_NUMBERS" can be set to true if the color of a text object included in the virtual license plate is a second color (e.g., white), and can be set to false if the color is a color different from the second color (e.g., a color different from white). "num_inverse_n" can be used to identify a text object (e.g., a number) included in one position of the virtual license plate (e.g., position information described later in FIG. 4). As an example, if a text object (e.g., a number) included in one position of the virtual license plate is 5, "num_inverse_5" can be set to true. However, this is not limited thereto. Information for identifying text objects (for example, numbers, symbols, or characters) in the license plate will be described below with reference to Tables 2 to 5. [Table 2] Table 2 may include information regarding text objects (e.g., characters) included in a license plate according to one embodiment. With reference to Table 2, an electronic device 101 according to one embodiment may identify text objects included in a position on a license plate. For example, "CHARACTERS" may be set to true if the color of the text object included in the license plate is a first color (e.g., black), and may be set to false if the color of the text object included in the license plate is a second color (e.g., a color different from black). "char_ga" through "char_bae" may represent variables indicating information about characters included in a position on a license plate.

[0041] For example, the variables may vary depending on the type of license plate, the region (or country) in which the license plate is used, and the language of the characters contained within the license plate. In one example, the variables may include Hangul, English, Kanji, and / or Hiragana words. In one example, the variables may include one or more characters. As an example, if the electronic device 101 identifies a text object corresponding to at least one of the variables, it may set the variable corresponding to the text object to true. The operation of the electronic device 101 to obtain a data set in which the variable corresponding to the text object is set to true will be described below with reference to Tables 4 to 5. [Table 3] Table 3 may include information regarding text objects (e.g., characters) included in the license plate according to an embodiment. For example, "INVERSE_CHARACTERS" may be set to true if the color of the text object included in the license plate is a first color (e.g., white) and may be set to false if the color of the text object included in the license plate is a second color (e.g., a color different from white). "char_inverse_ga" to "char_inverse_bae" may represent variables indicating information of characters included in one position of the license plate (e.g., "ga" or "bae"). For example, the variables may vary depending on the type of license plate, the region (or country) in which the license plate is used, and the language of the characters included in the license plate. In one example, the variables may include Hangul, English, Hanji, and / or Hiragana words. In one example, the variables may include one or more characters. As an example, when the electronic device 101 identifies a text object corresponding to at least one of the variables, the electronic device 101 may set the variable corresponding to the text object to true. However, the present invention is not limited thereto. For example, when the electronic device 101 obtains a data set based on the JSON format at the location 305-1, the data set may be shown as in Table 4. [Table 4] Referring to Table 4, Table 4 may include information regarding location 305-1 according to one embodiment. For example, "point" may include information regarding location information 306-1 (e.g., 1526.8438538205978, 929.5681063122923) and / or location information 307-1 (e.g., 1540.1328903654485, 953.156146179402) of location 305-1. For example, referring to Table 4, location 305-1 is an area in which a text object is set, so information indicating the type of license plate (e.g., "car", "truck", and / or "type01" to "typeETC") may be set to false. For example, referring to Table 4, location 305-1 includes a text object representing a number (e.g., 5), so a variable corresponding to this number (e.g., "num_5") may be set to true. The variables that are distinct from the variables corresponding to the numbers may be set to false. For example, referring to Table 4, the variables corresponding to the color of the text object (e.g., "NUMBERS") may be set to true, but are not limited to this. For example, when the electronic device 101 obtains a data set based on the JSON format at the position 305-2, the data set at the position 305-2 may be shown as in Table 5 described below. [Table 5] Referring to Table 5 according to one embodiment, "point" may represent a relative coordinate for the position information 306 of the virtual license plate 305 based on a shape (e.g., a rectangle) for distinguishing the position 305-2. "point" may represent a coordinate value representing the position information 306-2 and the position information 307-2. "point" may represent the position coordinates of the top left edge of the position (e.g., 1557.0764119601329, 930.5647840531561) and the position coordinates of the bottom right edge.

[0042] (e.g., 1572.0265780730897, 953.4883720930231), but is not limited thereto. For example, location 305-2 includes a text object representing a character (e.g., do), so the variable "num_n" (or "num_inverse_n") can be set to false. For example, location 305-2 includes a text object representing a character (e.g., do), so the variable for that character (e.g., "char_do") can be set to true and the other variables can be set to false. With reference to Table 5, the variables representing the type of license plate (e.g., "car", "truck", "type01" through "typeETC") can be set to false, but is not limited thereto. For example, electronic device 101 can use an image including an actual license plate to acquire data 310, and use the image and data 310 to train a neural network to acquire a virtual license plate. For example, the electronic device 101 can obtain the virtual license plate 305 by placing textual and / or visual objects corresponding to the type of virtual license plate at specified locations on the virtual license plate via a trained neural network. In this specification, the text object to be placed at a specified position on the virtual license plate includes only characters and numbers as an embodiment, but this is only one embodiment and the embodiment of the present invention is not limited thereto. For example, the text that can be placed at a specified position on the virtual license plate according to the present invention can include text generated based on a writing system for recording letters used around the world, such as ASCII (American Standard Code for Information Interchange) Code and / or Unicode. The text that can be placed at a specified position on the virtual license plate can include text having a standard defined for exchanging information via a computer.

[0043] An example of a virtual license plate acquired by the electronic device 101 according to an embodiment will be described below with reference to FIG.

[0044] FIG. 4 illustrates an example of a virtual license plate acquired by an electronic device according to an embodiment. The electronic device 101 of FIG. 4 may include the electronic device 101 of FIG. 1 to FIG. 3. At least one of the operations of FIG. 4 may be performed by the processor 120 of FIG. 1. Referring to FIG. 4, an example of a virtual license plate 401 acquired by the electronic device 101 according to an embodiment is illustrated.

[0045] For example, the electronic device 101 can obtain a virtual license plate 401-1 using the data 310 of Figure 3. The electronic device 101 can obtain the virtual license plate 401-1 based on the license plate rules corresponding to a first country (e.g., South Korea).

[0046] For example, the virtual license plate 401-1 can be obtained based on a specified color (e.g., white). The text objects included in the virtual license plate 401-1 can include different information depending on the location where they are placed on the virtual license plate 401-1. Depending on the number and / or location of the text objects included in the virtual license plate 401-1, the text objects can include time information (e.g., the revision year of the license plate rules). Tables showing text objects that include different information depending on the location where they are placed will be described below with reference to Figures 7 to 8b.

[0047] For example, a text object (e.g., 12) representing a number arranged at the first position 411 may indicate a type of vehicle (e.g., passenger car). For example, a text object (e.g., ga) representing a number arranged at the second position 412 may indicate a use of the vehicle. For example, a text object (e.g., ga) representing a number arranged at the third position 413 may indicate a serial number for identifying a vehicle. However, this is not limited to this. The position of the upper left edge and / or the position of the lower right edge of the first position 411 may be represented based on the "point" in FIG. 3. The number of text objects included in the first position 411 may be one or more. Hereinafter, overlapping descriptions in the description of the positions will be omitted.

[0048] According to an embodiment, the electronic device 101 can obtain the virtual license plate 401-2 using data corresponding to the virtual license plate 401-2 that is distinct from the virtual license plate 401-1. The virtual license plate 401-2 can include information about the region.

[0049] For example, a text object representing letters placed at a first position 421 may represent a region name (e.g., Seoul). A text object representing numbers placed at a second position 422 (e.g., 52) may represent a type of vehicle. A text object representing letters placed at a third position 423 (e.g., ba) may represent the use of the vehicle. A text object representing numbers placed at a fourth position 424 (e.g., 3108) may represent a serial number for identifying the vehicle to which the license plate is attached. However, the present invention is not limited to this.

[0050] For example, the electronic device 101 may process the virtual license plate 401-1 or the virtual license plate 401-2 based on grayscaling based on acquiring the virtual license plate 401-1 or the virtual license plate 401-2.

[0051] For example, the electronic device 101 may obtain a virtual license plate that includes a visual object (e.g., a Taegeuk symbol) representing a first country (e.g., South Korea) and / or a text object (e.g., KOR) representing the first country. The colors of different regions of the virtual license plate that include the visual object and / or the text object may be different.

[0052] According to an embodiment, the electronic device 101 can obtain the virtual license plate 401-3 according to the license plate rules corresponding to the second country (e.g., China). For example, the electronic device 101 can generate the virtual license plate 401-3 using data corresponding to the virtual license plate 401-3.

[0053] For example, the virtual license plate 401-3 can include information related to the region, information related to time, and / or information for identifying the vehicle. For example, a text object (e.g., Jing) representing the characters arranged at the first position 431 can represent information related to the region (e.g., Beijing). As an example, the text object representing the characters arranged at the first position 431 can be the abbreviation of a province included in the second country. A text object (e.g., N) representing the characters arranged at the second position 432 can include information related to the region or use. The text object representing the characters arranged at the second position 432 may include information related to the location of the provincial government, the city of the province, a city under the direct control of the central government, the use, and / or time included in the second country. However, it is not limited thereto. A text object (e.g., 888R8) representing a number or character arranged at the third position 433 can represent a serial number for identifying the vehicle.

[0054] For example, the color of the virtual license plate 401-3 can include information related to the vehicle according to the license plate rules corresponding to the second country. In one example, the first color (e.g., blue) can represent a passenger car. The second color (e.g., yellow) can represent a heavy passenger car, a bus, a truck, and / or a motorcycle. The third color (e.g., white) can represent the use of the vehicle (e.g., police car, military vehicle). The fourth color (e.g., black) can represent the use of the vehicle (e.g., a vehicle related to a foreign company, embassy or consulate). The fifth color (e.g., green) can represent the type of vehicle (e.g., electric vehicle). However, it is not limited thereto.

[0055] According to an embodiment, the electronic device 101 can obtain the virtual license plate 401-4 based on the license plate rules corresponding to a third country (e.g., Europe). For example, the electronic device 101 can obtain the virtual license plate 401-4 using data corresponding to the virtual license plate 401-4.

[0056] For example, the virtual license plate 401-4 may include information about the region. For example, the visual object located at the first position 441 may represent a union organization (e.g., EU) related to a third country. The text object (e.g., EST) representing the letters located at the second position 442 may represent information about the country. The spaces and text objects located at the third position 443, the fourth position 444, and the fifth position 445 may represent a serial number for identifying the vehicle. According to an embodiment, a visual object (e.g., a national flag) may be located at the third position 443, the fourth position 444, or the fifth position 445. According to an embodiment, the number of spaces and text objects located at the third position 443, the fourth position 444, and the fifth position 445 may be different. The color of the virtual license plate 401-4 may include information about the country (e.g., yellow or white for the UK). However, this is not limited thereto. For example, a virtual license plate corresponding to a fourth country (e.g., the USA) may further include a visual object.

[0057] According to an embodiment, the electronic device 101 can obtain the virtual license plate 401-5 based on the license plate rule corresponding to the fifth country (e.g., Japan). For example, the electronic device 101 can obtain the virtual license plate 401-5 using data corresponding to the virtual license plate 401-5. For example, the virtual license plate 401-5 can include information about the region. For example, the text object disposed at the first position 451 can represent the region. The text object disposed at the second position 452 can represent the business use for using the vehicle. The text object disposed at the third position 453 (e.g., 500) can represent the type and / or use of the vehicle. The text object disposed at the fourth position 454 (e.g., 46-49) can represent information for identifying the vehicle. The color of the virtual license plate 401-5 can be divided into a first color (e.g., white), a second color (e.g., green), a third color (e.g., yellow), and / or a fourth color (e.g., black) according to the use for using the vehicle. However, the color is not limited thereto.

[0058] For example, the electronic device 101 can identify sub-locations 451-1, 451-2, 452, 453-1, 453-2, 453-3, 454-1, 454-2, 454-3, 454-4, 454-5 for setting text objects according to the type of the virtual license plate 401-5. The number of sub-locations can vary according to the type of the virtual license plate. As an example, with reference to the drawings, the number of text objects included in a sub-location is shown as one, but according to an embodiment, it may include multiple text objects. The electronic device 101 can obtain a dataset based on the JSON format, such as Tables 1 to 9 in FIG. 3, based on one dataset related to the type of the virtual license plate 401-5 and a total of 12 datasets including the number of sub-positions 451-1, 451-2, 452, 453-1, 453-2, 453-3, 454-1, 454-2, 454-3, 454-4, 454-5 in the virtual license plate 401-5 (e.g., a total of 11 including four letters (451-1, 451-2, 452 and 454-3) and seven numbers (453-1, 453-2, 453-3, 454-1, 454-2, 454-4 and 454-5)). Each data set may include at least some different information based on the respective types (e.g., numbers or letters) of text objects contained within the sub-locations 451-1, 451-2, 453-1, 453-2, 453-3, 454-1, 454-2, 454-3, 454-4, 454-5. Based on the information corresponding to the type of the virtual license plate 401-5, the electronic device 101 may train a neural network to obtain the virtual license plate 401-5 by setting each of the sub-locations 451-1, 451-2, 452, 453-1, 453-2, 453-3, 454-1, 454-2, 454-3, 454-4, 454-5 to a text object (e.g., numbers or letters) designated according to the type of the virtual license plate 401-5.

[0059] In other words, the electronic device 101 according to an embodiment can generate characters specified within the coordinate values ​​of the virtual license plate 401-5 area according to the type of the virtual license plate 401-5 and the coordinate values ​​of the sub-areas 451-1, 451-2, 452, 453-1, 453-2, 453-3, 454-1, 454-2, 454-3, 454-4, and 454-5 in the virtual license plate 401-5 from a data file labeled in JSON format. Note that in this specification, a rectangle is used as the shape of the area specified in the virtual license plate 401-5, and therefore the specified area is expressed by only two values ​​representing the left-top coordinate and the right-bottom coordinate located in the virtual license plate 401-5, but this is not limited thereto. In other words, if the shape of the area specified within the virtual license plate 401-5 is another shape, such as an N-sided polygon or a circle, the values ​​necessary to easily specify the position of the area according to that shape can be used.

[0060] According to an embodiment, the electronic device 101 can obtain virtual license plates 401 based on various types according to license plate rules corresponding to a country. In FIG. 4, vehicle license plates stipulated in various countries such as Korea, the United States, Europe, China, and Japan are illustrated as examples of the virtual license plates 401, but this is merely an embodiment and does not limit the embodiment of the present invention. Then, based on obtaining the virtual license plates 401 based on the various types, the electronic device 101 can train a neural network for identifying license plates to identify virtual license plates based on various types. The electronic device 101 can set visual objects and / or text objects in different positions according to the license plate type (e.g., virtual license plates 401-1, 401-2, 401-3, 401-4, 401-5) and the license plate rule corresponding to a country. The electronic device 101 can train a neural network to identify (or set) text objects and / or visual objects included in the license plate based on position information according to the license plate type.

[0061] The virtual license plate 401 acquired by the electronic device 101 according to an embodiment as described above may not be an existing license plate. For example, the virtual license plate 401 acquired by the electronic device 101 may not match the vehicle included in the image in which the virtual license plate 401 is synthesized. The electronic device 101 may randomly combine text objects included in the virtual license plate 401 to generate the non-existent virtual license plate 401. The electronic device 101 may reduce the infringement of personal information based on generating the non-existent virtual license plate 401.

[0062] An example of a neural network for obtaining a virtual license plate according to an embodiment will be described below with reference to FIG.

[0063] 5 shows an example of a neural network used by an electronic device to obtain a virtual license plate according to an embodiment. The electronic device of FIG. 5 may include the electronic device 101 of FIG. 1 to FIG. 4. At least one of the operations of FIG. 5 may be performed by the processor 120 of FIG. 1.

[0064] 5, an electronic device according to one embodiment may include a neural network 505 (e.g., a conditional generative adversarial net (CGAN)). The neural network 505 may include one or more neural networks (e.g., a generator neural network or a discriminator neural network).

[0065] For example, the neural network 505 may include one or more layers. Each of the one or more layers may include a function and / or a filter for operating on data. Each of the one or more layers may be a convolution filter, a fully connected layer, or various types of filters or layers bundled together based on a special function or characteristic. The neural network 505 may form a deep neural network that includes one or more layers. Training a deep neural network is called deep learning.

[0066] For example, the nodes included in one or more layers may be connected to each other via connection lines having connection weights. Tuning of the neural network may be performed based on, for example, supervised learning, unsupervised learning, and / or adversarial learning. For example, the neural network 505 may include a generative neural network 505-1 for acquiring license plates and / or a discriminative neural network 505-2 for discriminating the acquired license plates. The electronic device may use the generative neural network 505-1 and / or the discriminative neural network 505-2 to generatively train a neural network for generating license plates. For example, in one embodiment, the electronic device may use a generative adversarial network (GAN) that competitively trains two neural networks simultaneously to generate license plates.

[0067] According to an embodiment, the electronic device can identify license plates based on a specified number (e.g., 30). The types of license plates can be different. The electronic device can identify information about at least one license plate 501 of the identified license plates. The electronic device can use a neural network 505 to generate the license plate 501 of the identified license plates.

[0068] For example, the electronic device can identify a dataset 502 of license plate 501. The dataset 502 can be input to a neural network 505. The electronic device can input the dataset 502 to the neural network 505 to train the neural network 505 to acquire an image corresponding to the license plate 501. In one example, the electronic device can identify a template corresponding to the license plate rules of the country of the license plate 501 to acquire an image of the license plate 501. The electronic device can acquire the license plate through the neural network 505 using the template, a specified number (e.g., 30), and a condition vector 503 corresponding to the license plate to be acquired. However, the electronic device is not limited thereto.

[0069] For example, the electronic device can input a captured license plate to the discriminative neural network 505-2 based on capturing the license plate using the generative neural network 505-1. The electronic device can use the discriminative neural network 505-2 to determine whether the captured license plate is one of the license plates captured or based on a specified number. The electronic device can capture license plates based on various types based on performing adversarial learning using the generative neural network 505-1 and the discriminative neural network 505-2.

[0070] An example of a virtual license plate acquired by the electronic device using the neural network 505 will now be described with reference to FIG.

[0071] 6 shows an example of a virtual license plate acquired by an electronic device using a neural network according to an embodiment. The electronic device 101 in FIG. 6 may include the electronic device 101 in FIG. 1 to FIG. 5. At least one of the operations in FIG. 6 may be performed by the processor 120 in FIG. 1.

[0072] Referring to FIG. 6, the electronic device 101 according to one embodiment can obtain an outline-based virtual license plate 611 using a template via the neural network 505 of FIG.

[0073] For example, the electronic device 101 can use the outline-based virtual license plate 611 to obtain a virtual license plate 612 that resembles a real license plate based at least in part on changing values ​​of pixels included in the outline-based virtual license plate 611. The electronic device 101 can obtain the virtual license plate 612 via a neural network (e.g., cycle GAN). For example, the electronic device 101 can train the neural network to change the outline-based virtual license plate 611 to a virtual license plate 612 that resembles a real license plate. For example, the virtual license plate 612 obtained by the electronic device 101 via the neural network can be stored in a memory as labeling data 615 for training a neural network to identify license plates. The electronic device 101 can store the labeling data 615 including one or more virtual license plates in a memory to train a neural network to identify license plates.

[0074] An example of a table showing at least a part of the country-specific license plate rules corresponding to the template used by the electronic device 101 to obtain a virtual license plate according to an embodiment will now be described with reference to Figs. 7 to 8b.

[0075] FIG 7 shows an example of a table showing license plates for identification by an electronic device according to an embodiment. FIG 8a-8b shows an example of a table showing types of license plates for identification by an electronic device according to an embodiment. The electronic device 101 of FIG 7-8b may include the electronic device 101 of FIG 1-6.

[0076] 7, a table 700 according to an embodiment may represent information of a text object used by the electronic device 101 to obtain a virtual license plate corresponding to a first country (e.g., South Korea). For example, referring to table 700, a classification column may be divided into vehicle type (e.g., by vehicle model) and vehicle use (e.g., by use).

[0077] For example, the category column can be categorized into passenger cars, vans, trucks, special cars such as cranes and snowplows, and emergency cars such as fire engines and police cars. The symbol column can be categorized by the range of numbers assigned to each vehicle type.

[0078] In one embodiment, the electronic device 101 can place a text object representing at least one number included in a range of numbers corresponding to a symbol column at a first position (e.g., 411) of FIG. 4 or a second position (e.g., 422) of FIG. 4.

[0079] For example, vehicles may be classified as for non-business use, for rental business, and for general business use. Vehicles for non-business use and / or rental business may correspond to a first range (e.g., 100-699). Vehicles for general business use may correspond to a second range (e.g., 01-69).

[0080] For example, vans can be classified as for non-business use, for rental business, and for general business use. Vans for non-business use and / or rental business may correspond to a third range (e.g., 700-799). Vans for general business use may correspond to a fourth range (e.g., 70-79).

[0081] For example, trucks can be classified as non-commercial and general commercial. Non-commercial trucks can correspond to a fifth range (e.g., 800-979). General commercial trucks can correspond to a sixth range (e.g., 80-97).

[0082] For example, a special-purpose vehicle may be classified as non-commercial, rental, and general commercial. The non-commercial and rental special-purpose vehicles may correspond to a seventh range (e.g., 980-997).

[0083] For example, emergency vehicles can be classified into police cars and fire engines. Emergency vehicles may correspond to an eighth range (eg, 998-999). However, the above examples are not limited thereto.

[0084] According to an embodiment, the use columns may be divided into non-business use, for automobile transportation businesses, and for diplomacy. The use columns may correspond to text objects representing characters.

[0085] According to an embodiment, the electronic device 101 can place at least one of the text objects at the second location 412 of FIG. 4 and / or the third location 423 of FIG.

[0086] For example, non-business vehicles can be classified as private cars, which can correspond to a first text object (e.g., ga, na, da, la, ma, geo, neo, deo, leo, meo, beo, seo, eo, jeo, go, no, do, lo, mo, bo, so, o, jo, gu, nu, du, lu, mu, bu, su, mu, ju).

[0087] For example, the automobile transport business can be classified into "for general" and "for rental businesses." For example, the automobile transport business for general can correspond to a second text object (e.g., ba, sa, a, ja, bae). The automobile transport business for rental businesses can correspond to a third text object (e.g., heo, ha, ho).

[0088] For example, diplomatic use can be classified into diplomatic use, consular use, quasi-diplomatic use, quasi-consular use, international organization use, and other diplomatic use. In one example, a text object related to diplomatic use can be located at a first position 421 in FIG. 4. For example, when the electronic device 101 uses a text object related to diplomatic use to obtain a virtual license plate, the color of the virtual license plate obtained using the text object can be a second color (e.g., indigo blue) different from the first color (e.g., white). However, the color is not limited thereto.

[0089] For example, a diplomatic use may correspond to a fourth text object (e.g., diplomacy); a consular use may correspond to a fifth text object (e.g., consul); a quasi-diplomatic use may correspond to a sixth text object (e.g., quasi-foreign); a quasi-consul use may correspond to a seventh text object (e.g., quasi-territorial); an international organization use may correspond to an eighth text object (e.g., national agency); and other diplomatic uses may correspond to a ninth text object (e.g., agreement, representative), but are not limited to these.

[0090] According to an embodiment, the electronic device 101 can obtain a virtual license plate by arranging at least one text object among a text object representing at least one number included in a number range corresponding to a vehicle type and a text object representing a character corresponding to a usage at a specified position in a background template. The electronic device 101 can obtain a virtual license plate similar to a real license plate based on identifying the text object representing a number and the text object representing a character using a table 700 corresponding to a license plate rule by country.

[0091] Referring to FIG. 8a, a table 800-1 showing types of license plates according to an embodiment is shown. Referring to FIG. 8b, a table 800-2 showing types of license plates according to an embodiment is shown. Tables 800-1 and 800-2 may include uses, specifications, and samples according to the types. In one example, the color of the license plate included in the sample may vary depending on the type. For example, the electronic device 101 may identify data for obtaining a virtual license plate using information on the type included in tables 800-1 and 800-2. For example, the information on the type may be included in the line information 313 of FIG. 3. When the electronic device 101 selects information on a first type (e.g., Type01) and does not select information on other types (e.g., Types02 to Type17) in the line information 313, the electronic device 101 may obtain a virtual license plate corresponding to the first type (e.g., a virtual license plate similar to the sample corresponding to Type01). However, the present invention is not limited thereto.

[0092] For example, electronic device 101 can obtain virtual license plates that resemble actual license plates of other countries using license plate rules corresponding to other countries that are distinct from the country associated with Table 700, Table 800-1, and / or Table 800-2 (e.g., South Korea).

[0093] As described above, the electronic device 101 according to an embodiment can obtain a virtual license plate based on a specified size by combining a text object representing at least one number included in table 700 and a text object representing at least one character included in tables 800-1 and 800-2. The electronic device 101 can obtain a virtual license plate similar to a real license plate using information included in tables 700, 800-1, and / or 800-2 corresponding to country-specific license plate rules. The electronic device 101 can train a neural network for identifying license plates based on obtaining a virtual license plate similar to a real license plate.

[0094] Hereinafter, an example of an operation in which the electronic device 101 according to an embodiment acquires position information of an actual license plate attached to a vehicle included in an image showing an actual environment will be described with reference to FIGS.

[0095] FIG. 9 shows an example of an image used by an electronic device according to an embodiment to acquire location information. FIG. 10 shows an example of an operation in which an electronic device according to an embodiment acquires location information of an actual license plate attached to a vehicle. The electronic device 101 in FIGS. 9-10 may include the electronic device 101 in FIGS. 1-8b. At least one of the operations in FIGS. 9-10 may be executed by the processor 120 in FIG. 1.

[0096] 9, an electronic device 101 according to an embodiment may identify, in a memory (e.g., memory 130 of FIG. 1), one or more images 900 including a vehicle. The one or more images 900 may represent an image showing a real environment including the vehicle. The one or more images 900 may include a second image 161. Each of the one or more images 900 may be combined with a first image (e.g., first image 160 of FIG. 1) including a virtual license plate acquired by the electronic device 101. The operation of the electronic device 101 to combine the second image 161 and the first image of the one or more images 900 will be described below with reference to FIGS. 14-15.

[0097] According to an embodiment, the electronic device 101 can identify a vehicle 905 in the second image 161 corresponding to the vehicle. For example, the electronic device 101 can identify an actual license plate 910 attached to the vehicle 905. The electronic device 101 can identify vertices 901 of the actual license plate 910. In an example, but not limited to, the electronic device 101 can identify an input indicating a selection of a vertex. For example, the electronic device 101 can use the vertices 901 to identify edges (e.g., edges that make up a bounding box) corresponding to the actual license plate 910.

[0098] 10, the electronic device 101 according to an embodiment can change the position of the edge of the real license plate 910 acquired using the vertices 901. For example, the electronic device 101 can use pixels included in the second image 161 to identify position information of the real license plate 910. For example, the electronic device 101 can use the position information of the vertices 901 to identify pixels included in the second image 161 that include the real license plate 910. The electronic device 101 can use the pixels that include the real license plate 910 to identify edges and / or vertices of the real license plate 910. In one example, the electronic device 101 can identify edges (e.g., edges acquired using the vertices 1010) adjacent to the edges of the real license plate 910 acquired using the vertices 901.

[0099] For example, a sub-image 1020 cropped from the second image 161 based on an edge of the actual license plate 910 may include a vertex 1010 of the actual license plate 910 .

[0100] For example, the electronic device 101 may change a first edge corresponding to the actual license plate 910 identified using the vertex 901 in FIG. 9 to a second edge of the actual license plate 910 identified using the pixels that include the actual license plate 910. The second edge may be identified based on the vertex 1010 in FIG. 10. In one example, the electronic device 101 may change the position of each of the vertices 901 in FIG. 9 to the position of each of the vertices 1010 in FIG. 10. In one example, the position of the vertex 901-1 may be changed to the position of the vertex 1010-1. The position of the vertex 901-2 may be changed to the position of the vertex 1010-2. The position of the vertex 901-3 may be changed to the position of the vertex 1010-3. The position of the vertex 901-4 may be changed to the position of the vertex 1010-4.

[0101] For example, the electronic device 101 can use the vertices 901 in FIG. 9 to identify position information (e.g., coordinate values) of each of the second edges that are located at a shortest distance (e.g., Manhattan distance) from each of the first edges corresponding to the identified actual license plate 910. The position information of the second edges can correspond to the position information of the actual license plate 910 included in the second image 161. For example, based on identifying each of the vertices 1010, the electronic device 101 can identify the position information of the actual license plate included in the second image 161. The position information can be used by the electronic device 101 to project the virtual license plate.

[0102] For example, the electronic device 101 can use the input to the vertices 901 in Fig. 9 to identify first position information of the actual license plate 910, and can use the general first position information of the identified actual license plate 910 to identify pixels included in the second image 161 that include the actual license plate 910. The electronic device 101 can identify edges that constitute the actual license plate 910 within the pixels that include the actual license plate 910. The electronic device 101 can obtain position information of the actual license plate 910 using the edges.

[0103] An example of an operation for changing the color of a virtual license plate acquired by the electronic device 101 using a template according to an embodiment will be described below with reference to Figs. 11 and 12.

[0104] Fig. 11 shows an example of an operation in which an electronic device according to an embodiment uses color information of a real license plate to change the color of a virtual license plate. Fig. 12 shows an example of an operation in which an electronic device according to an embodiment changes the color of a virtual license plate using a histogram showing the color of a real license plate. The electronic device 101 in Figs. 11-12 may include the electronic device 101 in Figs. 1-10.

[0105] 11 , the electronic device 101 according to an embodiment can identify an actual license plate 1110 in the second image 161. The electronic device 101 can identify color information of each pixel included in the actual license plate 1110 in the second image 161.

[0106] For example, the electronic device 101 may identify at least a portion of the actual license plate 1110. The electronic device 101 may identify a text object included in the actual license plate 1110. The electronic device 101 may sample pixels included in at least a portion of the actual license plate that is distinguished from other portions including the text object. The electronic device 101 may identify RGB values ​​corresponding to other portions for refraining from acquiring color information for the other portions including the text object. In one example, the electronic device 101 may temporarily refrain from acquiring color information of pixels having RGB values ​​less than a specified value (e.g., 90) among the RGB values ​​of each pixel included in the actual license plate 1110. The electronic device 101 may temporarily refrain from acquiring color information corresponding to the other portions based on the temporarily refraining from acquiring color information of pixels having RGB values ​​less than the specified value.

[0107] For example, the electronic device 101 can obtain color information 1120 including RGB values ​​1121 of a first region 1111 and RGB values ​​1122 of a second region 1112 of a portion of the actual license plate 1110. For example, the electronic device 101 can obtain color information 1120 including RGB values ​​of a plurality of regions (e.g., about 300 regions) including the first region 1111 and the second region 1112.

[0108] In one embodiment, the electronic device 101 can identify color information 1130 of the virtual license plate 1150 (or the first image 160 of FIG. 1 that includes the virtual license plate 1150). For example, the electronic device 101 can use the color information 1120 to change the color of at least one of the pixels included in the virtual license plate 1150.

[0109] For example, the electronic device 101 can change the color of at least a part of the virtual license plate 1150. The electronic device 101 can change the color of at least a part of the virtual license plate 1150 that is distinguished from other parts including text objects included in the virtual license plate 1150. For example, the electronic device 101 can obtain color information of the virtual license plate 1150 including the RGB value 1131 of the first region 1141 and the RGB value 1132 of the second region 1142 included in at least a part of the virtual license plate 1150. In one example, since the virtual license plate 1150 is generated by the electronic device 101, the color of at least a part of the virtual license plate 1150 may be substantially the same. However, the present invention is not limited thereto.

[0110] For example, the electronic device 101 can change the color of at least one of the pixels included in the virtual license plate 1150 (or the first image 160 including the virtual license plate 1150) based on changing each of the RGB values ​​(e.g., RGB value 1131 and RGB value 1132) corresponding to the color (e.g., color information 1130) of the virtual license plate 1150 using the color information 1120 indicating the RGB value corresponding to the color of the real license plate 1110. As an example, the electronic device 101 can change the RGB value of one region of the virtual license plate 1150 using at least one of the RGB values ​​included in the color information 1120. As an example, the electronic device 101 can change (or replace) the RGB value 1132 of the second region 1142 of the virtual license plate 1150 using the RGB value 1121 of the first region 1111 of the real license plate 1110. However, this is not limited to the above.

[0111] 12, the electronic device 101 according to an embodiment may acquire RGB values ​​of a first region 1111 having the largest RGB value among pixels included in the second image 161 and a second region 1112 having the smallest RGB value among pixels included in the second image 161. For example, an RGB value 1221 of the first region 1111 may be the largest RGB value among pixels included in the second image 161. For example, an RGB value 1222 of the second region 1112 may be the smallest RGB value among pixels included in the second image 161. The electronic device 101 may acquire color information 1220 including the RGB value 1121 and the RGB value 1222. The color information 1220 may be represented as a histogram 1230 based on color and frequency.

[0112] For example, the electronic device 101 can identify color information of the virtual license plate 1150 (or the first image 160 including the virtual license plate 1150). The color information of the virtual license plate 1150 can be represented as a histogram (not shown). The electronic device 101 can modify the histogram of the virtual license plate 1150 using the histogram 1230 of the real license plate 1110. The electronic device 101 may modify the histogram of the virtual license plate 1150 similarly to the histogram 1230 of the real license plate 1110. For example, based on the electronic device 101 modifying the histogram of the virtual license plate 1150, the color information of the virtual license plate 1150 corresponding to the histogram can be modified. For example, the electronic device 101 can use the histogram 1230 to modify the color of at least a portion 1240 of the virtual license plate 1150. The at least a portion 1240 can be distinguished from other portions, including text objects, in the virtual license plate 1150. For example, the electronic device 101 may use the number 1 to change the color of at least one of the pixels included in the virtual license plate 1150 (or the first image 160 including the virtual license plate 1150).

number

[0113] The electronic device 101 according to the above-mentioned embodiment can obtain a virtual license plate 1150 having a color similar to that of the actual license plate 1110 by changing the color of the virtual license plate 1150 using color information of the actual license plate 1110.

[0114] Hereinafter, with reference to FIG. 13, an example of an operation of the electronic device 101 according to an embodiment for acquiring noise using a virtual license plate 1150 with a changed color will be described.

[0115] Fig. 13 is a diagram illustrating an example of a virtual license plate based on noise acquired by an electronic device according to an embodiment. The electronic device 101 in Fig. 13 may include the electronic device 101 in Figs. 1 to 12. At least one of the operations in Fig. 13 may be performed by the processor 120 in Fig. 1. The virtual license plate 1300 may be referenced to the color-changed virtual license plate 1150.

[0116] According to an embodiment, the electronic device 101 can combine noise 1310 with the virtual license plate 1300. For example, the electronic device 101 can generate noise 1310 for the virtual license plate 1300 based on modifying parameters that represent pixels included in the virtual license plate 1300.

[0117] For example, the electronic device 101 may generate noise 1310 based on salt and pepper noise for generating at least one dot on the virtual license plate 1300. The electronic device 101 may generate noise 1310 based on Gaussian noise. The electronic device 101 may generate noise 1310 based on uniform noise. The electronic device 101 may generate noise 1310 based on at least one of resizing, gradation, a filter (e.g., a bilateral filter), and / or a morphology operation.

[0118] For example, the electronic device 101 may use one or more operations to generate the noise 1310. The electronic device 101 may obtain the noise 1310 using at least one of point noise, gradation, resizing, and / or Gaussian noise. For example, the electronic device 101 may obtain the noise 1310 using at least one of resizing, a morphological operation (e.g., dilation), a filter (e.g., bilateral filter), point noise, and / or gradation. For example, the electronic device 101 may render the first image 160 including the noise 1310 using a filter for modifying parameter values ​​representing pixels included in the virtual license plate 1300. The electronic device 101 may use the filter to perform an inverse operation of the filter for removing the noise.

[0119] For example, the electronic device 101 can modify the parameter values ​​of the pixels for the virtual license plate 1300 using a kernel (e.g., a kernel related to a morphological operation). The electronic device 101 can generate noise 1310 for the virtual license plate 1300 based on modifying the parameter values ​​to at least one of 0 or 1 using the kernel. The electronic device 101 can obtain a first image (e.g., the first image 160 of FIG. 1) including a virtual license plate similar to a real license plate included in a second image (e.g., the second image 161 of FIG. 1) based on generating the noise 1310. The electronic device 101 can store the first image and the second image based on generating the noise 1310 as labeling information for training a neural network to identify license plates.

[0120] The electronic device 101 according to one embodiment described above can obtain a virtual license plate similar to a real license plate based on obtaining noise 1310 on a color-changed virtual license plate 1300 .

[0121] Hereinafter, an example of an operation in which the electronic device 101 projects a first image including noise 1310 and a virtual license plate 1300 onto at least a portion of a second image will be described with reference to FIG.

[0122] 14 illustrates an example of an operation for an electronic device to project a first image onto a second image according to an embodiment. The electronic device 101 in FIG. 14 may include the electronic device 101 in FIGS.

[0123] 14, an electronic device 101 according to an embodiment can combine a first image 160 including a virtual license plate with a second image 161. The electronic device 101 can project the first image 160 onto at least a portion of the second image 161 based on a homography. The homography can represent a certain relationship between projected matching points when one image is projected onto another image.

[0124] For example, the electronic device 101 can project the first image 160 onto at least a portion of the second image 161 using position information of the real license plate identified using the vertices 1010 of FIG. 10. For example, the electronic device 101 can project a virtual license plate (e.g., a virtual license plate included in the first image 160) onto a real license plate included in the second image 161 using a matrix for mapping each of the vertices of the virtual license plate to each of the vertices 1010 of the real license plate using the position information. For example, the electronic device 101 can project the first image 160 onto at least a portion of the second image 161 based on the planar surface 1400. For example, the matrix can be expressed as follows:

number

[0125] 15 is a diagram showing an example of an operation of an electronic device according to an embodiment, in which a third image is acquired by combining a first image including a virtual license plate with a second image and storing the third image as labeling information. The electronic device 101 in FIG. 15 may include the electronic device 101 in FIG. 1 to FIG. 14. At least one of the operations in FIG. 15 may be executed by the processor 120 in FIG. 1.

[0126] 15, the electronic device 101 according to an embodiment can identify position information of a real license plate (e.g., real license plate 910 in FIG. 9) attached to a vehicle 905 in the second image 161. For example, the electronic device 101 can project each of the vertices of the first image 160 (or a virtual license plate included in the first image) onto each of the vertices of the real license plate based on a homography.

[0127] For example, the electronic device 101 can combine the first image 160 and the second image 161 based on the position information of the real license plate. The electronic device 101 can obtain a third image 162 including a vehicle 905 with a virtual license plate included in the first image 160 synthesized therewith.

[0128] For example, the electronic device 101 may store as labeling information 165 a text object (e.g., Gyeonggi 43 ga 5181) contained in the first image 160, location information indicating the location in the third image to which the first image 160 is combined, and the third image 162. The labeling information 165 may be used to train a neural network to identify license plates.

[0129] As described above, the electronic device 101 according to an embodiment can obtain a virtual license plate using a template based on a license plate rule for each country (e.g., the template 150 in FIG. 1). The electronic device 101 can identify color information of the actual license plate attached to the vehicle 905 included in the second image 161 and position information of the actual license plate in the second image 161. The electronic device 101 can change the color of at least one of the pixels included in the virtual license plate using the color information. The electronic device 101 can generate noise by changing parameters of the pixels included in the virtual license plate whose color has been changed. The electronic device 101 can obtain a first image 160 including the virtual license plate from which the noise has been generated. The electronic device 101 can project the first image 160 onto the actual license plate included in the second image 161 based on the homography using the position information. The electronic device 101 can use a third image 162 obtained by combining the first image 160 and the second image 161 to train a neural network. The electronic device 101 can store the third image 162 in memory as labeling information 165. The electronic device 101 can use the template to obtain the labeling information 165 that is used to train a neural network to identify license plates.

[0130] FIG 16 illustrates an example of a flow chart showing the operation of an electronic device according to an embodiment. At least one of the operations of FIG 16 may be performed by the electronic device 101 of FIG 1 and / or the processor 120 of FIG 1. The operations of FIG 16 may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0131] Referring to FIG. 16, in operation 1610, a processor according to an embodiment may use a template (e.g., template 150 in FIG. 1) representing a type of license plate to obtain a first image (e.g., first image 160 in FIG. 1) including a virtual license plate including text objects. For example, the type of virtual license plate obtained by electronic device 101 may vary based on country-specific license plate rules. The virtual license plate may include text objects representing numbers, text objects representing letters, visual objects representing locations (e.g., countries or regions), and / or figures. The color of the virtual license plate may vary depending on the type of vehicle. For example, the processor may utilize a neural network based on adversarial learning (e.g., neural network 505 in FIG. 5) to obtain the virtual license plate. This is not limiting.

[0132] Referring to FIG. 16, in operation 1620, a processor according to an embodiment can obtain position information of the actual license plate and color information of at least a part of the actual license plate based on identifying a second image (e.g., an actual license plate (e.g., actual license plate 910 in FIG. 9) attached to the vehicle in the second image 161 in FIG. 1) corresponding to a vehicle (e.g., vehicle 905 in FIG. 9). For example, the processor can represent the color information of at least a part of the actual license plate using a histogram. The processor can obtain the color information including RGB values ​​of at least a part of the actual license plate. The histogram can be represented based on each of the RGB values. For example, the processor can obtain RGB values ​​of pixels included in at least a part of the actual license plate that are distinguished from other parts corresponding to text objects included in the actual license plate. The processor can sample the RGB values ​​of the pixels based on a specified number.

[0133] Referring to FIG. 16, in operation 1630, a processor according to an embodiment can use the color information to change the color of at least one of the pixels included in the first image. For example, the processor can change (or replace) each of the RGB values ​​of the pixels included in the first image to each of the RGB values ​​corresponding to the actual license plate. For example, the processor can change a histogram representing the color of the pixels included in the first image based on the histogram corresponding to the actual license plate. For example, the processor can combine noise (e.g., noise 1310 of FIG. 13) based on at least one of point noise, Gaussian noise, uniform noise, resizing, gradient, filter, and / or morphological operation to the first image including at least one pixel with at least one changed color.

[0134] Referring to FIG. 16, in operation 1640, a processor according to an embodiment can combine the first image including at least one pixel with at least one color change and the second image based on the position information to obtain a third image including a vehicle with a virtual license plate synthesized therein (e.g., the third image 162 in FIG. 1). For example, the first image including at least one pixel with at least one color change can refer to the first image with noise combined therein. For example, the processor can combine the first image and the second image based on mapping each of the vertices of the first image to each of the vertices of the actual license plate included in the second image using the position information. The processor can project each of the vertices of the first image to each of the vertices of the actual license plate included in the second image based on the homography.

[0135] Referring to Fig. 16, in operation 1650, a processor according to an embodiment may store a pair of the third image, the text object, and the position information indicating the position in the third image where the first image is combined as labeling information (e.g., labeling information 165 in Fig. 1) for training a neural network. The processor may train a neural network for identifying license plates using the labeling information. The operation of the processor training the neural network may be an example of supervised learning. However, it is not limited thereto.

[0136] 17 is a diagram illustrating the operation of an electronic device 101 for training a neural network based on a set of training data, according to one embodiment. The operations described with reference to FIG. 17 may be performed by the electronic devices of FIGS. 1-16.

[0137] Referring to FIG. 17, in operation 1702, an electronic device according to an embodiment may obtain a set of training data. The electronic device may obtain a set of training data for supervised learning. The training data may include a pair of input data and ground truth data corresponding to the input data. The ground truth data may represent output data to be obtained from a neural network that receives the input data that is a pair of the ground truth data. The ground truth data may be obtained by the electronic device described with reference to FIGS. 1 to 12.

[0138] For example, when training a neural network to recognize images, the training data may include information about the images and one or more objects contained in the images. Such information may include a category or class of objects identifiable through the images. The information may include a position, width, height, and / or size of a visual object in the image that corresponds to the object. The set of training data identified through operation 1702 may include multiple training data pairs. In the above example of training a neural network to recognize images, the set of training data identified by the electronic device may include multiple images and base truth data corresponding to each of the multiple images.

[0139] Referring to Fig. 17, at operation 1704, an electronic device according to an embodiment may perform training of a neural network based on a set of training data. In an embodiment where a neural network is trained based on supervised learning, the electronic device may input input data included in the training data to an input layer of the neural network. An example of a neural network including the input layer is described with reference to Fig. 18. From an output layer of the neural network that receives the input data via the input layer, the electronic device may obtain output data of the neural network corresponding to the input data.

[0140] In one embodiment, the training of operation 1704 may be performed based on a difference between output data and underlying truth data included in the learning data and corresponding to the input data. For example, the electronic device may adjust one or more parameters (e.g., weights, described below with reference to FIG. 18) associated with the neural network so that the difference is reduced based on a gradient descent algorithm. The operation of the electronic device adjusting the one or more parameters may be referred to as tuning the neural network. The electronic device may perform the tuning of the neural network based on the output data using a function defined to evaluate the performance of the neural network, such as a cost function. The difference between the output data and underlying truth data may be included as an example of a cost function.

[0141] 17, in operation 1706, an electronic device according to an embodiment may identify whether valid output data is output from the neural network trained by operation 1704. The output data being valid may mean that the difference (or cost function) between the output data and the basis truth data satisfies the conditions set for using the neural network. For example, if the average and / or maximum value of the difference between the output data and the basis truth data is less than or equal to a specified threshold, the electronic device may determine that valid output data is output from the neural network.

[0142] If no valid output data is output from the neural network (1706-NO), the electronic device may repeatedly perform training of the neural network based on operation 1704. The embodiment is not limited thereto, and the electronic device may repeatedly perform operations 1702, 1704.

[0143] Upon obtaining valid output data from the neural network (operation 1706-YES), the electronic device according to an embodiment may use the trained neural network based on operation 1708. For example, the electronic device may input other input data, which is differentiated from the input data input to the neural network as learning data, to the neural network. The electronic device may use the output data obtained from the neural network that has received the other input data as a result of inference made on the other input data based on the neural network.

[0144] Figure 18 is a block diagram of an electronic device 101, according to one embodiment. The electronic device 101 of Figure 18 may include the electronic devices of Figures 1-17. For example, the operations described with reference to Figure 17 may be performed by the electronic device 101 of Figure 18 and / or the processor 1810 of Figure 18.

[0145] Referring to FIG. 18, a processor 1810 of the electronic device 101 can execute computations related to a neural network 1830 stored in a memory 1820. The processor 1810 may include at least one of a CPU (center processing unit), a GPU (graphic processing unit), or an NPU (neural processing unit). The NPU may be implemented as a chip separate from the CPU, or may be integrated into a chip such as the CPU in the form of a system on a chip (SoC). An NPU integrated into a CPU may be referred to as a neural core and / or an AI (artificial intelligence) accelerator.

[0146] Referring to FIG. 18, the processor 1810 may identify a neural network 1830 stored in the memory 1820. The neural network 1830 may include a combination of an input layer 1832, one or more hidden layers 1834, and an output layer 1836. The aforementioned layers (e.g., the input layer 1832, the one or more hidden layers 1834, and the output layer 1836) may include multiple nodes. The number of hidden layers 1834 may vary depending on the embodiment, and a neural network 1830 including multiple hidden layers 1834 may be referred to as a deep neural network. The act of training a deep neural network may be referred to as deep learning.

[0147] In one embodiment, when the neural network 1830 has the structure of a feed forward neural network, a first node included in a particular layer can be connected to all of the second nodes included in another layer before the particular layer. The parameters stored in the memory 1820 for the neural network 1830 can include weights assigned to connections between the second node and the first node. In the neural network 1830 having the structure of a feed forward neural network, the value of the first node can correspond to a weighted sum of the values ​​assigned to the second node based on the weights assigned to the connections connecting the second node and the first node.

[0148] In one embodiment, when the neural network 1830 has the structure of a convolution neural network, a first node included in a particular layer can correspond to a weighted weight for a portion of a second node included in another layer prior to the particular layer. Some of the second nodes corresponding to a first node may be identified by a filter corresponding to the particular layer. The parameters stored in the memory 1820 for the neural network 1830 can include weights representing the filters. The filter can include one or more of the second nodes used to calculate the weighted weight for the first node, and weights corresponding to each of the one or more nodes.

[0149] According to one embodiment, the processor 1810 of the electronic device 101 may perform training on the neural network 1830 using the training data set 1840 stored in the memory 1820. Based on the training data set 1840, the processor 1810 may perform the operations described with reference to FIG. 17 to adjust one or more parameters stored in the memory 1820 for the neural network 1830.

[0150] According to an embodiment, the processor 1810 of the electronic device 101 can perform object detection, object recognition, and / or object classification using the neural network 1830 trained based on the learning dataset 1840. The processor 1810 can input an image (or video) acquired via the camera 1850 to the input layer 1832 of the neural network 1830. Based on the input layer 1832 to which the image is input, the processor 1810 can sequentially obtain values ​​of nodes of the layers included in the neural network 1830 and obtain a set of values ​​of nodes of the output layer 1836 (e.g., output data). The output data can be used as a result of estimating information included in the image using the neural network 1830. The embodiment is not limited thereto, and the processor 1810 can input an image (or video) acquired from an external electronic device connected to the electronic device 101 to the neural network 1830 via the communication circuit 1860.

[0151] In one embodiment, the neural network 1830 trained to process an image can be used to identify regions in the image that correspond to objects (object detection) and / or to identify classes of objects depicted in the image (object recognition and / or object classification). For example, the electronic device 101 can use the neural network 1830 to segment regions in the image that correspond to the objects based on a rectangular shape, such as a bounding box. For example, the electronic device 101 can use the neural network 1830 to identify at least one class of a plurality of specified classes that matches the object.

[0152] As described above, according to one embodiment, the electronic device may be configured to include a memory and a processor. The processor may be configured to obtain a first image including a virtual license plate including a text object using a template representing a type of license plate. The processor may be configured to obtain position information of the real license plate and color information of at least a part of the real license plate based on identifying a real license plate attached to the vehicle in a second image corresponding to the vehicle. The processor may be configured to change the color of at least one pixel included in the first image using the color information. The processor may be configured to combine the first image and the second image including at least one pixel with the at least one color changed based on the position information to obtain a third image including the vehicle with the virtual license plate synthesized. The processor may be configured to store a pair of the third image, the text object, and position information indicating a position in the third image where the first image is combined as labeling information for training a neural network.

[0153] For example, the processor may be configured to project the virtual license plate in the first image onto the real license plate attached to the vehicle using the position information representing vertices of the real license plate, and the processor may be configured to train the neural network to identify other real license plates in a fourth image distinct from the real license plate using the labeling information.

[0154] For example, the processor may be configured to obtain the color information by sampling pixels included in at least a portion of the real license plate that is differentiated from other portions of the real license plate, including other text objects that are differentiated from the text object. The processor may be configured to use the color information to change the color of at least a portion of the virtual license plate that is differentiated from other portions of the virtual license plate, including the text object of the virtual license plate. The processor may be configured to generate the noise of the virtual license plate based on changing a parameter that represents pixels included in the virtual license plate with at least a portion of the color changed.

[0155] For example, the processor may be configured to use the template, which includes a background template representing a background of the virtual license plate and a text template representing the text objects based on a specified font, to combine at least one of the text objects, including a text object representing a number and a text object representing a letter, at a specified position of the background template to obtain the first image including the virtual license plate.

[0156] For example, the processor may be configured to use the position information to project the virtual license plate in the second image onto the real license plate using a matrix to map each of the vertices of the virtual license plate to each of the vertices of the real license plate.

[0157] For example, the processor may be configured to modify a color of at least one of the pixels of the first image based on modifying each of the RGB values ​​corresponding to a color of the first image using color information representing RGB values ​​corresponding to a color of the actual license plate.

[0158] For example, the processor may be configured to identify the location information of the actual license plate using a region in the second image that includes pixels corresponding to the actual license plate based on identifying pixels corresponding to the actual license plate attached to the vehicle using pixels included in the second image.

[0159] For example, the processor may be configured to use the template to combine the text objects having a specified size to obtain the first image including the virtual license plate including the combined text objects.

[0160] In a method performed by an electronic device according to one embodiment described above, the method may include an operation of acquiring a first image including a virtual license plate including a text object using a template representing a type of license plate. The method may include an operation of acquiring position information of the real license plate and color information of at least a part of the real license plate based on identifying a real license plate attached to the vehicle in a second image corresponding to the vehicle. The method may include an operation of changing a color of at least one pixel included in the first image using the color information. The method may include an operation of combining the first image including at least one pixel with the at least one color changed and the second image based on the position information to acquire a third image including the vehicle with the virtual license plate synthesized therewith. The method may include an operation of storing a pair of the third image, the text object, and position information indicating a position in the third image to which the first image is combined as labeling information for training a neural network.

[0161] For example, the method can include projecting the virtual license plate in the first image onto the real license plate mounted on the vehicle using the position information representing vertices of the real license plate. The method can include training the neural network to identify other real license plates in a fourth image distinct from the real license plate using the labeling information.

[0162] For example, the method may include sampling pixels in at least a portion of the real license plate that is distinct from other portions of the real license plate that include other text objects that are distinct from the text object to obtain the color information. The method may include using the color information to change a color of at least a portion of the virtual license plate that is distinct from other portions of the virtual license plate that include the text object of the virtual license plate. The method may include generating noise for the virtual license plate based on changing a parameter that represents pixels in the virtual license plate with the color changed for the at least a portion of the virtual license plate.

[0163] For example, the method may include an operation of combining at least one of the text objects, including a text object representing a number and a text object representing a letter, at a specified position of the background template using the template, including a background template representing a background of the virtual license plate and a text template representing the text objects based on a specified font, to obtain the first image including the virtual license plate.

[0164] For example, the method may include projecting the virtual license plate in the second image onto the real license plate using a matrix to map each of the vertices of the virtual license plate to each of the vertices of the real license plate using the position information.

[0165] For example, the method may include an act of modifying a color of at least one of the pixels of the first image based on modifying each of the RGB values ​​corresponding to a color of the first image using the color information representing RGB values ​​corresponding to a color of the actual license plate.

[0166] For example, the method may include an operation of identifying the location information of the actual license plate using a region in the second image that includes pixels corresponding to the actual license plate mounted on the vehicle based on identifying pixels included in the second image that correspond to the actual license plate mounted on the vehicle.

[0167] In a computer-readable storage medium having stored thereon one or more programs according to one embodiment described above, the one or more programs, when executed by at least one processor of an electronic device, may include instructions to cause the electronic device to obtain a first image including a virtual license plate including a text object using a template representing a type of license plate. The one or more programs, when executed by a processor of an electronic device, may include instructions to cause the electronic device to obtain position information of the real license plate and color information of at least a portion of the real license plate based on identifying a real license plate attached to the vehicle in a second image corresponding to the vehicle. The one or more programs, when executed by a processor of an electronic device, may include instructions to cause the electronic device to use the color information to change a color of at least one of the pixels included in the first image. The one or more programs, when executed by a processor of an electronic device, may include instructions to cause the electronic device to combine the first image including at least one pixel with the at least one color changed based on the position information and the second image to obtain a third image including the vehicle with the virtual license plate combined. The one or more programs can be configured to include instructions that, when executed by a processor of an electronic device, cause the electronic device to store pairs of the third image, the text object, and position information representing a position in the third image to which the first image is combined as labeling information for training a neural network.

[0168] For example, the one or more programs can be configured to include instructions that, when executed by the processor of the electronic device, cause the electronic device to project the virtual license plate included in the first image onto the real license plate mounted on the vehicle using the position information representing vertices of the real license plate. The one or more programs can be configured to include instructions that, when executed by the processor of the electronic device, cause the electronic device to train the neural network to identify other real license plates in a fourth image that are distinct from the real license plate using the labeling information.

[0169] For example, the one or more programs, when executed by the processor of the electronic device, can be configured to include instructions that cause the electronic device to sample pixels included in at least a portion of the real license plate that is distinct from other portions of the real license plate, including other text objects that are distinct from the text object, to obtain the color information. The one or more programs, when executed by the processor of the electronic device, can be configured to include instructions that cause the electronic device to use the color information to change the color of at least a portion of the virtual license plate that is distinct from other portions of the virtual license plate, including the text object of the virtual license plate. The one or more programs, when executed by the processor of the electronic device, can be configured to include instructions that cause the electronic device to generate noise for the virtual license plate based on changing a parameter that represents pixels included in the virtual license plate with the at least a portion of the color changed.

[0170] For example, the one or more programs, when executed by the processor of the electronic device, can be configured to include instructions that cause the electronic device to use the template, including a background template representing a background of the virtual license plate and a text template representing the text objects based on a specified font, to bind at least one of the text objects, including a text object representing a number and a text object representing a letter, to a specified position of the background template to obtain the first image including the virtual license plate.

[0171] For example, the one or more programs may be configured to include instructions that, when executed by the processor of the electronic device, cause the electronic device to use the position information to project the virtual license plate in the second image onto the real license plate using a matrix to map each of the vertices of the virtual license plate to each of the vertices of the real license plate.

[0172] The aforementioned devices may be implemented with hardware components, software components, and / or a combination of hardware and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Furthermore, the processing device may also access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, the processing device may be described as being one in which one is used, but a person skilled in the art may understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations are possible, such as parallel processors.

[0173] The software may include a computer program, code, instructions, or a combination of one or more of these, and may configure or instruct a processing device to operate in a desired manner, either individually or collectively. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device to be interpreted by a processing device or to provide instructions or data to a processing device. The software may be distributed over computer systems connected to a network, and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0174] The method according to the embodiment may be implemented in the form of program instructions that can be executed via various computer means and recorded on a computer-readable medium. In this case, the medium may continue to store the computer-executable program or may temporarily store it for execution or download. Furthermore, the medium may be various recording or storage means in the form of a single or multiple hardware combined, and may not be limited to a medium directly connected to any computer system, but may be distributed on a network. Examples of the medium may be those configured to store program instructions, including magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, ROMs, RAMs, flash memories, and the like. In addition, examples of other media include recording media and storage media managed by application stores that distribute applications, sites that supply or distribute various other software, and servers.

[0175] Although the embodiments have been described above with reference to the limited embodiments and drawings, those skilled in the art may make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from that described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a manner different from that described, or may be replaced or substituted by other components or equivalents, and still achieve suitable results.

[0176] Accordingly, other aspects, embodiments, and equivalents of the claims are within the scope of the following claims. [Explanation of symbols]

[0177] 101 Electronic equipment 120 processors 130 Memory 150 Templates 151 Background Templates 153 Text Templates 160 1st image 161 Second Image 162 Third Image 165 Labeling Information 1810 Processor 1820 Memory 1830 Neural Networks 1840 training data set 1850 Camera 1860 Communication Circuit

Claims

1. 1. An electronic device comprising: a memory storing instructions and including one or more storage media; at least one processor including processing circuitry; The instructions, individually or collectively, when executed by the at least one processor, cause the electronic device to: obtaining a first image including a virtual license plate including a text object using a template representing a type of license plate; Obtain position information of the virtual license plate and color information of at least a portion of the virtual license plate based on identifying a virtual license plate attached to the vehicle in a second image corresponding to the vehicle; Using the color information to modify the color of at least one pixel in the first image; combining the first image and the second image, including the at least one pixel with the at least one color changed based on the position information, to obtain a third image including the vehicle with the virtual license plate synthesized therein; and and storing pairs of the third image, the text object, and position information indicating a position in the third image to which the first image is combined as labeling information for training a neural network.

2. The instructions, individually or collectively, when executed by the at least one processor, cause the electronic device to: projecting the virtual license plate contained in the first image onto the real license plate attached to the vehicle using the position information representing the vertices of the real license plate; and 2. The electronic device of claim 1, wherein the labeling information is used to train the neural network to identify other real license plates in a fourth image that are distinct from the real license plate in the third image.

3. The instructions, individually or collectively, when executed by the at least one processor, cause the electronic device to: Sampling pixels in at least a portion of the actual license plate distinct from other portions of the actual license plate including other text objects distinct from the text object to obtain the color information; Using the color information, modifying a color of at least a portion of the virtual license plate that is distinct from other portions of the virtual license plate, including the text object of the virtual license plate; and 3. The electronic device of claim 2, further comprising: a processor configured to generate noise for the virtual license plate based on modifying a parameter representing pixels included in the at least a portion of the color-modified virtual license plate.

4. The instructions, individually or collectively, when executed by the at least one processor, cause the electronic device to:

4. The electronic device of claim 3, further comprising: a background template representing a background of the virtual license plate; and a text template representing the text object based on a specified font, and the template is used to combine at least one of the text objects, including a text object representing a number and a text object representing a letter, at a specified position of the background template to obtain the first image including the virtual license plate.

5. The instructions, individually or collectively, when executed by the at least one processor, cause the electronic device to:

3. The electronic device of claim 2, further comprising: a matrix for mapping each of the vertices of the virtual license plate to each of the vertices of the real license plate using the position information to project the virtual license plate in the second image onto the real license plate.

6. The instructions, individually or collectively, when executed by the at least one processor, cause the electronic device to:

2. The electronic device of claim 1, further comprising: a computer-readable storage device configured to store a plurality of color information representing a plurality of pixels of the first image, the plurality of pixels being arranged to have a color that is different from the plurality of pixels of the first image;

7. The instructions, individually or collectively, when executed by the at least one processor, cause the electronic device to:

2. The electronic device of claim 1, further comprising: a region in the second image including pixels corresponding to the actual license plate based on identifying pixels corresponding to the actual license plate attached to the vehicle using pixels included in the second image; and a region in the second image including pixels corresponding to the actual license plate, the electronic device identifying the location information of the actual license plate.

8. The instructions, individually or collectively, when executed by the at least one processor, cause the electronic device to:

2. The electronic device of claim 1, further comprising: a template for combining the text objects having a specified size to obtain the first image including the virtual license plate including the combined text objects.

9. 1. A method performed by an electronic device, comprising: obtaining a first image including a virtual license plate including a text object using a template representing a type of license plate; obtaining position information of a real license plate and color information of at least a portion of the real license plate based on identifying a real license plate attached to the vehicle in a second image corresponding to the vehicle; modifying the color of at least one pixel in the first image using the color information; combining the first image, including the at least one pixel with the at least one color changed, and the second image based on the position information to obtain a third image including the vehicle with the virtual license plate synthesized therein; and storing pairs of the third image, the text object, and position information indicating a location in the third image to which the first image is combined as labeling information for training a neural network.

10. The method comprises: - projecting the virtual license plate contained in the first image onto the real license plate mounted on the vehicle using the position information representing the vertices of the real license plate; and 10. The method of claim 9, further comprising: using the labeling information to train the neural network to identify other real license plates in a fourth image distinct from the real license plate in the third image distinct from the labeling information.

11. The method comprises: sampling pixels in at least a portion of the actual license plate distinct from other portions of the actual license plate including other text objects distinct from the text object to obtain the color information; using the color information to change a color of at least a portion of the virtual license plate that is distinct from other portions of the virtual license plate, including the text object of the virtual license plate; and 11. The method of claim 10, comprising generating noise for the virtual license plate based on modifying parameters representing pixels included in the at least a portion of the color-modified virtual license plate.

12. The method comprises:

12. The method of claim 11, further comprising: combining at least one of the text objects, including a text object representing a number and a text object representing a letter, at a specified position of the background template using the template, including a background template representing a background of the virtual license plate and a text template representing the text object based on a specified font, to obtain the first image including the virtual license plate.

13. The method comprises: using said position information, a matrix for mapping each of said virtual license plate vertices to each of said real license plate vertices; The method of claim 10, comprising projecting the virtual license plate in the second image onto the actual license plate.

14. The method comprises:

10. The method of claim 9, comprising modifying a color of at least one of the pixels of the first image based on modifying each of the RGB values ​​corresponding to a color of the first image using the color information representing RGB values ​​corresponding to a color of the actual license plate.

15. The method comprises:

10. The method of claim 9, further comprising: identifying the location information of the actual license plate using a region in the second image that includes the pixels corresponding to the actual license plate based on identifying pixels corresponding to the actual license plate attached to the vehicle using pixels included in the second image.

16. A non-transitory computer-readable storage medium having one or more programs stored thereon, The one or more programs, when individually or collectively executed by at least one processor of an electronic device, obtaining a first image including a virtual license plate including a text object using a template representing a type of license plate; Obtain position information of the real license plate and color information of at least a portion of the real license plate based on identifying a real license plate attached to the vehicle in a second image corresponding to the vehicle; Using the color information to modify the color of at least one pixel in the first image; Combining the first image, including the at least one pixel with the at least one color changed, and the second image based on the position information to obtain a third image including the vehicle with the virtual license plate synthesized therein; and and a non-transitory computer-readable storage medium configured to include instructions that cause the electronic device to store pairs of the third image, the text object, and position information representing positions within the third image to which the first image is combined as labeling information for training a neural network.

17. The one or more programs, when individually or collectively executed by the at least one processor, projecting the virtual license plate included in the first image onto the real license plate attached to the vehicle using the position information representing the vertices of the real license plate; and 17. The non-transitory computer-readable storage medium of claim 16, configured to include instructions that cause the electronic device to train the neural network to identify other real license plates in a fourth image that are distinct from the real license plate in the third image using the labeling information.

18. The one or more programs, when individually or collectively executed by the at least one processor, Sampling pixels in at least a portion of the actual license plate distinct from other portions of the actual license plate including other text objects distinct from the text object to obtain the color information; Using the color information, modifying a color of at least a portion of the virtual license plate that is distinct from other portions of the virtual license plate, including the text object of the virtual license plate; and 20. The non-transitory computer-readable storage medium of claim 17, configured to include instructions that cause the electronic device to generate noise for the virtual license plate based on modifying a parameter that represents pixels included in the at least a portion of the color-modified virtual license plate.

19. The one or more programs, when individually or collectively executed by the at least one processor, 20. The non-transitory computer-readable storage medium of claim 18, further comprising instructions for causing the electronic device to: combine at least one of the text objects, including a text object representing a number and a text object representing a letter, at a specified position of the background template using the template, including a background template representing a background of the virtual license plate and a text template representing the text object based on a specified font, to obtain the first image including the virtual license plate.

20. The one or more programs, when individually or collectively executed by the at least one processor, 20. The non-transitory computer-readable storage medium of claim 17, configured to include instructions that cause the electronic device to project the virtual license plate in the second image onto the real license plate using a matrix to map each of the vertices of the virtual license plate to each of the vertices of the real license plate using the position information.

Images