Electronic device for identifying external object based on inputting partial area of image to neural network and method for the same
The electronic device segments images into partial regions using a neural network to enhance object recognition, addressing inefficiencies in existing methods and improving accuracy for applications like autonomous driving and augmented reality.
Patent Information
- Application Number
- JP2024072232
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-04-26
- Filing Date
- 2024-04-26
- Publication Date
- 2024-11-08
AI Technical Summary
Existing methods for identifying external objects using neural networks are inefficient and lack effective mechanisms to segment and process images for accurate object recognition, particularly in applications like autonomous driving and augmented reality.
An electronic device with a camera and processor that segments images into partial regions based on a field of view, identifies a reference plane, and uses a neural network to detect external objects within these regions, training the network with data sets from multiple roadways to enhance object recognition accuracy.
The method enables precise identification of external objects by reducing processor load and improving accuracy through efficient image segmentation and neural network training, suitable for applications in autonomous driving and augmented reality.
Smart Images

Figure 2024159721000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an electronic device and method for identifying external objects based on inputting sub-regions of an image into a neural network. [Background technology]
[0002] As image object recognition technology develops, various forms of services are emerging. These services can be used for Automatic Driving, Augmented Reality, Virtual Reality, Metaverse, etc., and can be provided through electronic devices owned by different users, such as smartphones. The services can be related to hardware and / or software mechanisms that mimic human behavior and / or thinking, such as artificial intelligence. Technologies related to artificial intelligence can include technologies that utilize neural networks that simulate the neural networks of living organisms. Summary of the Invention [Problem to be solved by the invention]
[0003] A method may be sought for identifying external objects using neural networks. [Means for solving the problem]
[0004] According to an embodiment, an electronic device may include a camera and a processor. The processor may identify a width of a window used to divide the image based on a field of view (FOV) of an image acquired through the camera. The processor may identify a height of the window based on a first region including a visual object corresponding to a reference surface. The processor may divide the first region into a plurality of sub-regions using the window based on the width and the height. The processor may identify whether an external object is included in the first sub-region from a neural network to which a first sub-region of the plurality of sub-regions is input. The processor may obtain a second sub-region in the image separated from the first sub-region based on an interval identified based on whether the external object is included in the first sub-region.
[0005] According to an embodiment, an electronic device may include a memory and a processor. The processor may obtain a first data set related to a first vehicle based on identifying a first vehicle on a first road among a plurality of lanes separated by lanes in an image stored in the memory. The processor may obtain a second data set related to a second vehicle based on identifying a second vehicle on a second road among the plurality of lanes, the second road being different from the first road. The processor may obtain a third data set related to a third vehicle based on identifying a third vehicle on a third road among the plurality of lanes, the third road being different from the first road and the second road. The processor may train a neural network using truth data based on each of the first data set, the second data set, and the third data set.
[0006] According to one embodiment, a method of an electronic device includes identifying a width of a window used to divide an image based on a field of view of the image acquired via a camera. The method includes identifying a height of the window based on a first region including a visual object corresponding to a reference surface. The method includes dividing a plurality of overlapping sub-regions in the image based on the width and the height. The method includes identifying whether an external object is included in the first sub-region from a neural network inputted with a first sub-region of the plurality of sub-regions. The method includes obtaining a second sub-region in the image spaced apart from the first sub-region based on a distance identified based on whether the external object is included in the first sub-region. Effect of the Invention
[0007] According to the present disclosure, external objects can be identified using neural networks. [Brief description of the drawings]
[0008] [Figure 1] 1 illustrates an example block diagram of an electronic device according to one embodiment. [Diagram 2] 1 illustrates an example of an electronic device for training a neural network, according to one embodiment. [Figure 3a] FIG. 2 is an exemplary diagram illustrating information identified by an electronic device for training a neural network, according to one embodiment. [Figure 3b] FIG. 2 is an exemplary diagram illustrating information identified by an electronic device for training a neural network, according to one embodiment. [Figure 4a] FIG. 10 is a diagram for explaining an example of an operation of an electronic device according to an embodiment performed to identify a reference surface. [Figure 4b] FIG. 10 is a diagram for explaining an example of an operation of an electronic device according to an embodiment performed to identify a reference surface. [Figure 4c] FIG. 10 is a diagram for explaining an example of an operation of an electronic device according to an embodiment performed to identify a reference surface. [Figure 5a] FIG. 2 is a diagram illustrating the operation of an electronic device to identify an external object, according to one embodiment. [Figure 5b] FIG. 2 is a diagram illustrating the operation of an electronic device to identify an external object, according to one embodiment. [Figure 5c] FIG. 2 is a diagram illustrating the operation of an electronic device to identify an external object, according to one embodiment. [Figure 5d] 1 illustrates an example of an electronic device for identifying an external object, according to one embodiment. [Figure 6a]1 illustrates an example of an operation of an electronic device performed to identify an external object. [Figure 6b] 1 illustrates an example of an operation of an electronic device performed to identify an external object. [Figure 6c] 1 illustrates an example of an operation of an electronic device performed to identify an external object. [Figure 7] FIG. 1 illustrates an example of a block diagram illustrating an autonomous driving system for a vehicle, according to one embodiment. [Figure 8] FIG. 1 illustrates an example of a block diagram illustrating an autonomous vehicle, according to one embodiment. [Figure 9] FIG. 1 illustrates an example of a block diagram illustrating an autonomous vehicle, according to one embodiment. [Figure 10] 1 illustrates an example of a gateway associated with a user device in accordance with various embodiments. [Figure 11] 1 illustrates an example of a flowchart for the operation of an electronic device according to one embodiment. [Figure 12] 1 illustrates an example of a flowchart for the operation of an electronic device according to one embodiment. [Figure 13] FIG. 1 illustrates the operation of an electronic device for training a neural network based on a set of training data. [Figure 14] FIG. 1 is a block diagram of an electronic device according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] The electronic device according to various embodiments disclosed herein may be a device of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computing device, a portable multimedia device, a portable medical device, a camera, an electronic device, or a consumer electronic device. The electronic device according to the embodiments disclosed herein is not limited to the aforementioned devices.
[0010] It should be understood that the various embodiments and the terms used therein are not intended to limit the technical features described herein to a particular embodiment, but include various modifications, equivalents, or alternatives of the embodiments. With respect to the description of the drawings, similar or related components may use similar reference numerals. The singular form of a noun corresponding to an item may include one or more of the item, unless the relevant context clearly indicates otherwise. In this specification, each of phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may include any of the items listed together in the corresponding phrase of the phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish the component from other corresponding components, and do not limit the component to other aspects (e.g., importance or order). When a (e.g., first) component is referred to as being "coupled" or "connected" to another (e.g., second) component, either in combination with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., by wire), wirelessly, or through a third component.
[0011] The term "module" as used herein may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integrated component or the smallest unit or portion of a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).
[0012] Various embodiments of the present specification may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium (built-in memory or external memory) readable by a machine (e.g., electronic device 101). For example, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) may retrieve and execute at least one of the one or more stored instructions from the storage medium. This enables the machine to operate to perform at least one function according to the retrieved at least one instruction. The one or more instructions may include code generated by a compiler or code that may be executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" only means that the storage medium is a tangible device and does not include a signal (e.g., electromagnetic wave), and the term does not distinguish between a case where data is stored semi-permanently and a case where data is stored temporarily in the storage medium.
[0013] According to one embodiment, the method according to the various embodiments disclosed herein may be provided in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)) or may be distributed through an application store (e.g., the Play Store, etc.). TM ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a part of the computer program product may be at least temporarily stored or temporarily generated on a machine-readable storage medium, such as a memory of a manufacturer's server, an application store's server, or an intermediary server.
[0014] According to various embodiments, each of the components (e.g., modules or programs) may include one or more individuals. Some of the multiple individuals may be separately arranged in different components. According to various embodiments, one or more of the aforementioned corresponding components or operations may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into one component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as performed by the corresponding component of the multiple components before integration. According to various embodiments, the operations performed by the modules, programs, or other components may be performed sequentially, in parallel, iteratively, or empirically, or one or more of the operations may be performed in a different order, omitted, or one or more other operations may be added.
[0015] Various embodiments of the present specification will now be described with reference to the accompanying drawings.
[0016] FIG. 1 illustrates an example block diagram of an electronic device according to one embodiment.
[0017] Referring to FIG. 1, an electronic device 101 according to an embodiment may include at least one of a processor 120, a memory 130, a camera 140, a sensor 150, a display 160, or a speaker 170. The processor 120, the memory 130, the camera 140, the sensor 150, the display 160, and the speaker 170 may be electrically and / or operably coupled with each other by an electronic component such as a communication bus 110. Hereinafter, the term "operably coupled" may mean that a direct connection between the hardware or an indirect connection between the hardware is established by wire or wirelessly, such that a first piece of hardware controls a second piece of hardware. Although shown in different blocks, the embodiment is not limited thereto. Some of the hardware in FIG. 1 may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of hardware included in the electronic device 101 is not limited to that shown in FIG. 1. For example, electronic device 101 may include only a portion of the hardware shown in FIG.
[0018] According to one embodiment, the electronic device 101 may include hardware for processing data based on one or more instructions. The hardware for processing data may include a processor 120. For example, the hardware for processing data may include an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). The processor 120 may have a single-core processor structure, or may have a multi-core processor structure, such as a dual core, a quad core, a hexa core, or an octa core.
[0019] According to an embodiment, the memory 130 of the electronic device 101 may include hardware components for storing data and / or instructions input and / or output to the processor 120 of the electronic device 101. For example, the memory 130 may include a volatile memory, such as a random-access memory (RAM), and / or a non-volatile memory, such as a read-only memory (ROM). For example, the volatile memory may include at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM (Cache RAM), and a pseudo SRAM (PSRAM). For example, the non-volatile memory may include at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disk, a solid state drive (SSD), and an embedded multi-media card (eMMC). For example, the memory 130 of the electronic device 101 may include a neural network. The electronic device 101 can identify external objects based on the neural network stored in the memory 130. For example, the electronic device 101 can train the neural network based on images stored in the memory 130. The electronic device 101 can train the neural network based on external objects identified in the images. For example, the electronic device 101 can train the neural network using images corresponding to external objects and corresponding to vehicles traveling on a road. For example, the road can include lanes. For example, the road can include lanes separated by the lanes. The electronic device 101 can train the neural network using the vehicles identified on each road. The operation of training the neural network is described below in FIG. 2.
[0020] According to one embodiment, the camera 140 of the electronic device 101 may include a lens assembly, or an image sensor. The lens assembly may collect light emitted from a subject for which an image is to be captured. The lens assembly may include one or more lenses. According to one embodiment, the camera 140 may include multiple lens assemblies. For example, the camera 140 may include multiple lens assemblies, some of which may have the same lens characteristics (e.g., angle of view, focal length, autofocus, f number, or optical zoom), or at least one lens assembly may have one or more lens characteristics that are different from the lens characteristics of the other lens assemblies. The lens assembly may include a wide-angle lens or a telephoto lens. According to one embodiment, the flash may include one or more light-emitting diodes (e.g., red-green-blue (RGB) LEDs, white LEDs, infrared LEDs, or ultraviolet LEDs), or a xenon lamp. For example, the image sensor may capture an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly into an electrical signal. According to an embodiment, the image sensor may include one image sensor selected from image sensors having different attributes, such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same attribute, or a plurality of image sensors having different attributes. Each image sensor included in the image sensor may be realized using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.
[0021] According to an embodiment, the electronic device 101 may capture an image using the camera 140. The electronic device 101 may identify an external object in the captured image. The electronic device 101 may identify a visual object in the image. For example, the electronic device 101 may identify a region for identifying an external object based on identifying a visual object corresponding to a reference plane among the visual objects. For example, the reference plane may include a plane on which gravity acts in a direction perpendicular to one plane. The electronic device 101 may divide the region into a plurality of partial regions. For example, the electronic device 101 may identify a width of a window for dividing the region based on a field of view (FOV) of the camera 140 and / or an image. For example, the electronic device 101 may identify a height of a window based on a region including a visual object corresponding to a reference plane. The electronic device 101 may divide the region into a plurality of partial regions based on the width and the height. An operation for identifying the width and / or height of a window will be described later with reference to FIG. 4a to FIG. 4c.
[0022] According to an embodiment, the electronic device 101 can identify distortion in an image acquired based on the camera 140. The electronic device 101 can identify intrinsic parameters of the camera 140. The electronic device 101 can identify distortion based on identifying the intrinsic parameters of the camera 140. For example, the electronic device 101 can identify radial distortion and / or tangential distortion based on the intrinsic parameters. The electronic device 101 can acquire a second image compensated for the distortion based on identifying distortion in an image, the first image. The electronic device 101 can identify an external object based on the second image. For example, the electronic device 101 can identify an external object using a neural network based on inputting the second image into the neural network.
[0023] According to an embodiment, the sensor 150 of the electronic device 101 may include a sensor for measuring the inertia of the electronic device 101, such as an inertial measurement unit (IMU) sensor. For example, the IMU sensor may include, but is not limited to, a gyro sensor, an acceleration sensor, and / or a 6-DOF (degree of freedom) sensor. The electronic device 101 may use the sensor 150 to identify an angle between the electronic device 101 and a reference surface. For example, the reference surface may include a surface on which gravity acts in a direction perpendicular to one surface. According to an embodiment, the electronic device 101 may use the reference surface to divide a region for identifying an external object. The operation of dividing a region for identifying an external object using a reference surface will be described later with reference to FIGS. 4a to 4c.
[0024] According to an embodiment, the sensor 150 of the electronic device 101 may include a sensor for measuring the speed of the electronic device 101. For example, the sensor for measuring the speed of the electronic device 101 may include a global positioning system (GPS) sensor. For example, the electronic device 101 may identify the velocity and / or speed of the electronic device 101 based on a change in the position of the electronic device 101 acquired based on the GPS sensor. The electronic device 101 may change the intervals between the multiple partial regions based on the speed and / or speed of the electronic device 101. The operation of adjusting the intervals between the partial regions will be described later with reference to FIGS. 5c to 5d.
[0025] According to an embodiment, the display 160 of the electronic device 101 can output visualized information to a user. For example, the display 160 can be controlled by the processor 120 including a circuit such as a graphic processing unit (GPU) and can output visualized information to a user. The display 160 can include a flat panel display (FPD) and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs). According to an embodiment, the electronic device 101 can display an image captured by the camera 140 via the display 160. The electronic device 101 can highlight an external object of a specified type based on identifying an external object of a specified type identified in the image. For example, the electronic device 101 can display a visual object corresponding to the external object superimposed on the external object of the specified type. For example, the electronic device 101 may change the transparency of a visual object based on changing an alpha value of the visual object, and the electronic device 101 may highlight an external object by overlaying the visual object with the changed transparency on the external object.
[0026] According to an embodiment, the speaker 170 of the electronic device 101 may include at least one voice coil that provides vibrations in a diaphragm within the speaker 170, and a magnet that can form a magnetic field. When a current flows through the at least one voice coil, the magnetic field formed by the voice coil can interact with the magnetic field formed by the magnet to vibrate the voice coil. The diaphragm connected to the voice coil can vibrate based on the vibration of the voice coil. The speaker 170 can output an audio signal based on the vibration of the diaphragm. According to an embodiment, the electronic device 101 can identify an external object in an image acquired via the camera 140. For example, the electronic device 101 can identify a type of the external object. The electronic device 101 can output an audio signal via the speaker 170 based on identifying the specified type of external object. For example, the specified type of external object may include an external object corresponding to a type that impedes the movement of the vehicle, such as an obstacle.
[0027] According to an embodiment, the electronic device 101 can identify whether an external object is included in a first partial region among the plurality of partial regions. The electronic device 101 can identify a second partial region based on whether an external object is included in the first partial region. For example, the electronic device 101 can identify a second partial region separated from the first partial region by a first interval based on identifying an external object included in the first partial region. For example, the electronic device 101 can identify a second partial region separated from the first partial region by a second interval based on the first partial region not including an external object. For example, the first interval may be wider than the second interval. The electronic device 101 can input the partial regions separated by the first interval and / or the second interval to a neural network. The electronic device 101 can identify an external object based on the neural network to which the partial regions are input.
[0028] As described above, according to an embodiment, the electronic device 101 can divide an image based on the FOV of an image acquired via a camera. For example, the electronic device 101 can identify a width of a window used to divide the image based on the FOV. According to an embodiment, the electronic device 101 can identify a first region including a visual object corresponding to a reference plane (e.g., the ground). The electronic device 101 can identify a height of the window based on identifying the first region. For example, the electronic device 101 can divide the first region into a plurality of partial regions using the window based on the width and height of the window. For example, the electronic device 101 can divide the first region into the plurality of partial regions based on a first interval. The electronic device 101 can input a first partial region of the plurality of partial regions to a neural network. The electronic device 101 can identify whether an external object is included in the first partial region from the neural network to which the first partial region is input. The electronic device 101 can identify an interval between the partial regions based on whether an external object is included in the first partial region. The electronic device 101 may obtain a second partial region spaced apart from the first partial region based on a spacing between the plurality of partial regions. For example, the spacing may include a first spacing and / or a second spacing wider than the first spacing. The electronic device 101 may reduce usage of the processor 120 by identifying the external object based on the plurality of partial regions based on the first spacing and / or the second spacing wider than the first spacing.
[0029] Figure 2 illustrates an example of an electronic device for training a neural network, according to one embodiment. The electronic device 101 of Figure 2 may include the electronic device 101 of Figure 1. The operations of Figure 2 may be performed by the processor 120 of Figure 1.
[0030] According to one embodiment, the electronic device 101 can identify an image stored in a memory (e.g., memory 130 of FIG. 1). For example, the image can be acquired via a camera included in the electronic device 101 (e.g., camera 140 of FIG. 1). For example, the image can be transmitted from an external electronic device (e.g., a server). For example, the electronic device 101 can identify a plurality of lanes 250 in the image. For example, the plurality of lanes 250 can be separated by lanes 230. The electronic device 101 can train a neural network using truth data based on the vehicles identified in each of the plurality of lanes 250.
[0031] According to an embodiment, the electronic device 101 may identify a vehicle included in one of the multiple roads 250. For example, the electronic device 101 may identify a first vehicle 221 in a first road 251 identified on the left side of the multiple roads 250. The electronic device 101 may identify a first bounding box 241 based on identifying the first vehicle 221. The electronic device 101 may identify a shape of the first vehicle 221 based on identifying the first vehicle 221 in the first road 251. For example, the electronic device 101 may identify a right side and a rear side of the first vehicle 221 within the first bounding box 241. The electronic device 101 may identify a shape of the first vehicle 221 based on identifying the right side and the rear side of the first vehicle 221. The electronic device 101 may obtain a first data set associated with the first vehicle 221 based on identifying the first bounding box 241. The electronic device 101 may train a neural network with truth data based on the first data set. The truth data may be obtained based on the first vehicle 221 and a vehicle license plate identified within the first bounding box 241.
[0032] According to an embodiment, the electronic device 101 can identify vehicle license plates that include one or more texts within the first bounding box 241. The electronic device 101 can train a neural network using the vehicle license plates that include the one or more texts. For example, the electronic device 101 can perform labeling on the one or more texts. For example, this labeling may be referred to as data labeling. For example, the electronic device 101 can train a neural network using the labeling on the one or more texts.
[0033] According to an embodiment, the electronic device 101 can identify the second vehicle 222 in a second lane 252, which is a center lane among the multiple lanes 250. For example, the second lane 252 may be distinguished from the first lane 251 by the first lane 231. The electronic device 101 can identify a second bounding box 242 based on identifying the second vehicle 222. The electronic device 101 can identify a shape of the second vehicle 222 based on identifying the second vehicle 222 in the second lane 252. For example, the electronic device 101 can identify a rear surface of the second vehicle 222 within the second bounding box 242. The electronic device 101 can identify a shape of the second vehicle 222 based on identifying the rear surface of the second vehicle 222. The electronic device 101 can obtain a second data set related to the second vehicle 222 based on identifying the second bounding box 242. The electronic device 101 can train the neural network with truth data based on the second data set. The truth data can be obtained based on the second vehicle 222 and the vehicle license plate identified within the second bounding box 242.
[0034] According to one embodiment, the electronic device 101 may identify vehicle license plates that include one or more pieces of text within the second bounding box 242. The electronic device 101 may train a neural network using the vehicle license plates that include the one or more pieces of text. For example, the electronic device 101 may perform labeling on the one or more pieces of text. The electronic device 101 may train a neural network using the labeling on the one or more pieces of text identified within the second bounding box 242.
[0035] According to an embodiment, the electronic device 101 can identify the third vehicle 223 on the third road 253, which is a road on the right side of the multiple roads 250. For example, the third road 253 may be distinguished from the second road 252 by the second lane 232. The electronic device 101 can identify a third bounding box 243 based on identifying the third vehicle 223. The electronic device 101 can identify a shape of the third vehicle 223 based on identifying the third vehicle 2230 on the third road 253. For example, the electronic device 101 can identify a left side and a rear side of the third vehicle 223 within the third bounding box 243. The electronic device 101 can identify a shape of the third vehicle 223 based on identifying the left side and the rear side of the third vehicle 223. The electronic device 101 may obtain a third data set related to the third vehicle 223 based on identifying the third bounding box 243. The electronic device 101 may train the neural network with truth data based on the third data set. The truth data may be obtained based on the third vehicle 223 and the vehicle license plate identified within the third bounding box 243.
[0036] According to one embodiment, when acquiring the first data set, the second data set, and / or the third data set, the electronic device 101 may create respective bounding boxes 240 of the same size for acquiring each data set. For example, the widths w1, w2, and w3 of the bounding boxes 240 may be 640 pixels. For example, the heights h1, h2, and h3 of the bounding boxes 240 may be 640 pixels.
[0037] For example, each periphery of the bounding box 240 may be spaced the same distance from the center of the vehicle. The electronic device 101 may identify a first bounding box 241 centered at a point C1 that corresponds to the center point of the first vehicle 221 in the image. Similarly, a center point C2 of the second bounding box 242 may correspond to the center point of the second vehicle 222 in the image. Similarly, a center point C3 of the third bounding box 243 may correspond to the center point of the third vehicle 223 in the image.
[0038] For example, the edge of each of the bounding boxes 240 may be formed at a distance of 320 pixels away from the center of the vehicle. The electronic device 101 may identify each of the vehicle license plates within each of the bounding boxes 240. The electronic device 101 may perform labeling on each of the one or more pieces of text identified within the vehicle license plates. Operations related to said labeling are described below in FIG. 3a.
[0039] As described above, according to one embodiment, the electronic device 101 can train a neural network using truth data based on the morphology of the vehicle identified on each of the multiple vehicle lanes 250. The electronic device 101 can train a neural network for accurately determining the location of the vehicle by training the neural network using truth data based on the morphology of the vehicle identified on each of the multiple vehicle lanes 250. In FIG. 3a below, the operation of labeling each of one or more texts included in the vehicle license plate identified within the bounding box is described below.
[0040] FIG. 3a is an exemplary diagram illustrating information identified by an electronic device for training a neural network, according to one embodiment.
[0041] The operations of the electronic device 101 described with reference to FIG. 3a may be performed by the electronic device 101 of FIG. 1, FIG. 2, and / or the processor 120 of FIG.
[0042] Referring to FIG. 3a, according to one embodiment, the electronic device 101 may identify a bounding box (e.g., the first bounding box 241, the second bounding box 242, and / or the third bounding box 243 of FIG. 2) corresponding to a vehicle identified in an image 300. The electronic device 101 may identify a region 310 within the bounding box corresponding to a vehicle license plate. The electronic device 101 may identify one or more pieces of text 320 within the region 310. The electronic device 101 may perform labeling on each of the one or more pieces of text 320 based on identifying the one or more pieces of text. This labeling may be referred to as data labeling.
[0043] For example, the electronic device 101 can identify a first sub-region 331 corresponding to the first text 321. The electronic device 101 can label the first sub-region 331 with a first tag. For example, the electronic device 101 can identify a second sub-region 332 corresponding to the second text 322. For example, the electronic device 101 can label the second sub-region 332 with a second tag. The electronic device 101 can identify a third sub-region 333 corresponding to the third text 323. The electronic device 101 can label the third sub-region 333 with a third tag. For example, the electronic device 101 can identify a fourth sub-region 334 corresponding to the fourth text 324. The electronic device 101 can label the fourth sub-region 334 with a fourth tag. For example, the electronic device 101 can identify a fifth sub-region 335 corresponding to the fifth text 325. The electronic device 101 may label the fifth sub-region 335 with a fifth tag. The electronic device 101 may identify a sixth sub-region 336 corresponding to the sixth text 326. The electronic device 101 may label the sixth sub-region 336 with a sixth tag. For example, the electronic device 101 may identify a seventh sub-region 337 corresponding to the seventh text 327. The electronic device 101 may label the seventh sub-region 337 with a seventh tag. For example, the electronic device 101 may identify an eighth sub-region 338 corresponding to the eighth text 328. The electronic device 101 may label the eighth sub-region 338 with an eighth tag. The one or more texts 320 may include human readable characters.
[0044] According to one embodiment, the electronic device 101 can train a neural network using the sub-regions 330 labeled with each of the sub-regions 330 corresponding to one or more texts 320. The electronic device 101 can train the neural network using truth data including the sub-regions 330.
[0045] As mentioned above, according to one embodiment, the electronic device 101 can train a neural network using the sub-region 330 included in the image 300. By training a neural network using the sub-region 330, the electronic device 101 can identify external objects having a specified number of pixels or less in the image when the neural network is executed.
[0046] FIG. 3b is an exemplary diagram illustrating information identified by an electronic device to train a neural network, according to one embodiment.
[0047] The operations of the electronic device 101 described with reference to Fig. 3b may be performed by the electronic device 101 of Fig. 1, Fig. 2, and / or Fig. 3a, and / or the processor 120 of Fig. 1. With reference to Fig. 3b, the operations of the electronic device 101 in relation to one bounding box 340 in a set 360 of bounding boxes for vehicle license plates are described. The embodiments are not limited in this respect, and the electronic device 101 may perform similar operations for other bounding boxes in the set 360.
[0048] Referring to FIG. 3b, according to an embodiment, the electronic device 101 can identify a bounding box 340 including a vehicle license plate in an image. The electronic device 101 can identify vertices of the bounding box 340. The electronic device 101 can identify a first vertex 341 at the top left of the bounding box 340. The electronic device 101 can identify a first coordinate value in the image of the first vertex 341. The electronic device 101 can identify a second vertex 342 at the bottom right of the bounding box 340. The electronic device 101 can identify a second coordinate value in the image of the second vertex 342. The electronic device 101 can identify a vehicle type of a visual object including the vehicle license plate based on the first coordinate value and the second coordinate value. For example, the electronic device 101 can identify a vehicle type corresponding to the visual object based on a position of the vehicle license plate in the image. The above-mentioned operations can be performed by a processor of the electronic device 101 during identifying the vehicle license plate shown in FIG. 3a.
[0049] According to an embodiment, the electronic device 101 can identify a vehicle license plate using images acquired based on multiple cameras. For example, the electronic device 101 can identify a vehicle license plate based on images acquired at different angles. For example, the electronic device 101 can identify the vehicle license plate based on a projection technique such as homography. Based on identifying the vehicle license plate by the projection technique, the electronic device 101 can identify a bounding box 340 that includes the vehicle license plate.
[0050] According to one embodiment, the electronic device 101 can identify a bounding box 340 that includes the vehicle license plate in the image. The electronic device 101 can identify one or more texts 351, 353 and / or visual objects 352 within the bounding box 340. For example, the one or more texts 351, 353 can include a first text 351 of a first set for distinguishing a region. For example, the first set of texts 351 can include text for indicating a registration region of a vehicle to which the vehicle license plate is attached. For example, the one or more texts 351, 353 can include a second set of texts 353 that includes an identification number for distinguishing a vehicle. For example, the second set of texts 353 can include, but are not limited to, alphabets and / or numbers.
[0051] According to an embodiment, the electronic device 101 can identify characters of the first text 351 of the first set. Based on identifying the characters of the first text 351, the electronic device 101 can identify that the first text 351 represents a registration region of the vehicle. According to an embodiment, the electronic device 101 can identify characters of the second text 353 of the second set. Based on identifying the characters of the second text 353, the electronic device 101 can identify that the second text 353 represents an identification number for distinguishing the vehicle.
[0052] According to one embodiment, the electronic device 101 may identify visual objects 352 within the bounding box 340. For example, the visual objects 352 may include visual objects to represent a region in which the vehicle is registered. For example, the visual objects 352 may include natural objects, such as animals and / or plants. For example, the visual objects 352 may include man-made objects, such as, but not limited to, shields and / or helmets. For example, the visual objects 352 may include a combination of man-made and natural objects.
[0053] According to an embodiment, the electronic device 101 may perform labeling on each of the one or more texts 351, 353 and / or visual objects 352 included in the bounding box 340. For example, the electronic device 101 may label each of the one or more texts 351, 353 with a tag. The operation of labeling the tag may be substantially the same as the operation of FIG. 3a. For example, the electronic device 101 may perform labeling using a neural network model. The neural network model may be referred to as the neural network of FIG. 3a.
[0054] According to an embodiment, the electronic device 101 can generate an image including a vehicle license plate using a first neural network model. For example, the electronic device 101 can generate an image similar to the vehicle license plate included in the bounding box 340 based on inputting the bounding box 340 in the image to the first neural network model. For example, the first neural network model can include a neural network model for generating an image, such as a conditional generative adversarial nets (CGAN).
[0055] According to one embodiment, the electronic device 101 can input an image output through a first neural network model to a second neural network model different from the first neural network model. The electronic device 101 can train the second neural network model by inputting the image to the second neural network model. For example, the second neural network model can include a neural network model for identifying vehicle license plates in an image.
[0056] As described above, according to one embodiment, the electronic device 101 can use images acquired based on a first neural network model to train a second neural network model. The electronic device 101 can acquire images for training the second neural network model by acquiring the images based on the first neural network model.
[0057] 4a to 4c are diagrams for explaining an example of an operation of the electronic device 101 according to an embodiment performed to identify a reference surface. The operation of the electronic device 101 described with reference to Figs. 4a to 4c may be performed by the electronic device 101 of Figs. 1, 2, 3a, and / or 3b. The operation of Figs. 4a to 4c may be performed by the processor 120 of Fig. 1.
[0058] Referring to FIG. 4a, according to an embodiment, the electronic device 101 may include a camera (e.g., the camera 140 in FIG. 1) and a sensor (e.g., the sensor 150 in FIG. 1). For example, the electronic device 101 may use the sensor to identify a direction in which the camera is facing. For example, in FIG. 4a, the camera of the electronic device 101 may face in a +z-axis direction. For example, the +z-axis direction may be a first direction. For example, the +x-axis direction may be a second direction. For example, the +y-axis direction may be a third direction. For example, the first direction, the second direction, and the third direction may be perpendicular to each other. The electronic device 101 may identify a reference plane in the image 410 based on the direction of the camera. For example, the electronic device 101 may identify a focal length of the camera. The electronic device 101 may identify a reference point P for identifying the reference plane based on the focal length of the camera. For example, the reference point may be called a vanishing point. The electronic device 101 can identify a height of a window for dividing the image 410 based on a reference point P. For example, the electronic device 101 can identify the height of the window based on the reference point P and the height of the image 410. In FIG. 4a, the coordinates of the reference point P may be (x1, y1, z1). For example, the electronic device 101 can identify the height of the window based on the following equation 1:
number
[0059] According to one embodiment, the electronic device 101 can obtain the height of the image 410 based on the focal length and the reference point P. For example, the electronic device 101 can identify the height of the image 410 based on the following equation 2:
number
[0060] 4b, according to one embodiment, the electronic device 101 may identify a plurality of visual objects 420 in the image 410. The electronic device 101 may identify an area in the image 410 where a specified number or more of the visual objects 420 are identified. For example, the electronic device 101 may identify an area 430 for dividing the image 410 based on identifying a specified number or more of the visual objects 420. The electronic device 101 may divide the area 430 into a plurality of sub-areas using the window.
[0061] For example, the electronic device 101 can identify a plurality of visual objects 420 based on a neural network for image segmentation such as U-net in the region 430. The electronic device 101 can identify a region corresponding to the sky in the image 410 based on identifying the plurality of visual objects 420. The electronic device 101 can identify a visual object 440 corresponding to a reference plane (e.g., the ground) based on identifying the region corresponding to the sky. The electronic device 101 can identify a window height based on the region including the visual object 440 corresponding to the reference plane. The electronic device 101 can identify a window width based on the FOV of the image 410. For example, the window width can be obtained based on the FOV. For example, the window width can be obtained based on Equations 3 to 5 described below.
number
number
number
[0062] According to an embodiment, the electronic device 101 can use the window to divide an area including the visual object 440 corresponding to a reference plane into a plurality of sub-areas based on a width and a height of the window. The electronic device 101 can sequentially input the plurality of sub-areas into the neural network. The electronic device 101 can identify an external object in the sub-area based on the plurality of sub-areas input sequentially.
[0063] Referring to FIG. 4c, according to one embodiment, the electronic device 101 may perform image processing on the image 410. This image processing may be referred to as pyramid image processing. For example, the pyramid image processing may include a Gaussian pyramid, a Laplacian pyramid, and / or a steerable pyramid. For example, the electronic device 101 may perform image processing on the image 410 based on a morphological operation. The morphological operation may include erosion and / or dilation.
[0064] The electronic device 101 may obtain an image 415 based on image processing of the image 410. For example, the electronic device 101 may change colors of a plurality of visual objects identified in the image 410. For example, the electronic device 101 may change a first portion corresponding to a plurality of visual objects in the image 410 to a first color (e.g., black). The electronic device 101 may obtain an image 415 in which the first portion corresponding to the plurality of visual objects is changed to the first color. The electronic device 101 may change a second portion different from the first portion to a second color (e.g., white). The electronic device 101 may obtain an image 415 in which the colors of the first portion and the second portion are changed. The electronic device 101 may identify a reference surface in the image 415 based on a ratio of the first color being identified at or above a specified ratio in pixels arranged in a third direction of the image 415. For example, the electronic device 101 may identify an area for dividing the image 415 based on identifying the reference surface. The electronic device 101 can use the window to divide the region.
[0065] As described above, according to one embodiment, the electronic device 101 may segment an image based on a window. The electronic device 101 may sequentially input the image segmented into a plurality of sub-regions to a neural network. The electronic device 101 may identify an external object within the plurality of sub-regions based on inputting the plurality of sub-regions to the neural network. The electronic device 101 may reduce processor usage by identifying an external object based on the segmented image. The electronic device 101 may identify an external object within the sub-regions by identifying an external object based on the segmented image.
[0066] 5a-5c are diagrams illustrating the operation of an electronic device 101 for identifying an external object, according to one embodiment. The electronic device 101 of FIGS. 5a-5d may include the electronic device 101 of FIGS. 1, 2, 3a, 3b, 4a, 4b, and / or 4c. The operations of FIGS. 5a-5d may be performed by the processor 120 of FIG. 1. The operations of FIGS. 5a-5d may include identifying the external object using a neural network.
[0067] 5a-5b, according to an embodiment, the electronic device 101 may acquire an image 500 via a camera (e.g., the camera 140 in FIG. 1). According to an embodiment, the electronic device 101 may identify a width and a height of a window for dividing the image 500. The width and height of the window may be identified based on the operations of FIGS. 4a-4c. The electronic device 101 may acquire a plurality of partial regions 520 in a first region 510 using the window based on the width and height. For example, the partial regions 520 may be spaced apart by a first interval 531. For example, the first interval 531 may include an interval between first edges (e.g., left edges) of the partial regions 520. For example, the first interval 531 may include an interval between second edges (e.g., right edges) of the partial regions 520. For example, the first interval 531 may include an interval between center points of the partial regions 520. For example, the first interval 531 may be an interval corresponding to half the width of the window. For example, the first sub-region 521, the second sub-region 522, the third sub-region 523, the fourth sub-region 524, the fifth sub-region 525, the sixth sub-region 526, and / or the seventh sub-region 527 may be of the same size. The number of sub-regions 520 is not limited. According to an embodiment, the electronic device 101 can sequentially input the first sub-region 521 to the seventh sub-region 527 into the neural network. The electronic device 101 can identify an external object 501 included in at least one of the sub-regions 520 based on the sub-regions 520 input into the neural network.
[0068] According to one embodiment, the electronic device 101 can input the multiple regions 520 to the neural network. The electronic device 101 can identify the external object 501 within the multiple regions 520 based on inputting the multiple regions 520 to the neural network.
[0069] Referring to Fig. 5b, according to an embodiment, the electronic device 101 can input the sub-region 520 to the neural network. The electronic device 101 can identify whether the external object 501 is included in the first sub-region 521 from the neural network. The electronic device 101 can identify the first interval 531 and / or the second interval 532 based on whether the external object 501 is included in the first sub-region 521. For example, the electronic device 101 can identify the external object 501 in the first sub-region 521. The electronic device 101 can identify the second interval 532 based on identifying the external object 501 in the first sub-region 521. The electronic device 101 can obtain the fourth sub-region 524 based on the second interval 532. For example, the electronic device 101 may identify a second interval 532 corresponding to the sub-areas (e.g., the first sub-area 521, the second sub-area 522, and the third sub-area 523) in which the external object 501 is identified based on identifying the first sub-area 521. The electronic device 101 may obtain a fourth sub-area 524 separated by the second interval 532 from the first sub-area 521 based on identifying the second interval 532. The electronic device 101 may input the fourth sub-area 524 to the neural network based on obtaining the fourth sub-area 524.
[0070] 5c-5d, according to one embodiment, the electronic device 101 can identify a speed of the electronic device 101 based on a sensor (e.g., the sensor 150 of FIG. 1). The electronic device 101 can change a spacing between partial regions (e.g., the partial regions 520 of FIG. 5a and / or FIG. 5b) based on the speed. For example, the electronic device 101 can change a spacing between some of the partial regions based on the speed. According to one embodiment, the electronic device 101 can change a spacing between partial regions 540 in a region corresponding to a plurality of roads 250 based on the speed of the electronic device 101.
[0071] According to an embodiment, the electronic device 101 may be formed by overlapping a plurality of partial regions 520. The electronic device 101 may overlap a plurality of partial regions 520 based on a width of a window. For example, the electronic device 101 may overlap by a first ratio of the width of the window. For example, the first ratio may include about 50%. For example, the first ratio may be expressed as, but is not limited to, "1 / 2".
[0072] According to one embodiment, the electronic device 101 may identify a first speed of the electronic device 101 that is less than or equal to a first reference speed. For example, the first reference speed may include approximately 40 km / h. Based on identifying the first speed that is less than or equal to the first reference speed, the electronic device 101 may overlap the plurality of partial regions in a first sub-region that includes an edge of the image by a second ratio that is smaller than the first ratio. For example, the second ratio may be substantially equal to the square of the first ratio. For example, if the first ratio is provided as "(1 / n)", the second ratio may be provided as "(1 / n) m". For example, when n=m=2, the first ratio may be 1 / 2 and the second ratio may be 1 / 4. In the above example, the number of partial regions superimposed on the sub-region may differ depending on whether the speed of the electronic device 101 and / or the vehicle including the electronic device 101 is equal to or greater than a first reference speed of about 40 km / h or less. For example, when the speed of the electronic device 101 is equal to or greater than the first reference speed, the partial region may be superimposed on the first sub-region according to the first ratio (=1 / 2). For example, when the speed of the electronic device 101 is equal to or less than the first reference speed, the partial region may be superimposed on the first sub-region according to the second ratio (=1 / 4).
[0073] According to one embodiment, the electronic device 101 may identify a second speed of the electronic device 101 that exceeds a second reference speed. For example, the second reference speed may include approximately 60 km / h. Based on identifying the second speed that exceeds the second reference speed, the electronic device 101 may overlap the plurality of partial regions in a second sub-region that includes an edge of the image by a second ratio that is smaller than the first ratio. For example, the second ratio may be substantially equal to a square of the first ratio. For example, if the first ratio is provided as "(1 / n)", the second ratio may be provided as "(1 / n) m " may be provided.
[0074] Referring to FIG. 5c, according to an embodiment, the electronic device 101 can identify a speed of the electronic device 101 based on the sensor. The electronic device 101 can change the interval between the plurality of partial regions corresponding to the specified lane of the plurality of lanes 250 based on the speed. For example, the electronic device 101 can identify an external object using the partial regions 541, 542 in the sub-region including the second lane 252. For example, the electronic device 101 can identify a second speed exceeding a second reference speed. For example, the second reference speed can include about 60 km / h. The electronic device 101 can change the interval between the plurality of regions. For example, the electronic device 101 can change the interval between the first partial region 541 and the second partial region 542 to a first interval 533. The electronic device 101 can identify an external object within the plurality of partial regions of the changed interval.
[0075] Referring to FIG. 5d, according to an embodiment, the electronic device 101 may identify a speed of the electronic device 101 based on the sensor. The electronic device 101 may change an interval between a plurality of partial regions corresponding to a designated lane among the plurality of lanes 250 based on the speed. For example, the designated lane may include a first lane 251 and a third lane 253. The electronic device 101 may identify an external object using a partial region 543, 544 in a sub-region 540 including a left edge or a right edge. For example, the electronic device 101 may identify a first speed that is equal to or less than a first reference speed. For example, the first reference speed may include about 60 km / h. The electronic device 101 may change an interval between the plurality of regions. For example, the electronic device 101 may change an interval between a first partial region 543 and a second partial region 544 to a second interval 534. The electronic device 101 may identify an external object within the plurality of partial regions of the changed interval. The first spacing 533 in FIG. 5c and the second spacing 534 in FIG. 5d may be substantially the same.
[0076] As described above, according to one embodiment, the electronic device 101 can identify an external object based on a sub-region of an image input to the neural network. The electronic device 101 can sequentially input a plurality of sub-regions in an image to the neural network. The electronic device 101 can identify an external object in the sub-region based on the sequentially input sub-regions. If the electronic device 101 identifies an external object in the sub-region, it can input a different sub-region that does not include the sub-region to the neural network. By performing the above-described operations, the electronic device 101 can reduce processor usage.
[0077] 6a-6c show an example of operations of the electronic device performed to identify an external object. The operations of the electronic device 101 performed with reference to Figs. 6a-6c may be performed by the electronic device 101 of Figs. 1, 2, 3a, 3b, 4a, 4b, 4c, 5a, 5b, 5c, and / or 5d. The operations of Figs. 6a-6c may be performed by the processor 120 of Fig. 1. The operations of Figs. 6a-6c may include operations of identifying the external object using a neural network.
[0078] 6a-6c, according to an embodiment, the electronic device 101 may acquire an image 610 via a camera (e.g., the camera 140 in FIG. 1). For example, the image 610 may include an image of a different size from an image used for input to a neural network. The electronic device 101 may resize the image 610 with a size corresponding to the size of the image used for input to the neural network. For example, the electronic device 101 may identify an external object based on inputting a sub-region having a first size to the neural network. The electronic device 101 may acquire a sub-region 620 having a size larger than the sub-region used for the neural network. The electronic device 101 may resize the sub-region 620 based on acquiring the sub-region 620. For example, the electronic device 101 may resize the sub-region 620 based on image processing of the sub-region 620. For example, the electronic device 101 may perform down-scaling and / or up-scaling on the sub-region 620. The electronic device 101 may change the size of the sub-region 620 based on the downscaling and / or upscaling.
[0079] According to an embodiment, the electronic device 101 may execute a neural network using the sub-region 620 including the vertex at the top left corner of the first region 615 based on obtaining the sub-region 620 having a size larger than the sub-region used for the neural network. For example, the electronic device 101 may execute an operation of identifying an external object from the top left corner of the first region 615 to the top right corner of the first region 615.
[0080] Referring to FIG. 6b, according to an embodiment, the electronic device 101 may obtain a sub-region 630 in the image 610. The electronic device 101 may identify that the size of the sub-region 630 is larger than the size required for the neural network. The electronic device 101 may reduce the size of the sub-region 630 based on identifying the size of the sub-region 630 being larger than the size required for the neural network. For example, the electronic device 101 may reduce the size of the sub-region 630 based on downscaling the sub-region 630. The electronic device 101 may obtain a sub-region 635 that is a reduced size of the sub-region 630. The electronic device 101 may input the sub-region 635 processed by the downscaling to the neural network. The electronic device 101 may identify an external object in the sub-region 635 based on inputting the sub-region 635 to the neural network.
[0081] Referring to FIG. 6c, according to an embodiment, the electronic device 101 may obtain a sub-region 640 in the image 610. The electronic device 101 may identify that the size of the sub-region 640 is smaller than the size required for the neural network. The electronic device 101 may increase the size of the sub-region 640 based on identifying the size of the sub-region 640 being smaller than the size required for the neural network. For example, the electronic device 101 may increase the size of the sub-region 640 based on upscaling the sub-region 640. The electronic device 101 may obtain a sub-region 645 in which the size of the sub-region 640 is increased. The electronic device 101 may input the sub-region 645 processed by the upscaling to the neural network. The electronic device 101 may identify an external object in the sub-region 645 based on inputting the sub-region 645 to the neural network.
[0082] As described above, according to one embodiment, the electronic device 101 can identify an external object based on a sub-region of an image input to a neural network. For example, the neural network may include a neural network for identifying an external object using a sub-region of an image. The electronic device 101 can change the size of the sub-region of the image for input to the neural network. The electronic device 101 can identify an external object in images of various sizes by inputting the sub-region having the changed size to the neural network. The electronic device 101 can identify an external object having a relatively small size by using the neural network to identify the external object. The electronic device 101 can reduce processor usage by identifying the external object based on the sub-region.
[0083] FIG. 7 illustrates an example block diagram of an autonomous driving system for a vehicle, according to one embodiment.
[0084] The vehicle autonomous driving system 700 according to FIG. 7 may be a deep learning network including a sensor 703, an image preprocessor 705, a deep learning network 707, an artificial intelligence (AI) processor 709, a vehicle control module 711, a network interface 713, and a communication unit 715. In various embodiments, each element may be connected via various interfaces. For example, sensor data sensed and output by the sensor 703 may be fed to the image preprocessor 705. The sensor data processed by the image preprocessor 705 may be fed to the deep learning network 707 run by the AI processor 709. The output of the deep learning network 707 run by the AI processor 709 may be fed to the vehicle control module 711. An intermediate result of the deep learning network 707 run by the AI processor 709 may be fed to the AI processor 709. In various embodiments, the network interface 713 communicates with an in-vehicle electronic device to transmit autonomous driving route information and / or autonomous driving control commands for autonomous driving of the vehicle to the internal block configuration. In one embodiment, the network interface 713 may be used to transmit sensor data acquired via the sensor 703 to an external server. In some embodiments, the autonomous driving control system 700 may include additional or fewer components as appropriate. For example, in some embodiments, the image preprocessor 705 may be an optional component. In another example, a post-processing component (not shown) may be included in the autonomous driving control system 700 to perform post-processing on the output of the deep learning network 707 before the output is provided to the vehicle control module 711.
[0085] In some embodiments, the sensor 703 may include one or more sensors. In various embodiments, the sensor 703 may be mounted at different locations on the vehicle. The sensor 703 may be oriented in one or more different directions. For example, the sensor 703 may be mounted on the front, sides, rear, and / or roof of the vehicle to be oriented in a forward-facing, rear-facing, side-facing, etc. direction. In some embodiments, the sensor 703 may be an image sensor, such as high dynamic range cameras. In some embodiments, the sensor 703 includes non-visual sensors. In some embodiments, the sensor 703 includes RADAR, LiDAR (Light Detection And Ranging), and / or ultrasonic sensors in addition to the image sensor. In some embodiments, the sensor 703 is not mounted on the vehicle with the vehicle control module 711. For example, sensors 703 may be included as part of the deep learning system to capture sensor data and may be mounted in the environment or road and / or mounted in surrounding vehicles.
[0086] In some embodiments, an image pre-processor 705 may be used to pre-process the sensor data of the sensor 703. For example, the image pre-processor 705 may be used to pre-process the sensor data, to split the sensor data into one or more components, and / or to post-process the one or more components. In some embodiments, the image pre-processor 705 may be a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a specialized image processor. In various embodiments, the image pre-processor 705 may be a tone-mapper processor for processing high dynamic range data. In some embodiments, the image pre-processor 705 may be a component of the AI processor 709.
[0087] In some embodiments, the deep learning network 707 may be a deep learning network for implementing control instructions for controlling an autonomous vehicle. For example, the deep learning network 707 may be an artificial neural network, such as a convolutional neural network (CNN), trained using sensor data, and the output of the deep learning network 707 is provided to the vehicle control module 711.
[0088] In some embodiments, the artificial intelligence (AI) processor 709 may be a hardware processor for running the deep learning network 707. In some embodiments, the AI processor 709 is a specialized AI processor for performing inference via a convolutional neural network (CNN) on the sensor data. In some embodiments, the AI processor 709 may be optimized for the bit depth of the sensor data. In some embodiments, the AI processor 709 may be optimized for deep learning operations such as neural network operations including convolution, dot product, vector, and / or matrix operations. In some embodiments, the AI processor 709 may be implemented via multiple graphics processing units (GPUs) capable of efficiently performing parallel processing.
[0089] In various embodiments, the AI processor 709 can be coupled via an input / output interface to a memory configured to provide the AI processor with instructions that, while the AI processor 709 is running, cause the AI processor 709 to perform deep learning analysis on the sensor data received from the sensors 703 and determine the results of the machine learning used to operate the vehicle at least partially autonomously. In some embodiments, a Vehicle Control Module 711 can be used to process instructions for vehicle control output from the Artificial Intelligence (AI) processor 709 and translate the output of the AI processor 709 into instructions for controlling the various modules of the vehicle to control each of the modules of the vehicle. In some embodiments, the Vehicle Control Module 711 is used to control the vehicle for autonomous driving. In some embodiments, the Vehicle Control Module 711 can adjust the steering and / or speed of the vehicle. For example, the Vehicle Control Module 711 can be used to control the driving of the vehicle, such as deceleration, acceleration, steering, lane changing, lane keeping, etc. In some embodiments, the vehicle control module 711 can generate control signals to control vehicle lighting, such as brake lights, turn signals, headlights, etc. In some embodiments, the vehicle control module 711 can be used to control vehicle audio-related systems, such as the vehicle's sound system, the vehicle's audio warnings, the vehicle's microphone system, the vehicle's horn system, etc.
[0090] In some embodiments, the vehicle control module 711 can be used to control notification systems, including warning systems to inform passengers and / or drivers of access to an intended destination or of a driving event, such as a potential collision. In some embodiments, the vehicle control module 711 can be used to adjust sensors, such as the sensor 703, of the vehicle. For example, the vehicle control module 711 can modify the orientation of the sensor 703, change the output resolution and / or format type of the sensor 703, increase or decrease the capture rate, adjust the dynamic range, adjust the focus of the camera. Additionally, the vehicle control module 711 can turn on / off the operation of the sensors, individually or collectively.
[0091] In some embodiments, the vehicle control module 711 can be used to modify parameters of the image preprocessor 705, such as by changing the range of filter frequencies, adjusting edge detection parameters for feature and / or object detection, adjusting channels and bit depth, etc. In various embodiments, the vehicle control module 711 may be used to control autonomous driving of the vehicle and / or Driver assistance functions of the vehicle.
[0092] In some embodiments, the network interface 713 may be responsible for an internal interface between the block configuration of the autonomous driving control system 700 and the communication unit 715. Specifically, the network interface 713 may be a communication interface for receiving and / or transmitting data including voice data. In various embodiments, the network interface 713 may connect to an external server via the communication unit 715 to connect a voice call, receive and / or transmit a text message, transmit sensor data, update software of the vehicle in the autonomous driving system, or update software of the autonomous driving system of the vehicle.
[0093] In various embodiments, the communication unit 715 can include various wireless interfaces, such as cellular or WiFi. For example, the network interface 713 can be used to receive updates to operational parameters and / or instructions for the sensor 703, the image preprocessor 705, the deep learning network 707, the AI processor 709, and the vehicle control module 711 from an external server connected via the communication unit 715. For example, the machine learning model of the deep learning network 707 can be updated using the communication unit 715. According to yet another example, the communication unit 715 can be used to update operational parameters of the image preprocessor 705, such as image processing parameters, and / or firmware of the sensor 703.
[0094] In another embodiment, the communication unit 715 can be used to activate communications for emergency services and emergency contact in the event of an accident or near-accident. For example, in a crash event, the communication unit 715 can be used to call emergency services for assistance and to notify the emergency services of the crash details and vehicle location to the outside world. In various embodiments, the communication unit 715 can update or obtain an expected time of arrival and / or a destination location.
[0095] According to an embodiment, the autonomous driving system 700 shown in Fig. 7 may be configured in an electronic device of a vehicle. According to an embodiment, when an autonomous driving cancellation event occurs from a user during autonomous driving of the vehicle, the AI processor 709 of the autonomous driving system 700 can control the autonomous driving software of the vehicle to learn by controlling the input of autonomous driving cancellation event related information into the training set data of the deep learning network.
[0096] 8 and 9 show an example of a block diagram illustrating an autonomous vehicle according to an embodiment. Referring to FIG. 8, an autonomous vehicle 800 according to the present embodiment may include a control device 900, sensing modules 804a, 804b, 804c, and 804d, an engine 806, and a user interface 808.
[0097] The autonomous vehicle 800 may have an autonomous mode or a manual mode. As an example, the autonomous vehicle 800 may switch from the manual mode to the autonomous mode or from the autonomous mode to the manual mode according to user input received via the user interface 808.
[0098] When the vehicle 800 is operated in an autonomous driving mode, the autonomous driving vehicle 800 can operate under the control of the control device 900.
[0099] In this embodiment, the control device 900 may include a controller 920 including a memory 922 and a processor 924 , a sensor 910 , a communication device 930 , and an object detection device 940 .
[0100] Here, the object detection device 940 can perform all or part of the functions of a distance measurement device (eg, the electronic device 101).
[0101] That is, in this embodiment, the object detection device 940 is a device for detecting objects located outside the moving body 800, and the object detection device 940 can detect objects located outside the moving body 800 and generate object information according to the detection results.
[0102] The object information can include information regarding the presence or absence of an object, position information of the object, distance information between the moving body and the object, and relative speed information between the moving body and the object.
[0103] The objects may include various objects disposed outside the moving body 800, such as lanes, other vehicles, pedestrians, traffic signals, lights, roads, structures, speed limiters, topographical features, and animals. Here, the traffic signal may be a concept including traffic lights, traffic signs, and patterns or texts painted on the road surface. And the light may be light generated from lamps provided on other vehicles, light generated by street lamps, or sunlight.
[0104] And the structures may be objects located around the road and fixed to the ground. For example, the structures may include street lights, roadside trees, buildings, utility poles, traffic lights, bridges. The terrain objects may include mountains, hills, etc.
[0105] Such an object detection device 940 may include a camera module, and the controller 920 may extract object information from an external image captured by the camera module and process the information.
[0106] The object detection device 940 may further include an imaging device for recognizing the external environment. In addition to LIDAR, RADAR, GPS devices, odometry and other computer vision devices, ultrasonic sensors, infrared sensors, etc. may be used, which may be selected or operated simultaneously as needed to enable more accurate sensing.
[0107] On the other hand, a distance measuring device according to one embodiment of the present invention can calculate the distance between an autonomous vehicle 800 and an object, and control the operation of the vehicle based on the calculated distance in cooperation with a control device 900 of the autonomous vehicle 800.
[0108] As an example, when there is a possibility of a rear-end collision depending on the distance between the autonomous traveling vehicle 800 and the object, the autonomous traveling vehicle 800 can control the brakes to reduce or stop the vehicle. As another example, when the object is a moving object, the autonomous traveling vehicle 800 can control the traveling speed of the autonomous traveling vehicle 800 to maintain a predetermined distance or more from the object.
[0109] The distance measurement device according to one embodiment of the present invention can be configured as one module in the control device 900 of the autonomous vehicle 800. That is, the memory 922 and the processor 924 of the control device 900 can realize the collision prevention method according to the present invention in software.
[0110] Also, the sensor 910 can acquire various sensing information by connecting the internal / external environment of the moving object to the sensing modules 804a, 804b, 804c, and 804d. Here, the sensor 910 can include an attitude sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor, a collision sensor, a wheel sensor, a speed sensor, a tilt sensor, a weight detection sensor, a heading sensor, a gyro sensor, a position module, a moving object forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on the rotation of a steering wheel, a moving object internal temperature sensor, a moving object internal humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.
[0111] As a result, the sensor 910 can acquire sensing signals for mobile body posture information, mobile body collision information, mobile body direction information, mobile body position information (GPS information), mobile body angle information, mobile body speed information, mobile body acceleration information, mobile body inclination information, mobile body forward / reverse information, battery information, fuel information, tire information, mobile body lamp information, mobile body internal temperature information, mobile body internal humidity information, steering wheel rotation angle, mobile body external illuminance, pressure applied to the accelerator pedal, pressure applied to the brake pedal, etc.
[0112] In addition, the sensor 910 may further include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), and the like.
[0113] In this manner, the sensor 910 can generate mobile object status information based on the sensing data.
[0114] The wireless communication device 930 is configured to perform wireless communication between the autonomously traveling mobile body 800. For example, the wireless communication device 930 enables the autonomously traveling mobile body 800 to communicate with a user's mobile phone, another wireless communication device 930, another mobile body, a central device (traffic control device), a server, and the like. The wireless communication device 930 can transmit and receive wireless signals according to a connection wireless protocol. The wireless communication protocols are Wi-Fi, Bluetooth, LTE (Long-Term Evolution), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), and GSM (Global Systems for Mobile Communications), and the communication protocol is not limited thereto.
[0115] In addition, the autonomous traveling mobile body 800 in this embodiment can also realize communication between mobile bodies via the wireless communication device 930. That is, the wireless communication device 930 can communicate with other mobile bodies on the road and other mobile bodies by vehicle-to-vehicle (V2V) communication. The autonomous traveling mobile body 800 can transmit and receive information such as driving warnings and traffic information by vehicle-to-vehicle communication, and can also request information from other mobile bodies or receive requests. For example, the wireless communication device 930 can perform V2V communication by a short-range communication (DSRC, dedicated short-range communication) device or a C-V2V (Cellular-V2V) device. In addition to communication between vehicles, communication between vehicles and other objects (for example, electronic devices carried by pedestrians, etc.) (V2X, Vehicle to Everything communication) can also be realized via the wireless communication device 930.
[0116] In addition, the wireless communication device 930 can acquire information generated by various mobilities including infrastructure located on the road (traffic signals, CCTV, RSU, eNode B, etc.) or other autonomous driving / non-autonomous driving vehicles, via a non-terrestrial network rather than a terrestrial network, as information for performing autonomous driving of the autonomous driving mobile body 800.
[0117] For example, the wireless communication device 930 can perform wireless communication with a Low Earth Orbit (LEO) satellite system, a Medium Earth Orbit (MEO) satellite system, a Geostationary Orbit (GEO) satellite system, a High Altitude Platform (HAP) system, and the like, which constitute a non-terrestrial network, via an antenna dedicated to the non-terrestrial network mounted on the autonomous mobile body 800.
[0118] For example, the wireless communication device 930 may be a 5G NR NTN (5G NR NTN) currently being discussed by 3GPP and the like. th In accordance with wireless connection standards that comply with the NTN (New Generation New Radio Non-Terrestrial Network) standard, wireless communication can be performed with various platforms that make up NTN, but is not limited to this.
[0119] In this embodiment, the controller 920 can select a platform that can appropriately perform NTN communication, taking into account various information such as the position of the autonomous vehicle 800, the current time, and available power, and control the wireless communication device 930 to perform wireless communication with the selected platform.
[0120] In this embodiment, the controller 920 is a unit that controls the overall operation of each unit in the moving body 800, and may be configured by the manufacturer of the moving body at the time of manufacture, or may be further configured to execute an autonomous driving function after manufacture. Alternatively, the controller 920 may include a configuration for executing a continuous additional function through an upgrade of the controller 920 configured at the time of manufacture. Such a controller 920 may be called an ECU (Electronic Control Unit).
[0121] The controller 920 can collect various data from the connected sensors 910, the object detection device 940, the communication device 930, etc., and transmit control signals based on the collected data to the sensors 910, the engine 806, the user interface 808, the communication device 930, the object detection device 940, and other components in the vehicle. In addition, although not shown, the controller 920 can also transmit control signals to an acceleration device, a braking system, a steering device, or a navigation device related to the traveling of the vehicle.
[0122] In this embodiment, the controller 920 can control the engine 806, for example, by detecting the speed limit of the road on which the autonomous vehicle 800 is traveling, and controlling the engine 806 so that the traveling speed does not exceed the speed limit, or by controlling the engine 806 to accelerate the traveling speed of the autonomous vehicle 800 within a range that does not exceed the speed limit.
[0123] In addition, when the autonomous traveling vehicle 800 approaches a lane or deviates from a lane during the traveling of the autonomous traveling vehicle 800, the controller 920 can determine whether such approach and departure from the lane correspond to a normal traveling situation or other traveling situation, and control the engine 806 to control the traveling of the moving body according to the determination result. Specifically, the autonomous traveling vehicle 800 can detect lanes formed on both sides of the road on which the moving body is traveling. In this case, the controller 920 can determine whether the autonomous traveling vehicle 800 approaches a lane or deviates from the lane, and when it is determined that the autonomous traveling vehicle 800 approaches a lane or deviates from the lane, it can determine whether such traveling is due to a correct traveling situation or due to other traveling situations. Here, an example of a normal traveling situation may be a situation in which a lane change of the moving body is required. And, an example of another traveling situation may be a situation in which a lane change of the moving body is not required. When the controller 920 determines that the autonomously driving vehicle 800 is close to a lane or has deviated from the lane in a situation where the vehicle does not need to change lanes, the controller 920 can control the driving of the autonomously driving vehicle 800 so that the autonomously driving vehicle 800 does not deviate from the lane and drives normally.
[0124] If there is another moving body or an obstacle ahead of the moving body, the engine 806 or the braking system can be controlled to slow down the moving body, and in addition to the speed, the trajectory, the travel path, and the steering angle can be controlled. Alternatively, the controller 920 may generate necessary control signals to control the traveling of the moving body according to recognition information of other external environments such as the traveling lane of the moving body, the traveling signal, etc.
[0125] In addition to generating its own control signals, the controller 920 can also communicate with surrounding mobile units or a central server and, via the information received, control the movement of the mobile unit by sending commands to control the peripheral devices.
[0126] In addition, when the position of the camera module 950 is changed or the angle of view is changed, accurate recognition of the moving body or lane according to the present embodiment may be difficult. To prevent this, the controller 920 may generate a control signal to control the camera module 950 to perform calibration. Therefore, in the present embodiment, the controller 920 generates a calibration control signal in the camera module 950, so that even if the mounting position of the camera module 950 is changed due to vibration, impact, etc. generated with the movement of the autonomous traveling moving body 800, the normal mounting position, direction, angle of view, etc. of the camera module 950 can be continuously maintained. The controller 920 may generate a control signal to perform calibration of the camera module 920 when the initial mounting position, direction, angle of view information of the camera module 920 stored in advance and the initial mounting position, direction, angle of view information of the camera module 920 measured during the movement of the autonomous traveling moving body 800 change by more than a critical value.
[0127] In this embodiment, the controller 920 may include a memory 922 and a processor 924. The processor 924 may execute software stored in the memory 922 in accordance with control signals from the controller 920. In particular, the controller 920 may store data and instructions in the memory 922 for performing a lane detection method in accordance with the present invention, which instructions may be executed by the processor 924 to implement one or more methods disclosed herein. In this case, the memory 922 may be stored in a non-volatile recording medium executable by the processor 924. The memory 922 can store software and data via an appropriate internal or external device. The memory 922 can be composed of a random access memory (RAM), a read only memory (ROM), a hard disk, and a memory 922 device connected to a dongle.
[0128] The memory 922 may store at least an operating system (OS), user applications, and executable instructions, as well as application data and array data structures.
[0129] The processor 924 may be a microprocessor or any suitable electronic processor, controller, microcontroller, or state machine.
[0130] The processor 924 may be implemented with a combination of computing devices, which may comprise a digital signal processor, a microprocessor, or any suitable combination thereof.
[0131] Meanwhile, the autonomous vehicle 800 may further include a user interface 808 for user input to the control device 900 described above. The user interface 808 allows the user to input information through appropriate interaction. For example, the user interface 808 may be implemented as a touch screen, a keypad, an operation button, or the like. The user interface 808 transmits the input or command to the controller 920, and the controller 920 may execute a control operation of the vehicle in response to the input or command.
[0132] The user interface 808 is a device external to the autonomous vehicle 800 and can communicate with the autonomous vehicle 800 via a wireless communication device 930. For example, the user interface 808 can be linked to a mobile phone, a tablet, or another computer device.
[0133] Further, in this embodiment, the autonomous vehicle 800 has been described as including the engine 806, but may include other types of propulsion systems. For example, the vehicle may be powered by electric energy, hydrogen energy, or a hybrid system that combines these. Thus, the controller 920 may include the propulsion mechanisms of the autonomous vehicle 800's propulsion system and provide control signals to the configuration of each propulsion mechanism.
[0134] Hereinafter, with reference to FIG. 9, a detailed configuration of the control device 900 according to the present embodiment will be described in more detail.
[0135] The controller 900 includes a processor 924. The processor 924 may be a general purpose single-chip or multi-chip microprocessor, a special purpose microprocessor, a microcontroller, a programmable gate array, etc. The processor may be referred to as a central processing unit (CPU). Also, in this embodiment, the processor 924 may be used in combination with multiple processors.
[0136] The controller 900 also includes a memory 922. The memory 922 may be any electronic component capable of storing electronic information. The memory 922 may also include a single memory as well as a combination of memories 922.
[0137] Data and instructions 922a for executing the distance measurement method of the distance measurement device according to the present invention may be stored in the memory 922. When the processor 924 executes the instructions 922a, all or part of the instructions 922a and data 922b required for the execution of the instructions may be loaded 924a, 924b onto the processor 924.
[0138] The control device 900 may include a transmitter 930a, a receiver 930b, or a transceiver 930c to enable transmission and reception of signals. One or more antennas 932a, 932b may be electrically connected to the transmitter 930a, the receiver 930b, or each transceiver 930c and may further include an antenna.
[0139] The controller 900 may include a digital signal processor (DSP) 970. The DSP 970 may enable the mobile to rapidly process digital signals.
[0140] The control device 900 may include a communication interface 980. The communication interface 980 may include one or more ports and / or communication modules for connecting other devices to the control device 900. The communication interface 980 may allow a user to interact with the control device 900.
[0141] The various components of the controller 900 may be connected together by one or more buses 990, which may also include a power bus, a control signal bus, a status signal bus, a data bus, etc. Under the control of the processor 924, the components may communicate information to each other via the bus 990 to perform desired functions.
[0142] Meanwhile, in various embodiments, the control device 900 may be associated with a gateway for communication with the security cloud. For example, referring to FIG. 10, the control device 900 may be associated with a gateway 1005 for providing information acquired from at least one of the components 1001 to 1004 of the vehicle 1000 to the security cloud 1006. For example, the gateway 1005 may be included in the control device 900. In another example, the gateway 1005 may be composed of a separate device in the vehicle 1000 that is distinct from the control device 900. The gateway 1005 communicatively connects the software management cloud 1009 having different networks, the security cloud 1006, and the network in the vehicle 1000 protected by the in-vehicle security software 1010.
[0143] For example, the component 1001 may be a sensor. For example, the sensor may be used to obtain information regarding at least one of a state of the vehicle 1000 or a state surrounding the vehicle 1000. For example, the component 1001 may include the sensor 910.
[0144] For example, component 1002 may be an electronic control unit (ECU), which may be used for engine control, transmission control, airbag control, and tire pressure management, for example.
[0145] For example, component 1003 may be an instrument cluster. For example, an instrument cluster may refer to a panel of a dashboard that is located in front of the driver's seat. For example, the instrument cluster may be configured to display information required for driving to a driver (or passenger). For example, the instrument cluster may be used to display at least one of a visual element for indicating the revolutions per minute (RPM, revolutions per minute, or rotate per minute) of the engine, a visual element for indicating the speed of the vehicle 1000, a visual element for indicating the amount of remaining fuel, a visual element for indicating the status of the gear, or a visual element for indicating information obtained via component 1001.
[0146] For example, the component 1004 may be a telematics device. For example, the telematics device may refer to a device that provides various mobile communication services such as location information, safe driving, etc. in the vehicle 1000 by combining wireless communication technology and global positioning system (GPS) technology. For example, the telematics device may be used to connect the vehicle 1000 with a driver, a cloud (e.g., security cloud 1006), and / or a surrounding environment. For example, the telematics device may be configured to support high bandwidth and low latency for 5G NR standard technology (e.g., 5G NR V2X technology, 5G NR NTN (Non-Terrestrial Network) technology). For example, the telematics device may be configured to support autonomous driving of the vehicle 1000.
[0147] For example, the gateway 1005 can be used to connect a network in the vehicle 1000 to a software management cloud 1009 and a security cloud 1006, which are networks outside the vehicle. For example, the software management cloud 1009 can be used to update or manage at least one software required for running and managing the vehicle 1000. For example, the software management cloud 1009 can work with an in-car security software 1010 installed in the vehicle. For example, the in-car security software 1010 can be used to provide a security function in the vehicle 1000. For example, the in-car security software 1010 can encrypt data transmitted and received via the in-car network using an encryption key obtained from an external authorized server for encryption of the in-car network. In various embodiments, the encryption key used by the in-car security software 1010 can be generated corresponding to vehicle identification information (vehicle license plate, vehicle identification number (VIN)) or information uniquely assigned to each user (user identification information, etc.).
[0148] In various embodiments, the gateway 1005 can transmit data encrypted by the in-vehicle security software 1010 based on the encryption key to the software management cloud 1009 and / or the security cloud 1006. The software management cloud 1009 and / or the security cloud 1006 can identify which vehicle or which user the data is received from by decrypting the data using a decryption key that can decrypt the data encrypted by the encryption key of the in-vehicle security software 1010. For example, since the decryption key is a unique key corresponding to the encryption key, the software management cloud 1009 and / or the security cloud 1006 can identify the sender of the data (e.g., a vehicle or a user) based on the data decrypted via the decryption key.
[0149] For example, the gateway 1005 may be configured to support the in-vehicle security software 1010 and may be associated with the control device 2100. For example, the gateway 1005 may be associated with the control device 900 to support a connection between the client device 1007 connected to the security cloud 1006 and the control device 900. In another example, the gateway 1005 may be associated with the control device 900 to support a connection between the third party cloud 1008 connected to the security cloud 1006 and the control device 900. However, this is not limiting.
[0150] In various embodiments, the gateway 1005 can be used to connect the vehicle 1000 with a software management cloud 1009 for managing the operating software of the vehicle 1000. For example, the software management cloud 1009 can monitor whether an update of the operating software of the vehicle 1000 is requested and, based on monitoring that an update of the operating software of the vehicle 1000 is requested, provide data for updating the operating software of the vehicle 1000 via the gateway 1005. As another example, the software management cloud 1009 can receive a user request for an update of the operating software of the vehicle 1000 from the vehicle 1000 via the gateway 1005 and, based on the receipt, provide data for updating the operating software of the vehicle 1000. However, this is not limiting.
[0151] FIG. 11 illustrates an example of a flow chart for the operation of an electronic device according to an embodiment. The electronic device of FIG. 11 may include the electronic device 101 of FIG. 1, FIG. 2, FIG. 3a, FIG. 3b, FIG. 4a, FIG. 4b, FIG. 4c, FIG. 5a, FIG. 5b, FIG. 5c, FIG. 5d, FIG. 6a, FIG. 6b, and / or FIG. 6c. The electronic device of FIG. 11 may include the autonomous driving system 700 of FIG. 7. The electronic device of FIG. 11 may include the control device 900 of FIG. 8 and / or FIG. 9. The operation of FIG. 11 may be performed by the processor 120 of FIG. 1.
[0152] 11, in operation 1101, an electronic device according to one embodiment may capture an image via a camera (e.g., camera 140 of FIG. 1 ). The electronic device may identify a width of a window to be used to split the image based on the FOV of the image captured via the camera and / or the FOV of the camera.
[0153] In operation 1103, an electronic device according to an embodiment may identify a height of a window based on a first region that includes a visual object (eg, the ground) that corresponds to a reference plane.
[0154] In operation 1105, the electronic device according to an embodiment can divide the first region using the window into a number of sub-regions based on the width and height of the window. For example, the electronic device can divide the first region corresponding to the ground into sub-regions having a size equal to the size of the window.
[0155] In operation 1107, the electronic device according to an embodiment may identify whether an external object is included in a first sub-region from the neural network to which a first sub-region of the plurality of sub-regions is input. For example, the electronic device may sequentially input the plurality of sub-regions to the neural network. The electronic device may identify whether an external object is included in one of the plurality of sub-regions based on the sequential input of the plurality of sub-regions to the neural network. For example, the electronic device may identify that an external object is included in a first sub-region of the plurality of sub-regions. The electronic device may obtain a second sub-region spaced apart from the first sub-region by a specified distance based on identifying that an external object is included in the first sub-region.
[0156] In operation 1109, the electronic device according to an embodiment can obtain a second sub-region in the image spaced apart from the first sub-region based on the spacing identified based on whether the external object is included in the first sub-region. The electronic device can input the second sub-region that does not include the first sub-region into the neural network based on identifying the external object in the first sub-region.
[0157] As described above, an electronic device according to an embodiment can identify an external object based on a partial region of an image input to a neural network. For example, the neural network may include a neural network for identifying an external object using a partial region of an image. By identifying the external object using the neural network, the electronic device can identify an external object having a relatively small size. By identifying the external object based on the partial region, the electronic device can reduce processor usage.
[0158] FIG. 12 illustrates an example of a flow chart for the operation of an electronic device according to an embodiment. The electronic device of FIG. 12 may include the electronic device 101 of FIG. 1, FIG. 2, FIG. 3a, FIG. 3b, FIG. 4a, FIG. 4b, FIG. 4c, FIG. 5a, FIG. 5b, FIG. 5c, FIG. 5d, FIG. 6a, FIG. 6b, and / or FIG. 6c, and / or the electronic device of FIG. 11. The electronic device of FIG. 12 may include the autonomous driving system 700 of FIG. 7. The electronic device of FIG. 12 may include the control device 900 of FIG. 8 and / or FIG. 9. The operation of FIG. 12 may be performed by the processor 120 of FIG. 1.
[0159] 12, in operation 1201, an electronic device according to an embodiment may identify an image stored in a memory (e.g., memory 130 of FIG. 1). The electronic device may obtain a first data set associated with a first vehicle based on identifying a vehicle in the image stored in the memory in a first lane (e.g., first lane 251 of FIG. 2) of a plurality of lanes (e.g., plurality of lanes 250 of FIG. 2) separated by lanes (e.g., lane 230 of FIG. 2). For example, the first data set associated with the first vehicle may include data associated with a vehicle license plate of the first vehicle identified in the area identified as the first vehicle.
[0160] In operation 1203, the electronic device according to an embodiment may identify a second vehicle on a second road (e.g., second road 252 in FIG. 2 ) of the plurality of roads that is different from the first road. The electronic device may obtain a second data set related to the second vehicle based on identifying the second vehicle on the second road. For example, the second data set related to the second vehicle may include data related to a vehicle license plate of the second vehicle identified in the area identified as the second vehicle.
[0161] In operation 1205, the electronic device according to an embodiment may identify a third vehicle on a third road (e.g., the third road 253 in FIG. 2 ) of the plurality of roads, the third road being different from the first road and the second road. For example, the electronic device may obtain a data set related to the third vehicle based on identifying the third vehicle on the third road. For example, the third data set related to the third vehicle may include data related to a vehicle license plate of the third vehicle identified in the area identified as the third vehicle.
[0162] In operation 1207, the electronic device according to an embodiment may train a neural network with truth data based on each of the first data set, the second data set, and the third data set. The electronic device may train the neural network based on a data set related to a vehicle identified on different roads. For example, the data set may include data related to a road in which the vehicle was identified among a plurality of roads and / or a vehicle license plate included on the vehicle.
[0163] As previously mentioned, according to one embodiment, an electronic device can identify vehicles on a plurality of roads. The electronic device can obtain a data set associated with each of the identified vehicles on each of the plurality of roads. The electronic device can train a neural network based on the data set. The electronic device can obtain a neural network for identifying external objects included in subregions in an image by training the neural network based on the data set.
[0164] 13 is a diagram illustrating the operation of an electronic device 101 for training a neural network based on a set of training data, according to one embodiment. The operations described with reference to FIG. 13 may be performed by the electronic devices of FIGS. 1-12.
[0165] Referring to FIG. 13, in operation 1302, an electronic device according to an embodiment may obtain a set of training data. The electronic device may obtain a set of training data for supervised learning. The training data may include a pair of input data and ground truth data corresponding to the input data. The ground truth data may represent output data to be obtained from a neural network that receives the input data, which is a pair of the ground truth data. The ground truth data may be obtained by the electronic device described with reference to FIGS. 1 to 12.
[0166] For example, when training a neural network to recognize images, the training data may include information about the images and one or more objects contained in the images. The information may include a category or class of objects identifiable via the images. The information may include a position, width, height, and / or size of a visual object in the image that corresponds to the object. The set of training data identified via operation 1302 may include multiple training data pairs. In the above example of training a neural network to recognize images, the set of training data identified by the electronic device may include multiple images and basis truth data corresponding to each of the multiple images.
[0167] 13, in operation 1304, an electronic device according to an embodiment may perform training of a neural network based on a set of training data. In an embodiment in which a neural network is trained based on supervised learning, the electronic device may input input data included in the training data to an input layer of the neural network. An example of a neural network including the input layer is described with reference to FIG. 14. From an output layer of the neural network that receives the input data via the input layer, the electronic device may obtain output data of the neural network corresponding to the input data.
[0168] In one embodiment, the training of operation 1304 may be performed based on a difference between the output data and underlying truth data included in the learning data and corresponding to the input data. For example, the electronic device may adjust one or more parameters (e.g., weights, described below with reference to FIG. 14) associated with the neural network so that the difference is reduced based on a gradient descent algorithm. The operation of the electronic device adjusting the one or more parameters may be referred to as tuning the neural network. The electronic device may perform the tuning of the neural network based on the output data using a function defined to evaluate the performance of the neural network, such as a cost function. The difference between the output data and underlying truth data may be included as an example of a cost function.
[0169] 13, in operation 1306, an electronic device according to an embodiment may identify whether valid output data is output from the neural network trained by operation 1304. The output data being valid may mean that the difference (or cost function) between the output data and the basis truth data satisfies the conditions set for using the neural network. For example, if the average and / or maximum value of the difference between the output data and the basis truth data is less than or equal to a specified threshold, the electronic device may determine that valid output data is output from the neural network.
[0170] If the neural network does not output valid output data (operation 1306-no), the electronic device may repeatedly perform training of the neural network based on operation 1304. The embodiment is not limited thereto, and the electronic device may repeatedly perform operations 1302, 1304.
[0171] Upon obtaining valid output data from the neural network (operation 1306-YES), the electronic device according to an embodiment may use the trained neural network based on operation 1308. For example, the electronic device may input other input data to the neural network as learning data, which is differentiated from the input data input to the neural network. The electronic device may use the output data obtained from the neural network that has received the other input data as a result of inference made on the other input data based on the neural network.
[0172] Figure 14 is a block diagram of an electronic device 101 according to one embodiment. The electronic device 101 of Figure 14 may include the electronic devices of Figures 1 to 13. For example, the operations described with reference to Figure 13 may be performed by the electronic device 101 of Figure 14 and / or the processor 1410 of Figure 14.
[0173] 14, the processor 1410 of the electronic device 101 can execute computations related to the neural network 1430 stored in the memory 1420. The processor 1410 may include at least one of a center processing unit (CPU), a graphic processing unit (GPU), or a neural processing unit (NPU). The NPU may be implemented as a chip separate from the CPU, or may be integrated into a chip such as the CPU in the form of a system on a chip (SoC). An NPU integrated into a CPU may be referred to as a neural core and / or an artificial intelligence (AI) accelerator.
[0174] Referring to FIG. 14, the processor 1410 may identify a neural network 1430 stored in the memory 1420. The neural network 1430 may include a combination of an input layer 1432, one or more hidden layers 1434 (or intermediate layers), and an output layer 1436. The aforementioned layers (e.g., the input layer 1432, the one or more hidden layers 1434, and the output layer 1436) may include multiple nodes. The number of hidden layers 1434 may vary depending on the embodiment, and a neural network 1430 including multiple hidden layers 1434 may be referred to as a deep neural network. The operation of training a deep neural network may be referred to as deep learning.
[0175] In one embodiment, when the neural network 1430 has a structure of a feed forward neural network, a first node included in a particular layer can be connected to all of the second nodes included in another layer before the particular layer. The parameters stored in the memory 1420 for the neural network 1430 can include weights assigned to the connections between the second node and the first node. In the neural network 1430 having the structure of a feed forward neural network, the value of the first node can correspond to a weighted sum of the values assigned to the second node based on the weights assigned to the connections connecting the second node and the first node.
[0176] In one embodiment, when the neural network 1430 has the structure of a convolution neural network, a first node included in a particular layer can correspond to a weighted weight for a portion of a second node included in another layer preceding the particular layer. Some of the second nodes corresponding to the first node may be identified by a filter corresponding to the particular layer. The parameters stored in the memory 1420 for the neural network 1430 can include weights representing the filters. The filter can include one or more of the second nodes used to calculate the weighted weight for the first node, and weights corresponding to each of the one or more nodes.
[0177] According to one embodiment, the processor 1410 of the electronic device 101 can perform training on the neural network 1430 using a training data set 1440 stored in the memory 1420. Based on the training data set 1440, the processor 1410 can perform the operations described with reference to FIG.
[0178] According to an embodiment, the processor 1410 of the electronic device 101 can perform object detection, object recognition, and / or object classification using the neural network 1430 trained based on the learning dataset 1440. The processor 1410 can input an image (or video) acquired via the camera 1450 to the input layer 1432 of the neural network 1430. Based on the input layer 1432 to which the image is input, the processor 1410 can sequentially obtain values of nodes of the layers included in the neural network 1430 and obtain a set of values of nodes of the output layer 1436 (e.g., output data). The output data can be used as a result of estimating information included in the image using the neural network 1430. The embodiment is not limited thereto, and the processor 1410 can input an image (or video) acquired from an external electronic device connected to the electronic device 101 to the neural network 1430 via the communication circuit 1460.
[0179] In one embodiment, the neural network 1430 trained to process an image can be used to identify regions in the image that correspond to objects (object detection) and / or to identify classes of objects represented in the image (object recognition and / or object classification). For example, the electronic device 101 can use the neural network 1430 to segment regions in the image that correspond to the objects based on a rectangular shape, such as a bounding box. For example, the electronic device 101 can use the neural network 1430 to identify at least one class of a plurality of specified classes that matches the object.
[0180] As described above, an electronic device according to an embodiment may include a camera and a processor. The processor may identify a width of a window used to divide the image based on a field of view (FOV) of an image acquired through the camera, identify a height of the window based on a first region including a visual object corresponding to a reference surface, divide the first region into a plurality of subregions using the window based on the width and height, identify whether an external object is included in the first subregion from a neural network to which a first subregion of the plurality of subregions is input, and obtain a second subregion in the image separated from the first subregion based on an interval identified based on whether the external object is included in the first subregion.
[0181] According to one embodiment, the processor is capable of identifying distortion in an image, which is a first image taken by the camera, and identifying the external object based on a second image that differs from the first image and that has been compensated for the distortion.
[0182] According to one embodiment, the processor is operable to overlap the sub-regions by a first proportion of the width.
[0183] According to one embodiment, the electronic device may include a sensor, and the processor may identify a first speed of the electronic device based on the sensor, the first speed being less than or equal to a first reference speed, and may overlap the plurality of sub-regions with a first sub-region including an edge of the image by a second ratio less than the first ratio based on identifying the first speed.
[0184] According to one embodiment, the electronic device can include a sensor, and the processor can identify a second speed of the electronic device based on the sensor, the second speed being equal to or greater than a second reference speed, and can overlap the sub-regions by a second ratio less than the first ratio, with the second sub-region different from the first sub-region including an edge of the image based on identifying the second speed.
[0185] According to one embodiment, the electronic device may include a speaker, and the processor may output an audio signal based on identifying the external object to indicate that the external object has been identified.
[0186] According to one embodiment, the electronic device may include a display, and the processor may display a visual object corresponding to the external object based on identifying the external object.
[0187] According to one embodiment, the electronic device may include a sensor, and the processor may use the sensor to identify a direction in which the camera is pointing and identify the height based on identifying the direction within a specified range.
[0188] According to one embodiment, the height may include a first height, and the processor may identify a second height of the second region and adjust a size of the sub-regions capturing portions of the image based on identifying a difference between the second height and the first height.
[0189] According to one embodiment, the processor may identify a first subregion including a vertex of the second region based on identifying a first height that is smaller than a second height, and identify a second subregion that overlaps the first subregion and has the same size as the first subregion.
[0190] As described above, according to one embodiment, an electronic device may include a memory and a processor, the processor may acquire a first data set related to a first vehicle based on identifying a first vehicle on a first road among a plurality of lanes separated by lanes in an image stored in the memory, acquire a second data set related to a second vehicle based on identifying a second vehicle on a second road among the plurality of lanes different from the first road, acquire a third data set related to a third vehicle based on identifying a third vehicle on a third road among the plurality of lanes different from the first road and the second road, and train a neural network using truth data based on each of the first data set, the second data set, and the third data set.
[0191] According to one embodiment, the neural network may be trained using vehicle morphologies identified in each of the first data set, the second data set, and the third data set.
[0192] According to one embodiment, the neural network can be trained using a vehicle license plate within a first bounding box that includes one or more pieces of text contained within the first bounding box identified as a first vehicle.
[0193] According to one embodiment, the neural network may be trained with a labeling for one or more pieces of text contained in the second domain.
[0194] As described above, according to one embodiment, a method of an electronic device may include an operation of identifying a width of a window used to divide an image based on a field of view (FOV) of the image acquired via a camera, an operation of identifying a height of the window based on a first region including a visual object corresponding to a reference surface, an operation of dividing a plurality of partial regions that are partially overlapped with each other in the image based on the width and height, an operation of identifying whether an external object is included in the first partial region from a neural network to which a first partial region of the plurality of partial regions is input, and an operation of obtaining a second partial region in the image separated from the first partial region based on an interval identified based on whether the external object is included in the first partial region.
[0195] According to one embodiment, the method may include an operation of identifying a distortion of the image, which is a first image taken by the camera, and an operation of identifying the external object based on a second image that differs from the first image and that has been compensated for the distortion.
[0196] According to one embodiment, the method may include partially overlapping the sub-regions by a first proportion of the width.
[0197] According to one embodiment, the method may include an operation of identifying a first speed of the electronic device based on a sensor, the first speed being less than or equal to a first reference speed, and an operation of overlapping the multiple sub-regions by a second ratio less than the first ratio, with a first sub-region including an edge of the image, based on the identification of the first speed.
[0198] According to one embodiment, the method may include an operation of identifying a second speed of the electronic device based on a sensor, the second speed being equal to or greater than a second reference speed, and an operation of overlapping the multiple sub-regions by a second ratio less than the first ratio, with the second sub-region different from a first sub-region including an edge of the image, based on the identification of the second speed.
[0199] According to one embodiment, the method may include outputting an audio signal based on identifying the external object to indicate that the external object has been identified.
Claims
1. 1. An electronic device comprising: Camera and a processor; The processor, Identifying a width of a window used to segment the image based on a field of view (FOV) of an image acquired via the camera; Identifying a height of the window based on a first region containing a visual object corresponding to a reference surface; Dividing the first region into a plurality of sub-regions using the window based on the width and the height; identifying whether a first partial region of the plurality of partial regions includes an external object from the neural network to which the first partial region is input; The electronic device is configured to obtain a second partial region in the image spaced apart from the first partial region based on a distance identified based on whether the external object is included in the first partial region.
2. The processor, Identifying a distortion of the image, the first image, by the camera; The electronic device of claim 1 , configured to identify the external object based on a second image that is different from the first image and that is compensated for the distortion.
3. The processor, The electronic device of claim 1 , wherein the plurality of sub-regions are configured to overlap by a first proportion of the width.
4. Further comprising a sensor; The processor, Identifying a first speed of the electronic device based on the sensor, the first speed being less than or equal to a first reference speed; and 4. The electronic device of claim 3, further comprising: a first sub-region including an edge of the image that is configured to overlap the plurality of partial regions by a second ratio that is less than the first ratio based on the identification of the first speed.
5. Further comprising a sensor; The processor, Identifying a second speed of the electronic device based on the sensor, the second speed being greater than or equal to a second reference speed; and 4. The electronic device of claim 3, further comprising: a second sub-region that is different from a first sub-region that includes an edge of the image and that overlaps the plurality of sub-regions by a second ratio that is smaller than the first ratio based on the identification of the second speed.
6. Further including a speaker; The processor, The electronic device of claim 1 , configured to, upon identifying the external object, output an audio signal to indicate that the external object has been identified.
7. Further comprising a display; The processor, The electronic device of claim 1 , configured to display a visual object corresponding to the external object based on identifying the external object.
8. Further comprising a sensor; The processor, Identifying a direction in which the camera is facing, identified using the sensor; The electronic device of claim 1 , configured to identify the height based on identifying the direction within a specified range.
9. the height is a first height; The processor, identifying a second height of the second region; 2. The electronic device of claim 1, configured to adjust a size of the plurality of sub-regions capturing portions of the image based on identifying a difference between the second height and the first height.
10. The processor, identifying a first sub-region including a vertex of the second region based on identifying the first height being less than the second height; The electronic device of claim 9 , configured to identify a second sub-region that overlaps the first sub-region and has the same size as the first sub-region.
11. 1. An electronic device comprising: Memory, a processor; The processor, acquiring a first data set associated with a first vehicle based on identifying a first vehicle on a first road among a plurality of lanes separated by lines in the image stored in the memory; acquiring a second data set associated with the second vehicle based on identifying the second vehicle on a second road among the plurality of roads, the second road being different from the first road; acquiring a third data set related to the third vehicle based on identifying a third vehicle on a third road among the plurality of roads, the third road being different from the first road and the second road; 11. An electronic device configured to train a neural network using truth data based on each of the first data set, the second data set, and the third data set.
12. The neural network comprises: The electronic device of claim 11 , trained using vehicle morphologies identified from each of the first data set, the second data set, and the third data set.
13. The neural network comprises:
12. The electronic device of claim 11, trained with vehicle license plates within a first bounding box that includes one or more texts contained within the first bounding box identified as the vehicle.
14. The neural network comprises: The electronic device of claim 13 , trained with one or more text labelings contained in the second region.
15. 1. A method for an electronic device, comprising: Identifying a width of a window used to segment an image based on a field of view (FOV) of the image acquired via a camera; identifying a height of the window based on a first region containing a visual object corresponding to a reference surface; dividing the image into a plurality of overlapping sub-regions based on the width and the height; identifying, from a neural network to which a first subregion of the plurality of subregions is input, whether an external object is included in the first subregion; and The method includes an operation of obtaining a second sub-region in the image spaced apart from the first sub-region based on a distance identified based on whether the external object is included in the first sub-region.
16. The method comprises: identifying a distortion of the image, the first image from the camera; and The method of claim 15 , comprising an act of identifying the external object based on a second image that differs from the first image and that is compensated for the distortion.
17. The method comprises: The method of claim 15 , comprising overlapping the sub-regions by a first proportion of the width.
18. The method comprises: identifying a first speed of the electronic device based on the sensor, the first speed being less than or equal to a first reference speed; and 20. The method of claim 17, comprising overlapping the plurality of sub-regions by a second ratio less than the first ratio, with a first sub-region including an edge of the image based on identifying the first velocity.
19. The method comprises: identifying a second speed of the electronic device based on the sensor, the second speed being greater than or equal to a second reference speed; and 20. The method of claim 17, further comprising: overlapping the plurality of sub-regions by a second ratio less than the first ratio, with the second sub-region different from a first sub-region that includes an edge of the image based on identifying the second velocity.
20. The method comprises: The method of claim 15 , comprising the act of outputting an audio signal based on identifying the external object to indicate that the external object has been identified.
