Electronic device, method, and non-temporary computer-readable storage medium for identifying the direction of movement of an external object.

Abstract

The present invention provides an electronic device, method, and non-temporary computer-readable storage medium for identifying the direction of movement of an external object. [Solution] The electronic device includes an image sensor, a CPU, an NPU (neural processing unit), and a memory that stores instructions and includes one or more storage media. The CPU acquires an image 750 via the image sensor, controls the NPU to execute an object detection model that detects an external object 751 from the image, acquires coordinate values ​​761, 761-1 to 761-8 representing a part of the image related to the external object from the NPU, controls the CPU's processing circuit to perform a series of calculations based on the coordinate values ​​that define a direction identification model that identifies the direction of movement 753 of the external object, and outputs a notification regarding the direction of movement based on the direction of movement of the external object represented by the results of the series of calculations.

Description

Technical Field

[0001] The following description relates to an electronic device, a method, and a non - transient computer - readable storage medium for identifying the moving direction of an external object.

Background Art

[0002] The electronic device can be attached to a vehicle. The electronic device can execute the functions of an advanced driver assistance system (ADAS). The electronic device can prevent traffic accidents by using the advanced driver assistance system. For the convenience and safety of drivers, the advanced driver assistance system has been studied.

[0003] The foregoing information can be provided as background art (related art) for the purpose of assisting in the understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing can be applied as prior art related to the present disclosure.

Summary of the Invention

Means for Solving the Problems

[0004] An electronic device is provided. The electronic device may include an image sensor. The electronic device may include a CPU (central processing unit). The electronic device may include a NPU (neural processing unit). The electronic device may include a memory that stores instructions and includes one or more storage media. When the CPU executes the instructions, the electronic device may be prompted to acquire an image via the image sensor. When the CPU executes the instructions, the electronic device may be prompted to control the NPU to execute an object detection model configured to detect an external object from the image. When the CPU executes the instructions, the electronic device may be prompted to acquire coordinate values ​​from the NPU representing a portion of the image related to the external object. When the CPU executes the instructions, the electronic device may be prompted to control the CPU's processing circuitry to perform a series of calculations based on the coordinate values ​​that define a direction identification model configured to identify the direction of movement of the external object. When the instruction is executed by the CPU, it can cause the electronic device to output a notification regarding the direction of movement of the external object, based on the direction of movement of the external object, which is represented by the results of the plurality of calculations.

[0005] A method is provided. This method can be performed within an electronic device having an image sensor, a CPU, and an NPU. This method may include an operation to acquire an image via the image sensor. This method may include an operation to control the NPU to execute an object detection model configured to detect an external object from the image. This method may include an operation to acquire coordinate values ​​from the NPU that represent a portion of the image related to the external object. This method may include an operation to control the processing circuit of the CPU to perform a set of calculations based on the coordinate values ​​that define a direction identification model configured to identify the direction of movement of the external object. This method may include an operation to output a notification regarding the direction of movement of the external object based on the direction of movement of the external object represented by the results of the set of calculations.

[0006] A non-temporary computer-readable storage medium is provided. This non-temporary computer-readable storage medium can store one or more programs. When executed by the electronic device having an image sensor, a CPU, and an NPU, the one or more programs may include instructions to cause the electronic device to acquire an image via the image sensor. When executed by the electronic device, the one or more programs may include instructions to cause the electronic device to control the NPU to execute an object detection model configured to detect an external object from the image. When executed by the electronic device, the one or more programs may include instructions to cause the electronic device to obtain coordinate values ​​from the NPU representing a portion of the image related to the external object. When executed by the electronic device, the one or more programs may include instructions to cause the electronic device to control the processing circuit of the CPU to perform a plurality of calculations based on the coordinate values ​​that define a direction identification model configured to identify the direction of movement of the external object. When executed by the electronic device, the one or more programs may include instructions to cause the electronic device to output a notification regarding the direction of movement based on the direction of movement of the external object represented by the results of the plurality of calculations.

[0007] An electronic device is provided. The electronic device may include an image sensor. The electronic device may include a CPU (central processing unit). The electronic device may include a NPU (neural processing unit). The electronic device may include a memory that stores instructions and includes one or more storage media. When the CPU executes the instructions, the electronic device may be prompted to acquire an image via the image sensor. When the CPU executes the instructions, the electronic device may be prompted to control the NPU to execute an object detection model configured to detect an external object from the image. When the CPU executes the instructions, the electronic device may be prompted to acquire coordinate values ​​from the NPU representing a portion of the image related to the external object. When the CPU executes the instructions, the electronic device may be prompted to control the CPU's processing circuitry to perform a series of calculations based on the coordinate values ​​that define a direction identification model configured to identify the direction of movement of the external object. When the instruction is executed by the CPU, the electronic device may cause the electronic device to output a notification regarding the distance between the vehicle on which the electronic device is installed and the external object, based on the direction of movement of the external object and the coordinate values ​​represented by the results of the plurality of calculations.

[0008] A method is provided. This method can be performed within the electronic device having an image sensor, a CPU, and an NPU. The method may include an operation to acquire an image via the image sensor. The method may include an operation to control the NPU to execute an object detection model configured to detect an external object from the image. The method may include an operation to acquire coordinate values ​​from the NPU representing a portion of the image related to the external object. The method may include an operation to control the processing circuit of the CPU to perform a set of calculations based on the coordinate values ​​that define a direction identification model configured to identify the direction of movement of the external object. The method may include an operation to output a notification regarding the distance between the vehicle on which the electronic device is installed and the external object, based on the direction of movement of the external object represented by the results of the set of calculations and the coordinate values.

[0009] A non-temporary computer-readable storage medium is provided. The non-temporary computer-readable storage medium can store one or more programs. When executed by the electronic device having an image sensor, a CPU, and an NPU, the one or more programs may include instructions to cause the electronic device to acquire an image via the image sensor. When executed by the electronic device, the one or more programs may include instructions to cause the electronic device to control the NPU to execute an object detection model configured to detect an external object from the image. When executed by the electronic device, the one or more programs may include instructions to cause the electronic device to obtain coordinate values ​​from the NPU representing a portion of the image related to the external object. When executed by the electronic device, the one or more programs may include instructions to cause the electronic device to control the processing circuit of the CPU to perform a plurality of calculations based on the coordinate values ​​that define a direction identification model configured to identify the direction of movement of the external object. When executed by the electronic device, the one or more programs may include instructions to cause the electronic device to output a notification regarding the distance between the vehicle on which the electronic device is installed and the external object, based on the direction of movement of the external object represented by the results of the plurality of calculations and the coordinate values. [Brief explanation of the drawing]

[0010] [Figure 1] Examples of environments, including vehicles equipped with electronic devices, are shown. [Figure 2] This is a simplified block diagram of an exemplary electronic device. [Figure 3] This example demonstrates how to generate notifications of the movement direction of external objects using the NPU and CPU. [Figure 4][Figure 4a] An example of identifying external objects in an image using an object detection model. [Figure 4b] An example of identifying external objects in an image using an object detection model. [Figure 5] This example demonstrates how to identify regions within an image using an image segmentation model. [Figure 6] This example demonstrates selecting a portion of the data transmitted from the NPU so that multiple calculations of the direction discrimination model can be performed within a given timeframe. [Figure 7] [Figure 7a] This shows an example of calculating the distance between a vehicle equipped with an electronic device and an external object using coordinate values ​​representing the direction of movement of the external object and a portion of the image associated with the external object. [Figure 7b] This shows an example of calculating the distance between a vehicle equipped with an electronic device and an external object using coordinate values ​​representing the direction of movement of the external object and a portion of the image associated with the external object. [Figure 8] This example shows an environment in which an electronic device outputs a notification based on the direction of movement of an external object. [Figure 9] [Figure 9a] An example of an action performed by an electronic device in accordance with a notification. [Figure 9b] An example of an action performed by an electronic device in accordance with a notification. [Figure 10] An example of the structure of an artificial intelligence model is shown. [Figure 11] An example of a block diagram illustrating an autonomous driving system for a vehicle according to one embodiment is shown. [Figure 12] An example of a block diagram showing an autonomous mobile vehicle according to one embodiment is shown. [Figure 13] An example of a block diagram showing an autonomous mobile vehicle according to one embodiment is shown. [Figure 14] Examples of gateways related to user devices in various embodiments are shown. [Figure 15] This figure illustrates the operation of an electronic device for training a neural network based on a set of training data, according to one embodiment. [Figure 16] This is a block diagram of an electronic device according to one embodiment.

[0011] Throughout the drawings, it will be understood that the same reference number refers to the same part, component, and structure. [Modes for carrying out the invention]

[0012] The terms used in this disclosure are used solely to describe specific embodiments and are not intended to limit the scope of other embodiments. Singular expressions may, in context, include plural expressions unless otherwise specified. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by a person of ordinary skill in the art described herein. Terms used herein that are defined in a general dictionary may be interpreted as having the same or similar meaning as in the context of the relevant art, and not as ideally or excessively formal unless expressly defined herein. In some cases, terms defined herein may not be interpreted in a way that excludes embodiments of this disclosure.

[0013] The various embodiments of the Disclosure described below illustrate hardware-based approaches as examples. However, since the various embodiments of the Disclosure include techniques that use both hardware and software, the various embodiments of the Disclosure do not exclude software-based approaches.

[0014] Terms used to refer to data (e.g., data, information), terms used to refer to values (e.g., threshold values), terms used for operational states (e.g., operation, process), terms used to refer to objects (e.g., external objects), terms used to refer to network entities, terms used to refer to components of a device, etc. are exemplified for the convenience of explanation. Therefore, the present disclosure is not limited to the terms described below, and other terms having equivalent technical meanings can be used. Further, terms such as "... part", "... device", "... object", "... body" used below may mean at least one shape structure or a unit that processes functions.

[0015] Furthermore, in the present disclosure, expressions of exceeding or being less than can be used to determine whether specific conditions are satisfied or fulfilled, but this is merely an illustrative explanation and does not exclude the above or below descriptions. The conditions described as "above" can be replaced with "exceeding", the conditions described as "below" can be replaced with "being less than", and the conditions described as "above and below" can be replaced with "exceeding and being less than". Also, hereinafter, "A" to "B" mean at least one of the elements from A (including A) to B (including B). Hereinafter, "C" and / or "D" means at least one of "C" or "D", that is, it includes {"C", "D", "C and D"}.

[0016] FIG. 1 shows an example of an environment including a vehicle on which an electronic device is mounted.

[0017] Referring to FIG. 1, the environment 100 can include a vehicle 102 on which an electronic device 101 is mounted. The form in which the electronic device 101 shown in FIG. 1 is attached to the vehicle 102 is merely an example. For example, the electronic device 101 may be built into the vehicle 102. For example, the electronic device 101 may be built-in to the vehicle 102.

[0018] For example, the electronic device 101 can be described as a video processing device for a vehicle. For example, the electronic device 101 may include a black box and navigation. For example, the electronic device 101 can be realized as a product in various forms, such as a personal computer, laptop, tablet computer, smartphone, smart home appliance, intelligent automobile, or wearable device. However, it is not limited to these.

[0019] For example, the electronic device 101 may include an image sensor. For example, the electronic device 101 may acquire an image via the image sensor. For example, the image acquired by the electronic device 101 may include external objects (for example, external object 111, external object 112). For example, the electronic device 101 may store the image acquired via the image sensor in its own memory.

[0020] For example, vehicle 102 may collide with an external object 112. For example, electronic device 101 can perform functions of an advanced driver assistance system (ADAS) to prevent a collision between vehicle 102 and the external object 112. For example, the faster the processing speed of electronic device 101, the higher the quality of the functions of the advanced driver assistance system may be. For example, the processing speed of electronic device 101 can be increased as the data load decreases.

[0021] For example, the electronic device 101 can identify (or determine) the direction of movement of the external object 112 in order to prevent a collision between the vehicle 102 and the external object 112. For example, the electronic device 101 may need a method to identify the direction of movement of the external object 112 using the coordinate values ​​of the external object 112 in the image in order to reduce the data load. For example, the electronic device 101 may need a method to output a notification depending on the direction of movement of the external object.

[0022] For example, the electronic device 101 may determine the probability of a collision between the vehicle 102 and the external object 111. For example, the electronic device 101 can calculate the distance between the vehicle 102 and the external object 111 in order to determine the probability of a collision. For example, the electronic device 101 may need a method to determine the distance between the vehicle 102 and the external object 111 based on the coordinate values ​​of the external object 111 in the image and the direction of movement of the external object 111, for a relatively small data load.

[0023] For example, such a method can be carried out in an electronic device described later. For example, the electronic device described later may include components (or hardware components) for providing such a method. Such components are described and illustrated in more detail with reference to Figure 2.

[0024] Figure 2 is a simplified block diagram of an exemplary electronic device.

[0025] Referring to Figure 2, the electronic device 201 may include a CPU (central processing unit) 200, an NPU (neural processing unit) 210, a memory 220, an image sensor 230, a display 240, an LED (light-emitting diode) 250, and a speaker 260. For example, the electronic device 101 may be an example of the electronic device 201.

[0026] The CPU 200 can be represented as a hardware component for processing data based on the execution of instructions. For example, the CPU 200 may include processing circuits. The CPU 200 may include one or more cores. For example, the CPU 200 may have a multi-core processor structure such as a dual-core, quad-core, or hexa-core.

[0027] The NPU210 can be described as a hardware component for utilizing AI (artificial intelligence) software. For example, the NPU210 may include processing circuits. For example, the NPU210 may be usable for machine learning algorithms and / or deep learning algorithm calculations. For example, the NPU210 may be usable for running machine learning models and / or deep learning models.

[0028] Memory 220 may include hardware components for storing data and / or instructions that are input to and / or output from the CPU 200. Memory 220 may include volatile memory such as RAM (random-access memory) and / or non-volatile memory such as ROM (read-only memory). Volatile memory may include at least one of DRAM (dynamic RAM), SRAM (static RAM), Cache RAM, and PSRAM (pseudo SRAM). Non-volatile memory may include at least one of PROM (programmable ROM), EPROM (erasable PROM), EEPROM (electrically erasable PROM), flash memory, hard disk, compact disk, and EMMC (embedded multimedia card).

[0029] The image sensor 230 may include one or more optical sensors (e.g., CCD (charged coupled device) sensors, CMOS (complementary metal oxide semiconductor) sensors) that generate electrical signals indicating the color and / or brightness of light. The multiple optical sensors included in the image sensor 230 may be arranged in the form of a two-dimensional array. The image sensor 230 can acquire the electrical signals of each of the multiple optical sensors substantially simultaneously to generate an image containing multiple pixels arranged in two dimensions, corresponding to the light reaching the optical sensors in the two-dimensional array. For example, photographic data captured using the image sensor 230 may represent a single image acquired from the image sensor 230. For example, video data captured using the image sensor 230 may represent a sequence of multiple images acquired from the image sensor 230 according to a specified frame rate.

[0030] The display 240 may include hardware components of the electronic device 201 used to display the screen. For example, the display 240 may include light-emitting elements and circuits (e.g., transistors) that control the light-emitting elements to emit light. For example, each light-emitting element may include an OLED (organic light-emitting diode) or a micro-LED, but is not limited thereto. For example, the display 240 may include an LCD (liquid crystal display).

[0031] LED250 can emit light. For example, LED250 may be controlled by CPU200 to emit light. For example, LED250 can emit blue light, green light, and / or red light. However, it is not limited to these.

[0032] The speaker 260 can output an acoustic signal to the outside of the electronic device 201. For example, the speaker 260 can be used for general purposes such as multimedia playback or recording and playback.

[0033] For example, the CPU 200 can execute instructions stored in memory 220. For example, when such instructions are executed by the CPU 200, they can trigger the output of a notification regarding the direction of movement of an external object in an image acquired via the image sensor 230. These operations are described and illustrated in more detail with reference to Figures 3 to 10.

[0034] Figure 3 shows an example of generating notifications of the direction of movement of an external object using an NPU and a CPU. The following description is not limited to images acquired via an image sensor. For example, the following description can also be applied to videos acquired via an image sensor. For example, an electronic device may be configured to determine, identify, and / or estimate the direction of movement of an external object for each image frame contained in the video.

[0035] Referring to Figure 3, the CPU 200 can acquire an image 302 containing an external object 301 via the image sensor 230. For example, image 302 may represent an image of a road containing the external object 301. For example, the external object 301 may include a vehicle, but is not limited to this.

[0036] For example, CPU 200 can control NPU 210 to detect an external object 301 from image 302. For example, CPU 200 can control NPU 210 to execute an object detection model 311. For example, object detection model 311 can be represented as a pre-trained model. For example, object detection model 311 can be composed of a deep learning algorithm. For example, object detection model 311 is sometimes called an object detection model. For example, object detection model 311 can include a YOLO (You Only Look Once) model.

[0037] For example, CPU 200 can provide image 302 to object detection model 311 executed by NPU 210. For example, providing image 302 to object detection model 311 can indicate that image 302 is input to object detection model 311. For example, NPU 210 can use object detection model 311 to detect external objects 301 in image 302. For example, NPU 210 can use object detection model 311 to identify external objects 301 in image 302.

[0038] For example, the NPU210 can use the object detection model 311 to identify a portion of image 302 associated with an external object 301. For example, the NPU210 can use the object detection model 311 to obtain coordinate values ​​representing a portion of image 302 associated with an external object 301. For example, the NPU210 can use the object detection model 311 to obtain data 303 associated with a portion of image 302.

[0039] For example, the operation by which the NPU210 uses the object detection model 311 to identify a portion of the image 302 related to an external object 301 is described and illustrated in more detail with reference to Figures 4a and 4b.

[0040] Figures 4a and 4b show examples of identifying external objects in an image using an object detection model.

[0041] Referring to Figure 4a, the electronic device 201 is mounted on the vehicle 202. For example, the CPU 200 can acquire an image 302 via the image sensor 230. For example, the CPU 200 can provide the image 302 to an object detection model 311 executed by the NPU 210. For example, the NPU 210 can acquire an image 400 using the object detection model 311.

[0042] For example, image 400 includes external objects 401, 402, 403, and 404. For example, image 400 represents a portion of image 400 related to an external object (e.g., portion 411, portion 412, portion 413, portion 414).

[0043] For example, the NPU 210 can identify an external object 401 within the image 400 using the object detection model 311. For example, the NPU 210 can identify a portion of the image 400 related to the external object 401 using the object detection model 311. For example, a portion 411 of the image 400 related to the external object 401 can be described as a rectangular region surrounding the external object 401. The portion of the image shown in Figure 4a is shown as a rectangular region, but this is merely illustrative. In examples that are not limited, a portion of the image may include a circular region, an elliptical region, and a triangular region.

[0044] For example, the NPU 210 can acquire data related to part 411 using the object detection model 311. For example, the data related to part 411 may include coordinate values ​​representing part 411. For example, the coordinate values ​​may include the coordinate values ​​of at least one vertex of part 411. For example, the coordinate values ​​may include coordinate values ​​normalized using the resolution of image 400.

[0045] For example, data related to section 411 may include the width value and the height value of section 411. For example, the width value and the height value of section 411 can be normalized using the resolution of image 400.

[0046] For example, data related to part 411 may include data about the type of external object 401. For example, the type of external object 401 may be called the type of vehicle. For example, the data for the type of external object 401 may be represented as a medium-sized vehicle. However, it is not limited to this.

[0047] For example, the type of external object 401 and the type of external object 402 can be represented as the same type. For example, external object 402 can be located more adjacent to the vehicle 202 than external object 401. For example, the size of part 412 may be larger than the size of part 411. For example, the width of part 412 may be wider than the width of part 411.

[0048] For example, the size of section 411 can be calculated using data related to section 411. For example, CPU 200 can obtain data related to section 411 from NPU 210. For example, CPU 200 can calculate the size of section 411 by applying the width value of section 411 to the height value of section 411.

[0049] For example, if the types of external objects are the same, the larger the size of a part (e.g., part 412), the shorter the distance between the external object associated with that part (e.g., external object 402) and the vehicle 202 may be. For example, the closer the distance between the vehicle 202 and the external object, the higher the probability of a collision may be. For example, the closer the distance between the vehicle 202 and the external object, the higher the importance of the external object may be. For example, the larger the size of the part associated with the external object, the more the CPU 200 can prioritize executing multiple calculations that define the direction identification model 313.

[0050] For example, because the size of part 412 is larger than the size of part 411, the CPU 200 can perform several calculations to define the direction discrimination model 313 using the data associated with part 412, and then perform several calculations to define the direction discrimination model 313 using the data associated with part 411.

[0051] For example, because the size of part 412 is larger than the size of part 411, the CPU 200 can perform multiple calculations to define the direction discrimination model 313 using the data associated with part 412, and refrain from (or bypass) performing multiple calculations to define the direction discrimination model 313 using the data associated with part 411.

[0052] For example, the type of external object 403 can be represented as a bus (or large vehicle). For example, the type of external object 401 can be represented as a medium-sized vehicle. For example, external object 403 may be located further away from vehicle 202 than external object 401. For example, the size of part 413 may be larger than the size of part 411. For example, when the types of external objects are distinguished, they may be different from when the types of external objects are the same. For example, when the types of external objects are distinguished, the size of part 413 may be larger than the size of part 411, but the distance between vehicle 202 and external object 403 may be longer than the distance between vehicle 202 and external object 401.

[0053] For example, when the CPU 200 identifies the distance between the vehicle 202 and the external object, it may take into account the type of the external object. For instance, when the CPU 200 identifies the distance between the vehicle 202 and the external object, it may use values ​​corresponding to certain sizes and types of the external object.

[0054] For example, the CPU 200 may determine that the importance of external object 403 is lower than that of external object 401, based on the types of external object 403 and external object 401, even though the size of part 413 is larger than that of part 411. For example, the CPU 200 may perform several calculations to define the direction identification model 313 using data related to part 411, and then perform several calculations to define the direction identification model 313 using data related to part 413.

[0055] For example, because the importance of external object 401 is higher than that of external object 403, CPU 200 can perform multiple calculations to define the direction identification model 313 using data related to part 411, and refrain from (or bypass) performing multiple calculations to define the direction identification model 313 using data related to part 413.

[0056] Referring to Figure 4b, the CPU 200 can acquire image 302 via the image sensor 230. For example, the NPU 210 can acquire image 430 by providing image 302 to the object detection model 311. For example, image 430 may include a portion 431 of image 430 related to an external object 434. For example, portion 431 can be described as a region within image 430 that includes the external object 434. For example, image 430 may include portions 432 and 433 within portion 431. For example, portion 432 can be represented as part of image 430 related to a side of the external object 434. For example, portion 433 can be represented as part of image 430 related to the front of the external object 434. The portion 433 shown in Figure 4b is merely illustrative. For example, portion 433 can be represented as part of an image related to the back of the external object 434.

[0057] For example, the NPU210 can use the object detection model 311 to obtain some data related to the external object 434. For example, some data related to the external object 434 may include coordinate values ​​representing part 432. For example, the coordinate values ​​representing part 432 may include the coordinate values ​​of at least one vertex of part 432. For example, the coordinate values ​​representing part 432 may include coordinate values ​​normalized using the width value of part 431.

[0058] For example, some data related to the external object 434 may include the width value and height value of part 432. For example, the width value and height value of part 432 can be normalized using the width value of part 431.

[0059] For example, some data related to external object 434 may include coordinate values ​​representing part 433. For example, the coordinate values ​​representing part 433 may include the coordinate values ​​of at least one vertex of part 433. For example, the coordinate values ​​representing part 433 may include coordinate values ​​normalized using the width value of part 431.

[0060] For example, some data related to the external object 434 may include the width value and height value of part 431. For example, the width value and height value of part 431 can be normalized using the width value of part 431.

[0061] For example, data 303 in Figure 3 may include some data related to the external object 434.

[0062] Referring again to Figure 3, the CPU 200 can control the NPU 210 to perform image segmentation on image 302. For example, image segmentation can be described as identifying regions in image 302 occupied by external objects 301. For example, by performing image segmentation on image 302, the NPU 210 can identify pixels corresponding to regions in image 302 occupied by external objects 301. For example, image segmentation may include identifying regions in image 302 separated by lines. For example, by performing image segmentation on image 302, the NPU 210 can identify pixels corresponding to regions in image 302 separated by lines.

[0063] For example, CPU200 can control NPU210 to execute image segmentation model312. For example, image segmentation model312 can be represented as a pre-trained model. For example, image segmentation model312 can be constructed using a deep learning algorithm. For example, image segmentation model312 is sometimes called an image segmentation model.

[0064] For example, the CPU 200 can provide the image 302 to the image segmentation model 312 executed by the NPU 210. For example, providing the image 302 to the image segmentation model 312 can indicate that the image 302 is being input to the image segmentation model 312. For example, the NPU 210 can use the image segmentation model 312 to identify the regions occupied by external objects 301 within the image 302.

[0065] For example, the operation by which the NPU 210 uses the image segmentation model 312 to identify the region occupied by the external object 301 within the image 302 is explained and illustrated in more detail with reference to Figure 5.

[0066] Figure 5 shows an example of identifying regions within an image using an image segmentation model.

[0067] Referring to Figure 5, the CPU 200 can provide image 302 to the image segmentation model 312 executed by the NPU 210. For example, the NPU 210 can use the image segmentation model 312 to acquire image 500.

[0068] For example, image 500 can be described as an image containing a road. For example, image 500 may include external objects 511, 512, 513, and 514. For example, image 500 may include lines 510 within the road. For example, NPU 210 can recognize the road within image 500 using the image segmentation model 312. For example, NPU 210 can identify the area corresponding to the road demarcated by the center line 510-1 using the image segmentation model 312.

[0069] For example, the NPU 210 can recognize lanes within the image 500 using the image segmentation model 312. For example, the image segmentation model 312 may be configured to recognize the lane in which the vehicle 202 is located. For example, the NPU 210 can identify regions corresponding to lanes separated by lanes 510-2 and 510-3 using the image segmentation model 312. For example, the NPU 210 can identify regions 501, 502, 503, and 504.

[0070] For example, region 501 can be represented as the region corresponding to the lane in which the vehicle 202 is located. For example, since region 501 corresponds to the lane in which the vehicle 202 is traveling, the importance of external object 511 within region 501 may be higher than the importance of other external objects (e.g., external objects 512, 513, and 514).

[0071] For example, the CPU 200 can perform several calculations to define the direction identification model 313 using data related to external object 511, and then perform several more calculations to define the direction identification model 313 using data related to other external objects (e.g., external object 512, external object 513, and external object 514).

[0072] For example, the CPU 200 can perform multiple calculations that define the direction identification model 313 using data related to external object 511, and refrain from (or bypass) performing multiple calculations that define the direction identification model 313 using data related to other external objects (e.g., external objects 512, 513, and 514).

[0073] For example, NPU210 can demarcate roads in image 500 with a center line 510-1. For example, NPU210 can identify a first region (e.g., including regions 503 and 504) corresponding to roads demarcated by center line 510-1 where vehicle 202 is not located, and a second region (e.g., including regions 501 and 502) corresponding to roads demarcated by center line 510-1 where vehicle 202 is located. For example, the importance of external objects in the first region (e.g., external objects 513 and 514) may be lower than the importance of external objects in the second region (e.g., external objects 511 and 512).

[0074] For example, the CPU 200 can perform a series of calculations that define the direction identification model 313 using data related to external objects in the second region (e.g., external objects 511 and 512), and refrain from (or bypass) performing a series of calculations that define the direction identification model 313 using data related to external objects in the first region (e.g., external objects 513 and 514).

[0075] Referring again to Figure 3, the CPU 200 can obtain data 303 from the NPU 210. For example, data 303 may include data related to a portion of image 302 illustrated in Figure 4a. For example, data 303 may include data related to an external object 301 in image 302 shown in Figure 4b. For example, data 303 may include data from image 302 on which image segmentation has been performed, as illustrated in Figure 5.

[0076] For example, the CPU 200 can control its processing circuitry to execute the direction identification model 313. For example, the direction identification model 313 may be configured to identify the direction of movement of the external object 301. For example, the direction identification model 313 can output notifications about the direction of movement of the external object 301 as a series of calculations defining the direction identification model 313 are performed. For example, the CPU 200 can perform a series of calculations defining the direction identification model 313 based on data 303.

[0077] For example, the CPU 200 can obtain information on the direction of movement of the external object 301 by providing data 303 to the direction identification model 313. For example, the information on the direction of movement of the external object 301 can be expressed as an angle.

[0078] For example, the movement direction information of the external object 301 can be represented as a value obtained by applying a periodic function to the movement direction angle of the external object 301. For example, the periodic function may include trigonometric functions. For example, the periodic function may be of a form in which, as the input value increases, the output value repeatedly increases linearly from -1 to 1 and decreases linearly from 1 to -1. For example, the value obtained by applying a periodic function to the movement direction angle of the external object 301 has a value between -1 and 1, and has a relatively small difference with substantially adjacent angles (e.g., 359° and 1°, which have a difference of only 2°), thus demonstrating the similarity of the movement direction angles.

[0079] For example, the CPU 200 can calculate the direction of movement angle of the external object 301 by applying a trigonometric function to the direction of movement angle and then applying the inverse trigonometric function to that value. For example, when calculating the direction of movement angle, the CPU 200 can use data 303 from a portion of the image associated with the external object 301. For example, data 303 from a portion of the image associated with the external object 301 can indicate whether the portion of the image associated with the external object 301 includes the right side or the left side of the external object 301. For example, if the portion of the image associated with the external object 301 includes the right side of the external object 301, and if the portion of the image associated with the external object 301 includes the left side of the external object 301, the value obtained by applying a trigonometric function to the direction of movement is the same, but the direction of movement angle can be different. For example, if the portion of the image associated with the external object 301 includes the right side of the external object 301, the value obtained by applying the cosine function to the direction of movement is 0.5, but the direction of movement of the external object 301 may be 60°. For example, if a portion of the image associated with the external object 301 includes the left side of the external object 301, the value obtained by applying the cosine function to the direction of movement is 0.5, but the direction of movement of the external object 301 may be 300°. For example, the CPU 200 can calculate the direction of movement angle of the external object 301 more accurately by using data 303 of a portion of the image associated with the external object 301 when calculating the direction of movement angle.

[0080] For example, the CPU 200 can determine the probability of a collision between the external object 301 and the vehicle equipped with the electronic device 201, based on the movement direction information of the external object 301. For example, the CPU 200 can output a notification 304 based on the collision probability. For example, the CPU 200 can output a notification 304 in response to the collision probability exceeding a threshold set in advance by the user. For example, the notification 304 may include the collision probability between the external object 301 and the vehicle equipped with the electronic device 201.

[0081] For example, the CPU 200 can output distance information between the vehicle equipped with the electronic device 201 and the external object 301 based on the movement direction information and data 303 of the external object 301. For example, the CPU 200 can determine the probability of collision between the external object 301 and the vehicle 202 using the distance information and the movement direction of the external object 301. For example, the distance information may be included in the notification 304. For example, a method for outputting distance information between the vehicle equipped with the electronic device 201 and the external object 301 using the movement direction information and data 303 of the external object 301 is explained and illustrated in more detail with reference to Figures 7a and 7b.

[0082] For example, the CPU 200 can determine the probability of a collision between the external object 301 and the vehicle equipped with the electronic device 201 using the distance information, the movement direction information of the external object 301, and the data 303. For example, the CPU 200 can output a notification 304 based on the collision probability. For example, the notification 304 may include the probability of a collision between the external object 301 and the vehicle equipped with the electronic device 201.

[0083] For example, the CPU 200 can transmit the notification 304 to the display 240, LED 250, and / or speaker 260. For example, the operation of the display 240, LED 250, and / or speaker 260 in response to the notification 304 is described and illustrated in more detail with reference to Figures 9a and 9b.

[0084] Figure 6 shows an example of selecting a portion of the data transmitted from the NPU so that multiple calculations of the direction discrimination model can be performed within a given time period.

[0085] Referring to Figure 6, the NPU 210 can acquire an image 610 via the image sensor 230. For example, the NPU 210 can acquire data 620 by providing the image 610 to the object detection model 311. For example, the CPU 200 can acquire data 620 from the NPU 210. For example, data 620 can be represented as output data by providing the image 610 to the object detection model 311.

[0086] For example, image 610 may contain multiple external objects. For example, data 620 may contain data related to multiple external objects within image 610. For example, data 620 may contain data related to parts within image 610 that are each related to multiple external objects within image 610.

[0087] For example, the CPU 200 can obtain a period 601 based on the results of multiple calculations of the direction identification model 313. For example, period 601 can be described as the time required for the CPU 200's processing circuit to execute the multiple calculations. For example, the CPU 200 can calculate period 601 using other images acquired before acquiring image 610. For example, period 601 can be pre-set by the user of the electronic device 201. For example, period 601 can be represented as the period between the time the NPU 210 acquires the other image via the image sensor 230 and the time the CPU 200 outputs a notification 304 based on the other image.

[0088] For example, period 602 can be described as the time required for the CPU 200 to control the NPU 210 and perform image processing on the image 610. For example, period 602 can be represented as the period between the time when the NPU 210 acquires the other image via the image sensor 230 and the time when the NPU 210 acquires data 620 by providing the image 610 to the object detection model 311.

[0089] For example, period 603 can be represented as the period between the time when the CPU 200 acquires data 620 from the NPU 210 and the time when the CPU 200 outputs notification 304 by providing data 620 to the direction identification model 313. For example, since period 601 is fixed, the longer period 602, the shorter period 603 may be.

[0090] For example, since the CPU 200 must output notification 304 within period 603, if the amount of data 620 transmitted from the NPU 210 exceeds the amount of data that can be processed within period 603, the CPU 200 can select a portion of the data 620. For example, the CPU 200 can select a portion of the data 620 so that multiple calculations of the direction discrimination model 313 are performed within period 603.

[0091] For example, the CPU 200 can determine the amount of data that the direction recognition model 313 can process within a period 603, such that multiple calculations of the direction recognition model 313 are performed within that period 603. For example, the CPU 200 can determine the number of parts of images associated with each of multiple external objects that the direction recognition model 313 can process within a period 603.

[0092] For example, the CPU 200 can determine whether the number of parts within image 610 associated with multiple external objects within image 610 exceeds the determined number. For example, the CPU 200 can select a portion of data 620 by determining that the number of parts within image 610 associated with multiple external objects within image 610 exceeds the determined number.

[0093] For example, the CPU 200 can select a portion of the image 610 that is associated with multiple external objects within the image 610, and use that portion to perform multiple calculations of the orientation recognition model 313. For example, the CPU 200 can select a portion of the image 610 that is associated with multiple external objects within the image 610, and use that portion to perform multiple calculations of the orientation recognition model 313 based on the size of each portion within the image 610. For example, a larger portion of the image 610 may have a higher priority.

[0094] For example, the CPU 200 can select a portion of the image 610 associated with multiple external objects, based on the size of each portion of the image 610 and the type of external object, to be used to perform multiple calculations of the orientation recognition model 313.

[0095] For example, the CPU 200 can increase its computation speed by selecting a portion of the data 620 on which multiple calculations of the direction identification model 313 are performed.

[0096] Figures 7a and 7b show an example of calculating the distance between a vehicle equipped with an electronic device and an external object, using coordinate values ​​representing the direction of movement of the external object and a portion of the image associated with the external object.

[0097] Referring to Figures 7a and 7b, in operation 710, the CPU 200 can identify the coordinate values ​​representing the portion 752 of the image 750 associated with the external object 751, the direction of movement 753 of the external object 751, and the type of the external object 751 by performing the operation illustrated in Figure 3.

[0098] For example, in operation 720, the CPU 200 can identify point 754 of the external object 751 in the image 750 that is closest to the vehicle 202, using the coordinate values ​​representing part 752 and the direction of movement 753 of the external object 751. For example, the CPU 200 can identify the coordinate values ​​of point 754. For example, the CPU 200 can identify the boundary between the bounding box corresponding to the side of the external object 751 and the bounding box corresponding to the back (or front) of the external object 751. For example, point 754 can be represented within the boundary. For example, point 754 may include a point that touches the boundary with the ground in the image 750.

[0099] For example, in operation 730, the CPU 200 can identify points 761, 761-1 to 761-8 surrounding the external object 751 in image 760 based on point 754 and the type of external object 751. For example, the CPU 200 can identify the width, height, and dimensions of the external object 751 using the type of external object 751. For example, the CPU 200 can identify points 761 surrounding the external object 751 using the coordinates of point 754, the direction of movement 753, the width, height, and dimensions of the external object 751. For example, the CPU 200 can identify the coordinate values ​​of each point 761.

[0100] For example, in operation 740, the CPU 200 can calculate the distance 762 between the vehicle 202 and the external object 751 using the respective coordinate values ​​761, 761-1 to 761-8 of point 761. For example, the CPU 200 can identify the distance 762 between the vehicle 202 and the external object 751 using the respective coordinate values ​​of point 761.

[0101] Figure 8 shows an example of an environment in which an electronic device outputs a notification based on the direction of movement of an external object.

[0102] Referring to Figure 8, the electronic device 201 can be attached to the vehicle 202. For example, the vehicle 202 can be shown to be traveling in direction 810. For example, the vehicle 801 can be shown to be traveling in direction 811. For example, the vehicle 801 can be shown to be traveling in a lane adjacent to the lane in which the vehicle 202 is traveling.

[0103] For example, the CPU 200 can identify the direction of movement 811 of vehicle 801 by performing the operations illustrated in Figure 3. For example, the CPU 200 can determine the probability of a collision between vehicle 202 and vehicle 801 based on the directions of movement 810 and 811. For example, the CPU 200 can output a notification 304 according to this collision probability. For example, the CPU 200 can output a notification 304 when the collision probability exceeds a threshold set in advance by the user. For example, the driver of vehicle 202 can prevent a collision by recognizing the actions that the electronic device 201 will perform in accordance with the notification 304.

[0104] For example, the CPU 200 can determine the probability of a collision between vehicle 202 and vehicle 801 based on the distance between them, the direction of movement 810, and the direction of movement 811. For example, the CPU 200 can determine that the driver of vehicle 202 can prevent a collision by recognizing the actions that the electronic device 201 will perform in accordance with notification 304. For example, the actions that the electronic device 201 will perform in accordance with notification 304 will be described and illustrated in more detail with reference to Figures 9a and 9b.

[0105] For example, vehicle 802 can be described as a vehicle entering an intersection. For example, it can be indicated that vehicle 802 is traveling in direction 812. For example, the CPU 200 can identify the direction 812 of vehicle 802. For example, the CPU 200 can determine the probability of collision between vehicle 202 and vehicle 802 based on direction 810 and direction 812. For example, the predicted travel path of vehicle 802 according to direction 812 may overlap with the predicted travel path of vehicle 202 according to direction 810. For example, the CPU 200 can identify that the probability of collision between vehicle 202 and vehicle 802 is greater than the threshold. For example, the CPU 200 can output a notification 304 to vehicle 802.

[0106] For example, vehicle 803 can be depicted as a vehicle traveling on the opposite side of the road separated by a center line. For example, it can be indicated that vehicle 803 is traveling in direction 813. For example, the CPU 200 can identify the direction 813 of vehicle 803. For example, the CPU 200 can determine the probability of collision between vehicle 202 and vehicle 803 based on direction 810 and direction 813. For example, the predicted travel path of vehicle 803 according to direction 813 and the predicted travel path of vehicle 202 according to direction 810 may not overlap. For example, the CPU 200 can identify that the probability of collision between vehicle 202 and vehicle 803 is smaller than the threshold. For example, the CPU 200 can refrain from outputting a notification 304 to vehicle 803.

[0107] For example, the electronic device 201 can, by performing the aforementioned operations, cause the driver of a vehicle equipped with the electronic device 201 to take action to prevent a collision. For example, the aforementioned operations can increase the performance of the electronic device 201 against the computation speed of the electronic device 201 by having the direction identification model 313 in the CPU 200 process the data 303 of the image 302 processed by the NPU 210.

[0108] Figures 9a and 9b show an example of an action that an electronic device performs in response to a notification.

[0109] Referring to Figure 9a, the CPU 200 can be described as outputting a notification 304 based on the direction of movement of the external object 301. For example, the CPU 200 can transmit the notification 304 to the display 240, LED 250, and / or speaker 260. For example, the notification 304 may include the probability of a collision between the external object 301 and the vehicle 202 equipped with the electronic device 201.

[0110] For example, the CPU 200 can transmit notification 304-1 to the display 240. For example, the CPU 200 can display screen 910 via the display 240 based on the probability of a collision between an external object 301 and a vehicle 202 equipped with the electronic device 201 exceeding a threshold set by the user. For example, screen 910 may include content that warns the driver of the risk of collision. For example, screen 910 may include text that indicates danger.

[0111] For example, the CPU 200 can transmit notification 304-2 to the LED 250. For example, the CPU 200 can control the LED 250 to emit light based on whether the probability of collision between the external object 301 and the vehicle 202 equipped with the electronic device 201 exceeds a threshold set by the user. For example, the CPU 200 can control the LED 250 to use higher intensity light the higher the probability of collision between the external object 301 and the vehicle 202 equipped with the electronic device 201.

[0112] For example, the CPU 200 can control the LED 250 to make it blink. For example, the CPU 200 can control the LED 250 so that the blinking rate (or period) per unit of time increases as the probability of collision between the external object 301 and the vehicle 202 to which the electronic device 201 is installed increases.

[0113] For example, the CPU 200 can control the LED 250 to emit light of different colors. For example, the LED 250 can emit blue light, green light, and / or red light. For example, the CPU 200 can control the LED 250 to emit light of different colors depending on the probability of collision between an external object 301 and a vehicle 202 equipped with the electronic device 201. For example, the CPU 200 may control the LED 250 to emit green light when the collision probability is less than 0.2. For example, the CPU 200 may control the LED 250 to emit blue light when the collision probability is greater than 0.2 and less than 0.4. For example, the CPU 200 may control the LED 250 to emit red light when the collision probability is greater than 0.4. However, it is not limited to these.

[0114] For example, the CPU 200 can transmit notification 304-3 to the speaker 260. For example, the CPU 200 can control the speaker 260 to output an audio notification 920 based on whether the probability of collision between an external object 301 and a vehicle 202 equipped with the electronic device 201 exceeds a threshold set by the user. For example, the audio notification 920 may include, but is not limited to, "Danger". For example, the volume of the audio notification 920 may increase as the probability of collision between the external object 301 and the vehicle 202 equipped with the electronic device 201 increases.

[0115] Referring to Figure 9b, the CPU 200 can output distance information between the external object 301 and the vehicle 202 based on the direction of movement of the external object 301 and the data 303. For example, the CPU 200 can use the distance information and the direction of movement of the external object 301 to determine the probability of a collision between the external object 301 and the vehicle 202. For example, Figure 9b can be illustrated as an example where the CPU 200 outputs a notification 304 that includes the probability of a collision between the external object 301 and the vehicle 202. For example, the notification 304 may include distance information between the external object 301 and the vehicle 202.

[0116] For example, the CPU 200 can transmit notification 304-1 to the display 240. For example, the CPU 200 can display screen 930 via the display 240 based on the probability of a collision between an external object 301 and a vehicle 202 equipped with the electronic device 201 exceeding a threshold set by the user. For example, screen 930 may include content that draws the driver's attention. For example, screen 930 may include text indicating the presence of the external object 301. However, it is not limited to this. For example, screen 930 may include text indicating the distance between the external object 301 and the vehicle 202.

[0117] For example, the CPU 200 can transmit notification 304-2 to the LED 250. For example, the CPU 200 can control the LED 250 to emit light based on the probability of collision between an external object 301 and a vehicle 202 equipped with the electronic device 201 exceeding a threshold set by the user. For example, the CPU 200 can control the LED 250 to use higher intensity light the higher the probability of collision between the external object 301 and the vehicle 202 equipped with the electronic device 201. For example, the CPU 200 can control the LED 250 to flash light. For example, the CPU 200 can control the LED 250 to flash more times per unit of time the higher the probability of collision between the external object 301 and the vehicle 202 equipped with the electronic device 201.

[0118] For example, the CPU 200 can transmit notification 304-3 to the speaker 260. For example, the CPU 200 can control the speaker 260 to output an audio notification 940 based on whether the probability of a collision between an external object 301 and a vehicle 202 equipped with the electronic device 201 exceeds a threshold set by the user. For example, the audio notification 940 may include the message, "You are approaching a vehicle ahead." However, it is not limited to this. For example, the volume of the audio notification 940 may increase as the probability of a collision between the external object 301 and the vehicle 202 equipped with the electronic device 201 increases.

[0119] Figure 10 shows an example of the structure of an artificial intelligence model.

[0120] Referring to Figure 10, an example of model 1000 can be shown, which is represented by a set of parameters stored in the memory 220 of the electronic device 201. For example, object detection model 311, image segmentation model 312, and direction recognition model 313 may be examples of model 1000.

[0121] At least a portion of Model 1000 may include multiple layers. For example, Model 1000 may include an input layer 1010, one or more hidden layers 1020, and an output layer 1030. The input layer 1010 may receive a vector representing input data (for example, a vector having elements corresponding to the number of nodes included in the input layer 1010). Signals generated at each node in the input layer 1010, generated by the input data, may be transmitted from the input layer 1010 to the hidden layer 1020. The output layer 1030 may generate output data for Model 1000 based on one or more signals received from the hidden layer 1020. This output data may include, for example, a vector having elements corresponding to the number of nodes included in the output layer 1030.

[0122] Referring to Figure 10, one or more hidden layers 1020 can be located between the input layer 1010 and the output layer 1030, and can convert the input data transmitted through the input layer 1010 into predictable values. The input layer 1010, one or more hidden layers 1020, and the output layer 1030 can contain multiple nodes. The one or more hidden layers 1020 are not limited to the feedforward-based topology shown, and may be, for example, convolutional filters or fully connected layers in a CNN (convolutional neural network), or various types of filters or layers grouped based on special functions or features. In one embodiment, the one or more hidden layers 1020 may be layers based on a recurrent neural network (RNN) where the output values ​​are fed back into the hidden layer at the current time. In one example, the input layer 1010, one or more hidden layers 1020, and / or the output layer 1030 may be several layers of a transformer model. Model 1000, according to one embodiment, can form a deep neural network by including a large number of hidden layers 1020. Training a deep neural network is called deep learning. Among the nodes of Model 1000, those included in the hidden layer 1020 are called hidden nodes.

[0123] Nodes in the input layer 1010 and one or more hidden layers 1020 can be connected to each other via connection lines with connection weights, and nodes in the hidden layers and output layers can also be connected to each other via connection lines with connection weights. Tuning and / or training the model 1000 may mean changing the connection weights between nodes in each of the layers included in the model 1000 (e.g., the input layer 1010, one or more hidden layers 1020, and the output layer 1030). Tuning the model 1000 can be performed, for example, based on supervised learning and / or unsupervised learning.

[0124] For example, an electronic device can modify policy information used by Model 1000 to control an agent based on the interaction between the agent and the environment. The policy information is a set of rules used by the electronic device, using a neural network, to determine the agent's actions within the environment. The electronic device can train the neural network and modify its policy information based on the interaction between the agent and the environment. For example, the policy information may be modified to determine the optimal actions and / or sequences of actions for the agent to achieve its rewards and / or goals. An electronic device according to one embodiment can trigger a modification of the policy information by Model 1000 to maximize the agent's goals and / or rewards resulting from the interaction.

[0125] Model 1000 can extract feature values ​​from the input data and compare the similarity of the vector (or embedding vector) generated from the extracted feature values ​​with a reference vector (or reference embedding vector) stored in the electronic device 201. Based on the result of this comparison, it can generate identification information. Model 1000 can be trained via a loss function that corresponds to the difference between the ground truth and the output data. For example, this loss function can include a softmax loss function, a Euclidean distance-based loss function, or an angular-based (or cosine margin-based) loss function.

[0126] According to one embodiment, model 1000 may require training data for training. For example, the training data can be referred to as a dataset. For example, the training data for model 1000 may include simulation images (or videos) including virtual roads. For example, the training data for model 1000 may include simulation data including virtual vehicles. For example, by training model 1000 using simulation data, the cost used to train model 1000 can be reduced.

[0127] According to one embodiment, simulation data can be used to train the model 1000 (e.g., object detection model 311, image segmentation model 312). For example, the simulation data may include videos (or images) based on simulations. For example, each simulation-based video may have different virtual vehicle types, heights of image sensors mounted on the virtual vehicle, angles of the image sensors mounted on the virtual vehicle, field of view of the image sensors mounted on the virtual vehicle, and / or resolutions of images acquired through the image sensors. For example, virtual vehicle types can include buses, trucks, light SUVs, vans, sedans, and cargo trucks. For example, the height of the image sensor mounted on the virtual vehicle can be greater than 1.1m and less than 2.0m. For example, the field of view of the image sensor mounted on the virtual vehicle can be greater than 35° and less than 120°. For example, image resolutions can include 16:9 and 4:3.

[0128] According to one embodiment, simulation data can be used to train model 1000 (e.g., direction recognition model 313). For example, the simulation data may include data necessary to project a 3D image onto a 2D image. For example, the simulation data may include simulation-based videos. For example, each simulation-based video may have different types of virtual vehicles, heights of image sensors mounted on the virtual vehicles, angles of the image sensors mounted on the virtual vehicles, field of view of the image sensors mounted on the virtual vehicles, and / or resolution of images acquired through the image sensors. For example, the simulation data may include coordinate data of virtual vehicles within each simulation-based video. For example, the coordinate data of virtual vehicles may include the width of the bounding box corresponding to the virtual vehicle, the height of the bounding box, and the coordinate values ​​of at least one vertex of the bounding box. For example, the simulation data may have normalized values. For example, normalized simulation data may be represented by values ​​between -2 and 2. For example, the normalization of the simulation data may be based on the magnitude of the resolution.

[0129] According to one embodiment, the simulation data may include data on the direction of movement of a virtual vehicle within each of the simulation-based videos. For example, the data on the direction of movement of a virtual vehicle can be represented as a value obtained by applying a periodic function to the direction of movement angle of the virtual vehicle. For example, the periodic function may include a trigonometric function. For example, the periodic function may be of a form in which, as the input value increases, the output value repeatedly increases linearly from -1 to 1 and decreases linearly from 1 to -1. For example, the value obtained by applying a periodic function to the direction of movement angle of a virtual vehicle has a value between -1 and 1, and has a relatively small difference with substantially adjacent angles (e.g., 359° and 1°, which have a difference of only 2°), thus demonstrating similarity of the direction of movement angles.

[0130] For example, the quality of Model 1000 of the electronic device 201 (e.g., Direction Identification Model 313) can be enhanced by training it using simulation data, as it can accurately identify the direction angle of movement of a virtual vehicle. For instance, the direction identification quality of Model 1000 trained using simulation data may be higher than that of other models trained using actual vehicle data.

[0131] Figure 11 shows an example of a block diagram illustrating an autonomous driving system for a vehicle according to one embodiment.

[0132] The autonomous driving system 1100 for a vehicle shown in Figure 11 may include a deep learning network comprising a sensor 1103, an image preprocessor 1105, a deep learning network 1107, an artificial intelligence (AI) processor 1109, a vehicle control module 1111, a network interface 1113, and a communication unit 1115. In various embodiments, each element can be connected via various interfaces. For example, sensor data sensed and output by the sensor 1103 can be fed to the image preprocessor 1105. Sensor data processed by the image preprocessor 1105 can be fed to the deep learning network 1107, which is run by the AI ​​processor 1109. The output of the deep learning network 1107, run by the AI ​​processor 1109, can be fed to the vehicle control module 1111. Intermediate results of the deep learning network 1107, run by the AI ​​processor 1109, can be fed back to the AI ​​processor 1109. In various embodiments, the network interface 1113 transmits autonomous driving route information and / or autonomous driving control commands for autonomous driving of the vehicle to the internal block configuration by communicating with an in-vehicle electronic device (e.g., electronic device 201 in Figure 2). In one embodiment, the network interface 1113 may be used to transmit sensor data acquired via sensor 1103 to an external server. In some embodiments, the autonomous driving control system 1100 may include additional or fewer components as appropriate. For example, in some embodiments, the image preprocessor 1105 may be an optional component. In another example, a post-processing component (not shown) may be included in the autonomous driving control system 1100 to perform post-processing on the output of the deep learning network 1107 before the output is provided to the vehicle control module 1111.

[0133] In some embodiments, sensor 1103 may include one or more sensors. In various embodiments, sensor 1103 may be mounted at different locations on the vehicle. Sensor 1103 may be oriented in one or more different directions. For example, sensor 1103 may be mounted on the front, sides, rear, and / or roof of the vehicle so as to face directions such as forward-facing, rear-facing, and side-facing. In some embodiments, sensor 1103 may be an image sensor such as a high dynamic range camera. In some embodiments, sensor 1103 includes non-visual sensors. In some embodiments, sensor 1103 includes radar, LiDAR (Light Detection and Ranging), and / or ultrasonic sensors in addition to image sensors. In some embodiments, sensor 1103 is not mounted on a vehicle having a vehicle control module 1111. For example, sensor 1103 may be included as part of a deep learning system for capturing sensor data and may be mounted on the environment or road, and / or on surrounding vehicles.

[0134] In some embodiments, the image pre-processor 1105 can be used to pre-process sensor data from the sensor 1103. For example, the image pre-processor 1105 can be used to pre-process sensor data, to split sensor data into one or more components, and / or to post-process one or more components. In some embodiments, the image pre-processor 1105 may be a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a specialized image processor. In various embodiments, the image pre-processor 1105 may be a tone-mapper processor for processing high dynamic range data. In some embodiments, the image pre-processor 1105 may be a component of the AI ​​processor 1109.

[0135] In some embodiments, the deep learning network 1107 may be a deep learning network for executing control commands to control an autonomous vehicle. For example, the deep learning network 1107 may be an artificial neural network, such as a convolutional neural network (CNN), trained using sensor data, and the output of the deep learning network 1107 is provided to the vehicle control module 1111.

[0136] In some embodiments, the artificial intelligence (AI) processor 1109 may be a hardware processor for running a deep learning network 1107. In some embodiments, the AI ​​processor 1109 is a specialized AI processor for performing inference on sensor data via a convolutional neural network (CNN). In some embodiments, the AI ​​processor 1109 may be optimized for the bit depth of the sensor data. In some embodiments, the AI ​​processor 1109 may be optimized for deep learning operations such as neural network operations including convolution, dot product, vector, and / or matrix operations. In some embodiments, the AI ​​processor 1109 may be implemented via multiple graphics processing units (GPUs) capable of effectively performing parallel processing.

[0137] In various embodiments, the AI ​​processor 1109 can be coupled via an input / output interface to a memory configured to provide the AI ​​processor with instructions triggered to determine the results of machine learning used to operate the vehicle at least partially autonomously, while the AI ​​processor 1109 is running, and to perform deep learning analysis on sensor data received from the sensor 1103. In some embodiments, a Vehicle Control Module 1111 can process vehicle control instructions output from the artificial intelligence (AI) processor 1109 and can be used to translate the output of the AI ​​processor 1109 into instructions to control each vehicle module in order to control various modules of the vehicle. In some embodiments, the Vehicle Control Module 1111 is used to control the vehicle for autonomous driving. In some embodiments, the Vehicle Control Module 1111 can adjust the steering and / or speed of the vehicle. For example, the Vehicle Control Module 1111 can be used to control the driving of the vehicle, such as deceleration, acceleration, steering, lane changes, and lane keeping. In some embodiments, the vehicle control module 1111 can generate control signals for controlling vehicle lighting, such as brake lights, turn signals, and headlights. In some embodiments, the vehicle control module 1111 can be used to control vehicle audio-related systems, such as the vehicle's sound system, vehicle's audio warnings, vehicle's microphone system, and vehicle's horn system.

[0138] In some embodiments, the vehicle control module 1111 can be used to control notification systems, including warning systems for informing passengers and / or the driver of driving events such as access to an intended destination or a potential collision. In some embodiments, the vehicle control module 1111 may be used to adjust sensors such as the vehicle's sensor 1103. For example, the vehicle control module 1111 can modify the orientation of the sensor 1103, change the output resolution and / or format type of the sensor 1103, increase or decrease the capture rate, adjust the dynamic range, and adjust the camera focus. Furthermore, the vehicle control module 1111 can turn the operation of the sensors on or off individually or collectively.

[0139] In some embodiments, the vehicle control module 1111 can be used to modify the parameters of the image preprocessor 1105, such as by changing the frequency range of the filter, adjusting the edge detection parameter for feature and / or object detection, or adjusting the channels and bit depth. In various embodiments, the vehicle control module 1111 may be used to control the vehicle's autonomous driving and / or driver assistance functions.

[0140] In some embodiments, the network interface 1113 can serve as an internal interface between the block configuration of the autonomous driving control system 1100 and the communication unit 1115. Specifically, the network interface 1113 may be a communication interface for receiving and / or transmitting data, including voice data. In various embodiments, the network interface 1113 can connect to an external server to connect voice calls via the communication unit 1115, receive and / or send text messages, transmit sensor data, update the vehicle's software in the autonomous driving system, or to update the software of the vehicle's autonomous driving system.

[0141] In various embodiments, the communication unit 1115 may include various wireless interfaces, such as cellular or Wi-Fi. For example, a network interface 1113 can be used to receive updates to operating parameters and / or instructions for the sensor 1103, image preprocessor 1105, deep learning network 1107, AI processor 1109, and vehicle control module 1111 from an external server connected via the communication unit 1115. For example, the machine learning model of the deep learning network 1107 can be updated using the communication unit 1115. In yet another example, the communication unit 1115 may be used to update operating parameters of the image preprocessor 1105, such as image processing parameters, and / or the firmware of the sensor 1103.

[0142] In another embodiment, the communication unit 1115 can be used to activate communications for emergency services and emergency contact in the event of an accident or near-accident. For example, in a collision event, the communication unit 1115 may be used to call emergency services for assistance and to notify external parties of collision details and the location of the vehicle. In various embodiments, the communication unit 1115 may update or acquire the estimated time of arrival and / or the location of the destination.

[0143] According to one embodiment, the autonomous driving system 1100 shown in Figure 11 may be composed of the vehicle's electronic devices 201. According to one embodiment, the AI ​​processor 1109 of the autonomous driving system 1100 can be controlled to train the vehicle's autonomous driving software by inputting autonomous driving deactivation event-related information into the training set data of a deep learning network when an autonomous driving deactivation event occurs from the user during autonomous driving of the vehicle.

[0144] Figures 12 and 13 show an example of a block diagram illustrating an autonomous mobile vehicle according to one embodiment. Figure 14 shows examples of gateways related to user devices according to various embodiments.

[0145] Referring to Figure 12, the autonomous mobile unit 1200 according to this embodiment may include a control device 1300, sensing modules 1204a, 1204b, 1204c, 1204d, an engine 1206, and a user interface 1208.

[0146] The autonomous mobile unit 1200 may have an autonomous driving mode or a manual mode. For example, it may switch from manual mode to autonomous driving mode, or from autonomous driving mode to manual mode, according to user input received via the user interface 1208.

[0147] When the mobile unit 1200 is operating in autonomous driving mode, the autonomous mobile unit 1200 can be operated under the control of the control device 1300.

[0148] In this embodiment, the control device 1300 may include a controller 1320 comprising a memory 1322 and a processor 1324, a sensor 1310, a communication device 1330, and an object detection device 1340.

[0149] Here, the object detection device 1340 can perform all or part of the functions of the distance measuring device.

[0150] In other words, in this embodiment, the object detection device 1340 is a device for detecting objects located outside the moving body 1200, and the object detection device 1340 can detect objects located outside the moving body 1200 and generate object information according to the detection result.

[0151] Object information may include information about the existence of an object, the object's location, the distance between the moving object and the object, and the relative velocity between the moving object and the object.

[0152] The objects may include a variety of objects placed outside the moving body 1200, such as lanes, other vehicles, pedestrians, traffic signals, lights, roads, structures, speed limiters, terrain objects, and animals. Here, traffic signals may be a concept including traffic signal lights, traffic signs, patterns or text drawn on the road surface. And the lights may be light generated from lamps on other vehicles, light generated by streetlights, or sunlight.

[0153] Structures can be objects located around roads and fixed to the ground. For example, structures can include streetlights, street trees, buildings, utility poles, traffic lights, and bridges. Topographical features can include mountains, hills, and so on.

[0154] Such an object detection device 1340 may include a camera module. The controller 1320 can extract object information from an external image captured by the camera module and have the controller 1320 process that information.

[0155] Furthermore, the object detection device 1340 may further include an imaging device for recognizing the external environment. In addition to LIDAR, RADAR, GPS devices, odoometry devices and other computer vision devices, ultrasonic sensors, infrared sensors, etc., can be used, and these devices may be selected or operated simultaneously as needed to enable more accurate sensing.

[0156] On the other hand, the distance measuring device according to one embodiment of the present invention can calculate the distance between the autonomous mobile body 1200 and an object, and control the operation of the mobile body based on the calculated distance in cooperation with the control device 1300 of the autonomous mobile body 1200.

[0157] For example, if there is a possibility of collision depending on the distance between the autonomous mobile unit 1200 and the object, the autonomous mobile unit 1200 can control its brakes to reduce its speed or stop. As another example, if the object is a moving object, the autonomous mobile unit 1200 can control its speed to maintain a certain distance or more from the object.

[0158] Such a distance measuring device according to one embodiment of the present invention can be configured as a module within the control device 1300 of the autonomous mobile vehicle 1200. That is, the memory 1322 and processor 1324 of the control device 1300 can implement the collision prevention method according to the present invention in software.

[0159] Furthermore, sensor 1310 can connect to sensing modules 1204a, 1204b, 1204c, and 1204d to acquire various sensing information about the internal / external environment of the moving body. Here, sensor 1310 may include attitude sensors (e.g., yaw sensor, roll sensor, pitch sensor, collision sensor, wheel sensor, speed sensor, tilt sensor, weight sensor, heading sensor, gyro sensor, position module, forward / reverse moving body sensor, battery sensor, fuel sensor, tire sensor, steering sensor based on steering wheel rotation, internal temperature sensor of the moving body, internal humidity sensor of the moving body, ultrasonic sensor, illuminance sensor, acceleration pedal position sensor, brake pedal position sensor, etc.).

[0160] As a result, the sensor 1310 can acquire sensing signals for mobile body attitude information, mobile body collision information, mobile body direction information, mobile body position information (GPS information), mobile body angle information, mobile body speed information, mobile body acceleration information, mobile body tilt information, mobile body forward / reverse information, battery information, fuel information, tire information, mobile body lamp information, mobile body internal temperature information, mobile body internal humidity information, steering wheel rotation angle, external illumination of the mobile body, pressure applied to the accelerator pedal, pressure applied to the brake pedal, and so on.

[0161] Furthermore, the sensor 1310 may also include other sensors such as an acceleration pedal sensor, a pressure sensor, an engine speed sensor, an airflow sensor (AFS), an intake air temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a top-to-center temperature sensor, and a crank angle sensor (CAS).

[0162] In this way, the sensor 1310 can generate mobile object state information based on sensing data.

[0163] The wireless communication device 1330 is configured to perform wireless communication between autonomous mobile units 1200. For example, it enables the autonomous mobile unit 1200 to communicate with a user's mobile phone, or other wireless communication devices 1330, other mobile units, a central unit (traffic control device), a server, etc. The wireless communication device 1330 can send and receive wireless signals according to the connected wireless protocol. Wireless communication protocols include, but are not limited to, Wi-Fi, Bluetooth, LTE (Long-Term Evolution), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), and GSM (Global Systems for Mobile Communications).

[0164] Furthermore, the autonomous mobile unit 1200 in this embodiment can also achieve communication with other mobile units via the wireless communication device 1330. That is, the wireless communication device 1330 can communicate with other mobile units on the road via vehicle-to-vehicle (V2V) communication. The autonomous mobile unit 1200 can send and receive information such as driving warnings and traffic information via vehicle-to-vehicle communication, and can also request or receive information from other mobile units. For example, the wireless communication device 1330 can perform V2V communication using a dedicated short-range communication (DSRC) device or a cellular-V2V (C-V2V) device. In addition to vehicle-to-vehicle communication, communication between a vehicle and other objects (e.g., electronic devices carried by pedestrians) (V2X, Vehicle to Everything communication) can also be achieved via the wireless communication device 1330.

[0165] Furthermore, the wireless communication device 1330 can acquire information generated by various forms of mobility, including infrastructure located on roads (traffic signal lights, CCTV, RSU, eNode B, etc.) or other autonomous driving / non-autonomous driving vehicles, via a non-terrestrial network rather than a terrestrial network, as information for performing autonomous driving of the autonomous driving mobile unit 1200.

[0166] For example, the wireless communication device 1330 can communicate wirelessly with non-terrestrial network components such as low Earth orbit (LEO) satellite systems, medium Earth orbit (MEO) satellite systems, geostationary orbit (GEO) satellite systems, and high-altitude platform (HAP) systems via an antenna mounted on the autonomous mobile unit 1200 specifically for non-terrestrial networks.

[0167] For example, wireless communication device 1330 is currently being discussed by 3GPP, etc., regarding 5G NR NTN (5 th It can, but is not limited to, wireless communication with various NTN platforms in accordance with wireless connection standards compliant with the Generation New Radio Non-Terrestrial Network (DC) standard.

[0168] In this embodiment, the controller 1320 can select a platform capable of performing NTN communication appropriately, taking into account various information such as the position of the autonomous mobile vehicle 1200, the current time, and available power, and can control the wireless communication device 1330 to perform wireless communication with the selected platform.

[0169] In this embodiment, the controller 1320 is a unit that controls the overall operation of each unit within the mobile body 1200, and may be configured by the mobile body manufacturer at the time of manufacture, or may be further configured after manufacture to perform autonomous driving functions. Alternatively, it may include configurations to perform continuous additional functions through upgrades of the controller 1320 configured at the time of manufacture. Such a controller 1320 may be called an ECU (Electronic Control Unit).

[0170] The controller 1320 collects various data from connected sensors 1310, object detection devices 1340, communication devices 1330, etc., and based on the collected data, can transmit control signals to other components of the mobile body, including sensors 1310, engine 1206, user interface 1208, communication devices 1330, and object detection devices 1340. Although not shown, it can also transmit control signals to acceleration devices, braking systems, steering devices, or navigation devices related to the movement of the mobile body.

[0171] In this embodiment, the controller 1320 can control the engine 1206. For example, it can detect the speed limit of the road on which the autonomous mobile vehicle 1200 is traveling and control the engine 1206 so that the travel speed does not exceed the speed limit, or it can control the engine 1206 to accelerate the travel speed of the autonomous mobile vehicle 1200 within a range that does not exceed the speed limit.

[0172] Furthermore, the controller 1320 can determine whether the approach to or departure from a lane is in accordance with normal driving conditions or other driving conditions when the autonomous mobile unit 1200 is approaching or deviating from a lane while it is driving, and can control the engine 1206 to control the driving of the mobile unit according to the determination result. Specifically, the autonomous mobile unit 1200 can detect lanes formed on both sides of the road on which the mobile unit is driving. In this case, the controller 1320 determines whether the autonomous mobile unit 1200 is approaching or deviating from a lane, and if it determines that the autonomous mobile unit 1200 is approaching or deviating from a lane, it can determine whether such driving is due to normal driving conditions or other driving conditions. Here, an example of normal driving conditions may be a situation where the mobile unit needs to change lanes. An example of other driving conditions may be a situation where the mobile unit does not need to change lanes. If the controller 1320 determines that the autonomous vehicle 1200 is either close to or has deviated from its lane in a situation where the vehicle does not need to change its lane, it can control the autonomous vehicle 1200's movement so that it does not deviate from its lane and can drive normally.

[0173] If another moving object or obstacle is present in front of the moving object, the engine 1206 or braking system can be controlled to decelerate the moving object, and in addition to speed, the trajectory, route, and steering angle can be controlled. Alternatively, the controller 1320 may control the movement of the moving object by generating necessary control signals in response to recognition information of other external environments such as the lane the moving object is traveling in and traffic signals.

[0174] In addition to generating its own control signals, the controller 1320 can also control the movement of a mobile object by communicating with a peripheral mobile object or a central server and transmitting commands to control the peripheral device based on the received information.

[0175] Furthermore, if the position of the camera module (for example, included in the object detection device 1340) is changed or the field of view is changed, accurate recognition of the moving object or lane according to this embodiment may become difficult. Therefore, to prevent this, the controller 1320 can also generate a control signal to perform calibration of the camera module. Accordingly, in this embodiment, by generating a calibration control signal in the camera module, the controller 1320 can continuously maintain the normal mounting position, direction, field of view, etc. of the camera module even if the mounting position of the camera module is changed due to vibrations, shocks, etc. that occur as the autonomous mobile vehicle 1200 moves. The controller 1320 can generate a control signal to perform calibration of the camera module if the initial mounting position, direction, field of view information of the camera module stored in advance and the initial mounting position, direction, field of view information of the camera module measured during the movement of the autonomous mobile vehicle 1200 change by more than a critical value.

[0176] In this embodiment, the controller 1320 may include a memory 1322 and a processor 1324. The processor 1324 can execute software stored in the memory 1322 in accordance with control signals from the controller 1320. Specifically, the controller 1320 stores data and instructions for performing the lane detection method according to the present invention in the memory 1322, and these instructions can be executed by the processor 1324 to perform one or more of the methods disclosed herein.

[0177] In this case, memory 1322 may be stored on a recording medium executable by a non-volatile processor 1324. Memory 1322 can store software and data via appropriate internal and external devices. Memory 1322 can consist of RAM (random access memory), ROM (read-only memory), a hard disk, and a memory 1322 device connected to a dongle.

[0178] Memory 1322 can store at least the operating system (OS), user applications, and executable instructions. Memory 1322 can also store application data and array data structures.

[0179] The processor 1324 is a microprocessor or suitable electronic processor, which may be a controller, microcontroller, or state machine.

[0180] The processor 1324 can be implemented as a combination of computing devices, which can consist of a digital signal processor, a microprocessor, or an appropriate combination thereof.

[0181] On the other hand, the autonomous mobile vehicle 1200 may further include a user interface 1208 for user input to the control device 1300 described above. The user interface 1208 can allow the user to input information through appropriate interaction. For example, this can be done with a touchscreen, keypad, or operation buttons. The user interface 1208 transmits input or commands to the controller 1320, and the controller 1320 can perform control operations of the mobile vehicle in response to the input or commands.

[0182] Furthermore, the user interface 1208 is an external device of the autonomous mobile unit 1200 and can communicate with the autonomous mobile unit 1200 via the wireless communication device 1330. For example, the user interface 1208 can be linked with a mobile phone, tablet, or other computer device.

[0183] Furthermore, although the autonomous mobile vehicle 1200 has been described in this embodiment as including an engine 1206, it may also include other types of propulsion systems. For example, the mobile vehicle may be powered by electric energy, hydrogen energy, or a hybrid system combining the two. Therefore, the controller 1320 may include the propulsion mechanism of the autonomous mobile vehicle 1200's propulsion system and provide control signals therefor to the configuration of each propulsion mechanism.

[0184] The detailed configuration of the control device 1300 according to this embodiment will be described in more detail below with reference to Figure 13.

[0185] The control device 1300 includes a processor 1324. The processor 1324 may be a general-purpose single-chip or multi-chip microprocessor, a dedicated microprocessor, a microcontroller, a programmable gate array, etc. The processor is sometimes called a central processing unit (CPU). In this embodiment, the processor 1324 can also be used in combination with multiple other processors.

[0186] The control device 1300 also includes a memory 1322. The memory 1322 can be any electronic component capable of storing electronic information. The memory 1322 can also include a single memory or a combination of memories 1322.

[0187] The data and instructions 1322a for executing the distance measurement method of the distance measuring device according to the present invention may be stored in memory 1322. When the processor 1324 executes the instruction 1322a, all or part of the instruction 1322a and the data 1322b necessary for the execution of the instruction may be loaded onto the processor 1324 1324a, 1324b.

[0188] The control device 1300 may include a transmitter 1330a, a receiver 1330b, or a transceiver 1330c for transmitting and receiving signals. One or more antennas 1332a, 1332b may be electrically connected to the transmitter 1330a, the receiver 1330b, or each transceiver 1330c, and may also include additional antennas.

[0189] The control device 1300 may also include a digital signal processor (DSP) 1370. The DSP 1370 can enable the mobile device to process digital signals quickly.

[0190] The control device 1300 may include a communication interface 1380. The communication interface 1380 may include one or more ports and / or communication modules for connecting other devices to the control device 1300. The communication interface 1380 can enable interaction between a user and the control device 1300.

[0191] Various configurations of the control device 1300 may be connected together by one or more buses 1390, which may include a power bus, a control signal bus, a status signal bus, a data bus, and so on. Depending on the control of the processor 1324, the configurations can communicate information with each other via the buses 1390 and perform desired functions.

[0192] On the other hand, in various embodiments, the control unit 1300 may be associated with a gateway for communication with the security cloud. For example, referring to Figure 14, the control unit 1300 may be associated with a gateway 1405 for providing information obtained from at least one of the components 1401 to 1404 of the vehicle 1400 to the security cloud 1406. For example, the gateway 1405 may be contained within the control unit 1300. In another example, the gateway 1405 may consist of a separate device within the vehicle 1400 that is distinct from the control unit 1300. The gateway 1405 connects the networks within the vehicle 1400, which are protected by a software management cloud 1409, a security cloud 1406, and the in-vehicle security software 1410, which have different networks.

[0193] For example, component 1401 could be a sensor. For example, this sensor could be used to acquire information about at least one of the conditions of the vehicle 1400 or the conditions of the surroundings of the vehicle 1400. For example, component 1401 could include sensor 1310.

[0194] For example, component 1402 may be an ECU (electronic control unit). For example, an ECU can be used for engine control, transmission control, airbag control, and tire pressure management.

[0195] For example, component 1403 could be an instrument cluster. For example, an instrument cluster might mean a panel on the dashboard located directly in front of the driver's seat. For example, an instrument cluster may be configured to display information necessary for driving to the driver (or passenger). For example, an instrument cluster could be used to display at least one of the following: a visual element indicating the engine's revolutions per minute (RPM, or rotates per minute), a visual element indicating the vehicle's speed, a visual element indicating the amount of remaining fuel, a visual element indicating the gear status, or a visual element indicating information obtained via component 1401.

[0196] For example, component 1404 could be a telematics device. For example, this telematics device could mean a device that combines wireless communication technology and GPS (global positioning system) technology to provide various mobile communication services within the vehicle 1400, such as location information and safe driving. For example, the telematics device can be used to connect the vehicle 1400 with the driver, a cloud (e.g., security cloud 1406), and / or the surrounding environment. For example, the telematics device may be configured to support high bandwidth and low latency for 5G NR standard technologies (e.g., 5G NR V2X technology, 5G NR NTN (Non-Terrestrial Network) technology). For example, the telematics device may be configured to support autonomous driving of the vehicle 1400.

[0197] For example, gateway 1405 can be used to connect the network within the vehicle 1400 with the external networks, namely software management cloud 1409 and security cloud 1406. For example, software management cloud 1409 can be used to update or manage at least one piece of software necessary for the operation and management of vehicle 1400. For example, software management cloud 1409 can work in conjunction with in-car security software 1410 installed in the vehicle. For example, in-car security software 1410 can be used to provide security functions within vehicle 1400. For example, in-car security software 1410 can encrypt data transmitted and received over the in-car network using an encryption key obtained from an external authorized server for encryption of the in-car network. In various embodiments, the encryption key used by in-car security software 1410 can be generated in correspondence with vehicle identification information (vehicle license plate, vehicle VIN (vehicle identification number)) or information uniquely assigned to each user (such as user identification information).

[0198] In various embodiments, the gateway 1405 can transmit data encrypted by the in-vehicle security software 1410 based on the encryption key to the software management cloud 1409 and / or security cloud 1406. The software management cloud 1409 and / or security cloud 1406 can identify which vehicle or user the data was received from by decrypting the data encrypted by the encryption key of the in-vehicle security software 1410 using a decryption key. For example, since this decryption key is a unique key corresponding to the encryption key, the software management cloud 1409 and / or security cloud 1406 can identify the sender of the data (e.g., a vehicle or a user) based on the data decrypted via the decryption key.

[0199] For example, gateway 1405 may be configured to support in-vehicle security software 1410 and may be associated with control unit 1300. For example, gateway 1405 may be associated with control unit 1300 to support the connection between client device 1407 connected to security cloud 1406 and control unit 1300. In another example, gateway 1405 may be associated with control unit 1300 to support the connection between third-party cloud 1408 connected to security cloud 1406 and control unit 1300. However, it is not limited to these examples.

[0200] In various embodiments, the gateway 1405 can be used to connect the vehicle 1400 to a software management cloud 1409 for managing the vehicle's operating software. For example, the software management cloud 1409 can monitor whether an operating software update for the vehicle 1400 is required and, based on monitoring that an operating software update for the vehicle 1400 has been requested, provide data for updating the vehicle 1400's operating software via the gateway 1405. Alternatively, the software management cloud 1409 can receive a user request from the vehicle 1400 via the gateway 1405 to update the vehicle 1400's operating software and, based on this receipt, provide data for updating the vehicle 1400's operating software. However, it is not limited to these embodiments.

[0201] Figure 15 is a diagram illustrating the operation of an electronic device for training a neural network based on a set of training data, according to one embodiment.

[0202] The operation described with reference to Figure 15 can be performed by the aforementioned electronic device (electronic device 201 in Figure 2).

[0203] Referring to Figure 15, in operation 1502, an electronic device according to one embodiment can acquire a set of training data. The electronic device can acquire a set of training data for supervised learning. The training data may include input data and a pair of ground truth data corresponding to this input data. The ground truth data can represent output data that the neural network attempts to obtain from receiving the input data, which is this pair of ground truth data. The ground truth data can be acquired by the aforementioned electronic device.

[0204] For example, when training a neural network to recognize an image, the training data may include information about the image and one or more objects contained within the image. This information may include the classification (category or class) of the object identifiable through the image. This information may include the position, width, height, and / or size of the visual object corresponding to the object within the image. The set of training data identified through operation 1502 may include multiple pairs of training data. In the above example of training a neural network to recognize an image, the set of training data identified by the electronic device may include multiple images and the basis truth data corresponding to each of those multiple images.

[0205] Referring to Figure 15, in operation 1504, an electronic device according to one embodiment can train a neural network based on a set of training data. In one embodiment in which the neural network is trained based on supervised learning, the electronic device can input input data included in the training data into the input layer of the neural network. An example of a neural network including the input layer will be described with reference to Figure 16. From the output layer of the neural network that has received the input data via the input layer, the electronic device can obtain the output data of the neural network corresponding to the input data.

[0206] In one embodiment, training of operation 1504 may be performed based on the difference between the output data and the basis truth data included in the training data and corresponding to the input data. For example, the electronic device may adjust one or more parameters related to the neural network (e.g., weights, as described later with reference to Figure 16) based on a gradient descent algorithm so that the difference decreases. The operation of the electronic device to adjust one or more parameters is sometimes called tuning the neural network. The electronic device may perform tuning of the neural network based on the output data using a function defined to evaluate the performance of the neural network, such as a cost function. The difference between the output data and the basis truth data described above may be included as an example of a cost function.

[0207] Referring to Figure 15, in operation 1506, an electronic device according to one embodiment can determine whether valid output data has been output from the neural network trained by operation 1504. Valid output data can mean that the difference (or cost function) between the output data and the basis truth data satisfies the conditions set for using the neural network. For example, if the mean and / or maximum value of the difference between the output data and the basis truth data is less than or equal to a specified threshold, the electronic device can determine that valid output data has been output from the neural network.

[0208] If no valid output data is output from the neural network (operation 1506 - no), the electronic device can repeatedly perform training of the neural network based on operation 1504. The embodiments are not limited thereto, and the electronic device can repeatedly perform operations 1502 and 1504.

[0209] With valid output data obtained from the neural network (operation 1506 - yes), the electronic device according to one embodiment can use the trained neural network based on operation 1508. For example, the electronic device can input other input data, distinguished from the input data input to the neural network, as training data to the neural network. The electronic device can use the output data obtained from the neural network that has received the other input data as the result of inference of the other input data based on the neural network.

[0210] Figure 16 is a block diagram of an electronic device according to one embodiment.

[0211] The electronic device 1601 in Figure 16 may include the aforementioned electronic device 201.

[0212] For example, the operation described with reference to Figure 15 can be performed by the electronic device 1601 and / or the processor 1610 of Figure 16.

[0213] Referring to Figure 16, the processor 1610 of the electronic device 1601 can perform computations related to the neural network 1630 stored in memory 1620. The processor 1610 may include at least one of a CPU (central processing unit), a GPU (graphic processing unit), or an NPU (neural processing unit). The NPU may be implemented as a separate chip from the CPU, or it may be integrated into a chip such as a CPU in the form of a SoC (system on a chip). An NPU integrated into a CPU is sometimes called a neural core and / or an AI (artificial intelligence) accelerator.

[0214] Referring to Figure 16, the processor 1610 can identify the neural network 1630 stored in memory 1620. The neural network 1630 may include a combination of an input layer 1632, one or more hidden layers 1634 (or intermediate layers), and intermediate layers 1636. The aforementioned layers (for example, the input layer 1632, one or more hidden layers 1634, and the output layer 1636) may include multiple nodes. The number of hidden layers 1634 may vary depending on the embodiment, and a neural network 1630 containing multiple hidden layers 1634 may be called a deep neural network. The operation of training such a deep neural network is sometimes called deep learning.

[0215] In one embodiment, if the neural network 1630 has the structure of a feedforward neural network, a first node in a particular layer can be connected to all of the second nodes in another layer preceding that particular layer. In memory 1620, the parameters stored for the neural network 1630 may include weights assigned to the connections between the second nodes and the first nodes. In the neural network 1630 having the structure of a feedforward neural network, the value of the first node may correspond to a weighted sum of the values ​​assigned to the second node, based on the weights assigned to the connections between the second node and the first node.

[0216] In one embodiment, if the neural network 1630 has the structure of a convolutional neural network, a first node in a particular layer may correspond to a weighted sum of several second nodes in another layer preceding that particular layer. Some of the second nodes corresponding to the first node may be identified by a filter corresponding to a particular layer. In memory 1620, the parameters stored for the neural network 1630 may include weights representing the filter. The filter may include one or more of the second nodes used to calculate the weighted sum of the first nodes, and weights corresponding to each of the one or more nodes.

[0217] According to one embodiment, the processor 1610 of the electronic device 1601 can train the neural network 1630 using the training dataset 1640 stored in memory 1620. Based on the training dataset 1640, the processor 1610 can adjust one or more parameters stored in memory 1620 for the neural network 1630 by performing the operations described with reference to Figure 15.

[0218] According to one embodiment, the processor 1610 of the electronic device 1601 can perform object detection, object recognition, and / or object classification using a neural network 1630 trained on a training dataset 1640. The processor 1610 can input an image (or video) acquired via the camera 1650 into the input layer 1632 of the neural network 1630. Based on the input layer 1632 with the image input, the processor 1610 can sequentially acquire the node values ​​of the layers included in the neural network 1630 to obtain a set of node values ​​(e.g., output data) for the output layer 1636. The output data can be used as a result of estimating the information contained in the image using the neural network 1630. The embodiment is not limited thereto, and the processor 1610 can input an image (or video) acquired from an external electronic device connected to the electronic device 1601 via a communication circuit 1660 into the neural network 1630.

[0219] In one embodiment, a neural network 1630 trained to process an image may be used to identify regions corresponding to subjects within the image (object detection) and / or to identify classes of subjects represented within the image (object recognition and / or object classification). For example, the electronic device 1601 may use the neural network 1630 to segment regions corresponding to subjects within the image based on rectangular shapes such as bounding boxes. For example, the electronic device 1601 may use the neural network 1630 to identify at least one class from a plurality of designated classes that matches the subject.

[0220] The electronic device described above may include an image sensor. The electronic device may include a CPU (central processing unit). The electronic device may include a NPU (neural processing unit). The electronic device may include a memory that includes one or more storage media for storing instructions. When the instructions are executed by the CPU, they may cause the electronic device to acquire an image via the image sensor. When the instructions are executed by the CPU, they may cause the electronic device to control the NPU to execute an object detection model configured to detect an external object from the image. When the instructions are executed by the CPU, they may cause the electronic device to obtain coordinate values ​​from the NPU representing a portion of the image related to the external object. When the instructions are executed by the CPU, they may cause the electronic device to control the CPU's processing circuitry to perform a series of calculations based on the coordinate values ​​that define a direction identification model configured to identify the direction of movement of the external object. When the instructions are executed by the CPU, they may cause the electronic device to output a notification regarding the direction of movement based on the direction of movement of the external object, as represented by the results of the series of calculations.

[0221] According to one embodiment, the instruction can cause the electronic device to obtain a first size of the part and a second size of the other part, based on the execution of the CPU by executing the object detection model and obtaining other coordinate values ​​representing the other part of the image, along with the coordinate values ​​representing the part of the image. The instruction can cause the electronic device to output a notification of the direction of movement of the other external object, based on the direction of movement of the other external object corresponding to the other part, based on the result of the calculations performed on the other external object, which is greater than the first size, based on obtaining the second size which is greater than the first size.

[0222] According to one embodiment, the instruction can cause the electronic device to obtain from the NPU first data relating to the type of the external object and second data relating to the type of the other external object when executed by the CPU. The instruction can cause the electronic device to, when executed by the CPU, further based on the first data and the second data, refrain from performing the plurality of calculations based on the coordinate values, perform the plurality of calculations defining the direction identification model based on the other coordinate values, and output a notification regarding the direction of movement of the other external object, based on the direction of movement of the other external object corresponding to the other part, represented by the results of the plurality of calculations performed based on the other coordinate values.

[0223] According to one embodiment, when the instruction is executed by the CPU, the instruction can cause the electronic device to control the NPU to execute an image segmentation model configured to recognize the lane in which the vehicle equipped with the electronic device is located from the image. When the instruction is executed by the CPU, the instruction can cause the electronic device to cause the NPU to identify a region in the image corresponding to the lane in which the vehicle is located. When the instruction is executed by the CPU, the instruction can cause the electronic device to compare the portion and the region based on the identification of portions corresponding to a plurality of external objects in the image from the object detection model executed by the NPU. When the instruction is executed by the CPU, the instruction can cause the electronic device to control the processing circuit of the CPU to execute the plurality of calculations that define the direction identification model using the coordinate values ​​of the portion that overlaps with the region within the portion.

[0224] According to one embodiment, when the instruction is executed by the CPU, the instruction can cause the electronic device to control the NPU to execute an image segmentation model configured to recognize a road on which a vehicle equipped with the electronic device is located from the image. When the instruction is executed by the CPU, the instruction can cause the electronic device to cause the NPU to identify a region in the image that corresponds to a road, separated by a center line, on which the vehicle is located. When the instruction is executed by the CPU, the instruction can cause the electronic device to compare the portion and the region based on the identification of portions corresponding to a plurality of external objects in the image from the object detection model executed by the NPU. When the instruction is executed by the CPU, the instruction can cause the processing circuit of the CPU to cause the electronic device to refrain from performing the plurality of calculations that define the direction identification model using coordinate values ​​of a portion of the portion that does not overlap with the region. When the instruction is executed by the CPU, it can control the processing circuit of the CPU to cause the electronic device to perform the plurality of calculations that define the direction identification model using a portion of the coordinate values ​​that overlap with the region.

[0225] According to one embodiment, the instruction, when executed by the CPU, can cause the electronic device to obtain the duration during which the plurality of calculations were performed by the processing circuit of the CPU, based on the results of the plurality of calculations. The instruction, when executed by the CPU, can cause the electronic device to control the NPU to execute the object detection model using the other images, based on other images obtained from the image sensor after the image has been acquired. The instruction, when executed by the CPU, can cause the electronic device to determine the number of the other images, based on identifying from the NPU the portions of the other images associated with a plurality of external objects, such that the plurality of calculations of the orientation identification model associated with the other images are performed within the duration.

[0226] According to one embodiment, when the instruction is executed by the CPU, the instruction can cause the electronic device to select from the parts identified from the other image, based on the size of each part, the parts used to perform the plurality of calculations that define the orientation identification model, by determining that the number of parts identified from the other image exceeds the determined number.

[0227] According to one embodiment, the electronic device may include a speaker. When the instruction is executed by the CPU, the electronic device may be prompted to determine the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement of the external object represented by the results of the plurality of calculations. When the instruction is executed by the CPU, the electronic device may be prompted to output an audio notification via the speaker based on the collision probability.

[0228] According to one embodiment, the electronic device may include an LED (light-emitting diode). When the instruction is executed by the CPU, the electronic device may be prompted to determine the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement of the external object represented by the results of the plurality of calculations. When the instruction is executed by the CPU, the electronic device may be prompted to emit light through the LED based on the collision probability.

[0229] According to one embodiment, the electronic device may include a display. When the instruction is executed by the CPU, the electronic device may be prompted to determine the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement of the external object represented by the results of the plurality of calculations. When the instruction is executed by the CPU, the electronic device may be prompted to display a screen containing warning content via the display, based on the collision probability.

[0230] As described above, a method performed by an electronic device having an image sensor, a CPU, and an NPU may include an operation to acquire an image via the image sensor. This method may include an operation to control the NPU to execute an object detection model configured to detect an external object from the image. This method may include an operation to acquire coordinate values ​​from the NPU that represent a portion of the image related to the external object. This method may include an operation to control the processing circuit of the CPU to perform a set of calculations based on the coordinate values ​​that define a direction identification model configured to identify the direction of movement of the external object. This method may include an operation to output a notification regarding the direction of movement of the external object based on the direction of movement of the external object represented by the results of the set of calculations.

[0231] According to one embodiment, the method may include an operation to obtain a first size of the part and a second size of the other part, based on executing the object detection model to obtain other coordinate values ​​representing the other part of the image, along with the coordinate values ​​representing the part of the image. The method may include an operation to refrain from performing the plurality of calculations based on the coordinate values, based on obtaining the second size which is greater than the first size, and to perform the plurality of calculations that define the orientation identification model based on the other coordinate values, and to output a notification regarding the direction of movement of the other external object, based on the direction of movement of the other external object corresponding to the other part, as represented by the results of the plurality of calculations performed based on the other coordinate values.

[0232] According to one embodiment, the method may include operations to obtain from the NPU first data relating to the type of the external object and second data relating to the type of the other external object. The method may further include operations to refrain from performing the plurality of calculations based on the coordinate values, to perform the plurality of calculations defining the direction identification model based on the other coordinate values, and to output a notification regarding the direction of movement of the other external object based on the direction of movement of the other external object corresponding to the other part, as represented by the results of the plurality of calculations performed based on the other coordinate values.

[0233] According to one embodiment, the method may include controlling the NPU to execute an image segmentation model configured to recognize the lane in which the vehicle equipped with the electronic device is located from the image. The method may include the NPU identifying a region in the image that corresponds to the lane in which the vehicle is located. The method may include comparing the portion and the region based on the identification of portions corresponding to a plurality of external objects in the image from the object detection model executed by the NPU. The method may include controlling the processing circuit of the CPU to execute the plurality of calculations that define the direction identification model using the coordinate values ​​of the portion of the portion that overlaps with the region.

[0234] According to one embodiment, the method may include controlling the NPU to execute an image segmentation model configured to recognize a road on which a vehicle equipped with the electronic device is located from the image. The method may include the NPU identifying a region in the image that corresponds to a road, demarcated by a center line, on which the vehicle is located. The method may include comparing the portion and the region based on the identification of portions corresponding to a plurality of external objects in the image from the object detection model executed by the NPU. The method may include controlling the processing circuit of the CPU to refrain from performing the plurality of calculations defining the direction identification model using coordinate values ​​of the portion of the portion that does not overlap with the region. The method may include controlling the processing circuit of the CPU to perform the plurality of calculations defining the direction identification model using coordinate values ​​of the portion of the portion that overlaps with the region.

[0235] According to one embodiment, the method may include an operation to obtain the duration during which the plurality of calculations were performed by the processing circuit of the CPU, based on the results of the plurality of calculations. The method may include an operation to control the NPU to run the object detection model using the other images, based on other images obtained from the image sensor after acquiring the image. The method may include an operation to determine the number of the other images, based on identifying from the NPU the portions of the other images associated with a plurality of external objects, such that the plurality of calculations of the orientation recognition model associated with the other images are performed within the duration.

[0236] According to one embodiment, the method may include the action of determining that the number of parts identified from the other image exceeds the determined number, and then selecting from the parts identified from the other image, based on the size of each part, parts to be used to perform the plurality of calculations that define the orientation identification model.

[0237] According to one embodiment, the electronic device may include a speaker. The method may include determining the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement of the external object as represented by the results of the plurality of calculations. The method may include outputting an audio notification via the speaker based on the collision probability.

[0238] According to one embodiment, the electronic device may include an LED (light-emitting diode). This method may include determining the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement of the external object as represented by the results of the plurality of calculations. This method may include emitting light through the LED based on the collision probability.

[0239] According to one embodiment, the electronic device may include a display. This method may include determining the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement of the external object as represented by the results of the plurality of calculations. This method may include displaying a screen containing warning content via the display based on the collision probability.

[0240] As described above, in a computer-readable storage medium storing one or more programs, the one or more programs, when executed by an electronic device having an image sensor, a CPU, and an NPU, may include instructions to cause the electronic device to acquire an image via the image sensor. The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to control the NPU to execute an object detection model configured to detect an external object from the image. The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to obtain coordinate values ​​from the NPU representing a portion of the image related to the external object. The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to control the processing circuit of the CPU to perform a plurality of calculations based on the coordinate values ​​that define a direction identification model configured to identify the direction of movement of the external object. The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to output a notification regarding the direction of movement based on the direction of movement of the external object represented by the results of the plurality of calculations.

[0241] According to one embodiment, the one or more programs may include instructions to cause the electronic device to obtain a first size of the part and a second size of the other part, based on the execution of the electronic device by executing the object detection model and obtaining other coordinate values ​​representing the other part of the image, along with the coordinate values ​​representing the part of the image. The one or more programs may include instructions to cause the electronic device to refrain from performing the plurality of calculations based on the coordinate values, based on obtaining the second size which is greater than the first size, and to perform the plurality of calculations that define the orientation identification model based on the other coordinate values, and to output a notification regarding the direction of movement of the other external object, based on the direction of movement of the other external object corresponding to the other part, as represented by the results of the plurality of calculations performed based on the other coordinate values.

[0242] According to one embodiment, the one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to obtain from the NPU first data relating to the type of the external object and second data relating to the other type of external object. The one or more programs, when executed by the electronic device, may also include instructions to cause the electronic device to refrain from performing the plurality of calculations based on the coordinate values, based on the first data and the second data, perform the plurality of calculations defining the direction identification model based on the other coordinate values, and output a notification regarding the direction of movement of the other external object, based on the direction of movement of the other external object corresponding to the other part, represented by the results of the plurality of calculations performed based on the other coordinate values.

[0243] According to one embodiment, the one or more programs, when executed by the electronic device, may include instructions to trigger the electronic device to control the NPU and execute an image segmentation model configured to recognize the lane in which the vehicle equipped with the electronic device is located from the image. The one or more programs, when executed by the electronic device, may include instructions to trigger the electronic device to cause the NPU to identify a region in the image corresponding to the lane in which the vehicle is located. The one or more programs, when executed by the electronic device, may include instructions to trigger the electronic device to compare the portion and the region based on the identification of portions corresponding to a plurality of external objects in the image from the object detection model executed by the NPU. The one or more programs, when executed by the electronic device, may include instructions to trigger the electronic device to control the processing circuit of the CPU and execute the plurality of calculations that define the direction identification model using the coordinate values ​​of the portion that overlaps with the region.

[0244] According to one embodiment, the one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to control the NPU to execute an image segmentation model configured to recognize from the image a road on which a vehicle equipped with the electronic device is located. The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to cause the NPU to identify a region in the image that corresponds to a road, separated by a center line, on which the vehicle is located. The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to compare a portion and the region based on the identification of portions corresponding to a plurality of external objects in the image, respectively, from the object detection model executed by the NPU. The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to control the processing circuit of the CPU to refrain from performing the plurality of calculations that define the direction identification model using coordinate values ​​of portions of the portion that do not overlap with the region. The one or more programs, when executed by the electronic device, may include instructions that cause the electronic device to control the processing circuit of the CPU to perform the plurality of calculations that define the direction identification model, using coordinate values ​​of the portion of the part that overlaps with the region.

[0245] According to one embodiment, the one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to obtain, based on the results of the plurality of calculations, the duration during which the plurality of calculations were performed by the processing circuit of the CPU. The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to control the NPU to execute the object detection model using the other images, based on other images obtained from the image sensor after the image has been acquired. The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to determine the number of the other images, based on the NPU identifying the other images' portions associated with each of a plurality of external objects, such that a plurality of calculations of the orientation identification model associated with the other images are performed within the duration.

[0246] According to one embodiment, the one or more programs, when executed by the electronic device, may include instructions that cause the electronic device to select from the parts identified from the other image, based on the size of each part, a part to be used to perform the plurality of calculations that define the orientation identification model, by determining that the number of parts identified from the other image exceeds the determined number.

[0247] According to one embodiment, the electronic device may include a speaker. The one or more programs, when executed by the electronic device, may include instructions to trigger the electronic device to determine the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement of the external object represented by the results of the plurality of calculations. The one or more programs, when executed by the electronic device, may include instructions to trigger the electronic device to output an audio notification via the speaker, based on the collision probability.

[0248] According to one embodiment, the electronic device may include an LED (light-emitting diode). The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to determine the probability of a collision between the external object and the vehicle on which the electronic device is mounted, based on the direction of movement of the external object represented by the results of the plurality of calculations. The one or more programs, when executed by the electronic device, may include an operation to emit light through the LED based on the collision probability.

[0249] According to one embodiment, the electronic device may include a display. The one or more programs, when executed by the electronic device, may include instructions to trigger the electronic device to determine the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement of the external object represented by the results of the plurality of calculations. The one or more programs, when executed by the electronic device, may include instructions to trigger the electronic device to display a screen containing warning content through the display, based on the collision probability.

[0250] The electronic device described above may include an image sensor. The electronic device may include a CPU (central processing unit). The electronic device may include a NPU (neural processing unit). The electronic device may include a memory that stores instructions and includes one or more storage media. When the CPU executes the instructions, the electronic device may be prompted to acquire an image via the image sensor. When the CPU executes the instructions, the electronic device may be prompted to control the NPU to execute an object detection model configured to detect an external object from the image. When the CPU executes the instructions, the electronic device may be prompted to obtain coordinate values ​​from the NPU representing a portion of the image related to the external object. When the CPU executes the instructions, the electronic device may be prompted to control the CPU's processing circuitry to perform a series of calculations based on the coordinate values ​​that define a direction identification model configured to identify the direction of movement of the external object. When the instruction is executed by the CPU, the electronic device may cause the electronic device to output a notification regarding the distance between the vehicle on which the electronic device is installed and the external object, based on the direction of movement of the external object and the coordinate values ​​represented by the results of the plurality of calculations.

[0251] According to one embodiment, when the instruction is executed by the CPU, the electronic device may be prompted to obtain data relating to the type of the external object from the NPU. When the instruction is executed by the CPU, the electronic device may be prompted to output a notification relating to the distance between the vehicle on which the electronic device is installed and the external object, based on the data.

[0252] According to one embodiment, the electronic device may include a speaker. When the instruction is executed by the CPU, the electronic device may be prompted to determine the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement and distance of the external object, as represented by the results of the plurality of calculations. When the instruction is executed by the CPU, the electronic device may be prompted to output an audio notification through the speaker based on the collision probability.

[0253] According to one embodiment, the electronic device may include an LED (light-emitting diode). When the instruction is executed by the CPU, the electronic device may be prompted to determine the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement and distance of the external object, as represented by the results of the plurality of calculations. When the instruction is executed by the CPU, the electronic device may be prompted to emit light through the LED based on the collision probability.

[0254] According to one embodiment, the electronic device may include a display. When the instruction is executed by the CPU, the electronic device may be prompted to determine the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement and distance of the external object, which are represented by the results of the plurality of calculations. When the instruction is executed by the CPU, the electronic device may be prompted to display a screen containing warning content via the display, based on the collision probability.

[0255] As described above, a method performed by an electronic device having an image sensor, a CPU, and an NPU may include an operation to acquire an image via the image sensor. This method may include an operation to control the NPU to execute an object detection model configured to detect an external object from the image. This method may include an operation to acquire coordinate values ​​from the NPU that represent a portion of the image related to the external object. This method may include an operation to control the processing circuit of the CPU to perform a set of calculations based on the coordinate values ​​that define a direction identification model configured to identify the direction of movement of the external object. This method may include an operation to output a notification regarding the distance between the vehicle on which the electronic device is installed and the external object, based on the direction of movement of the external object represented by the results of the set of calculations and the coordinate values.

[0256] According to one embodiment, the method may include an operation to obtain data relating to the type of the external object from the NPU. The method may further include an operation to output the notification relating to the distance between the vehicle on which the electronic device is installed and the external object, based on the data.

[0257] According to one embodiment, the electronic device may include a speaker. This method may include determining the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement and distance of the external object as represented by the results of the plurality of calculations. This method may include outputting an audio notification via the speaker based on the collision probability.

[0258] According to one embodiment, the electronic device may include an LED (light-emitting diode). The method may include determining the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement and distance of the external object as represented by the results of the plurality of calculations. The method may include emitting light through the LED based on the collision probability.

[0259] According to one embodiment, the electronic device may include a display. This method may include determining the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement and distance of the external object as represented by the results of the plurality of calculations. This method may include displaying a screen containing warning content via the display based on the collision probability.

[0260] As described above, in a computer-readable storage medium storing one or more programs, the one or more programs, when executed by an electronic device having an image sensor, a CPU, and an NPU, may include instructions to cause the electronic device to acquire an image via the image sensor. The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to control the NPU to execute an object detection model configured to detect an external object from the image. The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to obtain coordinate values ​​from the NPU representing a portion of the image related to the external object. The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to control the processing circuit of the CPU to perform a plurality of calculations based on the coordinate values ​​that define a direction identification model configured to identify the direction of movement of the external object. The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to output a notification regarding the distance between the vehicle on which the electronic device is installed and the external object, based on the direction of movement of the external object represented by the results of the plurality of calculations and the coordinate values.

[0261] According to one embodiment, the one or more programs, when executed by the electronic device, may include instructions to trigger the electronic device to obtain data relating to the type of the external object from the NPU. The one or more programs, when executed by the electronic device, may also include instructions to trigger the electronic device to output a notification relating to the distance between the vehicle on which the electronic device is installed and the external object, based on the data.

[0262] According to one embodiment, the electronic device may include a speaker. The one or more programs, when executed by the electronic device, may include instructions to trigger the electronic device to determine the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement and distance of the external object represented by the results of the plurality of calculations. The one or more programs, when executed by the electronic device, may include instructions to trigger the electronic device to output an audio notification via the speaker based on the collision probability.

[0263] According to one embodiment, the electronic device may include an LED (light-emitting diode). The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to determine the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement and distance of the external object, as represented by the results of the plurality of calculations. The one or more programs, when executed by the electronic device, may include instructions to cause the electronic device to emit light through the LED based on the collision probability.

[0264] According to one embodiment, the electronic device may include a display. The one or more programs, when executed by the electronic device, may include instructions to trigger the electronic device to determine the probability of a collision between the external object and the vehicle on which the electronic device is installed, based on the direction of movement and distance of the external object, as represented by the results of the plurality of calculations. The one or more programs, when executed by the electronic device, may include instructions to trigger the electronic device to display a screen containing warning content through the display, based on the collision probability.

[0265] The devices described above can be implemented using hardware components, software components, and / or combinations of hardware and software components. For example, the devices and components described in the embodiments can be implemented using one or more general-purpose or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPGA (field programmable gate array), PLU (programmable logic unit), microprocessor, or any other device capable of executing and responding to instructions. The processing device can execute an operating system (OS) and one or more software applications that run on the OS. Furthermore, the processing device can access, store, manipulate, process, and generate data in response to the execution of software. For convenience of understanding, the processing device may sometimes be described as being used as one, but a person with ordinary skill in the art will see that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Furthermore, other processing configurations, such as parallel processors, are also possible.

[0266] Software may include computer programs, code, instructions, or a combination of one or more thereof, which can configure a processing unit to operate as desired, or which can instruct the processing unit independently or collectively. Software and / or data may be interpreted by a processing unit or embody in any type of machine, component, physical device, computer storage medium or device to provide instructions or data to a processing unit. Software may be distributed on a networked computer system and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0267] The methods according to the embodiments are implemented in the form of program instructions that can be executed via various computer means and can be recorded on a computer-readable medium. In this case, the medium can either continuously store computer-executable programs or temporarily store them for execution or download. Furthermore, the medium may be various recording or storage means in the form of a combination of one or more hardware components, and is not limited to a medium directly connected to any computer system, but may be distributed on a network. Examples of mediums may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical mediums such as floptical disks, ROMs, RAMs, and flash memory, which are configured to store program instructions. Another example of a medium is a recording medium or storage medium managed by application stores that distribute applications or other sites, servers that supply or distribute various software.

[0268] Although the embodiments have been described above with limited examples and drawings, various modifications and variations can be made from the above description by a person with ordinary skill in the art. For example, the described technology may be performed in a different order than described, and / or the components of the described system, structure, apparatus, circuit, etc. may be combined or combined in a different manner than described, or substituted or replaced by other components or equivalents, and the appropriate results may be achieved. Accordingly, other embodiments, other examples, and equivalents to the claims also fall within the scope of the claims described below.

[0269] The methods according to the embodiments described in the claims or specification of this disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0270] When implemented in software, a computer-readable storage medium can be provided that stores one or more programs (software modules). The one or more programs stored on the computer-readable storage medium are configured for execution by one or more processors in an electronic device. The one or more programs include instructions that cause the electronic device to perform a method according to the claims or specifications of this disclosure. The one or more programs may be provided in a computer program product. The computer program product can be traded as a commodity between a seller and a buyer. The computer program product can be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or online (e.g., downloaded or uploaded) via an application store (e.g., Play Store®), or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product can be at least temporarily stored or temporarily generated on a device-readable storage medium such as the memory of a manufacturer's server, an application store server, or an intermediary server.

[0271] Such programs (software modules, software) can be stored in random access memory, non-volatile memory including flash memory, ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), magnetic disc storage device, CD-ROM (compact disc-ROM), DVD (digital versatile discs), or other forms of optical storage, or magnetic cassette. Alternatively, they can be stored in memory consisting of some or all of these. Furthermore, each constituent memory may include multiple instances.

[0272] Furthermore, the program may be stored in an attachable storage device accessible via a communication network such as the Internet, intranet, LAN (local area network), WAN (wide area network), or SAN (storage area network), or a combination thereof. Such a storage device may be connected via an external port to an apparatus performing an embodiment of the disclosure. Furthermore, separate storage devices on the communication network may also be connected to an apparatus performing an embodiment of the disclosure.

[0273] In the specific embodiments of the present disclosure described above, the components included in the disclosure are represented singly or plurally according to the specific embodiments presented. However, the singular or plural representations are suitably selected for the context presented for the convenience of explanation, and the present disclosure is not limited to singular or plural components. Components represented plural may consist of singular components, and components represented singly may consist of plural components.

[0274] According to the embodiments, one or more of the corresponding components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (e.g., modules or programs) can be integrated into a single component. In this case, the integrated component can perform one or more functions of each of the multiple components in the same or similar manner as they were performed by the corresponding components of the multiple components before integration. According to various embodiments, the operations performed by modules, programs, or other components may be performed sequentially, in parallel, iteratively, or empirically, or one or more of the operations may be performed in a different order, omitted, or one or more other operations may be added.

[0275] While specific embodiments have been described in the detailed description of this disclosure, it goes without saying that various modifications are possible as long as they do not deviate from the scope of this disclosure.

Claims

1. An electronic device, Image sensor and CPU (Central Processing Unit) and, NPU (Neural Processing Unit) and A memory comprising one or more storage media for storing instructions, When the aforementioned instruction is executed by the CPU, An image is acquired via the aforementioned image sensor, The NPU is controlled to execute an object detection model configured to detect external objects from the image, From the NPU, obtain coordinate values ​​representing a portion of the image related to the external object, The CPU's processing circuit is controlled to perform a series of calculations based on the coordinate values, which define a direction identification model configured to identify the direction of movement of the external object. Based on the direction of movement of the external object, which is represented by the results of the aforementioned multiple calculations, a notification regarding the direction of movement is output. The aforementioned electronic device causes electronic equipment.

2. When the aforementioned instruction is executed by the CPU, Based on executing the object detection model and obtaining other coordinate values ​​representing other parts of the image along with the coordinate values ​​representing the part of the image, a first size of the part and a second size of the other part are obtained. Based on obtaining a second size that is larger than the first size, The aforementioned calculations will be performed without relying on the aforementioned coordinate values. The plurality of calculations that define the direction identification model are performed based on the other coordinate values, Based on the movement direction of the other external object corresponding to the other part, represented by the results of the multiple calculations performed based on the other coordinate values, a notification regarding the movement direction of the other external object is output. The aforementioned electronic device causes The electronic device according to claim 1.

3. When the aforementioned instruction is executed by the CPU, From the NPU, obtain first data relating to the type of the external object and second data relating to the type of other external objects. Based on the first and second data, The aforementioned calculations will be performed without relying on the aforementioned coordinate values. The plurality of calculations that define the direction identification model are performed based on the other coordinate values, Based on the results of the multiple calculations performed based on the aforementioned other coordinate values, and representing the direction of movement of the other external object corresponding to the other part, a notification regarding the direction of movement of the other external object is output. The aforementioned electronic device causes The electronic device according to claim 2.

4. When the aforementioned instruction is executed by the CPU, The NPU is controlled to execute an image segmentation model configured to recognize the lane in which the vehicle equipped with the electronic device is located from the image. From the NPU, the region in the image corresponding to the lane in which the vehicle is located is identified. Based on the object detection model performed by the NPU, which identifies parts corresponding to multiple external objects in the image, the parts and the region are compared. The processing circuit of the CPU is controlled to perform the plurality of calculations that define the direction identification model using a portion of the coordinate values ​​that overlap with the region. The aforementioned electronic device causes The electronic device according to claim 1.

5. When the aforementioned instruction is executed by the CPU, The NPU is controlled to execute an image segmentation model configured to recognize the road on which the vehicle equipped with the electronic device is located from the image. From the aforementioned NPU, identify the region in the image that corresponds to the road, separated by a center line, and where the vehicle is located. Based on identifying a portion corresponding to each of several external objects in the image from the object detection model executed by the NPU, the portion and the region are compared. By controlling the processing circuit of the CPU, the plurality of calculations that define the direction identification model are avoided using a portion of the coordinate values ​​that do not overlap with the region. Control the processing circuit of the CPU so that the plurality of calculations defining the direction identification model are performed using a portion of the coordinate values ​​that overlap with the region. The aforementioned electronic device causes The electronic device according to claim 1.

6. When the aforementioned instruction is executed by the CPU, Based on the results of the aforementioned multiple calculations, the duration during which the aforementioned multiple calculations were performed by the processing circuit of the CPU is obtained. After acquiring the aforementioned image, the NPU is controlled based on other images acquired from the image sensor, and the object detection model is executed using the other images. Based on identifying from the NPU a portion of the other image associated with each of the multiple external objects, the number of portions of the other image is determined such that multiple calculations of the orientation identification model associated with the other image are performed within the period. The aforementioned electronic device causes The electronic device according to claim 1.

7. When the aforementioned instruction is executed by the CPU, By determining that the number of parts identified from the other images exceeds the determined number, a selection is made from the parts identified from the other images to be used to perform the plurality of calculations that define the orientation identification model, based on the size of each of the parts. The aforementioned electronic device causes The electronic device according to claim 6.

8. Including speakers, When the aforementioned instruction is executed by the CPU, Based on the direction of movement of the external object represented by the results of the aforementioned calculations, the probability of collision between the external object and the vehicle on which the electronic device is installed is determined. Based on the collision probability, an audio notification is output via the speaker. The aforementioned electronic device causes The electronic device according to claim 1.

9. It further includes LEDs (light emitting diodes), When the aforementioned instruction is executed by the CPU, Based on the direction of movement of the external object represented by the results of the aforementioned calculations, the probability of collision between the external object and the vehicle on which the electronic device is installed is determined. Based on the collision probability, light is emitted through the LED. The aforementioned electronic device causes The electronic device according to claim 1.

10. Including the display, When the aforementioned instruction is executed by the CPU, Based on the direction of movement of the external object represented by the results of the aforementioned calculations, the probability of collision between the external object and the vehicle on which the electronic device is installed is determined. Based on the collision probability, a screen containing warning content is displayed via the display. The aforementioned electronic device causes The electronic device according to claim 1.

11. An electronic device, Image sensor and CPU (Central Processing Unit) and, NPU (Neural Processing Unit) and A memory comprising one or more storage media for storing instructions, When the aforementioned instruction is executed by the CPU, An image is acquired via the aforementioned image sensor, The NPU is controlled to execute an object detection model configured to detect external objects from the image, From the NPU, obtain coordinate values ​​representing a portion of the image related to the external object, The CPU's processing circuit is controlled to perform a series of calculations based on the coordinate values, which define a direction identification model configured to identify the direction of movement of the external object. Based on the direction of movement of the external object and the coordinate values ​​represented by the results of the aforementioned multiple calculations, the system outputs a notification regarding the distance between the vehicle on which the electronic device is installed and the external object. The aforementioned electronic device causes electronic equipment.

12. When the aforementioned instruction is executed by the CPU, The NPU obtains data relating to the type of the external object, Based on the aforementioned data, the system outputs the notification regarding the distance between the vehicle on which the electronic device is installed and the external object. The aforementioned electronic device causes The electronic device according to claim 11.

13. Including speakers, When the aforementioned instruction is executed by the CPU, Based on the direction of movement and distance of the external object represented by the results of the aforementioned calculations, the probability of collision between the external object and the vehicle on which the electronic device is installed is determined. Based on the collision probability, an audio notification is output via the speaker. The aforementioned electronic device causes The electronic device according to claim 11.

14. It further includes LEDs (light emitting diodes), When the aforementioned instruction is executed by the CPU, Based on the direction of movement and distance of the external object represented by the results of the aforementioned calculations, the probability of collision between the external object and the vehicle on which the electronic device is installed is determined. Based on the collision probability, light is emitted through the LED. The aforementioned electronic device causes The electronic device according to claim 11.

15. Including the display, When the aforementioned instruction is executed by the CPU, Based on the direction of movement and distance of the external object represented by the results of the aforementioned calculations, the probability of collision between the external object and the vehicle on which the electronic device is installed is determined. Based on the collision probability, a screen containing warning content is displayed via the display. The aforementioned electronic device causes The electronic device according to claim 11.

16. In a non-temporary computer-readable storage medium that stores one or more programs, When the one or more programs are executed by an electronic device having an image sensor, a CPU, and an NPU, An image is acquired via the aforementioned image sensor, The NPU is controlled to execute an object detection model configured to detect external objects from the image, From the NPU, obtain coordinate values ​​representing a portion of the image related to the external object, The CPU's processing circuit is controlled to perform a series of calculations based on the coordinate values, defining a direction identification model configured to identify the direction of movement of the external object. Based on the direction of movement of the external object, which is represented by the results of the aforementioned multiple calculations, a notification regarding the direction of movement is output. Including an instruction to trigger the aforementioned electronic device, A non-temporary, computer-readable storage medium.

17. When the one or more programs are executed by the electronic device, Based on executing the object detection model and obtaining other coordinate values ​​representing other parts of the image along with the coordinate values ​​representing the part of the image, a first size of the part and a second size of the other part are obtained. Based on obtaining a second size that is larger than the first size, The aforementioned calculations will be performed without relying on the aforementioned coordinate values. The plurality of calculations that define the direction identification model are performed based on the other coordinate values, Based on the movement direction of the other external object corresponding to the other part, represented by the results of the multiple calculations performed based on the other coordinate values, a notification regarding the movement direction of the other external object is output. Including an instruction to trigger the aforementioned electronic device, A non-temporary computer-readable storage medium according to claim 16.

18. When the one or more programs are executed by the electronic device, From the NPU, obtain first data relating to the type of the external object and second data relating to the type of other external objects. Based on the first and second data, The aforementioned calculations will be performed without relying on the aforementioned coordinate values. The plurality of calculations that define the direction identification model are performed based on the other coordinate values, Based on the results of the multiple calculations performed based on the aforementioned other coordinate values, and representing the direction of movement of the other external object corresponding to the other part, a notification regarding the direction of movement of the other external object is output. Including an instruction to trigger the aforementioned electronic device, A non-temporary computer-readable storage medium according to claim 17.

19. When the one or more programs are executed by the electronic device, The NPU is controlled to execute an image segmentation model configured to recognize the lane in which the vehicle equipped with the electronic device is located from the image. From the NPU, the region in the image corresponding to the lane in which the vehicle is located is identified. Based on identifying a portion corresponding to each of several external objects in the image from the object detection model executed by the NPU, the portion and the region are compared. The processing circuit of the CPU is controlled to perform the plurality of calculations that define the direction identification model using a portion of the coordinate values ​​that overlap with the region. Including an instruction to trigger the aforementioned electronic device, A non-temporary computer-readable storage medium according to claim 16.

20. When the one or more programs are executed by the electronic device, The NPU is controlled to execute an image segmentation model configured to recognize the road on which the vehicle equipped with the electronic device is located from the image. From the aforementioned NPU, identify the region in the image that corresponds to the road, separated by a center line, and where the vehicle is located. Based on the object detection model performed by the NPU, which identifies parts corresponding to multiple external objects in the image, the parts and the region are compared. By controlling the processing circuit of the CPU, the plurality of calculations that define the direction identification model are avoided using a portion of the coordinate values ​​that do not overlap with the region. Control the processing circuit of the CPU so that the plurality of calculations defining the direction identification model are performed using a portion of the coordinate values ​​that overlap with the region. Including an instruction to trigger the aforementioned electronic device, A non-temporary computer-readable storage medium according to claim 16.

Images