Electronic device and method for acquiring learning data

Abstract

To provide an electronic device configured to acquire an image with reduced noise or blur which has been generated due to environment, using learning data.SOLUTION: An electronic device can acquire a first image from a first camera. The electronic device can acquire a second image from a second camera which is lower in shutter speed than the first camera. The electronic device can distinguish the first image from the second image, the first image and the second image having been captured at corresponding time points. The electronic device can set a first region of interest in an object of a designated type, in the first image. The electronic device can set a second region of interest which is located in the same position as the first region of interest in the second image. The electronic device can generate the first image with the first region of interest set therein and the second image with the second region of interest set therein, as learning data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to electronic devices and methods for obtaining training data. [Background technology]

[0002] Electronic devices installed in vehicles to capture images of the external environment may have difficulty capturing clear images due to noise or blur caused by the environment (e.g., low light, dynamic environment).

[0003] Artificial intelligence can be used to obtain improved images through de-noising or de-blurring.

[0004] The preceding information may be provided as related art to aid in the understanding of the present disclosure. No assertion or determination is being made as to the applicability of any of the preceding as prior art pertaining to the present disclosure. Summary of the Invention [Means for solving the problem]

[0005] According to one embodiment, an electronic device can include a communications circuit. The electronic device can include a memory storing instructions. The electronic device can include at least one processor operatively connected to the communications circuit and the memory. When the instructions are executed by the processor, the electronic device can cause the electronic device to acquire a first image from a first camera via the communications circuit. When the instructions are executed by the processor, the electronic device can cause the electronic device to acquire a second image from a second camera via the communications circuit, the second image having a slower shutter speed than the first camera. When the instructions are executed by the processor, the electronic device can identify the first image and the second image as being acquired at corresponding times. When the instructions are executed by the processor, the electronic device can cause the electronic device to define a first region of interest within a specified type of object in the first image. When the instructions are executed by the processor, the electronic device can cause the electronic device to define a second region of interest in the second image at the same location as the first region of interest. When the instructions are executed by the processor, the electronic device can be caused to generate, as training data, a first image in which the first region of interest is defined and a second image in which the second region of interest is defined.

[0006] According to one embodiment, a method executed by an electronic device may be performed by an electronic device including a communication circuit. The method may include acquiring a first image from a first camera via the communication circuit. The method may include acquiring a second image from a second camera via the communication circuit, the second camera having a slower shutter speed than the first camera. The method may include identifying the first image and the second image as images captured at corresponding times. The method may include setting a first region of interest within a specified type of object in the first image. The method may include setting a second region of interest in the second image at the same position as the first region of interest. The method may include generating, as training data, a first image in which the first region of interest is set and a second image in which the second region of interest is set.

[0007] A non-transitory computer-readable storage medium may store a program including instructions. When the instructions are executed by a processor of an electronic device including a communication circuit, the electronic device may cause the electronic device to acquire a first image from a first camera via the communication circuit. When the instructions are executed by the processor, the electronic device may cause the electronic device to acquire a second image from a second camera via the communication circuit, the second camera having a slower shutter speed than the first camera. When the instructions are executed by the processor, the electronic device may identify the first image and the second image as being acquired at corresponding times. When the instructions are executed by the processor, the electronic device may cause the electronic device to define a first region of interest within a specified type of object in the first image. When the instructions are executed by the processor, the electronic device may cause the electronic device to define a second region of interest in the second image at the same location as the first region of interest. When the instructions are executed by the processor, the electronic device can be caused to generate, as training data, a first image in which the first region of interest is defined and a second image in which the second region of interest is defined. [Effects of the Invention]

[0008] According to one embodiment, the electronic device and method can obtain training data for training artificial intelligence for de-noising and / or de-blurring. [Brief explanation of the drawings]

[0009] [Figure 1] 1 shows an example of a block diagram of an electronic device. [Figure 2a] 1 shows an example of a frame captured via a camera. [Figure 2b] 1 shows an example of a frame captured via a camera. [Figure 2c] 1 shows an example of a frame captured via a camera. [Figure 2d] 1 shows an example of a frame captured via a camera. [Figure 3] 1 shows an example of a ground truth (GT) image and a blurred image acquired through a camera. [Figure 4] An example of the operation of cropping a GT image and a blurred image is shown below. [Figure 5] An example of the operation of extracting a region of interest from a GT image will be shown. [Figure 6] An example of an operation for setting a region of interest in a blurred image will be described. [Figure 7] 1 shows an exemplary flowchart for describing the operation of an electronic device, according to one embodiment. [Figure 8] FIG. 1 illustrates an example of a block diagram illustrating an autonomous driving system for a vehicle, according to one embodiment. [Figure 9] FIG. 1 illustrates an example of a block diagram illustrating an autonomous vehicle, according to one embodiment. [Figure 10] FIG. 1 illustrates an example of a block diagram illustrating an autonomous vehicle, according to one embodiment. [Figure 11] 1 illustrates an example of a gateway associated with a user device in accordance with various embodiments. [Figure 12] FIG. 1 illustrates the operation of an electronic device for training a neural network based on a set of training data, according to one embodiment. [Figure 13] FIG. 1 is a block diagram of an electronic device, according to one embodiment. [Figure 14] FIG. 10 is a diagram illustrating a state in which the tractor and trailer are not connected. [Figure 15] FIG. 1 is a diagram illustrating a state in which a tractor and a trailer are connected. DETAILED DESCRIPTION OF THE INVENTION

[0010] Various embodiments herein will now be described with reference to the accompanying drawings, in which like reference numerals may be used for similar or related components.

[0011] Over the years, the trucking industry has experienced steady growth and expanded its range of services to accommodate more complex supply chains. These services include last-mile deliveries, drop-trailer programs, and intermodal transportation at ports (moving cargo to its destination by two or more different modes of transportation (ship and rail, ship and air)).

[0012] As described above, since there are a great variety of ways to transport cargo, manufacturers of cargo transportation related equipment have designed different types of equipment for transporting cargo according to various transportation needs.

[0013] In this specification, a truck that pulls a trailer whose main purpose is to transport (carry or cater) freight will be referred to and described as a tractor.

[0014] The tractors described in this specification can be classified into conventional trucks (or bonneted trucks), cab-over trucks (or cab-over engines), and semi-conventional trucks, which are intermediate between conventional trucks and cab-over trucks, depending on the position and shape of the tractor's cab.

[0015] Conventional trucks are tractors primarily used in North America, with the engine and hood located on the front axle in front of the cab, and the driver sitting behind the front axle.

[0016] On the other hand, a cab-over truck is a type of tractor that has a structure in which the tractor cab is positioned at the front end of the tractor, with the driver sitting in front of the front axle. The front of the tractor is also known as a "flat face" (or flat nose) type, and the tractor engine is located below the driver. This type of tractor is mainly used in most countries in Europe and Asia.

[0017] Just as there are various types of tractors depending on their purpose and demand, there are also various types of trailers towed by tractors. The most common types of trailers are full trailers and semi-trailers. Full trailers and semi-trailers can be distinguished by whether the trailer has both a front axle and a rear axle. This trailer can be connected to a box truck or a tractor via a coupling device.

[0018] Specifically, a full-trailer is a commercial freight trailer with both a front and rear axle. Full-trailers are designed to support the entire load on the trailer alone, meaning they can completely support their own weight without relying on a tractor. They are equipped with a drawbar to connect to a hauling or towing unit like a tractor, and are primarily used in the United States and Canada.

[0019] A semi-trailer, on the other hand, is a cargo trailer equipped with only a rear axle and no front axle, and a large portion of its load can be supported by a tractor connected to a type of hitch called a "fifth wheel." When a semi-trailer is detached from the tractor and stationary, it can support the trailer's load by extending the landing gear attached to the bottom of the semi-trailer vertically to the ground. The combination of a semi-trailer and a tractor is called a semi-trailer truck (in the United States, it is also simply called a "semi-trailer," "tractor-trailer," "semi-truck," "big rig," or "semi"). The "fifth wheel" mentioned above is a horizontal wheel attached to the tractor axle of a trailer truck to facilitate turning the trailer, and is also called the fifth wheel. The "fifth wheel" is a device that allows the tractor and semi-trailer to be movably connected, and usually includes a lower part consisting of a trunnion plate and latch device that firmly secures the kingpin attached to the semi-trailer to the tractor.

[0020] Hereinafter, for the sake of convenience, the present specification will use the aforementioned tractor / trailer terminology to refer to a cargo transport vehicle connected to a trailer tractor, and a tractor to refer to a towing vehicle for moving the trailer. Furthermore, in order to maximize the elimination of limitations on the rights of the embodiments described in the detailed description, the present invention may also use the terms "towing vehicle" and "towed vehicle" to refer to a tractor that hauls / tows a trailer, and "towed vehicle" to refer to a trailer towed by a tractor.

[0021] For ease of explanation, it is preferable to understand that "trailer" referred to throughout this specification refers to "semi-trailer", but is not limited to this.

[0022] In one embodiment, the electronic device 101 and the cameras 151 and 155 may be included in (or mounted on) a vehicle (not shown). The electronic device 101 may correspond to or be included in an ECU (electronic control unit) in the vehicle. The ECU may also be referred to as an ECM (electronic control module). The electronic device 101 may be configured as independent hardware used in the vehicle to provide functionality according to embodiments of the present invention. The embodiment is not limited thereto, and the electronic device 101 may correspond to or be included in a device (e.g., a black box) installed in the vehicle.

[0023] Referring to FIG. 1 , an electronic device 101 according to one embodiment may include a communication circuit 110, a processor 120, and a memory 130. The communication circuit 110, the processor 120, and the memory 130 may be electrically and / or operably coupled to each other by electronic components such as a communication bus 140. Hereinafter, "operably coupled" hardware may refer to a direct or indirect connection between the hardware, established by wire or wirelessly, such that a first piece of hardware controls a second piece of hardware. Although shown as separate blocks, the embodiment is not limited thereto. Some of the hardware in FIG. 1 may be included in a single integrated circuit, such as a system on a chip (SoC). The types and / or number of hardware included in the electronic device 101 are not limited to those shown in FIG. 1 . For example, the electronic device 101 may include only some of the hardware shown in FIG. 1 .

[0024] According to one embodiment, the communication circuitry 110 of the electronic device 101 may include hardware components for supporting transmission and / or reception of electrical signals between the electronic device 101 and an external electronic device (e.g., cameras 151, 155). The communication circuitry 110 may include, for example, at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuitry 110 may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), Zigbee, long term evolution (LTE), 5G new radio (NR), and / or 6G.

[0025] According to one embodiment, electronic device 101 may include hardware for processing data based on one or more instructions. The hardware for processing data may include processor 120. For example, the hardware for processing data may include an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). Processor 120 may have a single-core processor architecture, or may have a multi-core processor architecture, such as a dual-core, quad-core, hexa-core, or octa-core architecture.

[0026] According to one embodiment, memory 130 of electronic device 101 may include hardware components for storing data and / or instructions input and / or output to processor 120 of electronic device 101. For example, memory 130 may include volatile memory, such as random-access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM). For example, volatile memory may include at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). For example, non-volatile memory may include at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, a hard disk, a compact disk, a solid state drive (SSD), and an embedded multi-media card (eMMC).

[0027] Although not shown, the electronic device 101 may further include various components. For example, the electronic device 101 may further include a display for displaying a user interface.

[0028] According to one embodiment, each of the cameras 151 and 155 may include a lens assembly or an image sensor. The lens assembly may collect light emitted from a subject being imaged. The lens assembly may include one or more lenses. According to one embodiment, each of the cameras 151 and 155 may include multiple lens assemblies. For example, each of the cameras 151 and 155 may have multiple lens assemblies with the same lens attributes (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens attributes that are different from the lens attributes of the other lens assemblies. The lens assembly may include a wide-angle lens or a telephoto lens. According to one embodiment, the flash may include one or more light-emitting diodes (e.g., red-green-blue (RGB) LEDs, white LEDs, infrared LEDs, or ultraviolet LEDs) or a xenon lamp. For example, the image sensor can capture an image corresponding to an object by converting light emitted from or reflected by the object and transmitted through the lens assembly into an electrical signal. According to one embodiment, the image sensor can include a selected one of image sensors with different attributes, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, or multiple image sensors with the same attribute, or multiple image sensors with different attributes. Each image sensor included in the image sensor can be realized using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

[0029] In one embodiment, the cameras 151, 155 may have different settings. In one embodiment, the cameras 151, 155 may have a difference in at least one of a plurality of settings. For example, the plurality of settings may include the respective angles of the cameras 151, 155 (e.g., tilt angle, roll angle, and / or pan angle). For example, the plurality of settings may include the angle of view, resolution, and / or lens attributes (e.g., angle of view, focal length, autofocus, f-number, ISO (International Organization for Standardization) sensitivity, or optical zoom). For example, the plurality of settings may include shutter speed. For example, the respective angles of the cameras 151, 155 may be set (or arranged) to be identical (or substantially identical) (or corresponding) to each other (by hardware configuration (e.g., a spirit level, a case, a fixed base)). For example, the resolution and / or lens characteristics (e.g., angle of view, focal length, autofocus, f-number, ISO sensitivity, or optical zoom) of each of cameras 151 and 155 may be set to be identical (or substantially identical) (or corresponding) to each other. For example, the shutter speeds of each of cameras 151 and 155 may be different from each other. For example, the shutter speed of camera 151 may be faster than the shutter speed of camera 155. In one embodiment, the faster the shutter speed of camera 151 is than the shutter speed of camera 155, the clearer the image captured via camera 151 may be than the image captured via camera 155. In one embodiment, the faster the shutter speed of camera 151 is than the shutter speed of camera 155, the more the image captured via camera 155 may exhibit at least one of resolution issues, white noise issues, or blur (or motion blur) issues than the image captured via camera 151. This may make it more difficult to identify character strings in the image captured via camera 155 than in the image captured via camera 151.

[0030] In one embodiment, the image captured via camera 151 may be referred to as a ground truth image. In one embodiment, the image captured via camera 155 may be referred to as a blur image (or motion blur image).

[0031] According to one embodiment, the cameras 151, 155 may be positioned (or arranged) toward one direction of the vehicle (not shown), for example, the cameras 151, 155 may be positioned (or arranged) toward the front direction and / or driving direction of the vehicle.

[0032] 1 illustrates cameras 151, 155 as being physically separate from electronic device 101, but this is merely an example. According to an embodiment, a portion of cameras 151, 155 may be integrally formed with electronic device 101. Alternatively, both cameras 151, 155 may be integrally formed with electronic device 101. For example, when both cameras 151, 155 are integrally formed with electronic device 101, communication circuit 110 may be a circuit for interfacing between cameras 151, 155 and processor 120 (e.g., MIPI (mobile industry processor interface)).

[0033] The operation of the electronic device 101 to collect learning data via the cameras 151 and 155 will be described below.

[0034] In one embodiment, the processor 120 of the electronic device 101 can instruct (or command) the cameras 151, 155 to take a picture. In one embodiment, the processor 120 of the electronic device 101 can instruct (or command) the cameras 151, 155 to take a picture at the same (or substantially the same) (or corresponding) time points as each other. In one embodiment, the processor 120 can instruct (or command) the cameras 151, 155 to take a picture based on vehicle movement. However, this is not limiting. In one embodiment, the processor 120 can instruct (or command) the cameras 151, 155 to take a picture regardless of vehicle movement. For example, the time points being the same (or substantially the same) (or corresponding) as each other can mean that the instruction (or command) to take a picture is generated and / or transmitted within a specified offset (or frame interval depending on the frame rate).

[0035] In one embodiment, processor 120 can acquire an image stream from each of cameras 151, 155. In one embodiment, the image stream can be a data stream of images acquired at a set frame rate (e.g., frames per second, FPS). In one embodiment, the set frame rate can be 30 FPS. However, it is not limited to this. In one embodiment, the set frame rate can be less than 30 FPS (e.g., 24 FPS) or greater than 30 FPS (e.g., 60 FPS).

[0036] In one embodiment, processor 120 can store in memory 130 (or a buffer in memory 130) a specified number (e.g., 10) of images acquired at a set frame rate after a capture command. In one embodiment, processor 120 can store in memory 130 (or a buffer in memory 130) images acquired during a specified time period (e.g., 0.3 seconds) after a capture command. For example, processor 120 can store in memory 130 (or a buffer in memory 130) a specified number (e.g., 10) of ground truth (GT) images acquired during a specified time period from camera 151 and a specified number (e.g., 10) of blurred images acquired during a specified time period from camera 155.

[0037] In one embodiment, the processor 120 can identify whether a vehicle is moving based on the GT image and the blurred image acquired from the cameras 151 and 155. In one embodiment, the processor 120 can identify whether a vehicle is moving based on a comparison between consecutive images. For example, the processor 120 can identify whether a vehicle is moving based on a comparison between consecutive GT images (or blurred images). Here, the comparison between consecutive images may be based on a comparison between a feature map of an image of a first frame and a feature map of an image of a second frame after the first frame. The comparison between consecutive images may be based on a differential image between an image of the first frame and an image of the second frame. Here, the second frame may be a frame acquired next to the first frame according to a set frame rate. Here, the feature map may be acquired based on a neural network. However, this is not limited to this.

[0038] In one embodiment, processor 120 can identify that a vehicle is stationary based on the difference between consecutive images being equal to or less than a specified reference difference. In one embodiment, processor 120 can identify that a vehicle is moving based on the difference between consecutive images exceeding a specified reference difference. For example, processor 120 can identify that a vehicle is moving based on the size of a region exceeding the reference difference being equal to or greater than a reference size in a difference feature map between a feature map of an image of a first frame and a feature map of an image of a second frame. For example, processor 120 can identify that a vehicle is stationary based on the size of a region exceeding the reference difference being less than a reference size in a difference feature map between a feature map of an image of a first frame and a feature map of an image of a second frame. For example, processor 120 can identify that a vehicle is moving based on the size of a region exceeding the reference difference being equal to or greater than a reference size in a difference image between a image of a first frame and an image of a second frame. For example, processor 120 can identify that the vehicle is stopped based on the size of the area in the difference image between the image of the first frame and the image of the second frame that exceeds the reference difference being less than the reference size.

[0039] In one embodiment, the processor 120 can identify duplicate images based on the GT images and blurred images acquired from the cameras 151 and 155. In one embodiment, the processor 120 can identify duplicate images based on a comparison between consecutive images. For example, the processor 120 can identify duplicate images based on a comparison between consecutive GT images (or blurred images). Here, the comparison between consecutive images may be based on a comparison between a feature map of an image of a first frame and a feature map of an image of a second frame after the first frame. The comparison between consecutive images may be based on a difference image between the image of the first frame and the image of the second frame. Here, the second frame may be a frame acquired next to the first frame according to a set frame rate. Here, the feature map may be acquired based on a neural network. However, this is not limiting.

[0040] In one embodiment, processor 120 may determine that consecutive images may be duplicate images based on the difference between the consecutive images being less than or equal to a specified reference difference, and may delete (or remove) one of the duplicate images.

[0041] In one embodiment, processor 120 may determine whether to process GT images and blurred images acquired from cameras 151, 155. For example, processor 120 may determine whether to process GT images and blurred images based on whether the GT images and blurred images were acquired while the vehicle was moving. For example, processor 120 may determine to process GT images and blurred images if the GT images and blurred images were acquired while the vehicle was moving. However, this is not limiting.

[0042] In one embodiment, processor 120 can align the time between a specified number of GT images and a blurred image. In one embodiment, when processor 120 determines to process GT images and blurred images, processor 120 can align the time between the specified number of GT images and a blurred image. For example, processor 120 can align the time by comparing the reference image among images acquired through a reference camera with images acquired through other cameras. For example, the reference camera may be camera 151, and the reference image may be an intermediate image (e.g., the fifth image) among the specified number of images. In this case, processor 120 can compare the fifth GT image with the ten blurred images to identify blurred images acquired at the same time (or substantially the same time) as the fifth GT image (or at times corresponding to each other).

[0043] For example, the comparison between the GT image and the blurred image may be based on a comparison between the feature map of the GT image and the feature map of the blurred image. For example, the electronic device 101 may identify the GT image and the blurred image as having been acquired at the same time based on comparing the size of the region exceeding the reference difference in the difference feature map between the GT image and the blurred image with a reference size. For example, the processor 120 may identify the GT image and the blurred image as having been acquired at a different time based on the size of the region exceeding the reference difference in the difference feature map between the GT image and the blurred image being equal to or greater than the reference size. For example, the processor 120 may identify the GT image and the blurred image as having been acquired at a different time based on the size of the region exceeding the reference difference in the difference feature map between the GT image and the blurred image being less than the reference size. Here, the feature map may be acquired based on a neural network. However, this is not limiting.

[0044] For example, the comparison between the GT image and the blurred image may be based on a difference image between the GT image and the blurred image. For example, the electronic device 101 can identify the GT image and the blurred image as having been acquired at the same time based on comparing the size of the region in the difference image where the difference exceeds the reference difference with a reference size. For example, the processor 120 can identify the GT image and the blurred image as having been acquired at the same time based on the size of the region in the difference image where the difference exceeds the reference difference being equal to or greater than the reference size. For example, the processor 120 can identify the GT image and the blurred image as having been acquired at the same time based on the size of the region in the difference image where the difference exceeds the reference difference being less than the reference size.

[0045] In one embodiment, the GT image and the blur image being acquired at substantially the same time (or at corresponding times) may include the acquisition time difference between the GT image and the blur image being equal to or less than a frame interval (or half the frame interval) according to a set frame rate. The comparison between the GT image and the blur image can be explained with reference to Figures 2a to 2d.

[0046] 2a to 2d show examples of frames captured via a camera.

[0047] Referring to FIG. 2a, the cameras 151 and 155 can start capturing images at the same time (or substantially the same time) (or corresponding times) (e.g., T1). In one embodiment, the cameras 151 and 155 starting capturing images at substantially the same time (or corresponding times) can include the difference in the capture start times between the cameras 151 and 155 being equal to or less than a frame interval (or half the frame interval) according to the set frame rate. Thus, in FIG. 2a, ten GT images 211-220 and ten blur images 221-230 can be acquired at the same time (or substantially the same time) (or corresponding times). For example, the GT image 211 and the blur image 221 can be acquired at time T1. For example, the GT images 212-220 and the blur images 222-230 can be acquired at each of the time points (e.g., T2-T10).

[0048] In one embodiment, the processor 120 can time-align the ten GT images 211-220 and the ten blurred images 221-230. For example, the processor 120 can compare the GT image 215 (or the reference image) in a specified order (e.g., the fifth) among the GT images 211-220 acquired via the camera 151 (or the reference camera) with each of the ten blurred images 221-230 acquired via the camera 155 (or another camera). Here, the comparison between the GT image 215 and each of the ten blurred images 221-230 may be based on a comparison between the feature map of the GT image 215 and each of the ten blurred images 221-230. The comparison between consecutive images may be based on a difference image between the GT image 215 and each of the ten blurred images 221-230. Here, the feature map may be acquired based on a neural network. However, this is not limiting.

[0049] In one embodiment, the processor 120 can determine that a blurred image, for which the difference between the GT image 215 and each of the ten blurred images 221-230 is equal to or less than a specified reference difference, is an image acquired at the same time as the GT image 215. For example, the processor 120 can determine that the GT image 215 and the blurred image 225 are images acquired at the same time. This allows the electronic device 101 to determine the image sets acquired at the same time based on the acquisition order (or frame order) of the ten GT images 211-220 and the ten blurred images 221-230. For example, the electronic device 101 can identify from the ten blurred images 221-230 the blurred image that has the same frame difference as the frame difference between a specific GT image and the reference image among the ten GT images 211-220. In one embodiment, the electronic device 101 can determine that the specific image and the identified blurred image are an image set acquired at the same time. For example, if it is determined that the GT image 215 and the blur image 225 are images obtained at the same time, it can be determined that the GT image 211 and the blur image 221 are images obtained at the same time (e.g., T1).

[0050] Referring to Fig. 2b, camera 151 can start capturing images at a time point (e.g., T3) later than camera 155 by a time corresponding to the time of two frames. As a result, after processor 120 issues a capture command, eight GT images 213-220 and ten blurred images 221-230 can be acquired in Fig. 2b. For example, blurred images 221 and 222 can be acquired at times T1 and T2. For example, GT images 213-220 and blurred images 223-230 can be acquired at each of the times (e.g., T3-T10).

[0051] In one embodiment, the processor 120 can time align the eight GT images 213-220 and the ten blurred images 221-230. For example, the processor 120 can compare the GT image 217 in a specified order (e.g., the fifth) among the GT images 213-220 acquired via the camera 151 with each of the ten blurred images 221-230 acquired via the camera 155.

[0052] In one embodiment, the processor 120 can determine that a blurred image whose difference between the GT image 217 and each of the ten blurred images 221-230 is equal to or less than a specified reference difference is an image acquired at the same time as the GT image 217. For example, the processor 120 can determine that the GT image 217 and the blurred image 227 are images acquired at the same time. This allows the electronic device 101 to determine which image sets were acquired at the same time according to the acquisition order (or frame order) of the eight GT images 213-220 and the ten blurred images 221-230. For example, if the GT image 217 and the blurred image 227 are determined to be images acquired at the same time, the GT image 213 and the blurred image 223 can be determined to be images acquired at the same time.

[0053] 2c, camera 155 can start capturing images at a time point (e.g., T2) later than camera 151 by a time corresponding to the time of one frame. As a result, after processor 120 issues a capture command, ten GT images 211-220 and nine blurred images 222-230 can be acquired in FIG. 2c. For example, GT image 211 can be acquired at time point T1. For example, GT images 212-220 and blurred images 222-230 can be acquired at each of time points (e.g., T2-T10).

[0054] In one embodiment, the processor 120 can time align the ten GT images 211-220 and the nine blurred images 222-230. For example, the processor 120 can compare the GT image 215 in a specified order (e.g., the fifth) among the GT images 211-220 acquired via the camera 151 with each of the nine blurred images 222-230 acquired via the camera 155.

[0055] In one embodiment, the processor 120 can determine that a blurred image, for which the difference between the GT image 215 and each of the nine blurred images 222-230 is equal to or less than a specified reference difference, is an image acquired at the same time as the GT image 215. For example, the processor 120 can determine that the GT image 215 and the blurred image 225 are images acquired at the same time. Therefore, the electronic device 101 can determine that the image sets were acquired at the same time according to the acquisition order (or frame order) of the ten GT images 211-220 and the nine blurred images 222-230. For example, if the GT image 215 and the blurred image 225 are determined to be images acquired at the same time, the GT image 212 and the blurred image 222 can be determined to be images acquired at the same time.

[0056] Referring to Fig. 2d, camera 155 can start capturing images at a time point (e.g., T4) that is later than camera 151 by a time corresponding to the time of three frames. As a result, after processor 120 issues a capture command, ten GT images 211-220 and seven blurred images 224-230 can be acquired in Fig. 2d. For example, GT images 211, 212, and 213 can be acquired at times T1, T2, and T3. For example, GT images 214-220 and blurred images 224-230 can be acquired at each of the times (e.g., T4-T10).

[0057] In one embodiment, the processor 120 can time align the ten GT images 211-220 and the seven blurred images 224-230. For example, the processor 120 can compare the GT image 215 in a specified order (e.g., the fifth) among the GT images 211-220 acquired via the camera 151 with each of the seven blurred images 224-230 acquired via the camera 155.

[0058] In one embodiment, the processor 120 can determine that a blurred image, for which the difference between the GT image 215 and each of the seven blurred images 224-230 is equal to or less than a specified reference difference, is an image acquired at the same time as the GT image 215. For example, the processor 120 can determine that the GT image 215 and the blurred image 225 are images acquired at the same time. Therefore, the electronic device 101 can determine that the image sets were acquired at the same time according to the acquisition order (or frame order) of the ten GT images 211-220 and the seven blurred images 224-230. For example, if the GT image 215 and the blurred image 225 are determined to be images acquired at the same time, the processor 120 can determine that the GT image 214 and the blurred image 224 are images acquired at the same time.

[0059] In one embodiment, the processor 120 may generate a training data set based on a GT image and a blurred image acquired at the same time (or substantially the same time) (or corresponding time points). The training data set may include images based on a pair of a GT image and a blurred image. The training data may include multiple training data sets. Generation of the training data set may be described with reference to FIGS. 3 to 6.

[0060] FIG. 3 shows an example of a ground truth (GT) image and a blur image acquired through a camera.

[0061] Referring to FIG. 3 , the GT image 211 and the blurred image 221 may have different fields of view (FOV) due to differences in the physical positions of the cameras 151 and 155. For example, the license plate 310 of the vehicle in the GT image 211 may straddle the left and right sides of the vertical center line 301 of the GT image 211, while the license plate 320 of the vehicle in the blurred image 221 may be located to the right of the vertical center line 301 of the blurred image 221. Although FIG. 3 illustrates the majority of the vehicle regions in the GT image 211 and the blurred image 221 as being located below the horizontal center line 305, the fields of view of the cameras 151 and 155 may differ from each other depending on the respective positions and / or angles of the cameras 151 and 155. Therefore, to generate training data based on the GT image 211 and the blurred image 221, certain image processing may be required for the GT image 211 and the blurred image 221. The image processing operations of the GT image 211 and the blur image 221 for generating learning data will be described below with reference to FIGS.

[0062] FIG. 4 shows an example of the operation of cropping the GT image 211 and the blur image 221.

[0063] In one embodiment, the processor 120 can crop the GT image 211. For example, the processor 120 can remove the outer region of the GT image 211 by cropping the GT image 211. In one embodiment, the processor 120 can obtain (or extract) (or generate) the cropped GT image 420 by cropping a region 410 of a specified size (e.g., 90% of the GT image 211) of the GT image 211. In one embodiment, the processor 120 can obtain the cropped GT image 420 by cropping a specified region 410 (e.g., a central region) of the GT image 211. However, this is not limiting. In one embodiment, the processor 120 can obtain the cropped GT image 420 by cropping a region including an object. In one embodiment, the processor 120 can identify an object through an object detection (OD) model in the GT image 211. In one embodiment, the processor 120 can obtain the cropped GT image 420 by cropping a region 410 including the identified object.

[0064] In one embodiment, the processor 120 can identify other regions 430 corresponding to the cropped GT image 420 from the blurred image 221 acquired at the same time as the GT image 211. In one embodiment, the processor 120 can compare the blurred image 221 with the cropped GT image 420 to identify the other regions 430.

[0065] For example, the comparison between the cropped GT image 420 and the blurred image 221 may be based on a difference image between each of the specific regions of the cropped GT image 420 and the blurred image 221. In one embodiment, the specific region may be the region compared to the cropped GT image 420. In one embodiment, the specific region may be the region where the cropped GT image 420 is located as it is shifted within the blurred image 221. For example, the electronic device 101 may identify the specific region associated with the difference image that has the least difference (or the highest degree of agreement) as the other region 430 of the blurred image 221. However, this is not limiting. In one embodiment, the electronic device 101 may identify the other region 430 of the blurred image 221 based on comparing feature maps between the cropped GT image 420 and the blurred image 221.

[0066] For example, the comparison between the cropped GT image 420 and the blurred image 221 may be based on the distance difference between the feature point coordinates between the cropped GT image 420 and each specific region of the blurred image 221. For example, the electronic device 101 may identify feature points in the cropped GT image 420. For example, the feature points may include specific positions (e.g., headlights, license plates, heads) of objects (e.g., vehicles, buildings, pedestrians) included in the cropped GT image 420. For example, the electronic device 101 may identify feature points in the blurred image 221. For example, the electronic device 101 may identify the distance between corresponding feature points in the cropped GT image 420 and the feature points in each specific region of the blurred image 221. In one embodiment, the electronic device 101 may identify a specific region with the shortest distance between feature points as the other region 430 of the blurred image 221.

[0067] For example, the comparison between the cropped GT image 420 and the blurred image 221 may be based on the normalized difference value (e.g., normalized between 0 and 1) of each difference image and the normalized distance of the distance in each of the particular regions. For example, the electronic device 101 may identify the particular region with the lowest weighting between the normalized difference value and the normalized distance as the other region 430 of the blurred image 221.

[0068] In one embodiment, the processor 120 may crop (or obtain) (or extract) another region 430 corresponding to the cropped GT image 420 in the blurred image 221 into the cropped blurred image 440. In one embodiment, the size of the cropped blurred image 440 and the size of the cropped GT image 420 may be equal to each other. In one embodiment, the type, size, and position of the object included in the cropped GT image 420 may be the same as the type, size, and position of the object included in the cropped blurred image 440.

[0069] According to an embodiment, the processor 120 may perform image correction on the cropped GT image 420. For example, the processor 120 may increase the brightness of the cropped GT image 420.

[0070] FIG. 5 shows an example of an operation for extracting a region of interest from a GT image 211.

[0071] 5, the processor 120 can identify objects in the cropped GT image 420. In one embodiment, the processor 120 can identify specified types of objects (e.g., vehicles, traffic signs, road markings) via an image segmentation model and / or an object detection (OD) model in the cropped GT image 420. In one embodiment, the processor 120 can identify objects in the cropped GT image 420 by adjusting the size of a mask for identifying the object. In one embodiment, the processor 120 can use the mask in a large order to identify the object.

[0072] In one embodiment, the processor 120 can extract (or obtain) a region 510 of the identified object (i.e., the automobile) in the cropped GT image 420 as a GT object image 520. In one embodiment, the processor 120 can obtain the GT object image 520 by cropping the region 510 that includes the identified object.

[0073] In one embodiment, the processor 120 can identify regions of interest 530 from the GT object image 520 that correspond to text regions of the object (e.g., a vehicle license plate, a traffic sign's traffic directions, or a road marking's road directions). In one embodiment, the processor 120 can identify regions of interest 530 from the GT object image 520 based on an image segmentation model. In FIG. 5, the GT object image 520 at the bottom left is an enlarged version of the GT object image 520 at the top right, and the GT object image 520 at the bottom left and the GT object image 520 at the top right may be the same image.

[0074] In one embodiment, the processor 120 can calculate (or identify) (or determine) the vertices 531, 533, 535, and 537 through geometric fitting (e.g., fitting via a triangle, a rectangle, a pentagon, or a polygon) within the region of interest 530. In one embodiment, the processor 120 can identify the coordinates of each of the vertices 531, 533, 535, and 537 through geometric fitting within the region of interest 530. For example, in a two-dimensional virtual coordinate system in which the vertex 531 at the bottom left corner of the vertices 531, 533, 535, and 537 is set as the origin, the electronic device 101 can identify the coordinates of each of the vertices 531, 533, 535, and 537. For example, the x-axis of the two-dimensional virtual coordinate system may be formed along the bottom edge of the GT object image 520. The y-axis of the two-dimensional virtual coordinate system may be formed along the left edge of the GT object image 520.

[0075] In one embodiment, the processor 120 can mark (or set) (or identify) (or designate) the object's region 510 and the fitted region of interest 540 in the cropped GT image 420 .

[0076] In one embodiment, the processor 120 may transform the fitted region of interest 540. For example, the processor 120 may transform the fitted region of interest 540 via a specified transformation algorithm (e.g., an algorithm for a rigid-body transformation, a similarity transformation, a linear transformation, an affine transformation, and / or a perspective transformation). In one embodiment, the processor 120 may transform the fitted region of interest 540 into a license plate-shaped figure (e.g., a rectangle), but is not limited to such.

[0077] In one embodiment, an algorithm (or operation) for identifying (or recognizing) the character string displayed on the license plate can be performed (or executed) on the (transformed) fitted region of interest 540 of the cropped GT image 420. The algorithm can include an algorithm for identifying the character string in the image (e.g., an algorithm using an optical character recognition (OCR) function). In one embodiment, the character string displayed on the license plate can be identified (or recognized) based on the OCR algorithm of the fitted region of interest 540 of the cropped GT image 420.

[0078] In one embodiment, the processor 120 may label the fitted region of interest 540 of the cropped GT image 420 with the text displayed on the license plate.

[0079] FIG. 6 shows an example of the operation of setting a region of interest in the blurred image 221. In FIG.

[0080] 6, the processor 120 can mark (or set) (or identify) (or designate) the region of the object 610 and the region of interest 640 in the cropped blurred image 440. In one embodiment, the processor 120 can mark (or set) (or identify) (or designate) the region of the object 610 and the region of interest 640 in the cropped blurred image 440 based on the region of the object 510 and the region of interest 540 in the cropped GT image 420. In one embodiment, the size and position of the region of the object 610 displayed in the cropped blurred image 440 can be the same as the size and position of the region of the object 510 displayed in the cropped GT image 420. In one embodiment, the size and position of the region of interest 640 displayed in the cropped blurred image 440 can be the same as the size and position of the region of interest 540 displayed in the cropped GT image 420.

[0081] In one embodiment, the processor 120 may label the region of interest 640 in the cropped blurred image 440 with the characters displayed on the license plate. In one embodiment, the characters may be the same as the characters identified in the fitted region of interest 540 in the cropped GT image 420.

[0082] In one embodiment, the processor 120 can set (or identify) (or acquire) as a training data set a cropped GT image 420 in which the object region 510 and the region of interest 540 are marked (or set) (or identified) (or specified), and a cropped blurred image 440 in which the object region 610 and the region of interest 640 are marked (or set) (or identified) (or specified).

[0083] In one embodiment, the processor 120 may repeat the operations performed on the GT image 211 and the blur image 221 for subsequent images (eg, the GT image 212 and the blur image 222).

[0084] As described above, the electronic device 101 can construct training data based on GT images and blurred images captured at the same time (or substantially the same time) (or corresponding times). The electronic device 101 can construct training data based on GT images and blurred images captured using different shutter speeds. This allows the electronic device 101 to construct training data based on GT images and blurred images with minimal user intervention. Furthermore, the electronic device 101 can construct a large amount of training data with minimal user intervention. Furthermore, the electronic device 101 can construct an AI model for deblurring and / or denoising by learning the output results of an AI model for blurred images based on the GT images. This allows the electronic device 101 to restore character strings in an image that are not identified due to at least one of resolution, white noise, or blurring. Furthermore, the electronic device 101 can construct a large amount of training data related to traffic signs and / or road markings as well as vehicle license plates.

[0085] FIG. 7 shows an exemplary flowchart illustrating the operation of an electronic device, according to one embodiment.

[0086] 7 can be described with reference to FIGS. 1 to 6. The operations of FIG. 7 can be performed by electronic device 101. The operations of FIG. 7 can be performed by executing instructions stored in memory 130. The operations of FIG. 7 may also be performed by processor 120 executing instructions stored in memory 130.

[0087] 7, in operation 710, electronic device 101 can acquire images from multiple cameras 151, 155. In one embodiment, cameras 151, 155 may have different shutter speeds. For example, the shutter speed of camera 151 may be faster than the shutter speed of camera 155. For example, the angles of cameras 151, 155 may be set (or positioned) to be identical (or substantially identical) (or corresponding) to each other. For example, the resolutions and / or lens characteristics (e.g., angle of view, focal length, autofocus, f-number, ISO sensitivity, or optical zoom) of cameras 151, 155 can be set to be identical (or substantially identical) (or corresponding) to each other.

[0088] In one embodiment, electronic device 101 can acquire images from cameras 151, 155 by instructing (or commanding) cameras 151, 155 to take pictures at times that are the same (or substantially the same) (or corresponding) to each other. For example, times that are the same (or substantially the same) (or corresponding) to each other may mean that the instruction (or command) to take pictures is generated and / or transmitted within a specified offset (or frame interval depending on the frame rate).

[0089] At operation 720, electronic device 101 may identify an overlapping area between a first image and a second image. In one embodiment, electronic device 101 may identify an overlapping area between a first image acquired from camera 151 and a second image acquired from camera 155 that is acquired at the same time as the first image.

[0090] In one embodiment, the electronic device 101 can identify an overlapping region between the first image and the second image based on cropping the first image. For example, the electronic device 101 can remove an outer region of the first image by cropping the first image. In one embodiment, the electronic device 101 can obtain (or extract) (or generate) a cropped first image by cropping a region of the first image that is a specified size (e.g., 90% of the first image). In one embodiment, the electronic device 101 can obtain a cropped first image by cropping a specified region of the first image (e.g., a central region).

[0091] In one embodiment, the electronic device 101 can identify other regions that correspond to the cropped first image in a second image that is captured at the same time as the first image. In one embodiment, the electronic device 101 can compare the second image with the cropped first image to identify the other regions.

[0092] For example, the comparison between the cropped first image and the second image may be based on a difference image between each of specific regions of the cropped first image and the second image. In one embodiment, the specific region may be a region compared to the cropped first image. In one embodiment, the specific region may be a region located as the cropped first image shifts from the second image. For example, the electronic device 101 may identify a specific region associated with the difference image that has the least difference (or the highest degree of match) as another region of the second image. However, this is not limiting. In one embodiment, the electronic device 101 may identify another region of the second image based on comparing feature maps between the cropped first image and the second image.

[0093] For example, the comparison between the cropped first image and the second image may be based on the distance difference between feature point coordinates between the cropped first image and each specific region of the second image. For example, the electronic device 101 may identify feature points in the cropped first image. For example, the feature points may include specific positions (e.g., headlights, license plates, heads) of objects (e.g., vehicles, buildings, pedestrians) included in the cropped first image. For example, the electronic device 101 may identify feature points in the second image. For example, the electronic device 101 may identify the distance between corresponding feature points in the cropped first image and feature points in each specific region of the second image. In one embodiment, the electronic device 101 may identify a specific region with the shortest distance between feature points as another region of the second image.

[0094] For example, the comparison between the cropped first image and the second image may be based on the normalized difference value (e.g., normalized between 0 and 1) of each difference image and the normalized distance of the distance in each of the particular regions. For example, the electronic device 101 may identify the particular region with the lowest weighting between the normalized difference value and the normalized distance as another region of the second image.

[0095] In one embodiment, the electronic device 101 may crop (or obtain) (or extract) another region from the second image that corresponds to the cropped first image as the cropped second image. In one embodiment, the size of the cropped second image and the size of the cropped first image may be equal to each other. In one embodiment, the type, size, and position of objects included in the cropped first image may be the same as the type, size, and position of objects included in the cropped second image.

[0096] In one embodiment, the electronic device 101 can identify the cropped first image and the cropped second image as an overlapping area between the first image and the second image.

[0097] In operation 730, the electronic device 101 may identify a first region of interest within the first overlap region of the first image. In one embodiment, the electronic device 101 may sequentially identify an object in the first overlap region of the first image and identify a first region of interest within the identified object.

[0098] In one embodiment, the electronic device 101 can identify an object from the cropped first image. In one embodiment, the electronic device 101 can identify a specified type of object (e.g., a vehicle, a traffic sign, a road marking) in the cropped first image via an image segmentation model and / or an object detection (OD) model. In one embodiment, the electronic device 101 can identify an object in the cropped first image by adjusting the size of a mask for identifying the object. In one embodiment, the electronic device 101 can use the mask from a large order to identify the object.

[0099] In one embodiment, the electronic device 101 can extract (or capture) a region of the identified object (i.e., the car) in the cropped first image as an object image. In one embodiment, the electronic device 101 can capture the object image by cropping a region that includes the identified object.

[0100] In one embodiment, the electronic device 101 can identify regions of interest corresponding to text regions of objects in the object image (e.g., license plates of vehicles, traffic directions on traffic signs, road directions on road markings). In one embodiment, the electronic device 101 can identify regions of interest in the object image based on an image segmentation model. In FIG. 5 , the object image at the bottom left is an enlarged version of the object image at the bottom left, and the object image at the top right and the object image at the top right may be the same image.

[0101] In one embodiment, the electronic device 101 can calculate (or identify) (or determine) the vertices through geometric fitting (e.g., fitting via a triangle, a quadrilateral, a pentagon, or a polygon) in the region of interest. In one embodiment, the electronic device 101 can identify the coordinates of each of the vertices through geometric fitting in the region of interest. For example, in a two-dimensional virtual coordinate system in which the lower leftmost vertex of the vertices is set as the origin, the electronic device 101 can identify the coordinates of each vertex.

[0102] In one embodiment, the electronic device 101 can identify a fitted region of interest within the region of the object in the cropped first image.

[0103] In operation 740, the electronic device 101 may identify a second region of interest corresponding to the first region of interest within the second overlap region of the second image. In one embodiment, the electronic device 101 may sequentially set an object region in the second overlap region of the second image and set a second region of interest within the object region.

[0104] In one embodiment, the electronic device 101 can mark (or set) (or identify) (or designate) regions of objects and regions of interest in the cropped second image. In one embodiment, the electronic device 101 can mark (or set) (or identify) (or designate) regions of objects and regions of interest in the cropped second image based on regions of objects and regions of interest in the cropped first image. In one embodiment, the size and position of the regions of objects displayed in the cropped second image may be equal to the size and position of the regions of objects displayed in the cropped first image. In one embodiment, the size and position of the regions of interest displayed in the cropped second image may be equal to the size and position of the regions of interest displayed in the cropped first image.

[0105] At operation 750, the electronic device 101 may label the object identified in the first region of interest in a second region of interest. In one embodiment, the electronic device 101 may label the character string of the object identified in the first region of interest in the second region of interest.

[0106] In one embodiment, the electronic device 101 can transform the fitted region of interest. For example, the electronic device 101 can transform the fitted region of interest via a specified transformation algorithm (e.g., an algorithm for a rigid-body transformation, an approximation transformation, a linear transformation, an affine transformation, and / or a perspective transformation). In one embodiment, the electronic device 101 can transform the fitted region of interest into a license plate-shaped figure (e.g., a rectangle), but is not limited to this.

[0107] In one embodiment, an algorithm (or operation) for identifying (or recognizing) characters displayed on the license plate can be performed (or executed) on the (transformed) fitted region of interest of the cropped first image. The algorithm can include an algorithm for identifying characters in the image (e.g., an algorithm using an optical character recognition (OCR) function). In one embodiment, the character string displayed on the license plate can be identified (or recognized) based on an OCR algorithm of the fitted region of interest of the cropped first image.

[0108] In one embodiment, electronic device 101 can label the region of interest in the cropped second image with a string of characters that appeared on the license plate, which in one embodiment can be the same string of characters that was identified in the fitted region of interest in the cropped first image.

[0109] The electronic device 101 can then set (or identify) (or acquire) a first cropped image in which the object area and the area of ​​interest are marked, and a second cropped image in which the object area and the area of ​​interest are marked, as a training data set.

[0110] The electronic device 101 can then repeat the operations performed on the first and second images for subsequent images (eg, the first and second images).

[0111] FIG. 8 illustrates an example block diagram illustrating an autonomous driving system for a vehicle, according to one embodiment.

[0112] The vehicle autonomous driving system 800 of FIG. 8 may be a deep learning network including a sensor 803, an image preprocessor 805, a deep learning network 807, an artificial intelligence (AI) processor 809, a vehicle control module 811, a network interface 813, and a communication unit 815. In various embodiments, each element may be connected via various interfaces. For example, sensor data sensed and output by the sensor 803 may be fed to the image preprocessor 805. The sensor data processed by the image preprocessor 805 may be fed to the deep learning network 807 executed by the AI ​​processor 809. The output of the deep learning network 807 executed by the AI ​​processor 809 may be fed to the vehicle control module 811. An intermediate result of the deep learning network 807 executed by the AI ​​processor 809 may be fed to the AI ​​processor 809. In various embodiments, network interface 813 communicates with on-board electronic devices (e.g., electronic device 101 and / or cameras 151, 155 in FIG. 1 ) to transmit autonomous driving path information and / or autonomous driving control commands for the autonomous driving of the vehicle to the internal block configuration. In one embodiment, network interface 813 may be used to transmit sensor data acquired via sensors 803 to an external server. In some embodiments, autonomous driving system 800 may include additional or fewer components as appropriate. For example, in some embodiments, image preprocessor 805 may be an optional component. In another example, a post-processing component (not shown) may be included in autonomous driving system 800 to perform post-processing on the output of deep learning network 807 before the output is provided to vehicle control module 811.

[0113] In some embodiments, the sensor 803 may include one or more sensors. In various embodiments, the sensor 803 may be mounted at different locations on the vehicle. The sensor 803 may be oriented in one or more different directions. For example, the sensor 803 may be mounted on the front, sides, rear, and / or roof of the vehicle so as to face in a direction such as forward-facing, rear-facing, or side-facing. In some embodiments, the sensor 803 may be an image sensor such as a high dynamic range camera. In some embodiments, the sensor 803 includes non-visual sensors. In some embodiments, the sensor 803 includes a RADAR, a Light Detection and Ranging (LiDAR), and / or an ultrasonic sensor in addition to an image sensor. In some embodiments, the sensor 803 is not mounted on the vehicle with the vehicle control module 811. For example, sensors 803 may be included as part of a deep learning system to capture sensor data and may be mounted in the environment or road and / or mounted in surrounding vehicles.

[0114] In some embodiments, an image pre-processor 805 may be used to pre-process sensor data from the sensor 803. For example, the image pre-processor 805 may be used to pre-process the sensor data, split the sensor data into one or more components, and / or post-process one or more components. In some embodiments, the image pre-processor 805 may be a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a specialized image processor. In various embodiments, the image pre-processor 805 may be a tone-mapper processor for processing high dynamic range data. In some embodiments, the image pre-processor 805 may be a component of the AI ​​processor 809.

[0115] In some embodiments, deep learning network 807 may be a deep learning network for implementing control instructions for controlling an autonomous vehicle. For example, deep learning network 807 may be an artificial neural network, such as a convolutional neural network (CNN), trained using sensor data, and the output of deep learning network 807 is provided to vehicle control module 811.

[0116] In some embodiments, artificial intelligence (AI) processor 809 may be a hardware processor for running deep learning network 807. In some embodiments, AI processor 809 is a specialized AI processor for performing inference via convolutional neural networks (CNNs) on sensor data. In some embodiments, AI processor 809 may be optimized for the bit depth of the sensor data. In some embodiments, AI processor 809 may be optimized for deep learning operations, such as neural network operations including convolution, dot product, vector, and / or matrix operations. In some embodiments, AI processor 809 may be implemented via multiple graphics processing units (GPUs) capable of efficiently performing parallel processing.

[0117] In various embodiments, AI processor 809 can be coupled via an input / output interface to memory configured to provide the AI ​​processor with instructions that, while running, cause the AI ​​processor 809 to perform deep learning analysis on sensor data received from sensors 803 and determine machine learning results used to operate the vehicle at least partially autonomously. In some embodiments, a Vehicle Control Module 811 can be used to process vehicle control instructions output from artificial intelligence (AI) processor 809 and translate the output of AI processor 809 into instructions for controlling various vehicle modules to control each vehicle module. In some embodiments, the Vehicle Control Module 811 is used to control the vehicle for autonomous driving. In some embodiments, the Vehicle Control Module 811 can adjust the steering and / or speed of the vehicle. For example, the Vehicle Control Module 811 can be used to control vehicle navigation, such as deceleration, acceleration, steering, lane changes, and lane keeping. In some embodiments, the vehicle control module 811 may generate control signals to control vehicle lighting, such as brake lights, turn signals, headlights, etc. In some embodiments, the vehicle control module 811 may be used to control vehicle audio-related systems, such as the vehicle's sound system, the vehicle's audio warnings, the vehicle's microphone system, the vehicle's horn system, etc.

[0118] In some embodiments, vehicle control module 811 may be used to control notification systems, including warning systems to inform passengers and / or the driver of driving events, such as failure to access an intended destination or a potential collision. In some embodiments, vehicle control module 811 may be used to adjust sensors, such as sensor 803, of the vehicle. For example, vehicle control module 811 may modify the orientation of sensor 803, change the output resolution and / or format type of sensor 803, increase or decrease the capture rate, adjust the dynamic range, or adjust the focus of a camera. Additionally, vehicle control module 811 may turn on / off the operation of sensors individually or collectively.

[0119] In some embodiments, the vehicle control module 811 may be used to modify parameters of the image preprocessor 805, such as by changing the range of filter frequencies, adjusting edge detection parameters for feature and / or object detection, adjusting channels and bit depth, etc. In various embodiments, the vehicle control module 811 may be used to control the autonomous driving of the vehicle and / or driver assistance features of the vehicle.

[0120] In some embodiments, network interface 813 may serve as an internal interface between the block configuration of autonomous driving system 800 and communication unit 815. Specifically, network interface 813 may be a communication interface for receiving and / or transmitting data, including voice data. In various embodiments, network interface 813 may connect to an external server via communication unit 815 to connect to voice calls, receive and / or transmit text messages, transmit sensor data, update vehicle software in the autonomous driving system, or update software in the vehicle's autonomous driving system.

[0121] In various embodiments, the communication unit 815 can include various wireless interfaces, such as cellular or WiFi. For example, the network interface 813 can be used to receive updates to operational parameters and / or instructions for the sensors 803, the image preprocessor 805, the deep learning network 807, the AI ​​processor 809, and the vehicle control module 811 from an external server connected via the communication unit 815. For example, the machine learning model of the deep learning network 807 can be updated using the communication unit 815. According to yet another example, the communication unit 815 can be used to update operational parameters of the image preprocessor 805, such as image processing parameters, and / or firmware of the sensors 803.

[0122] In another embodiment, the communication unit 815 can be used to activate communications for emergency services and emergency contact in the event of an accident or near-accident. For example, in a crash event, the communication unit 815 can be used to call emergency services for assistance and to notify emergency services of the crash details and vehicle location. In various embodiments, the communication unit 815 can update or obtain an expected time of arrival and / or destination location.

[0123] According to one embodiment, the autonomous driving system 800 shown in Fig. 8 may be configured in the vehicle electronic device 101. According to one embodiment, when an autonomous driving cancellation event is generated by the user while the vehicle is autonomously driving, the AI ​​processor 809 of the autonomous driving system 800 can control the input of information related to the autonomous driving cancellation event into the training set data of the deep learning network, thereby controlling the autonomous driving software of the vehicle to learn.

[0124] Figures 9 and 10 show example block diagrams illustrating autonomous vehicles according to one embodiment. Figure 11 shows an example gateway associated with a user device according to various embodiments.

[0125] Referring to FIG. 9, an autonomous vehicle 900 according to this embodiment may include a control device 1000, sensing modules 904a, 904b, 904c, and 904d, an engine 906, and a user interface 908.

[0126] The autonomous vehicle 900 can have an autonomous mode or a manual mode, and can switch from manual mode to autonomous mode or vice versa according to user input received via the user interface 908, as an example.

[0127] When the autonomous vehicle 900 is operated in the autonomous driving mode, the autonomous vehicle 900 can operate under the control of the control device 1000.

[0128] In this embodiment, the control device 1000 may include a controller 1020 including a memory 1022 and a processor 1024 , a sensor 1010 , a communication device 1030 , and an object detection device 1040 .

[0129] Here, the object detection device 1040 can perform all or part of the functions of the distance measurement device.

[0130] That is, in this embodiment, the object detection device 1040 is a device for detecting objects located outside the autonomously driving vehicle 900, and the object detection device 1040 can detect objects located outside the autonomously driving vehicle 900 and generate object information according to the detection results.

[0131] The object information can include information about the presence or absence of the object, position information of the object, distance information between the moving body and the object, and relative speed information between the moving body and the object.

[0132] The objects may include various objects located outside the autonomous vehicle 900, such as lanes, other vehicles, pedestrians, traffic signals, lights, roads, structures, speed limiters, terrain features, and animals. Here, the traffic signals may be a concept that includes traffic lights, traffic signs, and patterns or text painted on the road surface. The light may be light generated from lamps installed on other vehicles, light generated by street lamps, or sunlight.

[0133] The structures may be objects located around the road and fixed to the ground. For example, the structures may include streetlights, roadside trees, buildings, utility poles, traffic lights, bridges, etc. The terrain features may include mountains, hills, etc.

[0134] Such an object detection device 1040 may include a camera module, and the controller 1020 may extract object information from external images captured by the camera module and process the information.

[0135] The object detection device 1040 may further include an imaging device for recognizing the external environment. In addition to LIDAR, RADAR, GPS devices, odometry and other computer vision devices, ultrasonic sensors, infrared sensors, etc. may be used, and these devices may be selected or operated simultaneously as needed to enable more accurate sensing.

[0136] Meanwhile, a distance measurement device according to one embodiment of the present invention can calculate the distance between an autonomous vehicle 900 and an object, and control the operation of the vehicle based on the calculated distance in cooperation with a control device 1000 of the autonomous vehicle 900.

[0137] As one example, if there is a possibility of a rear-end collision depending on the distance between the autonomous vehicle 900 and the object, the autonomous vehicle 900 can control the brakes to reduce or stop the speed. As another example, if the object is a moving object, the autonomous vehicle 900 can control the traveling speed of the autonomous vehicle 900 to maintain a predetermined distance or more from the object.

[0138] The distance measurement device according to one embodiment of the present invention can be configured as one module in the control device 1000 of the autonomous vehicle 900. That is, the memory 1022 and the processor 1024 of the control device 1000 can realize the rear-end collision prevention method according to the present invention in software terms.

[0139] The sensor 1010 can acquire various sensing information about the internal / external environment of the vehicle by connecting to the sensing modules 904a, 904b, 904c, and 904d. Here, the sensor 1010 can include an attitude sensor (e.g., a yaw sensor, a roll sensor, and a pitch sensor), a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight sensor, a heading sensor, a gyro sensor, a position module, a vehicle forward / reverse sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on steering wheel rotation, a vehicle internal temperature sensor, a vehicle internal humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.

[0140] As a result, the sensor 1010 can acquire sensing signals for mobile body posture information, mobile body collision information, mobile body direction information, mobile body position information (GPS information), mobile body angle information, mobile body speed information, mobile body acceleration information, mobile body inclination information, mobile body forward / reverse information, battery information, fuel information, tire information, mobile body lamp information, mobile body internal temperature information, mobile body internal humidity information, steering wheel rotation angle, mobile body external illuminance, pressure on the accelerator pedal, pressure on the brake pedal, etc.

[0141] In addition, the sensor 1010 may further include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), etc.

[0142] In this way, the sensor 1010 can generate mobile object status information based on the sensing data.

[0143] The wireless communication device 1030 is configured to perform wireless communication between the autonomous mobile bodies 900. For example, the wireless communication device 1030 enables communication between the autonomous mobile body 900 and a user's mobile phone, another wireless communication device 1030, another mobile body, a central device (traffic control device), a server, etc. The wireless communication device 1030 can transmit and receive wireless signals according to a connection wireless protocol. Examples of wireless communication protocols include, but are not limited to, Wi-Fi, Bluetooth, LTE (Long-Term Evolution), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), and GSM (Global Systems for Mobile Communications).

[0144] Furthermore, the autonomously traveling vehicle 900 in this embodiment can also realize communication between vehicles via the wireless communication device 1030. That is, the wireless communication device 1030 can communicate with other vehicles on the road or other vehicles through vehicle-to-vehicle (V2V) communication. The autonomously traveling vehicle 900 can transmit and receive information such as driving warnings and traffic information through vehicle-to-vehicle communication, and can also request information from or receive requests for information from other vehicles. For example, the wireless communication device 1030 can perform V2V communication using a dedicated short-range communication (DSRC) device or a cellular-V2V (C-V2V) device. In addition to vehicle-to-vehicle communication, communication between a vehicle and another object (e.g., an electronic device carried by a pedestrian) (V2X, Vehicle to Everything communication) can also be realized via the wireless communication device 1030.

[0145] In addition, the wireless communication device 1030 can acquire information generated from various mobilities including infrastructure located on the road (traffic lights, CCTV, RSU, eNode B, etc.) or other autonomous driving / non-autonomous driving vehicles via a non-terrestrial network rather than a terrestrial network, as information for performing autonomous driving of the autonomous driving mobile body 900.

[0146] For example, the wireless communication device 1030 can perform wireless communication with a Low Earth Orbit (LEO) satellite system, a Medium Earth Orbit (MEO) satellite system, a Geostationary Orbit (GEO) satellite system, a High Altitude Platform (HAP) system, etc., which constitute a non-terrestrial network, via a dedicated antenna for the non-terrestrial network mounted on the autonomous vehicle 900.

[0147] For example, the wireless communication device 1030 may be a 5G NR NTN (5G NR NTN) currently being discussed by 3GPP and the like. TH It can communicate wirelessly with various platforms that make up NTN in accordance with wireless connection standards that comply with the NTN Generation New Radio Non-Terrestrial Network (NTN) standards, but is not limited to this.

[0148] In this embodiment, the controller 1020 can select a platform that can appropriately perform NTN communication, taking into consideration various information such as the position of the autonomous vehicle 900, the current time, and available power, and control the wireless communication device 1030 to perform wireless communication with the selected platform.

[0149] In this embodiment, the controller 1020 is a unit that controls the overall operation of each unit in the autonomous vehicle 900, and may be configured by the vehicle manufacturer at the time of manufacture, or may be further configured to perform autonomous driving functions after manufacture. Alternatively, the controller 1020 may include a configuration for performing continuous additional functions through an upgrade of the controller 1020 configured at the time of manufacture. Such a controller 1020 may be referred to as an ECU (Electronic Control Unit).

[0150] The controller 1020 collects various data from the connected sensors 1010, the object detection device 1040, the communication device 1030, etc., and based on the collected data, can transmit control signals to other components in the vehicle, including the sensors 1010, the engine 906, the user interface 908, the communication device 1030, and the object detection device 1040. In addition, although not shown, the controller 1020 can also transmit control signals to an acceleration device, a braking system, a steering device, or a navigation device related to the traveling of the vehicle.

[0151] In this embodiment, the controller 1020 can control the engine 906, for example, by detecting the speed limit of the road on which the autonomous vehicle 900 is traveling and controlling the engine 906 so that the traveling speed does not exceed the speed limit, or by controlling the engine 906 to accelerate the traveling speed of the autonomous vehicle 900 within a range that does not exceed the speed limit.

[0152] Furthermore, when the autonomous vehicle 900 approaches or departs from a lane while traveling, the controller 1020 can determine whether such approaching or leaving the lane corresponds to a normal driving condition or another driving condition, and can control the engine 906 to control the traveling of the vehicle according to the determination result. Specifically, the autonomous vehicle 900 can detect lanes formed on both sides of the road on which the vehicle is traveling. In this case, the controller 1020 can determine whether the autonomous vehicle 900 is approaching or departs from a lane, and when it determines that the autonomous vehicle 900 is approaching or departs from a lane, can determine whether such traveling is due to the correct driving condition or another driving condition. Here, an example of a normal driving condition may be a condition in which the vehicle needs to change lanes. An example of another driving condition may be a condition in which the vehicle does not need to change lanes. When the controller 1020 determines that the autonomously driving vehicle 900 is approaching a lane or leaving a lane in a situation where the vehicle does not need to change lanes, the controller 1020 can control the driving of the autonomously driving vehicle 900 so that the autonomously driving vehicle 900 does not leave the lane and drives normally.

[0153] If there is another moving body or an obstacle ahead of the moving body, the engine 906 or the braking system can be controlled to slow down the moving body, and in addition to the speed, the trajectory, travel path, steering angle, etc. Alternatively, the controller 1020 may generate necessary control signals to control the traveling of the moving body according to recognition information of other external environments such as the lane the moving body is traveling in, traffic signals, etc.

[0154] In addition to generating its own control signals, the controller 1020 can also control the movement of the vehicle by communicating with surrounding vehicles or a central server and, via the received information, sending commands to control peripheral devices.

[0155] In addition, when the position or angle of view of the camera module is changed, accurate moving body or lane recognition according to the present embodiment may be difficult. To prevent this, the controller 1020 may generate a control signal to control the camera module to perform calibration. Therefore, in the present embodiment, the controller 1020 generates a calibration control signal for the camera module, thereby continuously maintaining the normal mounting position, orientation, angle of view, etc. of the camera module even if the mounting position of the camera module is changed due to vibrations, shocks, etc. that occur with the movement of the autonomous mobile body 900. The controller 1020 may generate a control signal to perform calibration of the camera module when pre-stored initial mounting position, orientation, angle of view information of the camera module and initial mounting position, orientation, angle of view information, etc. of the camera module measured while the autonomous mobile body 900 is traveling change by more than a threshold value.

[0156] In this embodiment, the controller 1020 may include a memory 1022 and a processor 1024. The processor 1024 may execute software stored in the memory 1022 in accordance with control signals from the controller 1020. Specifically, the controller 1020 may store data and instructions in the memory 1022 for performing a lane detection method in accordance with the present invention, and the instructions may be executed by the processor 1024 to implement one or more of the methods disclosed herein.

[0157] In this case, the memory 1022 may be stored in a non-volatile recording medium executable by the processor 1024. The memory 1022 can store software and data via an appropriate internal or external device. The memory 1022 can be composed of a RAM (random access memory), a ROM (read only memory), a hard disk, and a memory 1022 device connected to a dongle.

[0158] The memory 1022 can store at least an operating system (OS), user applications, and executable instructions. The memory 1022 can also store application data and array data structures.

[0159] The processor 1024 may be a microprocessor or any suitable electronic processor, controller, microcontroller, or state machine.

[0160] The processor 1024 may be implemented as a combination of computing devices, which may comprise a digital signal processor, a microprocessor, or any suitable combination thereof.

[0161] Meanwhile, the autonomous vehicle 900 may further include a user interface 908 for user input to the control device 1000 described above. The user interface 908 allows the user to input information through appropriate interaction. For example, the user interface 908 may be implemented using a touch screen, a keypad, operation buttons, or the like. The user interface 908 transmits input or commands to the controller 1020, and the controller 1020 can execute control operations of the vehicle in response to the input or commands.

[0162] Additionally, the user interface 908 is a device external to the autonomous vehicle 900 and can communicate with the autonomous vehicle 900 via a wireless communication device 1030. For example, the user interface 908 can be linked to a mobile phone, tablet, or other computing device.

[0163] Additionally, although the autonomous vehicle 900 has been described in this embodiment as including an engine 906, it may include other types of propulsion systems. For example, the vehicle may be powered by electric energy, hydrogen energy, or a hybrid system that combines these. Thus, the controller 1020 may include the propulsion mechanisms of the autonomous vehicle 900's propulsion system and provide control signals to the configuration of each propulsion mechanism.

[0164] The detailed configuration of the control device 1000 according to this embodiment will be described below in more detail with reference to FIG.

[0165] The control device 1000 includes a processor 1024. The processor 1024 may be a general-purpose single-chip or multi-chip microprocessor, a special-purpose microprocessor, a microcontroller, a programmable gate array, or the like. The processor may be referred to as a central processing unit (CPU). In this embodiment, the processor 1024 may also be a combination of multiple processors.

[0166] The controller 1000 also includes a memory 1022. The memory 1022 can be any electronic component capable of storing electronic information. The memory 1022 can also include a single memory as well as a combination of memories 1022.

[0167] Data 1022b and instructions 1022a for executing the distance measurement method of the distance measurement device according to the present invention may be stored in memory 1022. When processor 1024 executes instructions 1024a, all or part of instructions 1024a and data 1024b required for executing the instructions may be loaded onto processor 1024.

[0168] The control device 1000 may include a transmitter 1030a, a receiver 1030b, or a transceiver 1030c to enable transmission and reception of signals. One or more antennas 1032a, 1032b may be electrically connected to the transmitter 1030a, receiver 1030b, or each transceiver 1030c and may further include antennas.

[0169] The control device 1000 may include a digital signal processor (DSP) 1070. The DSP 1070 may enable the mobile to quickly process digital signals.

[0170] The control device 1000 may include a communication interface 1080. The communication interface 1080 may include one or more ports and / or communication modules for connecting other devices to the control device 1000. The communication interface 1080 may allow a user to interact with the control device 1000.

[0171] The various components of the controller 1000 may be connected together by one or more buses 1090, which may also include a power bus, a control signal bus, a status signal bus, a data bus, etc. Under the control of the processor 1024, the components may communicate information with each other via the bus 1090 to perform desired functions.

[0172] Meanwhile, in various embodiments, the control device 1000 may be associated with a gateway for communication with the security cloud. For example, referring to FIG. 11 , the control device 1000 may be associated with a gateway 1105 for providing information acquired from at least one of the components 1101 to 1104 of the vehicle 1100 to the security cloud 1106. For example, the gateway 1105 may be included in the control device 1000. In another example, the gateway 1105 may be configured as a separate device in the vehicle 1100 that is distinct from the control device 1000. The gateway 1105 communicatively connects the software management cloud 1109, which have different networks, the security cloud 1106, and the network in the vehicle 1100 secured by the in-vehicle security software 1110.

[0173] For example, the component 1101 may be a sensor. For example, the sensor may be used to obtain information about at least one of a state of the vehicle 1100 or a state surrounding the vehicle 1100. For example, the component 1101 may include the sensor 1010.

[0174] For example, component 1102 may be an electronic control unit (ECU), which may be used for engine control, transmission control, airbag control, and tire pressure management.

[0175] For example, component 1103 may be an instrument cluster. For example, an instrument cluster may refer to a panel of a dashboard located in front of the driver's seat. For example, the instrument cluster may be configured to display information necessary for driving to the driver (or passengers). For example, the instrument cluster may be used to display at least one of a visual element for indicating the revolutions per minute (RPM, revolutions per minute, or rotates per minute) of the engine, a visual element for indicating the speed of the vehicle 1100, a visual element for indicating the amount of remaining fuel, a visual element for indicating the status of the gear, or a visual element for indicating information obtained via component 1101.

[0176] For example, the component 1104 may be a telematics device. For example, the telematics device may refer to a device that combines wireless communication technology and global positioning system (GPS) technology to provide various mobile communication services within the vehicle 1100, such as location information and safe driving. For example, the telematics device may be used to connect the vehicle 1100 with a driver, a cloud (e.g., the security cloud 1106), and / or the surrounding environment. For example, the telematics device may be configured to support high bandwidth and low latency for 5G NR standard technologies (e.g., 5G NR V2X technology, 5G NR NTN (Non-Terrestrial Network) technology). For example, the telematics device may be configured to support autonomous driving of the vehicle 1100.

[0177] For example, the gateway 1105 can be used to connect a network within the vehicle 1100 with a software management cloud 1109 and a security cloud 1106, which are networks outside the vehicle. For example, the software management cloud 1109 can be used to update or manage at least one software necessary for the operation and management of the vehicle 1100. For example, the software management cloud 1109 can interface with in-car security software 1110 installed in the vehicle. For example, the in-car security software 1110 can be used to provide security functions within the vehicle 1100. For example, the in-car security software 1110 can encrypt data transmitted and received over the in-car network using an encryption key obtained from an external authorized server for encryption of the in-vehicle network. In various embodiments, the encryption key used by the in-car security software 1110 can be generated corresponding to vehicle identification information (such as a vehicle license plate or a vehicle identification number (VIN)) or information uniquely assigned to each user (such as user identification information).

[0178] In various embodiments, the gateway 1105 can transmit data encrypted by the in-vehicle security software 1110 based on the encryption key to the software management cloud 1109 and / or the security cloud 1106. The software management cloud 1109 and / or the security cloud 1106 can identify which vehicle or which user the data was received from by decrypting the data using a decryption key that can decrypt the data encrypted with the encryption key of the in-vehicle security software 1110. For example, because this decryption key is a unique key corresponding to the encryption key, the software management cloud 1109 and / or the security cloud 1106 can identify the sender of the data (e.g., the vehicle or the user) based on the data decrypted using the decryption key.

[0179] For example, gateway 1105 may be configured to support in-vehicle security software 1110 and may be associated with control device 1000. For example, gateway 1105 may be associated with control device 1000 to support a connection between client device 1107 connected to security cloud 1106 and control device 1000. In another example, gateway 1105 may be associated with control device 1000 to support a connection between third party cloud 1108 connected to security cloud 1106 and control device 1000. However, this is not limiting.

[0180] In various embodiments, the gateway 1105 can be used to connect the vehicle 1100 with a software management cloud 1109 for managing the operating software of the vehicle 1100. For example, the software management cloud 1109 can monitor whether an update of the operating software of the vehicle 1100 is requested and, based on monitoring that an update of the operating software of the vehicle 1100 has been requested, provide data for updating the operating software of the vehicle 1100 via the gateway 1105. As another example, the software management cloud 1109 can receive a user request from the vehicle 1100 via the gateway 1105 for an update of the operating software of the vehicle 1100 and, based on this reception, provide data for updating the operating software of the vehicle 1100.

[0044] As an example, but not limited to, the software management cloud 1109 can receive a user request from the vehicle 1100 for an update of the operating software of the vehicle 1100 via the gateway 1105 and, based on this reception, provide data for updating the operating software of the vehicle 1100.

[0181] FIG. 12 is a diagram illustrating the operation of an electronic device for training a neural network based on a set of training data, according to one embodiment.

[0182] The operations described with reference to FIG. 12 may be performed by the electronic device previously described (eg, electronic device 101 of FIG. 1).

[0183] Referring to FIG. 12 , in operation 1202, an electronic device according to an embodiment may acquire a set of training data. The electronic device may acquire a set of training data for supervised learning. The training data may include a pair of input data and ground truth data corresponding to the input data. The ground truth data may represent output data to be obtained from a neural network that receives the input data that is the pair of the ground truth data. The ground truth data may be acquired by the electronic device.

[0184] For example, when training a neural network to recognize images, the training data may include information about the images and one or more objects contained in the images. Such information may include a category or class of the objects identifiable through the images. The information may include a position, width, height, and / or size of a visual object in the image corresponding to the object. The set of training data identified through operation 1202 may include multiple training data pairs. In the above example of training a neural network to recognize images, the set of training data identified by the electronic device may include multiple images and base truth data corresponding to each of the multiple images.

[0185] Referring to Figure 12, at operation 1204, an electronic device according to one embodiment may train a neural network based on a set of training data. In one embodiment in which the neural network is trained based on supervised learning, the electronic device may input input data included in the training data to an input layer of the neural network. An example of a neural network including the input layer is described with reference to Figure 13. From an output layer of the neural network that receives the input data via the input layer, the electronic device may obtain output data of the neural network corresponding to the input data.

[0186] In one embodiment, the training of operation 1204 may be performed based on a difference between the output data and underlying truth data included in the training data and corresponding to the input data. For example, the electronic device may adjust one or more parameters associated with the neural network (e.g., weights, as described below with reference to FIG. 13 ) based on a gradient descent algorithm to reduce the difference. The operation of the electronic device adjusting the one or more parameters may be referred to as tuning the neural network. The electronic device may tune the neural network based on the output data using a function defined to evaluate the performance of the neural network, such as a cost function. The difference between the output data and underlying truth data may be included as an example of the cost function.

[0187] 12 , in operation 1206, an electronic device according to an embodiment may identify whether valid output data is output from the neural network trained by operation 1204. Valid output data may mean that the difference (or cost function) between the output data and the basis truth data satisfies the conditions set for using the neural network. For example, if the average and / or maximum value of the difference between the output data and the basis truth data is equal to or less than a specified threshold, the electronic device may determine that valid output data is output from the neural network.

[0188] If the neural network does not output valid output data (1206-NO), the electronic device may repeatedly perform training of the neural network based on operation 1204. The embodiment is not limited thereto, and the electronic device may repeatedly perform operations 1202 and 1204.

[0189] After obtaining valid output data from the neural network (operation 1206—YES), the electronic device according to an embodiment can use the trained neural network based on operation 1208. For example, the electronic device can input other input data, distinct from the input data input to the neural network as learning data, to the neural network. The electronic device can use the output data obtained from the neural network that has received the other input data as a result of inferring the other input data based on the neural network.

[0190] FIG. 13 is a block diagram of an electronic device according to one embodiment.

[0191] The electronic device 1300 of FIG. 13 may include the electronic device 101 described above.

[0192] For example, the operations described with reference to FIG. 12 may be performed by electronic device 1300 of FIG. 13 and / or processor 1310 of FIG.

[0193] 13, a processor 1310 of an electronic device 1300 can perform computations related to a neural network 1330 stored in a memory 1320. The processor 1310 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), or a neural processing unit (NPU). The NPU may be implemented as a chip separate from the CPU or may be integrated into a chip such as the CPU in the form of a system on a chip (SoC). An NPU integrated into a CPU may be referred to as a neural core and / or an artificial intelligence (AI) accelerator.

[0194] Referring to FIG. 13 , the processor 1310 may identify a neural network 1330 stored in the memory 1320. The neural network 1330 may include a combination of an input layer 1332, one or more hidden layers 1334 (or intermediate layers), and an output layer 1336. The aforementioned layers (e.g., the input layer 1332, the one or more hidden layers 1334, and the output layer 1336) may include multiple nodes. The number of hidden layers 1334 may vary depending on the embodiment, and a neural network 1330 including multiple hidden layers 1334 may be referred to as a deep neural network. The operation of training the deep neural network may be referred to as deep learning.

[0195] In one embodiment, when the neural network 1330 has a feedforward neural network structure, a first node in a particular layer may be connected to all of the second nodes in another layer preceding the particular layer. The parameters stored in the memory 1320 for the neural network 1330 may include weights assigned to connections between the second node and the first node. In the neural network 1330 having a feedforward neural network structure, the value of the first node may correspond to a weighted sum of values ​​assigned to the second node based on the weights assigned to connections connecting the second node and the first node.

[0196] In one embodiment, when the neural network 1330 has a convolutional neural network structure, a first node included in a particular layer may correspond to a weighted combination of a portion of a second node included in another layer preceding the particular layer. The portion of the second node corresponding to the first node may be identified by a filter corresponding to the particular layer. Parameters stored in memory 1320 for the neural network 1330 may include weights representing the filters. The filter may include one or more of the second nodes used to calculate the weighted combination of the first node and weights corresponding to each of the one or more nodes.

[0197] According to one embodiment, processor 1310 of electronic device 1300 may train neural network 1330 using training data set 1340 stored in memory 1320. Based on training data set 1340, processor 1310 may perform the operations described with reference to FIG. 12 to adjust one or more parameters stored in memory 1320 for neural network 1330.

[0198] According to one embodiment, the processor 1310 of the electronic device 1300 can perform object detection, object recognition, and / or object classification using a neural network 1330 trained based on a learning dataset 1340. The processor 1310 can input an image (or video) acquired via a camera 1350 to an input layer 1332 of the neural network 1330. Based on the image input to the input layer 1332, the processor 1310 can sequentially obtain values ​​of nodes in the layers included in the neural network 1330 and obtain a set of node values ​​(e.g., output data) in the output layer 1336. The output data can be used as a result of estimating information included in the image using the neural network 1330. An embodiment is not limited thereto, and the processor 1310 can input an image (or video) acquired from an external electronic device connected to the electronic device 1300 to the neural network 1330 via the communication circuit 1360.

[0199] In one embodiment, neural network 1330 trained to process an image can be used to identify regions in the image that correspond to objects (object detection) and / or identify classes of objects depicted in the image (object recognition and / or object classification). For example, electronic device 1300 can use neural network 1330 to segment regions in the image that correspond to the objects based on rectangular shapes, such as bounding boxes. For example, electronic device 1300 can use neural network 1330 to identify at least one class of a plurality of specified classes that matches the object.

[0200] A conventional truck 10 is shown in FIGS.

[0201] FIG. 14 shows the tractor 12 and trailer 14 in an unconnected state.

[0202] 15 shows the tractor 12 connected to the trailer 14. In this embodiment of the invention, the trailer 14 is selectively connected by a steering wheel hitch 16 carried by the tractor 12, which engages a kingpin 18 fixed to the trailer 14 in a known manner.

[0203] Although the trailer 20 shown in FIG. 14 of this specification is shown in a "semi-trailer" configuration, this is for convenience of explanation, and it should not be understood that the embodiments of the present invention apply only to the "semi-trailer" configuration.

[0204] As described above, the electronic device 101 can include communications circuitry 110. The electronic device 101 can include memory 130 that stores instructions. The electronic device 101 can include at least one processor 120 operatively connected to the communications circuitry 110 and the memory 130. The instructions, when executed by the processor 120, can cause the electronic device 101 to acquire first images 211-220 from a first camera 151 via the communications circuitry 110. The instructions, when executed by the processor 120, can cause the electronic device 101 to acquire second images 221-230 from a second camera 155, the second camera having a slower shutter speed than the first camera 151, via the communications circuitry 110. When the instructions are executed by the processor 120, the electronic device 101 can be caused to identify the first image 211 and the second image 221, which were acquired at corresponding times, between the first image 211-220 and the second image 221-230. When the instructions are executed by the processor 120, the electronic device 101 can be caused to set, in the first image 211, a first region of interest 540 within a specified type of object. When the instructions are executed by the processor 120, the electronic device 101 can be caused to set, in the second image 221, a second region of interest 640 at the same position as the first region of interest 540. When the instructions are executed by the processor 120, the electronic device 101 can be caused to generate, as training data, the first image 211 in which the first region of interest 540 is set and the second image 221 in which the second region of interest 640 is set.

[0205] In one embodiment, the second camera 155 may be positioned at a corresponding angle to the first camera 151. In one embodiment, the lens attributes of the second camera 155 may be the same as the lens attributes of the first camera 151. When the instructions are executed by the processor 120, the electronic device 101 may cause the first camera 151 and the second camera 155 to take pictures at corresponding times.

[0206] When the instructions are executed by the processor 120, the electronic device 101 can cause the first camera 151 and the second camera 155 to capture first images 211-220 and second images 221-230 within a specified time period after instructing them to capture images at corresponding times. When the instructions are executed by the processor 120, the electronic device 101 can also cause the electronic device 101 to identify an image (e.g., image 225) captured at a corresponding time to a reference image (e.g., image 215) among the first images 211-220 based on a comparison between the reference image (e.g., image 215) and each of the second images 221-230. When the instructions are executed by the processor 120, the electronic device 101 can also cause the electronic device 101 to identify the second image 221 from the second images 221-230, the second image 221 having the same frame difference as the frame difference from the first image 211 to the reference image (e.g., image 215).

[0207] When the instructions are executed by the processor 120, the electronic device 101 can cause the electronic device 101 to generate a cropped first image 211 by cropping a region 410 of a specified size in the first image 211. When the instructions are executed by the processor 120, the electronic device 101 can cause the electronic device 101 to generate a cropped second image 221 based on the cropped first image 211 by cropping another region 430 of a specified size in the second image 221.

[0208] When the instructions are executed by the processor 120, the electronic device 101 can be caused to identify normalized difference values ​​between a plurality of regions of the cropped first image 211 and the second image 221. When the instructions are executed by the processor 120, the electronic device 101 can be caused to identify normalized distances between feature points of the cropped first image 211 and feature points of each of a plurality of regions of the second image 221. When the instructions are executed by the processor 120, the electronic device 101 can be caused to generate the cropped second image 221 by cropping a region having the lowest weighted average between the normalized difference values ​​and the normalized distances.

[0209] When executed by the processor 120, the instructions can cause the electronic device 101 to identify an object of a specified type in the cropped first image 211. When executed by the processor 120, the instructions can cause the electronic device 101 to set the first region of interest 540 within the object identified in the cropped first image 211. When executed by the processor 120, the instructions can cause the electronic device 101 to set the position of the object in the cropped first image 211 in the cropped second image 221. When executed by the processor 120, the instructions can cause the electronic device 101 to set the second region of interest 640 in the cropped second image 221 at the same position as the first region of interest 540 in the cropped first image 211.

[0210] The instructions, when executed by the processor 120, can cause the electronic device 101 to identify a string based on a string identification algorithm in the first region of interest 540. The instructions, when executed by the processor 120, can cause the electronic device 101 to label the identified string in the first region of interest 540 and the second region of interest 640.

[0211] When the instructions are executed by the processor 120, the electronic device 101 can be caused to transform the first region of interest 540 into a specified shape based on a specified transformation algorithm, and when the instructions are executed by the processor 120, the electronic device 101 can be caused to identify the character string in the first region of interest 540 transformed into the specified shape based on the character string identification algorithm.

[0212] When executed by the processor 120, the instructions can cause the electronic device 101 to identify whether the electronic device 101 is moving based on a difference between consecutive ones of the first images 211-220. When executed by the processor 120, the instructions can cause the electronic device 101 to delete the first images 211-220 and the second images 221-230 based on the electronic device 101 not moving. When executed by the processor 120, the instructions can cause the electronic device 101 to generate the training data based on the first images 211-220 and the second images 221-230 based on the electronic device 101 being moving.

[0213] As described above, the method can be performed by an electronic device 101 including a communication circuit 110. The method can include an operation of acquiring, via the communication circuit 110, a first image 211-220 from a first camera 151. The method can include an operation of acquiring, via the communication circuit 110, second images 221-230 from a second camera 155 having a slower shutter speed than the first camera 151. The method can include an operation of identifying the first image 211 and the second image 221-230, which were acquired at corresponding times. The method can include an operation of setting, in the first image 211, a first region of interest 540 within a specified type of object. The method can include an operation of setting, in the second image 221, a second region of interest 640 at the same position as the first region of interest 540. The method may include generating a first image 211 having the first region of interest 540 defined therein and a second image 221 having the second region of interest 640 defined therein as training data.

[0214] The method may include an operation of acquiring first images 211-220 and second images 221-230 within a designated time period after instructing the first camera 151 and the second camera 155 to capture images at corresponding times. The method may include an operation of identifying images acquired at corresponding times to a reference image (e.g., image 215) among the first images 211-220 based on a comparison between the reference image (e.g., image 215) and each of the second images 221-230. The method may include an operation of identifying the second image 221 from the second images 221-230, the second image 221 having the same frame difference as the frame difference from the first image 211 to the reference image (e.g., image 215).

[0215] The method may include generating a cropped first image 211 by cropping a region 410 of a specified size in the first image 211. The method may include generating a cropped second image 221 based on the cropped first image 211 by cropping another region 430 of a specified size in the second image 221.

[0216] The method may include identifying normalized difference values ​​between the cropped first image 211 and a plurality of regions of the second image 221. The method may include identifying normalized distances between feature points of the cropped first image 211 and feature points of each of a plurality of regions of the second image 221. The method may include generating the cropped second image 221 by cropping a region having the lowest weighted average between the normalized difference values ​​and the normalized distances.

[0217] The method may include an act of identifying an object of a specified type in the cropped first image 211. The method may include an act of setting the first region of interest 540 within the object identified in the cropped first image 211. The method may include an act of setting a location of the object in the cropped first image 211 in the cropped second image 221. The method may include an act of setting the second region of interest 640 in the cropped second image 221 at the same location as the first region of interest 540 in the cropped first image 211.

[0218] The method may include an act of identifying whether the electronic device 101 is moving based on a difference between consecutive images among the first images 211-220. The method may include an act of deleting the first images 211-220 and the second images 221-230 based on the electronic device 101 not moving. The method may include an act of generating training data based on the first images 211-220 and the second images 221-230 based on the electronic device 101 moving.

[0219] As described above, a non-transitory computer-readable storage medium can store a program including instructions that, when executed by a processor 120 of an electronic device 101 including a communication circuit 110, can cause the electronic device 101 to acquire first images 211-220 from a first camera via the communication circuit 110. When executed by the processor 120, the electronic device 101 can acquire second images 221-230 from a second camera 155, which has a slower shutter speed than the first camera 151, via the communication circuit 110. When executed by the processor 120, the electronic device 101 can identify first images 211 and second images 221-230 that were acquired at corresponding times between the first images 211-220 and the second images 221-230. When the instructions are executed by the processor 120, the electronic device 101 can be caused to set a first region of interest 540 within a specified type of object in the first image 211. When the instructions are executed by the processor 120, the electronic device 101 can be caused to set a second region of interest 640 in the second image 221 at the same position as the first region of interest 540. When the instructions are executed by the processor 120, the electronic device 101 can be caused to generate, as training data, the first image 211 in which the first region of interest 540 is set and the second image 221 in which the second region of interest 640 is set.

[0220] When the instructions are executed by the processor 120, the electronic device 101 can cause the electronic device 101 to generate a cropped first image 211 by cropping a region 410 of a specified size in the first image 211. When the instructions are executed by the processor 120, the electronic device 101 can cause the electronic device 101 to generate a cropped second image 221 based on the cropped first image 211 by cropping another region 430 of a specified size in the second image 221.

[0221] When the instructions are executed by the processor 120, the electronic device 101 can be caused to identify normalized difference values ​​between a plurality of regions of the cropped first image 211 and the second image 221. When the instructions are executed by the processor 120, the electronic device 101 can be caused to identify normalized distances between feature points of the cropped first image 211 and feature points of each of a plurality of regions of the second image 221. When the instructions are executed by the processor 120, the electronic device 101 can be caused to generate the cropped second image 221 by cropping a region having the lowest weighted average between the normalized difference values ​​and the normalized distances.

[0222] When the instructions are executed by the processor 120, the electronic device 101 can cause the electronic device 101 to identify an object of a specified type in the cropped first image 211. When the instructions are executed by the processor 120, the electronic device 101 can cause the electronic device 101 to set the first region of interest 540 within the object identified in the cropped first image 211. When the instructions are executed by the processor 120, the electronic device 101 can cause the electronic device 101 to set the position of the object in the cropped first image 211 in the cropped second image 221. When the instructions are executed by the processor 120, the electronic device 101 can cause the electronic device 101 to set the second region of interest 640 in the cropped second image 221 at the same position as the first region of interest 540 in the cropped first image 211.

[0223] When executed by the processor 120, the instructions can cause the electronic device 101 to identify whether the electronic device 101 is moving based on a difference between consecutive ones of the first images 211-220. When executed by the processor 120, the instructions can cause the electronic device 101 to delete the first images 211-220 and the second images 221-230 based on the electronic device 101 not moving. When executed by the processor 120, the instructions can cause the electronic device 101 to generate the training data based on the first images 211-220 and the second images 221-230 based on the electronic device 101 moving.

[0224] As described above, the electronic device 101 may include cameras 151, 155. The electronic device 101 may include a memory 130 storing instructions. The electronic device 101 may include at least one processor 120 operatively connected to the cameras 151, 155 and the memory 130. When the instructions are executed by the processor 120, the electronic device 101 may cause the electronic device 101 to acquire first images 211-220 from a first camera 151. When the instructions are executed by the processor 120, the electronic device 101 may cause the electronic device 101 to acquire second images 221-230 from a second camera 155 having a slower shutter speed than the first camera 151. When the instructions are executed by the processor 120, the electronic device 101 may identify the first images 211 and second images 221-230, which were acquired at corresponding times. When the instructions are executed by the processor 120, the electronic device 101 can be caused to set a first region of interest 540 within a specified type of object in the first image 211. When the instructions are executed by the processor 120, the electronic device 101 can be caused to set a second region of interest 640 in the second image 221 at the same position as the first region of interest 540. When the instructions are executed by the processor 120, the electronic device 101 can be caused to generate, as training data, the first image 211 in which the first region of interest 540 is set and the second image 221 in which the second region of interest 640 is set.

[0225] As described above, the method can be performed by the electronic device 101 including cameras 151 and 155. The method can include an operation of acquiring first images 211-220 from a first camera 151. The method can include an operation of acquiring second images 221-230 from a second camera 155 having a slower shutter speed than the first camera 151. The method can include an operation of identifying the first image 211 and the second image 221, which were acquired at corresponding times in the first images 211-220 and the second images 221-230. The method can include an operation of setting a first region of interest 540 within a specified type of object in the first image 211. The method can include an operation of setting a second region of interest 640 in the second image 221 at the same position as the first region of interest 540. The method can include an operation of generating, as training data, a first image 211 in which the first region of interest 540 is set and a second image 221 in which the second region of interest 640 is set.

[0226] As described above, a non-transitory computer-readable recording medium can store a program including instructions that, when executed by a processor 120 of an electronic device 101 including cameras 151 and 155, can cause the electronic device 101 to acquire first images 211-220 from a first camera 151. When executed by the processor 120, the electronic device 101 can acquire second images 221-230 from a second camera 155 having a slower shutter speed than the first camera 151. When executed by the processor 120, the electronic device 101 can identify the first images 211 and second images 221-230 that were acquired at corresponding times. When the instructions are executed by the processor 120, the electronic device 101 can be caused to set a first region of interest 540 within a specified type of object in the first image 211. When the instructions are executed by the processor 120, the electronic device 101 can be caused to set a second region of interest 640 in the second image 221 at the same position as the first region of interest 540. When the instructions are executed by the processor 120, the electronic device 101 can be caused to generate, as training data, the first image 211 in which the first region of interest 540 is set and the second image 221 in which the second region of interest 640 is set.

[0227] It should be understood that an embodiment of the present disclosure and the terms used therein are not intended to limit the technical features described herein to a specific embodiment, but include various modifications, equivalents, or alternatives of the embodiment. In describing the drawings, similar or related components may use similar reference numerals. The singular form of a noun corresponding to an item may include one or more of the item, unless the relevant context clearly dictates otherwise. In this specification, each of phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" may include any of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," "first," or "second" may be used simply to distinguish a component from other corresponding components and do not limit the component in other aspects (e.g., importance or order). When a (e.g., first) component is referred to as being "coupled" or "connected" to another (e.g., second) component, either in combination with or without the terms "functionally" or "communicatively," this means that the component may be connected to the other component directly (e.g., by wire), wirelessly, or through a third component.

[0228] In the specific embodiments of the present disclosure described above, elements included in the disclosure are expressed in the singular or plural form according to the specific embodiment presented. However, the expressions in the singular or plural form are selected to suit the presented situation for the convenience of explanation, and the present disclosure is not limited to a singular or plural element, and elements expressed in the plural form may be composed in the singular, and elements expressed in the singular may be composed in the plural.

[0229] According to various embodiments, one or more of the aforementioned components or operations may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as performed by the corresponding component of the multiple components before integration. According to various embodiments, the operations performed by modules, programs, or other components may be performed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be performed in a different order, omitted, or one or more other operations may be added.

[0230] In the detailed description of the present disclosure, specific embodiments have been described, but it goes without saying that various modifications are possible without departing from the scope of the present disclosure.

Claims

1. 1. An electronic device, comprising: communication circuits; a memory for storing instructions; and at least one processor operatively connected to said communication circuitry and said memory; When the instructions are executed by the processor, the electronic device: acquiring a first image from a first camera via the communication circuit; acquiring a second image from a second camera having a slower shutter speed than the first camera via the communication circuit; identifying the first image and the second image, the first image and the second image being acquired at corresponding times; establishing a first region of interest within a specified type of object in the first image; setting a second region of interest in the second image at the same position as the first region of interest; An electronic device that generates, as training data, a first image in which the first region of interest is set and a second image in which the second region of interest is set.

2. the second camera is positioned at a corresponding angle to the first camera; the lens attributes of the second camera are the same as the lens attributes of the first camera; When the instructions are executed by the processor, the electronic device: The electronic device of claim 1 , wherein the first camera and the second camera are caused to take pictures at corresponding times.

3. When the instructions are executed by the processor, the electronic device: Instructing the first camera and the second camera to take pictures at corresponding times, and then acquiring a first image and a second image for a specified time; identifying an image acquired at a time corresponding to the reference image based on comparing a reference image among the first images with each of the second images; 2. The electronic device of claim 1, wherein the second image having the same frame difference as the frame difference from the first image to the reference image is caused to be distinguished from the second image.

4. When the instructions are executed by the processor, the electronic device: generating a cropped first image by cropping an area of ​​a specified size in the first image; The electronic device of claim 1 , wherein the electronic device is caused to generate a cropped second image by cropping another area of ​​a specified size in the second image based on the cropped first image.

5. When the instructions are executed by the processor, the electronic device: identifying normalized difference values ​​between the cropped first image and a plurality of regions of the second image; identifying normalized distances between feature points of the cropped first image and feature points of each of a plurality of regions of the second image; 5. The electronic device of claim 4, wherein the electronic device is caused to generate the cropped second image by cropping a region where a weighted average between the normalized difference value and the normalized distance is lowest.

6. When the instructions are executed by the processor, the electronic device: Identifying objects of a specified type in the cropped first image; establishing the first region of interest within the object identified in the cropped first image; setting a position of the object in the cropped first image in the cropped second image; 5. The electronic device of claim 4, further comprising: causing the second region of interest in the cropped second image to be located at the same position as the first region of interest in the cropped first image.

7. When the instructions are executed by the processor, the electronic device: identifying a string of characters in the first region of interest based on a string identification algorithm; The electronic device of claim 6 , further comprising: causing the identified character string to be labeled in the first region of interest and the second region of interest.

8. When the instructions are executed by the processor, the electronic device: Transforming the first region of interest into a specified shape based on a specified transformation algorithm; The electronic device of claim 7 , further comprising: a character string identification algorithm for identifying the character string in the first region of interest converted into the specified shape.

9. When the instructions are executed by the processor, the electronic device: identifying whether the electronic device is moving based on differences between successive ones of the first images; deleting the first image and the second image based on the electronic device not being moved; The electronic device of claim 1 , wherein the electronic device is caused to generate the training data based on the first image and the second image based on the electronic device being moved.

10. 1. A method performed in an electronic device including a communications circuit, comprising: acquiring a first image from a first camera via the communication circuit; acquiring a second image from a second camera having a slower shutter speed than the first camera via the communication circuit; identifying the first image and the second image as being acquired at corresponding times; establishing a first region of interest within an object of a specified type in the first image; defining a second region of interest in the second image at the same location as the first region of interest; and A method comprising: generating, as training data, a first image in which the first region of interest is set and a second image in which the second region of interest is set.

11. an operation of instructing the first camera and the second camera to take pictures at corresponding times, and then acquiring a first image and a second image for a specified time; identifying an image acquired at a time corresponding to the reference image based on comparing a reference image of the first images with each of the second images; and 11. The method of claim 10, including the act of identifying the second image from the first image that has the same frame difference as the frame difference from the first image to the reference image.

12. generating a cropped first image by cropping an area of ​​a specified size in the first image; and 11. The method of claim 10, further comprising the act of generating a cropped second image by cropping another area of ​​a specified size in the second image based on the cropped first image.

13. identifying normalized difference values ​​between the cropped first image and a plurality of regions of the second image; identifying normalized distances between feature points of the cropped first image and feature points of each of a plurality of regions of the second image; and 13. The method of claim 12, comprising generating the cropped second image by cropping a region having the lowest weighted average between the normalized difference value and the normalized distance.

14. identifying an object of a specified type in the cropped first image; establishing the first region of interest within an object identified in the cropped first image; setting the position of the object of the cropped first image in the cropped second image; and 13. The method of claim 12, further comprising the act of placing the second region of interest in the cropped second image at the same location as the first region of interest in the cropped first image.

15. identifying whether the electronic device is moving based on differences between successive ones of the first images; deleting the first image and the second image based on the electronic device not being moved; and The method of claim 10 , comprising generating the training data based on the first image and the second image based on movement of the electronic device.

16. A non-transitory computer readable storage medium, comprising: storing a program containing instructions; The instructions, when executed by a processor of an electronic device including a communications circuit, cause the electronic device to: acquiring a first image from a first camera via the communication circuit; acquiring a second image from a second camera having a slower shutter speed than the first camera via the communication circuit; identifying the first image and the second image, the first image and the second image being acquired at corresponding times; establishing a first region of interest within a specified type of object in the first image; setting a second region of interest in the second image at the same position as the first region of interest; A non-transitory computer-readable storage medium that causes a first image in which the first region of interest is set and a second image in which the second region of interest is set to be generated as training data.

17. When the instructions are executed by the processor, the electronic device: generating a cropped first image by cropping an area of ​​a specified size in the first image; 17. The non-transitory computer-readable storage medium of claim 16, wherein the non-transitory computer-readable storage medium causes a cropped second image to be generated by cropping another area of ​​a specified size in the second image based on the cropped first image.

18. When the instructions are executed by the processor, the electronic device: identifying normalized difference values ​​between the cropped first image and a plurality of regions of the second image; identifying normalized distances between feature points of the cropped first image and feature points of each of a plurality of regions of the second image; 20. The non-transitory computer-readable storage medium of claim 17, causing the cropped second image to be generated by cropping a region where a weighted average between the normalized difference value and the normalized distance is lowest.

19. When the instructions are executed by the processor, the electronic device: Identifying objects of a specified type in the cropped first image; establishing the first region of interest within the object identified in the cropped first image; setting a position of the object in the cropped first image in the cropped second image; 20. The non-transitory computer-readable storage medium of claim 17, causing the second region of interest in the cropped second image to be set in the same position as the first region of interest in the cropped first image.

20. When the instructions are executed by the processor, the electronic device: identifying whether the electronic device is moving based on differences between successive ones of the first images; deleting the first image and the second image based on the electronic device not being moved; The non-transitory computer-readable storage medium of claim 16 , wherein the electronic device is caused to generate the training data based on the first image and the second image based on the electronic device being moved.

Images