Electronic device for collecting training data, operation method thereof, and storage medium

Abstract

The present invention provides an electronic device for collecting training data, an operation method thereof, and a storage medium, the electronic device according to an embodiment of the present invention being provided in a vehicle, the electronic device including: a communication circuit; a camera; a memory storing instructions; and a processor that, when the processor executes the instructions, causes the electronic device to be configured to: execute a first model to detect one or more objects in a first image acquired from the camera, and execute a second model using feature information acquired from the first model, and obtaining a second image based on the feature information.

Description

Technical Field

[0001] The present disclosure relates to an electronic device, a non-transitory computer-readable storage medium, and a method for collecting training data. Background Art

[0002] Neural networks such as deep neural networks (DNNs) can be used to provide autonomous driving information for vehicles. The performance of neural networks as described above is limited by the amount and quality of data used to train the neural networks.

[0003] The above information is provided as background technology to assist in understanding the present disclosure, and no claim or determination is made that any of the above content can be used as prior art related to the present disclosure. Summary of the Invention

[0004] In one embodiment, an electronic device is provided, which is arranged in a vehicle, and includes: a communication circuit; a camera; a memory for storing instructions; and a processor. When the processor runs the instructions, the electronic device is configured to: run a first model to detect one or more subjects in a first image obtained from the camera, and run a second model using feature information obtained from the first model to obtain a second image based on the feature information.

[0005] In one embodiment, a non-transitory computer-readable storage medium is provided, storing one or more programs. When a processor of an electronic device including a communication circuit and a camera runs the one or more programs, the electronic device is configured to: run a first model to detect one or more subjects in a first image obtained from the camera; and run a second model using feature information obtained from the first model to obtain a second image based on the feature information.

[0006] In one embodiment, a method for operating an electronic device is provided, wherein the electronic device includes a communication circuit and a camera, and the method for operating the electronic device includes: running a first model to detect the movement of one or more subjects in a first image obtained from the camera; and running a second model using feature information obtained from the first model to obtain the movement of a second image based on the feature information. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1A and Figure 1B An example of a conventional truck is shown.

[0008] Figure 1C is an exemplary block diagram of an in-vehicle electronic device according to an embodiment.

[0009] Figure 2 is an exemplary block diagram for illustrating a method for collecting training data for an autonomous driving system of a vehicle according to an embodiment.

[0010] Figure 3 is a flowchart illustrating operations of an electronic device according to an embodiment.

[0011] Figure 4 An example of a block diagram illustrating an autonomous driving system for a vehicle according to an embodiment is shown.

[0012] Figure 5 and Figure 6 An example of a block diagram illustrating an autonomously driven mobile object according to one embodiment is shown.

[0013] Figure 7 An example of a gateway associated with a user device is shown according to various embodiments.

[0014] Figure 8 is a diagram for explaining the operation of an electronic device for training a neural network based on a learning data set according to one embodiment.

[0015] Figure 9 is a block diagram of an electronic device according to an embodiment. DETAILED DESCRIPTION

[0016] The specific structural or functional descriptions of the embodiments of the concepts of the present invention disclosed in this specification are merely illustrative for the purpose of illustrating the embodiments of the concepts of the present invention. The embodiments of the concepts of the present invention can be implemented in various forms and are not limited to the embodiments described in this specification.

[0017] The embodiments based on the concepts of the present invention may be modified in various ways and may have various forms. Therefore, embodiments are shown in the drawings and described in detail in this specification. However, this does not limit the embodiments of the concepts of the present invention to the specific disclosed forms, but rather includes all modifications, equivalents, or alternatives within the spirit and technical scope of the present invention.

[0018] Although terms such as "first" or "second" can be used to describe various components, the components are not limited to these terms. These terms are only used to distinguish one component from another. For example, the first component can be named the second component, and similarly, the second component can be named the first component without departing from the scope of the present invention.

[0019] When a component is “connected” or “engaged” to another component, it should be understood that it can be directly connected or engaged to the other component, and other components may be present between the two components. Conversely, when a component is “directly connected” or “directly engaged” to another component, it should be understood that no other components are present between the two components. Expressions used to describe the relationship between components, such as “between,” “directly between,” or “directly adjacent to,” should also be interpreted similarly.

[0020] The terms used in this specification are only used to illustrate specific embodiments and are not intended to limit the present invention. Unless the context clearly indicates otherwise, expressions in the singular include expressions in the plural. In this specification, it should be understood that terms such as "including" or "having" are intended to specify the presence of the features, numbers, steps, actions, constituent elements, parts, or combinations thereof, but do not exclude in advance the possibility of the presence or addition of one or more other features or numbers, steps, actions, constituent elements, parts, or combinations thereof.

[0021] Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meanings as those commonly understood by those skilled in the art. Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology and should not be interpreted as having ideal or excessive meanings unless explicitly defined in this specification.

[0022] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. The same reference numerals mentioned in the drawings may represent the same structure, and repeated descriptions thereof will be omitted.

[0023] Figure 1A and Figure 1B An example of a conventional truck is shown. Over the years, the trucking industry has steadily grown and expanded its services to accommodate more complex supply chains. These services include last-mile deliveries, drop-trailer programs, and intermodal transportation at ports (where goods are transported by two or more different modes of transportation, such as ship and rail, or ship and aircraft, to their destination).

[0024] As described above, since there are many different methods for transporting goods, manufacturers of goods transport related equipment have designed equipment of different forms to transport goods according to various transportation needs.

[0025] In this specification, a truck that tows a trailer primarily for the purpose of freight transportation (carrying or catering) is generally referred to as a tractor.

[0026] The tractors described in this specification can be divided into conventional trucks (or bonneted trucks), cab-over trucks (or cab-over engines), and semi-conventional trucks, which are intermediate between conventional trucks and cab-over trucks, based on the position and shape of their cabs.

[0027] Conventional trucks have an engine and hood located above the front axle in front of the tractor's cab, with the driver sitting behind the front axle. These tractors are primarily used in North America, with the engine located in front of the driver.

[0028] In contrast, a cab-over truck has the tractor's cab positioned at the front end of the tractor, with the driver sitting in front of the front axle. This flat-front truck is also known as a "flat face" (or flatnose) truck, and is primarily used in most countries in Europe and Asia. The engine is located below the driver.

[0029] Just as tractors come in a variety of configurations depending on their purpose and needs, trailers towed by tractors also come in a variety of forms. The most representative types of trailers are full-trailers and semi-trailers. Full-trailers and semi-trailers can be distinguished by whether they have front and rear axles. These trailers can be connected to box trucks or tractors using a coupling device.

[0030] Specifically, a full-trailer is a commercial freight trailer equipped with a front axle and a rear axle. Designed so that its total load is fully supported by the trailer, it can fully support its own weight without relying on a towing vehicle, and can be equipped with a truck drawbar for coupling with a hauling unit or towing unit such as a tractor. Full-trailers are primarily used in the United States, Canada, and other regions.

[0031] On the other hand, a semi-trailer is a cargo trailer equipped with only a rear axle and no front axle, and most of its weight is supported by a tractor connected via a hitch called a "fifth wheel." When detached from the tractor and stationary, the semi-trailer is able to support the weight of the trailer by vertically deploying the landing gear installed on the bottom of the semi-trailer. The combination of a semitrailer and a tractor is called a semi-trailer truck (also referred to in the United States as a "semi-trailer," "tractor-trailer," "semi-truck," "big rig," or "semi"). The "fifth wheel" mentioned above refers to a horizontal wheel attached to the tractor axle of a trailer truck to facilitate directional changes of the trailer, also known as the fifth wheel. The "fifth wheel," as a device that enables the tractor and semitrailer to be flexibly coupled, generally consists of a trunnion plate and a latching device that securely secures the kingpin mounted on the semitrailer to the tractor.

[0032] For ease of explanation, this specification will use the term "trailer" to refer to a cargo transport vehicle connected to a tractor for a trailer, and "tractor" to refer to a towing vehicle used to move the trailer, based on the aforementioned tractor / trailer terminology. Furthermore, in order to minimize potential limitations on the rights of the embodiments described in the detailed description, the term "tractor" may be used interchangeably with the term "tracting vehicle" to describe a tractor hauling / towing a "trailer," and the term "towed vehicle" may be used interchangeably with the term "trailer towed by a tractor."

[0033] Furthermore, for the sake of convenience, the “trailer” described throughout this specification is preferably understood to refer to a “semi-trailer”, but is not limited thereto.

[0034] Reference Figure 1A and Figure 1B , the vehicle 1015 may include a tractor or a tractor unit 1051 and a semi-trailer 1052 . Figure 1A Indicates the state when the tractor 1051 and the semi-trailer 1052 are not connected. Figure 1B It shows the state when the tractor 1051 is connected to the semi-trailer 1052.

[0035] In one embodiment, the semi-trailer 1052 can be selectively connected using a steering wheel hitch 1056 carried by the tractor 1051, and the steering wheel hitch 1056 can be fastened to a tow pin 1058 fixed to the semi-trailer 1052 in a known manner. The vehicle 1015 including the tractor 1051 and the semi-trailer 1052 can be referred to as a truck. The vehicle 1015 can include only the tractor 1051. Figure 1A and Figure 1B The semi-trailer 1052 shown in the figure shows a “semi-trailer” form, but this is only for the convenience of explanation, and it should not be understood that the embodiments of the present disclosure are only applicable to the “semi-trailer” form. Figure 1A and Figure 1B The tractor 1051 shown in the figure shows a "flat-head truck" form, but this is only for the convenience of explanation and it should not be understood that the embodiments of the present disclosure are only applicable to the "flat-head truck" form.

[0036] In one embodiment, the semi-trailer 1052 may include a towing pin 1058 coupled to a steering wheel hitch 1056 of the tractor 1051, and a landing gear 1059 for supporting the semi-trailer 1052 on the ground when the semi-trailer 1052 is not coupled to the tractor 1051. The towing pin 1058 and the landing gear 1059 may be disposed (or arranged) at the bottom of the semi-trailer 1052.

[0037] In one embodiment, to facilitate travel on curved roads, the semi-trailer 1052 may be rotatably coupled to the tractor 151. For example, the tractor 1051 and the semi-trailer 1052 may be rotatably coupled via a coupling device including a steering wheel hitch 1056 and a towing pin 1058. However, the link mechanism between the tractor 1051 and the semi-trailer 1052 is not limited thereto.

[0038] Figure 1C FIG is an exemplary block diagram of an electronic device in a vehicle according to an embodiment. Figure 1C According to an embodiment, the electronic device 100 may include: a processor 110, a memory 150, a communication circuit 120, and a camera 130. In an embodiment, the processor 110, the memory 150, the communication circuit 120, and / or the camera 130 may be electrically connected and / or operatively connected to each other (electronically and / or operably coupled with each other) through an electronic component such as a communication bus. Hereinafter, the operative combination between hardware may refer to a direct connection or an indirect connection between hardware in a wired or wireless manner to control the second hardware through the first hardware in the hardware.

[0039] exist Figure 1C In FIG. 1 , the processor 110 , the memory 150 , the camera 130 and / or the communication circuit 120 are shown as different blocks, but are not limited thereto. Figure 1C Some of the hardware shown may also be implemented as a single integrated circuit (IC) or a portion of a single package, such as a system on a chip (SoC).

[0040] According to one embodiment, the memory 150 may store instructions. The processor 110 may be configured to process data based on the instructions stored in the memory 150. For example, the processor 110 may include an arithmetic and logic unit (ALU), a floating point processing unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor chip (AP). The processor 110 may have a single-core processor 110 structure, or a multi-core processor structure such as a dual-core, quad-core, hexa-core, or octa-core processor.

[0041] According to one embodiment, the memory 150 may include a hardware component for storing data and / or instructions that can be executed by the processor 110. For example, the memory 150 may include a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). For example, the volatile memory may include at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM (Cache RAM), and pseudo-static RAM (PSRAM). For example, the non-volatile memory may include at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, an optical disk, a solid state drive (SSD), and an embedded multi-media card (eMMC).

[0042] In one embodiment, the memory 150 of the electronic device 100 may include a neural network model. For example, the electronic device 100 may include a first model 151 , a second model 152 , a third model 153 , and a fourth model 154 stored in the memory 150 .

[0043] In one embodiment, the first model 151 may include a neural network model (e.g., an object detection model) for detecting and segmenting objects in an image. For example, the first model 151 may include a neural network such as a convolutional neural network (CNN). For example, the first model 151 may include an input layer, an intermediate layer (or hidden layer), and an output layer. The intermediate layer of the first model 151 may include a convolutional layer for extracting features from input data (e.g., an image input to the first model 151 or a feature map output by a convolutional layer) by applying a kernel (or fielder).

[0044] In one embodiment, the fourth model 154 may include a model for determining whether to use the analysis results (or detection results) of an image obtained by the first model 151. For example, the electronic device 100 may determine whether to send the detection results of an image obtained by the first model 151 to the server 160 by running the fourth model 154. For example, the electronic device 100 may determine whether to run the second model 152 based on the detection results of the image obtained by the first model 151 by running the fourth model 154. For example, when the detection results of an image do not meet pre-specified settings and / or the reliability of the detection results of the image is above a threshold, the electronic device 100 may use the fourth model 154 to send the detection results of the corresponding image to the server 160 and / or run the second model 152 based on the detection results of the corresponding image. For example, using the fourth model 154, the electronic device 100 may compare the training data used for training the first model 151 with information (e.g., analysis results) about the image input to the first model 151 and determine whether to expand the training data used for training the first model 151 using the image.

[0045] In one embodiment, the second model 152 and the third model 153 may constitute a generative adversarial network (GAN) (e.g., deep convolutional GAN ​​(DCGAN)). For example, the second model 152 may include a GAN generator, and the third model 153 may include a GAN discriminator. The second model 152 may be used to generate artificial data based on input data. The third model 153 may be used to distinguish whether the artificial data generated by the second model 152 is actual data based on similarity or probability. The second model 152 and the third model 153 may be configured to perform adversarial learning. Alternatively, the third model 153 may not be used to generate and collect data for machine learning of the autonomous driving system. When the third model 153 is used to generate and collect data for this purpose, it may be determined whether the data generated in the second model 152 is data that needs to be collected for machine learning of the autonomous driving system. The second model 152 and / or the third model 153 may include at least one of a GAN-based model such as CycleGan or StyleGAN, a convolutional neural network, a residual network (ResNet), a Transformer and / or a U-Net, but is not limited thereto.

[0046] In one embodiment, the actions of the electronic device 100 caused by the execution of the first model 151, the second model 152, the third model 153, and the fourth model 154 by the processor 110 will refer to Figure 2 This will be described later.

[0047] According to one embodiment, the communication circuit 120 can be used to communicate with an external electronic device in a wired and / or wireless manner. For example, the electronic device 100 can be configured to communicate with the server 160 in a wired and / or wireless manner using the communication circuit 120. The communication circuit 120 may include, for example, at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit 120 can support the transmission and / or reception of electrical signals based on various types of protocols such as Ethernet, local area network (LAN), wide area network (WAN), wireless fidelity (Wi-Fi), near field communication (NFC), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G (fifth generation), new radio (NR), 6G (sixth generation), and / or above-6G.

[0048] According to one embodiment, the camera 130 may include a lens assembly or an image sensor. The lens assembly may collect light emitted from a subject serving as an image capture object. The lens assembly may include one or more lenses. For example, the camera 130 may include multiple lens assemblies. For example, some of the multiple lens assemblies of the camera 130 may have the same lens properties (e.g., viewing angle, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that differ from those of the other lens assemblies. The lens assembly may include a wide-angle lens or a telephoto lens. For example, the electronic device 100 may include a flash device for the camera 130. The flash device may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LEDs, white LEDs, infrared LEDs, or ultraviolet LEDs) or a xenon lamp. For example, the image sensor may capture an image corresponding to the subject by converting light emitted from or reflected from the subject and transmitted through the lens assembly into electrical signals. According to one embodiment, the image sensor may include, for example, one image sensor selected from image sensors having different properties, such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor may be implemented using, for example, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

[0049] According to one embodiment, the electronic device 100 can use the camera 130 to obtain images of the surrounding environment (for example: Figure 2 For example, the electronic device 100 may acquire an image of the surrounding environment of a vehicle traveling on a road.

[0050] In one embodiment, the electronic device 100 may obtain or collect information for an autonomous driving system (e.g., Figure 4The server 160 may manage the data used in the training of the autonomous driving system. As machine learning models such as deep neural networks (DNNs) used to support autonomous driving of vehicles become increasingly complex, more and higher-quality training data or data sets are needed. When only images acquired from the camera 130 of the vehicle are used as training data, the amount of training data may be limited, and thus the performance of autonomous driving may also be limited. The electronic device 100 according to one embodiment may achieve a neural network with improved performance by collecting images generated based on images acquired from the camera 130 as training data.

[0051] Figure 2 is an exemplary block diagram for illustrating a method for collecting training data for an autonomous driving system of a vehicle according to an embodiment. Figure 2 The action described can be Figure 1C The electronic device 100 or the processor 110 of the electronic device 100 executes.

[0052] Reference Figure 2 , the electronic device 100 may acquire a first image 201 using the camera 130. The first image 201 may be an image of an environment in which a vehicle including the electronic device 100 is traveling. The first image 201 may include a road on which the vehicle is traveling and one or more subjects (or objects) such as other vehicles around the vehicle.

[0053] In one embodiment, the electronic device 100 may send the first image 201 from the camera 130 to the first model 151. In order to detect one or more subjects included in the first image 201, the electronic device 100 may run the first model 151. The electronic device 100 may obtain a detection result of the subject included in the first image 201 using the first model 151. For example, the electronic device 100 may obtain a detection result of one or more subjects included in the first image 201 from the first model 151 to which the first image 201 is input. The detection result may include, for example, information about a bounding box specified by coordinates on the first image 201, a category (or classification) of the subject associated with the bounding box, and a probability that the subject corresponding to the bounding box matches the category, but is not limited thereto.

[0054] In one embodiment, the electronic device 100 may send the first image 201 and the detection results of the first image 201 acquired using the first model 151 to the fourth model 154. To determine whether to collect the first image 201 and the detection results of the first image 201, the electronic device 100 may run the fourth model 154. Based on the determination result of the fourth model 154, the electronic device 100 may selectively send information associated with the first image 201 and the detection results of one or more objects within the first image 201 to the server 160.

[0055] In one embodiment, the electronic device 100 may obtain feature information (or intrinsic information) associated with the first image 201 from the first model 151 that has the first image 201 as input. For example, when it is determined that the first image 201 and the detection result of the first image 201 need to be collected (or sent to the server 160), the feature information associated with the first image 201 may be obtained. The electronic device 100 may send the feature information from the first model 151 to the second model 152. The feature information may be the output of an intermediate layer of the first model 151. For example, the feature information may be the output of a convolutional layer of the first model 151.

[0056] In one embodiment, the electronic device 100 may run the second model 152 using the feature information obtained from the first model 151. By running the second model 152 using the feature information obtained from the first model 151, the electronic device 100 may obtain the second image 202 based on the feature information. For example, the second model 152 may generate the second image 202 using the feature information associated with the first image 201. The second image 202 may be an image in which at least some of the features of the first image 201, such as weather (e.g., sunny, cloudy, rainy, or snowy), time period (e.g., dawn, daytime, or day and night), color of the subject, or pattern of the subject, are changed to new features. Additionally or alternatively, the first image 201 may be used together with the feature information to obtain the second image 202.

[0057] In one embodiment, the electronic device 100 may transmit the second image 153 from the second model 152 to the third model 153. The electronic device 100 may run the third model 153 using the second image 202. By running the third model 153 using the second image 202, the electronic device 100 may determine whether to collect the second image 202. For example, the electronic device 100 may determine whether to transmit the second image 202 to the server 160 using the third model 153 run based on the second image 202. For example, the electronic device 100 may determine whether to transmit the second image 202 to the server 160 by confirming whether a parameter associated with the second image 202 exceeds a preset threshold. For example, if the parameter associated with the second image 202 exceeds the threshold, the electronic device 100 may transmit the second image 202 to the server 160 via the communication circuit 120; otherwise, the second image 202 is not transmitted to the server 160. In this manner, data required for training a DNN for autonomous driving can be selectively collected.

[0058] In one embodiment, the electronic device 100 may generate multiple images using the feature information corresponding to the second image 202 determined to need to be collected, and may send the generated multiple images to the server 160. In this way, multiple high-quality training data may be collected.

[0059] In one embodiment, runtime delimiter reinforcement learning can be used as logic for collecting a large amount of training data that matches situations frequently encountered by the vehicle. For example, if the second image 202 is not collected, the third model 153 can be trained while the vehicle is driving using the recognition results stored in the buffer of the memory 150. For example, when the first model 151 recognizes a truck in the image, a label representing the recognition result can be generated. The generated label can be used to train the third model 153.

[0060] Figure 3 is a flow chart illustrating operations of an electronic device according to an embodiment. Figure 3 The action can be Figure 1C The electronic device 100 or the processor 110 of the electronic device 100 executes. Figure 3 Each action can be performed in sequence, but it is not necessary to perform it in sequence. For example, the order of each action can be changed, and at least two actions can also be performed in parallel.

[0061] Reference Figure 3In action 310, the electronic device 100 may execute the first model 151 to detect one or more objects from the first image 201 acquired by the camera 130. The electronic device 100 may obtain information on the detection results of the one or more objects in the first image 201 as an output of an output layer (e.g., a fully-connected layer) of the first model 151.

[0062] In action 320, the electronic device 100 may obtain feature information associated with the first image 201 using the first model 151. For example, the electronic device 100 may obtain the feature information associated with the first image 201 as an output of an intermediate layer of the first model 151.

[0063] In action 330, the electronic device 100 executes the second model 152 using the feature information acquired from the first model 151, thereby acquiring the second image 202. As described above, the second image 202 may be an image newly generated based on the first image 201. For example, the second image 202 may include features that are different from at least a portion of the features of the first image 201.

[0064] In action 340, the electronic device 100 may obtain information indicating whether the second image 202 is similar to an actual image by running the third model 153. For example, the electronic device 100 may obtain information indicating the similarity between the second image 202 generated by the second model 152 and an actual image (e.g., the first image 201), or the probability that the second image 202 is the actual image. However, action 340 may be omitted. In this case, the electronic device 100 may perform action 350 after performing action 330.

[0065] In act 350 , the electronic device 100 may transmit the second image 202 to the server 160 . For example, the electronic device 100 may transmit the second image 202 to the server 160 using the communication circuit 120 .

[0066] For example, when action 340 is omitted, the electronic device 100 may transmit the acquired second image 202 to the server 160 in action 330 .

[0067] For example, when executing action 340, the second image 202 may be sent to the server 160 based on the information obtained by running the third model 153. For example, when the similarity between the second image 202 and the actual image or the probability that the second image 202 is the actual image exceeds a critical value, the electronic device 100 may send the second image 202 to the server 160. Otherwise, the second image 202 may not be sent to the server 160. The second image 202 that is not sent may be used for training the first model 151, the second model 152, and / or the third model 153.

[0068] In action 360, the electronic device 100 may transmit the detection results of the first image 201 to the server. For example, the electronic device 100 may transmit information about the detection results of one or more objects in the first image 201 obtained using the first model 151 to the server 160 via the communication circuit 120. The detection results may include, for example, the location of the object included in the first image 201, the category of the object, and information about the probability that the object and the category are consistent.

[0069] Alternatively or additionally, the electronic device 100 may perform action 370 after performing action 310. In action 370, the electronic device 100 may determine whether to send the detection result of the first image 201 to the server 160 by executing the fourth model 154. The electronic device 100 may perform actions 320 and 360 based on the judgment result of action 370. For example, when it is determined that the detection result of the first image 201 is to be sent to the server 160, the electronic device 100 may send the detection result of the first image 201 to the server 160 in action 360. In addition, when it is determined that the detection result of the first image 201 is to be sent to the server 160, the electronic device 100 may obtain the feature information associated with the first image 201 as the output of the intermediate layer of the first model 151 in action 320.

[0070] Figure 4 An example of a block diagram illustrating an autonomous driving system for a vehicle according to an embodiment.

[0071] according to Figure 4The autonomous driving system 400 of a vehicle can be a deep learning network including a sensor 403, an image preprocessor 405, a deep learning network 407, an artificial intelligence (AI) processor 409, a vehicle control module 411, a network interface 413 and a communication unit 415. In various embodiments, the various elements can be connected through various interfaces. For example, the sensor data sensed and output by the sensor 403 can be pushed (feed) to the image preprocessor 405. The sensor data processed by the image preprocessor 405 can be pushed to the deep learning network 407 running (run) in the AI ​​processor 409. The output of the deep learning network 407 run (run) by the AI ​​processor 409 can be pushed to the vehicle control module 411. The intermediate results of the deep learning network 407 run (run) in the AI ​​processor 409 can be pushed to the AI ​​processor 409. In various embodiments, the network interface 413 communicates with the electronic devices in the vehicle (for example: Figure 1C The autonomous driving control system 400 may communicate with the electronic device 100 of the vehicle, and may pass autonomous driving path information and / or autonomous driving control instructions for autonomous driving of the vehicle to the internal block components. In one embodiment, the network interface 413 may be used to send sensor data acquired by the sensor 403 to an external server. In some embodiments, the autonomous driving control system 400 may include additional or fewer components as appropriate. For example, in some embodiments, the image preprocessor 405 may be an optional component. In another example, in order to perform post-processing on the output of the deep learning network 407 before providing the output to the vehicle control module 411, a post-processing component (not shown) may be included in the autonomous driving control system 400.

[0072] In some embodiments, sensor 403 may include more than one sensor. In various embodiments, sensor 403 may be attached to different locations on the vehicle. Sensor 403 may face one or more different directions. For example, sensor 403 may be attached to the front, sides, rear, and / or roof of the vehicle, facing forward, rear, or sideways. In some embodiments, sensor 403 may be an image sensor such as a high dynamic range camera. In some embodiments, sensor 403 includes non-visual sensors. In some embodiments, sensor 403 includes radar, lidar, and / or ultrasonic sensors in addition to image sensors. In some embodiments, sensor 403 is not mounted on the vehicle with vehicle control module 411. For example, sensor 403 may be included as part of a deep learning system for capturing sensor data and may be attached to the environment or road and / or mounted on surrounding vehicles.

[0073] In some embodiments, the image pre-processor 405 can be used to pre-process the sensor data of the sensor 403. For example, the image pre-processor 405 can be used to pre-process the sensor data, split the sensor data into one or more components, and / or post-process one or more components. In some embodiments, the image pre-processor 405 can be a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a specialized image processor. In various embodiments, the image pre-processor 405 can be a tone mapping processor for processing high dynamic range data. In some embodiments, the image pre-processor 405 can be a component of the AI ​​processor 409.

[0074] In some embodiments, the deep learning network 407 may be a deep learning network for implementing control commands for controlling the autonomous vehicle. For example, the deep learning network 407 may be an artificial neural network such as a convolutional neural network (CNN) trained using sensor data, and the output of the deep learning network 407 is provided to the vehicle control module 411.

[0075] In some embodiments, artificial intelligence (AI) processor 409 may be a hardware processor for running deep learning network 407. In some embodiments, AI processor 409 is a dedicated AI processor for performing inference on sensor data through a convolutional neural network (CNN). In some embodiments, AI processor 409 may be optimized for the bit depth of sensor data. In some embodiments, AI processor 409 may be optimized for deep learning computations such as neural network computations including convolution, inner product, vector, and / or matrix operations. In some embodiments, AI processor 409 may be implemented using multiple graphics processing units (GPUs) capable of efficiently performing parallel processing.

[0076] In various embodiments, the AI ​​processor 409 can be coupled to a memory via an input / output interface, the memory being configured to provide the AI ​​processor with instructions for initiating the following actions, such that during operation of the AI ​​processor 409, the AI ​​processor 409 performs deep learning analysis on sensor data received from the sensor 403 and determines machine learning results for use in causing the vehicle to operate at least partially autonomously. In some embodiments, the vehicle control module 411 processes vehicle control instructions output from the artificial intelligence (AI) processor 409 and can be used to translate the output of the AI ​​processor 409 into instructions for controlling various modules of the vehicle. In some embodiments, the vehicle control module 411 is used to control the vehicle for autonomous driving. In some embodiments, the vehicle control module 411 can adjust the steering and / or speed of the vehicle. For example, the vehicle control module 411 can be used to control vehicle movement, such as deceleration, acceleration, steering, lane changes, and lane keeping. In some embodiments, the vehicle control module 411 may generate control signals for controlling vehicle lighting, such as brake lights, turn signals, headlights, etc. In some embodiments, the vehicle control module 411 may be used to control vehicle audio-related systems, such as the vehicle's sound system, vehicle's audio warnings, vehicle's microphone system, vehicle's horn system, etc.

[0077] In some embodiments, the vehicle control module 411 can be used to control notification systems, including warning systems, that notify passengers and / or the driver of driving events, such as approaching an intended destination or a potential collision. In some embodiments, the vehicle control module 411 can be used to adjust sensors, such as the vehicle's sensors 403. For example, the vehicle control module 411 can perform the following actions: modifying the orientation of the sensors 403, changing the output resolution and / or format type of the sensors 403, increasing or decreasing the capture rate, adjusting the dynamic range, or adjusting the focus of the camera. In addition, the vehicle control module 411 can turn sensors on and off individually or collectively.

[0078] In some embodiments, the vehicle control module 411 can be used to change parameters of the image pre-processor 405 by, for example, modifying the frequency range of a filter, adjusting edge detection parameters for feature and / or object detection, adjusting channels and bit depth, etc. In various embodiments, the vehicle control module 411 can be used to control autonomous driving and / or driver assistance functions of the vehicle.

[0079] In some embodiments, the network interface 413 may serve as the internal interface between the components of the autonomous driving control system 400 and the communication unit 415. Specifically, the network interface 413 may be an interactive interface for receiving and / or transmitting data, including voice data. In various embodiments, the network interface 413 may connect to an external server via the communication unit 415 to facilitate voice calls, receive and / or transmit text messages, transmit sensor data, and update the vehicle's software via the autonomous driving system or the vehicle's autonomous driving system software.

[0080] In various embodiments, the communication unit 415 may include various wireless interfaces such as cellular or Wi-Fi. For example, the network interface 413 may be used to receive updates for operating parameters and / or instructions for the sensor 403, image preprocessor 405, deep learning network 407, AI processor 409, and vehicle control module 411 from an external server accessed through the communication unit 415. For example, the machine learning model of the deep learning network 407 may be updated using the communication unit 415. According to another example, the communication unit 415 may be used to update operating parameters of the image preprocessor 405, such as image processing parameters, and / or the firmware of the sensor 403.

[0081] In another embodiment, the communication unit 415 can be used to activate communications for emergency services and emergency contacts in the event of an accident or near-accident. For example, in the event of a collision, the communication unit 415 can be used to call emergency services for assistance and to notify emergency services of the details of the collision and the location of the vehicle. In various embodiments, the communication unit 415 can update or obtain an estimated time of arrival and / or destination location.

[0082] According to one embodiment, Figure 4 The autonomous driving system 400 shown in FIG4 may also be formed by the electronic device 100 of the vehicle. According to one embodiment, when an autonomous driving release event occurs from the user during autonomous driving of the vehicle, the AI ​​processor 409 of the autonomous driving system 400 controls the input of information related to the autonomous driving release event into the training set data of the deep learning network to control the autonomous driving software of the vehicle to learn.

[0083] Figure 5 and Figure 6 An example of a block diagram illustrating an autonomously driven mobile object according to one embodiment is shown. Figure 7 An example of a gateway associated with a user device is shown according to various embodiments.

[0084] Reference Figure 5 According to this embodiment, the autonomous driving mobile object 500 may include: a control device 600, sensor modules 504a, 504b, 504c, 504d, an engine 506 and a user interface 508.

[0085] The autonomous driving vehicle 500 can have an autonomous driving mode or a manual mode. As an example, based on user input received through the user interface 508, the vehicle can switch from the manual mode to the autonomous driving mode, or vice versa.

[0086] When the mobile body 500 operates in the autonomous driving mode, the autonomous driving mobile body 500 may operate under the control of the control device 600 .

[0087] In this embodiment, the control device 600 may include: a controller 620 including a memory 622 and a processor 624 , a sensor 610 , a communication device 630 , and an object detection device 640 .

[0088] The object detection device 640 may perform all or part of the functions of the distance measurement device.

[0089] That is, in this embodiment, the object detection device 640 serves as a device for detecting an object located outside the moving body 500 . The object detection device 640 can detect the object located outside the moving body 500 and generate object information corresponding to the detection result.

[0090] The object information may include information on the presence or absence of the object, position information of the object, distance information between the moving object and the object, and relative speed information between the moving object and the object.

[0091] Objects may include various objects located outside of the mobile object 500, such as lane markings, other vehicles, pedestrians, traffic signals, lights, roads, structures, speed bumps, terrain features, and animals. Traffic signals may include traffic lights, traffic signs, and patterns or text painted on the road surface. Furthermore, light may be light generated by lights on other vehicles, light generated by streetlights, or sunlight.

[0092] Furthermore, structures can be objects located around the road and fixed to the ground. For example, structures can include streetlights, roadside trees, buildings, utility poles, traffic lights, and bridges. Terrain can include mountains and hills.

[0093] The object detection device 640 may include a camera module. The controller 620 may extract object information from the external image captured by the camera module and process the information related thereto.

[0094] In addition, the object detection device 640 may also include an imaging device for identifying the external environment. In addition to LIDAR, radar (RADAR), GPS devices, odometry devices, other computer vision devices, ultrasonic sensors, and infrared sensors may also be used. These devices can be selected or operated simultaneously as needed to achieve more accurate sensing.

[0095] In addition, the distance measurement device according to an embodiment of the present invention calculates the distance between the autonomous driving mobile object 500 and an object, and can control the motion of the mobile object based on the calculated distance by communicating with the control device 600 of the autonomous driving mobile object 500.

[0096] For example, if the distance between autonomous vehicle 500 and an object indicates a potential collision, autonomous vehicle 500 may apply brakes to reduce speed or stop. As another example, if the object is moving, autonomous vehicle 500 may control its speed to maintain a predetermined distance from the object.

[0097] Such a distance measuring device according to an embodiment of the present invention may be configured as a module within the control device 600 of the autonomous driving vehicle 500. That is, the memory 622 and processor 624 of the control device 600 may implement the collision avoidance method according to the present invention in software.

[0098] In addition, the sensor 610 can be connected to the detection modules 504a, 504b, 504c, and 504d to detect the internal / external environment of the mobile object to obtain various detection information. The sensor 610 may include a posture sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor), a collision sensor, a wheel sensor, a speed sensor, a tilt sensor, a weight sensing sensor, a heading sensor, a gyro sensor, a positioning module, a mobile object forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor based on steering wheel rotation, a mobile object internal temperature sensor, a mobile object internal humidity sensor, an ultrasonic sensor, an illumination sensor, an accelerator pedal position sensor, a brake pedal position sensor, and the like.

[0099] Thus, the sensor 610 can obtain mobile body posture information, mobile body collision information, mobile body direction information, mobile body position information (GPS information), mobile body angle information, mobile body speed information, mobile body acceleration information, mobile body slope information, mobile body forward / backward information, battery information, fuel information, tire information, mobile body light information, mobile body internal temperature information, mobile body internal humidity information, detection signals for the steering wheel rotation angle, external illumination of the mobile body, pressure applied to the accelerator pedal, pressure applied to the brake pedal, etc.

[0100] In addition, the sensor 610 may also include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake air temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a time-to-digital conversion (TDC) sensor, a crank angle sensor (CAS), etc.

[0101] As described above, the sensor 610 may generate mobile object state information based on the detection data.

[0102] The wireless communication device 630 is configured to enable wireless communication between autonomous mobile objects 500. For example, the autonomous mobile object 500 can communicate with a user's mobile phone or other wireless communication device 630, other mobile objects, a central device (traffic control device), a server, etc. The wireless communication device 630 can transmit and receive wireless signals according to an access wireless protocol. The wireless communication protocol can be Wi-Fi, Bluetooth, Long-Term Evolution (LTE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), or Global Systems for Mobile Communications (GSM), but the communication protocol is not limited to this.

[0103] In addition, in this embodiment, the autonomous driving mobile body 500 can also realize inter-mobile communication through the wireless communication device 630. That is, the wireless communication device 630 can communicate with other mobile bodies on the road through vehicle-to-vehicle (V2V) communication. The autonomous driving mobile body 500 can send and receive information such as driving warnings and traffic information through inter-vehicle communication, and can also request information from other mobile bodies or receive requests. For example, the wireless communication device 630 can perform V2V communication through a dedicated short-range communication (DSRC) device or a cellular-V2V device. In addition, in addition to inter-vehicle communication, communication between vehicles and other objects (for example, electronic devices carried by pedestrians, etc.) (Vehicle to Everything communication, V2X) can also be realized through the wireless communication device 630.

[0104] In addition, the wireless communication device 630 can obtain information generated from various mobile bodies (Mobility) including infrastructure located on the road (traffic lights, CCTV, RSU, eNode B, etc.) or other autonomous driving (Autonomous Driving) / non-autonomous driving (Non-AutonomousDriving) vehicles through a non-terrestrial network (Non-Terrestrial Network) different from the terrestrial network (Terrestrial Network) as information for autonomous driving execution of the autonomous driving mobile body 500.

[0105] For example, the wireless communication device 630 can perform wireless communication with a non-terrestrial network dedicated antenna mounted on the autonomous driving mobile object 500 through a low earth orbit (LEO) satellite system, a medium earth orbit (MEO) satellite system, a geostationary orbit (GEO) satellite system, a high altitude platform (HAP) system, etc. that constitute a non-terrestrial network.

[0106] For example, the wireless communication device 630 can perform wireless communications with various platforms constituting the NTN based on the wireless access specifications of the 5G Generation New Radio Non-Terrestrial Network (5G NR NTN) standard specification currently under discussion in 3GPP, etc., but is not limited to this.

[0107] In this embodiment, the controller 620 can select a platform that can appropriately perform NTN communication by considering various information such as the position of the autonomous driving vehicle 500, the current time, and available power, and control the wireless communication device 630 to perform wireless communication with the selected platform.

[0108] In this embodiment, the controller 620, which controls all operations of the various units within the mobile object 500, can be configured by the mobile object's manufacturer during manufacturing or as an additional component after manufacturing to perform autonomous driving functions. Alternatively, the controller 620 configured during manufacturing can be updated to include components for continuously performing additional functions. This type of controller 620 may also be referred to as an electronic control unit (ECU).

[0109] The controller 620 can collect various data from connected sensors 610, object detection device 640, communication device 630, and the like, and transmit control signals based on the collected data to the sensors 610, engine 506, user interface 508, communication device 630, and object detection device 640, which are other components of the mobile body. Furthermore, although not shown, control signals can also be transmitted to an accelerator, brake system, steering system, or navigation system associated with the travel of the mobile body.

[0110] In this embodiment, the controller 620 can control the engine 506, for example, to sense the speed limit of the road on which the autonomous driving mobile body 500 is traveling and control the engine 506 so that the driving speed does not exceed the speed limit, or to control the engine 506 to accelerate the driving speed of the autonomous driving mobile body 500 within a range that does not exceed the speed limit.

[0111] Furthermore, during the driving of the autonomous vehicle 500, when the autonomous vehicle 500 approaches or deviates from a lane line, the controller 620 determines whether such approach or deviation from the lane line constitutes a normal driving condition or an alternative driving condition, and controls the engine 506 based on the determination result to control the vehicle's driving. Specifically, the autonomous vehicle 500 can detect lane lines formed on both sides of the lane in which the vehicle is traveling. In this case, the controller 620 determines whether the autonomous vehicle 500 is approaching or deviating from the lane line. If the autonomous vehicle 500 is determined to be approaching or deviating from the lane line, the controller 620 can determine whether such driving is a normal driving condition or an alternative driving condition. A normal driving condition may be, for example, a condition in which the vehicle needs to change lanes. An alternative driving condition may be, for example, a condition in which the vehicle does not need to change lanes. If the controller 620 determines that the autonomous vehicle 500 is approaching or deviating from a lane line under conditions in which the vehicle does not need to change lanes, the autonomous vehicle 500 can control the vehicle's driving to prevent the vehicle from deviating from the lane line and to ensure normal driving within the lane.

[0112] When another moving object or obstacle is in front of the moving object, the engine 506 or the braking system can be controlled to decelerate the moving object. In addition to speed, the trajectory, path, and steering angle can also be controlled. Furthermore, the controller 620 can generate necessary control signals to control the moving object based on information about the moving object's lane, driving signals, and other external environments.

[0113] In addition to generating its own control signals, the controller 620 can also communicate with surrounding mobile objects or a central server, and send instructions for controlling peripheral devices through the received information, thereby controlling the travel of the mobile object.

[0114] Furthermore, when the position or viewing angle of the camera module changes, it may be difficult to accurately identify the moving object or lane markings as in this embodiment. The controller 620 may also generate a control signal for executing camera module calibration to prevent this from occurring. Therefore, in this embodiment, the controller 620 generates a calibration control signal to the camera module to maintain the proper mounting position, orientation, and viewing angle of the camera module even if the camera module's mounting position changes due to vibration or impact generated by the movement of the autonomously driven moving object 500. When the pre-stored initial mounting position, orientation, and viewing angle information of the camera module differs from the initial mounting position, orientation, and viewing angle information of the camera module measured during driving of the autonomously driven moving object 500 by a threshold value or more, the controller 620 may generate a control signal to execute camera module calibration.

[0115] In this embodiment, the controller 620 may include a memory 622 and a processor 624. The processor 624 may execute software stored in the memory 622 in response to control signals from the controller 620. Specifically, the controller 620 stores data and instructions for executing the lane detection method according to the present invention in the memory 622, and the processor 624 executes these instructions to implement one or more methods disclosed herein.

[0116] In this case, the data and instructions may be stored in a non-volatile storage medium that can be executed by the processor 624. The memory 622 may store software and data via appropriate internal or external devices. The memory 622 may be composed of random access memory (RAM), read-only memory (ROM), a hard disk, or a memory device connected to a dongle.

[0117] The memory 622 can store at least an operating system (OS), user applications, and executable instructions. The memory 622 can also store application data and arrange data structures.

[0118] Processor 624 may be a microprocessor or a suitable electronic processor, such as a controller, microcontroller, or state machine.

[0119] The processor 624 may be implemented by a combination of computing devices, which may be constituted by a digital signal processor, a microprocessor, or a suitable combination thereof.

[0120] In addition, the autonomous driving mobile object 500 may also include a user interface 508 for user input to the control device 600. The user interface 508 can allow the user to input information through appropriate interactions. For example, it can be implemented through a touch screen, a keypad, operation buttons, etc. The user interface 508 can transmit the input or command to the controller 620, and the controller 620 executes the control action of the mobile object in response to the input or command.

[0121] In addition, the user interface 508 can enable devices external to the autonomous vehicle 500 to communicate with the autonomous vehicle 500 via the wireless communication device 630. For example, the user interface 508 can be configured to be linked to a mobile phone, tablet computer, or other computer device.

[0122] Furthermore, in this embodiment, while the autonomous mobile object 500 is described as including an engine 506, it may also include other types of propulsion systems. For example, the mobile object may be operated by electric energy, hydrogen energy, or a hybrid system combining these. Therefore, the controller 620 includes propulsion mechanisms corresponding to the propulsion systems of the autonomous mobile object 500 and may provide corresponding control signals to the components of each propulsion mechanism.

[0123] Below, refer to Figure 6 The detailed structure of the control device 600 according to the present invention will be described in more detail.

[0124] The control device 600 includes a processor 624. The processor 624 may also be a general-purpose single-chip or multi-chip microprocessor, a dedicated microprocessor, a microcontroller, a programmable gate array, or the like. The processor may also be referred to as a central processing unit (CPU). In addition, in this embodiment, the processor 624 may also be used in combination with multiple processors.

[0125] The control device 600 further includes a memory 622. The memory 622 may also be any electronic component capable of storing electronic information. Similarly, the memory 622 may include a combination of multiple memories 622 in addition to a single memory.

[0126] Data and instructions 622a for executing the distance measurement method of the distance measurement device according to the present invention may also be stored in the memory 622. When the processor 624 executes the instructions 622a, all or part of the instructions 622a and data 622b required for executing the instructions may also be loaded onto the processor 624 and used as the instructions 624a and data 624b executed on the processor 624.

[0127] The control device 600 may further include a transmitter 630a and a receiver 630b or a transceiver 630c that allow for the transmission and reception of signals. One or more antennas 632a, 632b may be electrically connected to the transmitter 630a, the receiver 630b or each transceiver 630c, and additional antennas may also be included.

[0128] The control device 600 may further include a digital signal processor (DSP) 670. The mobile object can quickly process digital signals through the DSP 670.

[0129] The control device 600 may further include a communication interface 680. The communication interface 680 may further include one or more ports and / or communication modules for connecting other devices to the control device 600. The communication interface 680 may enable a user to interact with the control device 600.

[0130] The various components of the control device 600 can be connected together via one or more buses 690 , which may include a power bus, a control signal bus, a status signal bus, a data bus, etc. Under the control of the processor 624 , the various components can communicate information with each other via the bus 690 and perform their intended functions.

[0131] Additionally, in various embodiments, the control device 600 may be associated with a gateway to communicate with a secure cloud. Figure 7 , the control device 600 may be associated with a gateway 705 for providing information acquired from at least one of the components (701 to 704) of the vehicle 700 to a secure cloud 706. For example, the gateway 705 may be included in the control device 600. As another example, the gateway 705 may be configured as an additional device within the vehicle 700 that is distinct from the control device 600. The gateway 705 communicatively connects the software management cloud 709, the secure cloud 706, and the network within the vehicle 700 secured by the in-vehicle security software 710, which have different networks.

[0132] For example, component 701 may be a sensor. For example, the sensor may be used to obtain information about at least one of the state of vehicle 700 or the state of the surrounding area of ​​vehicle 700. For example, component 701 may include sensor 610.

[0133] For example, component 702 may be an electronic control unit (ECU), which may be used for engine control, transmission control, airbag control, and tire pressure management.

[0134] For example, component 703 may be an instrument cluster. For example, the instrument cluster may be a panel located in front of the driver's seat in a dashboard. For example, the instrument cluster is configured to present information required for driving to the driver (or passenger). For example, the instrument cluster may be configured to display at least one of the following visual elements: a visual element indicating engine revolutions per minute (RPM), a visual element indicating the speed of vehicle 700, a visual element indicating the remaining fuel level, a visual element indicating a gear position, or a visual element indicating information obtained through component 701.

[0135] For example, component 704 may be a telematics device. For example, the telematics device may refer to a device that provides various mobile communication information such as location information and safe driving in the vehicle 700 by combining wireless communication technology with global positioning system (GPS) technology. For example, the telematics device may be used to connect the driver, the cloud (e.g., safety cloud 706) and / or the surrounding environment with the vehicle 700. For example, the telematics device may be configured to support high bandwidth and low latency for technologies specified in the 5G NR specification (e.g., 5G NR's V2X technology, 5G NR's NTN (Non-Terrestrial Network) technology). For example, the telematics device may be configured to support autonomous driving of the vehicle 700.

[0136] For example, gateway 705 can be used to connect the in-vehicle network of vehicle 700 with a software management cloud 709 and a security cloud 706, which are external networks. For example, software management cloud 709 can be used to update or manage at least one software required for driving and managing vehicle 700. For example, software management cloud 709 can be linked with in-car security software 710 installed in the vehicle. For example, in-car security software 710 can be used to provide security functions within vehicle 700. For example, to encrypt the in-vehicle network, in-vehicle security software 710 can use an encryption key obtained from an external authorized server to encrypt data sent and received over the in-vehicle network. In various embodiments, the encryption key used by in-vehicle security software 710 can be generated in accordance with vehicle identification information (vehicle license plate, vehicle identification number (VIN)) or information uniquely assigned to each user (e.g., user identification information).

[0137] In various embodiments, gateway 705 can transmit data encrypted by in-vehicle security software 710 to software management cloud 709 and / or security cloud 706 based on the encryption key. Software management cloud 709 and / or security cloud 706 then decrypt the data encrypted by the encryption key of in-vehicle security software 710 using a decryption key, thereby identifying the vehicle or user from which the data was received. For example, the decryption key is a unique key corresponding to the encryption key, so software management cloud 709 and / or security cloud 706 can identify the data sender (e.g., the vehicle or user) based on the data decrypted using the decryption key.

[0138] For example, gateway 705 is configured to support in-vehicle security software 710 and may be associated with control device 600. For example, gateway 705 may be associated with control device 600 to support a connection between client device 707 connected to secure cloud 706 and control device 600. As another example, gateway 705 may be associated with control device 600 to support a connection between third-party cloud 708 connected to secure cloud 706 and control device 600. However, the present invention is not limited thereto.

[0139] In various embodiments, gateway 705 can be used to connect a software management cloud 709 for managing the operating software of vehicle 700 with vehicle 700. For example, software management cloud 709 monitors whether the operating software of vehicle 700 needs to be updated and, based on monitoring the need for updating the operating software of vehicle 700, can provide data for updating the operating software of vehicle 700 via gateway 705. As another example, software management cloud 709 can receive a user request from vehicle 700 via gateway 705 to update the operating software of vehicle 700 and, based on the request, provide data for updating the operating software of vehicle 700. However, the present invention is not limited to this.

[0140] Figure 8 is a diagram for explaining the operation of an electronic device for training a neural network based on a learning data set according to one embodiment.

[0141] Reference Figure 8 The actions described can be performed by the electronic devices described above (e.g. Figure 1C electronic device 100) to execute.

[0142] Reference Figure 8 In action 802, an electronic device according to an embodiment may obtain a set of learning data. The electronic device may obtain a set of learning data for supervised learning. The learning data may include a pair of input data and ground truth data corresponding to the input data. The ground truth data may represent output data to be obtained from a neural network that receives the input data as the pair of ground truth data. The ground truth data may be obtained by the above-mentioned electronic device.

[0143] For example, when a neural network is trained for image recognition, the learning data may include an image and information about one or more subjects included in the image. The information may include a category (category or class) of a subject that can be recognized by the image. The information may include the position, width, height and / or size of the visual object corresponding to the subject within the image. The set of learning data identified by action 802 may include multiple pairs of learning data. In the example of training a neural network for image recognition, the set of learning data identified by the electronic device may include multiple images and ground truth data corresponding to the multiple images, respectively.

[0144] Reference Figure 8In action 804, the electronic device according to an embodiment may perform training for the neural network based on the set of learning data. In an embodiment of training the neural network based on supervised learning, the electronic device may input the input data included in the learning data into the input layer of the neural network. Figure 9 An example of a neural network including the input layer will be described. The electronic device can obtain output data of the neural network corresponding to the input data from the output layer of the neural network that receives the input data through the input layer.

[0145] In one embodiment, the training of action 804 may be performed based on the difference between the output data and the ground truth data included in the learning data and corresponding to the input data. For example, the electronic device may adjust one or more parameters associated with the neural network (e.g., referring to Figure 9 The electronic device may adjust the one or more parameters to reduce the difference. The electronic device may adjust the one or more parameters to reduce the difference. The electronic device may adjust the neural network based on the output data using a function defined for evaluating the performance of the neural network, such as a cost function. The difference between the output data and the ground truth data may be an example of the cost function.

[0146] Reference Figure 8 In action 806, the electronic device according to one embodiment may identify whether valid output data is output from the neural network trained in action 804. Valid output data may mean that the difference (or cost function) between the output data and the ground truth data satisfies the conditions set for using the neural network. For example, when the average value and / or the maximum value of the difference between the output data and the ground truth data is below a specified critical value, the electronic device may determine that valid output data is output from the neural network.

[0147] When no valid output data is output from the neural network (No in Action 806), the electronic device may repeatedly perform the training of the neural network based on Action 804. However, the embodiment is not limited thereto, and the electronic device may repeatedly perform Actions 802 and 804.

[0148] When valid output data is obtained from the neural network (yes in action 806), the electronic device according to one embodiment may use the trained neural network in action 808. For example, the electronic device may input other input data, which is different from the input data input to the neural network, as learning data to the neural network. The electronic device may use the output data obtained from the neural network that received the other input data as the result of inference performed on the other input data by the neural network.

[0149] Figure 9 is a block diagram of an electronic device according to an embodiment.

[0150] Figure 9 The electronic device 100 may include the aforementioned electronic devices.

[0151] For example, refer to Figure 8 The action described can be Figure 9 The electronic device 100 and / or Figure 9 The processor 910 is used to execute.

[0152] Reference Figure 9 , the processor 910 of the electronic device 100 can perform computations associated with the neural network 930 stored in the memory 920. The processor 910 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), or a neural processing unit (NPU). The NPU can be implemented as a chip separate from the CPU, or it can be integrated on a chip such as a CPU in the form of a system on a chip (SoC). The NPU integrated on the CPU can be called a neural core and / or an artificial intelligence (AI) accelerator.

[0153] Reference Figure 9, the processor 910 can identify the neural network 930 stored in the memory 920. The neural network 930 may include a combination of an input layer 932, one or more hidden layers 934 (or intermediate layers), and an output layer 936. The above-mentioned layers (e.g., the input layer 932, one or more hidden layers 934, and the output layer 936) may include multiple nodes. The number of hidden layers 934 may vary depending on the embodiment. The neural network 930 including multiple hidden layers 934 may be referred to as a deep neural network. The action of training the deep neural network may be referred to as deep learning.

[0154] In one embodiment, when neural network 930 has a feedforward neural network structure, a first node included in a specific layer may be connected to all second nodes included in other layers before the specific layer. Parameters stored for neural network 930 in memory 920 may include weights assigned to links between the second node and the first node. In neural network 930 having a feedforward neural network structure, the value of the first node may correspond to a weighted sum of values ​​assigned to the second node based on the weights assigned to the links connecting the second node and the first node.

[0155] In one embodiment, when the neural network 930 has a convolutional neural network structure, a first node included in a specific layer may correspond to a weighted sum of a portion of second nodes included in other layers before the specific layer. A portion of the second nodes corresponding to the first node may be identified by a filter corresponding to the specific layer. Parameters stored in the memory 920 for the neural network 930 may include weights representing the filter. The filter may include one or more nodes in the second node for calculating the weighted sum of the first node, and weights corresponding to the one or more nodes, respectively.

[0156] According to an embodiment, the processor 910 of the electronic device 100 can perform training for the neural network 930 using the learning data set 940 stored in the memory 920. Based on the learning data set 940, the processor 910 performs a reference Figure 8 The illustrated actions may adjust one or more parameters stored in memory 920 for neural network 930 .

[0157] According to one embodiment, the processor 910 of the electronic device 100 can perform object detection, object recognition and / or object classification using a neural network 930 trained based on a learning data set 940. The processor 910 can input an image (or video) acquired by the camera 950 into the input layer 932 of the neural network 930. Based on the input layer 932 to which the image is input, the processor 910 sequentially acquires the values ​​of the nodes of the layers included in the neural network 930, thereby acquiring a set of values ​​of the nodes of the output layer 936 (e.g., output data). The output data can be used as a result of inferring the information included in the image using the neural network 930. The embodiment is not limited thereto, and the processor 910 can input an image (or video) acquired from an external electronic device connected to the electronic device 100 through the communication circuit 960 into the neural network 930.

[0158] In one embodiment, the neural network 930 trained to process an image can be used to identify an area within the image corresponding to a subject (object detection) and / or identify a category of the subject presented in the image (object recognition and / or object classification). For example, the electronic device 100 can use the neural network 930 to segment the area corresponding to the subject within the image based on a rectangular form such as a bounding box. For example, the electronic device 100 can use the neural network 930 to identify at least one category that matches the subject from a plurality of specified categories.

[0159] In one embodiment, an electronic device in a vehicle includes a communication circuit, a camera, a memory storing instructions, and a processor, and when the processor runs the instructions, the electronic device can run a first model to detect one or more subjects in a first image obtained from the camera, and use feature information obtained from the first model to run a second model to obtain a second image based on the feature information.

[0160] In one embodiment, when the processor executes the instructions, it may cause the electronic device to use the second image to run a third model, thereby determining whether to send the second image to a server.

[0161] In one embodiment, when the processor executes the instructions, the electronic device may be caused to send the second image to a server through the communication circuit.

[0162] In one embodiment, when the processor executes the instructions, the electronic device may acquire data associated with the subject in the first image by executing the first model, and send the data to a server through the communication circuit.

[0163] In one embodiment, the first model includes a convolution neural network. When the processor executes the instructions, the electronic device can obtain the feature information from a hidden layer of the convolution neural network.

[0164] In one embodiment, the hidden layer may include a convolutional layer.

[0165] In one embodiment, the first model and the second model may constitute a generative adversarial network (GAN).

[0166] In one embodiment, a non-transitory computer-readable storage medium may store one or more programs. When a processor of an electronic device including a communication circuit and a camera executes the one or more programs, the electronic device may execute a first model to detect one or more subjects in a first image acquired from the camera, and execute a second model using feature information acquired from the first model to acquire a second image based on the feature information.

[0167] In one embodiment, when the processor runs the one or more programs, the electronic device may be caused to run a third model using the second image, thereby determining whether to send the second image to a server.

[0168] In one embodiment, when the processor runs the one or more programs, the electronic device may be caused to send the second image to a server through the communication circuit.

[0169] In one embodiment, when the processor runs the one or more programs, the electronic device may acquire data associated with the subject in the first image by running the first model, and send the data to the server through the communication circuit.

[0170] In one embodiment, the first model includes a convolution neural network. When the processor runs the one or more programs, the electronic device can obtain the feature information from a hidden layer of the convolution neural network.

[0171] In one embodiment, the hidden layer may include a convolutional layer.

[0172] In one embodiment, the first model and the second model may constitute a generative adversarial network (GAN).

[0173] In one embodiment, a method for an electronic device of a vehicle including a communication circuit and a camera may include: running a first model to detect the motion of one or more subjects in a first image obtained from the camera; and running a second model using feature information obtained from the first model to obtain the motion of a second image based on the feature information.

[0174] In one embodiment, the method may include an action of running a third model using the second image to determine whether to send the second image to a server.

[0175] In one embodiment, the method may include an action of sending the second image to a server via the communication circuit.

[0176] In one embodiment, the method may include: acquiring data associated with the subject in the first image by running the first model; and sending the data to a server through the communication circuit.

[0177] In one embodiment, the first model includes a convolution neural network, and the method may include: obtaining the feature information from a hidden layer of the convolution neural network.

[0178] In one embodiment, the hidden layer includes a convolutional layer, and the first model and the second model may constitute a generative adversarial network (GAN).

[0179] In one embodiment of this article and the terms used therein are not intended to limit the technical features described herein to specific embodiments, but should be understood to include various changes, equivalents or substitutes of the corresponding embodiments. In conjunction with the accompanying drawings, similar figure marks may be used for similar or associated constituent elements. Unless the context clearly indicates otherwise, the singular form of the noun corresponding to the component (item) may include one or more of the components. In this article, statements such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one of the items listed in the corresponding statements in these statements, or all possible combinations thereof. Terms such as "first", "second" or "first time" or "second time" are simply used to distinguish the corresponding constituent elements from other corresponding constituent elements and are not intended to limit the corresponding constituent elements in other aspects (e.g., importance or order). When any (e.g., a first) component is described as being “coupled” or “connected” to another (e.g., a second) component, whether or not it carries terms such as “functional” or “communicative,” it means that the any component can be connected to the other component directly (e.g., by wire), wirelessly, or through a third component.

[0180] In the specific embodiments of the present disclosure described above, the constituent elements included in the disclosure are expressed as a single or multiple elements depending on the specific embodiment proposed. However, the expression of single or multiple elements is a choice that is made for the convenience of explanation and is appropriate to the situation. The present disclosure is not limited to single or multiple constituent elements, and even constituent elements expressed as multiple elements can also be composed of a single element, and even constituent elements expressed as a single element can also be composed of multiple elements.

[0181] According to an embodiment, one or more constituent elements or actions of the aforementioned corresponding constituent elements can be omitted, or one or more other constituent elements or actions can be added. Alternatively or additionally, a plurality of constituent elements (e.g., modules or programs) can be integrated into one constituent element. In this case, the integrated constituent element can perform one or more functions of each of the plurality of constituent elements, which are identical or similar to the functions performed by the corresponding constituent elements in the plurality of constituent elements before the integration. According to an embodiment, the actions performed by modules, programs or other constituent elements can be performed in an orderly manner, in parallel, repeatedly or heuristically, or one or more actions in the actions can be performed in different orders, or one or more other actions can be omitted or added.

[0182] In addition, although the embodiments have been specifically described in the detailed description of the present disclosure, it is of course possible to implement various modifications thereto without departing from the scope of the present disclosure.

Claims

1. An electronic device, arranged in a vehicle, wherein: The electronic device comprises: Communication circuits, Camera, Memory, which stores instructions, and processor; When the processor executes the instructions, the electronic device is configured to: running a first model to detect one or more objects in a first image acquired from the camera, and The second model is run using the feature information obtained from the first model to obtain a second image based on the feature information.

2. The electronic device according to claim 1, wherein When the processor executes the instructions, the electronic device is configured to: A third model is run using the second image to determine whether to send the second image to a server.

3. The electronic device according to claim 1, wherein When the processor executes the instructions, the electronic device is configured to: The second image is sent to a server through the communication circuit.

4. The electronic device according to claim 1, wherein When the processor executes the instructions, the electronic device is configured to: acquiring data associated with the subject in the first image by running the first model; and The data is sent to a server through the communication circuit.

5. The electronic device according to claim 1, wherein The first model includes a convolutional neural network, When the processor executes the instructions, the electronic device is configured to: The feature information is obtained from a hidden layer of the convolutional neural network. The electronic device according to claim 5 , wherein: The hidden layer includes a convolutional layer.

7. The electronic device according to claim 1, wherein The first model and the second model constitute a generative adversarial network.

8. A non-transitory computer-readable storage medium storing one or more programs, wherein: When a processor of an electronic device including a communication circuit and a camera runs the one or more programs, the electronic device is configured to: running a first model to detect one or more objects in a first image acquired from the camera; and The second model is run using the feature information obtained from the first model to obtain a second image based on the feature information.

9. The non-transitory computer-readable storage medium according to claim 8, wherein: When the processor runs the one or more programs, the electronic device is configured to: A third model is run using the second image to determine whether to send the second image to a server.

10. The non-transitory computer-readable storage medium of claim 8, wherein: When the processor runs the one or more programs, the electronic device is configured to: The second image is sent to a server through the communication circuit.

11. The non-transitory computer-readable storage medium of claim 8, wherein: When the processor runs the one or more programs, the electronic device is configured to: acquiring data associated with the subject in the first image by running the first model; and The data is sent to a server through the communication circuit.

12. The non-transitory computer-readable storage medium of claim 8, wherein: The first model includes a convolutional neural network, When the processor runs the one or more programs, the electronic device is configured to: The feature information is obtained from a hidden layer of the convolutional neural network.

13. The non-transitory computer-readable storage medium of claim 12, wherein: The hidden layer includes a convolutional layer.

14. The non-transitory computer-readable storage medium of claim 8, wherein: The first model and the second model constitute a generative adversarial network.

15. A method for operating an electronic device, the electronic device comprising a communication circuit and a camera, in, include: running a first model to detect motion of one or more subjects in a first image acquired from the camera; and The second model is run using the feature information obtained from the first model to obtain a motion of the second image based on the feature information.

16. The operating method of the electronic device according to claim 15, wherein: include: An action is performed by running a third model using the second image to determine whether to send the second image to a server.

17. The operating method of the electronic device according to claim 15, wherein: include: An action of sending the second image to a server through the communication circuit.

18. The method for operating an electronic device according to claim 15, in, include: an act of acquiring data associated with a subject within the first image by executing the first model; and The action of sending the data to the server through the communication circuit.

19. The operating method of the electronic device according to claim 15, wherein: The first model includes a convolutional neural network, The operation method of the electronic device includes: The action of obtaining the feature information from the hidden layer of the convolutional neural network.

20. The operating method of the electronic device according to claim 19, wherein: The hidden layer includes a convolutional layer, The first model and the second model constitute a generative adversarial network.

Images