Electronic device and method for assisting driving of a vehicle

Abstract

An electronic device can include a processor, wherein the processor is configured to obtain a plurality of images, wherein the plurality of images include a first image captured by a first camera or a first image sensor and a second image captured by a second camera or a second image sensor. The processor can be further configured to determine whether first and second objects included in the first and second images, respectively, are the same object, and based on determining that the first and second objects are the same object, train a neural network data recognition model to recognize the first and second objects as the same object in both the first and second images by using the first and second images as training data for the neural network data recognition model.

Description

Technical Field

[0001] This disclosure relates to an electronic device and a method of operating the electronic device, and more specifically, to an electronic device and method for real-time detection of objects, such as assisting in driving a vehicle. Background Technology

[0002] With the integration of information and communication technologies and the automotive industry, automotive intelligence is developing rapidly. As intelligence evolves, cars are transforming from simple mechanical devices into intelligent vehicles, and autonomous driving, as a key technology for intelligent vehicles, is receiving increasing attention.

[0003] Fully automated driving is a technology where a vehicle (e.g., to its destination) moves without the driver operating the steering wheel, accelerator pedal, or brakes. Semi-automatic driving may involve some driver intervention.

[0004] Recently, various additional functions related to autonomous driving have been continuously developed, and there is a need for research on methods to provide occupants with a safe autonomous driving experience by controlling the vehicle by using various data to identify and determine the driving environment.

[0005] Furthermore, artificial intelligence (AI) systems are computer systems with human-like intelligence. Unlike previous rule-based intelligent systems, AI systems can train themselves, make decisions, and become increasingly intelligent. The more an AI system is used, the higher its recognition rate becomes, and the more accurately it can understand user preferences. Therefore, previous rule-based intelligent systems have been gradually replaced by AI, such as deep learning-based AI systems.

[0006] AI technology consists of machine learning (e.g., deep learning) and human element technologies that use machine learning.

[0007] Machine learning is an algorithmic technique that independently classifies / learns the characteristics of input data. Human element technology is a technique that uses machine learning algorithms (such as deep learning) to simulate the functions of the human brain (such as recognition, determination, etc.), and includes technical fields such as language understanding, visual understanding, reasoning / prediction, knowledge representation, and operational control.

[0008] AI technology can be applied to various fields, including: Language understanding, which is a technology for recognizing and processing human language / characters, encompassing natural language processing, machine translation, conversational systems, query response, speech recognition / synthesis, etc.; Visual understanding, which is a technology for recognizing and processing objects in human vision, including object recognition, object detection, image search, human recognition, scene understanding, spatial understanding, image enhancement, etc.; Reasoning and prediction, which is a technology for examining information and making logical inferences and predictions, including knowledge-based / probability-based reasoning, optimization prediction, preference-based planning, recommendation, etc.; Knowledge representation, which is a technology for automatically processing human experience data into knowledge data, including knowledge creation (data generation / classification) and knowledge management (data usage), etc.; and Operational control, which is a technology for controlling the autonomous driving of vehicles and the movement of robots, including motion control (navigation, collision avoidance, driving) and manipulation control (behavior control), etc. Summary of the Invention

[0009] Technical solution

[0010] An electronic device and method for assisting in driving a vehicle are provided. Attached Figure Description

[0011] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1 This is a schematic diagram illustrating an example of the operation of an electronic device for assisting in driving a vehicle according to an embodiment;

[0013] Figure 2 This is a flowchart of an operation method of an electronic device according to an embodiment;

[0014] Figure 3 This is a flowchart of an operation method for an electronic device and multiple vehicles according to an embodiment;

[0015] Figure 4 This is a view used to describe examples of multiple vehicles capturing images while driving, according to an embodiment;

[0016] Figure 5 This is a view used to describe an example of multiple images captured by multiple vehicles according to an embodiment;

[0017] Figure 6 This is a view used to describe an example of using images as training data for a data recognition model according to an embodiment;

[0018] Figure 7 This is a flowchart illustrating an example of determining whether a first object and a second object are the same object according to an embodiment;

[0019] Figure 8 This is a view used to describe an example of determining whether a first object and a second object are the same object according to an embodiment;

[0020] Figure 9 This is a flowchart illustrating an example of selecting an image based on an object recognition rate according to an embodiment;

[0021] Figure 10 This is a flowchart illustrating another example of selecting an image based on object recognition rate according to an embodiment;

[0022] Figure 11 This is a view used to describe an example of selecting an image based on object recognition rate according to an embodiment;

[0023] Figure 12 This is a block diagram of an electronic device according to an embodiment;

[0024] Figure 13 This is a block diagram of the processor of an electronic device according to an embodiment;

[0025] Figure 14 This is a block diagram of a data learner according to an embodiment;

[0026] Figure 15 This is a block diagram of a data recognizer according to an embodiment;

[0027] Figure 16 This is a view illustrating an example of an electronic device and an external server learning and recognizing data by interacting with each other according to an embodiment; and

[0028] Figure 17 This is a block diagram of a vehicle according to an embodiment. Detailed Implementation

[0029] An electronic device and method for assisting in driving a vehicle are provided. Furthermore, a non-transitory computer-readable recording medium on which a program for executing the method on a computer is provided is also provided. The technical problem to be solved is not limited to the technical problem described above, and other technical problems may exist.

[0030] Other aspects will be set forth in part in the description which follows, and will be apparent in part from the description, or may be learned by practicing the embodiments presented in this disclosure.

[0031] According to one aspect of this disclosure, an electronic device for assisting the driving of multiple vehicles includes: a communication interface; a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to perform the following operations: receiving, via the communication interface, multiple images captured by the multiple vehicles and multiple capture information corresponding to the multiple images; determining whether a first object and a second object included in a first image and a second image respectively in the multiple images are the same object, wherein, based on a first capture information corresponding to the first image and a second capture information corresponding to the second image, the first object and the second object are determined to be located at the same position at the same time point; and when it is determined that the first object and the second object are the same object, modifying and improving a data recognition model trained to identify objects from images by using the first image and the second image as training data for a data recognition model.

[0032] According to one aspect of this disclosure, a method for assisting driving of multiple vehicles includes: receiving, via a communication interface, multiple images captured by the multiple vehicles and multiple capture information corresponding to the multiple images; determining whether a first object and a second object included in a first image and a second image respectively in the multiple images are the same object, wherein, based on a first capture information corresponding to the first image and a second capture information corresponding to the second image, the first object and the second object are determined to be located at the same position at the same time point; and when it is determined that the first object and the second object are the same object, modifying and improving a data recognition model trained to identify objects from images by using the first image and the second image as training data for a data recognition model.

[0033] According to one aspect of this disclosure, a computer-readable recording medium having thereon a program for performing the above-described methods on a computer is provided.

[0034] According to one aspect of this disclosure, an electronic device may include a communication interface, a memory, and a processor.

[0035] According to one aspect of this disclosure, the processor may be configured to execute one or more instructions stored in the memory to perform the following operations: acquiring multiple images of a real-world scene via the communication interface, wherein the multiple images include a first image captured by a first camera or a first image sensor and a second image captured by a second camera or a second image sensor; determining whether a first object included in the first image and a second object included in the second image are the same object, wherein the first object and the second object are determined to be the same object based on the similarity between the first object and the second object and based on a first capture time and location information corresponding to the first image and a second capture time and location information corresponding to the second image corresponding to the same time point and the same location, and wherein the first capture time and location information includes information about the time point of capture of the first image and information about the location of capture of the first image, and the second capture time and location information includes information about the time point of capture of the second image and information about the location of capture of the second image; and based on the determination that the first object and the second object are the same object, training a neural network data recognition model to recognize the first object and the second object as the same object in both the first image and the second image by using the first image and the second image as training data for the neural network data recognition model.

[0036] According to one aspect of this disclosure, the processor may also be configured to execute one or more instructions to perform the following operations: based on determining that a first object and a second object are not the same object, based on the object recognition rate of the first image or the second image being less than a predefined threshold recognition rate, excluding the first image or the second image from the training data, and training a neural network data recognition model based on the training data to recognize the object.

[0037] According to one aspect of this disclosure, the processor may also be configured to execute the one or more instructions to perform the following operations: obtaining, via the communication interface, first capture time and location information corresponding to the first image and second capture time and location information corresponding to the second image.

[0038] According to one aspect of this disclosure, the processor may also be configured to execute the one or more instructions to determine the similarity between a first object and a second object by: calculating a similarity value by matching the first object with the second object; and determining that the first object and the second object are the same object when the similarity value is determined to be equal to or higher than a preset threshold and the first image and the second image are located at the same position at the same time point.

[0039] According to one aspect of this disclosure, the processor may also be configured to execute one or more instructions to perform the following operations: identify objects from a plurality of images using a neural network data recognition model; select images from the plurality of images whose object recognition rate is calculated to be equal to or lower than a preset threshold; and determine whether to use the selected images as training data for the neural network data recognition model.

[0040] According to one aspect of this disclosure, the processor may also be configured to execute the one or more instructions to perform the following operations: receiving information from a plurality of vehicles via the communication interface regarding the object recognition rate of objects identified in each of the plurality of images using a neural network data recognition model; selecting from the plurality of images an image whose object recognition rate is calculated to be equal to or lower than a preset threshold, and determining whether to use the selected image as training data for the neural network data recognition model.

[0041] According to one aspect of this disclosure, the processor may also be configured to execute the one or more instructions to perform the following operation: sending a trained neural network data recognition model to a plurality of vehicles via the communication interface, wherein the plurality of vehicles includes a first vehicle equipped with a first camera or a first image sensor and a second vehicle equipped with a second camera or a second image sensor.

[0042] According to one aspect of this disclosure, a method may include: acquiring multiple images of a real-world scene via a communication interface, wherein the multiple images include a first image captured by a first camera or a first image sensor and a second image captured by a second camera or a second image sensor; determining whether a first object and a second object included in the first image and the second image are the same object, wherein the first object and the second object are determined to be the same object based on the similarity between the first object and the second object and based on a first capture time and location information corresponding to the first image and a second capture time and location information corresponding to the second image corresponding to the same time point and the same location, wherein the first capture time and location information includes information about the time point of capture of the first image and information about the location of capture of the first image, and the second capture time and location information includes information about the time point of capture of capture of the second image and information about the location of capture of capture of the second image; and based on the determination that the first object and the second object are the same object, training a neural network data recognition model to recognize the first object and the second object as the same object in both the first image and the second image by using the first image and the second image as training data for the neural network data recognition model.

[0043] According to one aspect of this disclosure, the method may further include: excluding the first image or the second image from the training data based on determining that the first object and the second object are not the same object, based on the object recognition rate of the first image or the second image being less than a predefined threshold recognition rate, and training a neural network data recognition model based on the training data to identify the object.

[0044] According to one aspect of this disclosure, the method may further include: obtaining a first capture time and location information and a second capture time and location information corresponding to the first image and the second image, respectively, through the communication interface.

[0045] According to one aspect of this disclosure, the similarity between the first object and the second object is determined by: calculating a similarity value by matching the first object with the second object; and determining that the first object and the second object are the same object when the similarity value is determined to be equal to or higher than a preset threshold and the first image and the second image are located at the same position at the same time point.

[0046] According to one aspect of this disclosure, the method may further include: identifying an object from a plurality of images using a neural network data recognition model; selecting from the plurality of images an image whose object recognition rate is calculated to be equal to or lower than a preset threshold; and determining whether to use the selected image as training data for the neural network data recognition model.

[0047] According to one aspect of this disclosure, the method may further include: receiving information from a plurality of vehicles via the communication interface regarding the object recognition rate of objects identified from each of the plurality of images using a neural network data recognition model; selecting from the plurality of images an image whose object recognition rate is calculated to be equal to or lower than a preset threshold; and determining whether to use the selected image as training data for the neural network data recognition model.

[0048] According to one aspect of this disclosure, the method may further include: sending a trained neural network data recognition model to a plurality of vehicles via the communication interface, wherein the plurality of vehicles includes a first vehicle equipped with a first camera or a first image sensor and a second vehicle equipped with a second camera or a second image sensor.

[0049] According to one aspect of this disclosure, a non-transitory computer-readable recording medium may have a program recorded thereon for performing a method, the method comprising: acquiring a plurality of images of a real-world scene via a communication interface, wherein the plurality of images includes a first image captured by a first camera or a first image sensor and a second image captured by a second camera or a second image sensor; determining whether a first object included in the first image and a second object included in the second image are the same object, wherein the first object and the second object are determined to be the same object based on a similarity between the first object and the second object and based on a first capture time and location information corresponding to the first image and a second capture time and location information corresponding to the second image corresponding to the same time point and the same location, wherein the first capture time and location information includes information about the time point at which the first image was captured and information about the location at which the first image was captured, and the second capture time and location information includes information about the time point at which the second image was captured and information about the location at which the second image was captured; and, based on the determination that the first object and the second object are the same object, training a neural network data recognition model to recognize the first object and the second object as the same object in both the first image and the second image by using the first image and the second image as training data for the neural network data recognition model.

[0050] According to one aspect of this disclosure, a computer-implemented method for training a neural network for object detection may include: collecting a set of digital sample images from a database; inputting the collected set of digital facial images into a neural network recognition model; and training the neural network recognition model to recognize the first object as a specific object based on the similarity between a first object in a first digital image and a second object in a second digital image that co-located with the first object at the same time point.

[0051] According to one aspect of this disclosure, the method may further include: detecting objects in a collection of collected digital sample images and analyzing the detected objects; and determining that the first object and the second object are similar to each other when one or more appearance attributes of a first object in a first digital image are the same as one or more appearance attributes of a second object in a second digital image.

[0052] According to one aspect of this disclosure, the object recognition rate of the first digital image using a neural network recognition model for a specific object may be lower than a predefined recognition rate threshold for the neural network recognition model for a specific object, and the object recognition rate of the second digital image using a neural network recognition model for a specific object may be higher than the predefined recognition rate threshold.

[0053] In view of the function of the terms used in this disclosure, they are selected from commonly used terms that are currently widely used. However, the terms may differ depending on the intent of those skilled in the art, precedents, or the emergence of new technologies. Furthermore, in certain cases, the terms are arbitrarily chosen by the applicant of this disclosure, and the meanings of these terms will be described in detail in the relevant sections of the detailed description. Therefore, the terms used in this disclosure are not merely designations of terms, but are defined based on their meanings and the content throughout this disclosure.

[0054] Throughout this disclosure, when a component “comprises” an element, it will be understood that the component additionally includes other elements rather than excludes other elements, unless specifically stated otherwise. Furthermore, terms described in this disclosure (such as “unit,” “module,” etc.) may refer to a unit that can be implemented as hardware (e.g., a hardware processor) or software or a combination of hardware and software, performing at least one function or operation.

[0055] In the following description, embodiments will now be described more fully with reference to the accompanying drawings so that those skilled in the art can perform this disclosure without difficulty. However, this disclosure may be implemented in many different forms and should not be construed as limited to the embodiments set forth herein.

[0056] Throughout this disclosure, the expression "at least one of a, b or c" means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b and c, or variations thereof.

[0057] In the following description, embodiments will be described in detail with reference to the accompanying drawings.

[0058] Figure 1 This is a schematic diagram illustrating an example of the operation of an electronic device 100 for assisting in driving a vehicle according to an embodiment.

[0059] According to an embodiment, the electronic device 100 can operate as a server for assisting or controlling the driving operation of a vehicle.

[0060] According to embodiments, the first vehicle 200a, the second vehicle 200b, and the third vehicle 200c can be autonomous vehicles, or vehicles capable of automatically controlling some driving operations. For example, the first vehicle 200a, the second vehicle 200b, and the third vehicle 200c can be implemented to perform the functions of an autonomous driving system or an advanced driver assistance system (ADAS). Autonomous vehicles, or vehicles capable of automatically controlling at least one function, can use various sensors to identify objects around the vehicle while driving, and thus provide a stable driving environment.

[0061] According to the embodiments, an object may refer to an object included in an image obtained by capturing the surrounding environment (front, rear, sides, etc.) of the vehicle 200. Furthermore, an object may refer to an object detected in the surrounding environment (front, rear, sides, etc.) of the vehicle 200 while the vehicle 200 is being driven. For example, detectable objects may include other vehicles, pedestrians, obstacles in the driving path, etc., but are not limited to these.

[0062] Reference Figure 1 The first vehicle 200a, the second vehicle 200b, and the third vehicle 200c can perform relatively more accurate and stable autonomous driving operations by exchanging data with the electronic device 100.

[0063] The vehicle 200 according to an embodiment may correspond to at least one processor and at least one memory located in or installed in the vehicle 200. The vehicle 200 according to an embodiment may utilize an image sensor 228 ( Figure 17 The system captures the surrounding environment of the vehicle 200 and can identify objects (e.g., pedestrians, vehicles, road signs, etc.) from the images obtained by capturing the surrounding environment.

[0064] Image sensor 228 ( Figure 17 The image sensor 228 may include a still camera or a video camera configured to record the external environment of the vehicle. For example, the image sensor 228 may include multiple cameras, and the multiple cameras may be arranged in multiple locations inside and outside the vehicle 200.

[0065] According to an embodiment, vehicle 200 can utilize image sensor 228 ( Figure 17 It captures the surrounding environment and uses a data recognition model trained based on deep learning to identify objects from multiple image frames obtained by capturing the surrounding environment of vehicle 200.

[0066] According to an embodiment, vehicle 200 can extract candidate regions from image frames to identify objects from the image frames, and can estimate the type and location of objects from the extracted candidate regions by using a trained data recognition model.

[0067] Data recognition models can be, for example, neural network-based models. For example, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), bidirectional recurrent deep neural networks (BRDNNs) can be used as data recognition models, but are not limited to these.

[0068] For example, vehicle 200 can identify objects (e.g., pedestrians, vehicles, road signs, etc.) in an image frame by using a data recognition model based on a CNN that includes multiple layers. Each layer can receive data and process the received data to produce data output from each layer.

[0069] According to an embodiment, vehicle 200 may perform object recognition via a graphics processing unit (GPU). That is, according to an embodiment, at least one processor of the vehicle, which may be a general-purpose processor, can perform control to enable object recognition to be performed using a GPU, which may be a processor more powerful than the general-purpose processor. However, the vehicle is not limited in this respect. In an embodiment, the GPU may be used to perform the functions of a general-purpose processor. In an embodiment, the general-purpose processor may perform object recognition. Furthermore, vehicle 200 may perform object recognition via a dedicated hardware chip for artificial intelligence (AI), but is not limited thereto.

[0070] Figures 13 to 15 A processor 1300 corresponding to a GPU is shown. However, the processor 1300 is not limited to a GPU and can be used as other processors. The processor 1300 can be a hardware processor.

[0071] like Figure 13 As shown, the processor 1300 may include a data learner 1310 and a data recognizer 1320. The following description... Figures 13 to 16 The processor 1300 is described as the electronic device 100. However, according to an embodiment, the processor 120 of the vehicle 200 ( Figure 17 It can be operated. Figure 13 The operation of the processor 1300.

[0072] Furthermore, according to an embodiment, the electronic device 100 can identify objects from a plurality of image frames obtained by capturing the surrounding environment of a first vehicle 200a, a second vehicle 200b, and a third vehicle 200c, wherein the plurality of image frames are received from the first vehicle 200a, the second vehicle 200b, and the third vehicle 200c, and the identification is performed by using a data recognition model trained based on deep learning.

[0073] According to an embodiment, the data recognition model can be a pre-built model. For example, the data recognition model can be a pre-built model that receives basic training data (e.g., sample images, etc.).

[0074] Furthermore, according to embodiments, the data recognition model can be retrained using training data. For example, the data recognition model can be trained through unsupervised learning, thereby discovering criteria for situation determination by learning the data required for situation determination.

[0075] According to an embodiment, the electronic device 100 can train a data recognition model by using multiple image frames received from the vehicle 200 and the results of object recognition as training data.

[0076] According to the embodiment, even when the object recognition rate from an image is low, the electronic device 100 can use the image with the low object recognition rate and the results of data recognition as training data for the data recognition model when it is determined that the object is the same as an object in another image captured at the same time point. For example, when a first image has a low recognition rate for human objects, but it has been determined that the object in the first image is the same as an object in a second image with a high recognition rate for human objects, it can be inferred that the image with the low recognition rate actually does indeed contain a human object (the opposite of a low recognition rate).

[0077] For example, electronic device 100 can improve the object recognition rate of the data recognition model by training the data recognition model with the results of object recognition using the partial outline of the object (which is mostly covered by the outline of another object) and the object recognition results.

[0078] According to an embodiment, the electronic device 100 can send a modified and improved data recognition model to a first vehicle 200a, a second vehicle 200b, and a third vehicle 200c.

[0079] According to an embodiment, vehicle 200 can more accurately identify objects from images by performing object recognition using a modified and improved data recognition model. Therefore, vehicle 200 can perform relatively safer and more accurate autonomous driving.

[0080] The operation of the electronic device 100 according to the embodiment will now be described in more detail with reference to the accompanying drawings.

[0081] Figure 1 Examples have been shown, but this disclosure is not limited thereto.

[0082] Figure 2 This is a flowchart of the operation method of the electronic device 100 according to an embodiment. Figure 3 This is a flowchart of the operation method of the electronic device 100 and multiple vehicles according to an embodiment.

[0083] Figure 4 This is a view used to describe an example of multiple images captured by multiple vehicles according to an embodiment. Figure 5 This is a view used to describe an example of using images as input data for training a learning model, according to an embodiment.

[0084] By reference Figures 4 to 6 describe Figure 2 and Figure 3 The flowchart.

[0085] exist Figure 2In operation S201, the electronic device 100 can receive multiple images captured by multiple vehicles and multiple capture information corresponding to the multiple images from multiple vehicles.

[0086] According to an embodiment, vehicle 200 can utilize image sensor 228 ( Figure 17 ) capture the surrounding environment and communicate via interface 160 ( Figure 17 The device sends multiple images obtained by capturing the surrounding environment to the electronic device 100.

[0087] According to an embodiment, vehicle 200 can obtain capture information corresponding to the captured image. Vehicle 200 can transmit the capture information corresponding to the captured image to electronic device 100 via communication interface 160. Figure 17 ).

[0088] According to an embodiment, the capture information may include information about the time point in time the image was captured and information about the location of the captured image. For example, the information about the time point may include the hour, minute, and second at which the vehicle 200 captured the image. Furthermore, the information about the location may include, for example, the geographical location of an object in the image at the time the vehicle 200 captured the image (e.g., Global Positioning System (GPS) information). Additionally, the information about the location may include information about the relative distance between the location of the vehicle 200 and the location of an object in the image, as well as information about the relative direction. Furthermore, the information about the location may include, but is not limited to, information about the relative distance between an object in the image and another object, and information about the relative direction.

[0089] Figure 4 An example of a driving environment is shown. (See reference) Figure 4 The first vehicle 200a, the second vehicle 200b, and the third vehicle 200c can be driven by using image sensor 228 ( Figure 17 To capture what is in front. For example, according to an embodiment, a pedestrian 300 may cross the crosswalk in front of the first vehicle 200a, the second vehicle 200b, and the third vehicle 200c.

[0090] Figure 5 It shows in Figure 4 Examples of images captured by the first vehicle 200a, the second vehicle 200b, and the third vehicle 200c in a driving environment are shown.

[0091] Reference Figure 5A first image 400a captured by the first vehicle 200a may include a first object 500a. Furthermore, a second image 400b captured by the second vehicle 200b may include a second object 500b. Additionally, a third image 400c captured by the third vehicle 200c may include a third object 500c and a fourth object 500d.

[0092] According to an embodiment, a first vehicle 200a can send a first image 400a and a first capture message corresponding to the first image 400a to an electronic device 100. Furthermore, a second vehicle 200b can send a second image 400b and a corresponding second capture message to the electronic device 100. Additionally, a third vehicle 200c can send a third image 400c and a corresponding third capture message to the electronic device 100.

[0093] exist Figure 2 In operation S202, the electronic device 100 can determine whether a first object and a second object in a first image and a second image respectively included in a plurality of images are the same as each other, wherein, based on a first capture information and a second capture information corresponding to the first image and the second image respectively, the first object and the second object are determined to be located at the same position at the same time point.

[0094] According to an embodiment, the electronic device 100 can identify the time and location of the first image capture based on a first capture information corresponding to the first image. Furthermore, the electronic device 100 can identify the time and location of the second image capture based on a second capture information corresponding to the second image.

[0095] According to an embodiment, the electronic device 100 can detect a first object and a second object respectively included in a first image and a second image, wherein the first object and the second object are determined to be located at the same position at the same time point.

[0096] Reference Figure 4 and Figure 5 The first object 500a included in the first image 400a and the second object 500b included in the second image 400b can be pedestrian 300. Figure 4 ).

[0097] According to an embodiment, the electronic device 100 can determine whether a first object 500a included in a first image 400a and a second object 500b included in a second image 400b are the same object. Furthermore, according to an embodiment, the electronic device 100 can determine whether a second object 500b included in a second image 400b and a third object 500c included in a third image 400c are the same object. Furthermore, according to an embodiment, the electronic device 100 can determine whether a first object 500a included in a first image 400a and a third object 500c included in a third image 400c are the same object.

[0098] The following will refer to Figure 7 and Figure 8 A detailed example of a method for determining whether multiple objects are the same object, according to an embodiment.

[0099] exist Figure 2 In operation S203, when it is determined that the first object and the second object are the same object, the electronic device 100 can modify and improve the data recognition model trained to recognize objects from images (including actually performing training) by using the first image and the second image as training data for the data recognition model.

[0100] Reference Figure 5 When it is determined that the first object 500a and the second object 500b are the same object, the electronic device 100 can use the first image 400a and the second image 400b as training data for a data recognition model trained to identify objects from images. Furthermore, when it is determined that the second object 500b and the third object 500c are the same object, the electronic device 100 can use the second image 400b and the third image 400c as training data for the data recognition model.

[0101] like Figure 6 As shown, according to an embodiment, the electronic device 100 can modify and improve the data recognition model by using the first image 400a, the second image 400b, and the third image 400c as training data for the data recognition model.

[0102] Figure 3 This is a flowchart illustrating the operation of the electronic device 100 and multiple vehicles according to an embodiment.

[0103] exist Figure 3 In operation S301, the electronic device 100 can receive a first capture information and a second capture information corresponding to the first image and the second image, respectively.

[0104] According to an embodiment, the electronic device 100 can receive a first image and a first capture information corresponding to the first image from the first vehicle 200a. Furthermore, according to an embodiment, the electronic device 100 can receive a second image and a second capture information corresponding to the second image from the second vehicle 200b.

[0105] exist Figure 3 In operation S302, the electronic device 100 can determine whether a first object included in a first image and a second object included in a second image are the same object, wherein the first object and the second object are determined to be located at the same position at the same time point.

[0106] Reference Figure 4 and Figure 5 For example, each of the first vehicle 200a and the second vehicle 200b can capture a pedestrian 300 in front of it. The electronic device 100 can determine whether a first object 500a included in a first image 400a captured by the first vehicle 200a and a second object 500b included in a second image 400b captured by the second vehicle 200b are the same object.

[0107] In addition, refer to Figure 4 and Figure 5 For example, each of the first vehicle 200a and the third vehicle 200c can capture a pedestrian 300 in front of it. The electronic device 100 can determine whether a first object 500a included in a first image 400a captured by the first vehicle 200a and a third object 500c included in a third image 400c captured by the third vehicle 200c are the same object.

[0108] exist Figure 3 In operation S303, the electronic device 100 can modify and improve the data recognition model by using the first image and the second image as training data for the data recognition model.

[0109] Reference Figure 4 and Figure 5 For example, when it is determined that the first object 500a included in the first image 400a and the second object 50b included in the second image 400b are the same object (e.g., pedestrian 300). Figure 4 When using the first image 400a and the second image 400b as training data for the data recognition model, the data recognition model can be modified and improved.

[0110] In addition, refer to Figure 4 and Figure 5 For example, when it is determined that the first object 500a included in the first image 400a and the third object 500c included in the third image 400c are the same object (e.g., pedestrian 300). Figure 4When using the first image 400a and the third image 400c as training data for the data recognition model, the data recognition model can be modified and improved.

[0111] exist Figure 3 In operation S304, the electronic device 100 can send the modified and improved data recognition model to the first vehicle 200a and the second vehicle 200b.

[0112] According to an embodiment, the first vehicle 200a and the second vehicle 200b can use a data recognition model received from the electronic device 100 to identify objects from captured images. The first vehicle 200a and the second vehicle 200b can perform relatively more accurate object recognition by using a modified and improved data recognition model. Therefore, the first vehicle 200a and the second vehicle 200b can perform relatively safer and more accurate autonomous driving.

[0113] Figures 2 to 6 Examples have been shown, but this disclosure is not limited thereto.

[0114] Figure 7 This is a flowchart illustrating an example of determining whether a first object and a second object are the same object according to an embodiment. Figure 8 This is a view used to describe an example of determining whether a first object and a second object are the same object according to an embodiment.

[0115] According to an embodiment, the electronic device 100 can determine whether an object in an image captured by the vehicle 200 is the same object as an object in another image captured at the same time point as the captured image, in order to determine whether to use the image captured by the vehicle 200 as training data for a data recognition model.

[0116] Reference Figure 8 For example, the third object 500c is only partially visible in the third image 400c because it is hidden by the outline of the fourth object 500d. Therefore, the recognition rate of the electronic device 100 in identifying the third object 500c from the third image 400c may be low.

[0117] According to an embodiment, the electronic device 100 can use the results of object recognition from an image as training data for a data recognition model. Even if the electronic device 100 determines that the recognition rate of the third object 500c is low based on the results of recognizing the third object 500c, the electronic device 100 can still use the third image 400c as training data for a data recognition model when the third object 500c is determined to be the same object as the first object 500a and the second object 500b located at the same position as the third object 500c at the same time point.

[0118] Therefore, the data recognition model can learn a portion of the contour of the third object 500c (which is largely covered by the contour of the fourth object 500d) and the result of object recognition, thereby achieving an improved object recognition rate. In other words, according to the embodiment, the electronic device 100 can improve the accuracy of the data recognition model.

[0119] exist Figure 7 In operation S701, the electronic device 100 can calculate the similarity by matching the first object and the second object.

[0120] According to an embodiment, the electronic device 100 can determine whether a first object and a second object included in a first image and a second image are the same object, wherein the first object and the second object are determined to be located at the same position at the same time.

[0121] Reference Figure 8 For example, electronic device 100 can determine whether a first object 500a included in a first image 400a, a second object 500b included in a second image 400b, and a third object 500c included in a third image 400c are the same object.

[0122] Electronic device 100 can calculate similarity by matching at least two objects. According to an embodiment, electronic device 100 can compare the at least two objects by using the hue, saturation, value, color histogram, etc., of the at least two objects included in the image. However, it is not limited to this.

[0123] For example, electronic device 100 can match first object 500a and third object 500c using the hue, saturation, and brightness of the first object 500a and the third object 500c. Electronic device 100 can calculate similarity by matching first object 500a and third object 500c.

[0124] Reference Figure 8 For example, the similarity between the first object 1 and the third object 3 can be calculated as 0.9. Furthermore, for example, the similarity between the first object 1 and the second object 2 can be calculated as 0.8. Furthermore, for example, the similarity between the second object 2 and the third object 3 can be calculated as 0.8.

[0125] exist Figure 7 In operation S702, when the similarity is determined to be equal to or higher than a preset threshold, the electronic device 100 can determine that the first object and the second object are the same object.

[0126] Reference Figure 8For example, when the similarity between objects is determined to be equal to or higher than a preset threshold (e.g., a similarity of 0.8), the electronic device 100 can determine that the first object 1, the second object 2, and the third object 3 are the same object. However, it is not limited to this.

[0127] According to an embodiment, other image processing techniques may be applied to determine whether objects are the same object or to determine the similarity between objects, so as to determine whether objects included in different images are the same object.

[0128] Figure 7 and Figure 8 Examples have been shown, but this disclosure is not limited thereto.

[0129] Figure 9 This is a flowchart illustrating an example of selecting an image based on object recognition rate according to an embodiment. Figure 10 This is a flowchart of another example of selecting an image based on object recognition rate according to an embodiment. Figure 11 This is a view used to describe an example of selecting an image based on object recognition rate according to an embodiment.

[0130] exist Figure 9 In operation S901, the electronic device 100 can identify objects from multiple images by using a data recognition model.

[0131] According to an embodiment, the electronic device 100 can use a data recognition model trained to identify objects from an image to distinguish them from the vehicle 200. Figure 1 Identify objects in the received image.

[0132] exist Figure 9 In operation S902, the electronic device 100 can select from multiple images an image having an object recognition rate calculated to be equal to or lower than a preset threshold.

[0133] Reference Figure 11 For example, by inputting the first image 400a into the data recognition model, the object recognition rate of the first object 500a being identified as "human" can be 95%, the object recognition rate of the second object 500b in the second image 400b being identified as "human" can be 97%, and the object recognition rate of the third object 500c in the third image 400c being identified as "human" can be 48%.

[0134] For example, refer to Figure 11 The electronic device 100 can select a third image 400c from a plurality of images, which has an object recognition rate calculated to be equal to or lower than a preset threshold (e.g., 50%).

[0135] exist Figure 9In operation S903, the electronic device 100 can determine whether to use the selected image as training data for the data recognition model.

[0136] For example, refer to Figure 11 The electronic device 100 can match the selected third image 400c with another image (e.g., a first image 400a or a second image 400b) captured at the same time point as the third image 400c, and can determine that the third image 400c is used as training data for a data recognition model. The electronic device 100 can determine that a third object 500c included in the third image 400c with a low object recognition rate is the same object as a first object 500a included in the first image 400a captured at the same time point as the third image 400c, and can determine that the result of recognizing the third object 500c and the third image 400c with a low object recognition rate is used as training data for a data recognition model. Therefore, the data recognition model can be modified and improved to have enhanced performance. For example, the data recognition model can include images that would otherwise be excluded based on object recognition rate but actually contain the same object (e.g., a human) as evidenced by images captured by different co-located cameras, as training data.

[0137] Figure 10 This is a flowchart of an exemplary embodiment in which electronic device 100 identifies the object recognition rate of multiple images by receiving information about the object recognition rate of an image from vehicle 200.

[0138] exist Figure 10 In operation S1001, the electronic device 100 can receive information from multiple vehicles regarding the recognition rate of objects identified from multiple images.

[0139] According to an embodiment, vehicle 200 can identify objects from an image by using a data recognition model and can provide information about the object recognition rate of the image to electronic device 100.

[0140] exist Figure 10 In operation S1002, the electronic device 100 can select from multiple images an image whose object recognition rate is calculated to be equal to or lower than a preset threshold. Figure 10 In operation S1003, the electronic device 100 can determine whether to use the selected image as training data for the data recognition model. Operations S1002 and S1003 are related to... Figure 9 The corresponding operations are S902 and S903.

[0141] Figure 9 , Figure 10 and Figure 11 Examples have been shown, but this disclosure is not limited thereto.

[0142] Figure 12This is a block diagram of an electronic device 100 according to an embodiment.

[0143] According to an embodiment, the electronic device 100 may include a communication interface 1500, a memory 1700, and a processor 1300. Figure 12 The electronic device 100 shown includes only the components relevant to this embodiment. Therefore, those skilled in the art will understand that, in addition to... Figure 12 Other general-purpose components besides those shown may also be included in the electronic device 100.

[0144] According to an embodiment, processor 1300 may include at least one processor.

[0145] According to an embodiment, the processor 1300 may execute one or more instructions to receive, via a communication interface 1500, multiple images captured by the multiple vehicles 200 and multiple capture information corresponding to the multiple images.

[0146] Furthermore, according to an embodiment, the processor 1300 may execute one or more instructions to determine whether a first object and a second object in a first image and a second image respectively included in a plurality of images are the same object, wherein the first object and the second object are determined to be located at the same position at the same time point based on a first capture information and a second capture information corresponding to the first image and the second image respectively.

[0147] Furthermore, according to an embodiment, the processor 1300 may execute one or more instructions to calculate similarity by matching the first object with the second object, and when it is determined that the first object and the second object are the same object, the processor 1300 may execute one or more instructions to calculate similarity by matching the first object with the second object, and when it is determined that the similarity value is equal to or higher than a preset threshold ... first object and the second object are the same object.

[0148] Furthermore, according to an embodiment, based on determining that the first object and the second object are the same object, the processor 1300 may execute one or more instructions to modify and improve the data recognition model trained to identify objects from images by using the first image and the second image as training data for the data recognition model.

[0149] In addition, the processor 1300 can execute one or more instructions to identify objects from multiple images using a data recognition model.

[0150] In addition, the processor 1300 may execute one or more instructions to receive information about the recognition rate of objects identified from multiple images from multiple vehicles 200 via the communication interface 1500.

[0151] In addition, the processor 1300 can execute one or more instructions to select from a plurality of images an image having an object recognition rate calculated to be equal to or lower than a preset threshold.

[0152] In addition, the processor 1300 can execute one or more instructions to determine whether to use the selected image as training data for the data recognition model.

[0153] In addition, the processor 1300 can execute one or more instructions to send the modified and improved recognition model to multiple vehicles 200 via the communication interface 1500.

[0154] The communication interface 1500 may include at least one antenna for wireless communication with another electronic device.

[0155] The communication interface 1500, controlled by the processor 1300, can be hardware, such as a transceiver that transmits and receives signals (e.g., radio signals). For example, the processor 1300 can execute a program included in the memory 1700 that enables the communication interface 1500 to transmit and receive wireless signals from the cellular network.

[0156] The communication interface 1500 may include one or more components that enable the electronic device 100 to communicate with external electronic devices. For example, the communication interface 1500 may include a short-range wireless communication interface and a mobile communication interface.

[0157] Short-range wireless communication interfaces may include, but are not limited to, Bluetooth communication interfaces, Bluetooth Low Energy communication interfaces, near-field communication interfaces, WLAN (Wireless Local Area Network) communication interfaces, Zigbee communication interfaces, Infrared Data Association (IrDA) communication interfaces, Wi-Fi Direct (WFD) communication interfaces, Ultra-Wideband (UWB) communication interfaces, Ant+ communication interfaces, etc.

[0158] According to an embodiment, the mobile communication interface can transmit and receive wireless signals with at least one of a mobile communication network, a base station, an external terminal, or a server. Here, the wireless signals can include various types of data.

[0159] According to an embodiment, the memory 1700 may store programs (e.g., instructions, code, etc.) for processing and control of the processor 1300, and may also store data input to or output from the electronic device 100.

[0160] According to an embodiment, memory 1700 may include one or more of the following types of memory: flash memory, hard disk, multimedia card micro or card type (e.g., SD or XD memory). According to an embodiment, memory 1700 may include at least one type of storage medium selected from random access memory (RAM), static RAM (SRAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), programmable ROM (PROM), magnetic storage, magnetic disk, or optical disk.

[0161] Figure 13 This is a block diagram of the processor 1300 according to an embodiment.

[0162] Reference Figure 13 According to an embodiment, the processor 1300 may include a data learner 1310 and a data recognizer 1320.

[0163] The data learner 1310 can learn criteria for determining a situation. The data learner 1310 can learn criteria regarding which data to use to determine a specific situation and how to determine the situation using said data. The data learner 1310 learns the criteria for determining the situation by acquiring the data to be used for learning and applying the acquired data to a data recognition model, which will be described below.

[0164] The data recognizer 1320 can determine a situation based on data. The data recognizer 1320 can identify a situation based on specific data using a trained data recognition model. The data recognizer 1320 can obtain specific data according to pre-defined learning criteria, and can use the obtained data as input to use the data recognition model to determine a specific situation based on that specific data. Furthermore, the output value of the data recognition model using the obtained data as input can be used to modify and improve the data recognition model.

[0165] For example, the results of images captured by vehicle 200 and the identification of objects included in the images can be used to modify and improve data recognition models trained to identify objects from images.

[0166] At least one of the data learner 1310 or data recognizer 1320 may be formed as at least one hardware chip and installed in an electronic device. For example, at least one of the data learner 1310 or data recognizer 1320 may be formed as a dedicated hardware chip for AI, or may be formed as part of a previous general-purpose processor (e.g., a central processing unit (CPU) or application processor) or graphics dedicated processor (e.g., a GPU) and installed in the aforementioned various electronic devices.

[0167] In this configuration, the data learner 1310 and the data recognizer 1320 can be housed in a single electronic device, or each can be housed in a separate electronic device. For example, one of the data learner 1310 and the data recognizer 1320 can be included in an electronic device, while the other can be included in a server. Furthermore, the data learner 1310 and the data recognizer 1320 can communicate with each other via wired or wireless means to provide the data learner 1320 with model information built by the data learner 1310, or to provide the data learner 1310 with input data to the data recognizer 1320 as supplementary training data.

[0168] At least one of the data learner 1310 or data recognizer 1320 may be implemented as a software module. When at least one of the data learner 1310 or data recognizer 1320 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable recording medium. Furthermore, in this case, at least one software module may be provided by an operating system (OS) or a specific application. Optionally, a portion of at least one software module may be provided by the OS, and other portions of said at least one software module may be provided by a specific application.

[0169] Figure 14 This is a block diagram of the data learner 1310 according to an embodiment.

[0170] Reference Figure 14 The data learner 1310 according to one or more embodiments may include a data acquirer 1310-1, a preprocessor 1310-2, a training data selector 1310-3, a model trainer 1310-4, and a model evaluator 1310-5.

[0171] According to an embodiment, the data acquirer 1310-1 can acquire data required to determine a situation. For example, the data acquirer 1310-1 can acquire data required for learning to determine the situation. Furthermore, the data acquirer 1310-1 can receive data from a server.

[0172] For example, data acquisition unit 1310-1 can receive images of the surrounding environment of vehicle 200. The images of the surrounding environment may include multiple images (or image frames). For example, data acquisition unit 1310-1 can receive video via a camera that includes the electronics of data acquisition unit 1310 or an external camera (e.g., closed-circuit television (CCTV) or black box) capable of communicating with the electronics of data acquisition unit 1310. Here, the camera may include one or more image sensors (e.g., front or rear sensors), one or more lenses, an image signal processor (ISP), or a flash (e.g., a light-emitting diode (LED), xenon lamp, etc.).

[0173] Furthermore, for example, the data acquisition device 1310-1 can receive data via an input device of the electronic device (e.g., a microphone, camera, sensor, etc.). Alternatively, the data acquisition device 1310-1 can acquire data via an external device that communicates with the electronic device.

[0174] The preprocessor 1310-2 can preprocess the acquired data so that it can be used for learning to determine the situation. The preprocessor 1310-2 can process the acquired data into a preset format so that the model trainer 1310-4, described below, can use the acquired data for learning to determine the situation.

[0175] According to an embodiment, the training data selector 1310-3 can select the data required for learning from the preprocessed data. The selected data can be provided to the model trainer 1310-4. The training data selector 1310-3 can select the data required for training from the preprocessed data according to preset criteria used to determine the status. Furthermore, the training data selector 1310-3 can select data based on a benchmark set by the model trainer 1310-4 as described below.

[0176] For example, images captured by vehicle 200 and data about the results of identifying objects included in the images can be selected.

[0177] The model trainer 1310-4 can learn criteria on how to determine the state based on training data. Furthermore, the model trainer 1310-4 can learn criteria on which training data to use to determine the state.

[0178] Furthermore, the model trainer 1310-4 can train a data recognition model for determining a situation using training data. According to an embodiment, the data recognition model can be a pre-built model. For example, the data recognition model can be a pre-built model that receives basic training data (e.g., sample images, etc.).

[0179] Data recognition models can be developed by considering the application domain, learning objectives, or computing power of the device. Data recognition models can be, for example, neural network-based models. Examples include CNNs, DNNs, RNNs, and BRDNNs, but are not limited to these.

[0180] According to various embodiments, when multiple pre-established data recognition models exist, the model trainer 1310-4 can determine the data recognition model that has a high correlation between the input training data and the basic training data as the data recognition model for learning. According to embodiments, the basic training data can be pre-classified into multiple data types, and data recognition models can be pre-established for each data type. For example, the basic training data can be pre-classified based on various benchmarks, such as the region where the training data was generated, the time when the training data was generated, the size of the training data, the type of training data, the generator of the training data, and the type of objects in the training data.

[0181] Furthermore, the model trainer 1310-4 can train the data recognition model using, for example, learning algorithms (such as error backpropagation and gradient descent).

[0182] Furthermore, according to an embodiment, model trainer 1310-4 can, for example, train the data recognition model through supervised learning by using training data as input values. Furthermore, according to an embodiment, model trainer 1310-4 can, for example, train the data recognition model through unsupervised learning, thereby discovering criteria for determining the state by self-learning the state and determining the required type of data without specific instructions. Furthermore, according to an embodiment, model trainer 1310-4 can, for example, train the data recognition model through reinforcement learning, thereby using feedback on whether the results of the learned state determination are correct.

[0183] Furthermore, according to an embodiment, when the data recognition model is trained, the model trainer 1310-4 can store the trained data recognition model. In this case, the model trainer 1310-4 can store the trained data recognition model on the memory of an electronic device including the data recognizer 1320. Optionally, according to an embodiment, the model trainer 1310-4 can store the trained data recognition model on the memory of an electronic device including the data recognizer 1320, which will be described below. Optionally, according to an embodiment, the model trainer 1310-4 can store the trained data recognition model on the memory of a server connected to the electronic device via a wired or wireless network.

[0184] In this context, according to an embodiment, the memory storing the trained data recognition model may also store, for example, commands or data related to at least another component of the electronic device. Furthermore, the memory may store software and / or programs. Programs may include, for example, kernels, middleware, application programming interfaces (APIs), and / or applications (or "applications").

[0185] According to an embodiment, the model evaluator 1310-5 can input evaluation data into the data recognition model, and when the recognition result output from the evaluation data does not meet a preset benchmark, the model trainer 1310-4 can be made to relearn. In this case, the evaluation data can be data preset for evaluating the data recognition model.

[0186] For example, according to an embodiment, when the number of incorrectly identified evaluation data or the ratio of incorrectly identified evaluation data in the recognition results of the trained data recognition model for the evaluation data exceeds a preset threshold, the model evaluator 1310-5 can assess that the recognition results do not meet the preset benchmark. For example, according to an embodiment, when the preset benchmark is defined as 2%, and the trained data recognition model outputs incorrect recognition results for more than 20 (20) of the total 1,000 (1,000) evaluation data, the model evaluator 1310-5 can assess that the trained data recognition model is unsuitable.

[0187] Furthermore, according to the embodiment, when multiple trained data recognition models exist, the model evaluator 1310-5 can evaluate whether each trained data recognition model meets a preset benchmark, and can determine the model that meets the preset benchmark as the final data recognition model. In this case, when multiple models meet the preset benchmark, the model evaluator 1310-5 can determine a preset model or a preset number of models as the final data recognition model based on the order of the evaluation scores.

[0188] Additionally, at least one of the data acquirer 1310-1, preprocessor 1310-2, training data selector 1310-3, model trainer 1310-4, or model evaluator 1310-5 in the data learner 1310 can be formed as at least one hardware chip and can be installed in an electronic device. For example, at least one of the data acquirer 1310-1, preprocessor 1310-2, training data selector 1310-3, model trainer 1310-4, or model evaluator 1310-5 can be formed as a dedicated hardware chip for AI, or can be formed as part of a previous general-purpose processor (e.g., CPU or application processor) or graphics dedicated processor (e.g., GPU), and can be installed in the aforementioned various electronic devices.

[0189] Furthermore, the data acquirer 1310-1, preprocessor 1310-2, training data selector 1310-3, model trainer 1310-4, and model evaluator 1310-5 can be installed in a single electronic device, or each can be installed in a separate electronic device. For example, some of the data acquirer 1310-1, preprocessor 1310-2, training data selector 1310-3, model trainer 1310-4, and model evaluator 1310-5 can be included in an electronic device, and others can be included in a server.

[0190] Furthermore, at least one of the data acquirer 1310-1, preprocessor 1310-2, training data selector 1310-3, model trainer 1310-4, or model evaluator 1310-5 may be implemented as a software module. When at least one of the data acquirer 1310-1, preprocessor 1310-2, training data selector 1310-3, model trainer 1310-4, or model evaluator 1310-5 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable recording medium. Furthermore, in this case, at least one software module may be provided by an operating system (OS) or a specific application. Optionally, a portion of at least one software module may be provided by an OS, and other portions of said at least one software module may be provided by a specific application.

[0191] Figure 15 This is a block diagram of the data recognizer 1320 according to an embodiment.

[0192] Reference Figure 15 The data recognizer 1320 according to one or more embodiments may include a data acquirer 1320-1, a preprocessor 1320-2, a recognition data selector 1320-3, a recognition result provider 1320-4, and a model modifier and improver 1320-5.

[0193] The data acquisition unit 1320-1 can acquire the data required for situation determination, and the preprocessor 1320-2 can preprocess the acquired data for situation determination. The preprocessor 1320-2 can process the acquired data into a preset format so that the identification result provider 1320-4, which will be described below, can use the acquired data to determine the situation.

[0194] The identification data selector 1320-3 can select the data needed to determine the status from the preprocessed data. The selected data can be provided to the identification result provider 1320-4. The identification data selector 1320-3 can select part or all of the preprocessed data according to preset criteria used to determine the status. In addition, the identification data selector 1320-3 can select data according to preset criteria based on the learning of the model trainer 1310-4, which will be described below.

[0195] The recognition result provider 1320-4 can determine the status by applying the selected data to the data recognition model. The recognition result provider 1320-4 can provide recognition results according to the purpose of data recognition. The recognition result provider 1320-4 can apply the selected data to the data recognition model by using the data selected by the recognition data selector 1320-3 as input values. Furthermore, the recognition result can be determined by the data recognition model.

[0196] According to embodiments, the recognition results can be provided in the form of text, sound, video, image, or instructions (e.g., application execution instructions, module function execution instructions, etc.).

[0197] The model modifier and improver 1320-5 allows modification and improvement of the data recognition model based on the evaluation of the recognition results provided by the recognition result provider 1320-4. For example, the model modifier and improver 1320-5 can provide the recognition results provided by the recognition result provider 1320-4 to the model trainer 1310-4, allowing the model trainer 1310-4 to modify and improve the data recognition model by performing training.

[0198] Additionally, at least one of the data acquirer 1320-1, preprocessor 1320-2, recognition data selector 1320-3, recognition result provider 1320-4, or model modifier and improver 1320-5 in the data recognizer 1320 can be formed in the form of at least one hardware chip and can be installed in an electronic device. For example, at least one of the data acquirer 1320-1, preprocessor 1320-2, recognition data selector 1320-3, recognition result provider 1320-4, or model modifier and improver 1320-5 can be formed in the form of a dedicated hardware chip for AI, or can be formed as part of a previous general-purpose processor (e.g., CPU or application processor) or graphics dedicated processor (e.g., GPU), and can be installed in the aforementioned various electronic devices.

[0199] Furthermore, the data acquirer 1320-1, preprocessor 1320-2, identification data selector 1320-3, identification result provider 1320-4, and model modifier and improver 1320-5 can be installed in a single electronic device, or each can be installed in a separate electronic device. For example, some of the data acquirer 1320-1, preprocessor 1320-2, identification data selector 1320-3, identification result provider 1320-4, and model modifier and improver 1320-5 can be included in an electronic device, while others can be included in a server.

[0200] Furthermore, at least one of the data acquirer 1320-1, preprocessor 1320-2, identification data selector 1320-3, identification result provider 1320-4, or model modifier and improver 1320-5 can be implemented as a software module. When at least one of the data acquirer 1320-1, preprocessor 1320-2, identification data selector 1320-3, identification result provider 1320-4, or model modifier and improver 1320-5 is implemented as a software module (or a program module including instructions), the software module can be stored in a non-transitory computer-readable recording medium. Furthermore, in this case, at least one software module can be provided by an OS or a specific application. Optionally, a portion of at least one software module can be provided by an OS, and other portions of said at least one software module can be provided by a specific application.

[0201] Figure 16 This is a view illustrating examples of apparatus and servers, according to one or more embodiments, learning and recognizing data by interacting with each other.

[0202] Figure 16 This is a view of an example of how an electronic device 100 and an external server 2000 learn and identify data by interacting with each other, according to an embodiment.

[0203] Reference Figure 16 The external server 2000 can learn the criteria used to determine the situation, and the electronic device 100 can determine the situation based on the learning results of the external server 2000.

[0204] In this scenario, the model trainer 2340 on the external server 2000 can execute. Figure 14 The data learner 1310-4 shown has the following functionality. The model trainer 2340 of the external server 2000 can learn the criteria for determining a specific situation by which data to use and how to use said data to determine the situation. The model trainer 2340 can learn the criteria for determining the situation by obtaining the data that will be used for learning and applying the obtained data to the data recognition model described below.

[0205] Furthermore, the identification result provider 1320-4 of the electronic device 100 can determine the status by applying the data selected by the identification data selector 1320-3 to the data identification model generated by the external server 2000. For example, the identification result provider 1320-4 can send the data selected by the identification data selector 1320-3 to the external server 2000, and can request the external server 2000 to determine the status by applying the data selected by the identification data selector 1320-3 to the data identification model. Additionally, the identification result provider 1320-4 can receive information from the external server 2000 regarding the status determined by the external server 2000.

[0206] For example, electronic device 100 can send an image frame to external server 2000 and request external server 2000 to identify an object by applying the image frame to a data recognition model. Furthermore, electronic device 100 can receive information from external server 2000 regarding the identification of the object determined by external server 2000.

[0207] Optionally, the identification result provider 1320-4 of the electronic device 100 may receive an identification model generated by the external server 2000, and may determine the status by using the received identification model. In this case, the identification result provider 1320-4 may determine the status by applying the data selected by the identification data selector 1320-3 to the data identification model received from the external server 2000.

[0208] For example, electronic device 100 can identify objects by applying image frames to a data recognition model received from external server 2000.

[0209] Figure 17 This is a block diagram of vehicle 200 according to an embodiment.

[0210] The vehicle 200 may include a detection unit 110, a processor 120, an output interface 130, a memory (or storage device) 140, an input interface 150, and a communication interface 160.

[0211] According to an embodiment, the detection unit 110 may include multiple sensors configured to detect information about the surrounding environment in which the vehicle 200 is located, and may include one or more actuators configured to change the position and / or orientation of the sensors. For example, the detection unit 110 may include a GPS receiver 224, an inertial measurement unit (IMU) 225, a RADAR sensor 226, a LIDAR sensor 227, an image sensor 228, and a ranging sensor 230. According to an embodiment, the detection unit 110 may include at least one of a temperature / humidity sensor 232, an infrared sensor 233, an atmospheric sensor 235, a proximity sensor 236, or an RGB sensor (illuminance sensor) 237, but is not limited thereto. The function of each of the sensors can be intuitively inferred by those skilled in the art from their names, and therefore a detailed description thereof will be omitted.

[0212] In addition, the detection unit 110 may include a motion sensor 238 capable of detecting motion. The motion sensor 238 may include one or more of the following: a magnetic sensor 229, an accelerometer 230, or a gyroscope sensor 234.

[0213] GPS receiver 224 can be configured to estimate the geographic location of vehicle 200. That is, GPS receiver 224 may include a transceiver configured to estimate the position of vehicle 200 relative to the Earth.

[0214] According to an embodiment, the IMU 225 may be a combination of sensors configured to detect changes in the position and orientation of the vehicle 200 based on inertial acceleration. For example, the combination of sensors may include an accelerometer and a gyroscope.

[0215] According to an embodiment, the RADAR sensor 226 can be configured to sense objects in the driving environment where the vehicle 200 is located using radio signals. Furthermore, the RADAR sensor 226 can be configured to detect the speed and / or direction of the objects.

[0216] According to an embodiment, LIDAR sensor 227 may be configured to detect objects in the driving environment in which vehicle 200 is located using a laser beam. More specifically, LIDAR sensor 227 may include a laser source and / or laser scanner configured to emit a laser beam, and a sensor configured to detect reflections of the laser beam. LIDAR 227 may be configured to operate in coherent (e.g., using heterodyne detection) or incoherent detection modes. According to an embodiment, image sensor 228 may include a still camera or video camera configured to record the external environment of vehicle 200. For example, image sensor 228 may include multiple cameras, and the multiple cameras may be arranged at multiple locations inside or outside vehicle 200.

[0217] According to an embodiment, the ranging sensor 230 can estimate the position of the vehicle 200 and measure the distance traveled. For example, the ranging sensor 230 can measure the change in position of the vehicle 200 by using the number of rotations of the wheels of the vehicle 200.

[0218] According to an embodiment, memory 140 may include at least one of the following: a disk drive, an optical disk drive, or flash memory. Optionally, memory 140 may include a portable universal serial bus (USB) data storage device. Memory 140 may store system software for performing examples of one or more embodiments related to this disclosure. The system software for performing the examples related to this disclosure may be stored in a portable storage medium.

[0219] The communication interface 160 may include at least one antenna for wireless communication with another device. For example, the communication interface 160 may be used to wirelessly communicate with a cellular network or other wireless protocols and systems via Wi-Fi or Bluetooth. The communication interface 160, controlled by the processor 120, may transmit and receive radio signals. For example, the processor 120 may execute a program included in the memory 140 such that the communication interface 160 can transmit and receive wireless signals from the cellular network.

[0220] Input interface 150 represents a device for a user to input data for controlling vehicle 200. For example, input interface 150 may include one or more of the following: a keyboard, a dome switch, a touchpad (using capacitive touch, piezoresistive touch, infrared detection, surface ultrasonic conduction, overall tension measurement, piezoelectric effect, etc.), a scroll wheel, a microswitch, etc., but is not limited thereto. Furthermore, according to an embodiment, input interface 150 may include a microphone and may be configured to receive audio signals (e.g., voice commands) from occupants of vehicle 200.

[0221] The output interface 130 can output audio or video signals, and the output interface 130 may include a display 281 and a sound output interface 282.

[0222] Display 2811 may include at least one of liquid crystal display, thin-film transistor liquid crystal display, organic light-emitting diode, flexible display, three-dimensional (3D) display, or electrophoretic display. Output interface 130 may include two or more displays 281.

[0223] The audio output interface 282 can output audio data received from the communication interface 160 or stored in the memory 140. The audio output interface 282 may include a speaker, a buzzer, etc.

[0224] The input interface 150 and the output interface 130 may include a network interface and may be implemented as a touch screen.

[0225] The processor 120 can execute programs stored in the memory 140 to control the detection unit 110, communication interface 160, input interface 150, memory 140 and output interface 130 in general.

[0226] Electronic devices according to embodiments described herein may include a processor, memory for storing and executing program data, permanent memory such as a disk drive, a communication port for handling communication with external devices, and a user interface device, etc. Methods are implemented as software modules or may be stored on a computer-readable recording medium as computer-readable code or program instructions executable by a processor. Examples of computer-readable recording media may include magnetic storage media (e.g., read-only memory (ROM), random access memory (RAM), floppy disk, hard disk, etc.), optical reading media (e.g., CD-ROM, DVD, etc.). Computer-readable recording media may be distributed across computer systems connected to a network, and computer-readable code may be stored and executed in a distributed manner. The medium may be computer-readable, stored in memory, and executed by a processor.

[0227] Embodiments may be designated as functional block components and various processing operations. Functional blocks may be implemented as various numbers of hardware and / or software components that perform specific functions. For example, embodiments may implement direct circuit components, such as memory, processing circuitry, logic circuitry, lookup tables, etc., that can perform various functions under the control of one or more microprocessors or other control devices. Components of this disclosure may be implemented by software programming or software components. Similarly, embodiments may include various algorithms implemented by combinations of data structures, processes, routines, or other programming components, and may be implemented by programming or scripting languages ​​(such as C, C++, Java, assemblers, etc.). Functional aspects may be implemented by algorithms executed by one or more processors. Furthermore, embodiments may implement related techniques for electronic environment setup, signal processing, and / or data processing. Terms such as “mechanism,” “element,” “unit,” etc., are used extensively and are not limited to mechanical and physical components. These terms may refer to a series of software routines associated with processors, etc.

[0228] Specific implementations have been described as examples in the embodiments, and the scope of the embodiments is not limited to any particular method. For the sake of brevity, other functional aspects of electronic components, control systems, software, and systems according to the related art may not be described.

[0229] While embodiments have been described in detail, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope of this disclosure as defined by the appended claims. Therefore, the above embodiments should be interpreted as examples and not as limiting in any way. For example, each of the components described as a single unit may be executed in a distributed manner, and similarly, components described as distributed may be executed in a combined manner.

[0230] When the scope of the embodiments is not limited by the claims, all examples or example terms (e.g., etc.) used in the embodiments are for the purpose of describing the embodiments in detail and are not intended to limit the scope of the embodiments.

[0231] Furthermore, unless a component is specifically described using expressions such as “necessary” or “important”, it may not be a necessary component for applying this disclosure.

[0232] Those skilled in the art will understand that the embodiments may be implemented in modified form without departing from the spirit and scope of this disclosure.

[0233] Because this disclosure allows for various changes and numerous embodiments, it is not limited to the specific embodiments described herein, and it will be understood that all changes, equivalents, and alternatives that do not depart from the spirit and technical scope of this disclosure are included herein. Therefore, the embodiments described herein should be understood as illustrative in all respects and should not be construed as limiting.

[0234] The scope of this disclosure is defined by the appended claims rather than the detailed description above, and it will be understood that all modifications or variations derived from the meaning and scope of the claims and their equivalents are included within the scope of this disclosure.

[0235] The terms described in this disclosure (such as “unit”, “module”, etc.) refer to a unit that can be implemented as hardware or software or a combination of hardware and software to process at least one function or operation.

[0236] "Units" and "modules" can be stored in the storage medium to be addressed and can be implemented as programs that can be executed by a processor.

[0237] For example, "unit" and "module" can refer to components such as software components, object-oriented software components, class components, and task components, and can include processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables.

[0238] In this disclosure, the description “A may include one of a1, a2 and a3” can broadly indicate that examples that can be included in element A include a1, a2 or a3.

[0239] The description should not be interpreted as limiting the examples that can be included in element A to the meaning of a1, a2, and a3. Therefore, as examples included in element A, they should not be interpreted as excluding elements other than a1, a2, and a3.

[0240] Furthermore, this disclosure indicates that element A may include a1, a2, or a3. This disclosure does not imply that the elements included in element A must be selected from a specific set of elements. That is, the description should not be construed as implying that a1, a2, or a3 must be selected from a set including a1, a2, and a3 to be included in element A.

[0241] Furthermore, in this disclosure, the description "at least one of a1, a2, and a3" means one of "a1", "a2", "a3", "a1 and a2", "a1 and a3", "a2 and a3", and "a1, a2, and a3". Therefore, it should be noted that unless explicitly described as "at least one of a1, at least one of a2, and at least one of a3", the description "at least one of a1, a2, and / or a3" is not to be interpreted as "at least one of a1", "at least one of a2", and "at least one of a3".

Claims

1. An electronic device for assisting vehicle driving, comprising: Communication interface; Memory, which stores one or more instructions; as well as The processor is configured to execute one or more instructions stored in the memory to perform the following operations: Multiple images of a real-world scene are obtained through the communication interface, wherein the multiple images include a first image captured by a first camera or a first image sensor and a second image captured by a second camera or a second image sensor; Determining whether a first object included in a first image and a second object included in a second image are the same object, wherein the similarity value between the first object and the second object is determined to be equal to or higher than a preset threshold, and the first object and the second object are determined to be the same object based on the first capture time and location information corresponding to the first image and the second capture time and location information corresponding to the second image corresponding to the same time point and the same location, and wherein the first capture time and location information includes information about the time point of capture of the first image and information about the location of capture of the first image, and the second capture time and location information includes information about the time point of capture of the second image and information about the location of capture of the second image; Based on the determination that the first object and the second object are the same object, the neural network data recognition model is trained to recognize the first object and the second object as the same object in both the first image and the second image by using the first image and the second image as training data for the neural network data recognition model. From the plurality of images, images whose object recognition rate is calculated to be equal to or lower than a preset threshold are selected; and Determine whether to use the selected images as training data for the neural network data recognition model.

2. The electronic device as claimed in claim 1, wherein, The processor is also configured to execute one or more instructions to perform the following operations: based on determining that the first object and the second object are not the same object, based on the object recognition rate of the first image or the second image being less than a predefined threshold recognition rate, excluding the first image or the second image from the training data, and training the neural network data recognition model to recognize the object based on the training data.

3. The electronic device as claimed in claim 1, wherein, The processor is also configured to execute one or more instructions to perform the following operations: obtain, via the communication interface, a first capture time and location information corresponding to the first image and a second capture time and location information corresponding to the second image.

4. The electronic device as claimed in claim 1, wherein, The processor is also configured to execute one or more instructions to determine the similarity value between the first object and the second object by: The similarity value is calculated by matching the first object with the second object.

5. The electronic device as claimed in claim 1, wherein, The processor is also configured to execute one or more instructions to perform the following operations: The object is identified from the plurality of images by using the neural network data recognition model.

6. The electronic device as claimed in claim 1, wherein, The processor is also configured to execute one or more instructions to perform the following operations: Information about the object recognition rate, which is obtained from multiple vehicles via the communication interface, regarding the object recognition rate in each of the multiple images using the neural network data recognition model.

7. The electronic device as claimed in claim 1, wherein, The processor is also configured to execute one or more instructions to perform the following operation: sending a trained neural network data recognition model to a plurality of vehicles via the communication interface, wherein the plurality of vehicles includes a first vehicle equipped with a first camera or a first image sensor and a second vehicle equipped with a second camera or a second image sensor.

8. A method for assisting vehicle driving, comprising: Multiple images of a real-world scene are obtained through a communication interface, wherein the multiple images include a first image captured by a first camera or a first image sensor and a second image captured by a second camera or a second image sensor; Determine whether a first object and a second object included in a first image and a second image are the same object, wherein the similarity value between the first object and the second object is determined to be equal to or higher than a preset threshold, and the first object and the second object are determined to be the same object based on the first capture time and location information corresponding to the first image and the second capture time and location information corresponding to the second image corresponding to the same time point and the same location, wherein the first capture time and location information includes information about the time point of capture of the first image and information about the location of capture of the first image, and the second capture time and location information includes information about the time point of capture of the second image and information about the location of capture of the second image; Based on the determination that the first object and the second object are the same object, the neural network data recognition model is trained to recognize the first object and the second object as the same object in both the first image and the second image by using the first image and the second image as training data for the neural network data recognition model. From the plurality of images, images whose object recognition rate is calculated to be equal to or lower than a preset threshold are selected; and Determine whether to use the selected images as training data for the neural network data recognition model.

9. The method of claim 8, further comprising: Based on the determination that the first object and the second object are not the same object, the first image or the second image is excluded from the training data because the object recognition rate of the first image or the second image is less than a predefined threshold recognition rate, and the neural network data recognition model is trained based on the training data to identify the object.

10. The method of claim 8, further comprising: The first capture time and location information and the second capture time and location information corresponding to the first image and the second image are obtained through the communication interface.

11. The method of claim 8, wherein, The similarity value between the first object and the second object is determined by the following operation: The similarity value is calculated by matching the first object with the second object.

12. The method of claim 8, further comprising: The object is identified from the plurality of images by using the neural network data recognition model.

13. The method of claim 8, further comprising: Information regarding the object recognition rate, obtained from multiple vehicles via the communication interface, is received about the object recognition rate obtained from each of the multiple images using the neural network data recognition model.

14. The method of claim 8, further comprising: The trained neural network data recognition model is sent to multiple vehicles through the communication interface, wherein the multiple vehicles include a first vehicle equipped with a first camera or a first image sensor, and a second vehicle equipped with a second camera or a second image sensor.

15. A non-transitory computer-readable recording medium having a program for performing the method of claim 8 on a computer.

Images